How weapon is attached to a rigged animation

[slin]

Hello,

I am using this troll model for learning how to implement GPU instancing on rigged model: Troll - Download Free 3D model by SJunior3d (@sjunior3d) [bceb1cb] - Sketchfab

Generally, the troll animation works fine. I also made the weapon moves relative to its parent bone. But as shown in the screenshot, I can’t figure out how to calculate the transformation matrix to make the weapon attached to the right hand.

The node hierarchy is shown below:

This model actually has multiple meshes for main body hair, eyeballs, teeth etc. These meshes are node10, node12, node14, etc in the screenshot. Their positions are all correct. But the weapon mesh (mace_low_weapon_mat_0) is not. In the static pose opened in blender, the mace is at side of the model, not attached to the hand.

That’s why I want to know how the weapon is transformed to the hand when animation started in BJS. It would be great if you can briefly explain the general ideal about how to do it, or point me to which files I should read in BJS source code (so far I have no luck with it ).

[gryff]

This the kind of thing you are looking for :

Sword

Click on the simple sword to attach and remove, click on the figure for animation.

Below is a picture of the blender set up. The “sword” has its origin at 0.0.0 and I played with its rotation until it ended fitting into the hand nicely. Notice that the sword origin is not at the end of the sword but slightly in - so that the "handle will fit into the hand.

Feel free to steal my code for the interaction though it is kind of old.

Stay Safe, gryff

[gbz]

Hey @slin, if your troll model has (1, 1, 1) scaling, attachToBone is a good way to attach your weapon to the troll’s hand bone. (Without (1, 1, 1) scaling, attachToBone seems to change the weapon’s scale to that of the parent, which can be tricky to work with, especially if you plan on the troll picking up and dropping weapons)

The Playgrounds in ArcRotateCamera's Radius affects the Rotation/Position of attachToBone Meshes may be a good reference

However, it sounds like you already made the weapon move relative to its parent bone. I’m guessing that you achieved this through setting the position and rotation(Quaternion) of the weapon to that of the parent bone before every frame render? If so, all you would need is to apply an additional position and rotation offset to move the weapon inside the troll’s hand

Currently, I’m doing something like this that works fine, though there may be a more efficient way to accomplish this:

// You need to find (perhaps by trial and error) a, b, c, x, y, z such
// that the weapon is placed directly in the troll's hand. I'm currently
// using dat.gui to tune these values in real-time to quickly find these
const offsetPosition = new BABYLON.Vector3(a, b, c);
const offsetRotation = new BABYLON.Vector3(x, y, z);

// Update location of weapon before every frame render
scene.registerBeforeRender(() => {
    // Find absolute position and rotation of the trollMesh hand
    trollMesh.computeWorldMatrix(true);
    const scale = new BABYLON.Vector3();
    const rotation = new BABYLON.Quaternion();
    const position = new BABYLON.Vector3();
    const matrix = trollMeshSkeleton.bones[handBoneIndex].getWorldMatrix();
    const matrix2 = trollMesh.getWorldMatrix();
    const matrix3 = matrix.multiply(matrix2);
    matrix3.decompose(scale, rotation, position);

    // Set weaponMesh's absolute position and rotation to those of the
    // trollMesh hand
    weaponMesh.setAbsolutePosition(position);
    weaponMesh.rotationQuaternion = rotation;

    // Apply the position and rotation offsets found above (a, b, c, x, y, z)
    const rotationMatrix = new BABYLON.Matrix();
    rotation.toRotationMatrix(rotationMatrix);
    const translation = BABYLON.Vector3.TransformCoordinates(offsetPosition, rotationMatrix);
    weaponMesh.position.addInPlace(translation);
    weaponMesh.rotationQuaternion.multiplyInPlace(offsetRotation.toQuaternion());
});

The above code snippet allows for arbitrary initial scaling of the troll and weapon

[slin]

Hi @gbz,

Thanks for mentioning the scaling. I set the model back to scale (1,1,1) and the weapon is closer but still not quite right. I also tried to use a cube instead of the mace for testing.

Initially, I set the cube’s location as follows. The cube weapons do show up at the right place. I also tried to visualize the bones of the default pose on the side. The bone I want to attach weapon is highlighted with a larger sphere.

    if (bone.id === 'cave_troll_RArmPalm_052') {
      targetBoneMatrix = bone.getAbsoluteTransform();
    }

   const weaponTemplate = BABYLON.MeshBuilder.CreateBox('weapon', { size: 50 }, scene);
   targetBoneMatrix.decompose(weaponTemplate.scaling, weaponTemplate.rotationQuaternion, weaponTemplate.position);

The skin animation is driven by ‘#include’ in pbr vertex shader code. For GPU instanced animation, I replaced the default bonesVertex with a customized implementation that can accept baked animation as a uniform. And depending on the offset of the current frame and the influence bone, the corresponding transformation matrix is retrieved to calculate influence and execute:

finalWorld = finalWorld * influence;

I was thinking since I successfully applied the transformation matrix of the target bone to the skins around the bone. The same transformation can also be applied to animate the weapon. So I attached the material for the mace to the box weapon. And I have an instanced buffer mycontrol of type vec4. If mycontrol.y == 1.0, the following transformation is used instead of the default finalWorld * influence.

This code basically retrieves the boneSampler of the target bone for the current frame. And assuming all the vertices on the weapon are only influenced by the target bone. The result is the correct animation of the weapon. bakedAnimMatrices is the texture that store all the transformation matrix for all the bones and all the frames, bakedAnimFrames is total number of frames, animIndex is current frame.

But the position of the weapon is shifted away from the target bone position, as shown in the screenshot in my original question. I think there could be some gaps in my knowledge and my assumption to transform weapon the same way as transform skins influenced by the target bone is not correct.

if (mycontrol.y == 1.0) {
	float animIndex = mycontrol.x;
            // num of bones + 1 is used to store the transform matrix for the target bone
	float indexMace = 123.0;
            float numOfBones = 123.0;
            float myBoneTextureWidth = (numOfBones + 1.0) * 4.0;
            float dx = 1.0 / myBoneTextureWidth;
            float offsetY = animIndex / bakedAnimFrames;

	float offsetMace = indexMace * 4.0;
            vec4 mace0 = texture2D(bakedAnimMatrices, vec2(dx * (offsetMace + 0.5), offsetY));
	vec4 mace1 = texture2D(bakedAnimMatrices, vec2(dx * (offsetMace + 1.5), offsetY));
            vec4 mace2 = texture2D(bakedAnimMatrices, vec2(dx * (offsetMace + 2.5), offsetY));
            vec4 mace3 = texture2D(bakedAnimMatrices, vec2(dx * (offsetMace + 3.5), offsetY));
            mat4 maceMatrix = mat4(mace0, mace1, mace2, mace3);

            finalWorld = finalWorld * maceMatrix;
}

[gbz]

@slin Whoa, your custom implementation of GPU animation instancing sounds amazing! I wonder how difficult it is to generalize this approach, but it would be awesome if this became a feature of Babylon! @Deltakosh @Evgeni_Popov are far more experienced than I am in these areas, and hopefully they can share insights on GPU animation instancing and the weapon positioning issue

[Evgeni_Popov]

That’s hard to say without a repro as it is using a custom shader code and because the original file does have the mace correctly positionned in the hand of the model…

[slin]

@gbz @Evgeni_Popov

Thanks for helping. I will try to reproduce the issue in a PG.

[nasimiasl]

hey have a gpu picking system initialized in shader builder
that pic a pixel so you can pic any animation ( with or without shader vertx customaize)

https://www.babylonjs-playground.com/#JFW56V#18

it is 1 week task for append this system work on any vertex definition

[Raggar]

The way I attach weapons to characters using GPU instancing, is:

• Making sure both character and weapon have the same skeleton.
• Making sure both character and weapon have the same pivot point. This way I just position the weapon at the character’s position and either parent the weapon to the character, or update the weapon’s position and rotation every frame.
• Letting both have the same material/shader, so when you change the attributes of both meshes to your new animation frame, it simply works.

I think your issue might be with having different pivot points/transforms for both meshes. You can either manually figure out where this pivot should be, by changing the position of the weapon after you have parented it to the character, or by, and this is easier for me, give both of them the same pivot point before exporting.

[slin]

Hi guys,

Thank you very much for your inputs. I have made some progress last night. Now I am able to:

• Attach a separate mesh outside model (e.g. a box) to any bone I specify.
• Attach the original mace to be correctly hold in the hand of the troll model.

But I haven’t figured out how to attach the box to be hold in the hand like the mace. I think the problem is the box created by MeshBuilder is at the origin by default in the world space. But the weapon is at the side of the model in the static pose, not at world space origin. If I use the same approach for attaching mace, the box is shifted away. This could be the different pivot points @Raggar mentioned. I want to figure this out to have all the scenarios working, then I will share more details of how I did it.

Sadly I don’t work on BJS for living. Since holiday is over, I will not be able to reply back very soon. But as soon as I make the last scenario work, I will be back.

[Evgeni_Popov]

What you can do is shifting the coordinates of the box themselves. You can do this by setting a translation to your box and by calling box.bakeCurrentTransformIntoVertices(). In effect, that changes the origin of the box to be the translation you set before baking.

[slin]

Hello,

I have all the scenarios working.

1. Attach weapon to a bone.

The problem was the baked animation stores invertedBindingMatrix * worldMatrix for each bone. But the animation of a weapon only requires worldMatrix. So I included the worldMatrix of the bone that I want to attach the weapon. In the shader code, I extract the world matrix for this bone and call:

finalWorld = finalWorld * boneWorldMatrix;

2. Attach the original weapon included in the 3D model

In this model, the ‘leaf’ node for that right hand bone is ‘cave_troll_RAm22_052’. This bone has two transform nodes extended from it: mace_low and node93. The weapon mesh ‘mace_low_weapon_mat_0’ is attached to node93. As shown in the BJS inspector:

Inspired by 1, I tried to compute the worldMatrix of node93 as:

worldMatrixNode93 = localMatrixNode93 * localMatrixMaceLow * boneWorldMatrix;

And then I bake world matrix worldMatrixNode93 instead of boneWorldMatrix into the anim texture. In the shader I call:

finalWorld = finalWorld * worldMatrixNode93;

This gives the mace in the correct hand holding position in the animation.

3. Attach any other external weapon to the troll.

After some trial and error, it seems node93 is responsible for transforming the built-in mace from its location to origin. And transform node mace_low transforms to the right hand holding position. So for any other external mesh loaded at origin, I can calculate the worldMatrix of the correct hand holding position at:

worldMatrixMace = localMatrixMaceLow * boneWorldMatrix;

And in the shader:

finalWorld = finalWorld * worldMatrixNode93;

I attached an mp4 file to show the result of scenario 3: attach a box to the troll. Hope it is supported. I haven’t implemented interpolating the key frames yet, so the animation is not as smooth as it should be.