How to set position, rotation, scale of bones

Questions
,
1

Hello there,

As part of this: How to animate a model via a websocket - #11 by r0bert0.

I am trying to set properties e.g. rotation of the head, position of the head etc. of an imported glTF file directly via the transformNode() linked to the bones of the glTF file.

I have created a Playground to try to illustrate the various issues I am having here: https://playground.babylonjs.com/#FF3FD0#4 (Relevant code starts at line 37)

The animData object represents a subset of one frame of animation Data I am receiving via a WebSocket. For now I only try to set the properties relating to the animation of the head.

The animData represents absolute Transformations of the skeleton. Therefore, in my complete code I do a preorder traversal of the bones of the skeleton, which I then use the set the properties for the parents before their children. The array of 6 values for each bone is in order: eulerAngleX, eulerAngleY, eulerAngleZ, translationX, translationY, translationZ, scale.

I have simpilfied the PG example by ordering the bones in the correct way already so as to adhere to their hierarchy.

I then set the properties as shown in the PG.

Now this leads to a completely wrong orientation of my head. My main question is why is the orientation completely wrong? More specifically:

  1. Is my approach here correct, assuming I am getting absolute Transformations?

  2. Is this somehow a side effect of the .glTF importer?

Thanks for any help provided.

Tagging the bone doctor: @Deltakosh :wink:

2

One good way to learn how to do it is to attach a mesh to a node and then control that mesh (much easier and you don’t have to mess with the matrices ;))
Bones and Skeletons | Babylon.js Documentation (babylonjs.com)

3

But I am not dealing with matrices as above? Or would I have to? Also aren’t the bones of the skeletons already attached to the meshes of my .glTF file?

4

The attachToBone let you create a dummy mesh that will force the bone to follow.
Then the system will generate a correct bone matrix out of the dummy world matrix

Because else you have to provide a bone transform which is local. Not global.

5

Ah I didn’t realize I needed to provide local bone transforms. So an easier strategy is to attach the meshes of my glTF file to the correct bones and then do the transforms as above?

Just out of curiosity, how would I what would be the approach to get local bone transforms?

6

If you want to do it manually you can check how this code works:
Babylon.js/bone.ts at master · BabylonJS/Babylon.js (github.com)

7

Could you elaborate a bit on the attachToBone and how it would work for the .glTF file? My .glTF file already has meshes (i.e. TransformNodes) attached to the bones, so I don’t see why I need to attach meshes again. I access the transformNodes for each of the bones first and then apply my transform to the nodes.

I changed my code a bit, for each TransformNode linked to the bone I set the transforms by accessing the attributes: .absoluteRotationQuaternion, .absoluteScaling, .setAbsolutePosition, this seems to fix the head orientation and transforms, but the arms and fingers are still messed up. Judging from the documentation for the TransformNode absoluteRotationQuaternion, .absoluteScaling, can be used with absolute transforms in World coordinate space, so why do I still need to provide local transforms?

8

I tried doing this by creating the dummy mesh for the head of the .glTF and then setting the position of the dummyMesh, but the bone doesn’t seem to follow the mesh. That being said I am still unsure why I can’t just use the transformNodes directly as they seem to drive the transformations of my bones? Am I missing something with the .glTF importer or the way skinning works for my model? @bghgary could you explain how I can animate the .glTF file as I described above? My bone hierarchy != node hierarchy which could be the reason for the weird side effects.

9

I don’t know how to help you. There isn’t enough information. What does it look like on the other side? What are you sending over the wire?

What does this mean? Do you have a playground?

10

Sorry for the delayed response, I tried to set up a more detailed PG to illustrate what I am doing, due to the streaming of animation Data via a WebSocket it’s difficult to reproduce some parts of my code perfectly, but for now I think it captures how I apply the skeletal transformations to the .glTF model.

Playground: https://playground.babylonjs.com/#FF3FD0#9

I tried using scene.registerBeforeRender to have a more “live animation” but this causes my PG to crash. My code is most likely still pretty inefficient, but I am only focusing on why my transforms “break” the model for now.

Here is what I do:

  1. I do a preorder Traversal of the bone hierarchy, because the skeletonTransformations of the animData are absolute Transforms for each bone, so I ensure that parents bones are transformed before their children.

  2. getAnimData returns the .json file. In my full app I receive a renderFrame from my WebSocket and then parse that frame, here I simulate that process using the whole of the animation data.

  3. For each frame I transform all the bones of my .glTF file in the preorder from above. I use lookupJSONBoneTransforms to translate from the .glTF bone names to the bone names used in my animation Data.

  4. Specifically lines 70 - 93 are where I transform the bones of the .glTF model and I think this is where the error comes from.

  5. The scene that is rendered captures the last frame of the animation data. Head and body are intact, but the arms are not transformed correctly.

When I “animate” my model via the websocket, the head and torso mostly are correct but the arms and fingers are always transformed incorrectly. Am I approaching this problem in the wrong way?

Please ask for any clarifications and thank you for any help provided :slight_smile:

11

Sorry, I’m not asking you to put more code into the playground. I need to know why you expect transferring this information to work from one side to the other. Some users have asked similar questions on other threads (not through the network) where they are trying to retarget an animation. I think this is also what you are trying to do, except over the network? If so, the issue is whether the math works when you do this. If your skeleton is perfectly matched with identical bone matrices, it should work, but if not, then it gets complicated.

12

The absolute transforms are generated for the same .glb file in Unity i.e. same skeleton and bone hierarchy (just the names used for the bones are different) my goal is just to stream them “live”, so I thought just sending the absolute transform data for each individual bone and then setting .absolutePosition, .absoluteRotationQuaternion, .absoluteScaling for each of my bones in the correct order should work.

13

That sounds like it should work, but we have to make sure the way the bones are calculated are the same between Unity and Babylon. I wonder if handedness will be an issue as you are using absolute transforms.

14

I will check if bones are calculated the same way again, but I think it is the case. Don’t Babylon.js and Unity both use left-hand sided coordinates with y-up?

Another thing I noticed yesterday is that if I comment out the line where I set the absolute Position of my bone using .setAbsolutePosition, my model is not transformed at all. So the current transforms are only caused by .setAbsolutePosition. So I must not be setting rotation/scale correctly.

15

Yes, unless you change it for Babylon, which can be configured to be right-handed. glTF is right-handed. How the model is imported may be different.

16

I am trying to debug the reason why only setAbsolutePosition does anything and am running into weird effects when logging things to the console.

I wanted to check if I actually set the absolute Position of my transform Node correctly, but the changes aren’t reflected when I log absolute Position to console.

Doing something simpler to test this, I set the position of each transformNode (of each bone) to the zero vector, but the absolutePosition doesn’t change (at least in the console-the model is transformed correctly).

      for(const bone of importGLB.skeletons[0].bones){
            console.log("BEFORE", bone.getTransformNode().getAbsolutePosition());
            bone.getTransformNode().setAbsolutePosition(new Vector3(0, 0, 0));
            bone.getTransformNode().computeWorldMatrix();
            console.log("AFTER", bone.getTransformNode().getAbsolutePosition());
        }

Here is a playground: https://playground.babylonjs.com/#FF3FD0#10
Could someone please explain why the absolute Position doesn’t change, and how I can see the updated absolute position in the console? Thx

17

Try logging a clone of the absolute position, that way its values at that point in time are preserved. Otherwise the browser may only be showing you the most recent property values for the absolute position object. For example on Chrome on my MacBook this seems to fix the logging issue on your PG. :slight_smile:

1 Like
18

I do have some skeleton strange behaviors as well, but until now I’ve been able to implement workarounds.

One of them is that for one of my projects, I use a few Mixamo models, and they don’t come with exactly the same hierarchy, regardless of what I do. So, to fix that, I made a function which sorts the bones hierarchy. I use one skeleton as reference, then sort the other skeletons to have the same bones order. That way I can retarget my procedural animations with a fixed order.

That works well, but strangely enough, only after I do a first rotation on any of the bones – I chose the hips (index 0) and it fixed my problem. If I don’t do that, most of the time the skeleton is messed up after the sorting. The bone.setRotation() certainly triggers something that fixes the skeleton (for me, I mean).

Just commenting anyway, probably has nothing to do with this thread problem, but who knows…

1 Like
19

So after reading a bit further here: Babylon.js/transformNode.ts at master · BabylonJS/Babylon.js · GitHub

I think that only the position can be set globally using setAbsolutePosition and I can’t set the absoluteRotationQuaternion/absoluteScaling. So in my PG above I am only changing absolute position and not doing anything for absolute rotation, scaling of each bone.

So I was trying to implement setAbsoluteRotationQuaternion myself but am a bit unsure about the matrix maths required for this and would be grateful if someone could verify if this is a valid approach:

    public setAbsoluteRotationQuaternion(transformNode : TransformNode, rotation : Vector3) : TransformNode {
        if(!rotation){
            return transformNode;
        }
        if(transformNode.parent){
            const invertParentWorldMatrix = Matrix.Identity();
            transformNode.parent.getWorldMatrix().invertToRef(invertParentWorldMatrix);
            const result = Vector3.Zero();
            Vector3.TransformCoordinatesFromFloatsToRef(rotation.x, rotation.y, rotation.z, invertParentWorldMatrix, result);
            const q = Quaternion.RotationYawPitchRoll(result.y, result.x, result.z);
            transformNode.rotationQuaternion.set(q.x, q.y, q.z, q.w);
        }else{
           //World Space = Local Space
           const q = Quaternion.RotationYawPitchRoll(rotation.y, rotation.x, rotation.z);
           transformNode.rotationQuaternion.set(q.x, q.y, q.z, q.w);
        }
    
    }

I get the inverse World matrix of each parent bone and use that to translate my absolute rotation vector to the one in the parent’s coordinate system. Then I set the rotationQuaternion (which expects local transforms) of the transformNode of the bone.

20

It seems okay, but I’m not completely sure why you want to use absolute transforms. Why not transfer local transforms over the wire and then you won’t need to do this computation?