Return to Rest not Behaving as Expected?

Pryme8 · July 21, 2020, 5:52pm

Shouldn’t they move though? What would be keeping them in the action pose while the mesh updates?

Sorry if I am hammering on this, but it really seems like something that should be solved as it kinda is making me thing there is a underlying problem with our skeleton stuff.

Evgeni_Popov · July 21, 2020, 6:32pm

Yes I do think there’s a problem somewhere (or something more fundamental is escaping us), the thing is to find it…

Pryme8 · July 21, 2020, 6:35pm

Ok cool, just making sure we were on the same page. I feel like once we figure it out we are both gonna palm our foreheads.

Pryme8 · July 21, 2020, 7:07pm

I think I have narrowed it down.

In Blender the “restPose” for this model is when all the bones are set to (-0.707, 0, 0, 0.707) rotation, but I do not see these numbers in any matrix in BJS?

How is blender getting these numbers when they import the GLB its the same file so it has to be somewhere.

github.com

ksons/gltf-blender-importer/blob/master/addons/io_scene_gltf_ksons/vnode.py#L300


                        new_children.append(child)
                    vnode.mesh = None
                    vnode.children += new_children
                    vnode.mesh_moved_to = new_children

            for child in children:
                visit(child)

        visit(op.root_vnode)

# Here's the compilcated pass.
#
# Brief review: every bone in glTF has a local-to-parent transform T(b;pose).
# Sometimes we suppress the dependence on the pose and just write T(b). The
# composition with the parent's local-to-parent, and so on up the armature is
# the local-to-armature transform
#
#     L(b) = T(root) ... T(ppb) T(pb) T(b)
#
# where pb is the parent of b, ppb is the grandparent, etc. In Blender the
# local-to-armature is

^I think the solution might come from here?

Drigax · July 21, 2020, 7:38pm

Looking at the hierarchy, I can see that blender uses a scaling node to convert between the scene units:

This works for glTF, since we can set an arbitrary node as our skeleton root, but in babylon we extend the skeleton to use all nodes in its hierarchy, up to the scene root:

If we look at the implementation of returnToRest(): I believe that we’re currently resetting the scale of that root helper node in addition to the rest of the skeleton:

github.com

BabylonJS/Babylon.js/blob/22e572cc8da3887c059399f1a8514464f6d30dbe/src/Bones/skeleton.ts#L356-L360


public returnToRest(): void {

    for (var index = 0; index < this.bones.length; index++) {

        this.bones[index].returnToRest();

    }

}

currently if i disable resetting the bones that our loader adds, the rest position mesh starts looking a bit more proper:

    /**
     * Forces the skeleton to go to rest pose
     */
    public returnToRest(): void {
        for (var index = 0; index < this.bones.length; index++) {
            if(this.bones[index]._index != -1) {
                this.bones[index].returnToRest();
            }
        }
    }

yields:

But the skeleton helper is still generated in a posed position… even though we “returned” our skeleton to rest position, since the bones are linked to a transformNode that drives their transform, that may explain why it doesn’t seem to work in reverse. The transform nodes still appear to have their posed orientation and we may have reset and re-rendered the skeleton using the reset position, I’m not sure if this won’t be re-written by linked transform node…

Pryme8 · July 21, 2020, 7:48pm

Ohh dude you are getting soooo close! You are about to save the day I can feel it.

If we can now get the bones to fall into their correct position now to as to reflect the structure of that mesh then I think we got it!

Drigax · July 21, 2020, 7:56pm

maybe we can try adding a mechanism to update bone._linkedTransformNode on reset? the skeleton generator looks like it works from what I can gather, it may just be how we’re linking the bone and node transform…

Pryme8 · July 21, 2020, 8:05pm

Lets give that a shot! Ideally if I can see those bones at a tPose then all my problems are solved and I bet will work with my other fringe case GLBs and I have quite a few to test here.

The reason I am so pressed for this is I have a couple projects blocked by this problem so any sort of workaround/solution at this point will be a game changer.

Evgeni_Popov · July 21, 2020, 9:22pm

From what I can tell, the skeleton viewer does not use the linkTransformNode data.

It is doing bone.getLocalMatrix() and parentBone.getAbsoluteTransform() to compute the lines to display…

Drigax · July 21, 2020, 9:41pm

@Evgeni_Popov
Correct, but my understanding is that on render, Skeleton.prepare() sets bone._matrix = linkedTransformNode._localMatrix, so that the _linkedTransform matrix always drives the bone matrix.

github.com

BabylonJS/Babylon.js/blob/88df63fc49b71073946df818a75bfe073fcceb8a/src/Bones/skeleton.ts#L495-L506


public prepare(): void {

    // Update the local matrix of bones with linked transform nodes.

    if (this._numBonesWithLinkedTransformNode > 0) {

        for (const bone of this.bones) {

            if (bone._linkedTransformNode) {

                // Computing the world matrix also computes the local matrix.

                bone._linkedTransformNode.computeWorldMatrix();

                bone._matrix = bone._linkedTransformNode._localMatrix;

                bone.markAsDirty();

            }

        }

    }

EDIT:
To be honest, I’m not sure if this is a good way of going about it, it seems like this opens the door for other entry points to manipulate the bones just to be overwritten next render tick

Evgeni_Popov · July 21, 2020, 9:53pm

Yes, but as the mesh has now the correct rest position, that should mean the bones have the correct matrices (because the mesh appearance is driven by the bones), meaning the linkTransformNode has also the correct matrix as as you said it is the linkTransformNode matrix which is copied to the bone matrix.

So, it would seem everything has the right matrices everywhere, still the skeleton is not displayed correctly… To me it means that the localMatrix of the bones/linkTransformNodes are in fact wrong (they are responsible for the wrong skeleton line display) and the right rest display is coming from “something else” (I don’t know what).

gbz · July 21, 2020, 10:43pm

After importing the file from Dropbox - psylocke_ours.glb - Simplify your life into Blender, I saw as @Drigax pointed out,

There is a RootNode that adds extra rotation transform to the armature and mesh.

And there’s another node above the armature that adds more rotation transform and a scaling of 100X.

And there are a bunch of other transforms under the armature.

So I deleted those other transforms, and saw that the mesh itself has its rotation in Quaternion (WXYZ) rather than XYZ Euler.

I tried removing the two root nodes above the armature, and placing the armature directly under Collection.

Here’s the final look at the mesh with applied location, rotation, and scale.

However, with these changes, it’s not yet working: https://playground.babylonjs.com/#ZJIBBU#18

Evgeni_Popov · July 21, 2020, 10:55pm

By applying the correction by @Drigax (checking if _index !== -1 before updating the rest matrix), it does make the skeleton fit the mesh:

But calling skeleton.returnToRest() still does not work, it does not update the skeleton:

gbz · July 21, 2020, 11:10pm

https://playground.babylonjs.com/#ZJIBBU#22

Tried creating a NLA track for the rest pose in Blender. The animations seem to work okay, but the Skeleton Viewer is not updating.

Evgeni_Popov · July 21, 2020, 11:25pm

This does not work for two reasons:

you have to call skeletonViewer.update() to update the skeleton rendering
because of this bug: Bone: fix bug introduced by latest changes by Popov72 · Pull Request #8572 · BabylonJS/Babylon.js · GitHub

The fix for #2 is now live, so only #1 remains and is corrected in this PG:

https://playground.babylonjs.com/#ZJIBBU#23

Calling skeleton.returnToRest() does work in your PG, but the default position is also the rest position, so I guess it’s why it is working. You get the same mesh display by calling skeleton.returnToRest() or not:

https://playground.babylonjs.com/#ZJIBBU#24

Try to comment line 29: it does not change the display.

[EDIT]
So, if I understand correctly what you did, the problem comes from the two additional nodes that you removed in the latest file, leading to the default position of the mesh being the rest position?
[/EDIT]

Pryme8 · July 21, 2020, 11:31pm

I am so happy we are homing in on this!

But would really want things to work on the initial model without it being modified in a editor, because I will not have access to do that and we need these methods to work for all cases.

Evgeni_Popov · July 21, 2020, 11:49pm

Clear the cache of your browser. This PG does work for me:

https://playground.babylonjs.com/#ZJIBBU#23

The skeleton is following the animations.

gbz · July 21, 2020, 11:51pm

@Evgeni_Popov, whoa, amazing!

Pryme8 · July 21, 2020, 11:51pm

Is that on the original GLB or the one that was modified?

If it cant work on the original one then it wont work as a solution as this needs to work on all possible setups we might get passed. We are not going to be able to go back and modify the glb to ensure that its going to work.

And if Blender works with this file, there is nothing stopping us from making it work.

https://playground.babylonjs.com/#ZJIBBU#25

Evgeni_Popov · July 21, 2020, 11:53pm

It is using the last modified version by @gbz .

Topic		Replies	Views
Why stopping all animation groups won't make my character to return to rest pose? Questions animation , glb , animationgroups	3	51	March 25, 2024
What do the values in the joint rest matrix represent Questions	5	381	April 30, 2020
Bone rest pose and how it effects animations Questions	4	507	May 10, 2020
Skeleton failing to go to rest position Questions	1	373	December 5, 2019
I want to make a skeletal animation like the demo, but it seems that it can't be realized. Is it a problem with my model? Questions	4	359	September 6, 2021

Return to Rest not Behaving as Expected?

Related Topics