Return to Rest not Behaving as Expected?

Pryme8 · July 21, 2020, 11:56pm

Here ill post the links for example GLBs and it needs to work on all of these before we could consider it a complete solution.

https://dl.dropbox.com/s/pmqjuchgryngyyy/mutant.glb
https://dl.dropbox.com/s/lgu07fvjixd88h1/ironman.glb
https://dl.dropbox.com/s/aq1tgg4i7ou4uzf/gym-beasts.glb

and we wanna look at them in this context prolly:
https://playground.babylonjs.com/#ZJIBBU#26

gbz · July 22, 2020, 1:00am

@Pryme8, here is a video of the Blender modifications to the psylocke_ours.glb:

In general, before using in Babylon, we should make sure that the model and armature

has Location (0, 0, 0), Rotation XYZ Euler (0, 0, 0), and Scale (1, 1, 1), and we need to Apply All Transforms as shown at time 1:29 in the video
has a NLA track for the rest pose as shown at time 2:07 in the video

I’m not sure how the Blender exporters are implemented, but the above two items have allowed things to work for me in the past.

Pryme8 · July 22, 2020, 1:04am

Im not going to have control of the models though, so this will not work. It needs to be a 100% coverall solution. We are working on a replacement for the SkeletonViewer and in order to get it all working in one drawcall I need to be able to get to the same pose that the armature was weighted in. Like I said blender seems to be able to decompile it from the glb data so our best bet would be to see what is happening there that we are not doing.

gbz · July 22, 2020, 1:14am

It seems that https://sandbox.babylonjs.com/ has greater coverage of files in showing correct animations. I wonder if the Sandbox is implemented differently.

Pryme8 · July 22, 2020, 1:35am

I think @Drigax is really close with what he was brewing up. Now if we can get those lines to draw in the right spot in the one he was working on and then revert everything back to the animation pose we should be golden. I will be able to make it work from there.

Drigax · July 22, 2020, 7:07pm

We’re getting there, abeit slowly.

This definitely looks like a problem with the bone’s rest pose. I changed the logic of @Pryme8’s octahedral skeleton generating PG to decompose a calculated “absoluteRestPose” for each bone, used to initialize the debug skeleton bones, but it appears to be generated in the posed position. This is without skinning the mesh to the skeleton, we’re leaving the skinning buffers unmodified for now…
https://playground.babylonjs.com/#ZJIBBU#34

Disabling skinning, I can see that our mesh data is being exported as an A-Pose, so we can assume that we’re properly skinning the mesh in our exported glb scene…

As @Evgeni_Popov hinted to yesterday, we need to figure out what the heck is going on with our rest pose data, and why my modified returnToRest rendered properly…

Drigax · July 23, 2020, 4:34am

I think I understand what’s going on now…:

When we load glTF we do a multi-pass initialization when we asychronously load the different accessors for the data in our scene:

On the first pass as we load the bones referenced in a skin definition, we initialze the Bone using a null restPose:

github.com

BabylonJS/Babylon.js/blob/7b15ba42e23fb166a0fd6992c29eeb448a5b4b7e/loaders/src/glTF/2.0/glTFLoader.ts#L1096


        return babylonBone;
    }

    let babylonParentBone: Nullable<Bone> = null;
    if (node.parent && node.parent._babylonTransformNode !== this._rootBabylonMesh) {
        babylonParentBone = this._loadBone(node.parent, skin, babylonSkeleton, babylonBones);
    }

    const boneIndex = skin.joints.indexOf(node.index);

    babylonBone = new Bone(node.name || `joint${node.index}`, babylonSkeleton, babylonParentBone, this._getNodeMatrix(node), null, null, boneIndex);
    babylonBones[node.index] = babylonBone;

    node._babylonBones = node._babylonBones || [];
    node._babylonBones.push(babylonBone);

    return babylonBone;
}

private _loadSkinInverseBindMatricesDataAsync(context: string, skin: ISkin): Promise<Nullable<Float32Array>> {
    if (skin.inverseBindMatrices == undefined) {

We then use an asychronous callback to update the bone matrices using the loaded inverseBindMatrix that glTF uses to represent the world-to-bone for a given bone’s bind pose. via glTFLoader._updateBoneMatrices() (I’m going to use restPose and bindPose interchangably here, sorry):

Currently this callback sets the Bone._baseMatrix data via Bone.updateMatrix, but it looks like we overlook setting the rest position as well. My understanding is that the _baseMatrix represents the current local-space position of the bone, so I’ll keep to that pattern of representing the rest pose in local space as well.

The only issue with this notation is that it makes it somewhat difficult to quickly calculate the world space rest pose of a given bone, since we’d need to recurse the hierarchy. We may want to consider also add some additional members to represent this data as well, but I’ll err or making my change as minimal as possible.

So I made a modification to our initialization logic, and added a setter for Bone.setRestPose(), and its starting to look somewhat correct, we’re just missing something critical…:

    private _updateBoneMatrices(babylonSkeleton: Skeleton, inverseBindMatricesData: Nullable<Float32Array>): void {
        for (const babylonBone of babylonSkeleton.bones) {
            let baseMatrix = Matrix.Identity();
            const boneIndex = babylonBone._index!;
            if (inverseBindMatricesData && boneIndex !== -1) {
                Matrix.FromArrayToRef(inverseBindMatricesData, boneIndex * 16, baseMatrix);
                baseMatrix.invertToRef(baseMatrix);
            }

            const babylonParentBone = babylonBone.getParent();
            if (babylonParentBone) {
                baseMatrix.multiplyToRef(babylonParentBone.getInvertedAbsoluteTransform(), baseMatrix);
            }

            babylonBone.setRestPose(baseMatrix);
            babylonBone.updateMatrix(baseMatrix, false, false);
            babylonBone._updateDifferenceMatrix(undefined, false);
        }
    }

Drigax · July 23, 2020, 4:37am

disabling skinning, you can see what I mean…:

I think our skeleton is now properly created using the bind pose, but why is it not deforming properly?

EDIT:

I can see here that we’re hitting some sort of error when merging the skeleton meshes, as they don’t all have the same attributes…interesting…

EDIT:
It’s ok guys I’m just dumb. I forgot i commented out the spur and sphere vertex buffer setting code (PG @296-297)

@Pryme8 I think we got it buddy, can you work off of this iteration of the PG?
https://playground.babylonjs.com/#ZJIBBU#36

YET ANOTHER EDIT:
Fix is available here:

Evgeni_Popov · July 23, 2020, 10:03am

Great work!

I tested by applying your fix on my local repo but could not make it work.

Note that I had to apply your previous fix too (that is not in the PR), else the mesh is completely off:

public returnToRest(): void {
    for (var index = 0; index < this.bones.length; index++) {
        if (this.bones[index]._index !== -1) {
            this.bones[index].returnToRest();
        }
    }
}

With this fix and the PR, now we get:

when calling skeleton.returnToRest() (PG: Babylon.js Playground): the skeleton is ok but not the mesh.

Which is the reverse of what we get without the PR (but still with the fix above):

The mesh is ok but not the skeleton.

So, I updated skeleton.returnToRest to update the linked transform nodes:

public returnToRest(): void {
    const _localScaling = Vector3.Zero();
    const _localRotation = Quaternion.Zero();
    const _localPosition = Vector3.Zero();

    for (var index = 0; index < this.bones.length; index++) {
        const bone = this.bones[index];

        if (bone._index !== -1) {
            bone.returnToRest();
            if (bone._linkedTransformNode) {
                bone.getRestPose().decompose(_localScaling, _localRotation, _localPosition);

                bone._linkedTransformNode.position = _localPosition.clone();
                bone._linkedTransformNode.rotationQuaternion = _localRotation.clone();
                bone._linkedTransformNode.scaling = _localScaling.clone();
            }
        }
    }
}

and now the display is ok (after calling returnToRest()):

BUT it’s not the end of the story!

The display is ok when we call returnToRest at start but if starting an animation for some seconds, then stopping the animation and calling returnToRest it does not work (PG: https://playground.babylonjs.com/#2KRYP8#1): the skeleton is ok but the mesh has “disappeared”:

In fact the mesh is there: the bounding boxes in the screenshot are the bb of the 4 meshes, but the meshes are not displayed for some reason… Investigating…

[EDIT] In fact the bb are ok because they are never updated, but the transform matrices are wrong [/EDIT]

Evgeni_Popov · July 23, 2020, 10:51am

Ok, calling:

this.updateMatrix(this._restPose.clone(), false, false);

in bone.returnToRest instead of:

this.updateMatrix(this._restPose.clone());

seems to do the trick!

So, to sum up, applying all the below changes seem to make it work:

changes from the PR Fix rest pose of gltf imported Bone by Drigax · Pull Request #8589 · BabylonJS/Babylon.js · GitHub
Bone.returnToRest():

    public returnToRest(): void {
        this.updateMatrix(this._restPose.clone(), false, false);
    }

Skeleton.returnToRest():

public returnToRest(): void {
    const _localScaling = Vector3.Zero();
    const _localRotation = Quaternion.Zero();
    const _localPosition = Vector3.Zero();

    for (var index = 0; index < this.bones.length; index++) {
        const bone = this.bones[index];

        if (bone._index !== -1) {
            bone.returnToRest();
            if (bone._linkedTransformNode) {
                bone.getRestPose().decompose(_localScaling, _localRotation, _localPosition);

                bone._linkedTransformNode.position = _localPosition.clone();
                bone._linkedTransformNode.rotationQuaternion = _localRotation.clone();
                bone._linkedTransformNode.scaling = _localScaling.clone();
            }
        }
    }
}

Deltakosh · July 23, 2020, 1:46pm

Lol this is EPIC!

@Drigax can you add @Evgeni_Popov changes to your patch?

Drigax · July 23, 2020, 3:29pm

Thanks a ton @Evgeni_Popov, we’ve come full circle, I’ll add your modifications to the PR.

Pryme8 · July 23, 2020, 3:36pm

I’m so frickin proud of you two right now. . If this is then this is a fairly major fix for some cool things to happen now with the skeleton tools.

Drigax · July 23, 2020, 4:57pm

Checked @Evgeni_Popov’s changes over the current proposed fix. Looks great! we’re finally properly updating our linked transforms off of skeleton.returnToRest()

I noticed that it broke the latest iteration of the playground, however but it appeard to be caused by some of our logic we use to scale our skeleton debug meshes (@L332-336):

                            if(bone.getParent()){
                                let distanceFromParent = BABYLON.Vector3.Distance(bone.getParent().getPosition(BABYLON.Space.WORLD, this.mesh), bone.getPosition(BABYLON.Space.WORLD, this.mesh))
                                let factor = (distanceFromParent/(longestBoneLength*this.options.displayOptions.sphereFactor))
                                scale*=factor 
                            }

resulted in the meshes being far too large, since we’re using the posed mesh absolute positions…

changing this logic to instead use the local bindPose offset gave us a better result:

                            if(bone.getParent()){
                                let restPosePosition = new BABYLON.Vector3();
                                bone.getParent().getRestPose().decompose(null, null, restPosePosition);
                                let distanceFromParent = restPosePosition.length();
                                let factor = (distanceFromParent/(longestBoneLength*this.options.displayOptions.sphereFactor))
                                scale*=factor 
                            }

https://playground.babylonjs.com/#ZJIBBU#38

Ok! I think that settles it! I’ll update the PR with the additions. Thanks again!

Deltakosh · July 23, 2020, 5:05pm

YOu rock!

Evgeni_Popov · July 23, 2020, 9:57pm

The PR is live, you can see that this PG does work: https://playground.babylonjs.com/#2KRYP8#1

(I had to clean the cache of my browser to get the latest alpha release)

Pryme8 · July 23, 2020, 11:57pm

This one is close to perfect, I bet its just my Viewer that is off with the hip spheres.

not sure if this one is just a bad skeleton or not, Id have to look at the file will check it out asap.

Something up with this one as well.
https://playground.babylonjs.com/#ZJIBBU#40 <- and this if fairly nutty when scaling is applied.

Let me make sure that that data is correct before I jump the gun here tough and say those are wrong.

update
Some of these are working others are not the “gym-beasts” I think would be a good one to solve after the “psylock” model.

Evgeni_Popov · July 24, 2020, 7:42am

Here’s a smaller PG for the gym-beasts which is easier to deal with: Babylon.js Playground

What I could find is that if babylonBone.setRestPose(baseMatrix); is commented in GLTFLoader._updateBoneMatrices it seems it does work (however psylock won’t work anymore as it was one of the change that made it work…).

[EDIT]
Does exporting a .babylon file and use it in the PG work? If yes, could you provide the .babylon? Thanks!
[/EDIT]

Pryme8 · July 24, 2020, 3:12pm

The gym beast one I might think is out of scope

But ideally our stuff should pop up like this though so I might need to look at my bone and spur calculations now.

We should be able to get these methods to work on every file type though, do you still want me to send the .babylon? Ideally we would just work with the glb as I will have no control over the assets being presented but this still needs to work kinda no matter what.

Evgeni_Popov · July 24, 2020, 3:48pm

I have a simpler model that does not work, so I’m going to investigate with it first.

Topic		Replies	Views
There are issues with switching the blend model to the rest state Questions	8	41	December 23, 2024
Why stopping all animation groups won't make my character to return to rest pose? Questions animation , glb , animationgroups	3	107	March 25, 2024
What do the values in the joint rest matrix represent Questions	5	410	April 30, 2020
Bone rest pose and how it effects animations Questions	4	608	May 10, 2020
Unexpected bone rotations of skeleton from glb Questions	2	41	February 3, 2025

Return to Rest not Behaving as Expected?

Related topics