Exporting Skeleton Bone - Bind Poses

Questions
1

Hey Guys… (@Deltakosh)…

For a couple years now there has been a problem with the toolkit and skinning (Bone Influencers). Use the default XBot.fbx from Mixamo:

37%20PM
Screen Shot 2019-03-21 at 3.49.37 PM.png1387×975 72 KB

I have a WORK around to use Maya and my Reskin Tool to CLEANUP the skeleton.

Now In Unity to export a skeleton, we iterate thru all the bones and build a localTransform matrix to update each bone:

transformToBoneMap[unityBone].matrix = new[] {
                    localTransform[0, 0], localTransform[1, 0], localTransform[2, 0], localTransform[3, 0],
                    localTransform[0, 1], localTransform[1, 1], localTransform[2, 1], localTransform[3, 1],
                    localTransform[0, 2], localTransform[1, 2], localTransform[2, 2], localTransform[3, 2],
                    localTransform[0, 3], localTransform[1, 3], localTransform[2, 3], localTransform[3, 3]
                };

And we create that localTransform matrix by multiplying the INVERSE bindpose with the Parent bone’s INVERSE bindpose

 // Unity BindPose is already inverse so take the inverse again :-)
                if (transformToBoneMap.ContainsKey(unityBone.parent))
                {
                    var babylonParentBone = transformToBoneMap[unityBone.parent];
                    babylonBone.parentBoneIndex = babylonParentBone.index;
                    localTransform = bindPoses[babylonBone.parentBoneIndex] * bindPoses[i].inverse;
                }
                else
                {
                    if (unityBone != root) {
                        UnityEngine.Debug.LogWarning("Parent bone transform '" + unityBone.parent.name + "' missing weights for child bone '" + unityBone.name);
                    }
                    babylonBone.parentBoneIndex = -1;
                    localTransform = bindPoses[i].inverse;
                }

Now… this works but the SKIN JOB on the model has to be perfect. If it not the when you apply ANY animation to the model you get all Spaghetti Looking Model… Which i still get messages about the model being messed up and you have to use Maya Art Tools to Reskin the model.

While making the GLTF version which does not build a bone matrix for each bone but rather just export the array of bindposes… Everything looks perfect.

gltfSkin.InverseBindMatrices = ExportAccessor(mesh.bindposes, true);

So there is something wrong with the way Unity Babylon Toolkit is encoding the skeleton and bone data. Can you please help me fix this issue.

I tried looking a Babylon.Entities you use for 3DSMAX export. But i cant really tell whats going on in that code… It also looks like you might be decomposing the bind pose matrix and building it back up again and then assigning that to the bone.matrix… I dunno… But something needs to be done to the way babylon toolkit exports skeleton and bones and animations.

Can i count on you guys (the community) to help fix this issue :slight_smile:

1 Like
2

Hey @MackeyK24 … any news on this problem? I can’t remember WHO are the custodians of the 3dsMax exporter these days… but let’s try to get them pinged-into this conversation.

Perhaps @JCPalmer has info, too… gotten from dealing-with Blender exports.

These aged 0-reply threads… worry me… because they feel like “issue fell thru the cracks”. I’m not much good for more important things, but I can clean a crack. (huh?) :slight_smile:

3

The Blender exporter also experienced this. Here it was experienced anytime a mesh which had a skeleton have a scale of 1,1,1, and (rotation of 0,0,0, or rotationQuaternion of 0,0,0,0).

I got tired of the forum topics saying ‘my animation is all messed up’, so I put in a check for this. If encountered, put an error message in the log file, and terminate without generating a JSON file. Still get a topic sometimes, but no effort to try an see what is wrong.

Make sure that the mesh you are testing on does not have this issue, before thinking it is your code.

4

As @JCPalmer says there were issues with models from Mixamo and Blender export that go way back. In the image below you see such a character (Zed) . Note in the upper part how the rotation and scale values are not 0,0,0 for the armature. In the lower half, with the mesh selected, see that the origin of the mesh is not at 0,0,0. Jeff stopped the complaints by detecting these issues in the BJS export script from Blender.

That was with a model downloaded in 2017 - and earlier models could be lying on their backs or face down too. I don;t know if there have been any changes since then - but it was a pain. No idea what happens if you take it into Unity and then export to BJS, or what happens with FBX files as I used DAE files to get to Blender.

gryff :slight_smile:

errors1
errors1.jpg512×727 106 KB

5

I had this problem since the very beginning too. I cant get anyone to actually go int the code and try and fix… Or at least give it a valiant effort. @bghgary got it working for gltf… he does the inverse bind matrices LATE and multiplies by the node transform matrix. I cant get gary to help me translate why his working and our is not… So i can fix once and for all.

Note: @bghgary … not bitching at you… i appreciate all the shit you help me with thus far… i just thought i would stop bugging you directly for the final GLTF pieces i need to work out like… bone matrix and that pbr specular material issue that is overriding my custom material UNWINDING

Maybe you guys can ping him :slight_smile:

6

Shit… I never even noticed the rotation and scale on the ROOT BONE… that is what is probably the issue. When i re-skin… i think it reset the root bone rotation to zero…

Im gonna try and just FREEZE the transforms and NOT reskin and see if that works…

We might be able to compensate or some kind freeze rotation at export time

UPDATE: Still has funny rotation… but the reskin fixes the spaghetti model issue…

I still we need to get gary to spend a bit of time with us/me and hammer out whats what :slight_smile:

7

@MackeyK24 Sorry for the lack of communication on this stuff. There isn’t too much I can do without spending a lot of time on it.

Let me paste what I did to figure out the math for the Babylon glTF loader originally. I was originally comparing with three.js because it was working for three.js. I’m not totally sure it will help, but maybe it can give you a clue.

// bjs
bone.getInvertedAbsoluteTransform().multiplyToArray(bone.getWorldMatrix(), targetMatrix, index * 16);

// 3js
var matrix = bones[ i ] ? bones[ i ].matrixWorld : identityMatrix;
offsetMatrix.multiplyMatrices( matrix, boneInverses[ i ] );
offsetMatrix.toArray( boneMatrices, i * 16 );

// if
boneMatrices === targetMatrix (according to bjs/3js shader code)
boneInverses === inverseBindMatrix (according to 3js GLTF2Loader)

// then 
bone.getInvertedAbsoluteTransform() * bone.getWorldMatrix() = bone.inverseBindMatrix * bone.getWorldMatrix()

// bone.getWorldMatrix cancels out (not exactly because of root transform...)
bone.getInvertedAbsolutionTransform() = bone.inverseBindMatrix;

// given bjs _updateDifferenceMatrix code
bone._absoluteTransform = bone.baseMatrix * parentBone._absoluteTransform;

// expand and solve for bone.baseMatrix
bone.getInvertedAbsolutionTransform() = bone.inverseBindMatrix;
inverse(bone._absoluteTransform) = bone.inverseBindMatrix;
inverse(parentBone._absoluteTransform) * inverse(bone.baseMatrix) = bone.inverseBindMatrix;
inverse(bone.baseMatrix) = parentBone._absoluteTransform * bone.inverseBindMatrix;
bone.baseMatrix = inverse(bone.inverseBindMatrix) * inverse(parentBone._absoluteTransform);
8

Yo @bghgary Thanks for the response… (Although im not quite sure what your trying to get across)

I THINK… All that boils down to this:

bone.baseMatrix = inverse(bone.inverseBindMatrix) * inverse(parentBone._absoluteTransform);

Now i am doing this stuff in Unity so please help me convert this to unity

First off… WTH is AbsoluteTransform … In Unity would that be the world position vs local position

Second… WTH is bone.inverseBindMatrix in Unity… Would that be skin.sharedMesh.bindposes with the matching bone index

Unity Bindposes Says This:

Unity BindPose is already inverse so take the inverse again

Unity Pseudo Code:

int boneIndex = 0; // First Bone Example
var skin = SkinnedMeshRender;

skin.bones[boneIndex].matrix = skin.sharedMesh.bindposes[boneIndex].inverse * skin.sharedMesh.bindposes[parentBoneIndex].inverse;

Now this is basically what i have in the curent code:

            // Attaches Matrix and parent.
            for (var i = 0; i < bones.Length; i++)
            {
                var unityBone = bones[i];
                var babylonBone = transformToBoneMap[unityBone];
                
                Matrix4x4 localTransform;
                
                // Unity BindPose is already inverse so take the inverse again :-)
                if (transformToBoneMap.ContainsKey(unityBone.parent))
                {
                    var babylonParentBone = transformToBoneMap[unityBone.parent];
                    babylonBone.parentBoneIndex = babylonParentBone.index;
                    localTransform = bindPoses[babylonBone.parentBoneIndex] * bindPoses[i].inverse;
                }
                else
                {
                    if (unityBone != root) {
                        UnityEngine.Debug.LogWarning("Parent bone transform '" + unityBone.parent.name + "' missing weights for child bone '" + unityBone.name);
                    }
                    babylonBone.parentBoneIndex = -1;
                    localTransform = bindPoses[i].inverse;
                }
        }

Now you looked at the code before… You said… It need to take into account the local transform then i never heard back from you about what that means. Take what local transform into account… the bone or the parent… And where to take the ‘local transform’ into account.

Bro… Can you (OR SOMEBODY) please go into the code in the toolkit and try to fix her up… Send me that code and we are all golden… It would probably take you 10 mins to find and fix the problem if you just please go look at the code. Just download the current toolkit and look at file:

/Assets/Babylon/Sources/SceneBuilder.Meshes.cs

the function name

BabylonSkeleton ConvertUnitySkeletonToBabylon(SkinnedMeshRenderer skin, GameObject gameObject, float progress, ref UnityMetaData metaData)

And Bro… If its a mater of time and resources… I will gladly PAY YOU for your time and effort :slight_smile:

9

See https://github.com/BabylonJS/Babylon.js/blob/master/src/Bones/bone.ts#L225

I believe that is correct.

I’ll try to give you as much information as possible to fix it. I spent way more time than I’m willing to admit to get the glTF loader working correctly with skinning. :slight_smile: