GLTF Skeletons And Transform Nodes

Yo @bghgary … I got a couple questions about GLTF Skeletons…

1… What are you doing differently to a skeleton that links it with the transform nodes in the skinned mesh hierarchy ???

2… Why does playing animations on the transform nodes move the mesh. Its a transform node in the inspector, but you said something about those transform nodes are actually bones… How is that ???

Can you please point me to the code in GLTFLoader.ts that does this. I would like to see how and what you are actually doing to create a skeleton (that are linked to the transform nodes… i guess). I looked at the loadSkinAsync chain and i dont see anything that stands out :frowning:

Here is the PR: Add support to override mesh for skeleton and link bones to transform nodes by bghgary · Pull Request #5513 · BabylonJS/Babylon.js · GitHub

I guess my next question is Why?

Why do we have to do all that to create a skeleton… Why not just create the skeleton and put the animations on the bones.

It is to keep the TRANSFORMS in the loop so you can attach other meshes like a sword ???

I ask because i am trying to figure out why Animations played using the standard GLTF transform nodes DONT HAVE TO BE RE-SKINED. But if i put the animations directly on the bones… The skin job has to be PERFECT or you get spaghetti models when the animation plays.

I think is has to do with the bone local matrix…In unity is the parent bone bindpose (ALREADY INVERSED) multiplied by the bone inverse… This is he way the Babylon Entities serialze skeletons… then no need to save all the inverseBindMatrices and have to update the bone matrix at runtime since each bone has is inverse local matrix setup on the bone itself.

GLTF seems to NOT be effected by this issue because the animations are encoded on the transform nodes then the transform node WORLD matrix is copied to the linked bone each frame…

This somehow does NOT get the spaghetti model… But if i go the pure skeleton route… I gotta RE-SKIN the model in Maya to get around the spaghetti model…

Have you got ANY clues as to how to handle spaghetti looking animations BESIDE re skinning the model in Maya… Use another matrix or something ???

This is what used to happen, but it has problems that you reported. :slight_smile:

glTF skins are modeled from Unity which gets the bone transforms, etc. from nodes in the hierarchy. As seen in the issue you posted (HTML5 Game Devs Forum - HTML5GameDevs.com), if the runtime doesn’t match exactly with what is expected from glTF, unexpected things might happen.

When the glTF issue Sharing skeletons between two skinned mesh · Issue #1285 · KhronosGroup/glTF · GitHub gets fixed in future versions, this will be much less of an issue, but I can’t think of a good way to fix this with glTF 2.0 and Babylon.

Yeah… this was a tough one fo me too… I end solving the issue with metadata… for skeleton
then in my GLTF extension when serializing a skeleton, and assigning to the mesh… I check if that skeleton already exists and use that if so. But i am serializing the whole skeleton as metadata and NOT exporting a GLTF SKIN… Just the bone weights and bone indexes on the skinned mesh renderer. My extension create the skeleton just like the normal babylon parser then assign to mesh:


        private static DoParseSkeletonBones(parsedSkeleton: any, parser:BABYLON.MetadataParser, scene: BABYLON.Scene): BABYLON.Skeleton {
            const skeleton = new BABYLON.Skeleton(parsedSkeleton.name, parsedSkeleton.id, scene);
            skeleton.needInitialSkinMatrix = parsedSkeleton.needInitialSkinMatrix;
            if (parsedSkeleton.dimensionsAtRest) {
                skeleton.dimensionsAtRest = BABYLON.Vector3.FromArray(parsedSkeleton.dimensionsAtRest);
            }
            let index: number;
            for (index = 0; index < parsedSkeleton.bones.length; index++) {
                let parsedBone:any = parsedSkeleton.bones[index];
                let parentBone:any = null;
                if (parsedBone.parentBoneIndex > -1) {
                    parentBone = skeleton.bones[parsedBone.parentBoneIndex];
                }
                const localMatrix:BABYLON.Matrix = (parsedBone.matrix != null) ? BABYLON.Matrix.FromArray(parsedBone.matrix) : BABYLON.Matrix.Compose(
                    (parsedBone.scale != null) ? BABYLON.Vector3.FromArray(parsedBone.scale) : BABYLON.Vector3.One(),
                    (parsedBone.rotation != null) ? BABYLON.Quaternion.FromArray(parsedBone.rotation) : BABYLON.Quaternion.Identity(),
                    (parsedBone.translation != null) ? BABYLON.Vector3.FromArray(parsedBone.translation) : BABYLON.Vector3.Zero()
                );
                const bone = new BABYLON.Bone(parsedBone.name, skeleton, parentBone, localMatrix, null, null, index);
                if (parsedBone.id !== undefined && parsedBone.id !== null) {
                    bone.id = parsedBone.id;
                }
                if (parsedBone.length) {
                    bone.length = parsedBone.length;
                }
                if (parsedBone.metadata) {
                    bone.metadata = parsedBone.metadata;
                }
                if (parsedBone.animation) {
                    bone.animations.push(BABYLON.Animation.Parse(parsedBone.animation));
                }
            }
            // placed after bones, so createAnimationRange can cascade down
            if (parsedSkeleton.ranges) {
                for (index = 0; index < parsedSkeleton.ranges.length; index++) {
                    const data = parsedSkeleton.ranges[index];
                    skeleton.createAnimationRange(data.name, data.from, data.to);
                }
            }
            return skeleton;
        }

That way i end up with a native skeleton (NOT LINKED TO NODE TRANSFORM… Which i get rid of anyway)… When you have a native skeleton… no need for all those extra transform nodes… (Except if you wanna attach meshes like weapon to hand… which i am workin on a SOCKET mesh system for that… kinda like Unreal Sockets)

If you can translate Unity into something that always works in Babylon, then the loader should be able to also, since glTF works like Unity for skinning. But I suspect it doesn’t always work. Perhaps I’m wrong.

Yo @bghgary … Can you please point me at the part of gltfLoader.ts code where your skeleton is different than a skeleton built with entities… Basically what makes the skin NOT have the bounding box updated. Vs if you made a skeleton with babylon entities…

Is it override mesh ???

I am trying to BETTER understand the way you make skeletons and compute matrices so i can FIX the babylon skeleton issue i have in the current Unity Exporter.

It’s all in the PR I sent earlier: Add support to override mesh for skeleton and link bones to transform nodes by bghgary · Pull Request #5513 · BabylonJS/Babylon.js · GitHub

Yeah… I looked at that… But i cant figure out the WHY, what your doing to create a skeleton vs what the babylon entities does to create the skeletons.

HOW or WHY does override mesh effect the bound box issue vs a regular babylon skeleton.

Note: I simply did NOT create a gltf.skin but instead just created my own skeleton using the metadata (Basically i serialized the BabylonSkeleton from the entities) and i de-serialize the skeleton in the GLTF parser the same spot where you would normally create the gltf.skin… And beside the FACT that the bone matrix is still FICKED UP

It works fine…

So i am having a VERY HARD TIME trying to figure out the WHY for some other stuff your doing in the gltf loader.

WHY do we have an OVERRIDE mesh (if that is the culprit that causes the bounding box issue)… If a regular ol skeleton can be created instead… I dont get that…

Getting very frustrated… About to just use the regular GLTF SKINNING (even though i freaking HATE IT) and just call it a day. Everyone using the toolkit can deal with it their own way :frowning:

As I mentioned before, I added override mesh to skeletons to fix the issue you were having with skeletons. I was originally moving the skinned mesh to under the __root__ which caused the issues you were seeing. The reason any of this is a problem is because glTF skinned meshes must ignore its own transform. See second implementation note of the skinning section of the glTF spec.

Read this if you want to know the history of all of this: Add implementation notes about which transforms apply to skinned meshes by bghgary · Pull Request #1195 · KhronosGroup/glTF · GitHub

I’m sorry @bghgary … I didn’t meant WHY you did what you did… I really meant WHY the stuff you do works vs my code.

For example , the skeletons … WHY does your skeleton and animations work fine without altering the model what so ever.

Vs the current toolkit skeleton and animations … I have to re skin the model or I get spaghetti looking model… but your does not … WHY … is it the override mesh… if so WHY does having a override mesh fix that problem… or is it the bind poses.

When I keep saying why is this and that… I’m trying to figure out WHY your models work just fine GLTF but mine do not with Babylon meshes and skeletons… not without me fixing up the model skin in Maya … WHY is that is what I really meant to get across