Snapping Items to a Mesh to move with animations

Hi All,

Just me again, the humble potato on my journey and need some guidance :pray:

So we have a character model with a Skeleton to animate it, and we want to attach items to the character mesh to swap items in and out. To do this, we have created a small box mesh as an anchor point on the head mesh glb and made the head mesh the parent of the anchor, so in theory, when the head moves, the anchor point should move in relation to it :thinking:

Screenshot 2022-11-16 at 09.24.43

On the hair side of things, the hair also has an anchor point, and this time the anchor is the parent of the hair mesh so when the anchor moves so should the hair :wink:

Screenshot 2022-11-16 at 09.25.51

The challenge is that first when I export the model using the GLTF exporter, we get the usual mesh replication into a transform node, I believe this is expected behaviour? but the anchor point is not associated with the transform and not the mesh as it was in Blender, due to something that the GLTF exporter is deciding to do. It would firstly be good to know why it does this?

Screenshot 2022-11-16 at 09.49.28

When we try to reparent that anchor back to the head as it was in Blender when the head animates, the anchor stays in position. We essentially want to be able to create separate body parts and attach then to the main mesh and have them move in relation to it. My questions here for support are:

  • Is the approach we have taken even the right one?
  • Should we be merging the meshes instead of just parenting them?
  • Should we be attaching things to bones instead and if so how would we handle the positioning of the items in relation to the bones? Is this the art side or the Babylon side?

Really appreciate you getting this far with my post :pray: appreciate any help or guidance you can give

I will definitely redirect this one to our in house artists @PatrickRyan

@Anixan_Skystalker, I think I am following what you are seeing, but if I get anything wrong, please ask more questions.

From what I understand, you are seeing a couple of issues which are that you are not seeing an anchor point transform when importing the mesh and that the anchor point no longer follows the head when the head animates which appears that the anchor is no longer parented to the head mesh.

If this is correct, I believe that both of these issues are coming from the same root condition and, believe it or not, are acting as they should. To explain what is happening is a little abstract, so please bear with me and ask questions if I leave you confused.

When you skin a mesh to a skeleton, the vertices of the mesh follows the bones within the skeleton based on two things: the skinning weight to any given bone and the offset from that bone derived from the bind pose of the mesh. However, if you were to look at the bounding box of a mesh being deformed by the skeleton in Babylon.js, you will see that the bounding box does not move. Note in the gif below I have the bounding boxes displayed for the body and dome meshes while the animation is playing.

skinnedMeshBoundingBox

This is because the bounding box is computed before the skinning matrix is applied to the mesh. So if you are to parent your anchor to the mesh, it will inherit the position of the mesh, which is shown by the bounding box, before the skinning matrix is applied and would appear not to move with the skinned mesh.

The better way to do this is to parent your attachments to the bones in your skeleton since that is where the motion is derived from and is not affected by the lack of skinning matrix to compute the final position. But this also comes with some challenges if you have to also factor in an offset from the bone to get your attachment in the correct position. For example, if you want to put a hat on a character, the head for the bone may not be located where you would want the root position of the hat to be so you will need an offset.

There are two ways to do this and the first would be a slight modification of what you started with. Instead of creating a cube for your attachment position, use an empty in Blender. This still gives you a transform, but you don’t have any mesh associated with that transform so you don’t need to worry about more vertices or disabling the render once you are in engine. Then you will want to set the empty as a child of the bone driving the part of the mesh you want to attach to. If you want your attachment point for a hat, you would set your empty as a child of the bone driving the head and then set the position of the empty to be the attachment point of the hat. This offset will remain constant and the empty will move with the parent bone.

The second way is to create extra bones within your skeleton that are children of the bones driving the attachment point and position the bones in the location you want your attachment point to be. Again the joint offset will remain and then you want to set the parent of your attachment to be the attachment bone.

Some considerations on this approach would be to make sure that the orientation of the bones used for attachment align with the expected orientation of the attached mesh. If the orientation is wrong, the initial rotation of your attached mesh will be wrong. You will also want to make sure that no vertices have any weight influence being driven by the attachment bones. This could cause issues with the deformation of the mesh and simply giving these bones zero weight would be best.

There are drawbacks to both paths, however, and you need to choose the path that best suits your end goals. For empties as attachment points being children of the bones, you will end up with more transforms in your scene, which does have a compute cost. But this is a really flexible system because you can easily go back and add more attachment locations in the future if you see the system expanding. Using extra bones at attachment points is cheaper in the sense that you aren’t adding more transforms to the scene but may be less ideal to expand in the future. You may find that you need to reskin your meshes to add more attachment bones in the future. However, if you don’t plan to do it often, or if you are willing to take that maintenance on it may be the better option if you have a lot of attachments on your characters.

In terms of the transform node not showing on your cube when looking at it in the inspector, I believe this is due to the fact that your cube is not skinned to the skeleton. I believe we only display the transform of the mesh separately like this when the mesh is skinned.

I hope this helps, but please ping me back if anything isn’t clear.

4 Likes

Hi @PatrickRyan, the other potato here :slight_smile: I kid you not. We haven’t stopped since your post. Unfortunately, it’s like we’re a line of code away from greatness.

I popped together a playground in the hope that it shed some more light on what’s going:

We’ve probably made everything really dirty at this point, trying a million things, but this gives you an idea of the reference vs the attempt at rebuilding. It’s something about that head, but I just can’t seem to land it. I can see the influences look different on the head when I apply the debug layer.

We seem to have landed it with everything else, “I think”, but I noticed the other day that the reference body leans back too and our reconstituted effort doesn’t, so although the body looks right, I don’t think it is correct.

I have got faster at making a playground, though :slight_smile:

Incidentally parenting the head to Armature.Hips.Spine.Spine1.Spine2.Neck.Head transform feels closer on head movement but then nukes the hair.

@potato_noob, let me take some time to dig into your PG and I’ll ping back with what I find.

1 Like

You’re the best! I think we’re maybe running fowl of something in the GLTF mystery box :package: I’ve genuinely learnt a lot, and if it wasn’t for trying everything attempting to get this working, I wouldn’t have learned about weights and indices :slight_smile:

If anything needs cleaning up or simplifying or was unnecessary, humble potatoes are ready to fix :saluting_face:

1 Like

@potato_noob, I think this is what you were going for in terms of bringing in meshes from your asset container and assigning to a duplicate skeleton. I simplified it quite a bit though, so please ignore anything that doesn’t make sense.

The reason the head still doesn’t move with the head bone is because the glb for the head does not have the full skeleton. There is only the neck, head, and head end bones so they are missing the rest of the skeleton which does have an impact on the animation when we target a new skeleton.

There were also some extra things to pay attention to when creating the new hierarchy. Because Babylon is left-handed and glTF is right-handed, there is a -1 scale on Z in the __root__ transform of any imported glTF. To get your meshes to align with the intended animation and placement, you need to add a -1 scale to the abstract mesh rootTransform as well as a rotation of PI so it faces the correct direction. This will make both meshes act the same way and not mirror position of one another.

I believe the last step for you would be to make sure that the head mesh is skinned to the full skeleton and use that glTF to import into your scene and it should work. Let me know if you have more questions.

3 Likes

@PatrickRyan We need to know where to send the ko-fi! We’d never have managed this without your help; thank you so much!

2 Likes

@PatrickRyan You are a freakin rockstar :slight_smile: we really appreciate the help, and let us know your ko-fi!

Your responses were so detailed and well laid out; thanks again for taking us through this process step by step. Have a great weekend!

5 Likes

Thank you, @potato_noob and @Anixan_Skystalker! It’s my pleasure to help out and, as it happens, part of my job. Your questions have helped me as well as I am working on some documentation, demos, and tutorials around skeletal attachments. Digging into what you were doing helped me frame some of the context for that work so it was really a win-win here. I hope you are completely unblocked now, but feel free to ping with more questions as they arise.

1 Like