Multiple skeletons for single entity with multiple meshes?

Questions
1

Hey guys,

I’m having an issue where my character models aren’t correctly updating their bounding boxes. The character models are cloned skeletons and meshes that are animated (animations work perfectly), and when I use the mesh.refreshBoundingBox(true) method to update the bounding info for picking, the results will only be correct when all of the character models are performing the same animation. When the animation for one character changes, then the bounding box is distorted for any mesh doing a different animation.

I’ve been working on reproducing it in a Playground, and came across this: Use Bones and Skeletons - Babylon.js Documentation
It has an example where each mesh has a new skeleton cloned just for it:
dudes = [];

for (i = 0; i < 10; i++) { // 10 clones
    var xrand = Math.floor(Math.random() * 501) - 250;
    var zrand = Math.floor(Math.random() * 501) - 250;

    var c = [];

    for (j = 1; j < newMeshes.length; j++) {
        c[j] = newMeshes[j].clone("c" + j);
        c[j].position = new BABYLON.Vector3(xrand, 0, zrand);
        c[j].skeleton = newMeshes[j].skeleton.clone();
        scene.beginAnimation(c[j].skeleton, 0, 120, 1.0, true);
    }
    dudes[i] = c;
}

I’m wondering if it’s necessary to clone a new skeleton for every single mesh and then run the same animation on each skeleton? It seems like you should use a single skeleton (for each entity) for all of the sub meshes and just animate it once.

I have no idea if this is the cause of my problem either, so any ideas in that regard would also be super helpful! (Working on a PG)

2

You need to clone the skeleton if you want to run different animations for the different cloned meshes because the bone matrices are linked to the bones which are linked to the skeleton: because of this you can only apply a single animation to a skeleton.

In the dude example we could have reused the same skeleton for all clones because the animation played is the same. But if you wanted to change the speed ratio (for eg) on a per clone basis, you would need to clone the skeleton:

https://playground.babylonjs.com/#W1RP6N

Note that in the dude case, the model is made of 5 different meshes but they all use the same skeleton. So, when cloning, you can use a single cloned skeleton for all of the 5 meshes (that’s what I did in the PG above).

1 Like
3

Right. So in your PG (like in my game), you’re using a skeleton for each entity, and each mesh gets linked to that skeleton. In the example on the site, there is a skeleton cloned for every mesh. It sounds like my understanding is fine but the example uses extra skeleton clones unnecessarily.

Back to square 1 on my original problem :joy:

Any ideas why the refreshBoundingInfo() would result in incorrect bounding boxes if a separate cloned skeleton is performing a different animation?

4

Weird, I enabled the bounding box outline and it appears they are correct, but still the meshes aren’t picking…

picking
picking527×817 79.3 KB
not picking
not picking715×545 43.5 KB

The second image isn’t displaying the player’s username under the dot (center of the screen). When a player is hovered, it should display the name like in the first image.

It seems like it happens when the character is performing different animations than other characters, and only seems to happen when you pick from behind the character…

5

That’s hard to tell without a repro…

In this PG it seems to work as expected:

https://playground.babylonjs.com/#W1RP6N#2

Maybe you can update the PG above to load your character and see how it comes?

6

Okay, I whipped up a PG for the issue!

At first, the original player model is loaded in, and you can see the picking works decently (doesn’t update with the animation because I didn’t update the bounding info). I just made the ground change colors if a mesh is being picked.

After 3 seconds 3 new player models spawn in. I don’t update the bounding info, so you can see the bounding boxes are as if they’re in their default pose.

After 5 more seconds I update the bounding info for the spawned players (I could continually update it, but the problem exists either way). You’ll notice that the bounding boxes update on all of the spawned in meshes, but for the sleeping players it won’t accurately pick.

https://playground.babylonjs.com/#YA8851#8

Let me know if you see anything!

7

I’m starting to think this might just be a bug?

8

The thing is that the bounding boxes are used by the picking engine for a fast check against a mesh, but if the ray does intersect the box after this check, the engine goes further and looks for an actual intersection with the geometry.

The problem with skeleton animated meshes is that the geometry is the initial geometry, none of the transformations applied by the animation are reflected into the geometry because the computations are done in a vertex shader and so are not visible from the javascript code. So, the geometry intersection test is done against the untransformed geometry, that’s why it does not work as expected.

What I have just said is true if the bone computations are done in the vertex shader, which is the default. However, you can instruct the engine to perform those computations on the CPU (mesh.computeBonesUsingShaders = false), in which case everything will work as expected because the actual geometry data are really updated according to the animation:

https://playground.babylonjs.com/#YA8851#9

Two important things to make this work:

  • you should call mesh.refreshBoundingInfo() with a false parameter (or no parameter at all) and not with true, else you will apply two times the deformation from the skeleton, as the geometry data are already transformed by the animation!
  • you should call makeGeometryUnique on your cloned meshes, else they will all share the same geometry and so they will all be animated with the same animation

Of course, doing it this way is less performant than doing it on the CPU…

One possibility if you still want to use GPU computation is that we add a flag on the picking method so that it stops when the bounding box intersection is ok and does not go to the geometry intersection checking stage. The check won’t be pixel perfect but it could be enough in your case. I will make a PR for this as it is really easy (it’s already taken care in the computation but this flag is not visible from the user side).

Note that for both cases above to work, you should refresh the bounding info each frame as the methods rely on those boxes being right. It’s not so much a problem in the first case as the geometry data on the CPU side is already transformed by the animation, so it’s only a matter of scanning those data to extract the min/max vectors. However, in the second case, you will need to call refreshBoundingInfo(true) which will apply the skeleton transformation to the geometry data before the scan, in effect cancelling the benefit of doing the computations on the GPU… So, your option here would be to pre-compute the bounding boxes for each frame of each animation and simply sets the right bounding box depending on the current frame of the anim (the first case would also benefit from this, of course).

You have another option which is to use GPU picking. It’s more involved, you can look for more information in this forum, for eg: Speeding up scene.pick() on renderloop

1 Like
9

Wow. Thank you so much for the help!

I will temporarily use option #1 (as the live version of my game needs a working solution), but I will get to work on the GPU picking solution. The GPU picking post you linked is my post from earlier this week about an entirely different issue I’m having :sweat_smile: so now I have a classic two-birds-one-stone situation. Unfortunately, I know literally nothing about GPU picking, so this is going to be painful, but I will let you know how it goes :joy: :joy:

10

Be aware of the GPU picking solution: you need to get the texture from the GPU to the CPU to read back the color at the mouse location.

For your first problem where the mouse is always at the center of the screen you can make this fast because you can render in a very small texture, so reading this very small texture back is not necessarily a problem, but in your second case where you want to locate the mesh currently hovered by the mouse you would need a much bigger texture (not necessarily the size of the screen, but still big enough for the picking to be accurate). Your best bet here would be to retrieve the texture asynchronously so that it does not stall the pipeline, but that would mean you would get the result deferred by a number of frames, which may or may not be a problem for you.

11

Well, Babylon.js is so nicely done that it already exists: just use pickWithBoundingInfo instead of pick :slight_smile:

12

Wow thanks! That could be just the solution I’m needing - not use up all CPU even if it isn’t perfectly accurate. Sadly it looks like I need the preview version to use it, so I’ll have to work through all of the breaking changes to get it working. I imagine that will be easier than figuring out GPU picking at least :joy: