Action Manager on Animated Mesh Model not working as expected

Questions
1

https://www.babylonjs-playground.com/#2C9HSM#1

I have a model built similar to BrainStem, and I am trying to attach an Action Manager to the root mesh, so that whenever the mouse hovers over some part of the total model, an action occurs.

For some reason, actions can occur, but not in places that I thought they would. For the root mesh, it occurs around the feet. For all meshes, they seem to occur around the legs.

Any ideas why it is firing in seemingly random spots and how to fix? Preferably a fix that is performance friendly as well.

Thanks!

2

Pinging @PirateJC

3

I found this in the docs. Use Bones and Skeletons - Babylon.js Documentation

However, even trying to force the update, I don’t get the expected results.

https://www.babylonjs-playground.com/#2C9HSM#2

4

Hey @Devin_Wright - Great question!

What are you expecting to see exactly? An event fires at the center of mass of the character? An event fires over any piece of the character?

5

The latter preferably. And it would be ideal if it could fire with respect to the mesh as it is animating, although I would also be ok with firing when the mouse is over the entire meshes bounding box as well.

6

Sorry for the delay, this one was pretty tricky to think through. A lot of credit goes to @Deltakosh for helping me wrap my head around this.

Ok so here’s the problem. Babylon optimizes skeletal animations, by passing the work of the animation to the GPU. This means that when a mesh is animated by a bone, the CPU doesn’t actually inherently know that the mesh’s position has changed. So you have to update the CPU with the mesh’s new position. Why? Because in the case of the observables, which you’re using here, the CPU is what’s doing the work to “listen” for pointer/mesh interactions.

SO what happens…well as far as the CPU is concerned, the mesh is lying flat on the ground, not animating at all.

You can see that here in this playground:
https://www.babylonjs-playground.com/#2C9HSM#3

The animation is technically there, but I’ve disabled skeletons in the scene to represent what the CPU is seeing. So this now makes sense as to why the action manager appears to fire events at the “feet” of the robot. It thinks the robot is lying down.

SOOOOO what we have to do is update the CPU on the mesh’s new position each frame, as you point out in your second playground. However, this is a HUGE red flag, because telling the CPU where each vertex is of every mesh in the scene is CRAZY expensive.

So this means we actually need some sort of proxy for the CPU…bounding boxes is probably the best way to go about this.

Here’s a playground with the bounding boxes of each mesh being updated on the CPU every frame.
https://www.babylonjs-playground.com/#2C9HSM#4

So the good news is that we can get 75% of the way there by getting the CPU to have updated bounding info for every mesh in the scene.

So the final 25% would be using something like a "scene.pickBoundingInfo() to get the mesh that was selected based on the bounding box under the cursor when a click occurred. The bad news is that such a helper function doesn’t exist in Babylon…YET.

I’ve added this issue to add this helper to the core engine!

Sorry for the long winded response, but it’s been a tricky one to wrap my head around.

I am by no means an expert on this subject, but I also wonder if there are easier ways to go about this. Perhaps exploring raycasting for example?

https://doc.babylonjs.com/babylon101/raycasts

Hope this is helpful! Thanks a million for finding a gap in the platform! Definitely something we should add!

2 Likes
7

Thanks for the great response! I came to a lot of the same conclusions based on all the docs I read. Before your response I did actually go with a very crude solution of wrapping the mesh in a bounding box, and just using box for the hovering. This makes it so the CPU doesn’t need to be updated every time.

Very glad to see how the CPU sees the mesh, I could not figure out how to figure out how it was doing it’s scene picking.

I tried messing with ray picking and I think it would be basically the same. I did notice that there is a fastCheck for scene.pick that supposedly uses boundingInfo, but I couldn’t get it to work as expected with your example either.

Once that new helper is in, I will update to try it out, but for now looks like the current simple solution is my best bet w/o spending a lot more time on the problem.

Really appreciate your help!

1 Like
8

This code will work with next nightly:

function createScene() {
    var scene = new BABYLON.Scene(engine);

    var camera = new BABYLON.FreeCamera("camera1", new BABYLON.Vector3(0, 5, -10), scene);
    camera.setTarget(BABYLON.Vector3.Zero());
    camera.attachControl(canvas, true);

        let childMeshes = [];

    BABYLON.SceneLoader.ImportMesh("", "/playground/scenes/BrainStem/", "BrainStem.gltf", scene, function (meshes, particles, skeletons, animations) {
        scene.createDefaultCameraOrLight(true, true, true);
        scene.createDefaultEnvironment();


        childMeshes = meshes[0].getChildMeshes();

        for(let i = 0; i<childMeshes.length; i++){
            childMeshes[i].showBoundingBox = true
        }

        scene.onPointerDown = function(evt) {
            let pick = scene.pickWithBoundingInfo(scene.unTranslatedPointer.x, scene.unTranslatedPointer.y, (m) => m.showBoundingBox);

            if (pick.hit) {
                console.log(pick.pickedMesh.name)
            }
        }
    });

    scene.registerBeforeRender(() => {
        for(let i = 0; i<childMeshes.length; i++){
            childMeshes[i].refreshBoundingInfo(true);
        }
    });


    return scene;

}
1 Like
9

The relevant code!

1 Like
10

Speed is not something Babylon struggles with! LOL

Awesome! Thanks @Deltakosh!!!