Using scene.pick on a joint driven mesh

Questions
1

Hey all, I am working on a scene where I need to identify a material on part of a rigged object when the user clicks on the screen.
I have implemented scene.pick which works great in its default pose. However, when I deform the mesh during animation it still tests the mesh based on its default pose instead of where the geometry actually is.
Is there something I can do to test the mesh after it has been deformed by the joints?

2

Hiya CG, welcome to the BJS forum.

I’m far-from an expert in armatures/rigs -picking… but… look at this playground:

https://www.babylonjs-playground.com/#1UZIZC#8

Line 47 is continuously-updating the bounding box on a mesh. Now, to constantly update the bounding box of an animated rig… might be different. You might want to do that to the “root” of the rig.

AND, it could be performance-heavy. Perhaps do it ONLY-at onAnimationEnd observation/callback (but before user-picking is needed).

I’m just doing some guessing, here. :slight_smile: I hope this helps. More/wiser comments likely coming soon.

3

Thanks Wingnut, yeah I am thinking the solution is something along these lines as well. It seems that somehow I need to update the vertex buffer or something like that. Just am not 100% sure exactly what I need to do yet as the rig is fairly complex.

4

Did .refreshBoundingInfo() not work?

5

No not for what I am trying to do. Basically I have a mesh that is rigged, that mesh in turn has sub-meshes. It is the sub-meshes I am trying to identify on click.

Here is an example of the project I am working on. Basically I am trying to identify if a user clicks on one of the tabs. If the book is closed then “scene.pick” works fine and I am able to get the sub mesh of the tab, but if the book is open and try and click on the tab it does not identify that it has been moved by the joints.

Example Link

1 Like
6

Yeah but you can refreshBoundingInfo on the submeshes right?

7

I have tried that before with no luck. Perhaps I am missing something though. I have gotten it to work on meshes, just not submeshes.

8

Huh, I wonder if there is something obstructing the pick.

Or maybe just dig into it more: PickingInfo - Babylon.js Documentation

Try .subMeshId?

9

I am pretty sure there is nothing obstructing the pick as it actually picks the ground plane which is behind the tab. What is interesting is that the pick works prior to deformation and does return the subMeshID. But if I move the submesh with the joint and click on the new position it does not read the submesh, but if I click in the old position of the submesh it returns the submesh id as if it was still there in the old spot.

Here is a playground example of what I mean. https://playground.babylonjs.com/#EHL0MQ#2

If you change the x value of joint 0 by a value of 2 for example, you will see that the correct id is no longer printing in the console. However, if you click where it used to be it will still print out.

10

ok so I’ve found a way to help you (and probably @MackeyK24) and whoever wants to pick animated bones :slight_smile:
Here is the trick:

mainMesh.refreshBoundingInfo(true);
    var pickResult = scene.pick(scene.pointerX, scene.pointerY);
    if (pickResult.hit) {
        console.log('picked submesh id: ', pickResult.subMeshId);
    }

How it work:

  • You must call mesh.refreshBoundingInfo(true) on all candidate meshes
  • This will force the CPU to update the bounding info based on the deformation created by the bones. Please note that even if it as fast as possible it will still consume CPU so please be cautious
  • The picking will then use the updated version of the bounding box and will let you precisely pick your submeshes

This will work thanks to this commit: Allow picking to work with GPU bones · BabylonJS/Babylon.js@a28bf37 · GitHub

It will be in the nightly in a hour or so

2 Likes
11

Thank you so much Deltakosh, you guys are really amazing. We will try it out and provide feedback. :slight_smile:

12

documentation updated as well:Use Bones and Skeletons - Babylon.js Documentation

13

Just wanted to follow up. I tested it on the playground with a heavily rigged model. It seems to work great.

Playground Link: https://playground.babylonjs.com/#EHL0MQ#3

1 Like
14

I like when a plan comes together :slight_smile:

1 Like