New feature: GPU Picking

Hey team!

I’m happy to announce a new feature coming for Babylon 8 wave (but as usual already there for you to test ;)): GPU Picking

In a nutshell, instead of using the CPU to scan through all the geometries to intersect with the picking ray, GPU Picking is relying on the GPU to create a picking texture. This means rendering the scene with a unique color per mesh and then finding what color is below the mouse.

The main advantage is performance of course.

Sample code:

    var picker = new BABYLON.GPUPicker();
    picker.setPickingList(spheres);

    scene.onPointerObservable.add(() => {
        picker.pickAsync(scene.pointerX, scene.pointerY, scene, false).then(mesh => {
            if (mesh) {
                text1.text = mesh.name;
            } else {
                text1.text = "";
            }
        });
    });

More info: Mesh Picking | Babylon.js Documentation (babylonjs.com)

Demo: https://playground.babylonjs.com/#B5XGIP#37

32 Likes

Very cool. Thanks.

Very nice !

I like this kind of optimisation (fast picking using custom render pass). Reminds me about some work I did creating some artificial datasets for ADAS, we where rendering ID maps like this, in order to extract object segmentation. Watch out with AA, if by mistake it’s enabled, then the ID map is dead :grin:


About the function wrapping this new feature, does it mean that the output is only the mesh ID ? (no pickedPoint)

Thank you!

Fortunately render target are not multi-sampled by default :slight_smile:

3 Likes

A stress demo:

Run at 15fps on my CPU and solid 60 on the GPU

8 Likes

This is a great new feature. Regarding that pg, would it be possible to expose an onlyBoundingInfo to scene.pick? I modified the prototype in this pg to set it to true, and think it might be a useful option for when exact pick point/faceID aren’t required, and where an inexact pick is acceptable.

1 Like

@Deltakosh this is very cool but honestly this kills the show in many cases:

As the system will use vertex color to store the data, all your meshes with instances will see their vertex color channel overwritten.

Why did you opt for this solution?

You can do it already with the fastCheck parameter of scene.pick

What would be the alternative? Adding a second color channel?

In my case (Digital Twin) we have a lot of same objects, mainly chairs and tables (rendered as instances of course) and I was so happy when I started to read this topic because picking is a bottleneck in my app at this time.

To be more precise why I’m a bit sad after reading the GPU picking docs is because we use the vertex colors this way and it’s quite common in games to fake AO as well:

Anything what doesn’t kill the vertex colors :smiley:

1 Like

fastCheck is a different parameter and doesn’t do the same thing - that just returns on the first mesh hit - it still iterates through all the triangles though (try it in that pg, it’s much slower).

On my side I’m going from 8 fps CPU to 120 fps GPU… Impressive ! :+1:

(up :slight_smile: )

Ok gotcha! I will rework that with a second set of color buffer to not kill the main one:)

Stay tuned!

5 Likes

Yes, at the time it’s only the mesh id.

I guess the class could be extended to be able to also retrieve the point and face id.

3 Likes

@Evgeni_Popov had a lot of great suggestions that I will implement while pleasing @roland as well :slight_smile: Stay tuned ;D

1 Like

oh right! Well we could totally add another option to stop at bbox for sure!

Ok :+1:

Nice :slight_smile: I see an easy way which would be to use the (already existing) depth map to extract 3D pickedPoint (2D coord + Depth Map → 3D coord)

Ok folks!

Gpu pick2 by deltakosh · Pull Request #15175 · BabylonJS/Babylon.js (github.com)

I integrated most of the feedback and went to change one thing: you have to provide the list of objects everytime. The picker will not try to figure out which object to pick

So after that PR will be in this PG will work:
Babylon.js Playground (babylonjs.com)

I’ll update the doc now

7 Likes

@Deltakosh I appreciate it a lot master!