New feature: GPU Picking

Announcements
42

Hello everyone,

I am very excited about the appearance of this feature and immediately started using it in my project upon learning about it. However, when I call pickAsync, the console gives the following warning. I can confirm that it is caused by texture baking animation. I have been blocked by this issue for a few days, and although I have been reading the source code to find the problem, my technical level is relatively low. So, I would like to seek some help here.

image
image1562×780 33.4 KB

I created a PR: feat: Add support for GPU picker for vertex animation textures. by zhangyahan · Pull Request #15322 · BabylonJS/Babylon.js · GitHub

5 Likes
43

Great PR! thanks a lot! Please remove the package.json and I can merge it :slight_smile:

3 Likes
44

yes, i removed

45

There is another issue, which is the problem of picking up on screens of different resolutions. The input x and y need to be multiplied by window.devicePixelRatio

gpuPicker.pickAsync(
  scene.pointerX * window.devicePixelRatio, 
  scene.pointerY * window.devicePixelRatio
)
1 Like
46

This looks cool! Would this work with material.PointCloud = true, on meshes with vertex data only? Or does it require triangles?

47

This requires triangles (for now)

48

Please do the PR my friend!

1 Like
49

There is also a index issue when picking up thinInstance
PR： fix: change devicePixelRatio to hardwareScalingLevel and fix thinInstance pick index

1 Like
50

How could we use this in a scene that also contains 2d gui in 3d space where the gui occludes and gets occluded? If i were to add them to the gpupicker list, they don’t get their input events.

51

Unfortunately this is not supported. gpuPicking can only pick meshes not faces or precise pixels

52

A two pass system wouldn’t be too hard to do, right? First get a mesh using the gpu picker and then get the details on the cpu? The cpu then only would have to check against a single mesh which is probably still great for performance. Especially if you can get the details only when the mesh is one you care about.

I guess we can already do that with the scene picking predicate, but not sure if the ui events would still get fired correctly

53

yes that will work. CPU picking will provide all the required plumbing

54

I had to try a few things, but got it in a place where the gpu picker helps/overwrites the default picker

This method still allows any default behavior to still work, like UI or the scene.meshUnderPointer.

One thing I’m still kinda missing is being able to use a different gpu picker material for certain meshes. For example, to be able to click through the transparent areas of a material (just clipping based on the texture alpha). I might look into a solution another time, maybe will do a PR.

2 Likes
55

For getting point and normal, a depth map is needed according to GPU picking point and normal example - #35 by xiehangyun, so, in theory, it should be possible to use a MultiRenderTarget to get a depth map and a color texture in the same time.

1 Like
56

https://playcanvas.vercel.app/#/graphics/area-picker
Like PlayCanvas, could we add the box selection feature, or even beyond that, the capability to select a specific mesh shape?

I think this could be implemented using Stencil fairly well.

It would be very useful to create something like an editor

57

@Deltakosh Thank you so much for working on this, the performance is amazing.

Is there any chance to create a method that receives a Ray? I really want to use it on my VR apps!

Edit: I know this method needs a screen coordinate one idea that just occurred to me would be rendering the scene from the ray perspective as if it was a camera, so x and y would be the center of the texture.

58

Yes this is a good way to do it :wink:

As you stated this method works in screen mode so a ray is a bit tricky without doing what you already mentioned

1 Like
59

Will GPUPicking become the default picking for ActionManagers?

2 Likes
60

Probably not but I could add an option to enable it

2 Likes
61

GPU Multipicking added:

3 Likes