I also want to achieve such a function and look forward to your success 
2 Likes
Unfortunately I haven’t had any success yet solving the issue with the GPU Picker method, but will let you know if I do!
Thank you 
Hi!
Even though I have a sh*tload of work to do I’ve managed the multiPickAsync function to work correctly. 
@deltakosh I’ll tide up the code to be DRY compliant a bit later. It’ll stay in Draft until I do so. I’ll fix a bug in the existing pickAsync code which doesn’t pick anything when y === 0 as well.
Example PG running at 120 FPS on M1 Max.
3 Likes
Correct.
Can we see a full PG please?
1 Like
It’s just loading that model into the last playground I shared. I’ll see tomorrow if I can get permission to share the model.
The main overlap issue is also visible in your new playground with the dragon model though (nice work getting multiPickAsync set up btw!) - even labels in front of the model go transparent when there’s part of the model directly behind them (even though they’re closer to the camera). And so far I unfortunately haven’t been able to work out a way to fix it that doesn’t require raycasting, which would defeat the whole point of the GPU picker lol
Here’s an idea, modify the shader so that it can take in depth textures, calculate the camera distance in the vertex shader, and extract the distance in the depth texture in the slice shader for comparison to determine the final rendered alpha value.
I think it should be faster than GPUpicker because it doesn’t need readPixels.
2 Likes
I agree however in case of picking single, gpuPicker reads only one pixel and in case of multi pick the minimum area of pixels needed to pick the meshes is read from the RTT. So it doesn’t read pixels for the whole screen.
In addition the RTT is scissored on the gpu so only the this area is filled with pixels by the fragment shader.
So it’s quite optimized.
1 Like
Because of the time-consuming nature of the readPixels themselves, there is actually a rough scene complexity threshold for GPUPicker versus Ray; the higher the scene complexity and the more single-model facets there are, the more obvious the GPUPicker advantage is, and conversely Ray is a bit faster.
Reading data from GPU to CPU is a last resort, if the whole process can be realised on GPU, it will be the fastest option.
Ah - I’d briefly considered doing something similar to that but thought the performance hit would be too much. But it looks like with your new multiPickAsync it actually runs decently!
It does however seem to get a bit confused sometimes - do you think that’s an issue with multiPickAsync or with something else? As you can see, from this angle for example M0 and M19 go transparent even though their spheres are clearly visible. The picker seems to be returning null for them, which doesn’t make much sense 
It also has some issues if any of the labels are out of view (especially visible on mobile as it’s a narrow screen) which seem to be related to how multiPickAsync is implemented? The picker doesn’t pick anything past the first one that is out of view - so in this example it’s only picking up to 13, meaning that 16 - 19 aren’t being picked even though they’re visible.
Thanks again for all your help on this thread. I’m sure you must be busy so I appreciate you taking the time 
1 Like
That does sound like a good efficient solution (and shaders definitely seem to be how e.g. Sketchfab does it) - unfortunately my knowledge of shaders is too little to implement it lol. I’ll see if I can maybe find some time in September to learn a bit about modifying shaders in babylon…
But thank you for the idea, you’ve given me somewhere to start from if I do have time to look into it at some point 
1 Like
Surely it’s a case of using RenderingUtilityLayers, potentially having another scene to handle your dynamic 3D UI (always render over the top, bind input/camera, send events between scenes for hover etc). You could make use of scene depth and texture2D if you need further occlusion.
Without digging, it looks like the SketchFab viewer doesn’t care for the label order and maybe they’re rendering the labels scene twice: semi-transparent for the foreground, opaque for the background that’s occluded by the primary scene/mesh.
1 Like
I’ve noticed this behavior too when trying to zoom very close to the model and some of the picking coordinates went outside the canvas a few minutes ago.
I bet there are some issues, this is just the first working version. I’ll test it in various situations and fix all bugs I/we find so please keep posting your comments.
You’re welcome! Sometimes I have to deal with something other than work, because it would screw me up and then I make some time for these contributions.
I’m not sure if there is not an issue with the dragon model itself so I’ve crated the following PG for testing:
2 Likes
The issue seems to present itself at specific angles in that playground as well. Seems to happen mostly for labels near the top of the screen See label 0 here:
I believe the issue was caused by this line which sets alpha to 1 only when it picks the helper sphere:
marker.targetAlpha = mesh?.name.startsWith("sphere") ? 1 : 0.3
If it doesn’t pick anything the alpha must be set to 1 as well:
marker.targetAlpha = (mesh?.name.startsWith("sphere") || !mesh?.name) ? 1 : 0.3
Give it a try too please! Thanks!
1 Like
Unfortunately still seems to be happening. Here for marker 19 it’s picking box 12 for some reason (which isn’t the box just to the left)
Damned LOL!
I think the multi gpu pick stuff is working but we are sending wrong 2D coordinates sometimes. I’ve moved the picking to another observable.
Could you try this one please?
1 Like
Seems to still be happening. It’s only at quite specific angles though, so the right coordinates seem to be being sent most of the time…
And right at this angle 19 is visible when it shouldn’t be… But a few pixels left or right it correctly disappears.
EDIT: Ah - it could be just when some of the markers are off-screen, I’ve just noticed that as that stops all subsequent labels being picked it’s also affecting it…
EDIT 2: Well nope it’s not just that, they’re all on-screen here. 22 is picking as “undefined” for some reason…
And with all on screen still, box 19 seems to be incorrectly picking as the box next to it (which 17 and 18 are correctly covered by):
Oh, man. I need to setup a test PG with meshes at fixed positions and with more elaborate logging so I can see WTF is happening here.
However I spotted an issue. 40 markers but the result array contains only 17 values. I’ts because it is dropping the coordiantes outside the canvas and doesn’t push an undefined to the result.
I’ll get back to you in an hour/two.
Thanks for helping me to debug!
2 Likes