Test unicode | Babylon.js Playground (babylonjs.com)
In this example, the result of the mesh picking after deformation is still the same as before. How do I get the correct result for picking?
Picking happens before the morphs are applied to the mesh so iirc it’s not possible to get the morph-deformed result, @Evgeni_Popov is that right?
You could try GPU picking instead which would have the vertices already deformed, there’s a few topics on this forum about it if you search
Indeed, see Using forceSharedVertices() to enable ray picking breaks my morph targets for eg.
It seems to be just as I guessed.
I also read the article on GPU pickup schemes.
However, since detailed coordinates and normal direction need to be obtained at that time, ray picking can be optimized using Octree. I gave up delving into GPU pickup schemes.
Read this article：Using forceSharedVertices() to enable ray picking breaks my morph targets，You mentioned that you can use the depth buffer to obtain 3D coordinates, so is there a way to calculate the normal.
Yes, you can calculate the normal from the depth buffer by reconstructing a position then doing the cross product between the x and y derivatives. See Improved normal reconstruction from depth – Wicked Engine Net for eg.
2 Likes
Test unicode | Babylon.js Playground (babylonjs.com)
I created a case on playground to implement depth picking, but it seems to have precision problems, and there are abnormal normal lines after camera rotation and translation.
I can’t find the cause of the problem. 
GPU picking point and normal | Babylon.js Playground (babylonjs.com)
I have fixed the error in normal calculation. It occurs because the screen coordinates are mirrored with the depth texture coordinates on the y axis.
Now, the world coordinates and normal calculations are correct.
But the accuracy error still exists. In order to reduce the influence of precision error on normal calculation, I have to increase the distance of sampling coordinates in normal calculation.
This raises a new problem, when the boundary case occurs, the normal calculation will still be wrong.
Do you have any suggestions?
What do you mean by “boundary case”? Is it when the samples used to calculate the normal are not from the same object? If so, the link I gave above (Improved normal reconstruction from depth – Wicked Engine) addresses this problem.
Also, everything can be simplified if you use the geometry buffer renderer to render the normals directly into a render texture. This way you don’t need to calculate the normal, you just read it from the texture:
This example uses the geometry buffer position texture to get the 3D position of the pixel you clicked on the screen (search for gbr). You can easily get the normal by querying gbr.getGBuffer().textures[1]. And the depth can be retrieved by gbr.getGBuffer().textures[0].
1 Like
GPU picking point and normal | Babylon.js Playground (babylonjs.com)
I tried using gbr.getGBuffer().textures[1] to get the normal, but the result is still wrong.
It seems to have something to do with the attitude of the camera？
Using the geometry buffer renderer (gbr) was actually a bad idea…
The morph target does not update normals, only positions, so the normals produced by the gbr are not correct (plus they are in view space, so you would have to multiply them by the inverse matrix of the view camera to get them in world space). This will also fail if you use custom vertex deformations that are not handled by the gbr.
So using only the depth buffer to reconstruct the normal is more robust and will always work, as long as the objects write to the depth buffer (and it will also perform better).
GPU picking point and normal | Babylon.js Playground (babylonjs.com)
I followed the solution in the article to handle the boundary case, and now it works a lot better.
It’s still not perfect. 
If you force the depth buffer to be 32 bits it’s a lot better (third parameter of enableDepthRenderer):
2 Likes
GPU picking point and normal | Babylon.js Playground (babylonjs.com)
offsetBase is set to 1 and the effect is almost perfect!
Thanks for your help!
1 Like