Octree submesh picking returns incorrect subMeshId

Hello,

While attempting to use the submesh octree to speed up submesh selection, I came across what appears to be a bug in the submesh octree picking info result.

Here is a reproduction where if I am or am not using the octree, I get different results for the same ray when calling scene.pickWithRay: Babylon.js Playground

Based on the documentation for PickingInfo::subMeshId I would expect that subMeshId would return the id of the submesh as indexed by Mesh::subMeshes. However, when using the octree to accelerate selection, it returns the index of the submesh candidates that the octree narrowed down from the full set of possible submeshes. This means if I am using subMeshId to index back into Mesh::subMeshes, I will get an incorrect subMesh. In the above example, the octree returns 0 for both submeshes where-as without the octree it returns 0 and 1.

Based on my digging, the reason I get a different result is that the abstractMesh::intersects code calls down to the scene to intersect with its octree. When the octree is not being used, this function returns the list of all submeshes on the mesh. When the octree is being used, this function can return a subset of the submeshes. The index while iterating through this list will be relative to the candidates being searched and not absolute to the entire list of submeshes which lead to my confusion.

Please let me know if this is an actual bug, or if I have misunderstood something and what the ideal workflow is.

Thanks!

1 Like

cc @amoebachant

Actually I’ll check it nevermind :wink:

Nice one!!! Thanks @Scott_Baldric
Fix submesh identification with picking octrees by deltakosh · Pull Request #16924 · BabylonJS/Babylon.js

3 Likes

Awesome!

Thank you for the quick turn around. :slight_smile:

1 Like