Hi, its been years since I resumed my work with babylonjs hyped by the new MMD support. My Idea was to be able to make an VR/XR video editor so for example i can record action using my mmd model. I was trying to pose a model in VR but seems like ray picking is not working. I added a debugger and tried extending length, checked that every mesh is pickable etc… now im out of ideas!
For starters the idea was just to identify the correct bone. It works when clicked by mouse on pc when vr is not available and extension disabled.
Can you give me some tips? I will share the project folder.
Project: mmd-vr-editor.zip - Google Drive
For Quest 3: npm run host
For browser testing using: https://chromewebstore.google.com/detail/immersive-web-emulator/cgffilbpcibhmcfbgggfhfolhkfbhmik
cc @RaananW, as it seems it’s a VR problem only. If you can provide a minimal repro in the Playground it would probably help a lot.
1 Like
Here the playground: https://playground.babylonjs.com/#FY1L15#32
The ray doesnt work even on the ground, it will work only if I point straight down.
Maybe it has to do with ray distance? But I tried rising it and it didnt solve yet
@RaananW @Evgeni_Popov @qq2315137135 @noname0310
The strange thing is that the rays seems to be hitting the correct objects too! I can see the correct objects when logging them…but the .hit is a strange beast sometimes it sets pickedMesh to ground when pointing straight up 
scene.pickWithRay(ray, (mesh) => {
let valid = mesh.isPickable && mesh.isVisible;
if (valid) console.log(mesh.name, mesh.isPickable, mesh.isVisible);
return valid;
}, true)
The picking issue is not an issue with the MMD loader, I don’t have a clue
I’ve never really used Babylon.js’s Picking feature well, but to the best of my knowledge.
Even if the picking is fixed, if deformation is applied to the Skinned Mesh, it will not pick correctly from then on.
Just an idea for an implementation. Normally, most animation tools are designed to select the Skeleton GUI overlay when selecting a bone for a Skinned Mesh.
Even MMD does
I hope your issue is resolved and I apologize for not being able to help
1 Like
Let’s simplify the playground:
You are running 3 render loops and calling scene.render twice. Performance-wise it won’t be a good idea, and it is, in general, not needed.
Another thing - there is no need to construct your own rays from the controllers, though you can do that, of course. They way to do that would be to use the controller.getWorldPointerRayToRef
function, which is a helper function that generates a ray for you using the controller’s information.
And a last thing - the playground would not have worked outside of VR as well, as the bone names that you want to include are incorrect. You are not picking bones, you are picking meshes . And the meshes have totally different names. So for the time being i removed the usage of the array though you should decide yourself how to do it.
I hope this helps!
2 Likes
Looks like some progress! Teah instead of pickingMesh I’m using vertextData to get the bone i just skipped that part in the playground to make it simpler. Will update it now to see if it works. Thanks man.
2 Likes
Yep. Seems to be working! https://playground.babylonjs.com/#FY1L15#37
Still some mapping to fix but its the right way. Now the problem is another.
Once the part moved, i cant click it again to move another time.
But I can still moving if i click again on the area where it was before.
I tried running some update, marking dirty and disable enable the mesh yet the problem persists.
With a lil of patience I got a MikuWheel!
Havok Physics workin too! https://playground.babylonjs.com/#FY1L15#39