I am developing immersive VR applications based on web and babylonjs. Below is an example of a grabbing object interaction. You can view it using the immersive Web Emulator.
He works fine with quest equipment. But but I changed the view device to vision pro and I found that onButtonStateChangedObservable is not working properly.
I can see that when I pinch my index finger and thumb together, there are no rays in the scene but white dots appear on the grid indicating pickups.
Am I to assume that babylonjs is getting the finger pinch event but not triggering the onButtonStateChangedObservable correctly, or am I using it incorrectly? 
Correction, 7.14.0 you can see the white circle indicating ray pickup when you pinch your finger, and the latest 8.8.5 version of babylonjs white circle is not present.
This is because of the way interactions work in the vision pro.
The vision pro deals a bit differently with pinch and movement - the selection is gaze-based and not hands-based. And this is the way it works with babylon as well. so a pinch does not mean a selection
I can check why this specific observable is not triggered (i assume it is because apple doesn’t provide the event), but you should probably use the pointer selection extension and not the component directly, especially when using the visio pro.
Thanks for your reply.
Should I use the WebXRControllerPointerSelection class to unify the grabbing object functionality across different devices, is there a sample playground for reference.
From the performance of 7.14.0, babylonjs handtracking component UI does react to pinch gestures.
Right, but it can’t track the location. there is no ray from the hand, so you get the pinch event, but not wht was pinched. this comes from the gaze.
In version 7.14.0, the circle attached to the mesh surface when pinching the fingers together is not an eye-tracking focus point, but rather a point picked up from a ray shot from the hand as in QUEST’s hand tracking.
I think now with vision pro, hand pursuit hand rays with gesture pinch event support should all work fine.
The old implementation didn’t work correctly because of lack of support from apple. if they have changed something we can see what to do. The old implementation didn’t work with both vision os 1 and 2. What version of OS are you using?
The latest version of vision OS 2.5, on safari with version 7.14.0 of babylonjs, makes circles that look suspiciously like ray pickup indicator points appear when pinched by the finger.
But the question is - does the ray come from the hands, or from the gaze - what happends if you pinch with the hand at a lower or higher position?
It comes from pinching fingers together, not gazing.
It was stated earlier that in vision pro hand tracking mode, only the hand grid is shown, not the secondary tracking ray.
But when I pinch my thumb and index finger together, a circle appears on the pickable grid in front of me, at which point I move my hand’s orientation and the circle follows.
It looks consistent with the hand tracing ray on quest pro.
I reconfirmed the version number as vision os 2.5 (220473)
babylonjs version number is 7.14.0
Here’s the schematic.
1 Like
I can look into it, sure. Can’t promise a timeline, sadly.
1 Like
That’s okay, I’m doing the technical research phase right now. If there’s anything I need to do to work with it, message me. 
1 Like
