I agree, it would be explicitly out of the scope of hit testing but could work in theory…guess we can quickly check if hit-testing is even available.
Unfortunately, this is no longer working. Pinch changes the text in the gui but that’s it.
Which means that they are not sending handedness in the transient pointer. Which also means that there is no way of knowing which hand triggered the event, so we are only left with the apple option - gaze mode madness.
One option would be to emulate hand events from the pose like play canvas does for XrHand (line 225). In that reddit post above you can see the box being added to the individual hands after pinching, I am assuming that might be emulated now. Another approach would be something like handy.js which uses euclidian distance to estimate the correct pose.
I tried keeping a separation between the pointer emulation system and the hand feature, but it seems like it will be impossible to avoid checking tap on our own. this might take a little longer than I wished it would 
1 Like
@dsaffo , I have a few questions, if you get the chance.
I am trying to understand why this PR didn’t work as expected - BabylonServer
This one forces webxr events to any controller added, and should react correctly to transient pointers. It does work correctly in the emulator, which emulates transient pointers in a specific direction. What I wonder is whether the direction data is missing, or that we are reading the controller incorrectly. If you are able to connect to safari debugging session, it would be great to know a few things (and of course if you have the time) -
What are the values of an event (startenter) of a transient pointer that should be a successful hit (even looking at the floor should be enough). And (and this one is a bit more tricky) - what is the pick ray (and result) of the transient pointer - origin (or - what does babylon define as an origin), and direction. I will be happy to explain how to get that
1 Like
I am out of town this week but if no one else beats me to it I will try it out and report back as soon as I can next week! I can also ask around for some other vision pro owners if they can contribute.
1 Like
I am back but can’t seem to get the remote debugger to work no matter what I try, seems I am not alone but not a lot of guidance on how to fix it yet: Can't web inspect Safari in Vision… | Apple Developer Forums
1 Like
@dsaffo
I had this problem too. This steps helped me:
- computer and headset should be in one wi-fi network
- activate pairing of devices and check all settings for debugging for computer and headset Safari
- !!! update software for AVP (it was 5 days ago)
- reboot AVP and options ‘developer mode’ appears (it was miracle
)
1 Like
Unfortunately, I’ve gone through all these steps but still no dice. I am going to try re-installing xcode and if that doesn’t work a factory reset on the AVP.
I believe that in my case, updating the software was key.
Because I did the same thing before, but without success.
And even when I already had a successful debugging session, I lost this opportunity for no apparent reason. But after rebooting the device, it began to work more or less stably.
Device people, I wonder if you can test this:
WebXR basic example with teleportation | Babylon.js Playground (babylonjs.com)
My general assumption - looking at the sphere and tapping should work. Try looking straight, put the sphere at the center on the viewport. If a box is being shown on the sphere, try moving your hand while still holding the tap. The box should be visible until you pass the sphere.
Afterwards, try putting the sphere on the side of your viewport and look at it (kind of eye-tracking thingy). Check if it still work.
I thank you ALL for helping me with this one
Box was appears on the sphere surface when i aim by left hand or right hand (in second attempt) and do pinch.
Not succeed with eye-tracking (probably i do some wrong)? All behavior still the same.
1 Like
Ray is originating from the wrist on pinch. Here’s what that looks like: Watch RPReplay_Final1710384330 | Streamable
After debugging this quite a lot, I Believe i found the best way to integrate it into Babylon.
Having a device would 100% make it faster 
We have click action, and we have a weird form of gaze control. The reason I am saying weird is that it is impossible to show the user some form of a target BEFORE they are tapping their fingers. So the user will have to guess (!!) that they will select what they are looking at. Very bad UX, but there is no way around it.
I am able to get the ray for picking, and I am able to get it correctly. There is some form of a race condition in the current code that prevents from the pointer-down simulation to work correctly. Will require a bit of change in the code, but this will be done, hopefully, today and I will be able to share another playground (probably a snapshot playground) that has input on the vision pro working.
1 Like
This playground should work. Looking at the color picker and tapping will allow you to select a color. Looking at the sphere and tapping will rotate the sphere (it is low-poly, so you should be able to see its rotation). Otherwise, teleportation should work as well. The rays from the hands should be invisible. Picking is done by looking at things. PointerMove events should also work by tapping and moving the hand while still looking at the same point. TBH - I have no idea what happens if you look at something else while holding a tap. But that’s a different story altogether.
if it doesn’t (AAAAHHHHH), playground version #1569 should work. And if this one doesn’t as well, then #1570
Getting a 404 on those playgrounds
Trying this out, the PG doesn’t load on my AVP or iPad. I don’t get a 404, it just spins indefinitely (5+ minutes). Same for '69 and '70.
Link was missing the snippet id
1 Like
GOOD NEWS, THOSE ALL WORK!
Heres what it looks like: Watch RPReplay_Final1710518975 | Streamable (expires in 2 days)
The initial ray position will be at the eye gaze when you first pinch and if you hold pinch you can move the ray around.
It doesn’t come through in the video but the ray originates from the center of the camera which is fairly distracting. Maybe in this mode it’s better not to have a ray at all but we can configure that later.
2 Likes