So pinching is working with the native XR events, but doesn’t work with the babylon pointer system? might be related to the ray mode type.
i’ll create a playground for debugging later today, would be really great if you can test with it. Thanks a lot!!!
1 Like
Can you run this - WebXR basic example with teleportation | Babylon.js Playground (babylonjs.com) ?
The text above the color picker will change, depending on the controllers available and their ray target mode. The emulator is using transient-pointer, which somewhat make sense, but I would expect, in your case, that the text will be tracked-pointer true tracked-pointer true. My working assumption is that the main component doesn’t return or isn’t configured correctly.
Do you have access to the console?
It says tracked-pointer true when the session loads and transient-pointer true after pinching. I have been having trouble getting the remote inspector to work since the beta update, but I will try again later to see what the console says.
1 Like
i had the same issue with the latest emulator - can’t get it to debug, even in the technical preview. so I use GUI elements to debug. fun!
So they have a controller for the hands, it has a gamepad object, but it doesn’t use this controller to send select events. instead it generates a new controller (transient pointer, like in the emulator). it makes little sense to me, but we will need to adapt, i guess. i’ll think of a smart way to deal with that. theoretically it should already work, as we now also support transient pointers, but in screen space (emulator) and not the controller space. i assume that if you go close to the color picker you will be able to control it, or look at the floor and hold pinch you will be able to teleport. need to debug this…
Ok, I think I have a PR that will make this work. fingers crossed? 
This page will be available in a few minutes: WebXR PG Pointer events . It is the same playground with an extra flag - to force-use the standard webxr events instead of the processing babylon does. This is not yet in core, but if it works I will make sure to merge it and update the version.
BTW - the safari technical preview’s latest updated allowed me to debug the vision os session today, I hope it helps you as well
1 Like
Hi Raanan - I wasn’t able to select anything from the GUI from the AVP!
thanks a lot for reporting! I assume you are running it on 1.1 beta?
Is there any trigger working? Teleportation ring for example? Can you also verify that you see transient-pointer when clicking, but before it is tracked-pointer?
This playground seems to have the same behavior as before unfortunately.
The teleport ring also doesn’t appear in either example and holding pinch while targeting the ground doesn’t fire the teleport. I also can’t interact with the gui with the raycast, and since you cant walk around in VR mode with the AVP I can’t get close enough to try the near interaction just yet. I can modify this example to try that later. Ill also try out the safari technical preview to see if I can’t get you some logs.
1 Like
What i wonder is why do they create a new pointer when they already have a tracked-pointer in place. I wonder what are the transient-pointer coordinates in the pointer system frame loop and whether or not it even detects a click correctly. I will gladly give a few pointers if you get it to work (and if you have the time, of course).
This PR (and the playground associated with it) falls back to pure WebXR events (selectstart and selectend), which we have already made sure are working. It is using them to trigger a pointerdown event if the specific controller that has generated this event has collided with any pickable element in the scene (GUI, floor, etc’). It feels like the real solution here would be to use the tracked-pointer if there is an event generated by a transient-pointer, but that feels wrong to me. Maybe i’ll try hacking it in an provide you with a new link so you can test it. i’ll see what i can do.
1 Like
Ok - this PR will check if my suspicion is correct. I am sorry I have to do it this way, and I thank you SO MUCH for the time you invest in this
This page will be available once the PR is built. What it does is take the select event from the transient pointer and uses it on the first-added tracked-pointer controller, which I assume is one of the hands. I can’t tell if it is the right hand or left hand, as I can’t control the order in which the controllers are added.
Also, if you can test this one -
https://babylonsnapshots.z22.web.core.windows.net/refs/pull/14808/merge/index.html#9K3MRA#1498
This should change the color on pointer down/pointer up. Would be interesting to know if the pointer events are even triggered, just not in the right direction.
Thanks again!!!
Happy to help! I want to see this working just as much as you guys, so I’m happy to test it if it’s helpful!
I am still having trouble with the debugger, even with the technical preview, so I can’t share logs yet.
However, that playground seems to be working almost correctly!
Pinching with either hand does fire the teleport circle, it also changes the pointer color to blue. The near interaction with the GUI is also working.
Seems like only one pointer gets created on the hand that loads first (it switches sometimes), and it doesn’t switch hands to the one that pinched last. When I pinch to teleport, there is a new raycast that seems to be coming from the origin of the camera. Here is a video of what it all looks like: Watch RPReplay_Final1708985713 | Streamable (will expire in a few days)
That second example doesn’t seem to work, pointer doesn’t change color when pinching but the sphere does switch colors.
2 Likes
Oh wow, apple, what have you done…
Thanks, this video has been a huge help in understanding the behavior. This is what I get so far (for future me):
- The hand tracked-pointers are casting the ray correctly (though it is coming from a weird position of the hand, which is a decision made by apple).
- The tracked-pointers don’t react to input events at all. Not the gamepad’s main component (trigger-button) and not the webxr selectstart nor the selectend events.
- The tracked-pointers do have a gamepad object, but it is useless (see point 2)
- When tapping the fingers, the system triggers a transient-point, which, if i understand correctly, originates from the camera and not the hand that triggered it. More of a gaze behavior and less select with hands.
- My current hack does not distinguish between left and right hands. I will need to test if the transient pointers have their handedness defined, and if they do, find the right tracked-pointer and trigger a click there.
- The transient pointer’s ray (the one that babylon is generating) needs to be hidden. No need to display it.
So, I have two suggestions for solving this. The first is this:
Hands are only there for near interaction and tapping. When tapping the fingers, gaze-mode is triggered, and you select what you look at, and not what your hands are pointing at. It seems like this is what apple is expecting us to do, as the transient-pointers are triggered from the camera as an origin.
This will require changing the basic pointer system, removing the ray emitted from the hands, and ignoring the tracked-pointer completely.
The second:
Fight the apple implementation and pair transient pointers and tracked pointers. Which is the way every other implementation has worked thus far. This will only work, assuming that apple provides handedness to the transient pointers.
My gut feeling is to implement both (with configuration flags), but it will be quite hard without a device. Let’s start with the second one, as I am half-way there
P.S. - what I still find hard to process is the fact that the second ray, which as you say originates from the screen, is always hitting the center of the teleportation ring. It might be the way I am constructing this ray (taking the end point as the center of the ring) so I am ignoring it for now.
I think your intuition is correct here. Gaze and tap is the main and really only model of interaction on the vision pro right now so it makes sense they are translating that to webxr experiences. However gaze and tap is going to be horrible until eye tracking in implemented for webxr and also won’t play well with cross device experiences.
So supporting both styles in the long term I think is the right course of action but option two is the more attractive option in the short term.
1 Like
working on a solution
2 Likes
Looking at this reddit post is making me wonder if apple is setting the transient pointer position with the gaze data from the devices eye tracking and not just the center of the camera position.
Can you check this? Should work for both hands now - BabylonServer (windows.net)
So apple doesn’t support a pointer move while in eye tracking. Interesting. i’ll continue tinkering around, see how we solve this nicely
Seems to be a privacy thing, fair enough but why not just let me give consent for that data idk. They want to handle gaze data and object highlighting at the OS level. Not sure how they will achieve this for webxr (or if they will bother) but I guess time will tell. Triggering lots of pointer events to achieve this seems like a bad idea. Happy to test any examples you have tonight.
1 Like
I might be understanding this wrong, but maybe they are using their own hit-tests and returning an result when you look at something WebXR Device API - Hit Testing | hit-test I really need to get my de-buger working so I can snoop around lol.
oh, interesting. Hit testing is a ray against the real world, so I am not sure how they expect us to intercept it in scene without changing the way hit-tests work, but… I am never surprised by an apple decision.