WebXR Hand Tracking Error on Safari/Vision Pro

Bugs
1

Hello,

I am running into a an error specifically when I try to add XR-hand-tracking on safari on the Apple Vision Pro :
“undefined is not an object (evaluating ‘this._xrSessionManager.session.enabledFeatures.indexOf’)”

What’s strange is the hand tracking example from the playground runs just fine, and my code also works just fine on chrome and the quest browser.

Not sure what the best way to go about debugging this is but here is the relevant code and es6 imports

import { Engine, Color3, WebXRFeatureName, Scene, WebXRHandTracking} from '@babylonjs/core';

const babylonEngine = new Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true,  disableWebGL2Support: false});


var defaultXRExperience = await scene.createDefaultXRExperienceAsync( { floorMeshes: [env.ground]}  );

if (!defaultXRExperience.baseExperience) {
  console.log("No XR")
  } else {

    const featureManager = defaultXRExperience.baseExperience.featuresManager;

    if (!featureManager) {

      console.log("No Feature Manager")

  } else {

      defaultXRExperience.baseExperience.featuresManager.enableFeature(WebXRFeatureName.HAND_TRACKING, "latest", {
          xrInput: defaultXRExperience.input
      });
      
    }
}

babylonEngine.runRenderLoop(() => {
  scene.render()
})

window.addEventListener("resize", function () {
  babylonEngine.resize();
});
2

Heads up @RaananW , I am getting the same error here when trying to enter Frame from the AVP!

image
image2080×210 50.6 KB

3

This was fixed already, should already be available in the latest version

That’s why it works in the Playground :grinning:

3 Likes
4

Interesting , I thought I was on that version when I tested! I don’t see it in the 41 release notes either. Will test again later tonight!

1 Like
5

I am not sure why I marked it to skip the changelog TBH - Vision OS missing enabledFeatures by RaananW · Pull Request #14757 · BabylonJS/Babylon.js (github.com) . But it should be there. if it isn’t let me know, i’ll release a patch

6

I didn’t see it working in the 41, a patch would be awesome if you can swing it @RaananW

1 Like
7

checking and fixing

2 Likes
8

@RaananW @gabrieljbaker Its working for me now with version 6.41.1 thanks!

BTW happy to help test features or document whats available/missing for the AVP in the future.

3 Likes
9

@RaananW also confirming that worked nicely and I see my hands heh.

But I can’t get teleportation to work with them yet! Have you tried?

10

nice!

A question (as I couldn’t quite get the emulator to work correctly) - can you select using pinch gesture? If you can, try selecting the floor (the mesh you defined as floor) for 3-4 seconds, and check if the teleportation circle is showing. Would be great to know if that works :slight_smile:

11

@RaananW Pinch does not appear to trigger a select (the ray cast doesn’t change to blue like other platforms), and the ray cast currently comes straight out the center of the hand mesh instead of from between the thumb and pointer finger as it does on the quest. There is no teleportation circle showing up, either.

12

This data is vendor specific and is defined by the browser. That means that Apple defined the grip location differently than meta.
I don’t have a device and the emulator doesn’t support the hands feature, so i sadly can’t test it at the moment. I’m still trying to figure out what the best course of action is

13

The best course of action is that vendors send the babylon team a device! :smile:

3 Likes
14

Somewhere, in an office in Cupertino, someone is laughing out loud reading this! :wink:

1 Like
15

If you know how to log this information, I would be happy to run a script and record it.

1 Like
16

Let’s start with the simplest! :slight_smile:

WebXR basic example with teleportation | Babylon.js Playground (babylonjs.com)

I wonder if this simple scene works. It should change colors when you pinch (blue) and when you release (yellow). If that works, it means that the do support native WebXR events, but they dont support the hands fully. I will be able to patch babylon if I know this one works. I would expect the hands to show and move correctly as well.

17

I had to change the example a little to get it running : https://playground.babylonjs.com/#9K3MRA#1482

The hands show up and track well, and a raycast appears through the center of the hand from the wrist. However nothing happens on pinch. Here is the log:

Screenshot 2024-02-12 at 5.58.13 PM
Screenshot 2024-02-12 at 5.58.13 PM1992×1300 291 KB

2 Likes
18

Hmmm. That means Apple didn’t implement the most basic WebXR event. selectstart and selectend are not Babylon events, it’s pure WebXR.
OK. I’ll think of a better debugging Playground and will share it here.

5 Likes
19

I was experimenting with vision OS 1.1 beta just now. It seems like selectstart and selectend have now been implemented. the emulater, though, does not support hand tracking.

Has anyone tried 1.1 beta? any experience with proper hand tracking?

20

I can confirm the above example is working on os 1.1 beta 3. The ray cast is still off, but pinching will turn the sphere blue as intended.

1 Like