WebXR Handtracking callback when Hand Mesh loaded

Questions
,
1

Hey there,

I want to acces the Hand Mesh. I do that with this code:

const xrHandFeature = xrHelper.featuresManager.getEnabledFeature(WebXRFeatureName.HAND_TRACKING) as WebXRHandTracking;
xrHandFeature.onHandAddedObservable.add((newHand) => {
                // newHand.handMesh // Access handMesh here!
});

Sometimes the handMesh is undefined. I think thats because the hand mesh is not loaded yet. Is there a callback when the handMesh is loaded? There is a callback when a controller Mesh is loaded but this wont trigger for hand meshes. I tried this code:

 input.onControllerAddedObservable.add((controller) => {
     controller.onMotionControllerInitObservable.add((motionController) => {
            motionController.onModelLoadedObservable.add(() => {
                        const xrHandFeature = xrHelper!.featuresManager.getEnabledFeature(WebXRFeatureName.HAND_TRACKING) as WebXRHandTracking;
                        let newHand = xrHandFeature.getHandByHandedness("left");
                        if(!newHand)
                            xrHandFeature.getHandByHandedness("right");
                        // newHand!.handMesh // Access Hand Mesh here!
              });
       });
});

Its also really complicated to get the WebXRHand from a WebXRInputSource. You can acces the XRHand via controller.inputSource.hand but you wont get acces to the WebXRHand. Thats why I have this awkward code where I check for left or right hand. Maybe there is a more elegant way of accessing it which I dont see.

Thank you!

2

cc @RaananW

3

Good question!

As we try to mask the mesh loading during attachment, it is not really possible (with the current state) to know when the mesh itself was loaded. You can be notified when an WebXRHand was generated, but it is not guaranteed that the mesh was already loaded. Two ways to solve this (i mean, there are more, but let’s talk about the better 2 :wink: ):

  1. Load the meshes yourself and provide them as custom meshes. This way you know when the mesh was loaded, and you are loading them before initializing XR.
  2. I can add an observable to the WebXRHand class to notify when a mesh was set (i.e. when setHandMesh was called). This will cover both custom meshes and default hand meshes loading. If this can help you, I will be happy to do that!
1 Like
4

An observer would be really cool. I think its also interessting for people who just want to edit the Material of the hands and Change the color for example. Your first option works too but feels more like a workarround. I will use that for now but an observer would be a great feature.

5

Add observable when a hand mesh was set by RaananW · Pull Request #14967 · BabylonJS/Babylon.js (github.com)

:slight_smile:

2 Likes