WebXR Multiplayer with hand tracking does not work correctly in left handed scene

Questions
1

Hi everyone,

I’m making a multiplayer webxr experience using babylonjs (scene is using left handed system). I’m using Hand tracking feature instead of the motion controllers. For remote players, I’m spawning a cube for the head, and for the hands I’m spawning the default webxr hands provided by babylonjs. I noticed that if i use scene.useRightHandedSystem = true; the webxr data comes correctly and the remote hands syncs perfectly. But when i use in left handed system..the custom hand seems off/skewed. Any idea what’s happening? What am i missing here? Attaching a screenshot and 2 videos for reference.


Code sample:

      xr.baseExperience.onStateChangedObservable.add(async (newState) => {
        if (newState === WebXRState.IN_XR) {
          const testPlayer = new TestPlayer("test", this.scene!);
          await testPlayer.spawn();

          sessionManager.onXRFrameObservable.add((xrFrame) => {
            const pose = xrFrame.getViewerPose(sessionManager.referenceSpace);
            for (const input of xr.input.controllers) {
              const xrHand = input.inputSource.hand;
              if (!xrHand) return;
              for (const joint of xrHand.keys()) {
                const space = xrHand.get(joint)!;
                const jointPose = xrFrame.getJointPose?.(space, sessionManager.referenceSpace);
                if (!jointPose) return;
                const position = jointPose.transform.position;
                const orientation = jointPose.transform.orientation;

                const node = testPlayer.tNodes.get(`${joint}_${input.inputSource.handedness === "right" ? "R" : "L"}`);
                if (!node) return;
                node.setAbsolutePosition(new Vector3(position.x, position.y, -position.z));
                node.rotationQuaternion = new Quaternion(orientation.x, orientation.y, -orientation.z, -orientation.w);
              }
            }
          });
        }
      });

Screenshot 2025-04-26 153856
Screenshot 2025-04-26 1538561627×599 148 KB

2

cc @RaananW

3

This happens because the webxr data is right-handed. Babylon is converting the data to be able to show it in left-handed systems. This is why we abstract the WebXT hand concept in our own class. You can use the webXXR hands feature to achieve that, or run the same conversion we are running to support LHS

4

I already did the conversion. I’ve attached the code that I’m currently using. After doing the conversion only, the hands look like this.

5

Are these our hand meshes? There is one more transformation you will need to do (and this is why I still recommend using our implementation for simplicity :slight_smile: ). When loading the mesh in LHS you will need to transform the 2nd mesh received by the import function:

if (!scene.useRightHandedSystem) {
   handGLB.meshes[1].rotate(Axis.Y, Math.PI);
}

We also link transform nodes to the bones while keeping the joint meshes independent. And keep them both updated on each frame.