Quest 3 webxr body tracking: hand/forearm orientation can initialize incorrectly

Bugs
, , , ,
1

We are seeing what looks like a startup-time orientation bug in the Babylon.js WebXR body tracking path on Quest 3.

Setup:

  • Device: Meta Quest 3
  • Browser version: 146.1.0
  • OS version: 2.3.1026
  • Reproduced on older Quest browser / OS versions as well
  • Using the Babylon.js body tracking playground example: https://playground.babylonjs.com/#0FOISU#2

Repro steps:

  1. Open the Babylon.js body tracking playground example. https://playground.babylonjs.com/#0FOISU#2
  2. Before entering XR, hold the right arm forward with the palm facing down.
  3. Keep the left hand behind the back so it is not initially tracked.
  4. Enter XR.
  5. Observe the avatar’s right hand.
  6. Then bring the left hand into view in the same pose as the right hand.

Observed result:

  • The avatar’s right palm is initialized with the wrong orientation. In our case it faces left instead of down.
  • When the left hand starts tracking later and is brought into the same pose, the avatar’s left hand is oriented correctly.

Expected result:

  • Both hands should produce the same avatar orientation for the same real-world pose, regardless of whether the hand was already visible at XR entry or started tracking later.

What workarounds we already tried without success:

  • Delaying avatar setup or changing setup order around XR/session entry
  • Waiting for both hands before attaching the avatar / body tracking
  • Rebinding or rearming tracking after hand visibility changes
  • Wrist-specific orientation / aim corrections
  • Forearm-roll / chain-level correction attempts
  • Session-local per-side roll offset calibration ideas
  • Per-side delayed arm activation based on hand-tracking stability

If you could take a look when convenient, we would appreciate it. If this is ultimately a Meta Quest tracking/runtime issue rather than Babylon.js retargeting, guidance on that would also help so we can report it to Meta with the right framing.

A Quest 3 video showing the repro: https://drive.google.com/file/d/1sIRHHFloGa1IX9IIbysb3INhJB5RTY2Y/view?usp=sharing

An annotated screenshot from the video with the palm axis drawn to make the orientation error easier to see:

Screenshot from com.oculus.vrshell-20260526-114405-0.mp4
Screenshot from com.oculus.vrshell-20260526-114405-0.mp41920×1080 90.5 KB

2

cc @RaananW

3

checking now. Thanks for reporting!

4

I am waiting for the snapshot on this PR to fully test it:

EDIT - tried to reproduce according to your instructions, and it seems to be solved with this PR.

5

Thanks for the quick reply. I tested the fix and it seemed better now, but I noticed that the avatar’s hand goes upside down (a 180-degree roll offset) if I keep my palm up when I enter XR. The same thing happens if the left palm points left at the start or the right palm points right.

Repro video: https://drive.google.com/file/d/1j0nLBnqOZh9s5VpHP6SQ997X7fbRJDQq/view?usp=sharing

Screenshot from com.oculus.vrshell-20260527-113955-0.mp4
Screenshot from com.oculus.vrshell-20260527-113955-0.mp41024×1024 60.7 KB

6

Just bumping this thread. Regarding the second issue, I wanted to make sure it didn’t get missed since I haven’t heard back yet. Thanks for looking into it!

7

Sorry, was away for a week. i’ll look into that later today :slight_smile:

8

Can you check this snapshot

Does it still happen?

9

I was only able to test this now as it has been a really busy week.

It still happens. If this is the correct URL: https://playground.babylonjs.com/?snapshot=refs/pull/18530/merge#0FOISU#2

No errors in the console. Except the unrelated GL_INVALID_OPERATION?

logger.ts:107 BJS - [09:04:38]: Babylon.js v9.10.1 - WebGL2
index.js:52 Loading rigged character…
index.js:52 WebXR body tracking enabled.
Model: HVGirl.glb (Mixamo rig)

Put on a headset that supports ‘body-tracking’
(e.g. Meta Quest 3), and press ‘Enter VR’.
The avatar will mirror your body pose.
monaco-DbnHoJcc.js:1030 [Violation] ‘requestIdleCallback’ handler took 62ms
babylon.js?t=1780560326366:1 [Violation] ‘requestAnimationFrame’ handler took 242ms
/?snapshot=refs/pull/18530/merge#0FOISU#2:1 [.WebGL-0x7400767e00]GL ERROR :GL_INVALID_OPERATION : glFramebufferTexture2DMultisample: ← error from previous GL command
index.js:52 Body tracking started.
index.js:52 Body tracking ended.
10

Will try to reproduce this. i couldn’t initially, but i understand it is still present.