I’ve submitted a PR for this issue here:
master ← rebeckerspecialties:webxr-hands-optimization-snapdragon-ar2
opened 12:45AM - 10 Jan 26 UTC
This work was driven from gathering WebInspector performance profiles from the f… orthcoming Snap Spectacles device.
The killer on that device, which is actually rooted in WebKit in a way that's not platform-specific, was the texture upload for the hand tracking. Changing this to a uniform array update increased framerate, reduced latency, and reduced >30ms frame times on both Snap Spectacles *and* Apple Vision Pro. AVP was powerful enough to plow through this inefficiency in simple circumstances, but we saw framerate drops in app telemetry that we hadn't investigated. This fix solves both on-device measured issues. I've also advised platform holders to optimize WebKit so vertex animation texture uploads per frame (a primary way people scalable animate large crowds of high fidelity avatars).
There's also some minor GC and L1 cache optimizations that were also identified in the WebInspector profile data.
### WebXR
- **Optimized:** Hand tracking performance significantly improved by removing per-frame allocations in [updateFromXRFrame](cci:1://file:///Users/matt/src/Babylon.js/packages/dev/core/src/XR/features/WebXRHandTracking.ts:453:4-554:5).
- **Optimized:** Implemented L1-cache friendly matrix conversion strategy for hand joints, improving performance on AR/VR mobile chipsets (Snapdragon AR2/XR2).
- **Optimized:** Disabled `useTextureToStoreBoneMatrices` for default hand meshes to eliminate costly (in WebKit) `texImage2D` uploads on every frame, utilizing Uniform Arrays instead.
- **Fix:** Ensure zero-copy updates when using Thin Instances for hand tracking joints.
Before this fix, hand tracking was a few seconds behind and very low framerate on Snapdragon AR2 devices like Snap Spectacles. It turns out that this root problem was also the cause of our app’s frame time spikes on Apple Vision Pro, which we saw during longer sessions in user telemetry. Happy to discuss review/options here, but wanted to get the integration-tested change up for a reaction.
PS: GitHub repo asked me to label the PR, but I don’t have permissions to do that.
4 Likes
Tagged the PR for you
2 Likes
It’s Friday so it may take a little for people to get to review your PR, but thank you for this!