WebXRHandTracking on a mobile browser

Questions
, ,
1

i see in the docs ther is documentation for WebXRHandTracking

i also see the note about it being it being specialized hardware (understandable for a good experience with it).

is there a way i can get it to work on a regular mobile browser?

i can use the tensorflow implementation for hand tracking into to my app here. i would like to add a VR hand into the BabylonJS AR environment of my app as seen here. its a fairly basic implementation of BabylonJS cubes and textures.

is there anything like WebXRHandTracking already available for mobile browsers? im sure it wont be as good as using on vision pro, but BabylongJS AR works quite well on phones from a few years ago. it would be nice to have hand detection for interactions.

2

cc @RaananW

3

The hand tracking ability comes from WebXR. If the browser implementation supports the extension, we support the extension. Otherwise - your implementation would have to emulate the WebXR API for the hand tracking feature to be enabled.
I hope this makes senseā€¦

Of course, using external implementation without the WebXR hands tracking feature is possible, if you implement the data parsing yourself

4

thanks for the reply. thats unfortunate. iā€™ll see if i can figure anything out.

1 Like
5

To follow up, I recently open sourced my project and I wonder if I could get some guidance as described here: Reddit - Dive into anything

6

My presence on reddit is little to none :slight_smile:

I can answer your questions here, if itā€™s ok!

Babylon does have hand meshes. they are right/left hand models, based on the WebXR specs. If you are able to manipulate the mesh using the 25 points required for WebXR hand tracking, you can ā€œemulateā€ it. The WebXR specs can be found here - immersive-web/webxr-hand-input: A feature repo for working on hand input support in WebXR. Feature lead: Manish Goregaokar (github.com), and the code that adjust the hand meshes using this data is here - Babylon.js/packages/dev/core/src/XR/features/WebXRHandTracking.ts at master Ā· BabylonJS/Babylon.js (github.com)

7

great! thanks.

1 Like