WebXR Hand Functions Like Grabbing in Quest 3

Questions
1

Hello,

I just saw new release of Babylon and saw new WebXR functions, which makes me excited a lot! But I couldn’t find anything in documentations close to what I need.

I have Quest 3 device. Using WebXR I simply want to grab a box and drag it with hand controllers. I don’t want to use controllers or pointers, it should be just hand interaction.

2

cc @RaananW

3

@RaananW Created a Demo with Handtracking and Grabbing here: https://playground.babylonjs.com/#9M1I08#166

He basically checks for pointer events because the Pointer Down event with handtracking is like grabbing something (Index Finger and Thumb touch). In the PG these are the Lines 214-224 and 164-190.

When I had to implement Grabbing the Pointer Event didnt work out for me so I “manually” programmed grabbing. I get the Meshes of the Fingers I need and check the distane between them. When they are really close together and intersect with a Mesh I trigger my Grab event. Afterwards set the parent of the grabbed object to one of the fingers so they follow your hand.

let isGrabbed = false;
const xrHandFeature = xrHelper!.featuresManager.getEnabledFeature(WebXRFeatureName.HAND_TRACKING) as WebXRHandTracking;
xrHandFeature.onHandAddedObservable.add((hand) => {
       let indexFinger = hand.getJointMesh(WebXRHandJoint.INDEX_FINGER_TIP);
       let thumb = hand.getJointMesh(WebXRHandJoint.THUMB_TIP);

       scene.onBeforeRenderObservable.add(() => {
               let distance = Vector3.Distance(indexFinger.position, thumb.position);

                if(isGrabbed == false && distance < 0.03 && (indexFinger.intersectsMesh(this.collider!) || thumb.intersectsMesh(this.collider!)) ){ // Check Collision with Mesh to be grabbed!
                       // Object is Grabbed!
                       isGrabbed = true;
                       this.root!.setParent(indexFinger); // set parent of object to be grabbed to the indexFinger
                 }
                   
                 if(isGrabbed == true && distance > 0.03){
                       // Object no longer Grabbed!
                       this.isGrabbed = false;
                       this.root!.setParent(null);
                  }
       });
});

There is also the SixDofDragBehavior that allows Grabbing but I didnt work with this so far.
Documentation: Babylon.js docs
Playground: https://playground.babylonjs.com/#AZML8U#230

There are not many posts about Handtracking and Grabbing in the Forum or the documentation, maybe we can change that.

3 Likes
4

@Simon_Weck this is great answer! I tried each solution you provided and all of them worked. I believe hand tracking is very important functionality after Quest 3 and Vision Pro releases. I would like to see more documentations about them definitely.

2 Likes
5

nice, I think in the most cases SixDofDragBehavior is the way to go. It has callbacks when something is grabbed and also smoothes pointer jittering.

6

Yes, I was thinking the same. Is this SixDofDragBehavior was existed before or it comes with Babylon v7? If there is any recent improvements about WebXR functionalities, I would like to test them.

7

SixDofDragBehavior exists since 2021 or something, its not a new feature.

Babylonjs 7.0 didnt really add groundbreaking WebXR things. Most of the important Features were already available. Some cool things they added recently are these:

  • Full-screen GUI
  • Touchable UI elements. You touch Babylonjs GUI Elements with your fingers which is really cool for near interactions
  • Ability to use hands and controllers at the same time
2 Likes
8

Amazing answer :slight_smile:
thanks a lot for that.

SixDOF should work correctly with grabbing (single hand grabbing for now). If it doesn’t, I will be very happy to debug this. Two hands tracking is in the works!

I am planning to work on near interaction documentation which is still partly missing in the docs. There are a few flags, tips and tricks that might help in those scenarios.

9

Great news!

I started to use SixDOF so definitely will share if I see any problems/bugs during the process.

1 Like