Whenever performing a pinch gesture, the fingertips overlap. Another user has experienced the same issue.
This behavior does not occur with Meta’s native hand mesh, nor in the Three.js or A-Frame demos I tested.
Babylon.js (Playground):
Meta:
Three.js (three.js examples) :
A-Frame (A-Frame Examples):
cc @PatrickRyan - can you think of a reason for that? size of the finger/thumb maybe?
@RaananW, could be in the position of the bone within the finger mesh, but that should only account for some small overlap. This looks like we are either off on the pose of the thumb or we are off on the position of the wrist and the thumb and forefinger joints are compensating, but poorly. Looking at the angle of the thumb as to moves toward the wrist, it suggests the wrist bone is not aligned with the user’s wrist tracking point. The other hand looks misaligned in pose as well. Could this be bad tracking or are we somehow off in our skeleton origin?
we are running the same tracking as the other frameworks, so i will not expect that to be the issue. everything is possible though. we are converting RHS data to LHS which can always cause some issues. think it might be related?
I mean it’s always a possibility when we are converting data, right? I guess what I am seeing in the two that show the video frame and the mesh is that the mesh orientation from the wrist is more aligned with the video frame in the Meta experience than in ours. The wrist bone in ours is positioned in a very different position from the video frame. That said, we are looking at one frame and don’t know if this is a tracking problem on a single frame or an offset. I will have to dig out my headset and give this PG a try.
I thought about testing WebXR generic hand models in Babylon.js and downloaded them from here:
WebXR Generic Hand models
I only had to set rigMapping correctly and generate a new UV map in Blender using Project from View (Bounds) so that the babylon hand node material would work somewhat correctly.
For comparison, I put the generic hand model on the left hand and Babylon’s hand model on the right hand. The previously mentioned issues can be seen in these images:
Here’s both imported in Blender:
The generic hand is the same one used in other 3D engines, and it works really well. I suggest adding it as the default hand model in Babylon.js. Or make it an easy option.
Here’s the workaround for anyone interested:
import handsShaderUser from "../../assets/json/handsShaderUser.json"; // babylon.js hand mesh shader
import lHandRhsGlb from "../../assets/glb/left_new-uv.glb";
import rHandRhsGlb from "../../assets/glb/right_new-uv.glb";
protected async importHandMeshes(scene: Scene) {
const handMaterial = NodeMaterial.Parse(handsShaderUser, scene);
handMaterial.name = "handMaterial";
const tipFresnelColorBlock = handMaterial.getInputBlockByPredicate((b) => b.name === "tipFresnelColor");
const fingerColorBlock = handMaterial.getInputBlockByPredicate((b) => b.name === "fingerColor");
if (tipFresnelColorBlock && fingerColorBlock) {
tipFresnelColorBlock.value = fingerColorBlock.value;
}
this.leftHandMesh = await this.importHandMesh(scene, lHandRhsGlb, "leftHand", handMaterial);
this.rightHandMesh = await this.importHandMesh(scene, rHandRhsGlb, "rightHand", handMaterial);
}
protected async importHandMesh(scene: Scene, handGlb: any, handName: string, handMaterial: NodeMaterial | StandardMaterial): Promise<Mesh> {
const handImportResult = await SceneLoader.ImportMeshAsync("", "", handGlb, scene);
const handMesh = handImportResult.meshes[0] as Mesh;
handMesh.name = handName;
handMesh.scaling = new Vector3(-1, 1, 1);
// dispose and replace imported material
const handMeshChild = handMesh.getChildMeshes(true)[0];
handMeshChild.material?.dispose();
handMeshChild.material = handMaterial;
return handMesh;
}
const handTrackingFeature = featuresManager.enableFeature(WebXRFeatureName.HAND_TRACKING, "latest", {
xrInput: xr.input,
jointMeshes: {
invisible: true,
disableDefaultHandMesh: true,
handMeshes: {
right: this.rightHandMesh?.getChildMeshes(true)[0],
left: this.leftHandMesh?.getChildMeshes(true)[0],
},
// WebXR standard hand rig mapping
rigMapping: {
right: [
"wrist",
"thumb-metacarpal",
"thumb-phalanx-proximal",
"thumb-phalanx-distal",
"thumb-tip",
"index-finger-metacarpal",
"index-finger-phalanx-proximal",
"index-finger-phalanx-intermediate",
"index-finger-phalanx-distal",
"index-finger-tip",
"middle-finger-metacarpal",
"middle-finger-phalanx-proximal",
"middle-finger-phalanx-intermediate",
"middle-finger-phalanx-distal",
"middle-finger-tip",
"ring-finger-metacarpal",
"ring-finger-phalanx-proximal",
"ring-finger-phalanx-intermediate",
"ring-finger-phalanx-distal",
"ring-finger-tip",
"pinky-finger-metacarpal",
"pinky-finger-phalanx-proximal",
"pinky-finger-phalanx-intermediate",
"pinky-finger-phalanx-distal",
"pinky-finger-tip"
],
left: [
"wrist",
"thumb-metacarpal",
"thumb-phalanx-proximal",
"thumb-phalanx-distal",
"thumb-tip",
"index-finger-metacarpal",
"index-finger-phalanx-proximal",
"index-finger-phalanx-intermediate",
"index-finger-phalanx-distal",
"index-finger-tip",
"middle-finger-metacarpal",
"middle-finger-phalanx-proximal",
"middle-finger-phalanx-intermediate",
"middle-finger-phalanx-distal",
"middle-finger-tip",
"ring-finger-metacarpal",
"ring-finger-phalanx-proximal",
"ring-finger-phalanx-intermediate",
"ring-finger-phalanx-distal",
"ring-finger-tip",
"pinky-finger-metacarpal",
"pinky-finger-phalanx-proximal",
"pinky-finger-phalanx-intermediate",
"pinky-finger-phalanx-distal",
"pinky-finger-tip"
]
},
},
} as IWebXRHandTrackingOptions) as WebXRHandTracking;
The uv-modified generic hand models and babylon hand node material:
webxr generic hand in babylonjs.zip (130.8 KB)
I think it’s not entirely correct to compare WebXR with what happens in a native application.
The difference between Treejs and BabylonJS can be indicative, but only if the gesture and lighting conditions are the same (it seems quite dark in the frame).
I’ve noticed similar overlaps quite often, but I associated them with poor lighting in the scene.