Babylon.js and Mediapipe pose detection

Hello Community
I’m working with Babylon.js and Mediapipe pose detection in React.js and I have a problem show poses in Babylon.js.

initial Babylon.js scene

  const initScene= ()=> {
      engine = new Engine(canvasRef.current, true, { preserveDrawingBuffer: true, stencil: true });

      // This creates a basic Babylon Scene object (non-mesh)
      scene = new Scene(engine);
      scene.clearColor= new Color4(0,0,0,0)

      // This creates and positions a free camera (non-mesh)
      camera = new FreeCamera('camera1', new Vector3(0, 0, -1), scene);

      // This attaches the camera to the canvas
      camera.attachControl(canvasRef.current, true);

      const light = new HemisphericLight('light1', new Vector3(0, 1, 0), scene);
      light.intensity = 0.7;

     rightSholder = MeshBuilder.CreateBox("rightSholder", {size:0.5}, scene);  
  }

In detection results, I want to change position of Mesh

  const onResults=(results:any)=> {

    if (results.poseLandmarks) {
      const rightSholderTrk = new Vector3(results.poseLandmarks[12].x, results.poseLandmarks[12].y , 1)
      rightSholder.position  = rightSholderTrk;
    }
  }

not sure what is not working but a repro would help, maybe your camera is not looking at the cube ?

1 Like

Hello just checking in if you’d like any more help and if you can provide a repro

A bit late to this thread, but thought this might help: 3D Full-body pose estimation playground

1 Like