Single camera, in browser motion tracking/rendering with babylonjs

Hi everyone, I’m now here but I have been using babylonjs to build one of my projects. I figured it may be helpful to anyone who wants to do something similar, so let me share what I have.

Basically what I did is rendering a humanoid model with babylonjs, and at the same time using a third-party library (MediaPipe) to do the motion capture. I wrote the code that put all the parts together, convert pose data from MediaPipe to skeleton/bone rotations and applying them to the rendering part. I also made a demo site showing how to embed everything within a UI framework (ReactJS).

Here is the two repositories: v3d-core (phantom-software-AZ/v3d-core) is the part that is responsible for rendering, and v3d-web is the part that converts MediaPipe data to skeleton rotations.

Here is the demo site:

If you like the project, leave a star or sth, thanks!

4 Likes

Woot this is great buddy!

cc @piratejc

Love your project! I want to play with it someday to become a cute character :rofl:

This is pretty awesome! Keep up the good work!