FaceTrack using tensorflow (ml5)

As we prepare for the metaverse, I am getting myself more familiar with machine learning ai libraries like tensorflow, here’s a face tracker using ml5.js (a library under tensorflow) in conjuction with babylon.js

try it out ML5--tensorflow

compliments of the season

10 Likes

Pretty cool !!! mine is still stupidly sitting in there: https://playground.babylonjs.com/#H19VH7#42

4 Likes

This is really amazing :smiley: :+1: :+1:

i see this uses bounding box to place the helmet, nice!!

Wow, pretty neat!!!

1 Like

Thanks!!

aah - I was wondering who had made that playground - I found it on a curated babylon list. I wrote a Virtual try-on demo recently and that PG was super helpful - thanks for sharing!! I noticed that you had commented out looking up/down. I was looking at other 3d space solvers and then got distracted!

1 Like

Very cool. Does it give any depth information about eye position or its just x,y?

there is a position z defined but no idea how precise it is

1 Like

@kri100s
Yeah the z position is not that bad, i have updated the demo with a z coord

https://ml5-tests.glitch.me/#faceMesh

2 Likes

I used a kinect camera for parallax box effect. This could be cheap alternative if the z is reasonable and it seems fast. The kinect bodytracking was a bit slow.

1 Like

In typescript I read them as Record<string, Coords3D>.

This is how the eye position annotations look (there is also a big array ‘mesh’, ‘scaledMesh’ not annotated):

The bounding box with topLeft and bottomRight for the prediction is 2D.

1 Like

Video is mirrored, can you please let me how to fix this and also i am trying place some thing on head

can you please gudie what is logic for this

1 Like

how are you requesting your video stream?

const getStream = async (): Promise<MediaStream | null> => {
    if (navigator && navigator.mediaDevices && typeof navigator.mediaDevices.getUserMedia === 'function') {
      return await navigator.mediaDevices.getUserMedia({
        audio: false,
        video: {
          facingMode: 'user'
        }
      });
    }
    return null;
  };

I can’t remember if mine was mirrored that way.

even then it is mirrored , I think in the example playground we have apply this to layer, is layer is mirroring