FaceTrack using tensorflow (ml5)

As we prepare for the metaverse, I am getting myself more familiar with machine learning ai libraries like tensorflow, here’s a face tracker using ml5.js (a library under tensorflow) in conjuction with babylon.js

try it out ML5--tensorflow

compliments of the season

6 Likes

Pretty cool !!! mine is still stupidly sitting in there: https://playground.babylonjs.com/#H19VH7#42

2 Likes

This is really amazing :smiley: :+1: :+1:

i see this uses bounding box to place the helmet, nice!!

Wow, pretty neat!!!

1 Like

Thanks!!

aah - I was wondering who had made that playground - I found it on a curated babylon list. I wrote a Virtual try-on demo recently and that PG was super helpful - thanks for sharing!! I noticed that you had commented out looking up/down. I was looking at other 3d space solvers and then got distracted!

1 Like

Very cool. Does it give any depth information about eye position or its just x,y?

there is a position z defined but no idea how precise it is

1 Like

@kri100s
Yeah the z position is not that bad, i have updated the demo with a z coord

https://ml5-tests.glitch.me/#faceMesh

2 Likes

I used a kinect camera for parallax box effect. This could be cheap alternative if the z is reasonable and it seems fast. The kinect bodytracking was a bit slow.

1 Like

In typescript I read them as Record<string, Coords3D>.

This is how the eye position annotations look (there is also a big array ‘mesh’, ‘scaledMesh’ not annotated):

The bounding box with topLeft and bottomRight for the prediction is 2D.

1 Like