Mediapipe Face Tracking Playground

Demos and projects
1

Hey guys,

I have been working on some xr tracking code and wanted to share a playground with some source for attaching meshes to face tracking landmarks. The code works on both desktop and mobile, although some issues remain with aligning the scene camera to a webcam.

There are hotkeys (wasd) for correcting the camera position, and I will post an update when I come up with a better solution to align the scene. I also added filtering to the tracked landmark position to reduce jittering in the mesh.

(Thanks to @bghgary for the background scaling code, and to @imerso for mediapipe resources)

Let me know what you think,
r

6 Likes
2

This is so cool! :open_mouth: I tried and it worked very well just needed to adjust the camera a tiny bit

3

Hey Carol, and thanks. It turns out that getting pose and position is much easier than going through the screen based landmarks, although I need to derive the camera vertical fov and distance. Both should be fairly straightforward to gather from the camera, and should fix the scale issue you are seeing.

Here is a playground showing the improved camera orientation. The trackers have been removed temporarily while I decide what to do about normal recalculation.

r

2 Likes