I have been working on some xr tracking code and wanted to share a playground with some source for attaching meshes to face tracking landmarks. The code works on both desktop and mobile, although some issues remain with aligning the scene camera to a webcam.
There are hotkeys (wasd) for correcting the camera position, and I will post an update when I come up with a better solution to align the scene. I also added filtering to the tracked landmark position to reduce jittering in the mesh.
(Thanks to @bghgary for the background scaling code, and to @imerso for mediapipe resources)
Hey Carol, and thanks. It turns out that getting pose and position is much easier than going through the screen based landmarks, although I need to derive the camera vertical fov and distance. Both should be fairly straightforward to gather from the camera, and should fix the scale issue you are seeing.
Here is a playground showing the improved camera orientation. The trackers have been removed temporarily while I decide what to do about normal recalculation.
In the demo I am constructing a mesh using the triangles provide by Media Pipe, as it allows me to create new normals and guarantees the correct order of vertices. I am not sure that you could reliably upload a GLB and ensure that the vertex ordering matches their model, particularly if you have compressed or optimized the GLTF data.
It would be possible, and probably easier, to map the landmarks to an input model and to transfer the animated position data at each frame. If you can access the Face Mesh data on the GPU (where it is being computed) this could be run in a vertex shader, and it would be performant enough to suit most cases.
I tried and it worked very well just needed to adjust the camera a tiny bit