Background Scaling Issue on Mobile

Hello, and thanks for any help you might lend. I am working with a background layer to capture a webcam into a fullscreen texture, and found some useful code for resizing in another thread (@bghgary FullScreenLayer video; dynamically change the aspect ratio - #2 by carolhmj). The webcam image is being sent through a facemesh camera to allow me to track face landmarks in the image, and is showing some distortion on mobile. It is possible that the error is being introduced by scaling the texture which facemesh is using to create the landmarks, although the desktop version doesn’t show the issue.

I have tried pushing content through the scene to see if the distortion is in the 3d render, but it is only in the landmark positions (and generated mesh).


I see an offset for my desktop webcam also, so this is not limited to mobile.

The offset, I believe, is the result of the difference in angle between the scene and my desktop camera. I am looking at how to correct it using the face normals (and perhaps intraocular distance). On mobile (portrait) the facemesh generated model looks distorted in y, although the tracking objects appear to have the right proportions.

It is also possible I have been looking at this too long.

Thanks again.

I’m guessing you’ve already read this, but it seems to explain how the transform works.

Face Mesh | mediapipe (

It looks like we need a Weak perspective projection, which I’m not sure how to do in Babylon.js, but it should be possible.

1 Like

Thanks, and I should be able derive the perspective camera using the webcam focal distance, but will need to correct the camera pose to account for the location of the webcam. There are some tools included within mediapipe/camera_utils which are for that purpose.

I’ll update the demo when I have something.

1 Like

The pose transform matrix is included with the geometry in face mesh, and can be used to transform the vertices into correct 3d positions. I will update the demo in the other thread.

Thanks for the insight,