Apply Human Face(photo) to glb file

Hi,

We have female model(.glb) able to import into babylon.js successfully,
Now we want to change the face by applying new user photo. Pls let us know the approach
FYI:
https://d1a370nemizbjq.cloudfront.net/8bce1c02-0e76-4d89-9bca-f4561529731f.glb

Thanks
vij

This might be quite tricky :slight_smile:

Maybe @PatrickRyan has a nice idea to do it.

@vijay_krishna, Iā€™m not sure if I follow what you are asking to do. When you say ā€œchangeā€ the face by ā€œapplyingā€ the user photo, are you talking about morphing the geometry to match the user photo or changing the texture of the face only by adding the the user photo at the pixel level? In either case, there will need to be some tech for identifying the features in the face so you can either morph the geometry or for fine tweaking the position of the user photo since you will get a wide variety of source images. But further clarification of your goals would be great.

Thanks @sebavan for the referral

and Thanks @PatrickRyan for your attention, We are looking for face morphing by applying user photo. The input will be .glb file shared earlier and user will upload his photo as per our guidelines. Can you share some logic/example how to accomplish this.

.

HI @vijay_krishna and welcome to the forum.

Unfortunately it is still not clear what you need.

What have you achieved already? What is the next single step that you cannot do yet?

Hey, welcome.
Sorry that I have to give you this answer but ā€¦ ā€˜It is not possible yetā€™. Trust me, you are not the first who want to do this. I heard about this atleast 10 times till now.

2 big reasons.
1 - let say you have the object and you want to aply the photo on it, you canā€™t simply put an image over a mesh, ā€¦you need uvs, and the image need to look in some way to be aplied corectly, and the image will not look good if the geometry was done for other ā€˜faceā€™
2 -let say you want the mesh to change according to image, how do you think youā€™ll do this? A mesh with 8 morphtargets for eye, nose, mouth is very big, and the result is a very limited type of faces

As a conclusion, you canā€™t do this for random photo and random faces, maybe you can do this but for few only

But we have a hope, Iā€™m sure that one day an AI will do this for us.

I donā€™t want to discourage you, but unfortunately thatā€™s about the situation right now. You will find mobile apps or maybe web apps too, which say that they can do this, but ā€¦ they donā€™t, or their result is poor, not even close to a really human face. Or maybe youā€™ll find something which I donā€™t, so you can still search for it

Cheers :beers:

2 Likes

Actually, I understand that it can currently be done by a few. Not with a single photo, but a group. It cannot be done in this framework though. Perhaps try a search of 'deep fakes ā€™ will lead to place that has more emphasis on this.

This is totally possibleā€¦ like 10 years agoā€¦
Ready Player Me - Create a Full-Body 3D Avatar From a Photo is using babylon lol.
Babylon vr comes with a wasm build of opencv.

Search on github for PRNET, GAN, style transfer, mask, face3d, landmark detection, etc.

or a little less work
.GitHub - google/mediapipe: Cross-platform, customizable ML solutions for live and streaming media.
.Search Ā· mediapipe Ā· GitHub
.Search Ā· mediapipe Ā· GitHub

For consistency, you could start with a pool of known face types, similar to the concept of ecosystem biomes. Iā€™m sure thereā€™s some research out there on this topic. After you do feature detection, pick the prebuilt model closest to your user and apply a transform to it based on whatever feature detection algorithm you copy/pasted, making sure to keep everything symmetric.
Using .Merging Meshes | Babylon.js Documentation

Read the docs:
Playgrounds with MorphTarget
.Playground search page | Babylon.js Documentation

tons of info under .Meshes | Babylon.js Documentation

1 Like

Mmm, more or less agree with you.
It is one thing to read an image with opencv and based on some properties like hair color or wearing glasses return an 3d mesh from a galery which looks like, and other thing to create a 1 to 1 3d object based on an image, like photogrammetry.

It depends on point of view, I was talking about the second scenario