ReadyPlayerMe Avatar -- Talking Animation (Morph Targets)

Tutorials and tips
, , , ,
1

Hi group!!

This time I’m sharing a demo, a ReadyPlayerMe character, synchronizing the facial animation with previously generated audio using Morph Targets:

Live Demo
https://viseni.com/readyplayer_talk/
(Click or Tap on the BJS Logo to Start the Animation)

GitHub Code

The demo has some more details like animations and some interactions.

It’s important to download the ReadyPlayerMe models by applying the Morph Targets using GET:

https://models.readyplayer.me/--READYPLAYERME--.glb?morphTargets=ARKit&lod=1&textureFormat=webp

Then from the code it can be accessed using the morphTargetManager:

const morphTarget1 = youMesh.morphTargetManager.getTarget(2);

To synchronize the animation with the audio I have used BABYLON.Analyser:

myAnalyser = new BABYLON.Analyser(scene);
BABYLON.Engine.audioEngine.connectToAnalyser(myAnalyser);
myAnalyser.FFT_SIZE = 64;
myAnalyser.SMOOTHING = 0.07;
// Debug Analiser Canvas
// myAnalyser.drawDebugCanvas();

And once it detects that the audio is playing:

// RegisterBeforeRender Morph Target Mouth
scene.registerBeforeRender(function () {  
    var workingArray = myAnalyser.getByteFrequencyData();
    var jawValue = 0;

    if (talking) {
        // console.log("Frequency: " + workingArray[5] / 512);
        jawValue = workingArray[5] / 512 * morphMultiplier_1;
    }

    scene.getMeshByName("Wolf3D_Head").morphTargetManager.getTarget(16).influence = jawValue*2;
    scene.getMeshByName("Wolf3D_Head").morphTargetManager.getTarget(34).influence = jawValue;
    scene.getMeshByName("Wolf3D_Teeth").morphTargetManager.getTarget(34).influence = jawValue;
});

Check the log for available MorphTargets

console.log(scene.getMeshByName("Wolf3D_Head").morphTargetManager);

image
image446×734 35.4 KB

Also take a look at ReadyPlayerMe Animation Library on GitHub:

ReadyPlayerMe Animation Library

They are including very well configured animations.
To convert them to GLB in batch you can use this code

FBX2GLB-Batch-Convert-Optimizer

Hope it helps!

9 Likes
2

OMG this is AWESOME! @thomlucc @sebavan and @Evgeni_Popov will love to see this :slight_smile:
p.s: I cackled at “Steven NoJobs…” :rofl:

2 Likes
3

Yeah!!

We all love Steven NoJobs :crazy_face:

1 Like
4

This is awesome and derserves a spot in the doc :slight_smile: cc @PatrickRyan and @PirateJC

3 Likes
5

Thanks like always sir! :raised_hands::raised_hands::raised_hands:

1 Like
6

Very well done @DRLeria! I’m sure that is an example that will help many.

2 Likes
7

Thanks @thomlucc!! :raised_hands::raised_hands:

8

This is great @DRLeria !
Thanks a lot for sharing it :hearts:

1 Like
9

I love it. Thanks. I am updating my personal website :slight_smile:

2 Likes