ChatGPT & 3d talking models

Hi everyone :wave: I’ve been playing around with a little experiment, trying to see how a 3D talking model would look when paired with ChatGPT-generated messages and text-to-speech. Here’s what I got so far:

The 3d model has a couple of animations that loop while the character is talking. And the talking facial animation is just the mouth opening and closing randomly. Still a lot of stuff that could be improved, but decided to share anyway.

Hope you find this project interesting and I’d love to hear any feedback or suggestions you may have :pray:



Please make a version as amorphous neuralnet Eldritch pointcloud and run the speech synthesizer thru reverb and pitchshifters in Tone.js

1 Like

Cool. You can use blendshapes for the mouth per vowel and match the face animation to vowels. Most of the vrm avatars are already rigged for that.


This is amazing!!
Maybe you can use animation blending with a coroutine to smooth the character animation.
Here is a piece of code that may help you:

function* animationBlending(fromAnim, fromAnimSpeedRatio, toAnim, toAnimSpeedRatio, repeat)
    let currentWeight = 1;
    let newWeight = 0;;

    fromAnim.speedRatio = fromAnimSpeedRatio;
    toAnim.speedRatio = toAnimSpeedRatio;

    while(newWeight < 1)
        newWeight += 0.01;
        currentWeight -= 0.01;

// Run Coroutine //
//scene.onBeforeRenderObservable.runCoroutineAsync(animationBlending(fromAnim, 1.0, toAnim, 1.0, true));

This is very COOL! :star_struck: The model is so adorable, did you make it?

I do have a feedback: for me the colors in the chatbox have almost zero contrast so I can’t read it:

When looking for color combos, one website I really appreciate is Randoma11y since it gives pairs of colors with accessibility in mind :smiley:

Thank you! The model is a vroid model - this one, with a different uniform. I did some work on it though - added the animations in blender, converted shaders to a single Principled BSDF, and did some other minor tweaks here and there.

Apologies for the chat colors being messed up :grimacing: there’s definitely a bug somewhere. The text color should be black, and the input box background color should be white. I’ll look into it.

1 Like

Is this robot generated from code?

Hey wudao :wave:

I used VRoid studio to export the model. And then, I imported it on blender with a VRM blender extension. After that, I had some animations laying around, which I had to retarget to the VRM rig. I did this with the Rokoko blender plugin. After I’m done working on the model in blender, I export the model with the babylonJS blender plugin - this generates a .babylon file which I then import with SceneLoader.ImportMeshAsync.

The mouth movement is being done with the VRM model’s morph targets. To be more specific, I’m just using the “A” mouth morph target all the time. I have a talk() function that is called when there is sound available. Here’s what it looks like:

  talk() {
    if (this.isTalking) {

    const maxTime = 0.5;
    const minTime = 0.2;

    const aTarget = this.getMorphTargetByName("Face.M_F00_000_00_Fcl_MTH_A");

    const vowelAnimEnd = () => {
        [0, random(minTime, maxTime), random(minTime, maxTime)],
        [0, 1, 0],

    this.isTalking = true;
      [0, random(minTime, maxTime), random(minTime, maxTime)],
      [0, 1, 0],

The playMorphTargetAnim looks like this:

export const playMorphTargetAnim = (
  name: string,
  durationsSeconds: number[],
  values: number[],
  morphTarget: MorphTarget,
  endCallback: () => void,
  scene: Scene
) => {
  const keyFrames: IAnimationKey[] = [];
  const framesPerSecond = 60;

  let previousFrame = 0;
  for (let i = 0; i < values.length; i++) {
    const currentFrame = previousFrame + durationsSeconds[i] * framesPerSecond;

      frame: currentFrame,
      value: values[i],

    previousFrame = currentFrame;

  const lastFrame =;

  var morphAnimation = new Animation(
  morphTarget.animations = [];

  scene.beginAnimation(morphTarget, 0, lastFrame.frame, false, 1, endCallback);

The eye opening / closing is also done using morph targets. However, everything else are “regular” bone animations.

Does this answer your question? If not, let me know.


Very cool to see how it works! :smiley:

This is awesome. I have been trying to figure it out, as I really want to create a responsive AI for having my students practice speaking. All I need is a talking head (medium close-up) with a transparent background. I would like it to be a realistic looking model. It would only need slight facial movement. I want to put it in my Wordpress site, which I would likely have to link to Babylon somehow.

  1. Would this work?
  2. Would you be able to help me figure this out? I would happily pay you for your time, even though finances aren’t looking too good at the moment, lol.

Thanks for this information, by the way. It helps give me more of an idea of the process.

All the best,

It’s 100% possible. No need for payment. We can chat via telegram / WhatsApp / something else.

I’ve been willing to open source this project for a while now. Hopefully I’ll get around to it this weekend. I’ll post a new reply in this thread with more info once I open source this.

3 things to consider:

  1. You said you wanted a realistic looking model. VRoid studio (which is where this model came from) doesn’t have those I’m afraid. For realistic models, I know of 2 alternatives: we can use a makehuman model, or we can use a renderpeople model (they have a few free models). Not sure if they have eye closing / mouth movement morph targets. But, even if they don’t, these should be easy to create in blender.

  2. In this project, I’m using my own OpenAI API key. You will probably need one of those. Not to worry, as their API is pretty cheap, and getting cheaper by the day it seems.

  3. In this project I’m also using a Google Cloud API key for converting the text to speech. They give $300 credit for new users so no worries here as well.

I’m not 100% sure how to use babylonJS with wordpress, but looks like there are some docs on how to do this: Babylon.js Wordpress Integration | Babylon.js Documentation


Thank you so much! I am extremely grateful and will email you in the next day or two for sure.

I have been using a chatbot with an OpenAI API and an Eleven Labs API for speaking, but this is what I really want to create. Excited to learn more.

Have a great weekend and truly appreciate the help!

In case anyone wants to check out this project’s source code:


Fantastic! Beautiful innovation, wish you all the success . I am looking for a 3d talking model that can be powered by AI, to play games and teach kids on selected topics. The 3d talking model doesnt have to do much, just stand and make facial expressions and interact using AI. All the games & content ready, i only need the Ai-powered talking model. I had previously built a 3d model which can make lip movements but not connected to AI.

I need to build a prototype, I will pay you how much ever i can manage, give full credits on my website, mobile app and if you are open, you could even co-partner on my project as a side project of yours.

Can i have your email address so we can connect on zoom video, i am so excited to know more.

Hello Noel. Thank you for the kind words. I’ll send you a DM.


I’m really impressed by your work. As a Ph.D. candidate researching human-AI collaboration, focusing on different incarnations of AI, I find your project aligns closely with my study interests. Specifically, I’m exploring the incorporation of AI avatars, like a ChatGPT form, into interactive platforms. I’m curious if it’s possible to integrate a responsive, realistic-looking AI model into my research. Could we discuss the feasibility of this?

Thank you.

Best regards,

@GeraldoOfTheRiviera I’d appreciate the opportunity to speak with you about a commercial project we’re working on in the eldercare space. We can figure out a time to speak if you can DM me and I’m happy to share details. Great work here!

Sir I love your mindblowing project I am currently a 9th Grade student and I am from India in the past year I was planning to make a Chatting 3D model for introverts that can talk to them freely like a real friend and can be open source as well I am currently only good in Arduino Programming making some robots but for this project I am trying to learn 3D modeling, JavaScript, Python, And Kotlin also but due to my Confusion I don’t know fro where should I start. Sir I am also an introvert I love to watch anime so I want to make an anime character for this project Please guide me I will do any thing I only have my own laptop and for communication I am willing to use my mom’s smart phone so please sir reply to me its my dream project I have many more ideas since class 6th maybe they might help you to improve your project efficiency

Thank you very much
Your, follower
Abhinav Jaivardhan
Class 9th


Welcome to the forum Abhinav! It’s cool to see your excitement for the project :slight_smile: Since you’re taking your first steps with JavaScript and 3D programming, I highly recommend you check the tutorial here: Introduction to Babylon.js Features | Babylon.js Documentation ( It will give you a very nice introduction to 3D programming and Babylon.js, and it has a lot of examples to play with and learn from. And if you have questions, the community on here will be happy to help :smiley: I wish you great studies!

Thank you so so much Sir I am very grateful and I am gonna do my best for this project.