Babylon.js MMD Loader is here

TL;DR: Introduces “babylon-mmd,” a project that enables loading MikuMikuDance (MMD) assets into Babylon.js.

Hello, I am writing this post because I believe my project, which I started in May, has reached a state where it can be publicly shared.

About MMD:

MikuMikuDance (MMD) is a free 3D animation software developed by a programmer in Japan. Despite being around for quite some time, MMD still maintains a large community that shares high-quality 3D models and dance motions for free.

If we can load these assets directly into Babylon.js, it could greatly contribute to creating a wider range of web 3D content.

And so, I implemented it.

Here are the key features of babylon-mmd:

  • babylon-mmd consists of a loader for Babylon.js, allowing it to load MMD files, and a runtime that replicates MMD’s unique animation system.
  • The loader offers many optimization options and allows for conversion to a highly optimized custom format, resulting in very fast loading times (around 2 seconds).
  • There is comprehensive documentation, including two tutorials to guide users through the process of how to use.
  • It supports ES6 tree shaking.

I poured all my passion into this project within three months and strived to achieve the highest code quality and implementation methods.

Moving forward, here are the future project tasks:

I attempted rigid body and joint simulation using Havok Physics, but the results were quite strange compared to MMD. I’m not sure if it’s a bug in Havok Physics or if I’m using it incorrectly.

To address this, I plan to add a feature that uses Bullet Physics (the physics engine used by MMD) to handle rigid bodies and joints separately.

Currently, to overcome the physics engine issue, I am introducing a solution in the documentation that involves baking physics calculations. Thanks to Babylon-mmd’s typed array-based custom animation system, it handles a large number of physics bones’ keyframes quite efficiently.

Further reading:

Screenshots:

Music: 侵蝕 niki feat.星界
Model: YYB Miku Crown Knight by PilouLaBaka
Motion: 【餓狼MMD】侵蝕【モーション配布】 by 鯖味噌煮サンバ
Camera: 【MMD】侵蝕 - カメラ配布 | Camera DL by ひな

11 Likes

Wow, that’s great :heart_eyes:

Wow very impressive. On the camera animations/movement. Are they a part of MMD or are those programmed in later?

You should totally fork and PR the BabylonJS docs and put your documentation there as well!!!

I’m gonna give MMD a try, because I have been looking for some better ways to do character rigging and animating.

Edit;
I also now consider Babylon JS 100% feature complete since I can now make this;

4 Likes

The specification of mmd also includes camera motion.

I hope this can be added to the babylon.js documentation

THIS IS SUPERRRR AWESOME!!! I’ve always thought it was incredible the fun animations people create with MMD, and having them integrated with Babylon.js is very cool! I wholeheartedly agree with your post, the browser makes a great environment for playing MMD content. Maybe even with WebXR content :smiley:

About Havok, if you have any questions or find anything is behaving weird feel free to open a topic here so we can take a look!

And 100% this would be great at the documentation, we have the community section Community Extensions | Babylon.js Documentation (babylonjs.com) and we would love a PR for it Documentation/content/communityExtensions at master · BabylonJS/Documentation (github.com). If you have any questions about how to add things to the documentation too feel free to ask!

2 Likes
  • The reason why I hesitate to ask questions about Havok physics is that it is not easy to simplify the code that causes the bug to be reproducible in PG. I don’t know if I can get technical support to ask questions while showing the long code in the babylon-mmd…

  • Thank you for being positive. I will try to add a page to the document as soon as possible :>

+This is the webXR example that you probably imagined(It won’t work on IOS, maybe?):

https://noname0310.github.io/babylon-mmd-builds/melancholy_night_webxr/

credits

Music: メランコリ・ナイト

Model: YYB Hatsune Miku_10th

Motion / Camera: https://www.nicovideo.jp/watch/sm41164308

Only six lines of code made this possible, That’s truly incredible.

scene.createDefaultXRExperienceAsync({
    uiOptions: {
        sessionMode: "immersive-ar",
        referenceSpaceType: "local-floor"
    }
});
4 Likes

Oh, I see, yeah, it’s always harder to understand the cause of the bug when the system is complex :face_with_spiral_eyes:

But the WebXR example… IT’S SO COOL! It seems like it also doesn’t work in Firefox Android, but I tried on Chrome and it worked very well! It’s Miku in my office! :heart_eyes: :rofl:

5 Likes

Looking at the github repo, is the MmdCamera required? Will loading just the BPMX and BVMD be enough to draw an MMD animated character on a model?

No, most components are independent

MmdCamera is just a class of cameras that do minimal matrix operations to reproduce mmd cameras

BPMX and BVMD are also optional
With them, you can speed up the load time by about 1.5 times, but you can use the existing mmd formats of PMX and VMD

Here’s a video tutorial. If you look at the beginning, the model is loaded first without Mmd Camera.
In fact, this is a matter of course. There’s nothing to do with models and cameras

7 Likes

I tried out the BPMX converter at babylon-mmd but it seems to fail to load tga textures? they are in the same folder as the .pmx file and they load correctly when I load the model in MMD.

Just want to say the tutorial is fantastic! I love how you go through each step very detailedly and explains even the errors that can happen along the way :slight_smile: Your project is phenomenal and I hope a lot of people use it, MMD running live on web is such a fun idea :smiley:

1 Like

I’ll check. My test model doesn’t use tga very well, so it’s a little late to check for these issues.

If you could give me a link or name of the model that is experiencing the problem, it could be more helpful to solve the problem

It was simply an issue that I did not import the TGA Texture Loader side-effect. It’ll work fine now

It’s just this model: https://3d.nicovideo.jp/works/td78503

It’s the official MMD model of Hololive characters. Download 雪花ラミィ公式mmd_ver1.0.zip and extract it and it should be all in the same folder.

I imagine if you can get this to work, it’ll work for all Hololive MMD models

You can try again now. If not, try disabling cache and reload the page

4 Likes

yeah it’s working for me now, thanks! the model is quite complex so fingers crossed if it animates properly :crossed_fingers:

1 Like

ok, so I exported the bpmx to test out loading in BJS 6.19.0 but now it’s the bpmx file that loads without textures. attached the bpmx file. I reduced the size of textures so it’s not like 200MB :wink:


lamy.zip (5.1 MB)

Checkout the result that is output from SceneLoader.ImportMeshAsync,

it contains

  1. animationGroups:
  2. geometries: [Geometry]
  3. lights:
  4. meshes: [SdefMesh]
    • with 20 submeshes, each with different materialIndex and _currentMaterial which is MmdStandardMaterial but there’s no diffuseTexture or anything, only MmdPluginMaterial
  5. particleSystems:
  6. skeletons: [Skeleton]
  7. transformNodes:

It looks like you just didn’t import the side-effect

try

import "@babylonjs/core/Materials/Textures/Loaders/tgaTextureLoader";

it works well

here is my code ```typescript import "@babylonjs/core/Loading/loadingScreen"; import "@babylonjs/core/Lights/Shadows/shadowGeneratorSceneComponent"; import "@babylonjs/core/Materials/Textures/Loaders/tgaTextureLoader"; import "@babylonjs/core/Meshes/thinInstanceMesh"; import "@babylonjs/core/Rendering/prePassRendererSceneComponent"; import "@babylonjs/core/Rendering/depthRendererSceneComponent"; import "@babylonjs/core/Rendering/geometryBufferRendererSceneComponent"; import "babylon-mmd/esm/Loader/Optimized/bpmxLoader"; import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeCameraAnimation"; import "babylon-mmd/esm/Runtime/Animation/mmdRuntimeModelAnimation";

import { ArcRotateCamera } from “@babylonjs/core/Cameras/arcRotateCamera”;
// import { DirectionalLightFrustumViewer } from “@babylonjs/core/Debug/directionalLightFrustumViewer”;
import { SkeletonViewer } from “@babylonjs/core/Debug/skeletonViewer”;
import { Constants } from “@babylonjs/core/Engines/constants”;
import type { Engine } from “@babylonjs/core/Engines/engine”;
import { DirectionalLight } from “@babylonjs/core/Lights/directionalLight”;
import { HemisphericLight } from “@babylonjs/core/Lights/hemisphericLight”;
import { ShadowGenerator } from “@babylonjs/core/Lights/Shadows/shadowGenerator”;
import { SceneLoader } from “@babylonjs/core/Loading/sceneLoader”;
import { ImageProcessingConfiguration } from “@babylonjs/core/Materials/imageProcessingConfiguration”;
import { StandardMaterial } from “@babylonjs/core/Materials/standardMaterial”;
import { Color3, Color4 } from “@babylonjs/core/Maths/math.color”;
import { Matrix, Vector3 } from “@babylonjs/core/Maths/math.vector”;
import { CreateGround } from “@babylonjs/core/Meshes/Builders/groundBuilder”;
import type { Mesh } from “@babylonjs/core/Meshes/mesh”;
import { TransformNode } from “@babylonjs/core/Meshes/transformNode”;
import { HavokPlugin } from “@babylonjs/core/Physics/v2/Plugins/havokPlugin”;
import { DepthOfFieldEffectBlurLevel } from “@babylonjs/core/PostProcesses/depthOfFieldEffect”;
import { DefaultRenderingPipeline } from “@babylonjs/core/PostProcesses/RenderPipeline/Pipelines/defaultRenderingPipeline”;
import { SSRRenderingPipeline } from “@babylonjs/core/PostProcesses/RenderPipeline/Pipelines/ssrRenderingPipeline”;
import { Scene } from “@babylonjs/core/scene”;
import HavokPhysics from “@babylonjs/havok”;

import type { MmdAnimation } from “babylon-mmd/esm/Loader/Animation/mmdAnimation”;
import type { MmdStandardMaterialBuilder } from “babylon-mmd/esm/Loader/mmdStandardMaterialBuilder”;
import type { BpmxLoader } from “babylon-mmd/esm/Loader/Optimized/bpmxLoader”;
import { BvmdLoader } from “babylon-mmd/esm/Loader/Optimized/bvmdLoader”;
import { SdefInjector } from “babylon-mmd/esm/Loader/sdefInjector”;
import { StreamAudioPlayer } from “babylon-mmd/esm/Runtime/Audio/streamAudioPlayer”;
import { MmdCamera } from “babylon-mmd/esm/Runtime/mmdCamera”;
import { MmdPhysics } from “babylon-mmd/esm/Runtime/mmdPhysics”;
import { MmdRuntime } from “babylon-mmd/esm/Runtime/mmdRuntime”;
import { MmdPlayerControl } from “babylon-mmd/esm/Runtime/Util/mmdPlayerControl”;

import type { ISceneBuilder } from “…/baseRuntime”;

export class SceneBuilder implements ISceneBuilder {
public async build(canvas: HTMLCanvasElement, engine: Engine): Promise {
SdefInjector.OverrideEngineCreateEffect(engine);
const pmxLoader = SceneLoader.GetPluginForExtension(“.bpmx”) as BpmxLoader;
pmxLoader.loggingEnabled = true;
const materialBuilder = pmxLoader.materialBuilder as MmdStandardMaterialBuilder;
materialBuilder.alphaEvaluationResolution = 2048;
// materialBuilder.loadDiffuseTexture = (): void => { /* do nothing / };
// materialBuilder.loadSphereTexture = (): void => { /
do nothing / };
// materialBuilder.loadToonTexture = (): void => { /
do nothing / };
materialBuilder.loadOutlineRenderingProperties = (): void => { /
do nothing */ };

    const scene = new Scene(engine);
    scene.clearColor = new Color4(0.95, 0.95, 0.95, 1.0);

    const mmdCamera = new MmdCamera("mmdCamera", new Vector3(0, 10, 0), scene);
    mmdCamera.maxZ = 5000;

    const mmdRoot = new TransformNode("mmdRoot", scene);
    mmdCamera.parent = mmdRoot;
    mmdRoot.position.z -= 0;

    const camera = new ArcRotateCamera("arcRotateCamera", 0, 0, 45, new Vector3(0, 10, 0), scene);
    camera.maxZ = 5000;
    camera.setPosition(new Vector3(0, 10, -45));
    camera.attachControl(canvas, false);
    camera.inertia = 0.8;
    camera.speed = 10;

    const hemisphericLight = new HemisphericLight("hemisphericLight", new Vector3(0, 1, 0), scene);
    hemisphericLight.intensity = 0.4;
    hemisphericLight.specular = new Color3(0, 0, 0);
    hemisphericLight.groundColor = new Color3(1, 1, 1);

    const directionalLight = new DirectionalLight("directionalLight", new Vector3(0.5, -1, 1), scene);
    directionalLight.intensity = 0.8;
    directionalLight.autoCalcShadowZBounds = false;
    directionalLight.autoUpdateExtends = false;
    directionalLight.shadowMaxZ = 20;
    directionalLight.shadowMinZ = -20;
    directionalLight.orthoTop = 18;
    directionalLight.orthoBottom = -3;
    directionalLight.orthoLeft = -10;
    directionalLight.orthoRight = 10;
    directionalLight.shadowOrthoScale = 0;

    // const directionalLightFrustumViewer = new DirectionalLightFrustumViewer(directionalLight, mmdCamera);
    // scene.onBeforeRenderObservable.add(() => directionalLightFrustumViewer.update());

    const shadowGenerator = new ShadowGenerator(1024, directionalLight, true);
    shadowGenerator.usePercentageCloserFiltering = true;
    shadowGenerator.forceBackFacesOnly = false;
    shadowGenerator.bias = 0.01;
    shadowGenerator.filteringQuality = ShadowGenerator.QUALITY_MEDIUM;
    shadowGenerator.frustumEdgeFalloff = 0.1;

    const ground = CreateGround("ground1", { width: 120, height: 120, subdivisions: 2, updatable: false }, scene);
    const groundMaterial = ground.material = new StandardMaterial("groundMaterial", scene);
    groundMaterial.diffuseColor = new Color3(1.02, 1.02, 1.02);

    const mmdRuntime = new MmdRuntime(new MmdPhysics(scene));
    mmdRuntime.loggingEnabled = true;

    mmdRuntime.register(scene);
    mmdRuntime.playAnimation();

    const audioPlayer = new StreamAudioPlayer(scene);
    audioPlayer.preservesPitch = false;
    audioPlayer.source = "res/private_test/motion/patchwork_staccato/pv_912.mp3";
    mmdRuntime.setAudioPlayer(audioPlayer);

    const mmdPlayerControl = new MmdPlayerControl(scene, mmdRuntime, audioPlayer);
    mmdPlayerControl.showPlayerControl();

    engine.displayLoadingUI();

    let loadingTexts: string[] = [];
    const updateLoadingText = (updateIndex: number, text: string): void => {
        loadingTexts[updateIndex] = text;
        engine.loadingUIText = "<br/><br/><br/><br/>" + loadingTexts.join("<br/><br/>");
    };

    const promises: Promise<any>[] = [];

    const bvmdLoader = new BvmdLoader(scene);
    bvmdLoader.loggingEnabled = true;

    promises.push(bvmdLoader.loadAsync("motion", "res/private_test/motion/ruse/motion.bvmd",
        (event) => updateLoadingText(0, `Loading motion... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`))
    );

    pmxLoader.boundingBoxMargin = 60;
    promises.push(SceneLoader.ImportMeshAsync(
        undefined,
        "res/private_test/model/",
        "lamy.bpmx",
        scene,
        (event) => updateLoadingText(1, `Loading model... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`)
    ));

    pmxLoader.boundingBoxMargin = 0;
    pmxLoader.buildSkeleton = false;
    pmxLoader.buildMorph = false;
    promises.push(SceneLoader.ImportMeshAsync(
        undefined,
        "res/private_test/stage/",
        "Stage35_02.bpmx",
        scene,
        (event) => updateLoadingText(2, `Loading stage... ${event.loaded}/${event.total} (${Math.floor(event.loaded * 100 / event.total)}%)`)
    ));

    promises.push((async(): Promise<void> => {
        updateLoadingText(3, "Loading physics engine...");
        const havokInstance = await HavokPhysics();
        const havokPlugin = new HavokPlugin(true, havokInstance);
        scene.enablePhysics(new Vector3(0, -9.8 * 10, 0), havokPlugin);
        updateLoadingText(3, "Loading physics engine... Done");
    })());

    loadingTexts = new Array(promises.length).fill("");

    const loadResults = await Promise.all(promises);

    mmdRuntime.setManualAnimationDuration((loadResults[0] as MmdAnimation).endFrame);

    scene.onAfterRenderObservable.addOnce(() => engine.hideLoadingUI());

    scene.meshes.forEach((mesh) => {
        if (mesh.name === "skyBox") return;
        mesh.receiveShadows = true;
        shadowGenerator.addShadowCaster(mesh);
    });

    mmdRuntime.setCamera(mmdCamera);
    mmdCamera.addAnimation(loadResults[0] as MmdAnimation);
    mmdCamera.setAnimation("motion");

    {
        const modelMesh = loadResults[1].meshes[0] as Mesh;
        modelMesh.parent = mmdRoot;

        const mmdModel = mmdRuntime.createMmdModel(modelMesh, {
            buildPhysics: true
        });
        mmdModel.addAnimation(loadResults[0] as MmdAnimation);
        mmdModel.setAnimation("motion");

        const bodyBone = modelMesh.skeleton!.bones.find((bone) => bone.name === "センター");
        const meshWorldMatrix = modelMesh.getWorldMatrix();
        const boneWorldMatrix = new Matrix();
        scene.onBeforeRenderObservable.add(() => {
            boneWorldMatrix.copyFrom(bodyBone!.getFinalMatrix()).multiplyToRef(meshWorldMatrix, boneWorldMatrix);
            boneWorldMatrix.getTranslationToRef(directionalLight.position);
            directionalLight.position.y -= 10;

            camera.target.copyFrom(directionalLight.position);
            camera.target.y += 13;
        });

        const viewer = new SkeletonViewer(modelMesh.skeleton!, modelMesh, scene, false, 3, {
            displayMode: SkeletonViewer.DISPLAY_SPHERE_AND_SPURS
        });
        viewer.isEnabled = false;
    }

    const mmdStageMesh = loadResults[2].meshes[0] as Mesh;
    mmdStageMesh.receiveShadows = true;
    mmdStageMesh.position.y += 0.01;

    const useHavyPostProcess = true;
    const useBasicPostProcess = true;

    if (useHavyPostProcess) {
        const ssr = new SSRRenderingPipeline(
            "ssr",
            scene,
            [mmdCamera, camera],
            false,
            Constants.TEXTURETYPE_UNSIGNED_BYTE
        );
        ssr.step = 32;
        ssr.maxSteps = 128;
        ssr.maxDistance = 500;
        ssr.enableSmoothReflections = false;
        ssr.enableAutomaticThicknessComputation = false;
        ssr.blurDownsample = 2;
        ssr.ssrDownsample = 2;
        ssr.thickness = 0.1;
        ssr.selfCollisionNumSkip = 2;
        ssr.blurDispersionStrength = 0;
        ssr.roughnessFactor = 0.1;
        ssr.reflectivityThreshold = 0.9;
        ssr.samples = 4;
    }

    if (useBasicPostProcess) {
        const defaultPipeline = new DefaultRenderingPipeline("default", true, scene, [mmdCamera, camera]);
        defaultPipeline.samples = 4;
        defaultPipeline.bloomEnabled = true;
        defaultPipeline.chromaticAberrationEnabled = true;
        defaultPipeline.chromaticAberration.aberrationAmount = 1;
        defaultPipeline.depthOfFieldEnabled = true;
        defaultPipeline.depthOfFieldBlurLevel = DepthOfFieldEffectBlurLevel.High;
        defaultPipeline.fxaaEnabled = true;
        defaultPipeline.imageProcessingEnabled = true;
        defaultPipeline.imageProcessing.toneMappingEnabled = true;
        defaultPipeline.imageProcessing.toneMappingType = ImageProcessingConfiguration.TONEMAPPING_ACES;
        defaultPipeline.imageProcessing.vignetteWeight = 0.5;
        defaultPipeline.imageProcessing.vignetteStretch = 0.5;
        defaultPipeline.imageProcessing.vignetteColor = new Color4(0, 0, 0, 0);
        defaultPipeline.imageProcessing.vignetteEnabled = true;

        defaultPipeline.depthOfField.fStop = 0.05;
        defaultPipeline.depthOfField.focalLength = 20;

        // note: this dof distance compute will broken when camera and mesh is not in same space

        const modelMesh = loadResults[1].meshes[0] as Mesh;
        const headBone = modelMesh.skeleton!.bones.find((bone) => bone.name === "頭");

        const rotationMatrix = new Matrix();
        const cameraNormal = new Vector3();
        const cameraEyePosition = new Vector3();
        const headRelativePosition = new Vector3();

        scene.onBeforeRenderObservable.add(() => {
            const cameraRotation = mmdCamera.rotation;
            Matrix.RotationYawPitchRollToRef(-cameraRotation.y, -cameraRotation.x, -cameraRotation.z, rotationMatrix);

            Vector3.TransformNormalFromFloatsToRef(0, 0, 1, rotationMatrix, cameraNormal);

            mmdCamera.position.addToRef(
                Vector3.TransformCoordinatesFromFloatsToRef(0, 0, mmdCamera.distance, rotationMatrix, cameraEyePosition),
                cameraEyePosition
            );

            headBone!.getFinalMatrix().getTranslationToRef(headRelativePosition)
                .subtractToRef(cameraEyePosition, headRelativePosition);

            defaultPipeline.depthOfField.focusDistance = (Vector3.Dot(headRelativePosition, cameraNormal) / Vector3.Dot(cameraNormal, cameraNormal)) * 1000;
        });

        let lastClickTime = -Infinity;
        canvas.onclick = (): void => {
            const currentTime = performance.now();
            if (500 < currentTime - lastClickTime) {
                lastClickTime = currentTime;
                return;
            }

            lastClickTime = -Infinity;

            if (scene.activeCamera === mmdCamera) {
                defaultPipeline.depthOfFieldEnabled = false;
                scene.activeCamera = camera;
            } else {
                defaultPipeline.depthOfFieldEnabled = true;
                scene.activeCamera = mmdCamera;
            }
        };
    }

    // Inspector.Show(scene, { });

    return scene;
}

}

</details>
3 Likes

Thanks for the hard work! I was following your github readme for loading BPMX so I suggest adding the bit about importing for TGA side-effects to the readme? Update README.md with TGA texture help by vtange · Pull Request #14 · noname0310/babylon-mmd · GitHub

If this isn’t needed since it’s standard for BabylonJS - is there a place that lists all the possible side-effects one should be aware of?