Vertex Animation Texture module implementation

I’m considering implementing the VAT ideas from Vertex Animation Textures as a standalone module for a project of mine and publish it. I’m doing some performance tests and this is apparently the only way to run animations for tens of objects and keep the frame rate.

Is this something that could be integrated directly into BabylonJS? I’d be happy to send a PR to the repo instead of doing a separate module. If this is interesting, what would be the best way to define/discuss the API?

Also, are there any obvious limitations of using VATs with SPS or thin instances? I imagine if it works we could get crowds well into the 100s of objects.

Thanks

1 Like

This would be cool :slight_smile:

Keeping as a separate module before the PR will help a lot with not adding dependencies to it in the rest of the code so that then it would be all ready for your PR.

I absolutely love the idea

1 Like

Great. I’ll get the module ready over the next few days and post back here to get feedback, and if you like it enough I’ll send a PR to integrate it to the main tree :slight_smile:

2 Likes

Sounds like a wonderful plan !

1 Like

GitHub - Corollarium/babylonjs-vat: Vertex Animation Texture implementation for BabylonJS first draft up! But I’m having some trouble with materials, so it’s not working properly yet. How could I change an arbitrary material vertex shader?

The CustomMaterial approach from one of the demos seems interesting, but it doesn’t work for non StandardMaterial materials. I considered a secondary shader, but the binding/unbinding will be too costly and it won’t make sense to use a VAT then. I can’t see a way to monkey patch arbitrary shaders that would work reliably.

Any ideas?

I guess as soon as it is a standard feature, it would be its own include file and hidden by a #define so that ppl can use it in Shader or Node Material and we have add it to standard and PBR.

But first let s review what you have to see if you should move to main repo. @Evgeni_Popov and @Deltakosh would you mind having a look ?

I ll check tomorrow as well :slight_smile:

That seems good to me!

I guess we will need it to be handled a bit like the skeleton animation support, by adding some includes in the vertex shaders of the std/pbr material: have a look at Shaders/ShadersInclude/bonesDeclaration.fx and bonesVertex.fx and how they are injected in the standard material in Shaders/default.vertex.fx.

So, to go further I think we will need to have a PR against the Babylon.js repo, with the support added to StandardMaterial first (once it works, it should be easy to add support to the pbr/node materials).

1 Like

I see, that makes sense. I’ll finish the PoC demo with the proposed API on this separate repo, get your initial feedback and then fork BabylonJS and send a PR along the lines you mentioned. It’s always a pleasure to contribute to this project :slight_smile:

1 Like

First beta is up. GitHub - Corollarium/babylonjs-vat: Vertex Animation Texture implementation for BabylonJS It’s a PoC with some hardcoded data for the animation data (see below), but should be good enough for evaluation before moving to a fork of BabylonJS (and then adding automatic tests).

It works for the spider model (http://localhost:8080/) but the models are distorted for the shark model (http://localhost:8080/?scene=sharkVAT), which uses a PBR material. Certainly overriding the shader is not doing nice things to the poor shark.

Proposed API:


        // load our model
        const importResult = await SceneLoader.ImportMeshAsync(
            "",
            "https://raw.githubusercontent.com/RaggarDK/Baby/baby/",
            "arr.babylon",
            scene,
            undefined
        );

        // create a VAT for it
        const vat = new VAT(
            "VATspider",
            scene,
            importResult.meshes[0] as Mesh,
            importResult.skeletons[0],
            []
        );

        // bake VAT.
        vat.bakeVertexData().then(() => {
            vat.mesh.setEnabled(false);
            // create instances
            for (let i = 0; i < 20; i++) {
                const instance = vat.mesh.createInstance("spider" + i);
                instance.position.x += (i - 10.0) * 2;

                // set our animation parameters.
                instance.instancedBuffers.VATanimation = new Vector4(
                    0, // start
                    100, // end
                    i * 2, // offset
                    30.0 // speed in frames per second
                );
            }

            // Register a render loop to repeatedly render the scene
            const startTime = new Date().getTime();
            engine.runRenderLoop(() => {
                const endTime = new Date().getTime();
                const timeElapsed = (endTime - startTime) / 1000.0; // in s
                vat.updateTime(timeElapsed);
                scene.render();
            });
        });

There is a way to save the baked texture (actually, the vertex data Float32Array), serializing it to a JSON with a trick to keep it not too big (about 1.3x the size of the original array). Perhaps some utility using Puppeteer or something like it can be written later to bake models from the command line.

There’s one lingering problem that I’m not sure how to tackle. I thought I could always get the animations from the AnimationGroups, but that doesn’t seem to be the case. Is there some automatic way to get them? Or should we expect the user to send a [{name, startFrame, endFrame}] array of objects instead?

Also, on babylonjs-vat/vat.ts at master · Corollarium/babylonjs-vat · GitHub

        mat.onBindObservable.add(() => {
            // TODO: this should run only once
            mat.getEffect()
                .setFloat("singleFrameUVPer", 1 / this.frameCount)
                .setTexture("boneSampler1", this.boneTexture);
        });

Could this code be ran only once (and if so how) or does it need to be ran on every frame?

Looking forward to your feedback so we can integrate this soon :slight_smile: