Vertex Animation Texture module implementation

I’m considering implementing the VAT ideas from Vertex Animation Textures as a standalone module for a project of mine and publish it. I’m doing some performance tests and this is apparently the only way to run animations for tens of objects and keep the frame rate.

Is this something that could be integrated directly into BabylonJS? I’d be happy to send a PR to the repo instead of doing a separate module. If this is interesting, what would be the best way to define/discuss the API?

Also, are there any obvious limitations of using VATs with SPS or thin instances? I imagine if it works we could get crowds well into the 100s of objects.

Thanks

1 Like

This would be cool :slight_smile:

Keeping as a separate module before the PR will help a lot with not adding dependencies to it in the rest of the code so that then it would be all ready for your PR.

I absolutely love the idea

1 Like

Great. I’ll get the module ready over the next few days and post back here to get feedback, and if you like it enough I’ll send a PR to integrate it to the main tree :slight_smile:

2 Likes

Sounds like a wonderful plan !

1 Like

GitHub - Corollarium/babylonjs-vat: Vertex Animation Texture implementation for BabylonJS first draft up! But I’m having some trouble with materials, so it’s not working properly yet. How could I change an arbitrary material vertex shader?

The CustomMaterial approach from one of the demos seems interesting, but it doesn’t work for non StandardMaterial materials. I considered a secondary shader, but the binding/unbinding will be too costly and it won’t make sense to use a VAT then. I can’t see a way to monkey patch arbitrary shaders that would work reliably.

Any ideas?

I guess as soon as it is a standard feature, it would be its own include file and hidden by a #define so that ppl can use it in Shader or Node Material and we have add it to standard and PBR.

But first let s review what you have to see if you should move to main repo. @Evgeni_Popov and @Deltakosh would you mind having a look ?

I ll check tomorrow as well :slight_smile:

That seems good to me!

I guess we will need it to be handled a bit like the skeleton animation support, by adding some includes in the vertex shaders of the std/pbr material: have a look at Shaders/ShadersInclude/bonesDeclaration.fx and bonesVertex.fx and how they are injected in the standard material in Shaders/default.vertex.fx.

So, to go further I think we will need to have a PR against the Babylon.js repo, with the support added to StandardMaterial first (once it works, it should be easy to add support to the pbr/node materials).

1 Like

I see, that makes sense. I’ll finish the PoC demo with the proposed API on this separate repo, get your initial feedback and then fork BabylonJS and send a PR along the lines you mentioned. It’s always a pleasure to contribute to this project :slight_smile:

1 Like

First beta is up. GitHub - Corollarium/babylonjs-vat: Vertex Animation Texture implementation for BabylonJS It’s a PoC with some hardcoded data for the animation data (see below), but should be good enough for evaluation before moving to a fork of BabylonJS (and then adding automatic tests).

It works for the spider model (http://localhost:8080/) but the models are distorted for the shark model (http://localhost:8080/?scene=sharkVAT), which uses a PBR material. Certainly overriding the shader is not doing nice things to the poor shark.

Proposed API:


        // load our model
        const importResult = await SceneLoader.ImportMeshAsync(
            "",
            "https://raw.githubusercontent.com/RaggarDK/Baby/baby/",
            "arr.babylon",
            scene,
            undefined
        );

        // create a VAT for it
        const vat = new VAT(
            "VATspider",
            scene,
            importResult.meshes[0] as Mesh,
            importResult.skeletons[0],
            []
        );

        // bake VAT.
        vat.bakeVertexData().then(() => {
            vat.mesh.setEnabled(false);
            // create instances
            for (let i = 0; i < 20; i++) {
                const instance = vat.mesh.createInstance("spider" + i);
                instance.position.x += (i - 10.0) * 2;

                // set our animation parameters.
                instance.instancedBuffers.VATanimation = new Vector4(
                    0, // start
                    100, // end
                    i * 2, // offset
                    30.0 // speed in frames per second
                );
            }

            // Register a render loop to repeatedly render the scene
            const startTime = new Date().getTime();
            engine.runRenderLoop(() => {
                const endTime = new Date().getTime();
                const timeElapsed = (endTime - startTime) / 1000.0; // in s
                vat.updateTime(timeElapsed);
                scene.render();
            });
        });

There is a way to save the baked texture (actually, the vertex data Float32Array), serializing it to a JSON with a trick to keep it not too big (about 1.3x the size of the original array). Perhaps some utility using Puppeteer or something like it can be written later to bake models from the command line.

There’s one lingering problem that I’m not sure how to tackle. I thought I could always get the animations from the AnimationGroups, but that doesn’t seem to be the case. Is there some automatic way to get them? Or should we expect the user to send a [{name, startFrame, endFrame}] array of objects instead?

Also, on babylonjs-vat/vat.ts at master · Corollarium/babylonjs-vat · GitHub

        mat.onBindObservable.add(() => {
            // TODO: this should run only once
            mat.getEffect()
                .setFloat("singleFrameUVPer", 1 / this.frameCount)
                .setTexture("boneSampler1", this.boneTexture);
        });

Could this code be ran only once (and if so how) or does it need to be ran on every frame?

Looking forward to your feedback so we can integrate this soon :slight_smile:

I think you should focus first on adding the new feature to the standard material, meaning adding the shader code (in Shaders/ShadersInclude/) and the bits in Materials/standardMaterial.ts to enable/disable it. Building the texture is unrelated to this and can be tackled later on, maybe by adding an helper class in the Misc/ directory: you should assume the texture is already created and is an input of the new module.

So:

can be dealt with afterwards.

And:

won’t be handled that way as we won’t use a CustomMaterial, the binding code will be part of StandardMaterial.bindForSubMesh.

On the implementation side, you could do a bit like how the detail map has been implemented: see Materials/material.detailMapConfiguration.ts and look for “detailMap” occurrences in Materials/standardMaterial.ts.

Maybe the class can be called BakedVertexAnimation and implemented in Materials/material.bakedVertexAnimationConfiguration.ts?

Let’s see what @Deltakosh / @sebavan have to say about this before going on!

2 Likes

Thanks for the instructions @Evgeni_Popov, very helpful. I’ll take a look at the code you mentioned while I wait for @Deltakosh / @sebavan to give their feedback too.

I absolutely agree to keep it as a separate helper class, but I want to implement it as part of the initial PR, otherwise it’ll be essentially unusable as a feature. Serialization and loading can be handled later.

But since we’re on the topic, apparently the only image file format that supports float32 is DDS. Is that it, would it be a good choice? Babylon’s source code seems to support “R32G32B32A32F”, but I think there’s only a loader, not a saver, right?

What about starting a draft PR? It would be easier for review it (but overall I’m super positive for the change ;))

1 Like

Yes this is correct. DDS is fine as a storage file format and it can be produced by a lot of software. Ultimately how do you that artists and designer will create that texture?

I love it as well and a draft PR would help reviewing deeper :slight_smile: About Float texture, saving as .hdr or .exr might be possible and I thing @Evgeni_Popov worked on a .exr serializer ?

Also considering half floats instead might be a good options to optimize storage ?

What about starting a draft PR?

Working on it! :slight_smile:

Ultimately how do you that artists and designer will create that texture?

I’m not sure what would be the most common path.

My personal preference would be to generate it from BabylonJS somehow, since I would not be tied up to where the model was generated (and I’m a dev, not a designer, so it’s always nice if I don’t have to open Blender). I think it makes easier to do a generic workflow too, which would be my favorite: 1) get a model, no matter from where 2) run it through a command line tool 3) get a texture file.

Perhaps just to give a first tool for Babylon users, a JS script running on a browser that loads a generic model and generates the texture (with an option to save it locally) could be useful enough.

I read around and there are some script to export VATs from blender, such as Vertex Animation Script for Blender 3D Users - Work in Progress - Unreal Engine Forums. This would be quite helpful too.

I won’t write a plugin for Blender, but I can contribute with a baker script like the one made in the PoC repo. It’ll allow people use the feature without requiring a separate and as of now non-existing tool. And it can bake the texture at run-time too, which is not efficient but it’s easy for demos and documentation.

Also considering half floats instead might be a good options to optimize storage ?

Agreed.

Thanks for all the feedback! PR will be coming soon.

1 Like

I agree! Once the core of the functionality is implemented in the PR we can tackle the texture creation part.

Yes, a simple serializer for .exr is here: OceanDemo/exrSerializer.ts at main · Popov72/OceanDemo · GitHub

Note that it is very basic, it handles only float32 data. And you would also need a loader.

First PR is ready to be torn apart: Baked vertex animation textures by brunobg · Pull Request #11317 · BabylonJS/Babylon.js · GitHub. I imagine it makes more sense to keep the discussion on Github from now on.

3 Likes