Baked Texture Animations with Animation Groups?

Adding @brunobg the brilliant mind behind the animation texture.

2 Likes

I could never understand animation groups well enough to implement this, though I’d love too and actually was considering trying it again this week. There was a thread about this problem before: How to find the total number of frames of a skeleton animation?

From what I remember at the time, it seems that it’d require some manipulation of the RuntimeAnimation class to be able to set a specific frame and capture the transformations. I was a bit busy back then and I couldn’t work it out how that would work. I’ll try to look at it this week again (no promises!), and of course help is always appreciated.

Besides the basic problems of manipulating the AG directly to bake things, I remember there were two other minor issues to handle everything one would need. All this comes from what my memory is of the analysis I did back then, so this might all be way off.

First one was that applying VATs might clash with animation groups, but I think this was easy to handle by not using the AGs and VATs at the same time.

The second issue is that the VAT implementation does not handle composition of animations IIRC. So if you want to play and stop multiple groups it won’t work. My memory is fuzzy about this, but I think it’d be possible to use multiple textures (but not arbitrarily many), or do a small patch to the VAT so that it’d compose transformations if you had more than one applied (but I think it was not trivial since the order of transformations matters).

This is interesting and I didn’t know it. Do you mean that it exports textures that can be used with VAT, or that the BJS file has pre-baked animations stored in some form? I’m guessing the later but the docs mention the animation will be kind of baked during the export (hahaha, so very technical). I’d love to learn a bit more about this and if this baking enables 100% GPU animations.

Thank you for sharing the previous thread and your insights, really helps shed a light on the complexities of animations.

Regarding the BabylonJS Blender Exporter, my scenario is as follows:

  • I got an asset with an Armature/Skeleton structure and multiple meshes attached to it (one Mesh would be just fine but that’s how the asset I use is made, and it’s quite useful as well)
  • inside Blender, I’ve defined just one Action (Animation) in the Dope Sheet, and added all the Animations Keyframes sequentially into that one Action
  • exporting to a .babylon file, the Exporer actually creates the Animation on the Skeleton as a sequence of Keyframes
  • I have to manually define the ranges in code, since the getAnimationRanges() on the Skeleton returns the wrong values for some reason, so I cannot use them for the baking process. Since we only have one Blender Action, we will only have one complete AnimationRange anyways, so the ranges array has to be done manually in code.
  • the ranges array in the code have to match the Keyframe values in Blender, one range for each Animation (from/to), and the Animations bake & play correctly in the browser

In a nutshell having just one Blender Action with all animation keyframes sequentially layed out does seem to make the VAT work. In code, we have to specify the ranges array manually since the getAnimationRanges() on the skeleton will return one full range.

I’ve also tried many scenarios regarding multiple Actions in Blender (one for each Animation), as well as NLA tracks, but none seem to work correctly with getAnimationRanges(). The automated ranges seem to be off quite a bit, not sure why.

All of the above are for the .babylon asset format. Regarding the GLTF format, it will create those AnimationGroups instead of direct animations on the Skeleton, which don’t work for VAT as you mentioned in the previous thread.

1 Like

Yes, this is a problem I was having too at the time and that led to that other thread. I could never get the ranges properly with AGs either.

If you think your code would be helpful to improve the VAT system ping me and let’s get this into a PR.

1 Like

I tried to calculate VAT for animation groups. It should works with known limitations like only one anim at the same time, no blending etc.

I have an issue with blinking animation at the start of animation. I think it is something with starting offset of animation, or rounding numbers. Will appreciate any helps)

Another idea is to be independent of frame rate to render anim. Right now under the hood, it renders via Animation which calls once per frame. For many frames(>500) with 30fps the process too long. I know that we can PRE bake it into json, but maybe it is good option to configure how to bake.

@Deltakosh Maybe it can be integrated into Babylon.js/vertexAnimationBaker.ts at fdbf393d1d7699dc7cc69cec1dca0819ebd2622a · BabylonJS/Babylon.js · GitHub somehow?

100 instances with 5 animations:

4 Likes

@Evgeni_Popov might have some ideas ?

For some reason, the very first frame of the first animation group is generated two times. Skipping one fixes the problem:

That’s another thing we will have to figure out as part of the animation system revision for 7.0 (Animation Improvements · Issue #13534 · BabylonJS/Babylon.js · GitHub).

4 Likes

Hi @Evgeni_Popov, I was looking to integrate vat animations into my project but was wondering what are the benefits?

With a standard async import of 100+ entities, I get similar fps to your playground example? And that’s with 0 optimization, I’m sure I could improve that number. Hopefully, my question isnt too silly :slight_smile:

You should check the performance via Inspector->Statistics->Frame steps duration. Maybe you don’t see the difference because you’r not bound to CPU on your device. Try it on a mobile.

On your screenshot, you have 10FPS. Do you check what cause it?

1 Like

Ok that make sense,

Here is the frame steps, but I must admit, I not entirely sure what each line means.

image

So yeah, even for 30FPS you need 33ms per frame. So now you’r CPU and GPU bounded.

I think you can easily improve meshes selection(disable mesh selection for static objects like walls).

RenderTargets stat comes from UI if I’m not mistaken (in your case).

For GPU you need more info from COUNT tab. Cause it depends on many factors like draw calls, textures sizes etc.

I think you can easily improve meshes selection(disable mesh selection for static objects like walls).

Do you mean something like this:

m.checkCollisions = false;
m.isPickable = false;
m.receiveShadows = false;
m.freezeWorldMatrix();

I wonder why the render targets are so high, it’s probably the names above each entity. Anyway, just have to do some tests to pinpoint that.

In all cases, I will work on implementing VAT, once I found the right way to dynamically integrate it.

Ok once you start baking the animation, and doing a little optimization, VAT gives amazing results!

With 200 entities playing different animations… I get 30 fps on a low end computer.

Any idea why the weapon is not fully attaching to each instance? It’s initial position looks good, but the weapon does not follow the animation.
IF I move the player instance, the weapon does follow the player (but not the animation), it just stays in the same position as the player.

const weapon = weaponMeshMerged.createInstance("player_" + id + "_sword");
let bone = playerInstance.skeleton.bones[37];
weapon.attachToBone(bone, playerInstance);

I’m so close. Once I got this weapon issue sorted, I will implement vat + instances to my project.

(Apologies for spamming this post)

“Render targets” time is the time spent rendering all RenderTargetTexture textures. It takes into account the rendering of shadow maps and effects layers, among others.

Are you able to provide a small repro demonstrating the problem with the weapon?

Ok thanks for the clarifications @Evgeni_Popov, shadows were the cultprit, removing them brings render targets to close to 0


Abour the weapon issue, here is a playground, not quite the same code, but has the same with issue, it’s in the right position, but it’s not following the animation.

With BVA, there’s no more skeleton or bones, only a texture with the list of transformed vertices for all the frames of the animation, so attaching a mesh to a bone won’t work.

It’s possible to get it to work to some extent, though, but it’s a bit involved.

You need a skeleton to attach the weapon to, there’s no way around that. So, a skeleton should be created for each animation (5 in the PG). As each skeleton must drive a different transform node hierarchy (actually it’s the other way around for glTF animations, but it does not change the explanation), we need to duplicate the mesh hierarchy 5 times too and connect this hierarchy to the skeleton hierarchy: the 5 skeletons and the 5 mesh hierarchies are in the skeletonForAnim and rootForAnim arrays in the PG below, respectively.

Once these objects are created, you have to retarget the animation group so that it animates the transform nodes of the corresponding rootForAnim: AnimationHelper.RetargetAnimationGroupToRoot is doing that. Also, the skeleton must be retargeted so that the bones correspond to the transform nodes animated by the animation group: AnimationHelper.RetargetSkeletonToAnimationGroup is doing that.

Now that everything is in place, you have two things to do each frame (lines 139-147):

  • update the bones in the skeleton with the corresponding transform node matrices: call skeleton.prepare(). That’s normally done automatically for you by the framework for all meshes that have a skeleton, but in this case we don’t have any mesh that is using the skeletons we created.
  • synchronize the animations of the weapons (coming from the animation groups) with the BVA => we use the current time of the BVA manager to compute the frame to go to in the animation groups

Now, the caveat…

We have a single animation group / skeleton for each animation (run, idle, …), so the weapon animation for run / idle / … is the same for all instances that uses this animation. If you use a random starting offset for a BVA animation (line 96), then the weapon animation won’t be synchronized with the BVA animation. That’s why in the PG I use a constant offset for all instances. It does not have to be 0 (as you can see in the PG), but it must be the same value for all animations of the same type (you could have a different value for the run animation vs the idle animation, for eg).

If you want to be able to use different offsets for a given animation, you will have to duplicate the animation group / skeleton so that you have as many (animation group / skeleton) as there are different offsets (for a given animation).

PG: