Animations synchronized with the rendering loop

Hello everyone,
This work is inspired by the topic BABYLON.Animation is not synchronized with the rendering loop? By default the BABYLON animations are not synchronized with a gaming loop, but sometimes we desperately need those. I will explain later when and why.
I created a separate class that will help to make BABYLON animations work in synchronous way. It’s a demo, that you will be able to adapt and extend for your project needs.
My approach relies on some JavaScript very specific features. Before proceeding, please be sure that you are familiar with such features as difference between var and let, you know when JS copies values and when it copies links during assignment, and you know how JS closures work. This topic is for PRO! :slightly_smiling_face:
Let the game begin!

Foreword. When we need synchronous animations?
Animations can be:
a) Not synced. Such animations progresses apart from gaming loop. For those animations it doesn’t matter how many rendering events passed and how many frames the animation lost. Such animation tends to finish within time range of real world (for example, 5 seconds of real world time). For example, if you want to animate the watch that shows how long current gaming round goes, then I think you want to use this approach. In JavaScript not synced animations can be implemented using setInterval/setTimeout. And I guess, this is how BABYLON animations work.

b) Synced. Such animations progresses on every determined amount of rendering events (or ticks). Usually they progresse on every tick if there is no FPS lock or other rate function. Animation duration should be determined by the amount of frames instead of amount of seconds. For example, if you want an animation for approximately 5 seconds and your FPS is 60, then you will probably want to create animation consisting of 60 * 5 = 300 frames. Such animations don’t care about the real time, it will last exactly the determined amount of frames, despite how long these frames will be. In JS these animations could be achieved by using requestAnimationFrame. JS callbacks fire with some delays, so the longer animation you have the bigger gap with the real time you will get. This means that you should never think about such animations in terms of frames/fps = realTime. Compose such animations only relying on desired amount of frames that will fit the game flow. However, such animations have benefits. For example, if the length difference between two animations is 200 frames, then this difference will always be like that. No exceptions. Synced! This fact is very critical for animations that affect gameplay (and ability to win): weapon switching, skill reloading. Synced animations also allow to pause the game correctly. You might be sure that the game state will be exactly the same for the synced processes before and after the pause. In order to pause the game you just need to pause the game loop and all synced processes will be paused automatically.

When to use?
When animations are not important to game state it still could be not synced animations, because it’s easier to use them in BabylonJS. I assume that you will not worry much if blinking advertisement on the building or weapon bouncing will play some more frames after the pause, it doesn’t affect the gameplay at all and probably will not be noticed by the most of players.

However, such things as weapon switching, firing and reloading determine when you will be able to do the next shot, your life depends on it. Synced approach will provide the more consistent game state each round and your game strategy will be more protected from random factors, such as CPU performance.

Issues
Can we be sure that not synced animations always corresponds to real time? Of course no. JavaScript is single-threaded. If you have intensive function running right now, then your setTimout(myFunc, 5000) may fire much later.

How the screen rendering rate should be synchronized with animation frame rate? In the middle 10’s the 60 FPS was the de facto standard for screen refreshing rate. Running the game with higher FPS had no sense, so all animations could be bounded by 60 animation frames per second. Now when the high Hz monitors became widely available we run into a trouble. We don’t know the target refresh rate and we don’t know how the particular user PC will be able to maintain such rate. The 200 frames animation could take about 3.33 and 1.22 seconds on 60 and 164 frames per second. We can try to detect the current refresh rate and try to automatically generate missing frames for animations (using interpolation approach for example), but here is where PC performance come in play. If we have 164 Hz monitor it doesn’t mean that PC can maintain that all the time. Even checking the current FPS could be inconsistent in different levels for different computers. Of course we can still just bound our animations speed by old good 60 FPS. But when you are rendering the game at 164 FPS and your computer plays it smoothly, the 60 FPS animations look very unnatural. For example, you can even see that in Doom Eternal during some glory kills (some boss fights). So, yeah, this is a pretty big concern. And the creation of adaptive animations (with variable frame rate) that will be easy to create and use is not an easy task.

As I said before, the synced animations create a gap with a real time. If your animation is long and it suppose to produce sounds (for example, long weapon reloading animation), you can face the sound out-of-sync problem. So, advise here is not to use the single audio file to reproduce the whole sound sequence. Just split your animation into small parts and play each sound separately, when the target frame is reached. This will protect you even if the target PC did the bad performance at some point. This is fair advise for both types of animations, by the way.

0. What we have?
So, the BABYLON already has a good way to describe the animations: target property, keys and some other parameters. All we need is to make it run synchronously (I dream we could have just a flag in Babylon for that). So, all we need to change is the way how those animations are processed.

1. Preparation. Interpolation map.
Most key framed animations are based on interpolation. We calculate how we should change the value on each frame between two key frames. Of course, the value that we should add or subtract could be different on each segment of animation. For this purpose I create a so called animationMap from animation key frames. This map might be named also as incremental map or interpolation map.
It does the following:

// converts this
const frameRate = 60;
const keys = [];
    keys.push({
        frame: 0,
        value: -3
    });
    keys.push({
        frame: 2 * frameRate,
        value: 3
    });
    keys.push({
        frame: 3 * frameRate,
        value: 3
    });
    keys.push({
        frame: 5 * frameRate,
        value: -3
    });
// into this
{
    120: 0.05
    180: 0
    300: -0.05
}

Yeah, it just shows how much we should add to our target property before reaching the next key frame. So, before the frame 120 we add 0.05, before frame 180 we add 0, and before frame 300 we add -0.05.

Don’t forget to create animationMap, before playing the animation. I recommend to do that right after keys. For example:

sphereAnimation.setKeys(keys);
// !!! Additional important step.
sphereAnimation.animationMap = AnimationHelpers.getAnimationMap(sphereAnimation);

sphere.animations.push(sphereAnimation);

Here is the function for that:

    static getAnimationMap(animation) {
        const animationMap = {};
        const keys = animation.getKeys();
        if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_FLOAT) {
            for (let k = 0; k < keys.length - 1; k++) {
                const nextKeyFrame = keys[k+1].frame;
                const incrementValue = (keys[k+1].value - keys[k].value) / (nextKeyFrame - keys[k].frame);
                animationMap[Math.round(nextKeyFrame)] = incrementValue;
            }
        }
        else if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_VECTOR3) {
            for (let k = 0; k < keys.length - 1; k++) {
                const nextKeyFrame = keys[k+1].frame;
                const incrementValue = keys[k+1].value.subtract(keys[k].value).scale(1/(nextKeyFrame - keys[k].frame));
                animationMap[Math.round(nextKeyFrame)] = incrementValue;
            }
        }
        else if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_MATRIX) {
            for (let k = 0; k < keys.length - 1; k++) {
                const nextKeyFrame = keys[k+1].frame;
                const incrementValue = BABYLON.Matrix.Lerp(keys[k].value, keys[k+1].value, 1/(nextKeyFrame - keys[k].frame));
                animationMap[Math.round(nextKeyFrame)] = incrementValue;
            }
        }
        return animationMap;
    }

For now it supports types: FLOAT, VECTOR3 and MATRIX. You can add other types later. Be careful we can not use Lerp function from BABYLON libraries here. Because Lerp in BabylonJS is initialValue + incrementValue, but we need only the incrementValue for the map. Lerp is OK for MATRIX though, because MATRIX is used by imported animations mainly which describes every frame with no gaps. So later we will just overwrite the target property instead of incrementing it. However this will not work if our MATRIX animation has gaps between key frames.

You may notice some code duplication in the function above, but it’s intentional. For CPU it’s harder to cache for->if than if->for. I am not sure how it’s right for JavaScript, but in general it’s better to avoid if inside of for in performance critical places.

Since we are here, let’s check how we will modify the target property soon.

    static modifyTargetProperty(obj, animation, value) {
        const targetPropertyPath = animation.targetPropertyPath;
        const length = targetPropertyPath.length - 1;
        for (var i = 0; i < length; i++) {
            obj = obj[targetPropertyPath[i]];
        }

        if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_FLOAT) {
            obj[targetPropertyPath[i]] += value;
        }
        else if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_VECTOR3) {
            obj[targetPropertyPath[i]] = obj[targetPropertyPath[i]].add(value);
        }
        else if (animation.dataType === BABYLON.Animation.ANIMATIONTYPE_MATRIX) {
            obj.updateMatrix(value, false, true);
        }
    }

As I said, we increment FLOAT and VECTOR3 and overwrite the MATRIX. Be careful, we use var in the loop intentionally, because we want to use this value after the loop. We don’t traverse the property path to the end, otherwise we will receive the value itself instead of the ‘place’ where it should be saved. It’s PRO coding. :grin:

And here is the helper function that extracts current IncrementValue from the animationMap. Nothing hard, but I sort keys for extra safety. timer here represents the current frame.

function getAnimationCurrentStep(animationMap, timer) {
    const keys = Object.keys(animationMap).map((key) => parseInt(key)).sort((a, b) => a - b);
    const maxK = keys.length - 1;
    let k = 0;
    while (timer > keys[k] && k < maxK) {
        k++;
    }
    return animationMap[keys[k].toString()];
}

2. Main processing.
In BabylonJS we had an Animatable object. It is a special control object that keeps a link to running animation and allows us to control this animation. We need something similar for our synced animations, however I decided that we don’t need to recreate the interface of original Animatable object completely. Let’s just create only what we need for now. A helper function for that is:

    static generateAnimatableObject() {
        return {
            timer: 0,
            paused: false,
            stoped: false,
            pause: function () {
                this.paused = true;
            },
            stop: function () {
                this.stoped = true;
            },
            reset: function () {
                this.timer = 0;
            }
        
        };
    }

And here we go. The main function:

    static beginAnimation(scene, animation, animatedObject, startFrame, endFrame, onAnimationEnd, animatable) {
        animatable = animatable || AnimationHelpers.generateAnimatableObject();
        const keys = animation.getKeys();
        const lastFrame = keys[keys.length - 1].frame;
        endFrame = endFrame > lastFrame ? lastFrame : endFrame;
        animatable.timer = startFrame;
        const frameDuration = 1/animation.framePerSecond;
        let lastFrameTime = 0;
        animation.runtimeAnimations.push(animatable);
        const animationHandler = function () {
            if (animatable.stoped) {
                scene.onBeforeRenderObservable.removeCallback(animationHandler);
                animation.runtimeAnimations.splice(animation.runtimeAnimations.indexOf(animatable), 1);
            }
            if (!animatable.paused && AnimationHelpers.frameTimePassed(lastFrameTime, frameDuration)) {
                animatable.timer++;
                const incrementValue = getAnimationCurrentStep(animation.animationMap, animatable.timer);
                lastFrameTime = (new Date()).getTime();
                if (incrementValue !== 0) {
                    AnimationHelpers.modifyTargetProperty(animatedObject, animation, incrementValue)
                }
                if (animatable.timer >= endFrame) {
                    scene.onBeforeRenderObservable.removeCallback(animationHandler);
                    animation.runtimeAnimations.splice(animation.runtimeAnimations.indexOf(animatable), 1);
                    if (onAnimationEnd) {
                        onAnimationEnd();
                    }
                }
            }
        }

        scene.onBeforeRenderObservable.add(animationHandler);
        return animatable;
    }

First it generates the control object if it wasn’t provided. Then setups important variables.
Important step is animation.runtimeAnimations.push(animatable);. We may sometimes use runtimeAnimations to check that animation is currently running, don’t forget about that. Please also don’t forget that our animatable object is different, comparing to BABYLON Animatable.

Then the function creates animationHandler. It’s a callback function that runs on every tick, it increments timer and target property value accordingly. It also checks stoped and paused flags and fires onAnimationEnd callback when it’s time. animationHandler can also detach itself from rendering loop, when no longer needed.

frameTimePassed is a simple FPS lock:

static frameTimePassed (lastFrameTime, frameDuration) {
    return (((new Date()).getTime() - lastFrameTime) / 1000) > frameDuration;
}

Saving concerns.
mesh.serialize saves animations as well. It would be bad for our case. Because after loading we will have two different animation objects. One is from the code and one is from the save file. The animation inside of animated object and animation created in the code are not the same anymore.
Probably the easiest way is to do not save the synced animation:

for (let a = mesh.animations.length - 1; a >= 0; a--) {
    if (mesh.animations[a].animationMap) {
        serializationObject.animations.splice(a, 1);
    }
}

Just save animation name and current timer value somewhere. On loading just do again sphere.animations.push(sphereAnimation); and run beginAnimation with saved timer value as startFrame.

3. Group animation.
The most of the time we need to animate more than one property of animated object. For this purposes BabylonJS has grouped animations, which are processed by beginDirectAnimation function. And we need synced version of it as well.
First we need more complex control object. The helper function below creates the grouped control object of given capacity.

    static generateDirectAnimatableObject(n) {
        const animatables = [];
        for (let i = 0; i < n; i++) {
            animatables.push(AnimationHelpers.generateAnimatableObject());
        }
        return {
            animatables,
            pause: function () {
                for (let i = 0; i < n; i++) {
                    this.animatables[i].pause();
                }
            },
            stop: function () {
                for (let i = 0; i < n; i++) {
                    this.animatables[i].stop();
                }                
            },
            reset: function () {
                for (let i = 0; i < n; i++) {
                    this.animatables[i].reset();
                } 
            }
        }
    }

And below is the function itself.

    static beginDirectAnimation(scene, animations, animatedObject, startFrame, endFrame, onAnimationEnd, animatable) {
        const length = animations.length;
        animatable = animatable || AnimationHelpers.generateDirectAnimatableObject(length);
        let callbackCounter = 0;
        for (let a = 0; a < length; a++) {
            animatedObject = animations[a].animatedObject || animatedObject;
            AnimationHelpers.beginAnimation(scene, animations[a], animatedObject, startFrame, endFrame, function () {
                callbackCounter++;
                if (callbackCounter >= length && onAnimationEnd) {
                    onAnimationEnd();
                }
            }, animatable.animatables[a]);
        }
        return animatable;
    }

It’s pretty straightforward. We launch beginAnimation on every included animation passing to them the dedicated control object from grouped control object. The only interesting thing is how we fire the callback. We count how much of animations have finished. When all of them have finished we fire the callback of grouped animation.

4. Improvements and alternatives.
As you can see, my example doesn’t support animation speed and loop parameters. But it is easy to add them if you need.

Instead of creating incremental map we can just try to generate every missing frame between key frames, then we can just overwrite the target property on every tick. Probably it will increase the processing performance a little bit, however it will increase the preparation time and consuming of memory as well. It will also allow us to use Lerp function everywhere, like we did it for MATRIX values. Probably I should think about it more seriously. However we will lose the ability to skip the identical frames without comparing them. This is what we can do now if incrementValue is 0.

5. Objects Violations.
Some people might be not happy that we substitute some BabylonJS objects with objects that don’t obey the initial interface. TypeScript projects will be espicially angry about that. Well, to keep it simple I just implemented what we really need right now. If someone is not happy about that, it’s possible to implement the full interface or to extend the initial BabylonJS objects.

6. Demo.
Of course I did a playground. Unfortunately, I don’t know how to control the rendering loop in playground, that’s why my demo doesn’t show anything that original BABYLON animations can not do. However I left comments on how it can be used with rendering loop. Particularly like this:

engine.runRenderLoop(function () {
    if (!paused) {
        currentScene.render();         // Pause the main scene.
    }
    mainMenu.mainMenuScene.render();   // But keep the main menu always alive.
});

Manipulating the paused flag here you can stop and resume all synced animations instantly. No additional actions are needed.
I have put the whole AnimationHelpers class to demo, despite that I don’t use all features in this demo. But we have covered all of them in the text above.

The Link: https://www.babylonjs-playground.com/#EWLG0L#1

1 Like

and also onafterrenderobservable

The are other scene observables you can use.

I think you will find that it uses deltaTime

Hope these help in your quest for frame ‘synced’ animations.