Animations: storing key frames in typed arrays

[kzhsw]

Keyframes in Babylon.js are stored with array of objects. Some animations exported with millons of key frames, with the object mode, it would take more memory than stored with typed arrays, like Float32Array.
Without the object per frame, it would be much harder for features like per-frame interpolation and tangent to be implemented, so this could be an optional feature for those suffers too many keyframes without these advanced features.

See code in playground:

Babylon.js Playground

TODOs

• compact for gltf with 4-bytes alignment for each item
• deleteRange recalculate keys if removed first or last key
• AnimationCurveEditor (maybe convert it back to Animation)
• per-frame interpolation
• data.interpolation === ‘CUBICSPLINE’ and tangent

[sebavan]

You will at best only have half the data I believe by going from number to float32 ?

I am wondering if the scale of the win is worth the effort in this area or if we should look into other approaches.

@PatrickRyan, @bghgary are you aware of anything we could use ?

[bghgary]

@sebavan

You will at best only have half the data I believe by going from number to float32 ?

I’m not sure about this. JavaScript arrays probably have overhead compared to a typed arrays? It’ll probably be more than half.

It will be nice to be able to support typed arrays directly. The glTF loader will then be able to do less translation.

BabylonJS/Babylon.js/blob/master/packages/dev/loaders/src/glTF/2.0/glTFLoader.ts#L1669-L1713

[sebavan]

@kzhsw we are open to PR regarding this and @bghgary sold it to me with the gltf case It is quite compelling to have a direct load.

[kzhsw]

Here are some benchmark results after patching the glTF loader.

Heap Snapshot size(after GC):
BABYLON.Animation: 416MB

screenshot

CompactAnimation: 182MB

screenshot

gltf-transform inspect report of the model

 ANIMATIONS
 ────────────────────────────────────────────
┌───┬──────────┬──────────┬──────────┬──────────┬───────────┬──────────┐
│ # │ name     │ channels │ samplers │ duration │ keyframes │ size     │
├───┼──────────┼──────────┼──────────┼──────────┼───────────┼──────────┤
│ 0 │ Take 001 │ 1,215    │ 1,215    │ 66.667   │ 2,431,215 │ 18.87 MB │
└───┴──────────┴──────────┴──────────┴──────────┴───────────┴──────────┘

[kzhsw]

Initial POC on playground:

Using the original BABYLON.Animation:

Babylon.js Playground

Using CompactAnimation:

Babylon.js Playground

A demo model with 759,501 keyframes from glTF-Transform issue tracker:
Arm.zip (3.0 MB)

[kzhsw]

Morph Target supported.

Babylon.js Playground

Demo models:
AnimatedMorphSphere.glb
MorphStressTest.glb

[kzhsw]

Initial and incomplete support for GLTF serializer:

Babylon.js Playground

[kzhsw]

For AnimationCurveEditor it is hard to fully adapt it to CompactAnimation, maybe the fast way it to convert CompactAnimation to Animation, for which a function named compactAnimationToAnimation can be found in my last posted playground link.

If working with AnimationGroup the Animation and the target of animation can be found at AnimationGroup#targetedAnimations, where animations inside the animations property can be replaced with converted one, and animation inside TargetedAnimation itself, and animations inside AnimationGroup.animatables[number]._runtimeAnimations._animation.

It is also possible that the inspector targets an IAnimatable, eg. one Mesh, in this case all the AnimationGroups in scene and all the _activeAnimatables and animations of the scene should be iterated to find all references to the animations.

Targeting one Animation is simlar to the case above, but all the meshes, transformNodes, cameras, particleSystems, materials, morphTargetManagers, skeletons in the same scene should be iterated to find all references to the animations.

Another simple method is to check for existance of CompactAnimation and disable or hide the Edit button which opens the curve editor.

The event callback for AnimationCurveEditor, where all the convertion should be done before:

BabylonJS/Babylon.js/blob/9f3e6149731daf962f6325578d1d5aa8a17c61d7/packages/dev/inspector/src/components/actionTabs/tabs/propertyGrids/animations/curveEditor/animationCurveEditorComponent.tsx#L114

[kzhsw]

Since replacing animation object in scene is complicated, maybe there could be another way by replacing [[Prototype]] of the animation object using Object.prototype.__proto__ or Object.setPrototypeOf or Reflect.setPrototypeOf.

[sebavan]

Why not trying in the source code instead ?

[kzhsw]

Does the source code here refer to my last 2 replies about AnimationCurveEditor and replacing animation?

[sebavan]

@kzhsw

Since replacing animation object in scene is complicated, maybe there could be another way by replacing [[Prototype]] of the animation object using Object.prototype.__proto__ or Object.setPrototypeOf or Reflect.setPrototypeOf.

in reference to the above, I was wondering why not changing the system in the babylon source to prevent the need of prototype changes ?

[kzhsw]

Example of replacing animation in scene, could be imcomplete

/**
 * The callback for animation iteration in scene
 */
interface IterateAnimationCallback {
    /**
     * The callback for animation iteration in scene
     * @param animation current animation
     * @param key key of context
     * @param context object holding this animation
     * @returns false to stop the iteration
     */
    (animation: Animation, key: number | string, context: any): void | boolean;
}

function iterateAnimations(scene: Scene, callBack: IterateAnimationCallback): void {
    const {
        animations,
        animatables,
        animationGroups,
        transformNodes,
        meshes,
        materials,
        textures,
        cameras,
        particleSystems,
        morphTargetManagers,
        skeletons,
        spriteManagers,
        postProcesses,
    } = scene;

    let len = animations?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            if (callBack(animations[i], i, animations) === false) {
                return;
            }
        }
    }

    len = animatables?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const animatable = animatables[i];
            const runtimeAnimations = animatable.getAnimations();
            const length = runtimeAnimations?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const runtimeAnimation = runtimeAnimations[i];
                    const animation = runtimeAnimation.animation;
                    if (animation &&
                        callBack(animation, '_animation', runtimeAnimation) === false) {
                        return;
                    }
                    const target = runtimeAnimation.target;
                    if (target?.animations) {
                        const iAnimatable = target as IAnimatable;
                        const animations = iAnimatable.animations;
                        const animationsLength = animations?.length;
                        if (animations && animationsLength) {
                            for (let k = 0; k < animationsLength; k++) {
                                const animation = animations[k];
                                if (animation && callBack(animation, k, animations) === false) {
                                    return;
                                }
                            }
                        }
                    }
                }
            }
        }
    }

    len = animationGroups?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const group = animationGroups[i];
            const targetedAnimations = group?.targetedAnimations;
            let length = targetedAnimations?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const targetedAnimation = targetedAnimations[j];
                    const animation = targetedAnimation?.animation;
                    if (animation && callBack(animation, 'animation', targetedAnimation)) {
                        return;
                    }
                    const target = targetedAnimation.target;
                    if (target?.animations) {
                        const iAnimatable = target as IAnimatable;
                        const animations = iAnimatable.animations;
                        const animationsLength = animations?.length;
                        if (animations && animationsLength) {
                            for (let k = 0; k < animationsLength; k++) {
                                const animation = animations[k];
                                if (animation && callBack(animation, k, animations) === false) {
                                    return;
                                }
                            }
                        }
                    }
                }
            }
            const animatables = group?.animatables;
            length = animatables?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const animatable = animatables[i];
                    const runtimeAnimations = animatable.getAnimations();
                    const length = runtimeAnimations?.length;
                    if (length) {
                        for (let j = 0; j < length; j++) {
                            const runtimeAnimation = runtimeAnimations[i];
                            if (runtimeAnimation.animation &&
                                callBack(
                                    runtimeAnimation.animation,
                                    '_animation',
                                    runtimeAnimation
                                ) === false) {
                                return;
                            }
                        }
                    }
                }
            }
        }
    }
    len = morphTargetManagers?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const morphTargetManager = morphTargetManagers[i];
            const targetsLength = morphTargetManager.numTargets;
            if (!targetsLength) {
                continue;
            }
            for (let j = 0; j < targetsLength; j++) {
                const target = morphTargetManager.getTarget(j);
                const animations = target?.animations;
                const length = animations?.length;
                if (!length) {
                    continue;
                }
                for (let k = 0; k < length; k++) {
                    const animation = animations[k];
                    if (animation && callBack(animation, k, animations) === false) {
                        return;
                    }
                }
            }
        }
    }

    len = skeletons?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const skeleton = skeletons[i];
            const bones = skeleton?.bones;
            const length = bones?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const target = bones[j];
                    const animations = target?.animations;
                    const animationsLength = animations?.length;
                    if (!animationsLength) {
                        continue;
                    }
                    for (let k = 0; k < animationsLength; k++) {
                        const animation = animations[k];
                        if (animation && callBack(animation, k, animations) === false) {
                            return;
                        }
                    }
                }
            }
            const animations = skeleton?.animations;
            const animationsLength = animations?.length;
            if (animationsLength) {
                for (let j = 0; j < animationsLength; j++) {
                    const animation = animations[j];
                    if (animation && callBack(animation, j, animations) === false) {
                        return;
                    }
                }
            }
        }
    }

    len = spriteManagers?.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const spriteManager = spriteManagers[i];
            const sprites = spriteManager?.sprites;
            const length = sprites?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const target = sprites[j];
                    const animations = target?.animations;
                    const animationsLength = animations?.length;
                    if (!animationsLength) {
                        continue;
                    }
                    for (let k = 0; k < animationsLength; k++) {
                        const animation = animations[k];
                        if (animation && callBack(animation, k, animations) === false) {
                            return;
                        }
                    }
                }
            }
            const animations = spriteManager?.texture?.animations;
            const animationsLength = animations?.length;
            if (animationsLength) {
                for (let j = 0; j < animationsLength; j++) {
                    const animation = animations[j];
                    if (animation && callBack(animation, j, animations) === false) {
                        return;
                    }
                }
            }
        }
    }

    let iAnimatables: IAnimatable[] = [];

    len = transformNodes?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(transformNodes);
    }
    len = meshes?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(meshes);
    }
    len = cameras?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(cameras);
    }
    len = materials?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(materials);
    }
    len = textures?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(textures);
    }
    len = particleSystems?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(particleSystems);
    }
    len = postProcesses?.length;
    if (len) {
        iAnimatables = iAnimatables.concat(postProcesses);
    }

    len = iAnimatables.length;
    if (len) {
        for (let i = 0; i < len; i++) {
            const iAnimatable = iAnimatables[i];
            const animations = iAnimatable?.animations;
            const length = animations?.length;
            if (length) {
                for (let j = 0; j < length; j++) {
                    const animation = animations[j];
                    if (animation && callBack(animation, j, animations) === false) {
                        return;
                    }
                }
            }
        }
    }
}

/**
 * Replace animation from scene
 * @param scene The scene holding animation
 * @param animation The animation to be replaced
 * @param replacement The animation to be replaced to
 */
export function replaceAnimation(scene: Scene, animation: Animation, replacement: Animation): number {
    let count = 0;
    iterateAnimations(scene, (current, key, context) => {
        if (animation === current) {
            context[key] = replacement;
            count++;
        }
    });
    return count;
}

[kzhsw]

That reply is related to the AnimationCurveEditor in inspector, which can edit animation’s key frames with advanced features like per-frame tangent and interpolation, which is hard to implement for typed-array-based CompactAnimation.
To make CompactAnimation work with AnimationCurveEditor, the most likely possible way is to convert CompactAnimation back to regular Animation, and replace usages of the CompactAnimation in scene with converted Animation, the replies here and
here are different ways of replacing animation in scene, and the reply here is a demo of iterating all animations in scene and replace, which shoud be the prefered way since changing prototype is discouraged according to docs.

screenshots of advanced feature of AnimationCurveEditor and keys after using that

[kzhsw]

Initial PR here:

Animations: storing key frames in typed arrays

[sebavan]

Nice it will be easier to review and discuss the integration.

[kzhsw]

Playground targeting the PR:

Babylon.js Playground

Sandbox targeting the PR:

Babylon.js Sandbox - View glTF, glb, obj and babylon files

[jeremy-coleman]

I would just like to say, i am grateful for your effort and enthusiasm you’ve put into this and other refactoring projects. I feel that sometimes, efforts like this can seem to be met with a bit of a cold response, but just know this is great and thank you.