Ok, I see @labris I moved the camera around and did not get any lag. Gonna revisit webworkers then 
Sounds great but the PR says:
Not supported yet:
per-frame interpolation or tangent
Ohh I don’t know. This is a quite a limitation 
In babylon.js, the animation system allows some frames in one animation channel have tangent while some does not, but Float32Array is not flexable enough for data structures like that, so all frames in one animation channel need to have the same interpolation mode, and should be one of LINEAR, STEP, and CUBICSPLINE, per gltf spec.
The CUBICSPLINE mode where all frames in one animation channel have tangents are supported, but currently most animations in gltf are bakes into LINEAR mode by animation softwares like blender.
Found some time to work on this again.
Build pipeline now puts out individual animation files (intead of one larged merged file). I discovered that Blender lets you export an Armature-only glb (i.e. without mesh). 
Could have sworn this was not possible a couple of months ago.
Now that I do not have to generate a large file (full of animations) anymore, I can quickly edit animations and test them in-game.
Game code is bit crude at the moment. Animations are either “preload” or “on demand”. Currently, on demand means right before I want to play an animation, it gets streamed in if missing. While this pretty much eliminates loading times for animations, I do get a frame skip when streaming in from HD (probably gets worse when streaming from web server). So there needs to be some predictions when an animation is likely needed. Or I could keep them streaming in on player idle times until I have a full set (ready to be retargeted). idk yet.
Finally, might be obvious but, do not forget to look for existing animation groups first (scene.getAnimationGroupByName) before you stream them in.
The irony is even though this big chunk of loading time is gone now, I still need to get rid of 10 seconds 