This is a more general question about the animation capabilities of babylon.js
I would like to load a .glTF model and animate it, however I would like to be able to do so in a “live” fashion, meaning that I would periodically fetch the next frame of the whole animation e.g. via a websocket, then update my model, fetch the next frame and so on. This would enable a more adaptive animation to be played. Would something like this be possible using babylon.js?
As a first step I have been trying to see if it is possible, using scene.registerBeforeRender to play the animations part of my .glTF file “frame by frame”, so instead of playing the animation group, I try to use registerBeforeRender to fetch a frame. Currently I load the whole animation group, so this is sort of pointless still, as I am just using goToFrame to fetch the currFrame of the animation and then play that frame. See the playground for that here: Babylon.js Playground
However my idea is that we could do the same thing, if we just replace the .goToFrame(currFrame) (linewith a function that does sort of the following:
fetching the currFrame from the websocket
Update our model - not sure how this works with setting up the AnimationGroups yet as currently each frame is already part of the group, so what is the approach if I fetch the frames, without them being part of the AnimationGroup or attached to my model/skeleton already?
It still feels sort of rude to just randomly ping people but oh well: @Deltakosh, @bghgary seem to know a lot about the animation system in babylon.js
It doesn’t sound like you want to use animations at all. It sounds like you are streaming the frames down such that each frame contains all of the properties that are animating. If that’s the case, just set the properties directly?
You could do something like that but you are going to experience a ton of jitter in the animation from delays in the timing of frame packets arriving. You will probably want to use some sort of buffering strategy to try and account for this… i.e on its simplest level every XX milliseconds check if the next frame is available from the socket and goto it.
You would likely have better luck with something like the websocket just telling an animation to start playing rather syncing it on a frame by frame basis.
Yes, this scenario will have to deal with smoothing / synchronization / lag mitigation, etc. as any multi-user experience / streaming experience requires. I’m just saying that using animations / animation groups probably won’t help with this.
Yes, each frame will contain the properties and correct timing. Could you please elaborate on what you mean by setting the properties directly? How would that work in Babylon.js?
But I am unsure of what properties we are talking about. For example the playground animation group above consists of the targeted animations for each part of my skeleton. Now assuming I get a frame which will contain information about the position, quaternions, … of each bone at that frame, can I use methods described here: Bones and Skeletons | Babylon.js Documentation (e.g. bone.setPosition) to set the properties inside scene.registerBeforeRender and achieve animation in this way?
Yeah that is exactly the idea. You are just passing along the relevant transform data (position/rotation) of where any movable part should currently be.
What exactly are you trying to do? Just play an animation that multiple people can watch at the same time in sync?
Ok cool I will test it in the coming days and report back here. The whole idea is to stream animations from a server, to my web application. This is part of a larger project of 3D avatar animation on the web. The incoming animation data will be computed on the server-side and then sent to the app, hopefully improving the interactive aspect of the avatars. But I am still in the early stages of the Babylon.js learning journey as well as this project
Very cool project and good luck! This approach should work fine for you especially if a server is dynamically computing some animations. Just use the websocket to transmit where the various transforms should be for the frame being broadcast like @bghgary was suggesting. And then have the client app receive the data and assign the transforms to the correct place.
