Render animation to image sequence (or video) frame by frame

Questions
1

Hey folks,

I’m trying to render an animation (camera animation) out of a scene into an image sequence. I have been searching in the docs and forum as well as trying in a playground. This is what I have gotten so far:

In this PG, I imported a model then created an 240 frames camera animation fly around the model. My goal is render these 240 frames animation into 240 individual image file on disk. Then after I can use other approach (like ffmpeg) to convert this image sequence into whatever video format I need.

I was able to output some images, however, with some issues:

  1. I was expecting getting 240 .png files (from frame 0 to frame 239) on my local disk. But I only ended up getting around ~180 images. Some frames looks got dropping during exporting.
  1. The files written to disk have messed up names. The images before frame 100 can still exported with frame number in the file names, but any outputs after frame 100 all have random file naming. Are there any ways to name the output images in a better way such as: myOutput.####.png. (#### is 4 padding frame number)
  2. I was also trying to add camera motion blur. But somehow it returns error and how able to render motion blur at all.

Please let me know if there are solutions to these 3 issues.

Besides, if anyone has any other better ideas regarding how to render video from a scene please let me know too. As far as I know, render a image sequence first and convert to video file after is the safest way to implement this. Although I have found some other solutions here:

  • Use puppeteer: puppeteer-screen-recorder - npm
  • Use webrtc to stream the canvas to a separate server for rendering.
  • Use a worker thread to encode data from the canvas in an async way.
  • Use webcodecs to encode.

Some ideas come from this topic: Exact frame rendering of an animation into a video

I’m not sure if these above 4 solutions is better than the image sequence idea or not. Any insights and further details will be much appreciated.

Many thanks,
Tawibox

2

For video recording of the scene you can simply:

    const recorder = new BABYLON.VideoRecorder(engine)
    recorder.startRecording()

Have a look at the source how is it implemented: Babylon.js/packages/dev/core/src/Misc/videoRecorder.ts at master · BabylonJS/Babylon.js · GitHub

For image sequence I use and never had an issue:

    engine.stopRenderLoop()

    engine.runRenderLoop(async () => {
        scene.render()
        const data = await BABYLON.Tools.CreateScreenshotAsync(
            engine,
            camera,
        );
        // process data
    })Ï
3

@roland Thank you so much for the response!

For the image sequence:

I think the async/await might be the reason I couldn’t get every frame in the first place.

Can you add more detailed solution code in my original PG? I have been trying to apply your concept in my PG and unfortunately couldn’t be able to get it works…

Also do you have ideas for the motion blurs?

Many thanks!
Tawibox

1 Like
4

Here it is buddy:

EDIT:
Just counted the number of screenshots and I didn’t get 240 LOL… Hang on…

EDIT2:
It was ok, a lot of them were offscreen. I added a scrollbar to the image div so you can see them all :wink:

:vulcan_salute:

2 Likes
5

@roland Wow!! this is amazing!! Thank you so much for the super presentable code!!

Just wondering:

  • what was the root cause why the motion blur in my original PG didn’t work? My motion blur code looks pretty much as your code. Looks it is because of the rendering pipeline in my original PG?

  • is there a particular reason you changed the createScene function to a async func in your PG?

Appreciate the help again.
Tawibox

6

You are welcome!

You have to set the useGeometryBuffer to true when creating the post process when using with DefaultRenderingPipielin:

Otherwise you can’t use the await keyword.

1 Like
7

@roland
Thank you so much!! These all makes sense.

Sorry one last thing. I am trying to download all the image data with this function:

function downloadSerializedImage(serializedImageData, fileName) {
    // Create a link element
    const link = document.createElement('a');
    link.style.display = 'none';
    document.body.appendChild(link);
    // Set the href attribute of the link to the base64 image data
    link.href = serializedImageData;
    // Set a filename for the downloaded file
    link.download = fileName;
    // Simulate a click on the link to trigger the download
    link.click();
    // Clean up by removing the link from the DOM
    document.body.removeChild(link);
}

Trying to evoke this func after the addImage() function. However I can’t be able to download all 240 images. How should I do it? Is it still because of the async/await thing? Sorry I am very new to web dev. Still dont understand the entire Promise concept fully yet.

Many thanks,
Tawibox

8

I suggest to join all Base64 encoded pngs into one string, download it write a script which you can run locally. It could split the string into separate Base64 strings, convert them into binary pngs and save them to a local drive. If you’rein trouble doing this I can lend you a helping hand again :slight_smile:

Other solution is to create a REST API handler on a server (it can be your local machine) and fetch the data one by one to this API endpoint and save the data on a local drive. I’d prefer this solution but it’'require to run the Playground locally as well. Do you want to use the Playground to run your code or have you already set up you app to run on your local server?

Javascript runs in a single thread on the CPU. When you perform a time consuming operation this thread is blocked and the page becomes unresponsive unless you wrap that time consuming operation into with a Promise. You can use two approaches to achieve this.

1: using callbacks

function timeConsumingOperation() {
  return new Promise((resolve, reject) => {
    // do something time consuming here
    resolve() // call resolve when done (it's like a simple return from this function(
    resolve(someValue) // or you can resolve with a value (it's like returning a value from this function)
    reject() // or if something bad happens you can reject the Promise,
  })
}

You can then call this function:

timeConsumingOperation().then((someValue )=> {
  // do something with someValue
), () => {
  // rejected, do something
})

2: You can use async/await to achieve the same thing:

async function timeConsumingOperation() { // automatically wraps the function with a Promise
  return someValue
}

Calling the function:

const someValue = await timeConsumingOperation()

There is much more into JS Promises so I suggest you to look for more information on the web. It’s crucial to a web developer to master Promises.

1 Like
9

@roland

Thank you so much for all detailed responses!

I ended up successfully saving all png images to my local machine with the one long string solution you mentioned. (use python) This is a great step for me already in terms of prototyping! This entire video rendering process will eventually be running on a backend service. So this is a great enough for me now.

And thank you so much for the Promise explanation. I will definitely dive deeper for it. I have to master it soon or later haha.

Appreciate all the helps so far.
Best regards,
Tawibox

10

Anytime again :wink:

Good luck!

:vulcan_salute:

2 Likes