Custom Rendering of Scenes to Video

Questions
1

Hi all,

I’m new to BabylonJS. Super impressed with it’s power so far though.

My question is in regard to rendering scenes to video. According to the docs, the built-in RecordVideo feature only produces .webm files at the same dimensions as the canvas. However, I’d like to be able to render animated scenes to a series of still frames at any resolution, then use ffmpeg or similar tool to create a video from them in any format I choose.

The following code works somewhat but it appears to skip around 5 or so frames between each capture (assuming there are 60 frames for one second of playback). This seems to be consistently the case no matter how long each capture takes. Inserting deliberate random long wait periods between each capture seems to have no effect. I’ve also tried implementing this using (un)registerAfterRender() with similar results. Nothing I do seems to produce a sequence of images that produce smooth 60 FPS animation at a speed consistent with the live, uncaptured rendering rate directly to the display.

Any insight as to what I may be doing wrong would be very much appreciated. Thanks in advance!

var scene;
var engine;
var camera;
var frameNum = 0;
var screenshot_captured = true;

function post_image_to_http_server(encoded_image)
{
   var frameNumStr = ("0000" + frameNum).slice(-4);
   frameNum++;
   var xhttp;
   var encoded_image = encoded_image.replace(/^data:image\/(png|jpg);base64,/, "");
   xhttp = new XMLHttpRequest();
   xhttp.open("POST", "write_encoded_image_to_disk.php", true);
   xhttp.setRequestHeader( "Content-type", "application/x-www-form-urlencoded" );
   xhttp.send("fname=" + frameNumStr + "&fcontents=" + encodeURIComponent(encoded_image));
   screenshot_captured = true;
}

  function render_scene_frame_to_image_file()
  {
     BABYLON.Tools.CreateScreenshotUsingRenderTarget( engine, camera, {width:1920, height:1080}, 
        function(data)
        {
           post_image_to_http_server(data);
        }
     );
  }

  var canvas = document.getElementById("renderCanvas");
  function createScene () {
     scene = new BABYLON.Scene(engine);

     camera = new BABYLON.ArcRotateCamera("ArcRotateCamera", 1, 0.8, 5, new BABYLON.Vector3(0, 0, 0), scene);
        
     camera.attachControl(canvas, true);
    
     camera.lowerRadiusLimit = 2.5;
     camera.upperRadiusLimit = 10;
     camera.pinchDeltaPercentage = 0.01;
     camera.wheelDeltaPercentage = 0.01;
    
     scene.clearColor = new BABYLON.Color3(0.0, 0.0, 0.0);
       
     BABYLON.ParticleHelper.CreateAsync("sun", scene).then((set) => {
        set.start();
     });
    
     return scene;
  }

  engine = new BABYLON.Engine(canvas, true, { preserveDrawingBuffer: true, stencil: true });
  createScene();

  engine.runRenderLoop(function () {
     if (screenshot_captured) {
        screenshot_captured = false;
        scene.render();
        render_scene_frame_to_image_file();
     }
  });

  // Resize
  window.addEventListener("resize", function () {
     engine.resize();
  });
1 Like
2

Hello and welcome,

This is probably because of the time spent to send the data

The best option here is to force the system to use a fixed Delta time (time spent between two frames)

By default this value is dynamic to adapt to various hardware but in your case you want it to be fixed

To do so, call:

scene.useConstantAnimationDeltaTime = true;

You may also just need to no use the runRenderLoop but instead just call scene.render()

1 Like
3

Haha! That did the trick! All it was missing was the useConstantAnimationDeltaTime. Thanks a million! I spent so much time on this. Don’t think I would have ever thought of that on my own. You’ve been a huge help Deltakosh!

4

My pleasure:)

5

Could you be so kind to publish the whole solution (php included)?

6

Hi Darius and Deltakosh,

Like Darius I was having a few issues with the webM output which is down to the browser of course not BabylonJS.

Quality for me (linux) was best on Firefox but there is some kind of a memory thing going on in Firefox and if I made a video over a certain filesize it failed to save. Other browsers did not have this issue. Also my background is monochrome and flickers in the resultant video - both brightness but also colour value wobble in some areas. Anyway I did not want to spend time looking into issues specific to any particular browser so I followed your route.

Thanks for the code segment - all I had to do was add the PHP bit to save out the frame files.

I am running apache2 locally (linux) I got around the usual limitation that files cannot be saved to filespace outside of the locally running server by adding a linux link which made my working directory appear to be under the server but was actually in a more convenient location - this worked great.

I then started to look into issues of timing and synchronicity - the BabylonJS Render Loop and then the time spent to save files and so on.

After a short while I realised I did not need to get into this side of things since my project is not game based, there is no issue with video capture slowing down gameplay or anything like that - if everything slows down when filming then that is fine.

All I did in the end was add some logic to my animation logic timing loop that slowed it down dramatically when I am recording by this method but leaves it normal speed if I am settling for the lower quality of webM.

I simply look at how long the ‘frame save’ operation takes, multiply it by ten to be on the safe side and then go for a coffee when I want to film an animation in high quality - many performance issues in software can be rectified by reflecting on the fact that really what you wanted was a coffee. :grinning:

Great results.

7

I am working the same things. Is it possible to share us your write_encoded_image_to_disk.php file?