We noticed that someone else was having a similar issue HERE, but we are getting similar behavior when running Babylon in Chromium on NVidia Docker containers.
Here are some of the things we’ve noticed.
This does not happen when running on MacOS or Ubuntu. It only happens when inside a Docker container
This can be reproduced when on an EC2 running Docker
The screenshot tools in Babylon 5.xx have it happen almost 100% of the time. If I copy over the code from Babylon 4.2, the issue is fixed 99% of the time
Our code pauses the render loop so it should just wait until the render is complete and then attempt to pull the buffer off the canvas.
The difference I see in 5.xx and earlier code is the inclusion of the observable for framebuffer completion. This seems to have the opposite effect of its intention, in that it seems to always fire in the middle of the frame buffer flush. We’ve tried lots of variations here, and the one that works best is the 4.2 code along with something like this:
return new Promise((resolve: (value: string) => void) => {
// unpause render loop (omitted)
scene.onAfterRenderObservable.addOnce(async () => {
engine.flushFramebuffer();
const screenshot = await ScreenshotTools.CreateScreenshotAsync( // This code is copied over from 4.2 to a local file
engine,
scene.activeCamera as any,
{
...screenshotSize,
},
mimeType,
);
// pause render loop again (omitted)
resolve(screenshot);
});
scene.render();
});
Unfortunately, we can’t reproduce this in a playground, so maybe someone can help us understand how to correctly do the following:
Have you tried to wait for the next frame, and even the next 2 frames after the frame you actually want the screenshot to be taken from? I think once the gfx commands have been sent to the GPU (at the end of a frame) it can take one or two frames before the visible canvas is actually updated with those commands. Internally (in the browser), there are some buffering on the swap chain that may explain why you don’t see the updates you made in frame X in the screenshot you take at frame X: try to take the screenshot at frame X+1 or X+2.
To be sure everything is ready to render correctly your scene at frame X, you should use scene.executeWhenReady to know that all ressources are ready to be used for rendering.
I’ve tried all of that but I’ll give it another go for sanity’s sake. When you say wait for the next frame, you mean the browser’s animation frame, right?
So something like this should work, yeah?
function step() {
setTimeout(() => {
// this code is one animation frame and one event loop later.
// pull canvas pixels here...
}, 0)
}
// frame end observable fires =>>
myReq = requestAnimationFrame(step);
cancelAnimationFrame(myReq);
ETA: Oh, and everything is loaded. We’re hitting a server with color updates so it’s just updating uniforms, triggering a render, and returning the image. The results are that some meshes are just missing (as the link above showed)
That doesn’t fire in our setup. Our flow is that we communicate to the browser through puppeteer (page.evaluate) and wait for the updates to colors and textures to complete. When those are done, we use puppeteer again (page.evaluate) to grab the screenshot. So at the point of the code above the scene is already “ready” so there’s no event triggered. At this point, we want to trigger one render and do the screenshot. What we see locally is 100% success. On Docker with a smoking GPU (where we think it’s just much, much faster) we get a render where the framebuffer is just not totally flushed.
scene.executeWhenReady will always call the callback you pass in, even if the scene is already ready at the time you call it: if it’s not called there’s a problem somewhere else.
You can try to add a flushFramebuffer call before taking the screenshot.
If it’s still does not work, you may try to use the camera.outputRenderTarget property that will render the scene into the RTT you provide. Something like:
I implemented the render target texture approach and although I did not see the frame buffer issue, the overall render times increased almost 4x. It also doesn’t apply anti-aliasing which I was able to fix but that came with more performance loss.
I tried to simulate our basic setup using a Playground and discovered a way to demo the issue. For performance, we disable the render loop. What we’re trying to do here is:
Pause the scene after models and textures are fully loaded.
When an update comes in, apply new textures (if applicable), swap colors, and event that the render is ready to screenshot.
Render the scene ONCE
Grab data from the canvas using toDataUrl
The part we’re having trouble with is knowing when the scene is completely done without running the render loop.
The scene is normally ready when the callback passed to scene.executeWhenReady is called.
However, if you don’t setup a render loop, you need to call engine.endFrame() yourself as CreateScreenshot relies on the onEndFrameObservable observable which is notified by Engine.endFrame:
the overall render times increased almost 4x
I don’t really understand why, as the rendering is simply redirected to the render target instead of the canvas…
Indeed, you need to set rtt.samples to something like 4 to enable anti-aliasing on RTTs, but again I can’t explain why you see a significant loss of performance whereas you should see at most a few millisecond(s) more per frame… Are you sure it is not a software renderer which kicks in when creating the screenshot?
Definitely. We have verified that Chromium is running with GPU enabled. I’m happy to chalk this up to user error, though. Your example with engine.endFrame() looks promising. Testing now…
