Running Babylon.js on a nodejs server with headless-gl

I’m trying to get Babylon.js running on a nodejs server with headless-gl and was wondering if anyone has had any success doing something like this. What I’m ultimately trying to do is load a .gltf file, then run the render loop for a specified time while capturing the frames from the GL context. Since I’m running on a nodejs server, I don’t have access to the web browser APIs so I’m using the following to simulate a browser like environment:

  • jsdom for the DOM implementation
  • headless-gl for the WebGL implementation
  • node-canvas for Canvas support in jsdom

I have been bumping into a series of problems during the creation of the babylon.js engine and the load of the resources in the scene. I am hacking and slashing to get around some of these problems and have made some progress, but I’m wondering if anyone has success doing something similar. Are there settings that I can set on Babylon.js that remove the need for a browser like context (it doesn’t look like this is the case based on some of the Babylon.js code I have seen)? Are there other node modules that would work better for simulating the browser like environment?

Any help or feedback is appreciated.

1 Like

Hi Brentonator! Sorry for no replies. How is your progress on this? I wish you would have gotten some replies. (sniff) Maybe after this topic-bump… we will get more info.

We succeed in rendering server side with GitHub - GoogleChrome/puppeteer: Headless Chrome Node API

1 Like

Maybe have a look at the NullEngine() (Server Side - Babylon.js Documentation). Would that cover your use case? I have been successfully running a headless babylonjs on nodejs with it.

NullEngine Seems can’t output the image of the render result but it is needed for many cases on the server side.

But that is the whole point of the NullEngine. Running BabylonJS without any WebGL and rendering, thus of course also no image.
What is your usecase here? It sounds very much like the NullEngine is not the right tool here.

Sorry for so late replying, I just want to take some screenshots of the 3D model on the server-side.

HI there @vimcaw:

I’m trying to get some server side renders from BJS scenes, using NodeJS and Puppeteer, and facing the problem of how to know when the scene is completely created and shown, in order to call the Puppeteers’ screenshot API.

Have you got any tips on it?

1 Like

Hi there @sharp:

Any tips about how to know when the BJS’ scene to screenshot via Puppeteers is completely created and shown?

Thanks for your time.

Hi @paleRider you can trigger a custom-event puppeteer/custom-event.js at main · puppeteer/puppeteer · GitHub in your scene.executeWhenReady for example or see https://babylonjs.medium.com/understanding-asynchronous-code-in-babylon-js-cb534634beb for more details

1 Like

Thanks for your advice @sharp.

The case is that I can’t have it working, I’m already tried and checked several ways, but It refuses to work.

In my (static) HTML I have a JS object (Singleton) with the BJS functionality that renders the scene in a canvas fulfilling the whole page. The render is shown OK, and I fire the custom “sceneReady” event from scene.executeWhenReady as you state:

(Excerpt from com.vortice3d.bjs.js, called from render.html)

//2
_scene = new BABYLON.Scene(_engine);
_scene.clearColor = BABYLON.Color4.FromHexString('#00000000');
//
_scene.executeWhenReady(function () {
    _engine.runRenderLoop(function () {
        _scene.render();
     });
     //
     window.dispatchEvent(new Event("sceneReady"));
     console.log(_TAG+"sceneReady Dispatched!");
});

On the other side, my NodeJS module, using Puppeteer, is the classic for taking screenshots and for implementing custom-event listening:

var puppeteer = require('puppeteer');
var _TAG = "com.vortice3d.screenshot\t";
var _DEFAULT_VIEWPORT = {
    width: 1920,
    height: 1080,
    deviceScaleFactor: 1,
  };

async function takeScreenshot(params) {
    //1
    var browser = await puppeteer.launch({
        defaultViewport: _DEFAULT_VIEWPORT,
        headless: false
    });

    //2
    var page = await browser.newPage();
    //enable "console.log" inside "page.evaluate"
    page.on('console', function (consoleObj) {
        console.log(_TAG+consoleObj.text());
    });

    await page.exposeFunction('onCustomEvent', function(evt){
        console.log(`${evt.type} fired`, evt.detail || '');
    });

    await page.evaluateOnNewDocument(function () {
        document.addEventListener("sceneReady", function (evt) {
            window.onCustomEvent({
                type: "sceneReady",
                detail: evt.detail
            });
        });
    });

    var result = await page.goto(params.url, {
        /*waitUntil: "domcontentloaded"*/
        /*waitUntil: 'networkidle0'*/
    });
    console.log(_TAG + "Result: " + result.status());
    
    await page.waitForTimeout(2000);
    var canvas = await page.$('canvas');
    await canvas.screenshot({
        omitBackground: true,
        path: params.path
    });

    //await page.close();
    await browser.close();
}

module.exports = takeScreenshot;

With this I can get my screenshots with the BJS scene completely rendered, but only thanks to the addition of the await page.waitForTimeout(2000), that is not the desired way, as the exposed custom event handler (onCustomEvent) is never called. Timeover delay is a “magic number” and so the correct way is to go with the custom event instead.

Any tip with this?

Thanks for your time.

P.S:
Of course, in the case onCustomEvent is fired, the screenshot functionality would have to be moved there.
“Dispatched!” effectively is shown in my console, proving dispathEvent is called.
Using document.dispatchEvent(new Event(“sceneReady”)); doesn’t make the trick.

Hi @paleRider, sorry I can’t study your code right now but if you want a simple and basic solution to make screenshot you can expose a function like this :

const puppeteer = require('puppeteer');

(async () => {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();

    await page.exposeFunction('screenshot', async (e) => {
        await page.screenshot({ path: 'example.png' });
        await browser.close();
    });

    await page.goto('http://localhost/');
})();

and in your HTML :

_scene.executeWhenReady(function () {
    _engine.runRenderLoop(function () {
        _scene.render();
     });
     screenshot();
});
1 Like

Twice thanks indeed, @sharp:

Reviewing your code I’ve realized where was my error with the customEvent.

The case is that it must be:

 await page.exposeFunction('onCustomEvent', async function(evt){
        console.log(`${evt.type} fired`, evt.detail || '');
    });

instead of:

 await page.exposeFunction('onCustomEvent', function(evt){
        console.log(`${evt.type} fired`, evt.detail || '');
    });

That is, the callback must be stated as async.

Best regards.

1 Like

@sharp @vimcaw @jodo @Brentonator
Have you also found a way to manipulate model (like changing textures and all) before taking a screenshot?
I keep failing to do so.
Where exactly do I have to put

//2
_scene = new BABYLON.Scene(_engine);
_scene.clearColor = BABYLON.Color4.FromHexString('#00000000');
//
_scene.executeWhenReady(function () {
    _engine.runRenderLoop(function () {
        _scene.render();
     });
     //
     window.dispatchEvent(new Event("sceneReady"));
     console.log(_TAG+"sceneReady Dispatched!");
});

these?

were we able to resolve this issue? how want to use puppeteer to print out of Babylon scene?

@RaananW was doing this within our automated tests, maybe he has some inputs?

This is how we do it in our docs:

Documentation/lib/buildUtils/tools.ts at master · BabylonJS/Documentation (github.com)

And in my template you can find a way to run vis-tests using playwright:
babylonjs-webpack-es6/tests/validation.spec.ts at master · RaananW/babylonjs-webpack-es6 (github.com)

Eventually, playwright/puppeteer are just browser orchestrators. Anything you can run in a browser you can run there, and then ask for a screenshot.

1 Like

thank you @RaananW, It has been helpful.
when I hit the API which generates my pdf (from html), I am able to generate the pdf.
but once it is deployed, I am getting blank pdf

This is very hard to answer, without seeing exactly what you are trying to do. If it is blank, it is possible you are not waiting for the site to fully render.

if you are using puppeteer and deploying something like serverless - I think you need to make sure you have a GPU instance - I recall having these issues. I ended up building my own docker container from an node alpine image and then deploying to a GPU instance. That was close to 5 years ago, so the details have faded.

maybe if you can share more details on your solution - might point us in the right direction.

edit: I don’t have access to the original code, but I recall it had extra APKs related to graphics libraries. something like this though (that repo makes PDFs/images of website content):
crapdf/Dockerfile at master · brianzinn/crapdf (github.com)

edit2: before you spend too much time there I would try out Raanan’s suggestion - call scene.render and then on an after render observable - maybe add a DOM element and then in puppeteer wait for that?