[Babylon Native] get frame buffer data in c++ or expose requestScreenShot

Questions
1

Hello,
I’m trying out babylon native to be able to render a scene offscreen and send it through NDI.

I’m facing multiple problems, this first ones being :

  • Is is possible to have a scene render without showing it or passing the hwnd ?
  • How to retrieve the texture handle or pixel data that is rendered by the camera ?

I saw some methods like requestScreenShot and the NativeCapture class but they don’t seem documented and what I’ve found here and on github didn’t lead me anywhere interesting…

Thanks

2

Welcome to the community!

ccing the @BabylonNative gang

3

Here is an example:

call getFrameBufferData and provide a callback that will be fired when screen bits are available.

4

Thank you for the fast reply !
So what I understand is this is used to retrieve the framebuffer FROM Babylon JS side, not the C++ side.

My goal is to not show the render on the app but just generate the render and then use it to send through network (NDI). So i need to both have a windowless Babylon engine runnning, and retrieve the buffer from C++ to then send it through network. does that make sense ?

Thank you :slight_smile:

5

I don’t remember a c++ to do capture but it should be too hard to implement.
I believe you did tests some time ago @bghgary ?

6

So the method that is called from JS is this one, which points me to RequestScreenShot. I’m happy to see I was on the right path, but I can’t seem to access the graphics context from outside, so I can’t call RequestScreenShot.

 void NativeEngine::GetFrameBufferData(const Napi::CallbackInfo& info)
    {
        const auto callback{info[0].As<Napi::Function>()};

        auto callbackPtr{std::make_shared<Napi::FunctionReference>(Napi::Persistent(callback))};
        m_graphicsContext.RequestScreenShot([this, callbackPtr{std::move(callbackPtr)}](std::vector<uint8_t> array) {
            m_runtime.Dispatch([callbackPtr{std::move(callbackPtr)}, array{std::move(array)}](Napi::Env env) {
                auto arrayBuffer{Napi::ArrayBuffer::New(env, const_cast<uint8_t*>(array.data()), array.size())};
                auto typedArray{Napi::Uint8Array::New(env, array.size(), arrayBuffer, 0)};
                callbackPtr->Value().Call({typedArray});
            });
        });
    }
7

So maybe the thing to look for is a way to access a pointer to the NativeEngine instance that the Babylon side has created ? Or provide one from c++ to the interpreter environment ?

8

I think it’s possible the get the nativeEngine plugin instance from the engine instance. if, in your impl, it’s global, then you can get it.
or even simpler, you create the engine, then set a global variable as engine._engine
in your c++ code get that global var, unwrap it as a NativeEngine.

9

I tried and couldn’t find a way to do it clean, NativeEngine is not accessible from outside. And because of namespace magic, I couldn’t expose a global instance either.

If you have some sample code that would be great :slight_smile: Otherwise i’ll keep looking on how to modify the classes and propose some stuff

10

I would recommend rendering to a render target texture instead of trying to get the default back buffer. I wrote a blog about this late last year that should help.

Babylon Native in a Headless Environment | by Babylon.js | Medium

11

oh didn’t see the reply, thank you !

1 Like