Speeding up readPixels

Questions
1

Hi,
I’m building a robotics simulator, and am using an RTT camera to simulate a color sensor. I noticed that readPixels is extremely slow, even with a very low resolution (…I’m using 8x8 pixels). As the pixel data is being used to control the simulated robot movement, there’s no option for me to keep the data in GPU; it must be brought into main memory.

From this MDN article (WebGL best practices - Web APIs | MDN) it would appear that readPixels is blocking the thread, and it suggests using “…GPU-GPU readPixels in conjunction with async data readback”. How can I do this in Babylon?

2

They give some code:

function clientWaitAsync(gl, sync, flags, interval_ms) {
  return new Promise((resolve, reject) => {
    function test() {
      const res = gl.clientWaitSync(sync, flags, 0);
      if (res == gl.WAIT_FAILED) {
        reject();
        return;
      }
      if (res == gl.TIMEOUT_EXPIRED) {
        setTimeout(test, interval_ms);
        return;
      }
      resolve();
    }
    test());
  });
}

async function getBufferSubDataAsync(
    gl, target, buffer, srcByteOffset, dstBuffer,
    /* optional */ dstOffset, /* optional */ length) {
  const sync = gl.fenceSync(gl.SYNC_GPU_COMMANDS_COMPLETE, 0);
  gl.flush();

  await clientWaitAsync(gl, sync, 0, 10);
  gl.deleteSync(sync);

  gl.bindBuffer(target, buffer);
  gl.getBufferSubData(target, srcByteOffset, dstBuffer, dstOffset, length);
  gl.bindBuffer(target, null);

  return dest;
}

async function readPixelsAsync(gl, x, y, w, h, format, type, dest) {
  const buf = gl.createBuffer();
  gl.bindBuffer(gl.PIXEL_PACK_BUFFER, buf);
  gl.bufferData(gl.PIXEL_PACK_BUFFER, dest.byteLength, gl.STREAM_READ);
  gl.readPixels(x, y, w, h, format, type, 0);
  gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);

  await getBufferSubDataAsync(gl, gl.PIXEL_PACK_BUFFER, buf, 0, dest);

  gl.deleteBuffer(buf);
  return dest;
}

I don’t know how/if it can be added to Babylon, let’s wait for @sebavan or @Deltakosh inputs.

3

I love the idea of introducing this in babylon !!!

4

but it is normally what we have in engine._readPixelsAsync

5

not sure why we are not using it at the moment though @Deltakosh ?

6

We should:) I added the code but did not have time to wire it up (we need to test on IE for that for instance to be sure)

2 Likes
7

Thanks everyone. Reading the source for engine._readPixelsAsync gave me enough of a starting point to roll an async reads for my RTT texture.

I’m entirely unfamiliar with WebGL (…or any 3D rendering API), so while I’ve seen the sample code on MDN, it wasn’t much help. Even now, I still don’t quite understand what does “bindBuffer” do…

8

Strange… In _clientWaitAsync…

_clientWaitAsync = function(sync, flags = 0, interval_ms = 10) {
  return new Promise((resolve, reject) => {
    let check = () => {
      const res = gl.clientWaitSync(sync, flags, 0);
      if (res == gl.WAIT_FAILED) {
        console.log(gl.getError());
        reject();
        return;
      }
      if (res == gl.TIMEOUT_EXPIRED) {
        setTimeout(check, interval_ms);
        return;
      }
      resolve();
    };
    check();
  });
}

gl.clientWaitSync is returning WAIT_FAILED on Firefox, but works fine on Chrome.

gl.getError() is returning 0 (no error), and everything seems to be working fine if I just ignore the error and change reject() into resolve(). What could have caused the WAIT_FAILED?

9

gl is actually a state machine, so everytime if you want to modify etc for a buffer or texture etc, you need to first tell GL which is you target.

so bindBuffer is to tell GL about the target to be modified.
and then usually there will be some operation commands followed right after it.

WAIT_FAILED, no idea, google it. :slight_smile:

10

I was about to open a feature request on this, but +1 on providing an async implementation of readPixels for WebGL implementations that can support it.

I currently pay a heavy startup cost because I need to render some textures on the GPU and am forced to wait for the blocking roundtrip to get the pixel data back (often over a hundred millis).

@sebavan FYI

11

engine._readPixelsAsync should work :slight_smile: let me know if you have issues with it and if not let s promote it ???

12

Yep, specifically BaseTexture.readPixels does work but it’s not truly async. The API is async but WebGL implementation of ThinEngine is synchronous. (Babylon.js/engine.readTexture.ts at d25bc29091d47f51bd2f0f98fb0f16d25517675f · BabylonJS/Babylon.js · GitHub)

It eventually calls readPixels directly which means blocking on GPU. It would be great if the async approach was implemented on WebGL devices that support it.

13

Yup totally agree, this could be the fallback only. do you want to give it a try ?

1 Like