Speeding up readPixels

Hi,
I’m building a robotics simulator, and am using an RTT camera to simulate a color sensor. I noticed that readPixels is extremely slow, even with a very low resolution (…I’m using 8x8 pixels). As the pixel data is being used to control the simulated robot movement, there’s no option for me to keep the data in GPU; it must be brought into main memory.

From this MDN article (WebGL best practices - Web APIs | MDN) it would appear that readPixels is blocking the thread, and it suggests using “…GPU-GPU readPixels in conjunction with async data readback”. How can I do this in Babylon?

They give some code:

function clientWaitAsync(gl, sync, flags, interval_ms) {
  return new Promise((resolve, reject) => {
    function test() {
      const res = gl.clientWaitSync(sync, flags, 0);
      if (res == gl.WAIT_FAILED) {
        reject();
        return;
      }
      if (res == gl.TIMEOUT_EXPIRED) {
        setTimeout(test, interval_ms);
        return;
      }
      resolve();
    }
    test());
  });
}

async function getBufferSubDataAsync(
    gl, target, buffer, srcByteOffset, dstBuffer,
    /* optional */ dstOffset, /* optional */ length) {
  const sync = gl.fenceSync(gl.SYNC_GPU_COMMANDS_COMPLETE, 0);
  gl.flush();

  await clientWaitAsync(gl, sync, 0, 10);
  gl.deleteSync(sync);

  gl.bindBuffer(target, buffer);
  gl.getBufferSubData(target, srcByteOffset, dstBuffer, dstOffset, length);
  gl.bindBuffer(target, null);

  return dest;
}

async function readPixelsAsync(gl, x, y, w, h, format, type, dest) {
  const buf = gl.createBuffer();
  gl.bindBuffer(gl.PIXEL_PACK_BUFFER, buf);
  gl.bufferData(gl.PIXEL_PACK_BUFFER, dest.byteLength, gl.STREAM_READ);
  gl.readPixels(x, y, w, h, format, type, 0);
  gl.bindBuffer(gl.PIXEL_PACK_BUFFER, null);

  await getBufferSubDataAsync(gl, gl.PIXEL_PACK_BUFFER, buf, 0, dest);

  gl.deleteBuffer(buf);
  return dest;
}

I don’t know how/if it can be added to Babylon, let’s wait for @sebavan or @Deltakosh inputs.

I love the idea of introducing this in babylon !!!

but it is normally what we have in engine._readPixelsAsync

not sure why we are not using it at the moment though @Deltakosh ?

We should:) I added the code but did not have time to wire it up (we need to test on IE for that for instance to be sure)

2 Likes

Thanks everyone. Reading the source for engine._readPixelsAsync gave me enough of a starting point to roll an async reads for my RTT texture.

I’m entirely unfamiliar with WebGL (…or any 3D rendering API), so while I’ve seen the sample code on MDN, it wasn’t much help. Even now, I still don’t quite understand what does “bindBuffer” do…

Strange… In _clientWaitAsync…

_clientWaitAsync = function(sync, flags = 0, interval_ms = 10) {
  return new Promise((resolve, reject) => {
    let check = () => {
      const res = gl.clientWaitSync(sync, flags, 0);
      if (res == gl.WAIT_FAILED) {
        console.log(gl.getError());
        reject();
        return;
      }
      if (res == gl.TIMEOUT_EXPIRED) {
        setTimeout(check, interval_ms);
        return;
      }
      resolve();
    };
    check();
  });
}

gl.clientWaitSync is returning WAIT_FAILED on Firefox, but works fine on Chrome.

gl.getError() is returning 0 (no error), and everything seems to be working fine if I just ignore the error and change reject() into resolve(). What could have caused the WAIT_FAILED?

gl is actually a state machine, so everytime if you want to modify etc for a buffer or texture etc, you need to first tell GL which is you target.

so bindBuffer is to tell GL about the target to be modified.
and then usually there will be some operation commands followed right after it.

WAIT_FAILED, no idea, google it. :slight_smile: