Read pixels from raw camera texture

Hello, I’m trying to use webxr on a phone to simultaneously get phone pose and the camera image. In a later step I’d like to run some computer vision functions on the image.

I’m trying to combine these two examples:

This is the playground I’m using:

At the moment it can save the first image correctly, but all subsequent images come out black.

Additionally I’m concerned about the efficiency of read-pixels: Speeding up readPixels - #8 by Cort

Could anyone give any pointers as to whether I’m on the right track? I’m not currently clear if this will be achievable in babylon.js or not.

As an alternative I’ve found this example doing something similar in webxr (no babylon.js):
https://storage.googleapis.com/chromium-webxr-test/r1255390/proposals/camera-access-marker.html

That works well for me, but I’m not sure how to use it alongside babylon.js.

cc @RaananW

WebXR already has marker tracking, so there is no need to re-implement it. It is available on all mobile browsers that support raw camera access. Babylon has the feature as well.
Having said that - this demo is nothing more than a wrapper for readPixel calls (sync or async), that is passed to OpenCV.js. you can achieve the same with babylon.

I’ll have to look into that - this should work every time and not only on the first image.

Thank you, appreciate the input.

Yes, I’m not interested in the ability to do marker tracking per se, just the ability to capture the raw image and run some computer vision algorithms on it. The webxr demo is the only one I found that can read the camera texture at 30 Hz. As mentioned I’d like to achieve the same within babylon.js, but I’m unsure how to achieve that.

This playground works as expected - https://playground.babylonjs.com/#UV8FAB#27 (the ground is taking the canvas as a texture, just for visual approval of the readPixels procedure).

There is one issue on my end that needs to be solved, but this playground will continue working after I fix it (the texture is defined as cube, but it is not a cube texture).