Hello, I’m trying to use webxr on a phone to simultaneously get phone pose and the camera image. In a later step I’d like to run some computer vision functions on the image.
WebXR already has marker tracking, so there is no need to re-implement it. It is available on all mobile browsers that support raw camera access. Babylon has the feature as well.
Having said that - this demo is nothing more than a wrapper for readPixel calls (sync or async), that is passed to OpenCV.js. you can achieve the same with babylon.
I’ll have to look into that - this should work every time and not only on the first image.
Yes, I’m not interested in the ability to do marker tracking per se, just the ability to capture the raw image and run some computer vision algorithms on it. The webxr demo is the only one I found that can read the camera texture at 30 Hz. As mentioned I’d like to achieve the same within babylon.js, but I’m unsure how to achieve that.
This playground works as expected - https://playground.babylonjs.com/#UV8FAB#27 (the ground is taking the canvas as a texture, just for visual approval of the readPixels procedure).
There is one issue on my end that needs to be solved, but this playground will continue working after I fix it (the texture is defined as cube, but it is not a cube texture).
Hi @RaananW, I am trying to implement the same configuration and looking for some guidance. The https://playground.babylonjs.com/#UV8FAB#27 playground produce an error for me and I can’t seem to extract the image from the texture data. Here is the error code: “glReadPixels: format and type incompatible with the current read framebuffer”
Thanks! But it seems the same issue still persist after I tried to uncomment the readPixels() part. I am trying to get the image/pixels data from the texture and use it for a computer vision model.
on a samsung s23 the readpixel works and return pixels, but parsing it to a context2d doesn’t seem to work. however, if you don’t require this part and only need the actual texture it should be fine to use as well
Yeah, I wanted to put the pixels into a canvas element to feed into a CV model. I aim to deploy the model on the webpage as well, do you think it’s possible?
sure, should be possible, but I assume not in real time (or not in 30 FPS). But that’s just an assumption, it really depends what you want to do.
you get the pixel data from the readTexture call, read it using your CV library and process it as you wish. You will be at least 1 frame behind, but I assume this kind of delay is acceptable.
