My team is currently integrating OpenCV.js into an app and I am trying to avoid fetching an image twice (once for a Babylon texture and once for OpenCV.js) by re-using the texture pixels from Babylon.
One way to initialize a Mat() object in OpenCV.js is to provide it a buffer of pixel data. However, one also needs to provide the type of the data so OpenCV knows how to interpret it (i.e. it needs to be one of these types: opencv-ts/constants.d.ts at da6e203b7fdf26249ee0c494f93d407b0f393003 · peteruhnak/opencv-ts · GitHub)
I am now trying to user texture.readPixel() to get the pixel data and I was thinking of using textureType and textureFormat in order to figure out the type of data but I always get -1 for both of those.
Can someone confirm that the basic approach described above is valid and if so - why do I get -1 for textureType and textureFormat instead of one of the values defined here Babylon.js/constants.ts at master · BabylonJS/Babylon.js · GitHub?
Thank you!
It totally depends on how you load the texture, a repro in the playground could help narrowing the issue down ?
Sorry I wasn’t clear, the texture is loaded using the BABYLON.Texture constructor which receives an image URL.
Here is a playground demo: Babylon.js Playground
Pinging @sebavan just in case he missed your answer
In this case readPixels should be good by default: https://playground.babylonjs.com/#0YGIZR#1
@sebavan I am not sure I understand what you mean - my problem is NOT that readPixels() does NOT return anything, but that textureType and textureFormat return -1 so I am not sure how to interpret the data returned by readPixels().
For example, here is the doc for textureFormat:
"Get the texture underlying format (RGB, RGBA…)
If I get -1, how do I know what is the format?
Does this make sense or do I misunderstand the connection between readPixels(), textureType and textureFormat?
Oh sorry I missed your reply, so reading in RGBA should work fine for all png/jpg and so on you do not need to check the type, we actually only fill and use it for raw texture internally as this is the case you can not really know what would be in.
Thanks, @sebavan!
Just to make sure I understand - can I assume that any image I use, regardless of its type (e.g. png/jpg/gif/bmp/tiff) will be converted to RGBA as long as it is supported by Babylon?
Also, can I also assume that the format of the array will always be UINT8 (and NOT float or something else)?
yup rgba uint8 for all those from what I remember
Thanks, @sebavan.
However, something still doesn’t seem right or I am just misunderstanding. If I can just read the pixels from the texture as uint8 RGBA, I assume that I can read the pixels and put them in a canvas element, isn’t that right?
However, when I do this, the image is flipped upside-down, here is an example:
https://playground.babylonjs.com/#4HG4IC#2
Any idea why this happens?
Yup cause we flip texture by default in babylon to fit with our uv mapping. You could force not flipping it like this https://playground.babylonjs.com/#4HG4IC#4
The problem is that I am re-using an existing texture and I will not be allowed to change its creation (because this will probably trigger changes in many other parts of the code).
Is there an easy way to flip the pixels from readPixels (other than reverting them in 4-byte chunks manually)?
Wow that’s wild! Okay I feel like I’m missing something – why 2 canvases? It seems to work with one:
https://playground.babylonjs.com/#4HG4IC#8
Edit: whoops … no it does NOT work with one canvas. I still don’t understand what’s going on here 
I use 2 canvas for convenience: it is possible to copy one canvas into another one and invert on the Y axis easily in the process, by simply doing a translate + scale on the context. It’s easier than setting a loop and doing the inverting by hand 
1 Like
Ok, understood, thanks everyone.