# Read pixels from raw camera texture

**URL:** <https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358>\
**Category:** Questions\
**Tags:** camera\
**Created:** [June 15, 2024, 8:29pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358 "2024-06-15T20:29:52Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![qw8z5fUB](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/qw8z5fub/32/40380_2.png) [@qw8z5fUB](https://forum.babylonjs.com/u/qw8z5fUB)\
**Post date:** [June 15, 2024, 8:29pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/1 "2024-06-15T20:29:52Z")

</div>

Hello, I’m trying to use webxr on a phone to simultaneously get phone pose and the camera image. In a later step I’d like to run some computer vision functions on the image.

I’m trying to combine these two examples:

- raw camera access: [https://playground.babylonjs.com/#UV8FAB#2](https://playground.babylonjs.com/#UV8FAB#2)
- save a texture to .png: [https://playground.babylonjs.com/#EFFI0E#3](https://playground.babylonjs.com/#EFFI0E#3)

This is the playground I’m using:

> **[Babylon.js Playground](https://playground.babylonjs.com/#UV8FAB%2315)**
>
> Babylon.js playground is a live editor for Babylon.js WebGL 3D scenes

At the moment it can save the first image correctly, but all subsequent images come out black.

Additionally I’m concerned about the efficiency of read-pixels: [Speeding up readPixels - #8 by Cort](https://forum.babylonjs.com/t/speeding-up-readpixels/12739/8)

Could anyone give any pointers as to whether I’m on the right track? I’m not currently clear if this will be achievable in babylon.js or not.

As an alternative I’ve found this example doing something similar in webxr (no babylon.js):  
[https://storage.googleapis.com/chromium-webxr-test/r1255390/proposals/camera-access-marker.html](https://storage.googleapis.com/chromium-webxr-test/r1255390/proposals/camera-access-marker.html)

That works well for me, but I’m not sure how to use it alongside babylon.js.

---

<div class="post-metadata">

**Author:** ![Cedric](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/cedric/32/2742_2.png) [@Cedric](https://forum.babylonjs.com/u/Cedric)\
**Post date:** [June 17, 2024, 7:31am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/2 "2024-06-17T07:31:28Z")

</div>

cc @RaananW

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [June 17, 2024, 9:59am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/3 "2024-06-17T09:59:42Z")

</div>

> [@qw8z5fUB](#):
>
> As an alternative I’ve found this example doing something similar in webxr (no babylon.js):  
> [https://storage.googleapis.com/chromium-webxr-test/r1255390/proposals/camera-access-marker.html](https://storage.googleapis.com/chromium-webxr-test/r1255390/proposals/camera-access-marker.html)

WebXR already has marker tracking, so there is no need to re-implement it. It is available on all mobile browsers that support raw camera access. Babylon has the feature as well.  
Having said that - this demo is nothing more than a wrapper for readPixel calls (sync or async), that is passed to OpenCV.js. you can achieve the same with babylon.

> [@qw8z5fUB](#):
>
> At the moment it can save the first image correctly, but all subsequent images come out black.

I’ll have to look into that - this should work every time and not only on the first image.

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [June 17, 2024, 9:59am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/4 "2024-06-17T09:59:47Z")

</div>

Assigned @RaananW on June 17, 2024, 9:59am UTC



---

<div class="post-metadata">

**Author:** ![qw8z5fUB](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/qw8z5fub/32/40380_2.png) [@qw8z5fUB](https://forum.babylonjs.com/u/qw8z5fUB)\
**Post date:** [June 19, 2024, 7:22am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/5 "2024-06-19T07:22:16Z")

</div>

Thank you, appreciate the input.

Yes, I’m not interested in the ability to do marker tracking per se, just the ability to capture the raw image and run some computer vision algorithms on it. The webxr demo is the only one I found that can read the camera texture at 30 Hz. As mentioned I’d like to achieve the same within babylon.js, but I’m unsure how to achieve that.

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [June 19, 2024, 12:27pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/6 "2024-06-19T12:27:42Z")

</div>

This playground works as expected - [https://playground.babylonjs.com/#UV8FAB#27](https://playground.babylonjs.com/#UV8FAB#27) (the ground is taking the canvas as a texture, just for visual approval of the readPixels procedure).

There is one issue on my end that needs to be solved, but this playground will continue working after I fix it (the texture is defined as cube, but it is not a cube texture).

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [June 19, 2024, 1:36pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/7 "2024-06-19T13:36:02Z")

</div>

Unassigned @RaananW on June 19, 2024, 1:36pm UTC



---

<div class="post-metadata">

**Author:** ![byanraff](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/byanraff/32/43160_2.png) [@byanraff](https://forum.babylonjs.com/u/byanraff)\
**Post date:** [October 17, 2024, 12:33pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/8 "2024-10-17T12:33:23Z")

</div>

Hi @RaananW, I am trying to implement the same configuration and looking for some guidance. The [https://playground.babylonjs.com/#UV8FAB#27](https://playground.babylonjs.com/#UV8FAB#27) playground produce an error for me and I can’t seem to extract the image from the texture data. Here is the error code: “glReadPixels: format and type incompatible with the current read framebuffer”

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 21, 2024, 3:46pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/9 "2024-10-21T15:46:23Z")

</div>

What device are you running this on?

---

<div class="post-metadata">

**Author:** ![byanraff](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/byanraff/32/43160_2.png) [@byanraff](https://forum.babylonjs.com/u/byanraff)\
**Post date:** [October 22, 2024, 1:21pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/10 "2024-10-22T13:21:04Z")

</div>

Samsung Galaxy A52S and Magic Leap 2

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 22, 2024, 5:20pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/11 "2024-10-22T17:20:53Z")

</div>

Assigned @RaananW on October 22, 2024, 5:20pm UTC



---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 22, 2024, 5:26pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/12 "2024-10-22T17:26:05Z")

</div>

Seems like the browser(s) have changed the format without notifying. This is an issue stated in the specs:

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/9/b/9b1c270e5089022676455a0e31e727484882a232.png)

Sadly not defined correctly. I will play around with it, find out what happened. Thanks for reporting!

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 22, 2024, 5:40pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/13 "2024-10-22T17:40:53Z")

</div>

Also a question, what exactly are you trying to achieve?

This PG is working as expected - [https://playground.babylonjs.com/#UV8FAB#43](https://playground.babylonjs.com/#UV8FAB#43).  
Read texture also works as expected.

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 22, 2024, 6:36pm UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/14 "2024-10-22T18:36:57Z")

</div>

Unassigned @RaananW on October 22, 2024, 6:36pm UTC



---

<div class="post-metadata">

**Author:** ![byanraff](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/byanraff/32/43160_2.png) [@byanraff](https://forum.babylonjs.com/u/byanraff)\
**Post date:** [October 23, 2024, 8:16am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/15 "2024-10-23T08:16:06Z")

</div>

Thanks! But it seems the same issue still persist after I tried to uncomment the readPixels() part. I am trying to get the image/pixels data from the texture and use it for a computer vision model.

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 23, 2024, 9:04am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/16 "2024-10-23T09:04:23Z")

</div>

on a samsung s23 the readpixel works and return pixels, but parsing it to a context2d doesn’t seem to work. however, if you don’t require this part and only need the actual texture it should be fine to use as well

---

<div class="post-metadata">

**Author:** ![byanraff](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/byanraff/32/43160_2.png) [@byanraff](https://forum.babylonjs.com/u/byanraff)\
**Post date:** [October 23, 2024, 11:31am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/17 "2024-10-23T11:31:35Z")

</div>

Yeah, I wanted to put the pixels into a canvas element to feed into a CV model. I aim to deploy the model on the webpage as well, do you think it’s possible?

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 28, 2024, 10:17am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/18 "2024-10-28T10:17:42Z")

</div>

sure, should be possible, but I assume not in real time (or not in 30 FPS). But that’s just an assumption, it really depends what you want to do.  
you get the pixel data from the readTexture call, read it using your CV library and process it as you wish. You will be at least 1 frame behind, but I assume this kind of delay is acceptable.

---

<div class="post-metadata">

**Author:** ![Strik3agle](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/strik3agle/32/43470_2.png) [@Strik3agle](https://forum.babylonjs.com/u/Strik3agle)\
**Post date:** [October 29, 2024, 11:28am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/19 "2024-10-29T11:28:35Z")

</div>

Hi, I’ve managed to extract the texture by adding

```auto
texture.getInternalTexture().type = 0

```

> **[Babylon.js Playground](https://playground.babylonjs.com/#UV8FAB%2347)**
>
> Babylon.js playground is a live editor for Babylon.js WebGL and WebGPU 3D scenes

The problem is that [raw camera module describe texture as float](https://github.com/BabylonJS/Babylon.js/blob/master/packages/dev/core/src/XR/features/WebXRRawCameraAccess.ts#L171) but the actual data from webxr is unsigned int resulting in an array of zeros when readPixels() is called.

I’m happy to start a PR if my assessment is correct.

---

<div class="post-metadata">

**Author:** ![RaananW](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/raananw/32/28955_2.png) [@RaananW](https://forum.babylonjs.com/u/RaananW)\
**Post date:** [October 29, 2024, 11:31am UTC](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358/20 "2024-10-29T11:31:33Z")

</div>

i’d be happy to merge it

[Next page](https://forum.babylonjs.com/t/read-pixels-from-raw-camera-texture/51358.md?page=2)
