# Quest to create a Hologram with a JPEG! (how to? using Thin Instances, Matrix, Arrays)

**URL:** <https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988>\
**Category:** Questions\
**Tags:** documentation, material\
**Created:** [January 21, 2022, 2:43pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988 "2022-01-21T14:43:51Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 21, 2022, 2:43pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/1 "2022-01-21T14:43:51Z")

</div>

Is there a way of taking an image texture and mapping each pixel colour from the image to a “sea” array? of cubes, to look like a screen, and then offset the individual cubes in a sort of ripple or refresh type of movement pattern perpendicular to the screen – like a hologram?

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 21, 2022, 3:06pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/2 "2022-01-21T15:06:39Z")

</div>

Where there is a will there is a way 🙂

> [@HirangaG](#):
>
> image texture and mapping each pixel colour

You may use `texture.readPixels` - [Babylon.js docs](https://doc.babylonjs.com/typedoc/classes/babylon.texture#readpixels)

Thin instances ripple - [https://playground.babylonjs.com/#TWQZAU#1](https://playground.babylonjs.com/#TWQZAU#1)  
See also video about it  
[https://youtu.be/0DaSvI\_n7Us](https://youtu.be/0DaSvI_n7Us)

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 21, 2022, 3:58pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/3 "2022-01-21T15:58:26Z")

</div>

Omg. this sounds promising! I am excited for my tomorrow to try these methods out. Thanks for the lead!

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 23, 2022, 3:39am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/4 "2022-01-23T03:39:53Z")

</div>

Hey Labris. I’ve paused on my other challenges and come back to trying to figure out texture.readPixels.

Conceptually I think I know what to do:  
collect all the RGB values from the JPEG in particular order into a matrix?  
Then map the RGB colour as a material? to each thin instance object, via the matrix?

… I’ve just watched David’s tute: [Faster Scenes, Smaller Graphs with Thin Instances - YouTube](https://youtu.be/a8VY5qzyiSg?t=355)  
and Jason’s tute: [Fun with Legos Part 1: Thin Instances - YouTube](https://www.youtube.com/watch?v=0DaSvI_n7Us)

I successfully mapped the Tinstances with Jason’s code, but in trying to implement color, David’s seems to have the logic in his Line 29 …BUT…I’m confused between the difference in syntax, and how to implement…  
I’ve just copied David’s code and currently stuck with this error in VSCODE:

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/b/3/b30dcdd5bfd9083dbdc5c66ad68314b0ef41a262.png)

Any idea what I’m doing wrong? I’ve made a mockup in PG

> **[Babylon.js Playground](https://playground.babylonjs.com/#Q5AJPU)**
>
> Babylon.js playground is a live editor for Babylon.js WebGL 3D scenes

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 23, 2022, 4:22am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/5 "2022-01-23T04:22:43Z")

</div>

On mobile now so just couple of remarks:

Start with 8\*8 texture to simplify all things.

> [@HirangaG](#):
>
> Then map the RGB colour as a material? to each thin instance object, via the matrix?

Thin instance can have only the same material as mesh from which it was created.  
The color of material is defined by `diffuseColor` (`albedoColor` for PBR material) or/and `emissiveColor`.  
The error which you’ve got is related to the attempt to change unchangeable propeties. Thin instances should be used when you need a lot of static instances that you know won’t change often / at all. So, regular instances may still be the way to go, depending on your scene: if you must add/remove instances continuously, it may be better to use instances than thin instances.

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 23, 2022, 4:49am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/6 "2022-01-23T04:49:18Z")

</div>

Thx Labris… So with my intent, I want the TInstances to animate/move in the Depth Axis, similar to Jason’s waves (next task); and I’d like to colour the TInstances like in David’s PG example, where (from what I understand) he maps a value to R, G, B and A, according to the matrix … so I’m guessing it might be possible to map Matrix of TInstances with a Matrix of Color Values extracted using texture.readPixels ??? That viable with Thin Instances right?

But in order to test this, my error has me stumped… I think i get what you’re saying - have I used a Constant or something somewhere, erroneously?? I copied this chunk of code from Davids example and renamed the objects…

I’ve got both David’s and Jason’s code in the PG, commenting one out to test/understand the problem…

Jason, any ideas? @PirateJC

> **[Babylon.js Playground](https://playground.babylonjs.com/#Q5AJPU)**
>
> Babylon.js playground is a live editor for Babylon.js WebGL 3D scenes

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 23, 2022, 7:51am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/7 "2022-01-23T07:51:41Z")

</div>

I’ve just realised the error I’m having (snippy above) by copying Davids code may have to do with me trying to code in TS, instead of JS … 😑 Is this the problem causing the Matrix to only permit reading error???

Do i need completely different syntax here?  
[Babylon.js Playground](https://playground.babylonjs.com/#5V4209) \<\<\< this code only works in the JS Playground. Whereas I am building in VSC in Typescript …

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 23, 2022, 8:45am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/8 "2022-01-23T08:45:41Z")

</div>

Here is the working TS example - [https://playground.babylonjs.com/#HFN6RP#1](https://playground.babylonjs.com/#HFN6RP#1)

If you use TS be ready to add “any” as often as needed to make it work 🙂  
(of course, one may use more strict definitions)

`var m:any = BABYLON.Matrix.Identity();`

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 23, 2022, 1:53pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/9 "2022-01-23T13:53:44Z")

</div>

Oh man. amazing… “m:any” fixed it!  
I had another bug in one of my variables, resulting in a Null value 😐 Finally located that by checking through the original side by side. God it took so long to notice that null resulting equation… It has taken me 16hrs haha Slow progress… but that part is now working! woohoo!

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/c/a/ca702d2cc2070de7b60e9dbc88d531a6e86f6ce6.jpeg)

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 23, 2022, 1:59pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/10 "2022-01-23T13:59:35Z")

</div>

Labris, any tips on how I might use the readPixels function?  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/d/5/d5b42b70571f5ccac599d96cb43d19703efb6173.png)

I need to get better at understanding the Documentation (right now I don’t really get it…). I tried the following to; attempting to read pixels from the image loaded at L-150. I’m obviously doing it wrong. ???

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/2/e/2e51a8a9e33fb71866c17dfb0c37b9d78f4a879c.png)

And once I get those RBGA values do I need to format them into an array? or another matrix?  
I need to pass the results matrix? per index into this location (highlighted) I presume? 👇🏾 ??  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/8/7/870191b9f3652e30121cb7fa74d5d1a37b4dafa7.png)

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 23, 2022, 2:44pm UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/11 "2022-01-23T14:44:28Z")

</div>

Actually now I think you don’t need to readPixels function since you use just plain image.  
Put this image onto 2D canvas (for example, as dynamic texture).  
Then use something like (this is just quick example from the web, so you’'ll need to change some things according to your setup)

```auto
function getPixel(url, x, y) {
  var img = new Image();
  img.src = url;
  var canvas = document.createElement('canvas');
  var context = canvas.getContext('2d');
  context.drawImage(img, 0, 0);
  return context.getImageData(x, y, 1, 1).data;
}

getPixel('./bg.png', 10, 10); // [255, 255, 255, 0];

```

and cycle it across all your image coords to get a value for each image pixel.

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 24, 2022, 12:10am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/13 "2022-01-24T00:10:42Z")

</div>

Oh I think i get it! It seems I’m only getting an array with 4 values though when i console logged…

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/9/e/9e10f1be4763e578b0ccb276d32f2d43a1764b18.png)

Here’s what I did:  
Variables:  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/7/7/7753b3978eeee20c1a493794d2c3fb98ce91c159.png)

Image/Pixel functions:

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/0/d/0d803bc4bfe5044ea04ac5298c7b27928f1e32eb.png)

Mapping it to the Thin Instance:

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/f/9/f9f57ee66dc8d19c6e795ff0691b217cea3ae9d1.png)

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 24, 2022, 12:31am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/14 "2022-01-24T00:31:23Z")

</div>

As I mentioned, my example was just copy-paste from somewhere. You have already canvas element with Babylon scene. So you dont’ need to use in your function  
`var canvas = document.createElement('canvas');`  
Instead of this use dynamic texture for your 2d canvas where you put an image - [Dynamic Textures | Babylon.js Documentation](https://doc.babylonjs.com/divingDeeper/materials/using/dynamicTexture)  
Then get context  
`var ctx = myDynamicTexture.getContext();`  
and you are almost there 🙂  
Here are more links:  
[How to get Pixel from HTML Canvas ? - GeeksforGeeks](https://www.geeksforgeeks.org/how-to-get-pixel-from-html-canvas/#:~:text=To%20get%20the%20pixel%20of,object%20on%20the%20HTML%20canvas).  
[Pixel manipulation with canvas - Web APIs | MDN](https://developer.mozilla.org/en-US/docs/Web/API/Canvas_API/Tutorial/Pixel_manipulation_with_canvas)

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 24, 2022, 12:49am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/16 "2022-01-24T00:49:39Z")

</div>

Here is the example - [https://playground.babylonjs.com/#5ZCGRM#2222](https://playground.babylonjs.com/#5ZCGRM#2222) and the desired Uint8ClampedArray filled with values

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/b/5/b5461d4477f7361d309c733aea6db4825f6025bd.jpeg)

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 24, 2022, 4:33am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/17 "2022-01-24T04:33:41Z")

</div>

Thx… I’m piecing this together … and reading a whole lot of stuff to fix errors…

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 24, 2022, 5:30am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/18 "2022-01-24T05:30:00Z")

</div>

O . K . I finally got there…

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/0/1/0169b7a2fca801417870df52817e4420d1fd3198.png)

Array seems good…  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/d/7/d75d788b618b2524c296a936a887ce1537b4bf19.png)

Now i need to map them into the previous thing…

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 24, 2022, 5:51am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/19 "2022-01-24T05:51:54Z")

</div>

This is really visible progress which may lead to the final success 🙂

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 24, 2022, 6:02am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/20 "2022-01-24T06:02:45Z")

</div>

hahaha 👍🏾 \*needless to say i feel so stupid haha. Update: that “:any” thing worked once again _sort of_

 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/4/c/4c2b816b88d4e189cb20f786cd42f7dd7abd1cdb.png)  
My array returns correctly in ConsLog #1;  
and returns only 0’s in ConsLog #2.

I’ve noticed the original is a Float32Array:  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/3/4/346ad1310583f6feadad832481ae1169e1be2ac5.png)

How do i get the ConsLog #1 data out of there in the correct format?  
Should “: any” at L:173 be set to something else to declare an Array?  
 ![image](https://us1.discourse-cdn.com/flex024/uploads/babylonjs/original/3X/b/5/b5dda0afed3675ac85f9e1547c1e5b77c4cecb7a.png)

---

<div class="post-metadata">

**Author:** ![labris](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/labris/32/12576_2.png) [@labris](https://forum.babylonjs.com/u/labris)\
**Post date:** [January 24, 2022, 6:27am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/21 "2022-01-24T06:27:25Z")

</div>

As far as I can see your ConsLog #2 is outside a function which will be performed only after the image is loaded. That is why you have 0’s there. You need a callback or promise here, for example.  
As for the line 173 you may try `myInstance:MyArrayType[] = [];`  
See also [TypeScript Array Type](https://www.typescripttutorial.net/typescript-tutorial/typescript-array-type/)

---

<div class="post-metadata">

**Author:** ![HirangaG](https://sea2.discourse-cdn.com/flex024/user_avatar/forum.babylonjs.com/hirangag/32/14565_2.png) [@HirangaG](https://forum.babylonjs.com/u/HirangaG)\
**Post date:** [January 24, 2022, 6:32am UTC](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988/22 "2022-01-24T06:32:09Z")

</div>

Ah… okay…so that works! I just had to put the log back into the function…

So now, I need to make sure the Thin Instance creation occurs only after this Array has been created…

[Next page](https://forum.babylonjs.com/t/quest-to-create-a-hologram-with-a-jpeg-how-to-using-thin-instances-matrix-arrays/26988.md?page=2)