Quest to create a Hologram with a JPEG! (how to? using Thin Instances, Matrix, Arrays)

let pxlMat:any = new BABYLON.StandardMaterial("pxlMat", scene)

Doesn’t work either?

nope …
(mine is named “_scene” btw)

Oh. I pulled it outside of the function! Seemed to clear it in Console.Log
L: 271 moved to L:236 ! I think i fixed it !?!!?!

My pixel cubes are still black though…

Could you create a Github repo for your project or put it into PG?
Otherwise we can only guess what is happening actually.

yes! - let me get the link …

Okay. So I originally made my repo Private with a giant folder tree in there… *lessons learned! *
Mind if I paste here instead?

import * as BABYLON from 'babylonjs';
import { BaseTexture, PBRMaterial } from 'babylonjs';

export default class MyScene {
    private _canvas: HTMLCanvasElement;
    private _engine: BABYLON.Engine;
    private _scene: BABYLON.Scene;
    // private _camera: BABYLON.FreeCamera;
    private _camera: BABYLON.UniversalCamera;
    private _cameraF: BABYLON.FollowCamera;
    private _light: BABYLON.Light;

    constructor(canvasElement: string) {
        // Create canvas and engine.
        this._canvas = document.getElementById(canvasElement) as HTMLCanvasElement;
        this._engine = new BABYLON.Engine(this._canvas, true);
    }


    createScene(): void {
        // Create a basic BJS Scene object.
        this._scene = new BABYLON.Scene(this._engine);


        // EDITING CAMERA
        // Create a FreeCamera, and set its position to (x:0, y:5, z:-10).
        // this._camera = new BABYLON.FreeCamera('camera1', new BABYLON.Vector3(0, .2, -2), this._scene);

        // // Target the camera to scene origin.
        // this._camera.setTarget(BABYLON.Vector3.Zero());

        // // Attach the camera to the canvas.
        // this._camera.attachControl(this._canvas, false);

        // CAMERA TYPES 
        // UNIVERSAL CAMERA follow cursor aim
        this._camera = new BABYLON.UniversalCamera("UniCam", new BABYLON.Vector3(0, 0, -2), this._scene);
        // Target Cursor or object
        // this._camera.setTarget(BABYLON.Vector3.Zero()); //initial target
        this._camera.minZ = 0.01;
        this._camera.maxZ = 50.0;
        this._camera.speed = .1;
        this._camera.inertia = 0.8;
        // Enable mouse wheel inputs.
        this._camera.inputs.addMouseWheel();
        // this._camera.

        // Attach the camera to the canvas
        this._camera.attachControl(this._canvas, true);
        this._camera.keysUpward.push(87);
        this._camera.keysDownward.push(83);
        this._camera.keysLeft.push(65);
        this._camera.keysRight.push(68);



        // // FOLLOW CAMERA follow target //bounces around - so weird...
        // // Parameters: name, position, scene
        // this._cameraF = new BABYLON.FollowCamera("FollowCam", new BABYLON.Vector3(0, 0, -2), this._scene);
        // // The goal distance of camera from target
        // // this._cameraF.radius = 1;
        // this._cameraF.lowerRadiusLimit = -1;
        // this._cameraF.upperRadiusLimit = -2;
        // this._cameraF.radius = .01

        // // The goal height of camera above local origin (centre) of target
        // this._cameraF.heightOffset = 0;
        // // The goal rotation of camera around local origin (centre) of target in x y plane
        // this._cameraF.rotationOffset = 0;
        // // Acceleration of camera in moving from current to goal position
        // this._cameraF.cameraAcceleration = 0.001;
        // // The speed at which acceleration is halted
        // this._cameraF.maxCameraSpeed = 1;
        // // This attaches the camera to the canvas
        // this._cameraF.attachControl(true);
        // // NOTE:: SET CAMERA TARGET AFTER THE TARGET'S CREATION AND NOTE CHANGE FROM BABYLONJS V 2.5
        // // targetMesh created here.
        // // camera.target = targetMesh; // version 2.4 and earlier
        // // camera.lockedTarget = targetMesh; //version 2.5 onwards




        // // Pointer lock follows cursor aim
        // let isLocked = false;
        // this._scene.onPointerDown = evt => {
        //     if (!isLocked) {
        //         this._canvas.requestPointerLock = this._canvas.requestPointerLock || this._canvas.msRequestPointerLock || this._canvas.mozRequestPointerLock || this._canvas.webkitRequestPointerLock;
        //         if (this._canvas.requestPointerLock) {
        //             this._canvas.requestPointerLock();
        //             return;
        //         }
        //     }
        // }
        // const pointerlockchange = () => {
        //     // @ts-ignore
        //     const controlEnabled = document.mozPointerLockElement || document.webkitPointerLockElement || document.msPointerLockElement || document.pointerLockElement || null;
        //     if (!controlEnabled) {
        //         isLocked = false;
        //     } else {
        //         isLocked = true;
        //     }
        // };
        // document.addEventListener('pointerlockchange', pointerlockchange, false);
        // document.addEventListener('mspointerlockchange', pointerlockchange, false);
        // document.addEventListener('mozpointerlockchange', pointerlockchange, false);
        // document.addEventListener('webkitpointerlockchange', pointerlockchange, false);
        // // Pointer lock follows cursor aim


        // // Pick Target Ray //BREAK SOMETHING FOR FREE VIEWING 
        // this._scene.onPointerMove = function castRay() {
        //     let ray = this._scene.createPickingRay(this._scene.pointerX, this._scene.pointerY, BABYLON.Matrix.Identity(), this._camera);

        //     let hit = this._scene.pickWithRay(ray);

        //     if (hit.pickedMesh && hit.pickedMesh.metadata == "pixel") {
        //         // createGUIButton();
        //     }

        //     statusLamp.material = this.LightLamp();
        //     var gl = new BABYLON.GlowLayer("glow", this._scene);
        // }


        // LIGHTS
        // Create a basic light, aiming 0,1,0 - meaning, to the sky.
        // this._light = new BABYLON.HemisphericLight('light1', new BABYLON.Vector3(0, 1, 0), this._scene);

        // THIN INSTANCES VARS
        let imgW = 1920
        let imgH = 1920
        let imgTotPixels = imgW * imgH
        let pxlspacer = 1.5;
        let pxlsize = 0.010;
        let screenW = 1.920, screenH = 1.920, screenD = pxlsize;
        let screenWtot = pxlspacer * screenW;
        let screenHtot = pxlspacer * screenH;
        let screenDtot = pxlspacer * screenD;

        // let numPerX = 1920, numPerY = 1920, numPerZ = 1;
        let numPerX = screenW / pxlsize;
        let numPerY = screenH / pxlsize;
        let imgResFactor = (imgW * imgH) / (numPerX * numPerY); // orginal image W, divided by pixels per side == indexes to skip in Color Array mapping
        // let imgResFactorY = imgH / numPerY;
        let numPerZ = screenD / pxlsize; //flat or single layer
        let ofstX = screenWtot / (numPerX - 1);
        let ofstY = screenHtot / (numPerY - 1);
        // let ofstZ = screenDtot / (numPerZ - 1);

        // SCREEN
        const screen = BABYLON.MeshBuilder.CreatePlane('screen1', {
            width: screenWtot, height: screenHtot, updatable: true
        }, this._scene);
        screen.position = new BABYLON.Vector3(0, 0, 0);


        // IMAGE
        var decalMaterial = new BABYLON.StandardMaterial("decalMat", this._scene);
        decalMaterial.emissiveTexture = new BABYLON.Texture("src/npb_example.png", this._scene);
        decalMaterial.emissiveTexture.hasAlpha = false;
        decalMaterial.zOffset = -2;
        var decalSize = new BABYLON.Vector3(screenWtot, screenHtot, 0);


        // DYNAMIC TEXTURE
        var dynTextureImage = new BABYLON.DynamicTexture("dynTexture", { width: imgW, height: imgH }, this._scene, false);
        var dynTextureContext = dynTextureImage.getContext();

        var img = new Image();
        img.src = "src/npb_example.png";


        // let dynColorData = new Float32Array(4 * imgTotPixels )
        // let dynColorData: any = new Uint8ClampedArray(4 * imgTotPixels ) 
        let dynColorData: any = new Float32Array(4 * imgTotPixels)

        img.onload = function () {
            //Add image to dynamic texture
            dynTextureContext.drawImage(img, 0, 0);
            dynTextureContext.getImageData(0, 0, imgW, imgH);
            console.log("Log #1: ", dynTextureContext.getImageData(0, 0, imgW, imgH))


            dynColorData = dynTextureContext.getImageData(0, 0, imgW, imgH);
            console.log("Log #2: ", dynColorData);

            dynTextureImage.update();
            // dynColorData.update();
            
            makePxls();

        }





        // READ PIXELS
        // let imgPixels = BaseTexture.readPixels
        // decalMaterial.readPixels
        // function getPixel(url, x, y) {
        //     var img = new Image();
        //     img.src = url;
        //     // var canvas = document.createElement('canvas');
        //     // var context = canvas.getContext('2d');
        //     this.drawImage(img, 0, 0);
        //     return this.getImageData(x, y, 1, 1).data;
        // }

        // let imgColorData = getPixel("src/npb_example.png", numPerX, numPerY); // [255, 255, 255, 0];
        // // getPixel('./bg.png', 10, 10); // [255, 255, 255, 0];        
        // console.log(imgColorData)

        /*CREATE DECAL*/
        var decal = BABYLON.MeshBuilder.CreateDecal("decal", screen, { position: new BABYLON.Vector3(0, 0, 0), normal: new BABYLON.Vector3(0, 0, -1), size: decalSize });
        decal.material = decalMaterial;


        // CUBE PIXELS
        var pixels = BABYLON.MeshBuilder.CreateBox("pixel", { size: pxlsize }, this._scene);
        // pixels.position = new BABYLON.Vector3(0, 0, -pxlsize);
        // pixels.material = new BABYLON.

        // PIXEL GRID (THIN INSTANCES)

        // MAPPING POSITION & COLOR
        //David's method
        var m: any = BABYLON.Matrix.Identity();
        var col = 0, index = 0;

        let PXLinstanceCount = numPerX * numPerY * numPerZ;

        let pxlMatrixData = new Float32Array(16 * PXLinstanceCount);
        let pxlColorData = new Float32Array(4 * PXLinstanceCount); //number relates to the objects that will reside in array
        let pxlMat: any = new BABYLON.StandardMaterial("material", this._scene); 

        function makePxls() {
            console.log("Log async: ", dynColorData);
            for (var x = 0; x < numPerX; x++) {
                m.m[12] = -screenWtot / 2 + ofstX * x;
                for (var y = 0; y < numPerY; y++) {
                    m.m[13] = -screenHtot / 2 + ofstY * y;
                    for (var z = 0; z < numPerZ; z++) {
                        m.m[14] = -pxlsize;
                        // m.m[14] = -screenDtot / 2 + ofstZ * z;

                        m.copyToArray(pxlMatrixData, index * 16);

                        pxlColorData[index * imgResFactor + 0] = dynColorData[index * imgResFactor + 0];
                        pxlColorData[index * imgResFactor + 1] = dynColorData[index * imgResFactor + 1];
                        pxlColorData[index * imgResFactor + 2] = dynColorData[index * imgResFactor + 2];
                        pxlColorData[index * imgResFactor + 3] = 1.0;

                        // var coli = Math.floor(col);
                        // pxlColorData[index * 4 + 0] = ((coli & 0xff0000) >> 16) / 255;
                        // pxlColorData[index * 4 + 1] = ((coli & 0x00ff00) >> 8) / 255;
                        // pxlColorData[index * 4 + 2] = ((coli & 0x0000ff) >> 8) / 255;
                        // pxlColorData[index * 4 + 3] = 1.0;

                        index++;
                        col += 0xffffff / PXLinstanceCount;
                    }
                }
            }

            pixels.thinInstanceSetBuffer("matrix", pxlMatrixData, 16, false);
            pixels.thinInstanceSetBuffer("color", pxlColorData, 4, false);
            
            // let pxlMat: any = new BABYLON.StandardMaterial("material", this._scene); //nope
            // let pxlMat: any = new BABYLON.StandardMaterial("material", this._scene); //nope
            pxlMat.disableLighting = true;
            pxlMat.emissiveColor = BABYLON.Color3.White();
            
            pixels.material = pxlMat;
        }      
        

        //Jason's method
        // let pixelArray = [];

        // for (let x = -1 * (numPerX / 2); x < numPerX / 2; x++) {
        //     for (let y = -1 * (numPerY / 2); y < numPerY / 2; y++) {
        //         let matrix = BABYLON.Matrix.Translation(x * ofst + (pxlsize / 2), y * ofst + (pxlsize / 2), -ofstZ * 1);
        //         pixelArray.push(matrix);
        //     }
        // }
        // pixels.thinInstanceAdd(pixelArray, true);



        // //// David
        // var box = BABYLON.MeshBuilder.CreateBox("box", { size: pxlsize }, this._scene);

        // var numPerSide = 192, numPerSideZ = 3, size = screenHtot;
        // // let ofst = size / (numPerSide - 1);
        // let ofst = screenDtot / (numPerZ - 1);

        // var m: any = BABYLON.Matrix.Identity();
        // var col = 0, index = 0;

        // // let instanceCount = numPerSide * numPerSide * numPerSideZ;

        // let matricesData = new Float32Array(16 * PXLinstanceCount);
        // let colorData = new Float32Array(4 * PXLinstanceCount);

        // for (var x = 0; x < numPerX; x++) {
        //     m.m[12] = -screenWtot / 2 + ofst * x;
        //     for (var y = 0; y < numPerY; y++) {
        //         m.m[13] = -screenHtot / 2 + ofst * y;
        //         for (var z = 0; z < numPerZ; z++) {
        //             m.m[14] = -screenDtot / 2 + ofst * z;

        //             m.copyToArray(matricesData, index * 16);

        //             var coli = Math.floor(col);

        //             colorData[index * 4 + 0] = ((coli & 0xff0000) >> 16) / 255;
        //             colorData[index * 4 + 1] = ((coli & 0x00ff00) >> 8) / 255;
        //             colorData[index * 4 + 2] = ((coli & 0x0000ff) >> 8) / 255;
        //             colorData[index * 4 + 3] = 1.0;

        //             index++;
        //             col += 0xffffff / PXLinstanceCount;
        //         }
        //     }
        // }

        // box.thinInstanceSetBuffer("matrix", matricesData, 16, true);
        // box.thinInstanceSetBuffer("color", colorData, 4, true);

        // const boxMat: any = new BABYLON.StandardMaterial("material", this._scene);
        // boxMat.disableLighting = true;
        // boxMat.emissiveColor = BABYLON.Color3.White();

        // box.material = boxMat;

        // //// David





        //PIXEL COLOR_SUPPORT
        // const pixels.material = new BABYLON.ma readPixels

        // /*****************SET TARGET FOR CAMERA************************/
        // this._cameraF.lockedTarget = statusLamp;
        // /**************************************************************/


        // // Over/Out
        // var makeOverOut = function (mesh) {
        //     // mesh.actionManager.registerAction(new BABYLON.SetValueAction(BABYLON.ActionManager.OnPointerOutTrigger, mesh.material, "emissiveColor", mesh.material.emissiveColor));
        //     // mesh.actionManager.registerAction(new BABYLON.SetValueAction(BABYLON.ActionManager.OnPointerOverTrigger, mesh.material, "emissiveColor", BABYLON.Color3.White()));
        //     // mesh.actionManager.registerAction(new BABYLON.InterpolateValueAction(BABYLON.ActionManager.OnPointerOutTrigger, mesh, "scaling", new BABYLON.Vector3(1, 1, 1), 150));
        //     mesh.actionManager.registerAction(new BABYLON.InterpolateValueAction(BABYLON.ActionManager.OnPointerOverTrigger, mesh, "scaling", new BABYLON.Vector3(1.1, 1.1, 3.1), 150));
        // }

        // makeOverOut(statusLamp);
        // // makeOverOut(pixels);


        // // scene's actions
        // this._scene.actionManager = new BABYLON.ActionManager(this._scene);


    } //CREATE SCENE END

    doRender(): void {
        // Run the render loop.
        this._engine.runRenderLoop(() => {
            this._scene.render();
        });

        // The canvas/window resize event handler.
        window.addEventListener('resize', () => {
            this._engine.resize();
        });
    }

    // materials
    LightLamp(): PBRMaterial {
        const pbrRed = new PBRMaterial("pbrR", this._scene);
        pbrRed.emissiveColor = new BABYLON.Color3(255, 0, 0)
        return pbrRed;
    }
}

Here we are - https://playground.babylonjs.com/#XCYFIB#1
Let’s do all things there (it is not completely working but much cleaner already) :slight_smile:

2 Likes

Yup! Okay haha THX *i went down a rabbit hole to figure out how to make my repo folder public…

okay… so switched out the brokn link for testing purposes

Okay… so between your version in PG and my new version #7, I’ve managed to break the pixel grid somehow in PG. Yet in my Vscode that part still works…

No sure why - I compared both Playgrounds and the difference I can see relate only to moving the “pxlMat” constand to L121, and adding “imgURL” at L57.

:persevere:
I’m going back to your PG and rebuilding from there…

I went to your last saved PG#1 and I seem to be breaking the code that builds the thins instances, when I add a variable into the line 71 that declares img.src. My “imgURL” variable is correctly formatted right??

My code in Vscode starting to result in some lit pixels :open_mouth::muscle:t5:
but doesn’t work in Playground :man_facepalming:t5:

OKAY. Found a solution. Added this to it:
img.crossOrigin = "anonymous";

Solved it in PG, and I added it to my VSC file for good measure. :roll_eyes:

Next challenge: Why are the colors not mapping correctly from the array?
The quest continues!

Hey @roland I just noticed you’re quite the guru with Thin Instances haha - I was wondering if I could bounce off you, regarding the mapping of color to each instance?

I’ve been following David’s Tute/code (https://playground.babylonjs.com/#PYL7JG#1), and nutting through everything to arrive at the following PG, where we’ve managed to capture Pixel RGB values into an array. I’m just having trouble mapping it…
https://playground.babylonjs.com/#XCYFIB#13

Haha Unfortunately, no time to celebrate when my progress has been so slow Necips. Trying to absorb as much as possible through different resources and nut out the rest of it… I have a list of things I’m aiming to build neatly for my little project :frowning:

By chance do you have much bearing on how color is allocated in this part of the code to each Instance? I’m not quite understanding it I feel…

EDIT I think I’ve found my problem …

At the end you will be able to write a book “How to master Babylon.js in one week” :slight_smile:

3 Likes

Haha until I am able write such book, I remain humbly attentive and with gratitude everyone’s help here so far haha :pray:t5: :smiley:

1 Like

As an update, I’m up to mapping the colors. I’ve got it down to this point >>> https://playground.babylonjs.com/#XCYFIB#14

For some reason, the array seems to returning only 0’s and 1’s in the console.log, but the previous console logs are receiving the correct arrays… Any ideas?

edit: I wonder if the array or matrix “c.” is in the wrong format or has an incorrect setting… ?

Does this have sometihng to do with it? I don’t know why the Matrices have 12, 13 and 14 in the index … If i changed them to 1,2,3 it breaks…
image

Some solutions and clarity found in this thread:

The quest continues!!
PG update: https://playground.babylonjs.com/#XCYFIB#16

1 Like