Use of DynamicTexture from an HTML5 canvas or image

Hello!

I’m trying to composite user-defined images into a DynamicTexture for a 3D object.

I’d love to provide a Playground but I couldn’t get it to work (no access to head/body of HTML page, complaints that I hadn’t created a Scene when I had etc), so I’ll have to provide some example code instead. Sorry about that!

The Images class simply loads a user-defined set of images and blits them to an internal Canvas after loading:

class Images {
	constructor(lx, ly) {
		// Not added to document; off-screen!
		this.canvas = document.createElement('canvas');

		this.canvas.width = lx;
		this.canvas.height = ly;

		// Preview of the "merged" image we'd like to use on a 3D object
		this.merged = document.getElementById('imgMerged');
		
		// Internal store of images, number of loads outstanding 
		this.images = [];
		this.loading = 0;
	}

	// Load a set of image files; when all loaded, draw them onto the
	// internal canvas and call onAllLoaded()
	loadImages(files, onAllLoaded) {
		const N = files.length;

		this.images = [];
		this.loading = N;

		for (let i=0; i<N; i++) {
			let idx = this.images.length;
			let fn = (e) => {
				this.images[idx].src = e.target.result;
				this.loading--;
				if (this.loading<1) {
					this.drawImages();
					onAllLoaded();
				}
			};

			this.images.push(new Image()); // placeholder; overwritten in fn()

			let reader = new FileReader();
			reader.onload = fn;
			reader.readAsDataURL(files[i]);
		}
	}

	// Draw all images onto internal canvas, display in preview image
	drawImages() {
		let cvs = this.canvas;
		let ctx = cvs.getContext('2d');

		ctx.globalAlpha = 0.5;

		for (let i=0; i<this.images.length; i++ ) {
			const img = this.images[i];
			// This check seems to be triggered regularly, despite this
			// method only being called after onload() called for all image
			// loads? Weird, they should all be valid at this point.
			if (!img.complete || img.naturalWidth == 0) {
				console.log(`Problem: img=${i} complete=${img.complete} naturalWidth=${img.naturalWidth}`);
				return;
			}
			ctx.drawImage(img, 0,0);
		}

		// Update merged image preview using the canvas contents
		// Note; this works, but using the same canvas' toDataURL() method
		// fails to set the .src member of an Image in System.setTexture()
		// and produces only a black texture!
		this.merged.src = cvs.toDataURL('image/jpeg');
	}
}

There’s also a little wrapper for a Babylon scene:

class Babylon {
	constructor() {
		let PI = Math.PI;
		let V3 = (a,b,c) => { new BABYLON.Vector3(a,b,c); };

		this.canvas = document.getElementById('renderCanvas');
		this.engine = new BABYLON.Engine(this.canvas, true);
		this.scene = new BABYLON.Scene(this.engine);
		this.camera = new BABYLON.ArcRotateCamera('camera', PI/4, PI/3, 8, V3(0,0,0), this.scene);
		this.light = new BABYLON.HemisphericLight('light', V3(0,1,0));

		this.camera.attachControl(this.canvas, true);
	}

	startRenderLoop() {
		this.engine.runRenderLoop( () => { this.scene.render(); } );
	}
}

These two classes are wrapped by a System class that also creates a simple ground plane to texture and connects the file input to a handler that loads the image files and then tries to update the ground plane texture:

class System {
	constructor(lx,ly) {
		this.images = new Images(lx,ly);

		this.babylon = new Babylon();
		this.ground = BABYLON.MeshBuilder.CreateGround("ground1", {width: 20, height: 10, subdivisions: 25}, this.babylon.scene);
		this.babylon.startRenderLoop();

		let onChange = (e) => {
			this.images.loadImages(this.btnLoad.files, () => {
				this.setTexture();
			});
		}

		this.btnLoad = document.getElementById('btnLoad');
		this.btnLoad.addEventListener('change', onChange);
	}

	setTexture() {
		let mat = new BABYLON.StandardMaterial('dm', this.babylon.scene);
		let tex = new BABYLON.DynamicTexture('dt', this.images.canvas, this.babylon.scene);

		mat.diffuseTexture = tex;
		this.ground.material = mat;

		let img = new Image();
		img.onload = function() {
			tex.getContext().drawImage(img, 0,0);
			tex.update();
		}

		//
		// This is where it gets weird; setting img.src to ...
		//
		// 1. A hardwired path: WORKS
		// 2. Result of canvas.toDataURL('image/jpeg') DOES NOT WORK
		// 3. The .src member of the merged preview image: WORKS, even where
		//    that image had its .src member set to ... the results of the same
		//    Canvas.toDataURL() that doesn't work in (2)!
		//

//		img.src = 'whatever.jpeg'; // 1
		img.src = this.images.canvas.toDataURL('image/jpeg'); // 2
//		img.src = this.images.merged.src; // 3
	}
}

The System class is then created when everything else is ready to go:

window.onload = function(e) {
	let sys = new System(500,500);
}

All the code above is located in a file called test.js. The HTML document for testing is as follows:

<!DOCTYPE html>
<html>

<head>
	<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
	<script src="https://cdn.babylonjs.com/babylon.js"></script>
</head>

<body>
	<input id="btnLoad" type="file" value="Load" multiple="multiple" />
	<br />
	<img id="imgMerged" alt="merged image" />
	<br />
	<canvas id="renderCanvas"></canvas>

	<script type="module" src="test.js"></script>
</body>

</html>

I have two problems:

  1. There’s a race condition somewhere in the image loading: FileReader.onload() callbacks are being triggered, but the images are often not completely loaded (see test condition in Images.drawImages()). This is not a Babylon problem, but I’m struggling to track it down - if someone can tell me what dumb thing I’ve done that would be great!

  2. Attempting to draw an Image generated by Image.src = Canvas.toDataURL() onto the context of the DynamicTexture fails, giving only a black texture. However, if I set that same Image.src to the .src member of a different Image that was itself originally set using Canvas.toDataURL() it works (see System.setTexture())! I feel like this might be another race condition, but I’m not sure why, or how to fix it.

Can anyone see what I’m doing wrong?

Sorry about the rather long post; if anyone can tell me how to easily get this code into a Playground without any problems, I’d be very grateful!

Cheers!

As you said it sounds like a lot :frowning: and lots of things could go wrong about timing and such. I would advise to create a simpler playground let s say with only one image ? so we could remove issues one by one !!!

1 Like

Hi there!

If the problem is the actual image loading on the texture I would just isolate that section and try to get that part into the playground. If you get similar behavior that will indicate there is something wrong in that part of the code. :slight_smile:

Aha @sebavan 2 min faster than me :wink:

1 Like

Hello,

I’ve tried to create a simplified Playground version here:

https://playground.babylonjs.com/#V5TK5Z#20

It’s obviously not quite the same approach, as it can’t operate on user-specified files so I have to try to hardcode default texture paths instead. Hopefully it still captures the basic approach.

Thanks!

So you were setting back the source of the image, also you could use emissiveTexture to better see what happens and finally you should create the texture aside of the canvas to prevent blacking it out on creation: https://playground.babylonjs.com/#V5TK5Z#22 cause we reassign the size (a bug I am currently fixing)

1 Like

Thanks sebavan!

So you were setting back the source of the image

Can you please explain why this would be a problem?

The data has been loaded into memory, so I’m not quite understanding why re-assigning img.src to that data would be a problem (i.e., this.images[idx].src = e.target.result in the onload() callback)?

It’s inefficient, but I’m not sure why it would cause invalid image data when trying to use the image immediately afterwards, unless that operation is somehow deferred?

Thanks!

setting src will enforce a reload.

2 Likes