Multiple Persistent Shader Passes - how to?

Questions
, , ,
1

Hello, I’m struggling to find a way to do multipass shader - so that I can have multiple persistent samplers which I can play around with on a final pass (and do generative feedback loops). At the moment I have no idea how to configure this with RenderTargetTexture objects, and have tried a lot of things.

My shader is setup like this, which demonstrates the end result I’m aiming for:

#version 300 es
precision highp float;

in vec2 vUV;
in vec2 vNormal;

out vec4 outColor;

uniform mat4 worldViewProjection;
uniform int PASSINDEX;

uniform sampler2D pass0;
uniform sampler2D pass1;
uniform sampler2D pass2;
uniform sampler2D pass3;

void main() {

    vec2 xy = vec2(vUV);

    vec4 c0 = texture( pass0, vUV );
    vec4 c1 = texture( pass1, vUV );
    vec4 c2 = texture( pass2, vUV );
    vec4 c3 = texture( pass3, vUV );

    vec4 red = vec4(1.0,0.0,0.0,1.0);
    vec4 blue = vec4(0.0,0.0,1.0,1.0);
    vec4 green = vec4(0.0,1.0,0.0,1.0);
    vec4 white = vec4(1.0,1.0,1.0,1.0);

    if (PASSINDEX == 0) {
        outColor = white;
    } else if (PASSINDEX == 1) {
        outColor = mix( blue, c0, 0.0);
    } else if (PASSINDEX == 2) {
        outColor = mix( green, c1, 0.0);
    } else if (PASSINDEX == 3) {
        // outColor = mix( red, c2, 0.0 );
        // outColor = c0; // white?
        outColor = c1; // blue?
        // outColor = c2; // green?
        // outColor = c3; // red?
    }
}

In Babylon I’m attempting various things like this:


		let PASSINDEX = 0

		const passes = [
			new BABYLON.RenderTargetTexture('pass0', $_DIMENSIONS, scene),
			new BABYLON.RenderTargetTexture('pass1', $_DIMENSIONS, scene),
			new BABYLON.RenderTargetTexture('pass2', $_DIMENSIONS, scene),
			new BABYLON.RenderTargetTexture('pass3', $_DIMENSIONS, scene)
		]

		for ( let idx = 0; idx < passes.length; idx++ ) {
			object.material.setTexture( 'pass'+idx, passes[idx] )
		}

		engine.runRenderLoop( () => {

			checkIfCompiled( shaderMaterial )

			/* ------------- SHADER ------------- */

			object.material.setVector2('RENDERSIZE', new BABYLON.Vector2($_DIMENSIONS.width, $_DIMENSIONS.height))

			for (let i = 0; i < passes.length; i++) {
				PASSINDEX = i;
				object.material.setInt('PASSINDEX', i)
				passes[i].renderList = [ object ]
				scene.customRenderTargets = [ passes[i] ]
				scene.render()
			}	

		})

Though I have no idea how to do this properly. I’ve been able to get some strange visual feedback loops goingbut it’s not the result I was hoping for, and very laggy.

2

OK, I got it working! + ultra mega bloody chuffed that it runs smoothly on both WebGL and WebGPU (with lovely ES3 sampler2D array no less). Bigup to babylonjs once again. If anyone runs into the same troubles, this was my approach (perhaps not correct way - but it works + isn’t laggy):


		/* ------------- RENDER PASS BUFFERS ------------- */

		const DEBUG = false
		let PASSINDEX = 0
		let passes

		function assignTextures() {

			shaderMaterial.setTextureArray('passes', passes.map(pass=>pass.current))
		}

		function createTexture(idx) {

			const texture = new BABYLON.RenderTargetTexture('input' + idx, $_DIMENSIONS, scene, {
				isMulti: false
			})
			texture.renderList = [ object ]
			texture.onBeforeBindObservable.add( () => {
				if (DEBUG) console.log(idx, 'bind')
				for (let i = 0; i < passes.length; i++) {
					passes[i].current = i == PASSINDEX ? passes[i].output : passes[i].input
					shaderMaterial.setTexture( 'pass'+i, passes[i].current )
				}
				assignTextures()
				shaderMaterial.setInt('PASSINDEX', PASSINDEX)
				PASSINDEX += 1;
				if (PASSINDEX >= passes.length) PASSINDEX = 0
			})
			texture.onBeforeRenderObservable.add( () => {
				if (DEBUG) console.log(idx, 'render')
			})
			texture.onAfterUnbindObservable.add( () => {
				if (DEBUG) console.log(idx, 'unbind')
			})

			texture.setMaterialForRendering( object, shaderMaterial )

			return texture
		}

		function createPass(idx) {

			const input = createTexture(idx)
			const output = createTexture(idx)
			const current =  idx == 0 ? output : input
			return  {
				input,
				output,
				current
			}
		}

		passes = [
			createPass(0),
			createPass(1),
			createPass(2),
			createPass(3),
			createPass(4)
		]

		assignTextures()
		object.material = shaderMaterial

		scene.onBeforeRenderTargetsRenderObservable.add( () => {
			if (DEBUG) console.log('D')
		})
		scene.onBeforeRenderObservable.add( (o, a, b) => {
			if (DEBUG) console.log('B')
		})
		scene.registerBeforeRender( e => {
			if (DEBUG) console.log('C')
		})

		engine.runRenderLoop( () => {
			if (DEBUG) console.log('A')

			checkIfCompiled( shaderMaterial )

			/* ------------- SHADER ------------- */

			shaderMaterial.setVector2('RENDERSIZE', new BABYLON.Vector2($_DIMENSIONS.width, $_DIMENSIONS.height))
			scene.customRenderTargets = passes.map( o => o.input )
			scene.render()
		})

SHADER:


#version 300 es
precision highp float;

in vec2 vUV;
in vec2 vNormal;

out vec4 outColor;

uniform mat4 worldViewProjection;
uniform int PASSINDEX;

uniform sampler2D passes[4];

void main() {

    vec2 xy = vec2(vUV);

    vec4 c0 = texture( passes[0], vUV );
    vec4 c1 = texture( passes[1], vUV );
    vec4 c2 = texture( passes[2], vUV );
    vec4 c3 = texture( passes[3], vUV );

    vec4 red = vec4(1.0,0.0,0.0,1.0);
    vec4 blue = vec4(0.0,0.0,1.0,1.0);
    vec4 green = vec4(0.0,1.0,0.0,1.0);
    vec4 yellow = vec4(0.0,1.0,1.0,1.0);
    vec4 white = vec4(1.0,1.0,1.0,1.0);

    if (PASSINDEX == 0) {
        outColor = mix(red, c3, 0.5);
    } else if (PASSINDEX == 1) {
        outColor = mix( blue, c0, 0.0);
    } else if (PASSINDEX == 2) {
        outColor = mix( green, c1, 0.0);
    } else if (PASSINDEX == 3) {
        outColor = mix( white, c2, 0.0 );

        // outColor = c0; // white?
        outColor = c1; // blue?
        outColor = c2; // green?
        // outColor = c3; // red?
    } else {
        outColor = c3;
    }
    // outColor = vec4(1.0);
}
``
1 Like
3

If anyone can help - nextup is trying to work out how to render the passes into just rectangles until the final pass. Thinking so far:

  • the default normals and vectors from the vertex shader wont do this (correct?) - they are all clipped or uv mapped off the mesh object
  • so just add a 2d plane to the scene - switch to and show this mesh when doing the render passes and then hide it and switch to the main mesh object when doing the final pass

Seems a bit hacky so if there’s a better way please let me know

4

It’s not easy to understand exactly what you want to do without a repro. If you want to render only to a (rectangle) portion of the screen, you can use the scissor test for that:

2 Likes
5

That might be the one! Merci :blush:

Code is being refactored and I’m terribly privat until its ready for public consumption however I found the 2D matte normals which are positioned correctly - they are masked to the silhouette of the object - which makes sense since I can’t imagine a 3D renderer wants to render any more pixels than it has to do. Needs some more experimentation - but essentially canvas gets painted in 2D space and on the final pass wraps over the UVs - but quite important this happens in 2D space first so that the recursionis controllable … but then that also gets me thinking: if you’re going to all that effort of rendering an extra 3,4,5,6 times per visible frame you might aswell see what fun you can have in prepping the next recursive frame (ie. splaying the matte across 3D space in different ways)

6

This is not for a video game just something that will be good on drugs.