Using RenderTargetTexture with EffectWrapper / EffectRenderer

Hello, I’m trying to find my way to doing a multipass shader on the entire canvas without totally straying from how I’ve done it with a ShaderMaterial.

For this I have multiple RenderTargetTexture, and before render I set uniforms and a texture input of the previous pass. This works well.

A start would be if I could just get a single EffectWrapper and EffectRenderer working. My understanding is that if I call render without a 2nd texture input, it should render the effect to the canvas. But I see nothing?

			this.effectRenderer.render( this.effectWrapper )

NB. effectWrapper is compiled, ready to go, I can set uniforms etc, no errors - just I see nothing…

You should use Spector to understand what’s going on. If you can’t find the problem, I think we will need a repro in the Playground to be able to help.

No there just seems to be zero way to render multiple passes on the entire canvas - only with materials. I’ve tried everything and the only way within the API seems to be to create multiple concurrent shaders, which is really inefficient.

NB. this is something very easily done in any other language / framework, or just in vanilla JS - but within BabylonJS world there is a convoluted system of pipeline which obfuscate this very basic concept: multiple passes

Here is the work in progress. It gets ridiculous in the difference between config objects between shader materials, effects, postprocesses. Sometimes an array, sometimes an object, sometimes mixing up uniforms into “samplers” and “parameters”, sometimes an automatically-loaded “fx”, and sometimes not. Zero consistency in the API when it comes to shaders…


class ShaderBase extends Base {
	
	id = 'shaderSystem'

	fragment = FragmentTemplate
	vertex = VertexTemplate

	onBeforeRender = null
	onCompiled = null
	onError = null

	targets = []

	schema = {
		ready: {
			default: false,
			type: 'boolean',
			disabled: true
		},
		passes: {
			default: 4,
			type: 'number',
			onUpdate: 'reset'
		},
		uniformSchema: {
			default: {},
			type: 'object',
			onUpdate: 'reset'
		},
		needsAlphaBlending: {
			default: false,
			type: 'boolean'
		},
		needsAlphaTesting: {
			default: false,
			type: 'boolean'
		},
		backFaceCulling: {
			default: false,
			type: 'boolean'
		},
	}

	update() {
		if (this.shaderMaterial) {
			this.shaderMaterial.needsAlphaBlending = this.needsAlphaBlending
			this.shaderMaterial.needsAlphaTesting = this.needsAlphaTesting
			this.shaderMaterial.backFaceCulling = this.backFaceCulling
		}
	}

	constructor( id, params ) {
		super( id, params )
		this.initialise(id,params)
	}

	createAppropriateShader( configuration ) {
		SAY('THIS IS BASE')
	}

	reset() {

		if (!this.scene) return SAY('🚨 NO SCENE')

		this.ready = false

		SAY('SETUP')

		if (this.shaderMaterial) {
			this.shaderMaterial.dispose()
			this.shaderMaterial = null
			this.targets.forEach( texture => texture.dispose() )
			this.targets = []
		}

		const options = {
			attributes: [],
			uniforms: []
		}

		const attributes = []
		const samplers = []
		const parameters = []

		let header = 'precision highp float;'

		for (const [ category,list] of Object.entries(definitions.headerConfig)) {

			header += `\n\n// ${category.toUpperCase()}S\n` // TITLE

			for (const [ name, { type, description } ] of Object.entries(list)) {

				if (category == 'uniform') parameters.push(name)
				if (category == 'attribute') attributes.push(name)

				header += `\n${category} ${type} ${name};`
			}
		}

		header += `\n\n// CUSTOM UNIFORMS\n` // TITLE

		for ( const [name, { type }] of Object.entries( this.uniformSchema ) ) {

			if (type == 'sampler2D') {
				samplers.push(name)
			} else {
				parameters.push(name)
			}
			header += `\nuniform ${type} ${name};`

		}

		const code = {
			vertexSource: this.vertex
				.replace('// $HEADER', header)
				.replace('// $FOOTER', definitions.vertexFooter),
			fragmentSource: this.fragment
				.replace('// $HEADER', header.replaceAll('attribute', 'varying'))
				.replace('// $FOOTER', definitions.fragmentFooter),
		}


		BABYLON.Effect.ShadersStore[`${this.id}VertexShader`] = code.vertexSource
		BABYLON.Effect.ShadersStore[`${this.id}FragmentShader`] = code.fragmentSource


		const shaderMaterialConfig = {
			name: this.id, 
			scene: this.scene.instance, 
			code,
			options: {
				attributes,
				uniforms: [...parameters, ...samplers],
				needAlphaBlending: this.needAlphaBlending,
				needAlphaTesting: this.needAlphaTesting
			}
		}

		const postProcessConfig = {
			name: this.id,
			url: this.id,
			parameters,
			samplers,
			size: 1,
			camera: this.scene.instance.activeCamera,
			samplingMode: 0,
			engine: this.scene.instance.getEngine(),
			reusable: true
		}

		const args = {
			disableRewriting: false,
			prettyPrint: false,
			keepSymbols: false,
			globals: false
		}

		for (const which of ['vertexSource', 'fragmentSource']) {

			const results = parser.check( code[which] )
			for (const message of results.log.diagnostics) {
				const split = code[which].split('\n')
				console.log(`%c🚨 [${which}] ${message.text}`, 'color:darkred')
				let { start, end } = message.range
				for (start; start > 0; start--) if (code[which][start] == '\n') break
				for (end; end < code[which].length; end++) if (code[which][end] == '\n') break
				const line = code[which].substring(start, end).trim()
				console.log( `%c${line}`, 'color:red' )
			}
		}


		// ------ CREATE PASSES ------

		for (let passIndex = 0; passIndex < this.passes; passIndex++ ) {


			const width = this.scene.instance.getEngine().getRenderWidth()
			const height = this.scene.instance.getEngine().getRenderHeight()

			const renderTarget = new BABYLON.RenderTargetTexture(
				`${this.id}_renderTarget${passIndex}`,
				{ width, height },
				this.scene.instance,
				{ isMulti: false }
			)

			renderTarget.onBeforeRenderObservable.add( e => {

				// this.setInternalUniforms( passIndex, this.shaderMaterial )

				SAY('BEFORE RENDER')

			})

			this.scene.instance.customRenderTargets.push(renderTarget)
			this.targets.push( renderTarget )
		}

		this.createAppropriateShader( shaderMaterialConfig, postProcessConfig )

	}

	shaderMaterial = null
	postProcess = null


	setInternalUniforms( passIndex, object ) {

		if (this.onBeforeRender) {
			this.onBeforeRender( passIndex, object )
		}

		const width = this.scene.instance.getEngine().getRenderWidth()
		const height = this.scene.instance.getEngine().getRenderHeight()
		const inputSampler = passIndex === 0 ? this.targets[this.passes-1] : this.targets[i-1]

		object.setTexture('inputSampler', inputSampler )
		object.setInt('passIndex', passIndex)
		object.setInt('totalPasses', this.passes)
		object.setVector2('canvasSize', new BABYLON.Vector2(width, height))
	}

}

export class ShaderEffect extends ShaderBase {

	createAppropriateShader( matConfig, effectConfig ) {

		const engine = this.scene.instance.getEngine()
		const canvas = this.scene.instance.getEngine().getRenderingCanvas()
		const width = this.scene.instance.getEngine().getRenderWidth()
		const height = this.scene.instance.getEngine().getRenderHeight()
		const camera = this.scene.instance.activeCamera

		const parameters = []
		const samplers = []

		this.postProcess = new BABYLON.PostProcess( ...Object.values(effectConfig) )

		let inited = false

		this.postProcess.onApply = effect => {

			inited = true

		}

        this.scene.instance.onBeforeRenderObservable.add(() => {

        	if (!inited) return
        	for (let passIndex = 0; passIndex < this.passes; passIndex++) {

			    const target = this.targets[passIndex]
			    const effect = this.postProcess.getEffect()

		        engine.bindFramebuffer(target.getInternalTexture())
			    this.setInternalUniforms( passIndex, effect )
			    this.postProcess.activate(camera)
			    this.postProcess.apply()
		        engine.unBindFramebuffer(target.getInternalTexture())
			}
        })

		camera.attachPostProcess(this.postProcess)


	}

}

export class ShaderMaterial extends ShaderBase {


	createAppropriateShader( configuration ) {

		this.shaderMaterial = new BABYLON.ShaderMaterial( ...Object.values( configuration ) )
		this.shaderMaterial.name = this.id

		this.shaderMaterial.onCompiled = e => {
			SAY('SHADER MATERIAL COMPILED')
			this.ready = true
			if (this.onCompiled) this.onCompiled()
		}
		this.shaderMaterial.onError = (e, errors, other) => {
			SAY('🚨 SHADER MATERIAL ERROR', e, errors, other)
			if (this.onError) this.onError(errors)
		}

		// ------ BIND EVENTS ------

		if (this.onCompiled) this.shaderMaterial.onCompiled( this.onCompiled.bind(this) )
		if (this.onError) this.shaderMaterial.onError( this.onError.bind(this) )

	}

	bindMesh( mesh ) {
		mesh.instance.material = this.shaderMaterial
		this.targets.forEach((texture,i) => {
			if (!texture.renderList.includes(mesh.instance)) {
				texture.renderList.push(mesh.instance)
			}
		})
	}

}

I agree about the lack of consistency but as we always try to preserve backward compatibility we tend to not remove any “old” ways.

About your case, the simplest today would be to chain post processes (one per pass) or create a render pipeline.

The second more manual approach is to rely on an EffectRender with a bunch of EffectWrapper and this would only not work if the setup is not correct in which case a repro would help us help you.

Finally, as this is a space we wanted to improve a lot, we are planning on having a fully streamlined way of creating render pipelines in 8.0: Bringing balance in the force: A story about simplicity and control | by Babylon.js | Oct, 2024 | Medium

Not sure why you are so aggressive when asking for help but you must understand this will not go super far here.

We are kindly providing help to anyone asking for it, but you should really watch your tone.

One easy way to apply multipass on the full canvas as @sebavan mentioned is to use postprocesses:
How To Use Post Processes | Babylon.js Documentation

Remember that Babylon is a 10+ years old engine and we are sticking with our goal of protecting backward compatibility as much as possible.
In 10 years the JS/TS ways of doing things evolved and we tried to stick with the trends as much as possible without compromising on our fundamentals.
At the same time, the Engine abstracts WebGL, WebGPU, Null and Native layers. So it has to come with some complexities.

I’m sad that you find that ridiculous but this is who we are.