Feedback on frame graphs

[phaselock]

Repro: https://playground.babylonjs.com/#OQTYRG#1

There are 2 identical FrameGraphGeometryRendererTasks where the object list of one is empty and the other is set to all meshes in scene. I was expecting 2 different depth textures but it seems like the last task added to the frame graph overwrites the texture result of the previous, as shown in the Inspector. If I set the samples differently, the result is wai. You can swap the addTask around to see the difference.

wai?

[Evgeni_Popov]

@phaselock

There are 2 identical FrameGraphGeometryRendererTasks where the object list of one is empty and the other is set to all meshes in scene. I was expecting 2 different depth textures but it seems like the last task added to the frame graph overwrites the texture result of the previous, as shown in the Inspector. If I set the samples differently, the result is wai. You can swap the addTask around to see the difference.

This is expected, due to texture reuse optimization.

The texture created by the first geometry renderer is not used in the graph afterwards, so the optimizer assumes it can be reused if another task uses a texture with the same dimensions/type/format, which is the case for the 2nd geometry renderer.

There are two ways to make this work:

1. disable texture optimization: frameGraph.optimizeTextureAllocation = false;
2. create dependencies between tasks to extend the lifetime of textures and prevent the optimizer from reusing textures too early: laterTask.dependencies = new Set([textureHandleFromEarlierTask])

For e.g., using the “dependencies” method:

Babylon.js Playground

I added allDepthTask.dependencies = new Set([noDepthTask.geometryNormViewDepthTexture]); to ensure that the geometry depth texture from noDepthTask lives long enough that the optimizer cannot reuse it to generate the geometry depth texture of allDepthTask. You could also use copyToBackbufferTask.dependencies = new Set([noDepthTask.geometryNormViewDepthTexture, allDepthTask.geometryNormViewDepthTexture]);. This way, both textures remain active until the end of the graph, and you can freely change the order of noDepthTask and allDepthTask in the task list.

I will look into the other issues as soon as possible.

[Evgeni_Popov]

Regarding the “context lost” problem, it seems to be a chromium bug only (Chrome/Edge). I was able to repro it in a simple PG:

Babylon.js Playground

Note: you may have to increase the “900” values to make it fail.

This PG works in Firefox.

The PG doesn’t crash if createMSAATexture is set to false. Could you please retest your use case by setting samples = 1 everywhere so that MSAA textures are not created, and see if that works for you? This would be an additional proof that the problem lies in the deletion/recreation of MSAA textures.

[phaselock]

@Evgeni_Popov

Note: you may have to increase the “900” values to make it fail.

Yes, it seems your repro is showing the identical screen flash and context lost. Just loading the pg was able to trigger it.

I do not remember the exact setup used (crash froze my tab and pg lost), so I repro a simple one with lesser textures: Babylon.js Playground

I was able to get the same context crash just by running the pg and mousing over spheres aggressively w/o having to tab away. But also managed to crash it permanently with the tab away (cf below ss). Can’t crash the context with samples=1 so far even with a 15min tab away. I do not know if it is definitive but its a definite lead.

[phaselock]

@Evgeni_Popov

Thin instances are part of a mesh: if a mesh is in a frame graph object list, its thin instances will be displayed. It’s not specific to frame graphs, it’s the same thing if you add a mesh with thin instances to a RTT.renderList: all thin instances of the mesh will be displayed in the RTT.

I forgot about this. When you add thin instances to an RTT, you can leverage rtt.setMaterialForRendering(mesh,shaderMaterial); + thin instance buffers to render all other thin instances as transparent and the desired thin instance as non-transparent. Is there an equivalent process for frame graphs? ie, user pointer over a thin instance, fg texture renders only specific thin instance based on its index. or it could render all thin instances but only the selected thin instance as non-transparent, ditto for depth/normals etc…

[Evgeni_Popov]

@phaselock

I forgot about this. When you add thin instances to an RTT, you can leverage rtt.setMaterialForRendering(mesh,shaderMaterial); + thin instance buffers to render all other thin instances as transparent and the desired thin instance as non-transparent. Is there an equivalent process for frame graphs?

It’s no different. You can get the ObjectRenderer from the object renderer task, and call setMaterialForRendering on this object. Something like:

const renderTask = new BABYLON.FrameGraphObjectRendererTask("renderObjects", frameGraph, scene);

renderTask.objectRenderer.setMaterialForRendering(meshOrMeshes, materialOrMaterials);

[phaselock]

Thank you! Now I remember why I forgot about it, cos it was solved the same way in another code snippet!! Bloody hell, today is my brain fart day.

Regarding dependencies for identical FrameGraphTasks as above, the doc expressly states:

Note that this dependencies input is specific to the node render graph framework: frame graph tasks do not have this property!

I guess this needs an update?

[phaselock]

scene.dispose() currently does not dispose any associated frame graphs. Is there any particular reason why it should not automatically deal with it?

[phaselock]

repro: https://playground.babylonjs.com/#OQTYRG#5

I’m getting artifacts when using setMaterialForRendering for a mesh on one of the FrameGraphGeometryRendererTask. The textures in the inspector are similarly affected. I can shift frameGraph.addTask(renderTask); earlier to resolve screen visuals but the tex are still affected. Is this wai?

Seems like the framegraph doesn’t like the depth texture shared between 2 tasks?

[Evgeni_Popov]

Regarding the crash problem, I created a repro outside of Babylon and created an issue in the Chromium repository:

Chromium

@phaselock

Regarding dependencies for identical FrameGraphTasks as above, the doc expressly states:

Note that this dependencies input is specific to the node render graph framework: frame graph tasks do not have this property!

I guess this needs an update?

Indeed, here’s the update:

FrameGraph: Update dependencies input explanation in notes

@phaselock

scene.dispose() currently does not dispose any associated frame graphs. Is there any particular reason why it should not automatically deal with it?

It’s an oversight, it’s fixed in this PR:

FrameGraph: Disposes of frame graphs when the scene is disposed

@phaselock

I’m getting artifacts when using setMaterialForRendering for a mesh on one of the FrameGraphGeometryRendererTask. The textures in the inspector are similarly affected. I can shift frameGraph.addTask(renderTask); earlier to resolve screen visuals but the tex are still affected. Is this wai?

Seems like the framegraph doesn’t like the depth texture shared between 2 tasks?

The problem is due to the order of operations and the resulting floating point precision when the sphere is drawn by the default vertex shader the first time and by the custom vertex shader the second time. The depth value is not generated in exactly the same way, resulting in depth fighting.

You must generate gl_Position in exactly the same way as the default vertex shader:

vec4 worldPos = world * vec4(position, 1.);
gl_Position = viewProjection * worldPos;

Babylon.js Playground

Note that doing gl_Position = viewProjection * world * vec4(position, 1.); doesn’t work!

Babylon.js Playground

Passing viewProjection as a uniform rather than via the Scene ubo doesn’t work either:

Babylon.js Playground

I created a ticket in the Chromium repository for this last point two years ago (Chromium). Please feel free to vote for it to improve its visibility!

[phaselock]

@Evgeni_Popov

Regarding the crash problem, I created a repro outside of Babylon and created an issue in the Chromium repository:

Awesome, thanks!

@Evgeni_Popov

It’s an oversight, it’s fixed in this PR

Thank you!

@Evgeni_Popov

You must generate gl_Position in exactly the same way as the default vertex shader

repro: https://playground.babylonjs.com/#OQTYRG#21
I’m still getting artifacts with a textured mesh. Setting needAlphaBlending = true resolves it. Is there a uniform I forgot to bind or is this wai? Not seeing anything in the material/tex related to alpha…hmm… nvm, not a biggie cos its not my asset.

@phaselock

what is the equivalent of effectRenderer.render(effectWrapper,outputTexture) using framegraph textures, ie, the internal textures managed by the FrameGraphTextureManager?

In case this was missed. Do I need to submit a PR/feature request? Or is there another way around it? I love how neat my previous engine.onResizeObservable is…hehe

[Evgeni_Popov]

@phaselock

repro: https://playground.babylonjs.com/#OQTYRG#21
I’m still getting artifacts with a textured mesh. Setting needAlphaBlending = true resolves it. Is there a uniform I forgot to bind or is this wai? Not seeing anything in the material/tex related to alpha…hmm… nvm, not a biggie cos its not my asset.

The rabbit has bones, so there is additional code in the vertex shader. I updated the PG to support morphs, bones and (thin/) instances:

Babylon.js Playground

@phaselock

@phaselock

what is the equivalent of effectRenderer.render(effectWrapper,outputTexture) using framegraph textures, ie, the internal textures managed by the FrameGraphTextureManager?

In case this was missed. Do I need to submit a PR/feature request? Or is there another way around it? I love how neat my previous engine.onResizeObservable is…hehe

It’s on my todo list (see Frame Graph implementation v1 · Issue #16536 · BabylonJS/Babylon.js · GitHub), stay tuned!

[phaselock]

@Evgeni_Popov

The rabbit has bones

Aye, I figured it was missing declarations after comparing with my src. Thanks!

@Evgeni_Popov

It’s on my todo list

yay!

[phaselock]

Repro: Babylon.js Playground

Playing around with the water shader, the water mesh is subjected to 2 shader materials (water + task). Dependencies are set correctly, not seeing anything wrong with the testShader either. But the water surface is showing artifacts, wai?

[Evgeni_Popov]

Would you have the same (working!) PG without frame graphs? It will be easier to investigate the issue.

[phaselock]

here: https://playground.babylonjs.com/#ESQ4BY#1

Its the stylized water shader with the refraction removed. Testing what happens with multiple depth tasks and different materials rendered into the textures, ie, overload+stress.

My turn to hit the sack, cheers!

[Evgeni_Popov]

@phaselock

here: https://playground.babylonjs.com/#ESQ4BY#1

The shader uses the depthTex sampler, but this sampler is never set anywhere. It’s undefined behavior to do so, the shader will receive a “random” texture (I checked with Spector, the shader gets crate.png for the depthTex texture).

If you enable the default depth renderer and set depthTex from it, you get:

Babylon.js Playground

The default depth renderer generates normalized view depths by default, so you should use PREPASS_NORMALIZED_VIEW_DEPTH_TEXTURE_TYPE in the frame graph version:

Babylon.js Playground

With this change, we get the same output in both cases.

@phaselock

My turn to hit the sack, cheers!

Good night!

[phaselock]

I stripped the original depthTex out from the water shader when I removed the frame graph. You were looking at the wrong pg, my bad!

Simplified the problem down: Babylon.js Playground

This is educational. So, if you have a mesh with a shaderMaterial that has vertex code
gl_Position = worldViewProjection * vec4(position, 1.0);
This works correctly in normal scene.render. And will work in the frame graph IF the mesh does not have another shaderMaterial used with setMaterialForRendering for a frame graph task! If you do, you get the artifacts! In short, it doesn’t matter how many or what type of materials (plugin/custom/shader etc) are associated, the culprit is from the vertex code.

This is important for users who want to write shaders to the frame graph in future! At least until Chrome fixes their end. As it has yet to be triaged, the fix isn’t coming soon.

The fix: https://playground.babylonjs.com/#J0F7M8#1

You want to port existing shaders to the frame graph? Make sure any vertex code is updated to #include<sceneUboDeclaration> +uniformBuffers: ["Scene"]

vec4 worldPos = world * vec4(position, 1.);
gl_Position = viewProjection * worldPos;

Tripped by it twice now, leaving it here as reminder.
Reference: Feedback on frame graphs - #31 by Evgeni_Popov

[Evgeni_Popov]

This issue is not related to frame graphs, but to the shader code and how you render in your scene: you will encounter the same issues without a frame graph if you do the same things than the frame graph PG does.

Here’s a PG that does the same thing than the frame graph in your PG:

• It creates two depth renderers, d1 and d2, with d2 using the same depth attachment texture as d1.
• It sets testMaterial for the sphere when rendered in d2.
• It sets customMaterial for the sphere when rendered in the normal scene
• It creates a pass post-process so that the scene is rendered in a specific rtt, allowing us to attach the depth of d1 to this rtt

Babylon.js Playground

You can see the same artifacts as with the frame graph version.

We can even simplify the PG and use a single depth renderer while still retaining the artifacts:

Babylon.js Playground