DOF + Vertex Shader Issue

In my game I have trees waving in the wind and for that I use a custom Vertex Shader. I’m running into an issue where the DOF post process doesn’t take into account Vertex Shader displacement resulting in my trees looking really odd.

Here is a PG demonstrating what I’m seeing:

https://playground.babylonjs.com/#4YBB22#6

Unfortunately, the depth renderer used for the DoF effect is using a custom effect to generate depth values that can’t handle custom vertex/fragment code that would be used in the regular material of the meshes…

There’s currently no simple way to fix the problem, I think we would need to think about that. Pinging @Deltakosh and @sebavan to let them know.

If you still really need it to work in the current state, here’s a way to do it:

https://playground.babylonjs.com/#4YBB22#8

It creates a specific material customMaterialDepth that will be used in place of the default depth material to render the bottle into the depth map. It is a CustomMaterial instead of a PBRCustomMaterial because the former is a bit lighter than the latter on the GPU side and we don’t loose anything by using a standard material as we are only interested in storing the depth value in the fragment shader.

Once we have this material, we need to use it over the one from the depth renderer. To do that, I have overriden the Mesh._processRendering method as at this point the depth renderer has already done its job and bound its effect to the pipeline, so we can override safely the effect with our own. The overrides consists in making sure our effect is ready to be used (the isReadyForSubMesh call) and use it for the subsequent rendering (the call to saveProcessRendering).

That’s not safe code as it is using methods / properties with a leading underscore, but at least it should work in your project if you use 5.0. In 4.2, there’s no _getCustomEffect method on subMesh, but you can easily overcome this:

https://playground.babylonjs.com/#4YBB22#9

This one will work in 4.2 and 5.0.

1 Like

What do we need to expose to make it work (officially)?

The best way in my opinion would be to have something like DepthWrapper (in the same way we have ShadowDepthWrapper) to provide the material to use for depth rendering. Basically, DepthWrapper would be nearly identical to ShadowDepthWrapper but a little simpler when doing shader code injection as it only needs to keep track of the depth (whereas ShadowDepthWrapper also needs to handle bias / normal bias / …).