I have been going down the rabbit hole trying to figure out why SSAO2 produces some odd banding for us. Doing so I have discovered that this also happens in the public playgroup demo and that there seems to be an unrelated error in the ssao2 shader.
Test case: Babylon.js - Screen Space Ambient Occlusion demo
1. Shader is always bilateral, no matter if you enable or disable it in code?
If bilateral blur should be used
SSAO2RenderingPipeline | Babylon.js Documentation
That flag gets converted to the define “
EXPENSIVE” set to either 1 or 0 on line 147 and 151 here:
The SSAO2 shader looks at the
EXPENSIVE define and if set runs an inline bilatteral filter in main (line 176), and if not set, calls
blur13Bilateral on lines 227 and 230.
blur13Bilateral also contains a bilateral filter, but with a fix set of samples.
So no matter what
expensiveBlur is set to, from my undersstanding of the code, you end up with a bilateral blur instead of one of the two normal blurs also defined in the shader.
Another thing is that I can’t see a single pixel change in the demo linked above when you tick/untick the checkbox, which is odd since the two algorithms aren’t identical…
2. Bilateral blur messes up edges it is supposed to preserve
This is the cause of the banding in our product, and it is visible in the demo as well.
Zooming in on one of the lines that ought to be occluded, after incresing the strength a bit to see it clearer:
I can understand that a blur would result in that dark areas in the generated occlusion layer would spill over, but one would expect it to spill over in a more blurred and gradual way, not a sold bar like that.
If you use Spector to see the result of the SSAO2 buffer, it becomes clear that something isn’t right.
Initial SSAO2 pass, things look great. Notice the very strong black lines on the walls.
After the horistontal blur pass, notice how the vertical line in the middle got very distorted. As a side note, the floor has been blurred a lot, but the right wall looks almost untouched. This makes sence, the bilateral part of the shader seem to look at the Z diffs of the samples ignore samples with a big z-diff.
This is after the vertical blur is applied, and represents the final SSAO2 layer generated, which is then blended with the color buffer. Notice how the strong horisontal lines have been completely obliterated as well now, completely messing up the generated occlusions in those areas. Also notice that the floor once again got blurred, but not the walls, which makes sense.
The dark lines do probably have heavy diffs in Z values between the samples, but in that case, the algorithm should almost only use the center sample?
Also, both in the demo linked above and in our deploy, the clear bands show up, always the same number of pixels wide. My guess here is that all samples with almost the same Z as the center sample will be weighted the same, no matter how far from the center sample it is. If so, is that really logical for a blur filter?