SSAO2 blur shader issues

[ChristofferA]

I have been going down the rabbit hole trying to figure out why SSAO2 produces some odd banding for us. Doing so I have discovered that this also happens in the public playgroup demo and that there seems to be an unrelated error in the ssao2 shader.

Test case: https://www.babylonjs.com/demos/ssao2/

1. Shader is always bilateral, no matter if you enable or disable it in code?

expensiveBlur
If bilateral blur should be used
Returns boolean
SSAO2RenderingPipeline | Babylon.js Documentation

That flag gets converted to the define “EXPENSIVE” set to either 1 or 0 on line 147 and 151 here:

BabylonJS/Babylon.js/blob/master/packages/dev/core/src/PostProcesses/RenderPipeline/Pipelines/ssao2RenderingPipeline.ts

The SSAO2 shader looks at the EXPENSIVE define and if set runs an inline bilatteral filter in main (line 176), and if not set, calls blur13Bilateral on lines 227 and 230.

blur13Bilateral also contains a bilateral filter, but with a fix set of samples.

So no matter what expensiveBlur is set to, from my undersstanding of the code, you end up with a bilateral blur instead of one of the two normal blurs also defined in the shader.

Another thing is that I can’t see a single pixel change in the demo linked above when you tick/untick the checkbox, which is odd since the two algorithms aren’t identical…

2. Bilateral blur messes up edges it is supposed to preserve
This is the cause of the banding in our product, and it is visible in the demo as well.

Zooming in on one of the lines that ought to be occluded, after incresing the strength a bit to see it clearer:

I can understand that a blur would result in that dark areas in the generated occlusion layer would spill over, but one would expect it to spill over in a more blurred and gradual way, not a sold bar like that.

If you use Spector to see the result of the SSAO2 buffer, it becomes clear that something isn’t right.

Initial SSAO2 pass, things look great. Notice the very strong black lines on the walls.

After the horistontal blur pass, notice how the vertical line in the middle got very distorted. As a side note, the floor has been blurred a lot, but the right wall looks almost untouched. This makes sence, the bilateral part of the shader seem to look at the Z diffs of the samples ignore samples with a big z-diff.

This is after the vertical blur is applied, and represents the final SSAO2 layer generated, which is then blended with the color buffer. Notice how the strong horisontal lines have been completely obliterated as well now, completely messing up the generated occlusions in those areas. Also notice that the floor once again got blurred, but not the walls, which makes sense.

The dark lines do probably have heavy diffs in Z values between the samples, but in that case, the algorithm should almost only use the center sample?

Also, both in the demo linked above and in our deploy, the clear bands show up, always the same number of pixels wide. My guess here is that all samples with almost the same Z as the center sample will be weighted the same, no matter how far from the center sample it is. If so, is that really logical for a blur filter?

[shaderbytes]

Wow hope all your effort results in some fixes and better results!

Im not clued up enough on post process code like this to be able to help myself , so I really appreciate you digging into it. I do know the current SSAO 1 or 2 in babylon is in drastic need of some improvements to actually be usable.

[ChristofferA]

I hope I can get some fix in as well, but first I need to know that my analysis is right. I have never written shader code before nor tried to build Babylon. I can for sure give it a shot though, but that probably can’t happen on company time, unfortunately.

Here is how it looks in our production code, running Babylon 4.2.

SSAO off

SSAO2 on, before the blur steps

This is of course very noisy but shows where the occlusions should be. Notice how dark it is between the drawer fronts, as expected.

SSAO2 on, including the blur steps

Notice how the occluded areas in between the drawer fronts are suddenly no longer occluded, but the extremely non-occluded areas on the top and bottom of the drawer fronts are suddenly rendered as they were occluded.

Manually blurred SSAO2

For this image, I extracted the SSAO2 buffer using Spector. Then I manually blurred the SSAO2 buffer using a gaussian blur and composited that on top of the first image using paint.NET. The results are in my opinion much, much better. However, there you get the occluded areas spilling over into areas with very different Z values, which is what the bilatteral filter tries to remove. Or at least I think it does.

[shaderbytes]

Here is a video again to try get the devs engadged again over the SSAO issues because I have made threads about this before and it didnt result into anything useful. Basically more samples were added to try hide the shading errors.

SSAO problem

In the video i point out the banding issue once again and that it it present no matter what the settings. The current solution put forward is always " it just needs the right settings" which is actually always just a case of “just reduce all the settings to try hide the errors” and that is not a solution. Besides this banding shading error , as you point out the actual overall result of the effect is not good anyway.

I did also now show the results of another online rendering engine in this video, I know they are not open source and all that , but its fair to point out the drastic difference in quality and user experience here.

This is a bitter honest pill that has to be swallowed, the current SSAO in babylon is broken and not usable in any production. I would think this should be on the top of the list for the rewards program for bug hunts

( ps dont flame me for being heated in my words , i dont mean to offend , I’m just passionate about wanting this fixed haha )

[Evgeni_Popov]

Here’s a fix for the expensive blur that could not be disabled:

SSAO2: Fix expensive blur that could not be disabled

SSAO2 is under the radar for an overhaul: Improve SSAO / Mirror reflections / Refractions · Issue #12630 · BabylonJS/Babylon.js · GitHub

I can’t help much more because I don’t know the algorithm and it is a bit complex. Maybe @CraigFeldspar may be able to help if he is around. There has been some discussing about SSAO2 in Ssao 2 has different results each time i add it to the scene ( video showing it ).

[ChristofferA]

@shaderbytes , interesting video. I haven’t seen that issue I think. The best input you could provide at this point to complement that video would be to use Spector to grab the SSAO output buffer before the blurring stages. That way you can see if the SSAO2 algorithm or the bilateral blur algorithm is to blame.

Given that from what I can read, basic bilateral blur filters are prone to both banding and “gradient reversal” perhaps it is to blame in both cases. If so, perhaps the issue will be improved for you if you could switch to a simple regular blur instead, i.e. that flag that isn’t working.

@Evgeni_Popov , regarding your fix; if the shader code has booth #ifdef and #if in it, why isn’t the old code correct? Shouldn’t #if actually look at the value of the define rather than the fact that it is defined? (Note that I haven’t written shader code, so please educate me.)

---

I am really excited to try and debug and fix (or at least improve) the SSAO2 shader code and will try to set up the dev environment today.

I would also try at some point to implement a temporal SSAO2 algorithm, by jittering the randomness texture and somehow accumulating the resulting SSAO buffer over time. I have seen an older thread talking about doing this for soft shadow (by jittering the lights) by porting over some Three.js code, but I don’t think that happened. Does anyone have any pointers to existing code to look at? Perhaps some motion blur implementation?

[Evgeni_Popov]

@ChristofferA

if the shader code has booth #ifdef and #if in it, why isn’t the old code correct? Shouldn’t #if actually look at the value of the define rather than the fact that it is defined?

This doesn’t work because we wrote our own preprocessor parser and it doesn’t support the #if identifier syntax in the same way as full-fledged preprocessor parsers: it is interpreted as #ifdef identifier instead.

You must explicitly add an arithmetic operator like #if identifier != 0 to get the same behavior as other preprocessor parsers.

[ChristofferA]

@Evgeni_Popov: Got it, thanks! Unexpected though and thus error-prone. Perhaps have a pipeline check for lines with an #if without a conditional expression? Something like: [whitespace]#IF[whitespace][non-whitespace][whitespace]\n

In any case, I got the dev environment up using gitpods. That was super easy, what a fantastic system.

I am currently in the progress of fixing so expensiveBlur does what it is supposed to, and also identified a few other code errors along the way.

How much are you allowed to change in a pull request? For example, can I add and expose a setting for controlling how many samples are used in the blur filters and default it to 16 which is hard-coded today, or would that fall under a bigger refactoring?

[Evgeni_Popov]

You are allowed everything in a PR if it improves or fix things .

One important thing to remember is that we should not break backward compatibility, except if it is a bug fix. So, adding a parameter to control the number of samples is no problem, as long as you default it to 16 (as you said).

Looking forward for your PR!

[shaderbytes]

@ChristofferA

That way you can see if the SSAO2 algorithm or the bilateral blur algorithm is to blame.

i did that now , so interesting finds are , its sometimes both , and sometimes only the blur stages and this is tied to using sample values ( or at least it appears to be )

eg lower samples values like 13 , the actual algorithm produces the banding :

its really from about 16 samples that the banding seems to not be visible from the algorithm :

but then comes back with the bluring ( first pass always there already )

Now for a very weird observation :

If I load my model , and I translate and zoom the camera only , no rotation , then this bug is not present, i can change samples and strength and the banding is not there. As soon as I rotate the camera in any manner , then the banding is there and its impossible to get back to the state of before rotation by changing any settings.

[ChristofferA]

@shaderbytes: Thanks, odd indeed.

Just to be sure, have you tried lowering the “max Z” as far as possible until the SSAO effect starts fading out in your scene?

The reason I ask is that your images really look like z-fighting, and AFAK having too high “max Z” could cause that due to limited Z-steps in your scene.

[ChristofferA]

OK, think I reproduced it here by disabling the blur filter and experimenting.

A sample size of 1 looks OK:ish.
Sample sizes 2-14 all look very, very bad with no obvious improvements.
Sample sizes 15 and above all look OK, and improve with more samples.

So unless I broke something locally, or it is extremely scene-dependent, sample sizes of 2-14 are all bugged.

[Evgeni_Popov]

Actually, samples < 16 uses a different algorithm than samples >= 16 to generate the random samples, and I did notice that the algorithm used for samples >= 16 is better.

Note that 5 months ago a bug has been fixed in the algorithm used when samples < 16, but I guess you are testing with the latest version?

[shaderbytes]

here is a video to show the weird quirk i discovered mentioned above , if you dont rotate the camera … the bug does not happen! Once rotated the bug is there

SSAO2 debugging

[shaderbytes]

@ChristofferA

Just to be sure, have you tried lowering the “max Z” as far as possible until the SSAO effect starts fading out in your scene?

based on the quick i outlined now , maz Z cant be a factor since it is untouched. Its view direction of some sort?

( ps , also , I did mention that somehow these threads always end up with proposed solutions like “just dial in the settings better” which do not fix the problem , they just making it slightly less visible )

There should be no setting of any sort that can make it bug out anyway. I pointed out in other engines that you can play with any combination of the full modulation range for each property , like strength , bias , radius and it never bugs out.

The code should be adjusted to not allow a user to set a value that causes problems. Eg in babylon , drag min Z to zero and it bombs out.

[Evgeni_Popov]

Would you be able to share the model you use in the video, as the problem can clearly be reproduced there?

[ChristofferA]

@shaderbytes

The code should be adjusted to not allow a user to set a value that causes problems. Eg in babylon , drag min Z to zero and it bombs out.

I agree there. The new code I’m adding follow those thought, but if it doesn’t follow Babylon code conventions I can of course remove it.

As for z-fighting issues, those usually do change with camera rotations. It could be with trying out at least.

[ChristofferA]

@Evgeni_Popov

Note that 5 months ago a bug has been fixed in the algorithm used when samples < 16, but I guess you are testing with the latest version?

I’m running against master, that is correct, right?

[shaderbytes]

@Evgeni_Popov

Would you be able to share the model you use in the video, as the problem can clearly be reproduced there?

sure here it is :

scene.glb

[Evgeni_Popov]

Thanks for the file!

The problem comes from the fact that the pre-pass renderer is using a 16 bits float texture for the depth and there is not enough precision. I have a PR in progress (for SSR) that will let change the precision of this texture, but in the meantime you can force using the geometry buffer:

Babylon.js Playground

16 bits precision (prepass renderer):

32 bits precision (geometry buffer renderer):

I’m running against master, that is correct, right?

That’s correct!