How do I make MSAA affect my post-process outline shader

Questions
,
1

Playground link: https://playground.babylonjs.com/#9465JA#7

I’m drawing an outline around scene objects by feeding the depth buffer output to a post-process shader which uses a Sobel filter to get outlines, and I’m using Gaussian blurring to make it smoother. But I’m still getting jaggies, and MSAA doesn’t seem to affect the outline. How do I get smooth outlines?

2

MSAA won’t help because AA is applied when triangles are rendered, and in your case aliasing occurs because of your custom shader.

Blurring just the sobel value won’t help much, aliasing occurs mostly because of the hard step between Sobel/No Sobel, that is edgeDetection = step(outlineThreshold, sobel);. You can try to use a smoothstep function instead of step:

This helps a little:

You can also use a AA post-process, like FXAA.

1 Like
3

I’d tried those two things, it doesn’t help a lot (and FXAA was giving me performance issues in my private scene). Looks like I’ll have to take another approach, but thanks for your help

Edit: Not sure why, but FXAA performs pretty decently now. Just using FXAA is good enough.

1 Like