separateCullingPass knocks out twoSidedLighting

Bugs
,
1

In short
Enabling separateCullingPass knocks out twoSidedLighting setting in materials.

What we want to achieve
Render a double sided, semi-transparent material with proper lighting (flipped normals for back side facing triangles) while also using the separateCullingPass option to first render the inside and then the outside of the mesh to minimize alpha z-issues.

Why do we want to do that
Imaging rendering a semitransparent open box. For alpha to work properly you need to render the inside of the box first, otherwise you will end up with the wrong z-order from some angles.

The meshes and materials come from an external source and we must be as smart as possible to make things work in the most number of scenarios, thus we can’t for example turn off the z-buffer or things like textures with semi-transparent holes in them will fail.

Suggestion for fix

  • Change code so twoSidedLighting always flips the normals when the “back side” pass of separateCullingPass is performed.
  • Either do this conditionally on backfaceCulling being false or update the documentation of twoSidedLightning so it applies both if backfaceCulling is false and/or during the “back side” pass of separateCullingPass.

Playground
A playground says more than a thousand words, so please have a look here:
https://www.babylonjs-playground.com/#GY9NG6#7

I commented the code quite a bit, here is most of the comments and a screenshot of how it looks. The boxes are numbered from left to right. All the boxes uses the same geometry which is a box with no top or bottom. The single light is indicated by the yellow arrow.

boxes
boxes1106Ă—322 64.8 KB

Box 1
The inside of the open box is rendered “incorrectly”:

  • Back and left inside is black even though the light is shining on it.
  • Right and front inside is lit even though it should be in shadow.

Box 2
We fix the lighting on the inside by enable twoSidedLighting which flips the normals on the “back facing” triangles in the shader.

Box 3
We now enable transparency, but since the back of the box is drawn before the front, we get very incorrect rendering when the camera is in front of the boxes, for example looking straight along the z-axis.

Note 1: Simply changing the order of the indices isn’t enough, it will always be wrong from some camera angle, so that is not an workaround.

Note 2: We ignore facet sorting since it isn’t feasible in real life scenarios.

Box 4
A very good way to solve the issue of rendering both inside and outside of the meshes is to first render only the back faces then rendering only the front faces. Babylon.js has this built in via the separateCullingPass option.

Enabling that fixes the alpha issues, however we are back at the lighting from box 1. Looking at the Babylon source code it appears that separateCullingPass renders the mesh twice (as expected) by simply changing the orientation setting causing the back triangles to turn to front triangles etc.

However, looking at the source code for twoSidedLighting in the shader code it only turns on if the triangles are back facing, which they never are due to the change of orientation by separateCullingPass. Hence twoSidedLighting is in practice disabled by separateCullingPass.

Box 5
Scanning the shader source for a workaround I found forceNormalForward which seems to be an effective workaround when separateCullingPass is enabled. We now have a case where both the alpha and lighting issues are sorted! =)

However, forceNormalForward has some drawbacks:

  1. It looks much more costly in the shader than twoSidedLighting.
  2. Normals will not be flipped, but rather forced to always face the same direction on the triangle, no matter what they where to begin with. This can be correct, but might not be wanted/expected in edge cases where normals might intentionally be “the wrong way” in the loaded mesh.
  3. It doesn’t seem like this option is available on StandardMaterial, only PBRMaterial?

Attempted workarounds
I have tried loads and loads of combinations and hacks to get this working before I found forceNormalForward. I could not find a single one that worked without either manually modifying a clone of the Geometry to flip all the normals and manually rendering it first to get the inside. This however must be much more expensive compared to how separateCullingPass works.

forceNormalForward seems to do the trick for us, but see the drawbacks mentioned above. Also, I feel it lacks documentation to state exactly what it does. The shader code in pbrBlockNormalFix.fx isn’t easy for a novice like myself to figure out, especially since it uses vEyePosition which makes me a bit vary if it is really the right workaround for us.

2 Likes
2

Thanks for the really thorough writeup @ChristofferA! This one is definitely one for the eyes of @Deltakosh

3

Thanks @PirateJC, just hope I got it right. =)

I can also add that this issue is sort of touched upon in the how-to on transparency. There is suggests using Mesh.DOUBLESIDE and turning off backfaceCulling. This is more or less exactly what we want to do, while also turning on twoSidedLighting.

However, Mesh.DOUBLESIDE is only available in the mesh builders, and looking at the source for that in the vertex buffer code (IIRC) that option simply doubles the amount of indices and manually reverses the normals and vertices.

That doubles the geometry data, takes some time to process for very large meshes and also means cloning and modifying existing meshes if you already downloaded and built them.

4

adding @sebavan or @Evgeni_Popov to see if they can take care of it :slight_smile:

1 Like
5

I think this is what we need to fix :wink:

6

@Evgeni_Popov can you have a look ? else I ll only be able to tackle it down on Tuesday :frowning:

7

I think that forceNormalForward has been added precisely to handle this case!

Indeed, we would need to use a different shader when drawing the back faces than when drawing the front faces because the check normalW = gl_FrontFacing ? normalW : -normalW; would need to be flipped to normalW = gl_FrontFacing ? -normalW : normalW;. But creating two effects for the same material is not supported, it would be too costly.

There are two additional lines in the shader to handle forceNormalForward, but when you take into account the whole shader code I don’t think it’s worth worrying.

Also, you don’t need twoSidedLighting in this case so set it to false to save a line in the shader, meaning this option only adds a line to the shader compared to using twoSidedLighting = true and forceNormalForward = false.

I don’t understand this one, it’s really a flipping that is performed: the normal is multiplied by the sign of a dot product (so either +1 or -1), so it will invert (or not) the result, the same way normalW = gl_FrontFacing ? normalW : -normalW; is also (potentially) inverting the normal.

It is not available indeed, if it’s something that is needed an issue should be created in the github repo.

forceNormalForward is the flag you need.

vEyePosition.w (note the .w) is only used as a multiplier (+1 or -1) to invert the computed normal if we are in a right handed system and/or the camera is mirrored. The eye position itself (vEyePosition.xyz) is not used in the computation.

I have updated the doc about that:
https://github.com/BabylonJS/Documentation/pull/2049

8

This was a bit over my current level of understanding, but I am curious to learn more if you got the time. =)

Why can’t the shader compiled when separateCullingPass is enabled take an additional input telling it to always / never flip the normals and set it during back-side pass when the widing order has been changed? Then it wouldn’t actually need the check that twoSidedLighting does since it can assume that all non-culled faces are actually originally back faces.

Again, a novice in this area, so I might be talking gibberish.

From the docs:
twoSidedLighting - “If sets to true and backfaceCulling is false, normals will be flipped on the backside.”
forceNormalForward - “Force normal to face away from face”

I would say that the two is semantically different.

The first one says it is flipping the normals on backside triangles. Backside here is based on the winding order of the vertices relative to the camera, correct? So it says that given a back facing face it will always flip the normal, no matter which direction the normal was to begin with.

The second one says it will force the normal to point “forwards” so to speak, but isn’t very clear on that to be honest. Both side of a face is “away” from it as long as it isn’t parallell to the face surface.

So take an edge case where a backside face has a normal pointing the “wrong” way for some bizarre lighting trick; from my understanding of the docs and novice eyes looking at the shader code, the result would actually end up different?

I wouldn’t say that it is terribly important, but it does read like forceNormalForward is rather supposed to be used when one suspect bad normals in ones imported geometry.

Cheers, thanks for that! Makes me feel safe with my current workaround.

Oh, thanks for clearing that up for me.

9

That’s because material properties are meant to be set in a specific method, namely bind / bindForSubMesh. Not doing so will lead to problems, and notably it won’t work with WebGPU. We could do without adding a new parameter, but then we are back to what I mentioned regarding the fact we would need to compile the shader two times (with a different #define to activate one case or the other) and so we would have to deal with two effects.

Also, I’m not sure we want to add a new parameter just for this (especially as we already handle this use case in another way), as we already have a lot of parameters for the PBR material…

[wording in the docs]

I think the wording for forceNormalForward is indeed a bit misleading / not clear, but the one for twoSidedLighting looks ok to me, what you describe is what it does.

forceNormalForward will always make sure the normal points toward the camera, meaning its direction will be toward the camera, not away: the normal will be flipped accordingly (it’s still a flip, so a -1 multiply).

It is indeed different from twoSidedLighting in case the normal of your back face is wrong and is pointing to the camera: in this case, twoSidedLighting = true will flip it whereas forceNormalForward = true won’t.

1 Like
10

Thanks again @Evgeni_Popov for the explanations.

I’m content with using foceNormalForward for our use case now that I feel confident that it does what we need and I find it highly unlikely that any of the models has their normals “the wrong way” intentionally. Only having it in PBRMaterial is also fine for us since we only use PBR. So :+1:

1 Like