I work in a project that uses Babylon.js, and yesterday I found an issue with meshes that have a negative world transform determinant and a two-sided material (i.e. no backface culling).
I believe I found the reason for the issue and was able to make a minimal reproducing example that showcases my theory of the bug.
Playground link: https://playground.babylonjs.com/#0XYMA9#167
In the scene, you see a black plane that represents a ceiling, as well as a small black sphere which shows where a point light has been placed (which is below the ceiling). We are looking up at the ceiling.
Note that the ceiling has a negative world transform determinant.
The material on the ceiling has two-sided lighting and no backface culling, and I would expect the point light to illuminate the ceiling.
If you look at line 15, there is a boolean to enable a “workaround”, which after 1 second will produce the expected results (point light illuminates the ceiling).
If you use Spector to inspect the GL draw calls, you can see that the difference between the working state and the broken state differ only by a different glFrontFace call/state, which in turn would affect the shader’s statement
gl_FrontFacing which flips the normals if needed.
My theory/actual solution
I tried to find the issue inside Babylon, and I believe this issue arises from the code found in this if and else block in the mesh.render function.
More specifically, this code either evaluates and sets the
instanceDataStorage.sideOrientation (taking the world transform determinant into account), or just fetches the value from
For it to evaluate the
sideOrientation, the material has to enable backface culling or we have to set an override on the mesh, none of which is true for our case. Instead, for us, the code never evaluates the
sideOrientation and the world transform determinant is never taken into account, causing the flipped normals for us.
The workaround in the playground example tries to make sure that the mesh is rendered at least once with a material that has backface culling, causing the rendering logic to evaluate (and store)
sideOrientation, which is then only read when the actual wanted material is applied later.
(I haven’t tested it, but I would assume if I changed the determinant sign again with the wanted material, it would have the incorrect flipped normals again.)
I read this code in the mesh rendering logic as “If backface culling isn’t enabled, then we don’t need to care about which face is which”, which is true for a lot of cases, but we do need to care when twoSidedLighting is enabled as well, as it affects the normal flipping in the shader.
But this is just my theory of the issue, please let me know if you have other ideas. I’m not well-versed with Babylon’s code either, so I’m not sure how difficult it would be to add “twoSidedLighting” check next to the backface culling check either. But this is an issue for us in our project, so I hope we can find a solution.
I’m not sure how this is done usually, but please let me know what our next steps could be