Transparent shadows?

Questions
2

Welcome aboard!

Soft transparent shadows are not supported with node materials. However, it is possible to make them work with a bit of work:

https://playground.babylonjs.com/#CJEJD1#6

NME: https://nme.babylonjs.com/#Q85QUS#10

You need to generate yourself a Bayer dither pattern which is used to reject a fragment based on the current alpha value.

For this, I have created a custom block (new in 5.0) with the Bayer Dither code. Here’s the json file for this custom block:

{
    "name": "BayerDither",
    "comments": "Generates a Bayer Dither pattern",
    "target": "Neutral",
    "inParameters": [
        {
            "name": "input",
            "type": "Vector2"
        },
        {
            "name": "dummy",
            "type": "Float"
        }
    ],
    "outParameters": [
        {
            "name": "output",
            "type": "Float"
        }
    ],
    "functionName": "BayerDither8",
    "code": [
        "float BayerDither2(vec2 _P) {",
        "    return mod(2.0 * _P.y + _P.x + 1.0, 4.0);",
        "}",
        "float BayerDither4(vec2 _P) {",
        "    vec2 P1 = mod(_P, 2.0);",
        "    vec2 P2 = floor(0.5 * mod(_P, 4.0));",
        "    return 4.0 * BayerDither2(P1) + BayerDither2(P2);",
        "}",
        "void BayerDither8(vec2 _P, float dummy, out float res) {",
        "    vec2 P1 = mod(_P, 2.0);",
        "    vec2 P2 = floor(0.5  * mod(_P, 4.0));",
        "    vec2 P4 = floor(0.25 * mod(_P, 8.0));",
        "#ifdef SM_SOFTTRANSPARENTSHADOW",
        "    res = 4.0 * (4.0 * BayerDither2(P1) + BayerDither2(P2)) + BayerDither2(P4);",
        "#else",
        "    res = 0.0;",
        "#endif",
        "}"
    ]    
}

Note that there is currently a bug (will be fixed by NME: Fix missing comma in function call (CustomBlock) by Popov72 · Pull Request #11598 · BabylonJS/Babylon.js · GitHub) when there is a single input parameter for a custom block. So I have added a second dummy parameter as a hack to overcome the bug. Once the PR is merged this parameter can be removed.

This block is used like this:

image
image1305×315 41.4 KB

It’s simply the transcription of the GLSL code used by the soft transparent shadow code:

if ((bayerDither8(floor(mod(gl_FragCoord.xy, 8.0)))) / 64.0 >= alpha) discard;

You need to encapsulate this node material inside a ShadowDepthWrapper to use it as the custom code to generate the shadow map (see Shadows | Babylon.js Documentation) - line 35 in the PG.

There’s still a problem, though: the Dither Bayer pattern is also used to display the mesh and not only the shadow:

image
image625×513 10.6 KB

You need to disable the discard code when it is not used to generate the shadow map. The easiest way I found is that BayerDither returns 0 when SM_SOFTTRANSPARENTSHADOW is not defined (see the code in the json above): SM_SOFTTRANSPARENTSHADOW is defined only when the shader is used to generate the shadow map.

4 Likes