Transparent shadows?

Hi there!

Recently started using Babylon.js and have been loving it! I’m having a bit of trouble understanding how to get transparent shadows working.

In the shadow documentation it says:

“It works by generating some dithering patterns in the shadow map, based on the alpha value of the fragment.”

In the soft shadow example, the transparent shadows were coming from the diffuse texture’s alpha channel. I’d like to have the transparent shadows controlled by whatever I’ve hooked up in the object material’s fragment output.

Below is a simple scene and the shader on the sphere.

https://playground.babylonjs.com/#CJEJD1#2
https://nme.babylonjs.com/#Q85QUS#1

What am I doing wrong? Thanks for taking the time out to read this post.

Welcome aboard!

Soft transparent shadows are not supported with node materials. However, it is possible to make them work with a bit of work:

https://playground.babylonjs.com/#CJEJD1#6

NME: https://nme.babylonjs.com/#Q85QUS#10

You need to generate yourself a Bayer dither pattern which is used to reject a fragment based on the current alpha value.

For this, I have created a custom block (new in 5.0) with the Bayer Dither code. Here’s the json file for this custom block:

{
    "name": "BayerDither",
    "comments": "Generates a Bayer Dither pattern",
    "target": "Neutral",
    "inParameters": [
        {
            "name": "input",
            "type": "Vector2"
        },
        {
            "name": "dummy",
            "type": "Float"
        }
    ],
    "outParameters": [
        {
            "name": "output",
            "type": "Float"
        }
    ],
    "functionName": "BayerDither8",
    "code": [
        "float BayerDither2(vec2 _P) {",
        "    return mod(2.0 * _P.y + _P.x + 1.0, 4.0);",
        "}",
        "float BayerDither4(vec2 _P) {",
        "    vec2 P1 = mod(_P, 2.0);",
        "    vec2 P2 = floor(0.5 * mod(_P, 4.0));",
        "    return 4.0 * BayerDither2(P1) + BayerDither2(P2);",
        "}",
        "void BayerDither8(vec2 _P, float dummy, out float res) {",
        "    vec2 P1 = mod(_P, 2.0);",
        "    vec2 P2 = floor(0.5  * mod(_P, 4.0));",
        "    vec2 P4 = floor(0.25 * mod(_P, 8.0));",
        "#ifdef SM_SOFTTRANSPARENTSHADOW",
        "    res = 4.0 * (4.0 * BayerDither2(P1) + BayerDither2(P2)) + BayerDither2(P4);",
        "#else",
        "    res = 0.0;",
        "#endif",
        "}"
    ]    
}

Note that there is currently a bug (will be fixed by NME: Fix missing comma in function call (CustomBlock) by Popov72 · Pull Request #11598 · BabylonJS/Babylon.js · GitHub) when there is a single input parameter for a custom block. So I have added a second dummy parameter as a hack to overcome the bug. Once the PR is merged this parameter can be removed.

This block is used like this:

It’s simply the transcription of the GLSL code used by the soft transparent shadow code:

if ((bayerDither8(floor(mod(gl_FragCoord.xy, 8.0)))) / 64.0 >= alpha) discard;

You need to encapsulate this node material inside a ShadowDepthWrapper to use it as the custom code to generate the shadow map (see Shadows | Babylon.js Documentation) - line 35 in the PG.

There’s still a problem, though: the Dither Bayer pattern is also used to display the mesh and not only the shadow:

You need to disable the discard code when it is not used to generate the shadow map. The easiest way I found is that BayerDither returns 0 when SM_SOFTTRANSPARENTSHADOW is not defined (see the code in the json above): SM_SOFTTRANSPARENTSHADOW is defined only when the shader is used to generate the shadow map.

4 Likes

Hi @ShaderRig ! Just checking in if you have any more questions about this topic :slight_smile: