ShaderMaterial seems to occlude the GlowLayer, even when the ShaderMaterial’s shader returns an alpha of 0 or close to it. Not sure if there is some setting that I am missing or if this is a bug?
Here is a demo which cycles through some materials on an occlusion plane. The ShaderMaterial (red) seems to block the glow, whereas the PBRMaterial (green) does not occlude the glow. The latter behaviour is what I would expect to happen from the ShaderMaterial as well.
Also a note: setting material1.alphaMode = BABYLON.Constants.ALPHA_ADD; will result in the ShaderMaterial not blocking the glow, but this change is not desirable for my use case.
Another solution that worked for me was to just assign emissiveColor = new BABYLON.Color3(0, 0, 0) to the ShaderMaterial as an expando property. Then the if check linked above will also pass.