I think what you want is to keep the shadow and the mirror and have everything else transparent?
Maybe this will work for you:
https://playground.babylonjs.com/#9KLUBA#4
If the shadow variable is strictly less than 1 it means the pixel is at least partly in shadow. And the if (color.r != 0.2 || color.g != 0.2 || color.b != 0.3) check is to test against the clear color.