Shadow from partly transparent (video/chromakey)texture

Questions
1

Hallo, and sorry for my terrible english :wink:
Is there a way to get the shadow from an plane with chroma-keyed video-texture not to throw the shadow form the plane itself but from the not transparent parts of the texture.
Iā€™m paying an greenscreen-video with an person; i make the green pixels transparent by using following function:

videoMat.Fragment_Before_FragColor(vec3 chromaKey = vec3(0.15,0.64,0.15); // 0.15,0.64,0.15 // 0.40,0.90,0.52 vec3 tolerance = vec3(0.18); vec3 limitMin = vec3(chromaKey-tolerance); vec3 limitMax = vec3(chromaKey+tolerance); limitMin.r = max(limitMin.r, 0.); limitMin.g = max(limitMin.g, 0.); limitMin.b = max(limitMin.b, 0.); limitMax.r = min(limitMax.r, 1.); limitMax.g = min(limitMax.g, 1.); limitMax.b = min(limitMax.b, 1.); if( ((color.r >= limitMin.r) && (color.r <= limitMax.r)) && ((color.g >= limitMin.g) && (color.g <= limitMax.g)) && ((color.b >= limitMin.b) && (color.b <= limitMax.b)) ){ discard; }else{ color.a = 1.; });

Now iā€™d like to cast only the shadow of the person, not the videoplane itself.

MANY thanks for any help.

2

Welcome to the forum!

You can use BABYLON.ShadowGenerator, and push your render list into the render loop.

Hereā€™s the documentation:
https://doc.babylonjs.com/babylon101/shadows

Galen

3

thanks for reply.
BUT, ialready pushed it to the shadowgenerator, but i only get an rectangular shadow of the whole plane.
Anything else i could do to get the shadow from the non transparent parts of the video-texture only?
i managed to get the correct shadow by using an transparent png as texture, but NOT using the video.

var videoTexture = new BABYLON.VideoTexture("s",["./cam4.mp4"], vrscene);

var videoMat = new BABYLON.CustomMaterial(ā€œvideo materialā€, vrscene);
videoMat.Fragment_Before_FragColor(
vec3 chromaKey = vec3(0.15,0.58,0.18); // 0.15,0.64,0.15 // 0.40,0.90,0.52 vec3 tolerance = vec3(0.25); vec3 limitMin = vec3(chromaKey-tolerance); vec3 limitMax = vec3(chromaKey+tolerance); limitMin.r = max(limitMin.r, 0.); limitMin.g = max(limitMin.g, 0.); limitMin.b = max(limitMin.b, 0.); limitMax.r = min(limitMax.r, 1.); limitMax.g = min(limitMax.g, 1.); limitMax.b = min(limitMax.b, 1.); if( ((color.r >= limitMin.r) && (color.r <= limitMax.r)) && ((color.g >= limitMin.g) && (color.g <= limitMax.g)) && ((color.b >= limitMin.b) && (color.b <= limitMax.b)) ){ //discard; color.a = 0.1; discard; }else{ color.a = 1.; });
videoMat.diffuseTexture = videoTexture;
videoMat.diffuseTexture.hasAlpha = true;
videoMat.backFaceCulling = false;
videoMat.texture = videoTexture;
videoPlane.material = videoMat;
var shadowGenerator = new BABYLON.ShadowGenerator(1024, light);
shadowGenerator.getShadowMap().renderList.push(sphere);
shadowGenerator.getShadowMap().renderList.push(sphere2);
shadowGenerator.getShadowMap().renderList.push(videoPlane);

4

You have a valid point. Perhaps a bug? If youā€™re using a hemispheric light then the shadow generator wonk work as hemispheric light donā€™t cast shadows. But hereā€™s a PG scene I modified which uses a point ligh and I would expect it to cast a shadow from the plane.

https://www.babylonjs-playground.com/#P75330#31

Hereā€™s a PG scene with a texture.
https://www.babylonjs-playground.com/#P75330#32

If I place a sphere in the scene it works fine. Anyone have any ideas? I believe I tried everything I know.

Galen

1 Like
5

Unfortunately video do not contains alpha value so you also have to update the shadow generator code to update it to take your chroma key change into account

1 Like
6

could you give me any hint, how to do this??? :wink:

7

So maybe first step would be for you to prepare a plyaground which repro where you are at the moment and then we could help you :slight_smile:

8

First of all: Thanks for your help!

This is the playground:

https://playground.babylonjs.com/#II3Z1J#1

Problem:
.) the shadow of the video-texture is the whole plane-rectangle. (seems, i need to set the pixel alpha anyhow, instead of using discard)
.) is there a way to use the video without a light, so i donā€™t get the lightā€™s reflection when doing chroma-key?

9

Here is an example on how to use your own shader for the shadow map as well:
https://www.babylonjs-playground.com/#WE7DRF#4

10

Following up on this thread. Anyone figure out a working version? (last PG throws a bunch of errors).

11

I would recommend to use Node Material Editor now for the same question

12

@Deltakosh, can you elaborate on your last response? I am trying to get shadows to work with transparent videos, and havenā€™t quite got things working with all the good stuff in this thread.

13

I meant: NME is a good candidate to help writing shaders (for both material or shadows)

14

You can use the new ShadowDepthWrapper class to easily generate shadows for your custom shaders:

https://playground.babylonjs.com/#II3Z1J#3

See Custom shadows with ShadowDepthWrapper

1 Like