My scene consists of a bunch of painting meshes (of arbitrary sizes) that are mounted on a wall. I’d like to add a shadow around and below them. I used blender to capture the shadow map around a painting as a template. Dropbox - Canvas_Shadow_map.png - Simplify your life
My intention is to stretch this template on a plane of arbitrary size and place that plane behind the painting mesh and front of the wall. I’d like to understand the best way to stretch the shadow map. I can think of few options, but not sure if any of these are feasible. So before I go too further down in a single path, wanted to get advice from folks:
option 1: Create the shadow plane and stretch the current texture to fit the plane. The issue with this is that the template consists of 2 regions - the dark area behind the painting and the soft shadows around it. The soft shadows will be highly distorted. Say if I stretch the texture 3X its original size then then soft shadows on the left will be stretched by 3x and wont look realistic. What I want is the soft shadows to be of fixed width (but variable length).
option 2: write your own glsl shader. Never done this before so don’t know where to start or if its even possible. Any examples here will be greatly appreciated.
option 3: use Node Material Editor. Again never done it before, so if this is the easiest route - would love to see some examples of similar ways textures are used.
Well, I guess there’s no secret on this matter. You have a number of options (more or less dynamic), either dev/maths- or design- based, and you already stated some in your post.
Obviously, scaling greater than ~1,5 a static shadow map projected on a plane will not provide good results. I guess a possible solution would be to create a set of these scalable up to ~1,5 textures and use the one most adapted to the original size/ratio of the mesh.
Going more dynamic, NME could be a good solution, I believe. I don’t think it would be all that hard to set-up an nme creating a gradient from the mesh size (although I never did it so far, but seems quite feasible in theory).
Twisting things a little and not using shaders, may be setting-up a nine-patch image for a 2D GUI for mesh could also be an option. You could make sure your shadow gradients always remain the same whilst your inner part covers the size/ratio of your mesh.
I guess at some point, there are many ways one could imagine (either on the side of dev or design). Being essentially a designer, I can straight think of another way to twist things. What if for each painted mesh, you would create all assets with the outer shadow in the picture. Next, crop your image/mesh with an opacityTexture; Create a plane behind the mesh of the same size and simply use an inverted opacityTexture and a black or dark color as a BG. I’m sure that could also do the job and would be quite ez to handle in a workflow.
I’m actually eager to read @PatrickRyan answer for this. See if he has other ideas or a different approach (and I’m sure there are many others).
@evidanary, I think that you will likely get the outcome you are looking for by baking your shadow texture in a ray tracer. To use this you would need to employ a nine-patch approach like @mawa mentioned. This would be the best option for retaining the quality you get from a ray traced shadow while reducing the distortion in the sides and corners. We have that built into the GUI system, but in terms of applying it to a mesh, you would need to do a projection on mesh of the ADT, but there is some overhead there since need to include the GUI system just to render your texture.
So that leaves a custom approach with a node material. Unfortunately, we didn’t have any examples of this kind of technique, so I put one together for you. I made a simple texture so we can easily see what is happening with the technique:
You would start with your texture divided up like this. In this example, I used a uniform nine-patch where each patch is one-third the width and height of the quad. You can change up the shader to make this more flexible but I did not in this example because the shader is already complex.
Increasing the width will keep the corner patches from scaling and also keep the vertical patches from scaling in width and the horizontal patches from scaling in height. You will get stretching parallel to the sides of the texture though, which is pretty standard with nine-patch.
The same happens when stretching the height. In this example, I am using the GUI sliders to control the height and width of the quad, so there is some calculation going on with starting size and scale factor driving into the shader. If you are not dynamically changing, though, you can simply push the size of each dimension into the shader as you create the shadow for each frame.
One addition you could make is to tile the middle patches on the edges in whole units. This would reduce some of the distortion parallel to the edges, but you would need to come up with extra operations to determine if the middle patches are larger than X scale and then increase the tiling of the middle patches by one whole tile and compress the extra tile in. This may not be necessary if your shadows were made to tile along the edges nicely, but if you have detail in the shadows you want to keep, you would need to go this extra step to ensure you don’t see too much stretching.
I know that the shader is pretty complicated to look at if you aren’t used to node material, so let me know if you have questions.
You are amazing. I believe it would be good to be added to the doc, yes? In fact, any more examples for creating node materials would be (in my opinion) a very useful and nice addition to the doc and help get a grasp on NME. As for me, I bookmarked this one already
@mawa, thanks for the feedback! One of the reasons I made the example is to add it to our docs. The list of new content I am writing is growing, so it may be a little bit before it gets up, but it’s on the list. At the minimum, I am going to add it to the playground examples on one of the NME pages later today while the docs are written.
Wow! @PatrickRyan that’s amazing! Thanks so much for creating that NM. This works amazing.
I noticed the alpha of the fragoutput is not populated. I tried it with my shadow file and put a plane behind it so I can see some transparency effect https://playground.babylonjs.com/#KY4129#2 . Looks like the alpha isn’t coming through and I see a solid black texture.
@evidanary, the alpha wiring is exactly the same as the texture wiring. You just need to mix the alpha of each texture in the same way as you mixed the textures and then wire that to the frag out. I updated the playground with those changes for you to see. You will still need to tweak your texture and/or the shader to make sure that you are getting the final look you are after, but this should get you a good distance to that end goal.
I believe it would be good to be added to the doc, yes? In fact, any more examples for creating node materials would be (in my opinion) a very useful and nice addition to the doc and help get a grasp on NME. As for me, I bookmarked this one already 
