Shadow effect in the corners (Shadows/Mesh)

In this screenshot, an interesting effect appears in the corners. Any ideas what it is and how to achieve this effect?

Is that a 3d object? In any case looking at the entire model/drawing this shadow is baked/brushed. You can see it by looking at the angle and comparing the shadows on the different sides. They are all straight following the edges so it cannot be a ‘real’ shadow.

1 Like

Yes, it’s a 3d object. Are there any other options besides texture baking to achieve this effect?

Nah, I don’t think so. Nothing that will integrate it smoothly with the texture (and then may be with varying the size of blur on different edges. Except of course a shader or NME. I can’t think of an existing one but I don’t know them all. May be someone else or @PatrickRyan if he has a minute would know of a pre-existing you could rapidly twist and tweak to your likin?

Edit: Actually just thinking of it. May be you could twist it from the base of a toon shader @Lun created some Could be something to use as a base? Not sure though, it’s just an idea that came to me like that.

1 Like

@Nawar, I can’t think of a way to do this automatically without a geometry shader, which we don’t have in WebGL. This is because the returned color is darkened only for inside corners, not outside corners which means we need to know the normal vectors of neighboring vertices and compare vectors to isolate any interior corners. Since the edges in the model are hard, each vertex along a hard edge will be split into two vertices sharing the same location with perpendicular normal vectors. This is what we would be testing, neighboring vertices with the same world position.

There is a work around here, though, but it would be a manual process akin to baking. Though you might have a difficult finding a method to bake the map that is as clean as what is shown above. Curvature masking is one of the ways you could try but it wouldn’t give you as clean a line along the interior corner as that method us used to isolate concave surfaces that might gather dirt in a natural way or how a convex surface might show wear.

If you were to assign vertex color to your mesh and use a particular color for your inside corner vertices, you can isolate them by filtering for a particular vertex color. For example, if you assign Color3(1, 0, 0) to all inside corner vertices and black to every other vertex, you can use the red channel of mesh.color in node material to generate a gradient from each corner vertex to each other vertex. You can then use a pow node to control the length of the gradient. This method would likely need to have some vertices physically split where you have an inside corner meeting an exterior face so that you don’t get spill over onto the exterior face. However, since you are dealing with hard edges anyway, the splits won’t matter. And in glTF, any vertex that is on a UV seam will get split anyway, so there are already many split vertices in your file if using glTF.

This is a more manual process, but if you are looking for the perfect consistency in your application of the darkening of the inside corners, this will likely be needed - not to mention way cheaper to render - without a geometry shader.

2 Likes

OK. Thanks for the detailed answer @PatrickRyan!

Suppose I would like to find all corners and paint the vertices with a gradient.

Where should I start?

You should be able to get this effect with SSAO, like in:

4 Likes

@Nawar, can you give a little more context to your question? What do you mean when you say “find all corners”? And when you say “paint the vertices with a gradient” where is this happening?

Are you looking to find all corners automatically? If so, where are you doing that? In a DCC tool like Max or Blender or are you thinking this will be at runtime? At runtime we don’t have the ability to create a geometry shader in WebGL, so the options are limited. I suppose you could scrub the triangle list looking at normal vectors and positions in your js/ts code and make some inferences based on comparing them as to what may be a corner. But I don’t think this will be an easy lift depending on the mesh.

I figured rather than using more words to describe how I would approach this, which can lead to confusion, I would quickly put together a simple example. This is using the vertex color technique I loosely described above.

There was an unexpected wrinkle in this technique, which needed to be accounted for. That is the fact that vertex color blending between vertices sometimes makes unexpected interpolations depending on the complexity of the selection. To combat this, I used two channels of the vertex color, one for horizontal corners (red) and one for vertical corners (blue).

You can see that the red and blue gradients behave as expected. If I used red for all of the corner pixels, I would get one face with a diagonal gradient rather than “L” shaped following the corner. The extra resolution in the center of the faces also helps shape the gradients. The faces with vertex color other than black are extracted from the original mesh and then combined again. This separates all vertices along the edge and still allows me to combine them back again to only have one mesh. This is why I can have extra resolution on these faces without artifacts.

This shader is a very simplified version for this example, but the critical parts of isolating the corner gradients is complete. The color on the mesh is just a simple dot product interpolation between light and dark colors based on surface normal and view angle. The part of the shader that adds in the darkened pixels on the inside corners is this bit:

This is simply taking the max of the red and blue channels of mesh.color and doing a max operation on them to get the gradient for all inside corners.

Then we use a pow block to control the gradient a bit:

Then multiply by a value between 0 and 1 to control the strength of the corner:

Then invert with a oneMinus block and use a max with the color for the corner:

This is then just multiplied over the rest of the diffuse contribution on the shader.

6 Likes

Wow! This is a very comprehensive answer! I don’t know how to thank you for taking the time to answer this. Thank you!

I was thinking about generating all of this in the process of rendering, but it turned out to be many times more difficult.

I think the SSAO from the answer above is completely in line with the topic of the question.

2 Likes