I am working on this problem for several days now and it is freaking me out. My app needs to render into stencil at some point (for sth like a portal). A simple mask. Additionally, I am using a HighlightLayer. If I look at the portal from the back, nothing should be rendered. But still, the highlight of an object behind is interrupted by the portal. I think this might either be due to depth or stencil of the portal accidentally being written. Now I’d like to know in more detail what is going on in the stencil buffer etc, but am struggling to debug it properly.
So I am asking more generally: Do you have experience with using stencil and the Highlight Layer in conjunction? How do I need to tweak/”hack” the engine to get this to work? My research so far yielded, that a single HighlightLayer should only use the first two bits of the stencil. So if I write in the third, there shouldn’t be a problem…
Currently, using the highlight layer is not really compatible with custom usages of the stencil buffer, unfortunately.
The layer sets the stencil reference before drawing a mesh (if it is part of the layer). If you set the same value (ThinHighlightLayer._instanceGlowingMeshStencilReference) through the mesh material stencil block, then it’s fine. However, if you add other bits in the reference value, it won’t work as expected in the composition phase, because we don’t set the stencil function mask: we don’t use a restricted number of bits but the full range of the stencil value. That’s because each highlight layer has its own stencil reference value: 2 for the first layer, 3 for the second layer, and so on.
If you know the number of layers you will use is limited and the stencil reference value will never use more than X bits, we could set the function mask to 2^X-1 and let 8-X bits available for the user…
You can try it by overriding the ThinHighlightLayer._internalCompose method and by calling setStencilFunctionMask yourself:
BABYLON.ThinHighlightLayer.prototype._internalCompose = function(effect, renderIndex) {
// Texture
this.bindTexturesForCompose(effect);
// Cache
const engine = this._engine;
engine.cacheStencilState();
// Stencil operations
engine.setStencilOperationPass(Constants.REPLACE);
engine.setStencilOperationFail(Constants.KEEP);
engine.setStencilOperationDepthFail(Constants.KEEP);
// Draw order
engine.setStencilMask(0x00);
engine.setStencilBuffer(true);
engine.setStencilFunctionReference(this._instanceGlowingMeshStencilReference);
engine.setStencilFunctionMask(0x03); // 3 is ok if you use a single highlight layer.
// 2 passes inner outer
if (this.outerGlow && renderIndex === 0) {
// the outer glow is rendered the first time _internalRender is called, so when renderIndex == 0 (and only if outerGlow is enabled)
effect.setFloat("offset", 0);
engine.setStencilFunction(Constants.NOTEQUAL);
engine.drawElementsType(Material.TriangleFillMode, 0, 6);
}
if (this.innerGlow && renderIndex === 1) {
// the inner glow is rendered the second time _internalRender is called, so when renderIndex == 1 (and only if innerGlow is enabled)
effect.setFloat("offset", 1);
engine.setStencilFunction(Constants.EQUAL);
engine.drawElementsType(Material.TriangleFillMode, 0, 6);
}
// Restore Cache
engine.restoreStencilState();
}
If this works for you, we will implement a feature that allows you to define the stencil function mask to be used during the composition phase (or you can create a PR for this, if you wish!).
Thank you very much for your detailed and fast response, I appreciate it!
So I tried your code and it didn’t solve the problem. I stumbled across other stuff though that I didn’t quite understand, maybe you can help me out. In the ThinHighlightLayer constructor, the _instanceGlowingMeshStencilReference attribute is initialized with ThinHighlightLayer.GlowingMeshStencilReference++. I guess, this is supposed to let different layers write to different stencil bits, correct? I would expect that this reference should always only have a single bit set? But shouldn’t it then be ThinHighlightLayer.GlowingMeshStencilReference *= 2 instead of just ++ after the assignment? I also couldn’t really figure out what is happening with these counters for me: After I select an object, GlowingMeshStencilReference is 4, and _instanceGlowingMeshStencilReference is 3. As if something else snatched reference 0x02, although I can’t see what that should be in my app.
Either way, adding mask 0x03 did not help. I noticed another thing though: When the highlight layer is active (i.e. at least one mesh is added to it), the meshes that depend on my custom stencil bit, don’t seem to write into the depth buffer. Other objects that are behind and come in the next rendering group overwrite the pixels (I do call setRenderingAutoClearDepthStencil though). Furthermore, I guess, when the highlight layer is active, the stencil buffer is written to, even for fragments that didn’t pass my custom stencil test (even if I use “unused” stencil bits). When the highlight layer is active, apparently many stencil and depth related properties of my materials are overwritten. But I couldn’t find the code that’s doing this. If you could point me out to it, I could maybe finally understand what’s happening!
About the stencil in general: Imo, it would be best if different parties of an app (including e.g. the highlight layer) could allocate stencil bits via a centralized API. This API could look for a sequence of free adjacent stencil bits of a given length (like free memory management) and return a handle to the caller. The main purpose of the handle is to set the mask properly. Users of it could then still specify the stencil function and operations but ideally don’t get in touch with the mask anymore directly. That way, all stencil users are completely independent of each other and there can’t be such side effects. Just a proposal though, e.g. I don’t know how to align this with the existing material properties exactly…
Hope, you guys can still help me out with my issue!
We’re missing a number of elements in the highlighting layer and in the engine to achieve this. I’ll get back to it as soon as possible, but we’re currently in the code freeze phase and preparing for the v9.0 release, so I can’t devote too much time to it at the moment.
Sure, I understand. Could you maybe tell me about the other places where the engine sets stencil values when a highlight layer is active? I would then just try to hack around those for now. Any help appreciated!
The PR adds two new properties to the highlight layer:
stencilReference. Let’s you get the stencil value used by the layer, so you can factor it with your code
numStencilBits. The number of bits the layer should use. It’s 8 by default, but if you set it to 2, for example, then you can use the lowest 6 bits of the stencil reference value for your own use.
So apparently, I still haven’t understood fully how the highlight layer works. Till then, I can’t judge if your proposed change would solve my problem.
If I only use a single highlight layer, what stencil values can I expect at the end? If I check ThinHighlightLayer.GlowingMeshStencilReference and ThinHighlightLayer.NormalMeshStencilReference I would expect that all pixels belonging to a highlighted mesh should be 0b00000010 and all other pixels 0b00000001. But one thing confuses me here already: NormalMeshStencilReference seems to be applied only for meshes that were registered on the highlight layer via addMesh or addExcludedMesh. What about all other meshes in the scene? Does the stencil buffer stay 0b00000000 for them? Why do we enter sth in the stencil buffer for excluded meshes but not for all the other meshes of the scene?
The example you sent doesn’t exactly cover my case, I think. A bit more details about my scene to be able to imagine better: I have a room that is looked into from the outside (back-face culling removes the faces on the camera side such that you can always look into the room). One side of the room has windows with reveals (four interior sides of the window). Now when I look at the room from the window side, back-face culling doesn’t suffice and I have floating window reveals. That’s why I went to using the stencil buffer. Before rendering the reveals, a quad is drawn for every window area into the stencil buffer. Afterwards, the window reveals are rendered with a stencil test for these values on. Now if I look at the room from the window side, the quads are culled, nothing goes to the stencil buffer, and the reveals are not shown. Here some example pictures to explain:
But if I highlight an object behind the wall, the highlights are interrupted by the window reveals in the front, even though they shouldn’t have been rendered (and obviously they at least don’t make it to the color buffer):
In my understanding, for the highlight layer all meshes are rendered again (in ThinEffectLayer). And maybe there, there is no stencil test for custom stencil usage, so the reveals do get rendered and occlude the highlight effect.
GlowingMeshStencilReference is used for meshes added to the layer by addMesh
Meshes added to the layer by addExcludedMesh are not rendered to the stencil buffer
All other meshes are rendered to the stencil with NormalMeshStencilReference
Note that we are speaking of the rendering of the meshes to the scene.
Regarding the rendering to the layer texture (that will be composited later on with the scene), it works like this:
meshes added to the layer by addMesh are rendered to the layer (either with the fixed color passed to addMesh, or with their own material, depending on the settings)
meshes added to the layer by addExcludedMesh are not rendered to the texture
all other meshes are rendered with a (0,0,0,0) color so that they don’t participate in the highlighting BUT still write to the depth buffer, for proper culling with highlighted meshes
I think what’s confusing in the code is that:
the check if (this._excludedMeshes && this._excludedMeshes[mesh.uniqueId]) in the addMesh method has no meaning, I don’t see how it can be true in that place…
to render meshes not added to the layer, we rely on the fact that the stencil value is reset to NormalMeshStencilReference when a highlighted mesh has been rendered (the _defaultStencilReference call in mesh.onAfterRenderObservable). So, it is the value used if the next rendered mesh has not been added to the layer, because it’s just the current value.
I’ll discuss it with the team to make sure I’m not missing anything, and we’ll simplify the code.
Yes, meshes are rendered again in the layer texture, that’s why you have to enable the stencil buffer for the layer, by passing generateStencilBuffer: true when you create the layer (see my post above).
@Evgeni_Popov Thanks again for the detailed response!
So you say other meshes will get NormalMeshStencilReference because it is set after a highlighted mesh is rendered. That requires a highlighted mesh to be rendered first in the scene doesn’t it? Is that ensured?
Another question: Why do we even set the stencil to 1 for non-highlighted meshes? Wouldn’t it suffice to just keep it 0?
And I think this must be the central part for my problem. It seems like the window reveals are rendered in this pass even though they don’t pass my custom stencil test. I wonder, if maybe stencil testing is manually disabled or sth during this pass?
I tried this now. Unfortunately, it didn’t fix the problem. Btw, the fix should also adapt sth here, because generateStencilBuffer is not part of the HighlightLayer constructor types and can thus only be passed by using any. I also didn’t find another way of setting it.
I’m gonna try to create an example now with the link you sent.
No, that’s a weak point of the code, but after the first frame, the current stencil value will have been set to 1 at the end of the rendering, so it will be the right value if the first mesh to render in the next frames has not been added to the layer.
Yes, I think using 0 would also work, I don’t know why we use a specific value. Something to discuss with the team too.
I updated the PR accordingly.
Thanks! It will be easier to understand what’s going on.
Ah ok, so the stencil of the layer is not reset every frame? That might explain sth that confused me while debugging, which is a certain stencil state only flashing for a single frame when starting the app…
Thanks!
So I created an example now, and your fix does seem to work. Here is the example without the fix: Babylon.js Playground and with the fix: https://playground.babylonjs.com/?snapshot=refs/pull/17934/merge#00PM1W#55. In the example, the alien also kinda looks through a window (plane in front of it, created first in the code). For the window, I added sth like a frame in the middle (second plane, perpendicular to the first one). This second plane should only show up when looking through the window from the alien’s perspective. When looking from the other side (i.e. facing the alien), the second plane should be invisible. This works. But in the current engine version without the fix, the highlight of the alien gets interrupted.
Great that your fix solves it! When do you think it will be available in a new Babylon version? Should I include your fix as a hack on my side for now? Btw what did you think about my a bit more sophisticated approach:
For now, I’d also go for a quick solution. But long-term, my proposal could maybe be a safer solution?
I’m going to discuss with the team today. If it’s ok, it should be merged quickly!
I like it!
However, it’s a bit more involved, and we currently have only one component which uses the stencil buffer (the highlight layer).
I think we will revisit this issue once we have another component using the stencil buffer, to have a better view of what’s needed and how to implement it.
But yeah of course, maybe it’s sth for another day.