Please explain it like I was 5 - Stencil testing / Masking / Rendering Groups

Hi folks! I have been playing around with a scene which is sort of a protype for an effect I would like to use on my own personal blog site. The idea is to have some small “Portals” which act as doorways to the various sections of my site. Each cube will contain distinct meshes and backgrounds and I will overlay an HTML link on each portal. The plan is to have the scene remain even during site navigation as a sort of background with content layered on top of it in a synchronized way, (I will load in content via turbolinks.js). When someone clicks on a link I will make the babylon camera zoom in and enter the corresponding cube and the assets within it will serve as that pages distinct 3D background. I will have maybe 5 or 6 of these cubes in the scene. Outside the cubes will be some other geometry, not decided on that yet.

Anyway. Before I can acheive this effect I really need to develop a good understanding of how the various stencil testing functions work. I found some examples, but not enough for me to really understand how stencil masks and rendering groups etc work together. I can get close to the effect I need just by clumsily copying code from other playgrounds, but not close enough. I need to really understand well how these stencil functions work. I don’t have a background in 3D so am useless with shaders currently, but hope to learn it all!

Here is a playground link:

The idea is that the “portal” can be any transparent mesh, (I probably wont stick to just using simple cubes, they might end up being stone shapes made in Blender…). I want the portal mesh to be able to use the skybox cube texture. I was also experimenting with refractions for the inside faces of the cube, but can’t figure out a way to achieve this in a performant way yet. I tried creating two cubes and hiding the backfaces of one and setting the other to use a reflection cube texture, but I couldnt get it to look decent on a mobile with only 512px for the size, so am abandoning that idea for now and just having the box be transparent with no refractions.

So what am hoping is that some of you wizards will find the time to explain to me like I am 5 yrs old exactly what’s happening in the “PORTAL CODE” section of my playground which I mostly just copied over from another playground. The result I want is for the “portal” to have reflections from the scene’s skybox and for the “ground” mesh, (the big sphere), to not be visible inside the cube though it intersects the portal’s “inner” geometry.

I think I understand the basic idea of rendering groups, they are like layers, the default one is 0, higher ids will mean the layer is painted over objects in lower rendering groups regardless of depth. Right?

Things I don’t understand and can’t find documentation on:

“scene.setRenderingAutoClearDepthStencil(1, false);”
the API docs say that this function “Specifies whether or not the stencil and depth buffer are cleared between two rendering groups.” What exactly does “between two rendering groups” mean? The default one, 0, and the one referenced as a parameter?

There is not much info in the API docs about the following functions:


I can’t see a guide to using them. I can guess at what they do, but would really appreciate an authoritative explanation so I can properly understand. Am assuming this is not something that you were expecting beginners like me to mess with.

Am guessing I need to add the ground/sphere to its own rendering group and then maybe somehow create a second stencil mask which only discards the piece of the ground mesh which is visible within the portal. Am just new sure how to do it. Am guessing that I will get the cube texture reflections to appear on the portal cube once I have added it to the right rendering group. Just not sure which one.

If I can understand these things well, with your help, I would like to write up a small guide that could be added to the docs, as I think these techniques are probably more useful even to beginners than you might have realized. I might even create a demo playground scene with some useful utility functions that folk can use to organise their all their meshes and scenes into “layers” easily.

Any guidance at all would be appreciated!

P.S. I think the forum could maybe make more use of the tags feature. Maybe am looking at it wrong, but there only seem to be a handful of tags. I can see the use for more detailed tags such as “stencil” or “masking” etc

1 Like

Here are 2 of the best references I used to learn stencil and depth operations:

Hope that helps but here it means you sill need to render about 7 scenes composed with html. I am afraid your perf would not be what you expect mainly due to the composition with html.


Thanks for the links! I will read them tomorrow, (going out to a film now).

I don’t think I will need to render multiple scenes, the html is just going to be an overlay on top of one scene. I will make sure each portal’s “internal” geometry does not overlap, they are only small boxes, so I can have them all exist in the same scene. In other words, the portals are not “real” portals to another scene, it will just look a bit like that. I am just syncing the movement of the camera in this one scene with the html that gets overlaid. Hope that makes sense :grin: Anyway I will post a playground with the finished project at some point so you see what I mean.

Thanks again!

1 Like

Rendering manager renders rendering groups in ascending order.
By default, it clears depth and stencil buffer before rendering each group.
In this case, I see rendering groups as Photoshop layers.

If manager is told to not auto-clear depth and stencil buffer for, say group 1,
then depth and stencil info. of group 0 are kept during rendering group 1.
In this case, I see rendering groups as collections.

E.g. meshes in same group

Meshes in different groups (uncomment line 11 to 12)
groups = Photoshop layers

Keep depth (uncomment line 14)
groups = a way to organize meshes, which won’t interfere with rendering.

Stencil is a region for testing if a location is drawable or not. :joy:
The 3 functions you mentioned are configuring the region (kind of)


which works like this

(R & M) F (valueAtLocationInBuffer & M)     // = pass


// (1 & 0xff) equal (valueInBuffer & 0xff)    // pass 

The setStencilMask is a mask to determine which bits are updatable in stencil buffer.


engine.setStencilMask(0xff); // the 8 least significant bits are updatable

You’re probably interested in operating the stencil buffer for below cases

// case: outside region(stencil fails)
// case: inside region(stencil passes) and obscured (suppose depth test is LESS, fails) 
// case: inside region(stencil passes) and not obscured(depth test passes) 

According to the gl pipeline, stencil test comes before depth test. If the stencil test fails, subseq. tests, including depth are skipped. Thus, there is no “setStencilOperationFailDepthFail”.

E.g. Sphere in a magic box

E.g. Scan enermies behind buildings


Thank you so much for the explanation! Am starting to get it. :joy: Am a bit slow tbh so thanks for bearing with me!

Sorry to barge in like this on a problem that is maybe already solved, but I’d like you to consider moving away from HTML and rather use the Babylon GUI. It’s maybe more cumbersome/unfamiliar, but there is a huge performance benefit to it.

I speak from experience. We made some prototypes for our RPG menus (inventory etc.) as HTML overlays and any time there was a DOM reflow we had massive frame drops (from 60 FPS down to 1 for about a second on Chrome and Edge, even worse on Firefox). We could not even deal with that using requestAnimationFrame for large updates; that just dragged everything out to two or three seconds with 2 FPS. The amount of reflows is not such a big factor as the fact that any reflow occurs at all. Why, I have no idea, you’d have to ask the browser developers. Unlike Babylon’s native GUI, you also can’t use Web Workers for DOM updates, so they block the main thread every time.

We are using the GUI now and everything runs extremely smoothly; the performance impact is not even perceptible. Maybe your use case is slightly better because you only use absolute/fixed positioning, but still I wouldn’t risk frame drops like this.

1 Like

Oh dear, that is something I hadn’t considered. I had assumed that as long as I didn’t reload the page that a canvas element would happily carry on as normal. I will do some tests on that today. I don’t think the GUI would be appropriate for what am building, so if it doesn’t work out I will just have to abandon the idea of having babylon sit in the background of my site. Thanks for the warning! I will check it out.

Stupid question out of curiosity: Why not? According to the README of Turbolinks you can just call


which should also work in a babylon callback. The only drawback of course is the frame drop when the actual content comes in, but that can’t really be avoided except if you build the entire site in babylon.

Styling-wise the GUI is pretty flexible, it does work differently from CSS, but almost anything is possible.

1 Like

Oh no, that isn’t a stupid question, am a total beginner to babylon, so I could making all kinds of stupid assumptions here. The reason I assumed the GUI would not be any use is because the type of content am thinking of overlaying has nothing to do with the Babylon scene, it’s not a control or anything. This is not for a game or app, the scene will literally just be a background to a blog basically. The only link between the scene and the HTML will be that I will be trying to synchornize camera movement with navigation around the site. I will be wanting to style the HTML with CSS and have the same level of accessibility for the content that you would have on any other blog. As far as I understood it, the GUI would not suit that purpose well, because it can’t be styled via CSS and it is not accessible to the same degree as standard HTML. Is that right?

1 Like

OK I think I get what you are saying, but to maintain accessibility I think you have to offer an auxiliary, HTML-only navigation anyway. I have no idea how screen readers actually react to links being jumbled around with absolute positioning, but possibly not very well.

I would just play around a little bit and see if it fits your requirements.

1 Like

Aye, as you say this will require a lot of playing about. I was thinking of augmenting the onscreen navigation with content inside the “subDOM” of the canvas element itself. Another thing I will have a play around with today. Thanks for the advice, this is certainly make me think more clearly about this “plan” of mine :smile:

Hey @ycw, any chance you can share the “BabylonJS Scan Behind” sample scene :sweat_smile:?

1 Like

This topic is exactly what I was planning to ask about but couldn’t even form the question properly. @Richard thanks for posting this and thanks to everyone else for responding. There is lots of good stuff here and even though some of it is pretty intimidating, this thread just made my day. :crazy_face:


@sebavan any chance you could recommend good starter for someone who haven’t done any shaders before?

1 Like

Thank you @Necips

Just wrote a demo Babylon.js Playground

Blog updated.


Thank you very much @ycw and everyone else for helping!

1 Like