Because I thought your question was about picking, not rendering. No idea if you can use the stencil buffer like that in Babylon.js
This guy: YouTube (linked from your github link) uses render targets as well (at least according to his video). He claims the effect he’s after is not possible in Unity, probably because of the non-euclidean aspects.
Here is a tutorial with portals and render textures in Unity where it works: YouTube But he’s not doing anything non-euclidean of course.
I mention Unity only because it’s comparable to Babylon in its abstraction level, so most stuff that is possible in Unity will be in Babylon.js as well.
The non-euclidean idea is really cool. You could make a highly surreal game with it. I hope you find a solution because I’m looking forward to your game