Create a hole in the wall effect with WebXR

I’m trying to achieve a similar effect using Babylonjs as shown in the screenshot below, taken from a page with three JS-Ar examples (Three.js and AR.js - examples).

Based on this playground I’m looking for a way to create a box under the tracker to position the model inside the box but in the ground. But I have problems with the visibility of the box walls.

Screenshot from 2023-08-30 12-41-43

I tried backface culling, but did not create the right effect.

Thank you in advance for any suggestions or hint to the right direction. :hugs:

Welcome abroad! This was a fun one to try to make, through I always struggle with stencils :sweat_smile: But you can do it like this :slight_smile: Stencil hole | Babylon.js Playground (babylonjs.com)

3 Likes

Thank you very much for your quick response and solution.

I could not finde any documentation referring to stencil buffers in Babyonjs, looks like this is
a very advanced topic :sweat_smile:

I reckon we need to write some :slight_smile: Can I make an issue on the docs to remind us? @Evgeni_Popov @PirateJC @PatrickRyan

Sorry, but I am stuck again. The final goal is to have an AR scene with image tracker. The tracker marks the hole in the wall. Inside the wall I want to place other object with animations and so on. With your help I have finaly the hole but as I understand, the hole is more of an optical illusion, right? So it is not realy a hole in 3D space, where I could place other object.

Yes, using the stencil buffer is a trick. If you really want to create a hole in another object, you may want to use CSG instead:

In your case, you will want to use a subtraction:

Be aware that CSG can be slow, if your meshes have a lot of faces.

@_Mike, the other way to think about this problem is a more simple, direct route of faking it. If you are in an AR space, you already have a texture from the camera feed to work with. If you create a separate piece of geometry that has a hole in it that is large enough to cover the open cube, you can just create a shader that applies the camera feed to it in screen space. That way you have geometry sorting out visibility of anything inside your open cube and can have multiple meshes inside the cube playing animation or whatever. You could also move meshes into an out of the open cube in real time making it appear that there really is a hole in the wall.

Other than the shader displaying the camera feed in screen space on the mesh, there is little overhead for this kind of approach. It’s fake, but it’s an inexpensive fake since you have a clean camera feed already and there is no need for additional render textures.

@Evgeni_Popov Maybe I did not understand your suggestion, but there is no real mesh to subtract from, as the “ground” is the real surface from the AR camera. I need to hide the cube (hole in the wall) from the side perspectives. Tricky :thinking:

you could create a fake almost flat box to use in CSG substract to act as a blocker if you position it “on” the wall ?

@sebavan Sorry but I still don’t get the part where I substract :sweat_smile: But meanwhile I found a playground, which comes closer to what I need. Still don’t know if this works in AR. :crossed_fingers:

Something like this: Babylon.js Playground

where basically you have an occluder

that wont be visible

1 Like

@sebavan Thank you for your suggestion! In regular 3D view this looks exactly like the effect that I need. But unfortunately it looks distorted in AR view. :worried:

cc @RaananW as it should work like any other xr experience. It does not rely on anything special :frowning:

So, a few things :slight_smile:

First - in XR units are in meters. so a hole the size of 4 units would mean 4 meters wide. That’s probably your entire device screen space, in this proximity.
Second - you missed 2 things from the original PG - first, the hole walls should be moved down, otherwise it looks odd (the walls are above the ground), and the second is setting the occluder to visibility 0.0001 - visible, but as little as possible. And a small note - making the occluder mesh only 5 by 5 will prevent the effect from working. Make it 50 by 50 or even more.

I added an arc rotate camera to the scene so you could see how it looks like before

Also note that you are setting the root position to be the position of the marker. I can only assume you are using the marker on the floor? Try first looking at the floor without a marker - you will notice it is working as expected already:

This is the PG i am using (didn’t try with the marker)

2 Likes

Thank you very @RaananW for your reply! I have to confess, that I still don’t get the occluder magic :sweat_smile: On the floor your PG works very well. Unfortunately I am trying to achieve the effect on a vertical wall. Using the tracking marker, the cube is place on the right place, but the effect breaks.

you will need to place the occluder differently. Instead of a ground, you will need to rotate it. For example:

Image Tracking Demo | Babylon.js Playground (babylonjs.com)

This of the occluder as a mesh that doesn’t render anything that is “deeper” that itself. Setting its visibility to 0.0001 make it almost transparent (or - transparent to the human eye? :slight_smile: ). So this mesh will occlude everything that comes after, which is exactly what you are trying to achieve.

Now I think I understand the occluder magic, thank you for the explanation :smile:

I tried your updated PG with the rotated ground. Now standing right in front of the demo it looks like a hole in the wall. But looking from the side it still looks like the screenshot above. I it is possible, could you try your PG with the image marker, and see if it works? I am starting to feel crazy and annoying. :crazy_face:

Oh, wow, that was a lot of fun :slight_smile: . Took a little while, but a lot of fun nonetheless!

there are a few reasons for that - first, the occluder doesn’t have the root as a parent. Second, the scene needs to be prepared in 0 rotation and 0 position, if you want to just apply the transformation of the marker. Otherwise you will need to transform the scene correctly.

Anyhow, try this - Image Tracking Demo | Babylon.js Playground (babylonjs.com)

This is a screen capture of my test device (that had this silly out-of-focus issue, which made tracking fail for a bit). Still, you can see the effect, and how awesome it is. gives me so many ideas!! :slight_smile:

2 Likes

This is super awesome, exactly what I was looking for!
Thank you for all the responses, this forum is really a great support.
:smiling_face_with_three_hearts:

1 Like