I would like to remove the overlapping(intersected) faces.
For the following image;
Boxes are intersected. I want to delete the small box’s all sides that intersected except the marked face which is not intersected with the big box.

To achieve the effect I think I understand of eliminating the intersecting faces between the two boxes, you will need to perform boolean operations on the meshes of the mesh.

In what I know of Babylon.js, I haven’t found the ability to do built-in boolean operations for mesh intersections directly, but you can use a combination of approaches, including custom shaders or external libraries like CSG (Constructive Solid Geometry).

Here is an example of using the CSG library. I don’t know if that’s what you were looking for, but I hope it helps:

Meshes use triangles. If you are looking for faces, that has to be defined somehow for the mesh, as it’s not clear what a face is for a generic mesh.

Assuming we can associate a triangle with a face somehow, then a brute force way is to intersect the triangles of the small cube against the triangles of the big cube. If they intersect, then remove it along with associated triangles of the same face from the small cube. If not, then you keep it.

I don’t see any code in Babylon.js that is available to call directly that gives you this Boolean operation of whether two triangles intersect. I found one based on three.js here that maybe can be ported.

I don’t know what you mean by checking vertexes. A triangle-to-triangle intersection in 3D space will certainly involve checking vertexes. Maybe you mean if there is a way to check if a vertex is inside a volume? That should work, but I’m not sure if the algorithm is simpler than a triangle-to-triangle intersection in 3D space. If your scenario guarantees everything is a cube, then these checks are easy.

When you delete the 4 vertices of the blue box inside the green box woudl this not leave the desired face? So, iterate vertices (world space!), do point versus box check, if point inside box, delete?

Seems like it would except if a face is not defined as a triangle, in which case my other point stands. Also, the point inside box might be point inside mesh. Other than that, I don’t see why it wouldn’t work.