Your illustration show cubes but you say your meshes are complex.
I think you supplied a simple example to illustrate your need but actually you are thinking more complex?
I am also interested in this area, I would like it for my Babylon project.
I used to work as a developer on a major CADCAM software engine used under the hood in the majority of high end solid modellers and we had a hidden line removal algorithm. I am interested in hidden line removal in my project.
You might think “Great someone who knows about this” but hold your enthusiasm its been 15 years since I worked in this area and I am dusty and rusty and in addition we were working with stuff like bspline surfaces not triangular meshes, we had a topology structure way in advance of the topology ‘structure’ found in meshes, the module was a huge chunk of code developed over decades by a team of full time developers.
I am not sure that knowledge (or rather what I can actually remember of it ) will actually be that much help on mesh hidden line removal and in addition we were not dealing with planar mesh geometries - I am kind of tempted to play around, look at research papers and dabble, see what turns up.
In general the issues tend to be.
- Near coincidence - resolution issues - maths that breaks down due to precision issues.
- Brute force methods are the most reliable but slow.
- Optimisations often involve applying results calculated in one place to make inferences in another - downside tends to be error propagation which is why brute force methods have their fans.
- You do not need horrible triangles to encounter horrible problems - a large set of connected triangles that are nearly but not quite co-planar when seen ‘end on’ will generate resolution issues.
I do not mean this as a gloom and doom - its really just my two cents worth seeing your simple cube example and your statement that your meshes are complex.