Custom highlightLayer when the sign of the scaling vector is different between an instance and the root mesh

[KevinWu]

I use instance buffer and custom material to distinguish between instance mesh renderings with the same source mesh. Everything was going well until I had the following situation: he sign of the scaling vector is different between an instance and the root mesh. Looking through the documents, I learned that Instances with a world matrix where determinant is different than root mesh world matrix will be rendered separately (like a regular mesh).

According to my understanding, although this instance mesh will use a separate draw call, it should share materials with the source mesh?But in the effectLayer, I see that the instance mesh seems to use a different effect, which seems to be causing some problems with my rendering results.

I expect only the leftmost sphere to have the highlighting effect，but now，the third sphere also has the effect of highlighting…(Forgive me for not writing the STENCIL BUFFER code for convenience reasons

PG: https://playground.babylonjs.com/#HRBTTJ#2

[KevinWu]

class: core/src/Layers/effectLayer.ts
method: renderSubMesh

renderingMaterial here is undefined when effectiveMesh is an instance mesh

[KevinWu]

but I used a custom highlight material

[Evgeni_Popov]

The fix in this case is to add the instance(s) that have a determinant different from the determinant of the master mesh to the highlight layer:

Babylon.js Playground