PBRCustomMaterial: compiledEffects memory leak

Hi again,

Still hunting memory leaks in our app, after the fix from my previous post ( PBRCustomMaterial: ShadersStore memory leak on dispose() ) has been merged. They both come from the way the entry key in caches is computed for PBRCustomMaterial.

The problem

When you mutate a property that changes shader #defines on a PBRCustomMaterial at runtime (e.g. transparencyMode, albedoTexture, backFaceCulling…), BabylonJS recompiles the shader and adds a new entry to engine._compiledEffects. The previous entry (compiled with the old defines) is never removed. It becomes a permanently orphaned Effect.

This happens because PBRCustomMaterial builds its cache key as custompbr_<uniqueId>+<defines>. When defines change, both the old and the new key are unique by construction and will never be reused. dispose(true, true) only cleans up whichever key is current at the time of disposal => the orphaned one stays forever.

Interestingly, this also occurs with PBRMaterial, but is bounded at most one orphaned entry per define combination, since the key doesn’t include uniqueId and can be reused across instances. With PBRCustomMaterial, the uniqueId in the key guarantees a new orphan on every cycle, making the leak unbounded.

Reproduction

Playground: Babylon.js Playground

Click the button repeatedly and watch _compiledEffects grow by 1 on every click.

_compiledEffects is logged:

The pattern that leaks on each cycle:

// 1. Material is created and rendered — Effect A compiled
const mat = new BABYLON.PBRCustomMaterial("pbr", scene);
mesh.material = mat;

// 2. Define-changing property mutated — Effect B compiled, Effect A orphaned
mat.transparencyMode = BABYLON.PBRMaterial.PBRMATERIAL_ALPHABLEND;

// 3. Material disposed — Effect B removed, Effect A stays forever
mat.dispose(true, true);

Thanks for looking into this!

Thanks for the detailed repro! We have a fix for this now.

The PR will be created once this larger PR has been merged first: Tree-shaking - the pure barrel by RaananW · Pull Request #18441 · BabylonJS/Babylon.js · GitHub

PR has been created:

Thanks a lot for the quick fix !

Hello @Evgeni_Popov
I may be wrong, but since your fix is included in the last BJS release (9.9.0), I expected that previous playground won’t show leaks anymore. It seems there’s a remaining issues about effects leaks as countor still increases.
Thanks a lot !

Thanks for checking 9.9.0. You were right: there was still a remaining cleanup path.

The first fix released the previous custom effect when a new one replaced it, but the effect cache can return the same PBRCustomMaterial effect more than once and increment its ref count. On dispose we were only releasing one reference, so the unique custompbr_* key could stay in engine._compiledEffects.

I opened a follow-up PR that tracks the effects generated for a PBRCustomMaterial instance and force-releases those custom effects when the material is disposed: [Materials] Fix PBRCustomMaterial cached effect cleanup by Popov72 · Pull Request #18493 · BabylonJS/Babylon.js · GitHub

Thanks again @Evgeni_Popov , it works like a charm with your fix in 9.9.1 ! :ok_hand: