Hmm, would overriding scene.getActiveMeshCandidates make sense for your use case? It returns scene.meshes by default but you could return the subset instead. Babylon does something similar internally.
Comparison: https://playground.babylonjs.com/#BOGC77#2