This is done by the system to see if some meshes are pickable and need to trigger their actions (onPointerMove).
This is the predicate used to select if a mesh needs an action:
Babylon.js/scene.inputManager.ts at master · BabylonJS/Babylon.js (github.com)
So if you have a lot of meshes in your scene the system will go through all of them on every pointer move.
I will introduce an option on the scene to skip that feature entirely if you do not need it