This is an expected behavior as the hover is used to also change the cursor so you know that there is something you can click there.
To change that behavior:
https://playground.babylonjs.com/#J2SJEZ#1
See line 6-8:
scene.pointerMovePredicate = function(mesh) {
return mesh.isPickable && mesh.isVisible && mesh.isReady() && mesh.isEnabled();
}