Yeah but this is inside a this.preventDefaultOnPointerDown
check so this should be controllable:
https://github.com/BabylonJS/Babylon.js/blob/master/src/scene.ts#L2045
same for up:
https://github.com/BabylonJS/Babylon.js/blob/master/src/scene.ts#L2089
Did you try to set both up and down booleans to false?