An orthographic camera does not behave like a perspective one. When you zoom you are not getting the camera closer as the position is defined by the orthoXXX properties
I would recommend to disable the radius control (inherited from the arcrotatedirectly) and instead do your own changes when zooming:
camera.inputs.attached.mousewheel.detachControl(canvas)
something like that:
var zoomScale = 1;
scene.onPointerObservable.add(p => {
var event = p.event;
let wheelDelta = 0;
if (event.wheelDelta) {
wheelDelta = event.wheelDelta;
} else {
wheelDelta = -(event.deltaY || event.detail) * 60;
}
zoomScale += wheelDelta / 5000;
camera.orthoTop = 100 * zoomScale;
camera.orthoBottom = -100 * zoomScale;
camera.orthoLeft = -100 * zoomScale;
camera.orthoRight = 100 * zoomScale;
}, BABYLON.PointerEventTypes.POINTERWHEEL);
Demo: https://www.babylonjs-playground.com/#KUM5WC#27