Ok looking into it further, I am not able to repro outside of VR.
The camera needs to be at the center or close to the center of the sphere relative to the sky sphere size.
In your example @RaananW you have 1500 unit diff between both center so looking right or left you would be either close or far from the wall dislplaying the video changing the perception of zoom level.
dome.position = new BABYLON.Vector3(1500, 0, 1500);
vrHelper.position = new BABYLON.Vector3(15, 0, 15);
Actually @johannesmueller you have the same issue by offsetting the view position of 1.7m, you would be closer from the top than the bottom probably resulting in the feeling of zoom while looking up.
The only position that do really work is camera at exactly the center of the dome which was the case whilst filming.
So you if you move the camera up you should try to move the dome up ?
I guess… 