Great question! I was waiting to answer that one 
XR’s session manager requires a scene in its constructor, which is required to generate the resources needed to start an XR scene. This is a limitation in Babylon’s engine as well (as the render target texture used to render the scene is also scene-dependent). Apart from that there are the assets for the controllers, hands (if enabled), that are all scene-dependent.
We had a discussion once about multi-scene support, but I never dug into it too much. I will be happy to spend some time and see if we can separate XR and the current scene, but that won’t happen in the near future. Until then I would recommend using methods like the assets container to manipulate a single scene with different asset-groups to simulate multiple scenes