The issue here is the mouse picking of this complex scene. Babylon runs a picking function on each frame, which uses your current mouse position over the canvas. when dealing with complex meshes it might take quite some time, because it runs picking against the actual triangle of each mesh it collides with (or better - with its bounding box/sphere). To eliminate this you can set each mesh you loaded to be not-pickable (or at least the more complex meshes).
isPickable is set to true per default so you will need to actively set it to false.
Now, saince you are using the emulator, the mouse events are still executed on the canvas. this is why the FPS drops even when you are in XR. when outside of the emulator you won’t have the mouse issue, BUT you will still experience the same behavior when moving your xr controllers. that it because the xr controllers are emulating the pointer system! And they technically work the same as a mouse, but in the 3D space (as opposed to the mouse that works on the 2D canvas and always points “forward”). To disable this behavior you can detach the pointer system (which is part of the default xr experience), or, just like in the case of the mouse, make all meshes not-pickable (line 9, a very simple approach that you might want to adjust in your scenes):
Hill Valley in WebXR using Babylon.js | Babylon.js Playground (babylonjs.com)