There was a new feature added that allows using the controllers more like the free camera experience, instead of the standard teleportation feature. You can read more about it here:
WebXR Selected Features | Babylon.js Documentation (babylonjs.com)
It’s only available in v5 alpha.25+. Please post any questions and also look forward to any ideas for improvements - here is a PG that I used for testing (same as the docs page):
XR Movement Feature | Babylon.js Playground (babylonjs.com)