Are there any examples or tools out there that demonstrate using React to create, manage, and render a user interface onto a 3D Plane inside of Babylon?
Specifically need to create some user interfaces that can be used in XR (HMD). It would sure be nice to use some common web tools like React and Semantic UI.
There is a great babylon-react project made by the hero @brianzinn - GitHub - brianzinn/react-babylonjs: React Components for BabylonJS , which might be a good starting point for html GUI.
Having said that - it will sadly not work in XR. XR gets the canvas only, and nothing more. to create UI in XR you will need to use a UI lib that actually renders the items on the canvas using 3D objects inside your scene. The Babylon GUI is a great starting point for that -
https://doc.babylonjs.com/how_to/gui
and
Thanks for the reply. That is what I was afraid of. I have use React extensively as UI around and overlaid on my Babylon canvas, which works great for a usual, non-XR experience. I was hoping I could just reuse that code but render it inside Babylon when the user flips to an XR experience, thus maintaining the same feel.
I’ve had some success with using puppeteer server-side to render react + canvas to render image for VR display - you could use that only as a non-interactive XR/VR surface unfortunately. The react-babylonjs library supports then declaratively putting that image on a 3D gui plane. I made a game in React and when I wanted VR then I had to rethink everything - I put all UI into 3D planes using GUI declaratively and it worked out, but I didn’t have all the niceties like Semantic UI or custom controls/css.
I suspect that you are after an interactive one and you would need to look at html to canvas for that - I’m not aware of any advances in that area. I think if you asked separately about rendering html to webgl that the solution working backwards from there would turn out to not be the hard part 