HTML Content in BabylonJS Scenes

Interesting. Were you frame grabbing the HTML on the page and then rendering it to a video texture? The basic problem that we have to solve in XR is how or where do I render HTML so we can frame grab it and push it to a texture. If we don’t have overlays, then we have to do it headlessly somewhere. As I see it we can either do it on the users device (using a companion app) or off the user’s device. The latter gets costly as we have to run a headless browser instance for every concurrent user session, though I have some ideas for some ways to reduce that. The former will be a non-started for alot of users. After all if they were using WebXR they probably have no interest in downloading an app to support it. So unless I’m missing something, that’s basically where we are, short of the Three.js HtmlMesh rendering raw DOM text to a canvas using html2canvas or similar, that are just too limited to be broadly useful.

Yes. Kinda. Can’t we use an offscreen iframe to capture the content? Just guessing, I never played with babylon.js in VR mode so I don’t have a clue what are the possibilities here :frowning:

I had not thought of an iframe. I’d be shocked if we can use an iframe in webxr (after all, if I had a DOM I could add content to I wouldn’t have the issue in WebXR in the first place), but it is worth a quick test.

1 Like

Makes sense :crazy_face:

LOL, I just noticed we’re working on similar things. I recently created babylon-html npm package (though it’s more intended for webxr). Been looking at extending to add interactivity to DOM elements, IDK if you’d want to collaborate. I took a slightly different approach in that I (right now) render html passed as a string onto a plane (which works in immersive webxr experiences). Also added hover and pointerup/down states so you can have different css render different things as folks interact with the item.

Lemme know if you want to rap!

1 Like

Originally it was just to make buttons more easily with existing skillset (html css) versus using current GUI system, but now I’m debating on expanding and doing things like interactive forms (which can be done with existing solution, it would just create more meshes to accomplish). Ideally I’d like to be able to have designers copy/paste pretty html elements into a babylon file (or something like that) and have it show up in the experience with very little effort. Styling the existing UI components is a PITA.

Thanks. I’ll DM you.

Thanks to @ckeiner, HtmlMesh now works in right hand coordinate systems! Grab the latest NPM package if you need HtmllMesh in a scene that uses right hand CS.

Will this not work in a webxr capable browser, e.g. Quest Browser?

It will not currently work in webxr immersive augmented or virtual reality, because there is no dom layer on which to render the HTML. There are some solutions that will work in webxr but they rely on rendering the HTML to a canvas at some frame rate using 3rd party libraries. These may work for some applications but don’t fully support everything you can do in HTML and CSS and don’t work with iframes which is what is needed to stream video from a site such as YouTube. The only comprehensive solution for immersive-ar or immersive-vr is a streaming web browser which requires either a service (expensive) or a companion app (which defeats a major reason for using web in the first place (to avoid native app installs).

XR running through a browser doesn’t have access to the browser?

It can’t access the DOM, or at least it can’t render the DOM. DOM Overlay Layers would, if supported, allow rendering of dom content in front of the XR scene, however it would not allow rendering of dom content “in” the XR scene (so that HTML content could be occluded by objects in the scene) because that actually requires rendering the DOM behind the XR canvas.

1 Like

Huh. I was figuring using this method for menus in XR. Guess not.t

You may be able to leverage an approach similar to the one used here. Basically, he’s using a dom-to-image package to render dom content to a texture on some subinterval of scene render (so it works with animations). Be advised, this won’t work in Safari, but then again, neither does webxr, so that isn’t really a new limitation.

2 Likes

I don’t get to use the menu buttons on the xr controllers anyways

last I checked that was AR only. OP is discussing controllers, so I don’t think applies here.