HTML + CSS to texture in the browser

Firstly I’ll prefix this by saying the babylonj.js GUI is amazing, so certainly that has it’s place and is better integrated with controls and interaction, but let’s say we just want to generate images of HTML and drop them in our scene. Maybe even this image could be an underlay and we get flexbox/gradients/etc with GUI controls on top.

Seen some questions lately regarding how to put HTML content to textures especially in XR where we don’t have access to DOM. I used to recommend using headless chromium server-side - I did a talk on that a few years ago (github.com/brianzinn/crapdf: react pdf render puppeteer). I also added xr-dom-overlay support to babylon, but that has limited usefulness as well. I wanted to bring HTML right into a texture in the browser without hitting a server (by the way I’m pretty sure this will work native)…

It’s cool to use DOM on web and do CSS transitions to make the HTML + CSS appear in the scene (and for Video this makes more sense - see the CSSRenderer posts here in the forum), but I came across a (new to me) highly cool library from vercel vercel/satori: Enlightened library to convert HTML and CSS to SVG (github.com) that let’s us take HTML straight to our immersive scene.

Check out the demo here (brianzinn.github.io/satori-gl/).

Use the dropdown to choose different HTML that we convert to SVG and load in the scene on the fly:
image

Pictured here is the Call to Action with Tailwind:

all source is here: satori-gl/App.tsx at main · brianzinn/satori-gl (github.com)

NOTE that the JSX is optional you can go from JSON representation.

ps: use Chrome - I’m missing a polyfill at least on Firefox ( Intl.Segmenter).

@brianzinn , this is super interesting. Am I right that we shouldn’t expect the buttons to have interactivity in your demos? The rendering of the UI looks great, but not sure anything is interactive or what the path to that would be. We’ve also looked a bit into Skia and some others.

3 Likes

You are exactly right about the interactivity and it is a huge drawback - that’s what I was alluding to with using Satori as an underlay with potentially transparent background GUI (buttons/sliders) in the forefront. With CSS positioning (even tailwind + flexbox) mostly working should be able to work the two together. It’s a bit of a hack in some regards, but I think potential lies there to even perhaps bring CSS into babylon.js GUI if somebody really had the inclination.

1 Like