Would you like to include HTML content in your scenes for instructional/educational purposes? Or stream video from YouTube, Vimeo, TikTok, Instagram, etc… in a scene? Or to allow users to fill out a form or poll in app? Or display an in-app browser, so a user can take some action, e.g. sign up for an account, purchase something, join an email list without leaving the experience? Or create a rich UI that can leverage all the capabilities of CSS and be driven by a CMS? Or maybe you just want to make a box where each side is a different site?
The HtmlMesh Babylon Extension is live and supports all of these use cases. This is Babylon’s version of ThreeJS CSS3DRenderer that uses CSS transforms to render HTML content in the scene with perspective and rotation to make it appear to the user to be in the scene. However, the BabylonJS version uses a depth mask to allow HTML content to be occluded by other content in the scene to increase the sense of immersion. It also has a pointer capture mechanism that allows users to instantly interact with the HTML content simply by moving the pointer over it.
Unfortunately, since this relies on the browser for rendering the HTML, this won’t work in native apps; the html content will not show if the app is run in a non-browser context.
You can check out the HtmlMesh in the BabylonJS Extensions repo and on NPM. Feel free to hit me up here or on github with questions or suggestions.
I have followed the examples and deployed it in a vue environment. It seems like it never is rendering in the correct size and I can not even see it unless I set the css-groups z-index to something really high.
Not really sure what we are doing wrong, but some help getting this working correctly would be greatly appreciated! We are excited to incorporate it into Frame.
Thanks for giving it a shot and for the feedback. I think your size issues are probably a misunderstanding caused by poor documentation on my part. The renderer will set a width and height for your element that matches the mesh aspect ratio and fills as much of the screen as possible. It will then apply a scale factor to get it to fit the mesh. I have found that scaling the content in this way provides the clearest text for the HTML content. For this reason, you need to specify the size of the mesh in BJS units when you call setContent. Once you’ve done this, you should be able to adjust the scale through mesh.scaling, however you will want to make sure that you update the size (mesh scaling * original dimensions passed to set content) and pass the updated size to setContent when you hydrate the html mesh from the server. You should not need to set z-Index of the HTML content directly (the renderer will overwrite it when you call setContent. Do you have a frame I can look at to help you debug the issues?
Note that one issue with the scaling approach is that it doesn’t take advantage of any responsive design by the site devs to make the site more functional on a smaller display (in our case the mesh). I plan to enhance the extension to use the dimensions of the html element so it will leverage any responsive design, but right now it ignores them.
Looking at the code, it seems that would happen if the onPointerMove function was picking meshes that do not have the capture behavior, but there is a predicate on the pick that prevents that. Without a reproduction it is hard to say. Does it happen everytime the pointer moves over the HtmlMesh? Or is it intermittent?
Currently it doesn’t, because I was focussed on HTML content that can be fully part of the scene, i.e. can occlude and be occluded by other elements in the scene; and that requires that the canvas be fully transparent in way of the HTML so that the HTML comes through. However, I think that is a simple change to have an option to not use a depth mask and just set a z order that brings that content in front of the canvas. Such meshes would always appear on top of everything in the scene regardless of z value, but would still have rotation and perspective. If this is of interest to you, please create an issue in the repo (GitHub - BabylonJS/Extensions: Extensions for Babylon.js) or feel free to submit a PR.
This is a great extension, and I have modified the code based on it. Now it can support both internal and top-level display, achieving a semi transparent effect when displayed at the top level. In addition, I have fixed some bugs, such as positioning errors when htmlMesh is not in the root node. I have initiated a PR and really like this extension，I have created a playground where can view the effects HtmlMesh Example PG | Babylon.js Playground (babylonjs-playground.com)
In addition, I optimized the zoom display of HTML. For example, I created a 100px * 50px HTML and a 2 * 1 mesh in the scene. Now, the HTML will automatically zoom to the size that fits the mesh and it looks very good
I believe the latest NPM package (PR pending) fixes all the issues @Pryme8 and @Kelede were seeing. Thanks so much to both of you for your feedback and contributions.
HtmlMesh is written in typescript. Make sure that you are using the ES6 core babylon package. Per the docs:
This package depends on BabylonJS, specifically the Core ES6 Supported package @babylonjs/core. This is done to support apps that are taking advantage of dependency optimization to reduce the size of the included BabylonJS dependencies. See the documentation for BabylonJS ES6 Support for details. Note that it will not work if your app is using the legacy babylonjs package. If this is an issue, please create an issue on github and I’ll look into creating a legacy compatible version.
If you are using the correct BabylonJS package and having issues, please create an issue on Github.
