I’d like to add rich HTML annotations to the scene like SketchFab annotations. These annotations must be attached to numbered hotspots on the mesh and their visibility must be controllable.
I understand there are several ways to achieve this effect such as Babylon GUI, etc. but those don’t meet our need. For example babylon gui does not have an API surface that allows passing HTML. Second, with babylon GUI you have to worry about things like AdaptToDeviceRatio performance issues. It is better to let the browser handle the complexities of the markup we want to show. That way we can extend it to show rich media like images, links, videos, etc.
I’ve looked at quite a few forum posts discussing similar topics (such as CSS3DRenderer) but none that solves our problem fully. CSS3DRenderer link shows how to have a fixed dom element be part of the scene. Whereas we want a floating DOM element overlaid on top of the scene that is selectable (to do copy/paste, etc.).
The only viable solution seems to involve projection of the mesh on the viewport and then passing that position along to the DOM (via updating the CSS) that @sebavanhinted here.
So I set out to create a playground and quickly got stuck. Part of it because of playground quirks, but part of it because I’m not sure if the projection function is correct. Babylon.js Playground
As you can see the div follows box.getBoundingInfo().boundingBox.centerWorld. A better way would be to follow a particular point on the GLB mesh. For example this playground https://playground.babylonjs.com/#3E0RTK#9 I’d like the div to follow the location of the beak of the seagull.
Any idea on how to get the projection of a particular point of the GLB model on viewport?
Let me propose another solution. If you want fine grained control on positioning the annotations I suggest to add markers in the GLB (or whatever format you use).
Example for Blender:
Parent your mesh with an Empty (becomes a TransformNode in babylon.js).
Add an Empty for each your annotation position. I added two, one in the corner of the cube and one in the middle of one side of the cube,
With this approach you can absoutely precisely position your annotations even on complex meshes. And they naturaly follow the transformation of the mesh. The code is mucb more simpler and readable.
This isn’t really true, at least not using the HtmlMesh extension. You can have an overlay mesh that can be positioned relative to any other mesh (via setting the parent and transform). Be aware however that HtmlMesh will not work in XR or native wheras the BJS GUI will.
IMHO it’s better to have the long description part somewhere at a static position, always at the same, where it doesn’t cover the model and the other markers and I would implement it rather in plain HTML to have reach features available and stick it to the right upper corner of the screen with a close button.
One thing I learned and wanted to share here for posterity is that this will not work for WebXR. I read in a post that BABYLON.GUI.AdvancedDynamicTexture.CreateFullscreenUI is not supported in webxr.
So when comparing the tradeoffs of GUI vs. HTML Overlay vs. HTMLMesh that must be considered. For now I’m leaning towards calling defeat in making the annotations work with the same code for webxr and regular alike and going with the GUI approach. May come back to rewriting this if/when overlays are supported in webxr (which i believe is the limiting factor)
Maybe you could opt for regular meshes in billboard mode. That means you could create a plane with a texture which will always face the camera. You could create the texture dynamically for each marker, that’s very easy to do with babylon.js.
Something like this:
If you don’t want to have the markers always on top just remove this line:
