Hello all,
I have experimented with Babylon’s GUI, but would like to integrate custom HTML elements into the app and have them tracked in the 3D scene. I assume this is possible, but I can’t find anything in the docs.
Thanks in advance!
Hello!
you can project 3D coordinates into screen and then use absolute placement of your HTML controls
This is how we do it for GUI:
1 Like
I’ll give it a try!
Thanks!
@Deltakosh So I’ve had success implementing the HTML annotations, however, on mobile devices it doesn’t show up. I’m thinking it may have to do with setHardwareScalingLevel but I can’t quite pin the issue on it. Here’s what I’ve got:
// Update 2D Vector Position
const testPos = BABYLON.Vector3.Project(
pinVector1,
BABYLON.Matrix.IdentityReadOnly,
scene.getTransformMatrix(),
camera.viewport.toGlobal(
engine.getRenderWidth(),
engine.getRenderHeight(),
),
);
// Update HTML Element Position
pinPos1.style.left = testPos.x + 'px';
pinPos1.style.top = testPos.y + 'px';Sorry but with no repro I cannot even guess 
@Deltakosh I’m in a bit of a tough situation, as the work I’m doing is shrouded by an NDA until the project is complete, so I’m unable to share a repo. However, I have made considerable headway, and my questions now regard performance.
Originally, I was able to make HTML annotations by replicating what your team has done for the Babylon GUI, but upon testing it on lower-end devices, I noticed a significant performance drop. So I recreated the annotation using Babylon’s GUI, and performance was only slightly better. Is this because of the Vector3.Project() calculation happening every frame? Or the communication between Babylon and the CSS property?
I have been reading similar cases online, and some have suggested using Pixi.JS because it is also utilizing WebGL, but I have reservations in jumping in if calculating the position in 3D space is going to slow it down regardless.
Any suggestions you have would be greatly appreciated, and I thank you for your help thus far!
GUI does not use CSS so it should not be related.
What you could do is to use f12 profiler to see what is sucking your CPU and report it here. We may find creative ways to help 
For sure, do you have any documentation on how to set it up? I haven’t done any analysis outside the Babylon Debugger.
A simple google search should do it 
I ended up using the Microsoft Edge performance tab, as the f12 profiler seemed WordPress Specific – or maybe I’m just the ultimate noob! 
It looks like “Paint” was a recurring thing that was related to visual changes to the DOM. Could this be it?
Ok so now we have the culprits: intersectsTriangles is definitely too high.
Are you using scene.pick a lot?
@Deltakosh Nope, haven’t used it once. Is it being called automatically by the GUI class?
class createPin {
constructor(x, y, z, name, URL, panel, scale, scene) {
this.name = name;
this.scale = (scale * 0.1) + 1;
var location = new BABYLON.TransformNode(name, scene);
location.position.x = x;
location.position.y = y;
location.position.z = z;
this.label = new BABYLON.GUI.Rectangle(this.name);
this.label.height = (75 * this.scale) + "px";
this.label.alpha = 1;
this.label.width = (150 * this.scale) + "px";
this.label.thickness = 0;
this.label.linkOffsetY = -38 * this.scale;
panel.addControl(this.label);
this.label.linkWithMesh(location);
this.image = new BABYLON.GUI.Image("pin " + this.name, URL);
this.image.width = 0.25 * this.scale;
this.image.height = 0.55 * this.scale;
this.image.top = 18 * this.scale;
this.label.addControl(this.image);
this.text1 = new BABYLON.GUI.TextBlock();
this.text1.text = this.name;
this.text1.fontSizeInPixels = 15 * this.scale;
this.text1.fontWeight = "bold";
this.text1.shadowColor = "#000";
this.text1.shadowBlur = 1.5;
this.text1.shadowOffsetY = 2.5;
this.text1.color = "white";
this.text1.top = -15 * this.scale;
this.label.addControl(this.text1);
}
}
It should not this is why we need a repro
1 Like
Agreed, but I think you just pointed me in the right direction. Thanks
1 Like
@Deltakosh getting closer I think. Does the Advanced Dynamic Texture calculate intersectingTraingles for GUI elements, or could that be the Transform Node that the GUI panel is attached to?
@Deltakosh So after reverse engineering what I have setup, the stuttering; which only happens for 5-10 seconds on the first camera move, and randomly afterwards, goes away when I don’t linkWithMesh() the GUI elements. Also, I’m testing on a Galaxy S9, which runs my web-app flawlessly at 60 fps up until this point. It doesn’t matter what they’re linked to either. I’ve linked to Transform Nodes and actual mesh objects loaded into the scene. Aside from the createPin class I shared earlier, I’m not doing any collision detection in my scene, or raycasts, picking, you name it – and I haven’t done anything with the AdvancedDynamicTexture besides make it fullscreen. If anything, something gets turned on when I linkWithMesh(), and I can’t find anything in the docs aside from the basics. If the lag has to do with depth sorting GUI elements based on position, I don’t need it.
I’m officially stumped 
Well, how many pins do you have?
At the moment, 3…