@RaananW Thanks to your earlier help, the project has started to take shape. But now, I’ve run into a new issue. I’ve embedded an external webpage using an iframe on my website. It looks good on Pico Browser in Pico 4, but when I switch to XR mode, things seem to go wonky. Any idea what might be causing this? By the way, I’m seeing the same issue on Microsoft Edge browser too.
Hi!
Can I ask - how are you rendering iframe on the desktop applications?
import * as BABYLON from '@babylonjs/core';
class CSS3DObject extends BABYLON.Mesh {
constructor(element, scene) {
super();
this.element = element;
this.element.style.position = 'absolute';
this.element.style.pointerEvents = 'auto';
}
}
class CSS3DRenderer {
constructor() {
const matrix = new BABYLON.Matrix();
this.cache = {
camera: { fov: 0, style: '' },
objects: new WeakMap(),
};
let domElement = document.createElement('div');
domElement.style.overflow = 'hidden';
this.domElement = domElement;
this.cameraElement = document.createElement('div');
this.isIE = !!document['documentMode'] || /Edge/.test(navigator.userAgent) || /Edg/.test(navigator.userAgent);
if (!this.isIE) {
this.cameraElement.style.webkitTransformStyle = 'preserve-3d';
this.cameraElement.style.transformStyle = 'preserve-3d';
}
this.cameraElement.style.pointerEvents = 'none';
domElement.appendChild(this.cameraElement);
}
getSize() {
return {
width: this.width,
height: this.height,
};
}
setSize(width, height) {
this.width = width;
this.height = height;
this.widthHalf = this.width / 2;
this.heightHalf = this.height / 2;
this.domElement.style.width = width + 'px';
this.domElement.style.height = height + 'px';
this.cameraElement.style.width = width + 'px';
this.cameraElement.style.height = height + 'px';
}
epsilon(value) {
return Math.abs(value) < 1e-10 ? 0 : value;
}
getCameraCSSMatrix(matrix) {
let elements = matrix.m;
return (
'matrix3d(' +
this.epsilon(elements[0]) +
',' +
this.epsilon(-elements[1]) +
',' +
this.epsilon(elements[2]) +
',' +
this.epsilon(elements[3]) +
',' +
this.epsilon(elements[4]) +
',' +
this.epsilon(-elements[5]) +
',' +
this.epsilon(elements[6]) +
',' +
this.epsilon(elements[7]) +
',' +
this.epsilon(elements[8]) +
',' +
this.epsilon(-elements[9]) +
',' +
this.epsilon(elements[10]) +
',' +
this.epsilon(elements[11]) +
',' +
this.epsilon(elements[12]) +
',' +
this.epsilon(-elements[13]) +
',' +
this.epsilon(elements[14]) +
',' +
this.epsilon(elements[15]) +
')'
);
}
getObjectCSSMatrix(matrix, cameraCSSMatrix) {
let elements = matrix.m;
let matrix3d =
'matrix3d(' +
this.epsilon(elements[0]) +
',' +
this.epsilon(elements[1]) +
',' +
this.epsilon(elements[2]) +
',' +
this.epsilon(elements[3]) +
',' +
this.epsilon(-elements[4]) +
',' +
this.epsilon(-elements[5]) +
',' +
this.epsilon(-elements[6]) +
',' +
this.epsilon(-elements[7]) +
',' +
this.epsilon(elements[8]) +
',' +
this.epsilon(elements[9]) +
',' +
this.epsilon(elements[10]) +
',' +
this.epsilon(elements[11]) +
',' +
this.epsilon(elements[12]) +
',' +
this.epsilon(elements[13]) +
',' +
this.epsilon(elements[14]) +
',' +
this.epsilon(elements[15]) +
')';
if (this.isIE) {
return (
'translate(-50%,-50%)' +
'translate(' +
this.widthHalf +
'px,' +
this.heightHalf +
'px)' +
cameraCSSMatrix +
matrix3d
);
}
return 'translate(-50%,-50%)' + matrix3d;
}
renderObject(object, scene, camera, cameraCSSMatrix) {
if (object instanceof CSS3DObject) {
let style;
let objectMatrixWorld = object.getWorldMatrix().clone();
let camMatrix = camera.getWorldMatrix();
let innerMatrix = objectMatrixWorld.m;
// Set scaling
const youtubeVideoWidth = 4.8;
const youtubeVideoHeight = 3.6;
innerMatrix[0] *= 0.01 / youtubeVideoWidth;
innerMatrix[2] *= 0.01 / youtubeVideoWidth;
innerMatrix[5] *= 0.01 / youtubeVideoHeight;
innerMatrix[1] *= 0.01 / youtubeVideoWidth;
innerMatrix[6] *= 0.01 / youtubeVideoHeight;
innerMatrix[4] *= 0.01 / youtubeVideoHeight;
// Set position from camera
innerMatrix[12] = -camMatrix.m[12] + object.position.x;
innerMatrix[13] = -camMatrix.m[13] + object.position.y;
innerMatrix[14] = camMatrix.m[14] - object.position.z;
innerMatrix[15] = camMatrix.m[15] * 0.00001;
objectMatrixWorld = BABYLON.Matrix.FromArray(innerMatrix);
objectMatrixWorld = objectMatrixWorld.scale(100);
style = this.getObjectCSSMatrix(objectMatrixWorld, cameraCSSMatrix);
let element = object.element;
let cachedObject = this.cache.objects.get(object);
if (cachedObject === undefined || cachedObject.style !== style) {
element.style.webkitTransform = style;
element.style.transform = style;
const objectData = { style: style };
this.cache.objects.set(object, objectData);
}
if (element.parentNode !== this.cameraElement) {
this.cameraElement.appendChild(element);
}
} else if (object instanceof BABYLON.Scene) {
for (let i = 0, l = object.meshes.length; i < l; i++) {
this.renderObject(object.meshes[i], scene, camera, cameraCSSMatrix);
}
}
}
render(scene, camera) {
let projectionMatrix = camera.getProjectionMatrix();
let fov = projectionMatrix.m[5] * this.heightHalf;
if (this.cache.camera.fov !== fov) {
if (camera.mode == BABYLON.Camera.PERSPECTIVE_CAMERA) {
this.domElement.style.webkitPerspective = fov + 'px';
this.domElement.style.perspective = fov + 'px';
} else {
this.domElement.style.webkitPerspective = '';
this.domElement.style.perspective = '';
}
this.cache.camera.fov = fov;
}
if (camera.parent === null) camera.computeWorldMatrix();
let matrixWorld = camera.getWorldMatrix().clone();
let rotation = matrixWorld.clone().getRotationMatrix().transpose();
let innerMatrix = matrixWorld.m;
innerMatrix[1] = rotation.m[1];
innerMatrix[2] = -rotation.m[2];
innerMatrix[4] = -rotation.m[4];
innerMatrix[6] = -rotation.m[6];
innerMatrix[8] = -rotation.m[8];
innerMatrix[9] = -rotation.m[9];
matrixWorld = BABYLON.Matrix.FromArray(innerMatrix);
let cameraCSSMatrix = 'translateZ(' + fov + 'px)' + this.getCameraCSSMatrix(matrixWorld);
let style = cameraCSSMatrix + 'translate(' + this.widthHalf + 'px,' + this.heightHalf + 'px)';
if (this.cache.camera.style !== style && !this.isIE) {
this.cameraElement.style.webkitTransform = style;
this.cameraElement.style.transform = style;
this.cache.camera.style = style;
}
this.renderObject(scene, scene, camera, cameraCSSMatrix);
}
}
export const initExternalWebPage = (webURL, scene, engine, position, rotation) => {
let existingRenderer = document.getElementById('css-container');
if (existingRenderer) existingRenderer.remove();
let renderer = setupRenderer();
createCSSobject(webURL, scene, 'qgKbpe4qvno', renderer, position, rotation);
createMaskingScreen('1', scene, engine, position, rotation);
};
const createMaskingScreen = (id, scene, engine, position, rotation) => {
const idPlane = BABYLON.MeshBuilder.CreatePlane('plane', { width: 1, height: 1,sideOrientation: 0 }, scene);
const sceneMaterial = new BABYLON.StandardMaterial('sceneMaterial', scene);
sceneMaterial.diffuseColor = new BABYLON.Color4(0, 0, 0, 0);
idPlane.material = sceneMaterial;
idPlane.scaling.x = 5;
idPlane.scaling.y = 2;
idPlane.position = position;
idPlane.rotation = rotation;
let depthMask = new BABYLON.StandardMaterial('matDepthMask', scene);
depthMask.backFaceCulling = false;
idPlane.material = depthMask;
idPlane.onBeforeRenderObservable.add(() => engine.setColorWrite(false));
idPlane.onAfterRenderObservable.add(() => engine.setColorWrite(true));
let mask_index = scene.meshes.indexOf(idPlane);
scene.meshes[mask_index] = scene.meshes[0];
scene.meshes[0] = idPlane;
};
const setupRenderer = () => {
let container = document.createElement('div');
container.id = 'css-container';
container.style.position = 'absolute';
container.style.width = '100%';
container.style.height = '100%';
container.style.zIndex = '1';
container.style.pointerEvents = 'none';
let canvasZone = document.getElementById('canvasZone');
canvasZone.insertBefore(container, canvasZone.firstChild);
let renderer = new CSS3DRenderer();
container.appendChild(renderer.domElement);
renderer.setSize(canvasZone.offsetWidth, canvasZone.offsetHeight);
window.addEventListener('resize', (e) => {
renderer.setSize(canvasZone.offsetWidth, canvasZone.offsetHeight);
});
return renderer;
};
let rendererStatus = 0
const createCSSobject = (webURL, scene, videoID, renderer, position, rotation) => {
let width = 2500;
let height = 805;
if (rendererStatus === 0) {
scene.onBeforeRenderObservable.add(() => {
renderer.render(scene, scene.activeCamera);
});
rendererStatus = 1
}
let div = document.createElement('div');
div.id = 'iframebox';
div.style.width = width + 'px';
div.style.height = height + 'px';
div.style.backgroundColor = '#fff';
div.style.zIndex = '-1';
div.style.pointerEvents = 'none';
const CSSobject = new CSS3DObject(div, scene);
CSSobject.name = 'ExternalWebPage';
CSSobject.position = position;
CSSobject.rotation = rotation;
let iframe = document.createElement('iframe');
iframe.id = 'web';
iframe.style.width = '2500px';
iframe.style.height = '805px';
iframe.style.border = '0px';
iframe.allow = 'autoplay';
iframe.src = `${webURL}`;
iframe.setAttribute('title', 'myiframe')
setTimeout(() => {
div.appendChild(iframe);
}, 0);
};
Do you mean code? Like this?
Nope, just wanted to see if you use the html directly or render it to a texture.
You cannot embed html elements nor use CSS in XR, as it didn’t render the DOM, only the canvas.
1 Like
You may be able to use DOM overlays, but they will only work on Android devices, not sure about Meta. Also the overlays will occlude everything else in the scene. I’m looking into some potential solutions, but nothing to announce yet. BTW, for non-XR applications you can use the HtmlMesh Babylon extension for this. It encapsulates all the CSS rendered stuff and adds some nice features.
Sorry if this has been raised before and I’ve missed it when searching the forum, but Three.js’s version of HTMLMesh renders as a texture in WebXR: Demo, Source, Implementation
Any reason why we couldn’t port the Three.js solution to BJS?
This is (probably) because they have some sort of HTML → Image or HTML → Canvas convertor.
I assume it doesn’t really work with each and every HTML element, and I assume some CSS is not being applied correctly to the page. Just an assumption though!
Also, not sure how they deal with interactions, forms, things like that.
There is currently no plan to create an HTML->Image convertor, but we are always happy to see contributions!
EDIT!
Just saw the HTMLMesh class. As I assumed - it is an HTML-to-canvas conversion that is then being used as a dynamic texture. My assumption is that it behaves wonderfully with simple HTML structures, but is quite slow to render when a more complex HTML page is rendered. This is, by no means, a negative opinion. I think it’s a great piece of code 
Yea, I realized after I posted this that the three.js implementation probably gets you simple things that cover lots of interesting use cases (“enter a password” or dat.gui) but probably not to a full iframe’d webpage.
1 Like