Dom in XR scene

Tutorials and tips
1

Hi everyone,
I am trying to display an HTMLCanvasElement over a mesh in an xr scene, I wish I could update the texture every time the canvas is modified. To do so I want to create an observable that would monitor the state of my canvas (html) and every time the canvas is updated I update my texture. My issue is that I can’t update my texture, it always stay black even when I set the backgroundcolor property to ‘FFFF00’ (for example)
Here is my code sample. I am testing in a playground. Is there somethig I need to know about texture in a xr context ?

var camera;

var createScene = async function () {
var scene = new BABYLON.Scene(engine);

var light = new BABYLON.PointLight("Omni", new BABYLON.Vector3(0, 100, 100), scene);
camera = new BABYLON.ArcRotateCamera("Camera", 0, 0.8, 50, BABYLON.Vector3.Zero(), scene);
camera.attachControl(canvas, true);

var ground = BABYLON.Mesh.CreateGround("ground", 500, 500, 10, scene)

let canvas1 = document.createElement(‘canvas’);

canvas1.style.background = ‘fff00’
const myDynamicTexture = new BABYLON.DynamicTexture(‘’, { width: canvas1.width, height: canvas1.height }, scene);
myDynamicTexture._canvas = canvas1;
myDynamicTexture.update();
var material = new BABYLON.StandardMaterial(“”, scene);
material.diffuseTexture = myDynamicTexture;
material.useAlphaFromDiffuseTexture = true;

var box3 = BABYLON.MeshBuilder.CreateBox(‘box3’, {size: 4})
box3.position = new BABYLON.Vector3(-2, 1.7, 3);
box3.material = material;

//  const environment = scene.createDefaultEnvironment();

// XR
const xrHelper = await scene.createDefaultXRExperienceAsync({
    floorMeshes: [ground]
});

let mesh;

xrHelper.input.onControllerAddedObservable.add((controller) => {
    controller.onMotionControllerInitObservable.add((motionController) => {
        if (motionController.handness === 'left') {
            const xr_ids = motionController.getComponentIds();
            let triggerComponent = motionController.getComponent(xr_ids[0]);//xr-standard-trigger
            triggerComponent.onButtonStateChangedObservable.add(() => {
                if (triggerComponent.changes.pressed) {
                    // is it pressed?
                    if (triggerComponent.pressed) {
                        mesh = scene.meshUnderPointer;
                        console.log(mesh && mesh.name);
                        if (xrHelper.pointerSelection.getMeshUnderPointer) {
                            mesh = xrHelper.pointerSelection.getMeshUnderPointer(controller.uniqueId);
                        }
                        console.log(mesh && mesh.name);
                        if (mesh === ground) {
                            return;
                        }
                        mesh && mesh.setParent(motionController.rootMesh);
                    } else {
                        mesh && mesh.setParent(null);
                    }
                }
            });
        }
    })
});


return scene;

}

2

Hello and welcome ! :slight_smile:

You should have a look at this topic : HTML Content in BabylonJS Scenes

In this post, @ericwood73 was speaking about the issue with XR and Dom. Maybe it will help

++
Tricotou

1 Like
3

You can use a dynamic texture when in XR, but you can’t display DOM elements directly. If you want to reproduce the issue on the playground we might be able to point out what went wrong and where.

4

This is the closest thing I have seen to what you are trying to accomplish, but as noted in the docs, you could have CORs issues, it won’t work with cross origin iframes, and it won’t work in Safari/webkit (meaning it won’t work at all on iOS browsers outside DMA, but then that’s not that big a deal since webxr won’t work on iOS outside the DMA). The recommended solution to these issues in the docs is the same conclusion I have come to: the only way to reliably and fully support any HTML in webxr is to render it on the server (or a companion app) and ship frames to the client to be rendered as textures. I’m sorry to be the bearer of bad news, but I can save you alot of the headache I have gone through researching solutions and hitting dead end after dead end.

1 Like
5

thanks everyone for your responses, I think I will work on other features before coming back to this one. I think you saved me a ton of time

1 Like