Babylonjs-gui seems to not work in webview (tauri)

Description

I am developing an App which runs on two Platforms (desktop and web). For the desktop variant i’m using tauri. In the App there is a babylon scene which (in the end) consists of a couple of meshes. And if I click on some of the meshes i let a line, rectangle and textblock appear with babylonjs-gui. Which works just fine in the browser (Firefox 91.12.0esr (64-Bit)) but doesnt work in the desktop app (webview). So my guess is that babylonjs-gui just doesnt work in the tauri webview.

This is how it works in the browser:

demo_web_graue_kugeln_geht

In the desktop variant however it doesnt work (and yes I’m clicking):

demo_desktop_graue_kugeln_gehen_nicht

Reproduction

I’m using the following babylon versions:

"babylonjs": "5.18.0",
"babylonjs-gui": "5.18.0",
"babylonjs-loaders": "5.18.0",

And this is my onPointerDown function:

I also get “we hit gold” logged in the desktop app so the code which should display the gui elements gets executed but it just doesnt display it :frowning: .

// TODO i dont know if we need var to use it in the onPointerDown callback down below
var label: GUI.TextBlock;
var rect1: GUI.Rectangle;
var line: GUI.Line;
var sphere: BABYLON.Mesh; // The point to which the line rect and label get attached

// TODO this is kinda bad but since we need it in the callback
var scene = this.scene;
var camera = this.camera;
var errors = this.errors;

this.scene.onPointerDown = function castRay(evt: PointerEvent, pickInfo: BABYLON.PickingInfo, type: BABYLON.PointerEventTypes) {
    // src: https://playground.babylonjs.com/#AC8XPN, https://www.youtube.com/watch?v=dgsWKpa7RcY
    let ray = scene.createPickingRay(scene.pointerX, scene.pointerY, BABYLON.Matrix.Identity(), camera);	
    console.log('in onPointerDown');

    let hit = scene.pickWithRay(ray, (mesh: BABYLON.AbstractMesh) => {
        // some filtering
    });

    function disposeBox() {
        label?.dispose();
        rect1?.dispose();
        line?.dispose();
        sphere?.dispose();
    }

    // TODO could do hit.pickedMesh?.metadata ??
    if(hit.pickedMesh && hit.pickedMesh.metadata) {
        console.log('we hit gold');
        disposeBox();
        sphere = BABYLON.MeshBuilder.CreateSphere("a", {});		// Create a sphere with the given name
        sphere.visibility = 0;															// Set visability 0 = no visable 1 = visable
        sphere.position = hit.pickedPoint;
        let advancedTexture = GUI.AdvancedDynamicTexture.CreateFullscreenUI("UI", true, this);		// Enables GUI Elements for the scene
        advancedTexture.renderAtIdealSize = true;
        line = new GUI.Line("line");	
        line.lineWidth = 4;					// Width of the line
        line.color = "red";					// Color of the line
        advancedTexture.addControl(line);		
        
        line.linkWithMesh(sphere); 			// Link with given Mesh

        // Texbox
        rect1 = new GUI.Rectangle("rect1");		// Create Textbox
        rect1.adaptHeightToChildren = true;		
        rect1.adaptWidthToChildren = true;
        rect1.cornerRadius = 20;					// radius for the corners
        rect1.color = "red";						// Border color
        rect1.thickness = 4;						// Thickness of the border
        rect1.background = "white";					// Background color
        advancedTexture.addControl(rect1);	
        rect1.linkWithMesh(sphere);   				// Link with given mesh 
        rect1.linkOffsetY = -150;					// Let it float above  
        
        // Text
        label = new GUI.TextBlock("label");		// Create Lable for text
        label.text = hit.pickedMesh.metadata.split(';').join('\n');							// Set the displayed text
        label.resizeToFit = true;					// Reisize to content
        label.color = "Black"						// Fontcolor
        label.paddingTop = 10;						// Set the padding
        label.paddingBottom = 10;					// Set the padding
        label.paddingRight= 10;						// Set the padding
        label.paddingLeft = 10;						// Set the padding
        rect1.addControl(label);
        
        // Connect Line with TextBox
        line.connectedControl = rect1;		// Connect with TextBox
    } else {
        disposeBox();
    }
}

Can you create some GUI without being related to an event, like a label: https://playground.babylonjs.com/#XCPP9Y#1

If not I would suspect it is a tauri problem unfortunately either not supporting webgl or canvas 2d correctly cause they are both at play in GUI.

Thanks for the quick reply. So it seems like the button is there but it just doesnt get displayed since the onPointerUp event gets fired when I click into where the button should be displayed.

meme_shit

As far as I know the Button doesnt have a visibility attribute which I could just set to 1, right?

so definitely smthg not working in this browser :frowning: do you have an error message ? could you check on their github or forum ? sounds like a missing support somewhere

Thats the weird thing, no exceptions at all. All I get logged to the console is this:

[Log] [vite] connecting... (client, line 184)
[Log] [vite] connected. (client, line 212)
[Warning] <Component> was created with unknown prop 'paramsFromPrevAnalysis' (Component.svelte, line 802)
[Warning] <Component> was created without expected prop 'prevParameters' (Component.svelte, line 968)
[Warning] <Dialog> was created without expected prop 'value' (Dialog.svelte, line 468)
[Log] Babylon.js v5.18.0 - WebGL1 (chunk-Q5RSHIEU.js, line 4975)

According to here tauri uses [webkitgtk](https://webkitgtk.org/) as a browser/webview. And they say that they are supporting webgl:

"

3D CSS and accelerated rendering

WebKitGTK can use the GPU to enable smooth page compositing and scrolling, as well as 3D CSS transforms and 3D HTML canvas (otherwise known as WebGL). This makes WebKitGTK suitable for a whole range of games and visualization applications.
"

which makes sense. because if it didn’t support webgl it wouldn’t even be possible to display a mesh or anything with babylon. and of course the boundingbox of the rotor is also missing in the original post.

Unfortunately, when it comes down to those kind of issues specific to one browser, there is not much we can do on the babylon side.

The gui is relying on a canvas 2d uploaded as a texture with WebGLRenderingContext.texImage2D() - Web APIs | MDN and this fully works in Safari so I doubt the issue would be in webkit or might be the version used in Tauri would be older ?

ok. I dont really know how to check which version of webkit is beeing used (it isn’t written in the dev-tools e.g. there isn’t a about page). All I know is that I had to install the following packages:

sudo apt update
sudo apt install libwebkit2gtk-4.0-dev \
    build-essential \
    curl \
    wget \
    libssl-dev \
    libgtk-3-dev \
    libayatana-appindicator3-dev \
    librsvg2-dev

of course i could check with apt show which version of the packages are installed.

anyway… i noticed that in the dev-tools of webkit there is a Graphics tab. Which shows this:

Seems like both canvases are rendered correctly, but the dynamic texture is not updated. Can you try some other dynamic texture samples and see if that’s the issue here? something like this:

Dynamic Texture Example | Babylon.js Playground (babylonjs.com)

@sebavan , any reason dynamic textures will be ignored like that?

None except a broken support :frowning: