In the combination of multi-canvas and 3D GUI, problems occur in 3D GUI pointer position detection due to the execution order of registerView

My application uses a single engine, multiple canvases, and places a 3D GUI in the scene.

Also, the canvas that is the input source to the engine is set with “engine.inputElement()”.

https://doc.babylonjs.com/features/featuresDeepDive/scene/multiCanvas#performance
As described in the documentation, it is recommended to make the canvases the same size in the case of multi-canvas,
but at the moment, it is Not possible to make all canvases the same size for reasons of placement in my application.

In such usage, there will be a problem with 3D GUI pointer position detection depending on the execution order of “registerView” to the engine.

It’s not a recommended usage, but it gives strange results, so I reproduced it in the playground.

https://playground.babylonjs.com/#9BTUAI#7

https://playground.babylonjs.com/#9BTUAI#9

Currently, it is avoided by executing “registerView” for the canvas set by “engine.inputElement()” at the end.

Is there a better way to do this than changing the order of “registerView” execution with “unRegisterView” + “registerView”?

— The difference between the two playgrounds is -----------------------------------------------
< PG#7>

  • Main canvas ( renderCanvas2.style.width =“100%” / .height =“100%” )
  • Small canvas (newCanvas.style.width/.height not specified)

< PG#9>

  • Main canvas ( renderCanvas2.style.width =“1000px” / .height =“1100px” )
  • Small canvas (newCanvas.style.width = “100%” / .height = “100%”)
  • Add white square frame

— How to play playground ------------------------------------------------------------------------------

(1) Open the playground.

(2) In the initial state
・The relationship between the ground and the pointer is normal
When the mouse moves over the ground, the cursor changes from an arrow to a pointer.
· It also recognizes that it has moved above the button.
(You can check it in the console output.)

(3)Click the button at this point.
Programmatically change the registration order of registerView after 1 second.
(UnRegisterView “newCanvas” once and registerView “newCanvas” again.)

— For PG#7 —
(4) At this point
・When you move the mouse over the ground, it changes from an arrow to a pointer.
・It will no longer recognize that you have moved over the button.
(Nothing happens when you click the button.)

(5) Now click the mouse on the ground avoiding the button and the sphere.

⇒Go to (6)

— For PG#9 —

(4) At this point
・Even if you move the mouse over the ground, the arrow will not change to a pointer.
・It will not recognize that you moved over the button.

(5) However, when the mouse is moved into the square frame on the upper left of the screen
The mouse cursor will change from an arrow to a pointer.
Click the mouse where the cursor has changed to a pointer.

⇒Go to (6)

— PG#7 and PG#9 —
(6) If you check the console output, it is recognized that you clicked on the ground.

(7) In the playground, if the ground is clicked in (5),
Programmatically change the registration order of registerView.
(Once unRegisterView “renderCanvas2” and registerView “renderCanvas2” again. )

(8) As a result of (7), it returns to the state of (2) and operates normally.

cc @carolhmj

Hello! The playgrounds aren’t the best use case for multi canvas, as it’s recommended to not directly use the main canvas that was used to create the engine, but I did try to reproduce on another multi canvas setup and I can confirm the 3D GUI doesn’t seem to respond to events in this situation, so I’ll investigate further.

2 Likes

Ok, investigated! So, the issue with GUI 3D controls when you have a multi-canvas setup is that, due to the multiple cameras involved, the 3D GUI is bound to the last camera created, which is not necessarily the same that’s receiving inputs. You can solve this by manually setting which camera is used for the 3D GUI layer like this: quick-demos/test-multiple-canvas.js at main · carolhmj/quick-demos (github.com)

2 Likes

Thank you for your prompt reply.
I found out that the camera can be specified by executing “.utilityLayer.setRenderCamera(camera)” for GUI3DManager.

I reproduced the demo provided and
when the canvases are all the same size (see red arrow in the figure below) and
when there are canvases of different sizes (see the red and yellow arrows in the figure below) compared.

as a result
Even if you reproduce the provided demo, it seems that the position detection is strange if there are canvases with different sizes.
plerase see figure.

< same size >

< different size >

Although it is difficult to understand without enlarging the figure, the actual mouse position (a) and the position detection mark (b) displayed on the 3D button do not match.
< different size detail >

Please let me know if there is a way to avoid this issue even if I have canvases of different sizes.

I did some testing with canvas of different sizes and saw the behavior, due to the canvas dimensions changing between views. I set the view that corresponds to the inputElement to always be rendered last, so the canvas size should be consistent now. The PR is up here: When rendering in a multi-canvas setup, always render the input view … by carolhmj · Pull Request #13320 · BabylonJS/Babylon.js (github.com)

2 Likes

Thanks for investigating and fixing. :heart_eyes:
Wait for the merge. :smiley:

1 Like

I have confirmed that the previous sample fixed the issue. :laughing: :heart_eyes:

This is what I noticed using GUI3DManager’s .utilityLayer.setRenderCamera(camera).
:arrow_down: :arrow_down: :arrow_down:
My application goes back and forth between 3D and XR.
In such a case, when using “.utilityLayer.setRenderCamera(camera)”,
the 3D camera and the XR camera change when 3D <=> XR switches,
so If you do not execute “.utilityLayer.setRenderCamera (xr-camera)”
or “.utilityLayer.setRenderCamera (3D-camera)”
in accordance with the switching between 3D and XR,
the display destination of the 3D-GUI will not switch, so be careful. .

I forgot this and was in a panic when the 3D-GUI was not displayed on the XR. :scream:

1 Like