Pointer through multiple cameras

Questions
1

Hi,

I’m trying to use 2 cameras in a scene, one to render the 3D world (perspective view), and one to render a 3D UI (orthographic view).
The seconds overlays the first (scene.activeCameras = [persp_camera, ortho_camera]).

I can interact with my meshes viewed by the ortho_camera as expected, but objects on my second camera never receive the mouse.
From what I saw here and here, this is an expected behaviour.
I can choose to pick on one camera or the other, but not both.
How could I achieve picking on both ? Is there a limitation that wouldn’t allow it (focus issue…) ?

Here is a snapshot of my scene if that help to understand my goal :

  • The blue lines on the piece of paper are InputText.
  • The 4 objects (cube to sphere) on the bottom are 3D objects to control the scene “mode”.
    Currently, I can either choose between :
  • Having the 4 buttons at the bottom working, but the InputText never receive any pointer event
  • Having the InputText working, but the buttons stop receiving pointer events (if I set scene.cameraToUseForPointers = persp_camera).

Thanks

Capture.PNG
Capture.PNG1469×975 427 KB

2

If you want both scene, there is an option (as always with Babylon ;))

You can use the utility layer scenes. There are meant to be child scene of a main scene (in your case, this seems to be the right choice):
https://doc.babylonjs.com/how_to/utilitylayerrenderer

3

Thanks for the tip, but it seems it doesn’t work with multiple cameras and pick events together…

Here is a simple PG of what I had so far : https://www.babylonjs-playground.com/#3QW4J1#838
The ball is clickable, because it is on the last active camera of the scene (if I got that correctly), but the box is not.

And here is one where I tried using the utilityLayer : https://www.babylonjs-playground.com/#3QW4J1#843
At line 49 and down, I try to play with which camera is active in the main scene, but whichever option I apply, I can’t get both the ball and the box visible, and both of them clickable…

Do you have any idea of what I’m doing wrong ?
Should I proceed another way ?

Thanks

4

Pinging @Cedric who knows the utility layer and will be able to guide you :slight_smile:

5

I investigated a bit more for a possible solution, with and without utilityLayer.
I tried to capture pointer events on the scene level, and retry a pick for those that had no hit on the main camera to the second :

scene.onPointerObservable.add((e) => {
    if (!e.pickInfo.hit) {
        let pi = scene.pick(e.event.x, e.event.y, null, false, camera_1);
        switch (e.type) {
        case BABYLON.PointerEventTypes.POINTERDOWN :
            scene.simulatePointerDown(pi);
            break;
        case BABYLON.PointerEventTypes.POINTERUP :
            scene.simulatePointerUp(pi);
            break;
        case BABYLON.PointerEventTypes.POINTERMOVE :
            scene.simulatePointerMove(pi);
            break;
        case BABYLON.PointerEventTypes.POINTERPICK :
            // No method to simulate this (?). Is PICK generated from UP and DOWN ?
            break;
        }
    }
})

It seems to work to a very limited extend : I can have a cursor in a TextInput on a mesh for a very brief moment, then it loses the focus. Clickable meshes (using action managers) don’t react at all (so the pointer event never gets propagated that far).

In the PG, the snippet above results in a stack overflow, as it seems my implementation makes it recursive (it seems logical, yet I don’t have that issue on my project).

I tried to limit the depth of recursivity by putting a flag on the PickingInfo (as it seemed to be reused, as could be expected), so that it gets dropped on a second passage, but that only makes it an infinite loop (instead of filling the stack - it’s actually worse because I had to kill my browser to get rid of it).

So no real progress…
If anyone has a better idea, I’m all ears…

Thanks

6

I would expect all cameras to trigger an action, not just the last active.
I’ll check how the action is filtered and if it’s possible to change that.

7

Picking uses scene.activeCamera once a mesh is picked, action is triggered. There is no easy way to use all or a subset of activeCameras
So, for 1 camera, the ray is computed and the raycast is done against all mesh. As the mesh are displayed with a camera, only one mesh can be hit.

You can try to things:

  • with scene.onPointerObservable iterate over all your cameras and do the picking. You should get any mesh hit depending on the camera. You need to remove the actionManager.registerAction
  • or keep the actionManager.registerAction and in scene.onPointerObservable swap the camera and call the scene.simulatexxx functions. beware of too many recursive call

I think it’s better to swap the active camera when you get a hit. You should’nt get any weird behavior. It would be like setting focus to a specific camera depending on last ray hit success.

3 Likes
8

Sounds good enough for me for now !
I’ll let you know how it goes.

Thanks !

9

Hi,

Sorry for the long delay.
Finally, I had this bit of code working, if anyone is interested :

scene.onPointerObservable.add((e) => {
    if (!e.pickInfo.hit) {
        scene.cameraToUseForPointers = (scene.cameraToUseForPointers == cameraScene ? cameraUI : cameraScene)
    }
})

I use a highlight on pointer over, and it blinks a little when the mouse is on the border of a pickable element, but that’s understandable with this way of picking. I’ll try and make a smarter condition to switch from one camera to the other.

It works alright mostly because my meshes of interest on the 2 cameras are never on top of each other on screen, otherwise, it could randomly take the second mesh rather than the first (“randomly” because the choice to pick on either camera depends on which mesh return a hit first, which can depends on how the pointer enters either mesh, and which camera was being picked at just before).

So I’m still interested in a solution that evaluates pick for the first camera, then (if pick has not hit), pick on the second camera.

Cheers !