Mouse pointer lock and OnPickTrigger

Hello Everyone,

I am trying to create first-person controls and let players aim with a reticule and click to interact with things, however I want to still make use of the standard OnPickTrigger.

Is there a way to have a pointer screen lock and still use the regular method of mouse clicks? The problem is when the screen locks, the pointer disappears and it’s likely not centered where the reticule is. (It is in the original place when the screen locked) After this happens, the original pointer is invisible, but still works. If the original pointer could be repositioned to where the reticule is, everything would be all good, but as I understand it there’s no way to do that. (Security reasons)

Perhaps using rays to activate the onPickTrigger as if it was a regular mouse click would work? Does anyone know how to do this? I can’t find info on it. Or does anyone have a better solution to this problem?

Here’s a playground:

The end goal is to make this work with and without VR without needing separate logic. The standard onPickTrigger works well with VR.

I found some code and kind of have it working now:

But if anyone else has thoughts please let me know.


Can you explain ‘kind of’ ? Looks to me like this is doing the job.

You are right, it seems ok but there is still a leftover side effect. When the pointer lock is activated, the reticule click works as expected but the original mouse pointer also still triggers a click.

For example, position the mouse pointer off to a bottom corner, click and lock the screen then move the screen so the invisible old mouse pointer points to the sphere and click. (The corner of the screen is overtop the sphere) It triggers a click but I want only the reticule to trigger clicks when the screen is locked.

@PolygonalSun might have some tips?

Yes, I have been able to reproduce this. I’m eager to read about the fix (may be not as much as you, but still :wink:). Hope it won’t take all too much time :hourglass_flowing_sand:
Meanwhile, have a great day :sunglasses:

This feels hacky, but it works:

Four planes positioned so there’s a tiny hole in the middle for the ray to shoot through… or even for the original mouse pointer to click through if it happened to lock at the center. The planes are parented to the camera and invisible and they can block the default click.

Uncomment the line 97: visibility = 0 to see it.

Yes, that seems very hacky :wink:
I was just thinking. Since you cannot move the pointer towards the center when you lock it, may be you could work it the other way round. Try center the camera on pointer position when you lock. Will next the pointer follow along with the camera is the question that remains…

Hey @vx9,
So I took a look at your PGs (particularly, the second one), and here’s what I’m seeing. You have an action set up to handle a pick when it happens, as well as performing a manual pick on a pointerdown event. Inside of scene.onPointerDown, you process the trigger if you get a hit on your sphere. When you enter a Pointer Lock, you’ll have one of two scenarios.
1.) If you click on anything that isn’t the sphere to start your pointer lock, it should work as expected because the ray is firing where you want it to fire.
2.) If you click on the sphere to start your pointer lock, your action will double activate.

The reason for scenario 2 is because when you request a pointer lock, any pointer events fired during that lock will not change the absolute coordinates in the pointer event. Only deltas (movementX/Y) will be updated. This means that every click you perform will be done at the exact same place, as far as the browser is concerned.
For the action OnPickTrigger, this action will automatically be processed as part of the “behind the scene” work between the InputManager and the ActionManager (this happens during the pointerup step). With this in mind, you’re processing your trigger from your down event and then potentially processing another with your up event.

To fix this, you’ll either need to not use the ActionManager and just work within either the callback or observable OR you can just create a boolean to skip the trigger body:

// Some boolean to track when to skip
var skipPick = false;
// Inside of your Action
    new BABYLON.ExecuteCodeAction(BABYLON.ActionManager.OnPickTrigger, function(){
        if (!skipPick) {
            console.log("clicked using pickTrigger");
            sphere.position.x += 0.1;

// Using onPointerObservable just as a way to organize scene inputs
// If you use this, you will need to remove your onPointerDown code
  scene.onPointerObservable.add(eventData => {
      const e = eventData.event;
      if (e.button === 0) {
          if (eventData.type === BABYLON.PointerEventTypes.POINTERDOWN) {
                let hit = castRay(scene);
                if (hit.pickedMesh && === "sphere") {
                    console.log("clicked on sphere using ray",
                skipPick = true;
          // This just handles the up event, if you want you could just add this part to an onPointerUp callback and keep your down callback
          else {
              skipPick = false;
// As a note, this mask just makes it so we only do something for UP or DOWN only
  }, BABYLON.PointerEventTypes.POINTERDOWN | BABYLON.PointerEventTypes.POINTERUP);

Keep in mind, this is just one approach but hopefully this helps.