Re-attach control on pointer lost if outside canvas

For projects that have an independent UI, it is important that the user can control focus which is acceptable in terms of accessibility and UX.

I couldn’t find an implementation in the engine that handles detaching the controls if effectively losing pointer capture (say by clicking DOM UI component outside of the canvas). The problem I had is that control events sometimes persisted, such as the camera position when dragging-out of canvas. The user would have to trigger the onkeydown back in the canvas to regain correct control/reset inertia.

Below is my solution which seems to be ideal for my project, without messing with inertia. This post is for others/discuss as a feature request if helpful. If there’s a better solution then please ridicule this method! Apologies for lack of repo at this time.

@babylonjs/core: ^5.40.0
@babylonjs/gui: ^5.40.0

export function addLostPointerCaptureListenerOnPointerOut(canvas: HTMLCanvasElement, scene: Scene) {
  canvas.addEventListener(
    "lostpointercapture",
    (ev) => {
      const rec = canvas.getBoundingClientRect();
      const x: number = ev.clientX - rec.left;
      const y: number = ev.clientY - rec.top;
      const outLeft: boolean = x < 0;
      const outTop: boolean = y < 0;
      const outRight: boolean = x > canvas.width;
      const outBottom: boolean = y > canvas.height;
      const pointerOutOfCanvas = [outLeft, outTop, outRight, outBottom].some(Boolean);
      if (pointerOutOfCanvas) {
        scene.activeCamera!.detachControl();
        scene.activeCamera!.attachControl();
      }
    },
    false
  );
}
1 Like

cc @PolygonalSun

I believe this topic is worth a discussion. It probably was part of many discussions.
And then, I believe your approach is not ‘ridiculeous’. I guess the one problem we are facing is that, when loosing context, we eventually do not get the callback from all browsers, apps and context.
So, trying to work this from within the canvas seems appropriate (in most cases).
But then,I really don’t have the expertise on this. All I can do is acknowledge and talk about my user experience. Yes, we keep on having issues with loosing context, that’s all I can say.
I’m also certain that this has been a concern and a study for many of the ‘big-brains’ within the BJS team and the community. But then I guess, one more brain on this can only be welcomed.

@mawa thank you, that’s encouraging.

I don’t think we should put up constraints on the engine but maybe provide utility functions for the scenarios and accessibility best-practice documentation, mentioning these utility functions.

These three scenarios covers two focus contexts:

basically… canvasFocusOnly: false

If false then something outside the <canvas> may be intended to be focusable (for pointer, touch, controllers & XR). This could be an engine constant to provide a basis for the control managers.

Design best practice in my opinion:

  • if canvasFocusOnly === true then canvas should be the size of the window
  • if the pointer is consumed, then the only focus should be possible to the canvas (no other UI outside)

Of course this is currently just design conjecture.

Accessibility compliance is a legal challenge for web developers and clients. In my experience, complying as best as possible can greatly increase the usability and engagement of a product for everyone. Microsoft are hot on accessibility, I’m sure they’ve discussed this with their metaverse projects if anyone from MS is listening? I’ve dabbled with SharePoint’s Spaces (Babylonjs) but don’t think it’s quite mature enough for accessibility.

Some reading: XR Accessibility User Requirements

It would be great to see some Babylonjs examples around accessibility needs!

1 Like

cc @carolhmj about the accessibility part.

That’s a great discussion and I agree that making stuff more accessible benefits everyone design and technology-wise :smiley: We’re starting to include some features designed for accessibility, such as some support for screen readers ( Accessibility Scene Tree for Screen Readers | Babylon.js Documentation (babylonjs.com)), ( Canvas Accessibility and GUI Animations with Babylon.js | by Babylon.js | Dec, 2022 | Medium), but we have a long way going yet. These first features were thought around full screen Babylon experiences, so it’s very good that you brought up the non full screen case!