Support for Latest WebXR Features in Babylon.js

Feature requests
, , , ,
1

Dear Babylon.js Team,

I’m currently using Babylon.js version 7.32.0 on a Meta Quest 2 device running build 69.0, with browser version 35.3. I wanted to inquire about the support for the latest WebXR features in Babylon.js. I couldn’t find all the information in the GitHub issues labeled VR/AR/XR (Issues · BabylonJS/Babylon.js · GitHub), so I decided to ask here.

I’ve noticed that multimodal input doesn’t seem to be fully functional yet. For example, if I start with two Quest controllers and set one aside, a hand doesn’t appear in place of the controller. Conversely, if I start with two hands and pick up a controller with my right hand, the controller appears but the left hand disappears.

According to Meta’s browser release notes (https://developers.meta.com/horizon/documentation/web/browser-release-notes):

WebXR Multimodal Input: If hands are enabled in WebXR, you can now get one hand and one controller. In addition, the non-primary inputs can now be tracked in the session’s trackedSources attribute.
Logitech MX Ink: WebXR support for new 6DoF pen controller with 2 buttons and 2 pressure sensitive areas

I’m expecting to receive the XR pen soon (MR Stylus for Meta Quest 3, Meta Quest 2 | Logitech). I would like to get multimodal input working so I can use hand tracking with my left hand for object manipulation and the pen in my right hand for more precise interactions. The pen was developed in collaboration with Meta and has received positive reviews. There’s a Three.js template available (GitHub - fcor/mx-ink-webxr: This template uses Three.js and WebXR to provide an immersive 3D painting experience using Logitech MX-INK.), but I couldn’t find a Babylon.js equivalent.

Additionally, the release notes mention:

WebXR: add experimental support for unbounded spaces

In AR, Meta’s boundaries appear unnecessarily, and objects added to the scene don’t stay in place when referenceSpaceType is set to “unbounded”.

WebXR: transform is now provided for the origin reset event

I’m not entirely sure of the implications, but it might be useful in certain scenarios.

WebXR: Depth sensing implementation has been updated to support the latest spec. Older implementations will continue to work with a warning.

There’s already an issue on GitHub regarding this. It could be beneficial for AR experiments.

I also read that body tracking support is coming to Babylon.js ([WebXR] Implement body tracking · Issue #14960 · BabylonJS/Babylon.js · GitHub). I’m curious about how this will work with multimodal input. Will both the body and controllers be rendered simultaneously, or only the controllers? What’s the current status of the body tracking implementation?

I’m interested in any updates on these new WebXR features and any other recent developments.

Thank you for your time, and I appreciate all the great work you’re doing with Babylon.js!

Best regards,

Tapani

2

Regardless of WebXR, I didn’t even know about this multimodal input stuff. Is it commonly used ? I have both Quest 2 & Quest 3, and I think I have never used at the same time a hand and a remote, but maybe I just never tried ^^

3

I haven’t tried multimodal input yet either, but it seems very useful, especially with the XR pen. Typically, it’s used with one Quest controller, but for new users, it would be easier if they could use hand tracking alongside the pen. This way, they wouldn’t need to learn how to use the Quest controller, and everyone already understands how to handle a pen.

According to Meta’s Unity SDK documentation and recent news articles, multimodal input is supported on the Quest 2:

According to the documentation:

Benefits of Multimodal Input

  • Hand+controller game-play: Use a controller in one hand as a tool or weapon (high accuracy, haptics, wide FoV), while keeping your other hand free to interact with objects or cast spells (high immersion).
  • Single controller game play: if your experience only requires one controller for most interactions, allow the users to only pick up one controller, and use their other hand for immersion and casual interactions when needed (eg. use a controller as a racket, and use your free hand to pick up the ball and interact with menus).
  • Use hands and controllers interchangeably for comfort: Use one hand for direct interactions with controls, while the other hand is holding a controller as a “remote control” for low effort indirect interactions.
  • Instant transition between hands and controllers: With hand tracking active, we can instantly identify when the controllers are no longer in the hand. This minimizes the need for manual transitions, and solves the problems of slow or failed auto transitions.
  • Find my controllers: With controllers now tracked when using hands, the app can show the user where the controllers are when they want to pick them up. This allows smoother transition back to controllers without having to break immersion by turning on passthrough/ taking off the headset.

I tried Meta’s own app https://www.meta.com/en-gb/experiences/interaction-sdk-samples/5605166159514983/ , “Concurrent Hands and Controllers (multimodal)” scene in the app. However, it didn’t work on my Quest 2 for some reason. Perhaps it hasn’t been updated recently, and since Quest 2 only received multimodal input support recently, that might be why. I’ll have to try it on the Quest 3 when I get the chance.

4

I found a related discussion on WebXR Discord about multimodal input and Logitech MX Ink: Discord

5

So, quick and concise answer :slight_smile:

  1. Multimodal input is already supported and working for quite some time now. You can try it with hands and controllers, but it should work with any other type of controller supported by the system.
  2. The logitech MX Ink should be supported out of the box as well. I don’t see a reason why not, as our controller system is controller-agnostic and supports any type of input webxr supports. If it doesn’t, please let me know!

We do support unbounded spaces. This is actually the default for AR sessions.
Depth sensing is partly supported alreadyy, but not on headsets like the oculus, because the specs have changed. This is the next feature on my todo list after I am done with something else. Afterwards comes body tracking.

Regarding body-tracking, the feature will allow you to connect the body tracking data to a human model, or provide your own meshes for each part. The controllers will still show up in their right place, because otherwise you will be lost when handling them. Of course they will work in parallel.

2 Likes
6

This could be the reason why multimodal input doesn’t work on my Quest 2.

https://developers.meta.com/horizon/documentation/unity/unity-multimodal/#troubleshooting

Switching between controllers and hands doesn’t work in the sample.
Ensure that you’re running on Quest 2 (with Meta Quest Pro controllers paired to your headset), Quest 3, or Quest Pro.

I thought they updated it so the pro controllers are not required anymore. I don’t really know.
https://gamerant.com/meta-quest-2-multimodal-hand-tracking/

Meta Quest 2 now has access to Multimodal, a form of control that implements the ability to freely utilize the controllers and hand controls at the same time. This feature was previously limited to the newer versions of the devices. While the Meta Quest 2 had a version of the mode, it required that users own the Touch Pro controllers, not the default Meta Quest 2 controllers. While the Meta Quest line of VR headsets has had hand controls for four years now, Multimodal is a step-up in convenience and capability.

Meta’s documentation could be misleading when it says multimodal works with Quest 2 but you actually might also need these expensive self-tracked pro controllers with Quest 2
https://www.meta.com/fi/en/quest/accessories/quest-touch-pro-controllers-and-charging-dock/

Anyways, I will use Quest 3 to test multimodal input, MX Ink and AR to see how they work with babylon.js.

1 Like
7

Here’s a Babylon.js demo video featuring Logitech MX Ink with Meta Quest 3. I do some writing, 2D & 3D drawing and multimodal input testing.

Multimodal input without the stylus works fine as you can see.

MX Ink doesn’t support hand tracking as we already knew and there’s nothing we can do about it at the moment.

https://drive.google.com/file/d/1vFGySAeIeSyg-TkQuKEC2PUvIFeDeBvj/view?usp=sharing

2 Likes
8

That’s an awesome demo :slight_smile:
Any chance of getting a link to try it ourselves? I want to draw on the floor as well

9

Here it is now, enjoy. :smiley:

Demo: http://bit.ly/3OOoT0j
The instructions are in the repository: GitHub - FinweLtd/mx-ink-webxr-babylonjs

1 Like
10

“unbounded” referenceSpaceType

Documentation link

This now appears to work on Quest 3, and it works quite well!

However, this statement didn’t hold true when I tested the examples:

“We do support unbounded spaces. This is actually the default for AR sessions.”

For example, in the first AR example, a boundary appears:
Example Playground

Depth Sensing

You mentioned that this feature is being worked on next, which is great to hear. It seems like an important feature.

For the documentation, could we get a Playground example similar to this Three.js example?
Three.js Example “xr / dragging / custom / depth”

It’s a very cool demo—check out the video I recorded: com.oculus.vrshell-20241211-104830-0_cut.mp4 - Google Drive

11

Unbounded does not mean the underlying system won’t show you their boundries, if configured by the user. It just means that it is expected that you will be able to cross this boundry. Or have I misunderstood something?

Yes! it is important for me to have this feature in babylon 8.0, so I the minute I have time to work on WebXR features I will work on this feature.

12

I’m not familiar enough with the specs. Based on my tests, the Quest 3 doesn’t show the boundary with the unbounded referenceSpaceType. When I crossed the boundary for the first time, the device displayed a brief message saying something like, “This app has boundary disabled,” and allowed me to continue using the app outside the boundary. However, the message appears only once, and I haven’t seen it again since.

With these options:

const xr = await scene.createDefaultXRExperienceAsync({
    uiOptions: { "sessionMode": "immersive-ar" },
    optionalFeatures: true
});

It sets referenceSpaceType to “local-floor” by default:

image
image871×107 20.8 KB

the boundary appears, and the app can’t be used outside of it. I get the “Return to your boundary” message from Meta.

But with these options:

const xr = await scene.createDefaultXRExperienceAsync({
    uiOptions: { "sessionMode": "immersive-ar", "referenceSpaceType": "unbounded" },
    optionalFeatures: true
});

The boundary doesn’t appear, and I can even leave my apartment while the app continues running without any issues.

1 Like
13

got it. Very possibly a change I was not aware of :confused:
We should have unbounded as default for ar sessions, so this is an issue I will need to address. Thanks for reporting :slight_smile:

EDIT - sorry, not a bug. a feature!
Initially we changed the defaults, but eventually decided to just show a warning in the console, which I assume you can see. I’m reconsidering again forcing unbounded on immersive-ar sessions.

1 Like
14

Unbounded is the best option for Quest 3 AR, but Quest 2 doesn’t seem to support the unbounded referenceSpaceType. It defaults to the viewer referenceSpaceType, which is very problematic because room-scale is not then active:

"immersive-ar" & "unbounded" on Quest 2
"immersive-ar" & "unbounded" on Quest 21348×121 112 KB

I propose defaulting to “unbounded,” and if that is not available, then defaulting to “local-floor.” This way, with Quest 2, you can still experience AR like on Quest 3, except you can’t leave the boundary.

Even though Quest 2 is becoming outdated with its low-resolution black-and-white passthrough, it can still be useful as a development device. It continues to receive updates; I just updated it to the latest Meta Build v72.