WebXR Plane Detection demo not working?

Ankit_Yadav · September 8, 2020, 1:24pm

Babylon.js Playground.

Ankit_Yadav · September 8, 2020, 1:36pm

Working after enabling webxr incubation

adam_breachvr · September 8, 2020, 1:42pm

Doesn’t seem to work in the iOS XRViewer, however…

Ankit_Yadav · September 8, 2020, 1:53pm

https://immersive-web.github.io/webxr-samples/report/

adam_breachvr · September 8, 2020, 2:02pm

Hmm, everything except for core is reported as unsupported, even though I have tested e.g. hit tests and they work. Also, pressing Begin Session on immersive-ar doesn’t do anything.

RaananW · September 8, 2020, 2:12pm

iOS doesn’t support WebXR officially, so everything there is very experimental.

The plane detection demo doesn’t need hit test only. it needs the plane detection feature, and i believe it also uses anchors to add meshes on top of those planes.

adam_breachvr · September 8, 2020, 2:20pm

The anchors feature is neither enabled nor used in that demo ( Babylon.js Playground ). Incidentally, trying to enable it on iOS crashes the whole AR session.

Planes are detected on iOS, as evidenced by the debug draw, however something is wrong in the API either on side of the WebXR implementation or Babylon and this information is not communicated. Will attach screenshot later.

adam_breachvr · September 8, 2020, 2:34pm

RaananW · September 8, 2020, 2:37pm

Yup, you are right - no anchors in this one. Thought you are referring to a different demo.

As it is working in chrome, and chomre is very good in keeping up with the specs, I would guess that it is not an API issue. I don’t know the viewer, and I sadly can’t test.

Is there any exception thrown, is there any way to debug this?

As plane detection is not an (optional) feature of XR, i would assume planes will be shown on every ar scene. Is it so?

adam_breachvr · September 8, 2020, 2:57pm

Yeah, I think the plane detection runs in the background automatically on every scene. There is no exception thrown from my code that initializes the Babylon plane detection feature, but I don’t know if there isn’t any error thrown from anywhere else. I’ll see if I can find some way to debug this further.

RaananW · September 8, 2020, 3:06pm

Seems like they didn’t expose the API. They just show you how XR is seeing the environment.

adam_breachvr · September 8, 2020, 3:07pm

Alright, will try ask folks over at the viewer app. Thanks for the help!

adam_breachvr · September 11, 2020, 9:10am

Well, they have their own demo of rendering the detected planes and it’s working just fine: webxr-ios-js/index.html at master · MozillaReality/webxr-ios-js · GitHub

So I think maybe there are some slight inconsistencies in the API between Chrome and Firefox, with the Babylon implementation only accounting for Chrome?

RaananW · September 11, 2020, 9:43am

This demo does not use plane detection. it uses anchors and hit tests like this demo - https://playground.babylonjs.com/#GG06BQ#3

adam_breachvr · September 11, 2020, 10:17am

The description reads: Render the anchors, including planes and face geometry, detected by the platform.

It uses something referred to in code as “world sensing”, which seems to refer to, among others, plane detection.
Initialized here: webxr-ios-js/index.html at master · MozillaReality/webxr-ios-js · GitHub
Calllback here: line 210
Plane mesh creation here: line 310

Screenshot:

RaananW · September 11, 2020, 11:07am

The plane detection explainer (which is the standard we are following) can be found here:

github.com

immersive-web/real-world-geometry/blob/master/plane-detection-explainer.md#plane-detection---quick-start

# Plane detection explainer
## Introduction
This document presents an overview of the plane detection API. The idea is that in order for some Augmented Reality scenarios to work nicely, developers need to know about surfaces present in the user’s environment. Example use cases:
- Placing virtual objects in user’s environment.
- Measuring user’s environment.
- Real-world physics interactions.
It’s possible that access to detected plane information will be useful for occlusion as well, though limitations of current RWG runtimes may limit the effectiveness.

Later sections of this document show how applications can enable plane detection and retrieve information about the planes. More advanced use shows how plane lifetime management could be performed in an application. Please note that this document is not supposed to serve as an API reference - it only shows by example how the API could be used.

## Overview
The API is synchronous and frame-based. If plane detection is enabled in an XRSession, each XRFrame will have a set populated with planes tracked by the session. The advantage of a synchronous API is that we can guarantee that the set of returned planes is valid only during that frame, and this allows the User Agent to change their properties as they get refined by the underlying AR SDKs.

The returned set will contain all planes tracked in the current frame, including planes that may not be currently visible. If the plane object was present in the XRFrame’s set in frame N and is not present in the XRFrame’s set in frame N+1, it means that the tracking of that plane was lost / the plane is no longer present and all properties on the object will throw exceptions if the application attempts to access them. If the same plane is detected both in frame N and in frame N+1, it will be represented by the same XRPlane object, with its attributes / state possibly updated.

The information stored in an XRPlane consists of plane orientation (if known), a convex polygon approximating detected plane, and the pose of the plane’s center that can be retrieved given a reference space. The center’s pose describes a new frame of reference in such a way that the Y axis is a plane’s normal vector, and X & Z axes are right and top vectors, respectively. The polygon vertices are specified in the reference space described by the plane center’s pose. The information retrieved from a plane is valid only during the session’s `requestAnimationFrame` callback, same as the XRFrame object from which it was obtained.

Plane tracking is enabled via parameters passed to `configureWorldTracking`. This signals that WebXR should start populating the `detectedPlanes` attribute for each subsequent XRFrame delivered to `requestAnimationFrame` callback. This approach also allows us to extend the plane detection configuration easily if we choose to do so (for example to enable pre-filtering of planes based on their orientation).

## Plane detection - quick start

This file has been truncated. show original

What mozilla are doing is not part of the standard, and we cannot support it. You can, of course, implement this yourself, but it will only work in firefox reality (and will probably not work in any other AR-based device)

eddie_romper · May 22, 2021, 7:26pm

You must have a new android phone. Setting the flag to enable is not a good solution when default setting works in ThreeJS and older phones. Its a babylonJS bug

RaananW · May 25, 2021, 5:22pm

Can you elaborate? Is plane detection not working?

thangld · November 7, 2022, 11:00am

Can someone update the demo code please. https://playground.babylonjs.com/#98TM63#131
It cannot detect surface on Chrome on Samsung Galaxy S10. Thanks!

RaananW · November 7, 2022, 1:06pm

seems to work for me on my S10. Have you turned on the incubation flag in chrome://flags ?

Topic		Replies	Views
WebXR Plane detection API not working in WebXR Viewer Bugs	1	452	September 8, 2020
[XR] WebXRPlaneDetector not finding planes Bugs	7	792	October 6, 2020
Web XR - Ar are specifically - plane detection - future? Questions webxr , ar	4	1316	February 17, 2023
Babylon WEB -AR is not rendering in IOS - WebXR Viewe‪r‬ Questions documentation	1	1194	March 26, 2021
Any working AR example available on PlayGround? Questions	25	1873	September 21, 2022

WebXR Plane Detection demo not working?

Related topics