How can you track a marker while also utilising webXR/webAR technology?

Hello :wave:

I’d like to use a tracking marker (perhaps via ar.js or artoolkit/artoolkit5) to accurately place a 3d object in a specific real world location.

I would then like to use webXR/webAR technology so that the the 3d object will remain in place even if the camera moves away from the marker or the marker is obstructed/hidden from view.

I’ve been playing with the demos below but I am struggling to work out how to marry the two aspects as I’m new to the libraries.

Babylon examples

It looks like to me that the most important part to get right is the camera position between the webXR and marker tracking libraries as well as the camera’s controller but I’m not sure how to do this.

If anyone can help that would be awesome!
Thanks :slight_smile:

These are some three.js examples…

Adding @RaananW to the thread but as he is in vacation at the moment the answer might have some delay probably until Jan.

Hi Jammer,

If all you’re looking for is an example of how to integrate a marker tracker with Babylon.js rendering, our old Babylon AR extension did exactly that, connecting OpenCV’s ArUco marker tracking compiled for WebAssembly to a small TypeScript layer to make it easy to consume. I’m not familiar with the tracker you’re referencing, so I don’t know how directly you might be able to apply the same approaches for consumption in Babylon.js, but it it should at least serve as an example of how you might go about it. The approach we used was to create a Babylon.js node in the scene that a background system interacting with the tracker simply kept up-to-date; then we could parent whatever we wanted to the node, and it would move around the scene the same way the tracker saw the marker moving. The entry point to this logic is this file. Hope this helps, or at least provides an interesting reference point!

4 Likes

WebXR has a native image tracking capabilities (experimental at the moment). We have the feature implemented. You can see an example on how to use it here:

[XR] AR image tracking feature by RaananW · Pull Request #9512 · BabylonJS/Babylon.js (github.com)

It works using latest canary on android

2 Likes

Thanks for helping everyone! I’ve just been able to work on this again today.

The native webXR capabilities look really interesting! I hope it becomes stable soon :grin:

1 Like

Hey, and I have entered the discussion quite a bit later but would appreciate some advice:

I have created the following playground using the example code linked in this thread:

On any device I have tried with I get the following error:
Unrecognized feature requested: image-tracking

(mobile: Pixel 6 Pro using Chrome Canary)
(desktop: MB Air M1)

I have looked around for flags which I can set to enable features in the browser, or to install sw versions > recommended versions, but without improvement.

r

The flag you need to enable is this: chrome://flags/#webxr-incubations

This should work on the Pixel 6. Not sure about the Mac.

1 Like

Shameless plug. @RaananW and I did a presentation demoing this recently.

MR and the Web with Babylon.js 5.0 - Events | Microsoft Docs

4 Likes