NFT image tracking AR with babylon.js

Hello @javismiles, do you have any update on those two issues discussed here, I’m bumping in the same problem…
Cheers

yes, my update is that NFT image tracking on the web is a waste of time
until WebXR becomes cross platform for real, at that point in time then yes
it will be super awesome :slight_smile:

Ok. Maybe Three.js does not need to switch the Chrome flag… Any idea? I will try to find out

All the web libraries have the same capabilities in this regard. It is a browser limitation not a lib one.

Hi, I know it’s been a couple of years but I was taking a look at this example you put together and it doesn’t seem to be working. WebXR samples on this page for example seem to work without issue, but for your playground sample it doesn’t even load the camera. Any idea why that might be?

I’m using AR.js for an image tracking project as Safari doesn’t support WebXR… But it would be nice to have babylon.js + WebXR for the devices that do support it, as the tracking quality on AR.js is pretty bad!

Thanks!

If it doesn’t work I will be really happy to debug that! Can you share the playground tat isn’t working?

Thank you! It’s the playground you shared in your original reply to this thread: Babylon.js Playground

Seems to work for me. But I noticed you said the webxr samples do work for you? I assume you mean the AR ones?

Yes, the AR ones seem to work fine at that sample page. But the playground shows the Babylong.js loading symbol, then goes white for a few seconds, then goes to a blank purple background. It has the AR headset icon in the bottom right, but clicking that pops up “ERROR” over it.
The babylon.js AR Portal example does work, so AR etc. with babylon does definitely work on my phone.
Thanks for offering your help!

Can I ask if you have the WebXR incubation flag turned on in chrome://flags ? I am not sure TBH if it is still needed for image recognition, but it might be the issue here

Also - are you connected to a debug session? any errors in the console?

The flag was indeed the issue! I thought that image tracking was one of the “stable” parts of webxr but I guess not yet. Guess I can’t actually use it in production even for Android devices for now then :cry: I’ll have to stick with AR.js’ shaky tracking… And I wonder if Apple will ever get around to implementing WebXR on iOS…

Thanks so much for your help!

Yeah, i find it weird that it is still behind a flag. It has been like that for at least a year (if not longer)…

This is why I’ve paused my Baby AR project :slightly_frowning_face: raise the flags already, browsers!

I was looking for an AR image tracking solution for BabylonJS, but sadly it seems the WebXR flag business still isn’t sorted out.

However, I would like to make a strong case for MindAR. I have used it in several projects and the performance is great.
I had the same issues with jittery content, but you need to adjust the filterMinCF and filterBeta parameters of the mindar-image component. I currently use filterMinCF: 0.001; filterBeta: 0.01; which gives very stable and smooth results.
See Tracking Config | mind-ar-js

Too bad it’s a-frame/Three based, but until there is a decent solution for BabylonJS, this is a solid alternative.

cc @RaananW and @PirateJC as I guess we should contact them and see if they could help adding support for Babylon as well ?

Seems to be tensorflow-based, and very deeply integrating three.js/aframe. Would be interesting to see if they can add a different engine. Looking at the code ( hiukim/mind-ar-js: Web Augmented Reality. Image Tracking, Face Tracking. Tensorflow.js (github.com)) I can see the aframe and three.js integration, which seem to be separated. The tracker/detector sections are engine-agnostic.

It unfortunately seems that both mind-ar and AR.js (pretty much the most popular open source AR marker/image tracking solution) are not being developed/supported any more. Neither have had any commits / pull request approvals in 7-9 months, and mind-ar even has a couple of GitHub issues about it being abandoned. So likely not the best things to put effort into?

I would love an AR marker/image tracking solution for Babylon.js though. It’s unfortunately the one thing stopping me from switching fully over to Babylon for a 3D/AR university platform being developed… But with AR.js and mind-ar seemingly at the end of their updates there unfortunately don’t seem to be many other options :frowning:

I saw there used to be a BabylonAR project - I wonder if maybe that would be worth reviving one day… As seemingly, widely-available WebXR image tracking won’t be around for the foreseeable future.

BabylonAR has also been partly abandoned, mainly due to the promising progress of WebXR. And then, WebXR didn’t move forward as much as we wanted it to. I find it ridiculous that marker tracking is still behind a flag on chrome. I find it even more ridiculous that Apple doesn’t ever try to support it.

But I get the frustration. Of course - there should be a proper engine-agnostic marker-tracking solution, and it shouldn’t be that hard. so many projects have been doing that for so long already.

Yeah - even if Chrome did enable it by default there’d still be the big issue of Safari support. I was hopeful when they announced WebXR for their VR headset but so far there’s been no indication that they’re planning on also adding support on standard Safari. And as much as I favour Android unfortunately I can’t ignore iPhones when writing software for university use lol :sweat_smile:

It does surprise me that there aren’t more open-source marker-tracking solutions - it would seem like such an important thing to have, as currently good AR tracking is basically locked behind expensive APIs of a few select companies if available at all. But I guess there’s maybe not yet high-enough adoption of AR outside of WebXR supported headsets for there to be more than the couple that currently exist (AR.js and mind-ar), but of course both are unfortunately not maintained now… If I had the knowledge and time I’d look at whether it would be realistic to e.g. revive BabylonAR - of course neither of those things are easy to come by :laughing:

Hey guys,

I’ve been developing an open-source WebAR engine called encantar.js. It’s GPU-accelerated and features image tracking on any device. It might be just what you’re looking for :slightly_smiling_face: