Hi!
I’m working with the Babylon React Native project and I’m trying to convert screenspace coordinates to ‘real world’ Vector 3 coordinates.
I know I can get them converted through Scene.pointer to Vector3 or Screen xy to world coordinates or something similar and am having pretty good results on my test device. It’s not exactly in the right place, but I feel confident I can dial this in to fit right.
However, I was wondering that if I dail in the conversion on my test device, will this translate to other devices?
Or will other devices then have an offset since their camera could be slightly different, thus giving different outputs?
@HedwigAR can you describe a bit more what you mean when you say “it’s not exactly in the right place?” We are doing this with good results in a production app. There is only one case I know of where the behavior might be off (we are still investigating), and that is with any “Plus” model of iPhones. Which device are you testing on?
@ryantrem The not-in-the-right-place is on me, I’ll just elaborate what I’m currently trying.
I’m still figuring out a way to get a sort of image tracking working. And I thought maybe I could just use the react-native-camera qr code tracking. So I’m currently working on a little proof of concept for that.
So that’s basically what I have, a proof of concept. I can track a QR code and use that as a starting point.
Now, it’s not working well at all but it’s working enough to show the concept can or cannot work.
Unfortunately this approach doesn’t work well enough (so the swapping from react-native-camera to babylon xr), but the qr code tracking itself does. So I’m hoping that when the camera support is added (Babylon React Native snapshot without 3D objects - #3 by Peter) I could maybe use firebase ml (@react-native-firebase/ml - npm) to process the images or stream coming from that to do QR code tracking.
My question is basically if this has a chance of success and if the translation from screen to AR world is going to be consistent on different phones.
However, you saying you’re already doing this with good results in a production app answers that already, so thanks!
If it’s image tracking specifically that you are interested in, I would take a look at this issue and see if it sounds like the same thing you are looking for: WebXRFeatureName.IMAGE_TRACKING support · Issue #619 · BabylonJS/BabylonNative (github.com)
No one has had time to dig into that work, but it sounds like what you want then drop a comment there indicating your interest and use case. If you are comfortable writing C++, you could take a stab at implementing the feature as well. 
Hi @ryantrem, thanks for answering 
I am aware of that, and if I was able to add that feature myself, I would. But the closest to C++ I know is C#. You wouldn’t appreciate the interesting memory leaks I’d probably add when attempting it.
I also already wanted to have QR code tracking instead of image tracking anyway so I thought I’d just try if I can’t cook up something. I’ve tried jsQR (GitHub - cozmo/jsQR: A pure javascript QR code reading library. This library takes in raw images and will locate, extract and parse any QR code found within.) with the snapshot feature, but it’s not really quick enough and the Uint8ClampedArray is also kind of hard to parse to, jpeg-js (GitHub - jpeg-js/jpeg-js: A pure javascript JPEG encoder and decoder for node.js) is able to do it and give some results but it’s also synchronous and freezes babylon when it’s calculating.
I’m now looking to mlkit or the react native firebase implementation to see if that can maybe do this quick enough. Since they accept more image formats.
Anyhow, thanks again for getting back to me and answering so quick. It really helps!