Using WebXRImageTracking in a bundled React Native app

Hi!

I’m trying to use WebXRImageTracking in a bundled React Native app and running into an issue with getting an image source.
I’m using Image.resolveAssetSource to transform a number representation of an image to the uri needed for WebXRImageTracking. This is working fine in an unbundled app, I’m getting something like http://10.0.70.18:8083/assets/img/etcetcetc. However, in a bundled app, this same code is getting a img_pathtotheimageetcetc instead, which the WebXRImageTracking doesn’t seem to want.

It’s throwing a Error loading images for tracking, WebXRImageTracking disabled for this session.

I’m not getting a specific error, but I changed some stuff and got this more specific error: Error loading image with exception: Error: Image with provided source returned empty response.

Any idea what I can do to make this work?

Seems like the error is coming from here: BabylonNative/Image.cpp at 08cec4cacf4d187eb839b6318a7414860f8fde77 · BabylonJS/BabylonNative · GitHub

So it seems that it’s trying to parse the image in there, but it’s empty. Hardcoding a url does work in a bundled app, but how do I provide a local image (I do see it in the android project, so it is there) so that it doesn’t get seen as empty?

Maybe take a look at this issue which has information about how to load any asset from the bundle.

Hi! Thanks for the answer. I’ve looked through the link you sent, and some other threads on this topic. I’m not entirely sure what the exact solution is though? :sweat_smile:

I have a location of the bundled image, and when I put this in a normal image component it renders it. Is there another way to get the image uri in such a way that the image tracking will accept it? From what I’ve read a common solution is to convert the image to base64? But I don’t think that would work for this feature.

A require was also suggested, but that also doesn’t seem to get accepted by the image tracking as a src.

I’ve also tried adding the prefix file://, but that also gave an error.

Unfortunately, I don’t know since I haven’t investigated this myself. @Cedric/@ryantrem maybe can help.

I guess this happens on Android? Did you try it with iOS? It looks like the same issue with resources loading on Android in release build.

Hi! I just checked, and I don’t seem to have the same issue with iOS. Do you maybe know of a way to make this work on Android as well? Any way to convert the bundled image so that it is accepted by the image tracking?

You may get some tricks here : importing a .glb file fails if only local path is provided and not served through HTTP · Issue #165 · BabylonJS/BabylonReactNative · GitHub
For some projects, we used base64 encoded images. Not great but if you only have a few, it’s fine.

Hi! I saw that, but I wasn’t sure if base64 would work as src for the ImageTracking, I thought it had to be a url specifically. Did I misunderstand that?

No, you are right! base64 will not work with that. I’m adding an issue for that.

Thanks so much :smiley: if there’s any other way to do this I’d also be very open to that.

1 Like

Have you tried expo-asset as mentioned in the linked issue? It seems like other folks were having success using it for resolving uris for embedded/bundled assets.

1 Like

Hi! Yeah I saw that as well, but unfortunately we don’t have expo in the project so using expo-asset isn’t an option. I’ll take a look if there’s another way to transform the link.
What does expo-asset output that babylon does understand?

I haven’t tried it myself, but I believe expo-asset can be used as a standalone npm package in an existing react-native app: expo-asset - npm

Also, my understanding is that it correctly produces a valid local file path in all cases (debug/release for both iOS/Android).

I would suggest trying to experiment with it a bit and then share findings here.

Ah thanks for the tip, I’ll give that a try then!

Hi all!

This issue became higher priority lately in my project, thanks to all the advice I’ve managed to read the file with react-native-fs (there’s no expo in the project). I have it now as a base64, but I saw that the option to be able to use base64 images isn’t implemented yet and doesn’t seem to be a current priority (Base64 support for Canvas Image source · Issue #1227 · BabylonJS/BabylonNative · GitHub).

I haven’t been able to find a way to get an exact file path for stuff in the drawable folder. For normal use in images it’s apparently enough just to have the name of it in the drawable folder: Images · React Native

I have been able to use react-native-fs converting images in the bundle to textures like mentioned here, but I’m currently at a loss how to figure this out for WebXRImageTracking specifically.

I’ll continue working on it, but if anyone has figured this out before I’d be all ears :smiley:

This is how I managed it for normal textures, in case anyone comes across this:

import fs from "react-native-fs";
const resolvedImage = Image.resolveAssetSource(props.MyImage).uri;
fs.readFileRes(`${resolvedImage}.png`, "base64")
    then(base64Image => {
        setTexture(Texture.CreateFromBase64String(`data:image/png;base64,${base64Image}`, "texture", scene));
    })
    .catch(error => {
        console.error("Error: " + error);
    });

Oh! I’ve found a way around it, it’s not pretty, but I’m able to move the file to the cache folder and read it from there :woman_shrugging:

const resolvedImage = Image.resolveAssetSource(props.MyImage).uri;
const destinationPath = fs.CachesDirectoryPath + resolvedImage + ".png";
fs.copyFileRes(resolvedImage + ".png", destinationPath)
    .then(() => setImageURL(`file://${destinationPath}`))
    .catch(error => {
        console.error("Error: " + error);
    });

I think this’ll work as a workaround, but if anyone knows how to get the uri for the file in the drawable folder directly, I’m still very interested :smiley:

Does it work with Android on Release build?

Oh, that’s ominous :sweat_smile:, I’ve not built a release build on android before but I’ll give it a whirl.

1 Like