I’m here to get help understanding a weird behavior I noticed about WebXR augmented reality (AR) on the react web app I am currently working on.
On the app, the idea is to enter in AR mode to display a 3D object inside it. In AR, I use hit-test, anchors and dom-overlay features to place the 3D object on the floor and provide interactions.
It works fine at start but the problem is that I experience some weird behavior when I exit AR mode and go back again in it without reloading the page.
For example, the objects appearance seems different and the hit test marker that I use to display where the object will be placed is now badly oriented. I also have a plane with a texture that I animate with the renderloop (make it translate on y axis) and it is then translating on other weird random axis depending on the case. I also experience performance issues in the “standard” 3D scene once the first exit is done.
However, when I reload the page, all works fortunately fine again for the first AR launch.
After some research on my side, I tried the babylonjs hit test example playground in the docs and found out that I experienced similar problems on it too: https://playground.babylonjs.com/#XWBES1
Here, when I enter in AR mode, all works fine and the hit test is well displayed on the floor. Next, I go out of the AR session with the back button on my phone and then, when I enter again in AR without reloading the page. The hit test marker seems to be upside down (or maybe the shading is or something else).
I also noticed that in the first example of the docs with the sphere in AR mode (https://playground.babylonjs.com/#F41V6N#32) that the sphere does not appear in the second launch of AR mode.
I know it’s quite vague but I don’t know what suppositions could be relevant here and if it is a real bug or a misunderstanding on my side.
Could it be some assets, values, etc that are not well disposed or reset at exiting ? Or could it be related to the phone used ? the optional features behaviors ? I’m a bit confused here
I’m still discovering the framework so I’m probably missing something. Do you have any idea of what is happening here, or some things I could do to find what the real problem is ?
It’s not running on an official build of babylonj.js, but I will update that today as well and if you can reproduce the same behaviour there would be great. There are lifecycle events when leaving and re-entering AR mode that the features need to handle correctly.
This scene is not being cleaned up when quitting XR, so it is somewhat expected that it will not work as expected if you placed an object in XR. otherwise, it should work every time you go in.
I’ll debug this and let you know if I find something. If you enter and exit your scene and things don’t work, it’s clearly a bug and we need to squash it.
I did a bit of experimentation. Seems like the y-coordinates get a bit crazy and then you see that. I’m on a Pixel 4 Android 11. I’m using the DOM overlay to show current active camera position.
Can you try this code and see how it goes for you? setText can log to console if you are using device inspector in chrome://inspect/#devices or you can write to DOM overlay as I have in screenshot above.
const textHandle = { current: null };
xr.baseExperience.onStateChangedObservable.add((webXRState: WebXRState) => {
if (textHandle.current !== null) {
clearInterval(textHandle.current);
}
textHandle.current = setInterval(() => {
setText(`pos:${scene.activeCamera?.position.x.toFixed(2)},${scene.activeCamera?.position.y.toFixed(2)},${scene.activeCamera?.position.z.toFixed(2)}`);
}, 1000) as any as number;
});
I was getting some slightly negative numbers, but you can see it went to -15K. What I would try next is to track the torus position and light direction and then it would probably click what is going on. Perhaps you are looking up up at the torus that is from below and lit from above?? I think it may have to do with the transfer of camera positions and then offsetting, but it looks like AR has a different mechanism than VR and I haven’t looked at that before. Sorry, but I am on the slopes for next 3 days - will try to connect when back on Monday. Hopefully that is enough and somebody else will pick up the baton or enough clue for you to see what is happening.
Thank you for your quick answers and your support.
I effectively experienced the y coordinates shift but don’t figure out what the problem behind it is.
I’ll try to make a playground that makes something similar to my app to test it with a model. I’ll keep you updated if I find something too
That seems very wrong. Those positions should stay the same every time we enter AR. It should depend on the reference mode that you set, but this might have an effect on the Y axis, Z and X should technically stay the same. Well, at least not go to 15k.
When exiting AR (made with the back button of the phone, don’t know if it could be relevant to know), it does not go through the EXITING_XR state and the model sometimes looses its color in the classic scene.
When re-entering AR, the model is weirdly rotated around the hitTest point during the hit test phase. The y position coordinate of the model node (logged in the console when positioning it on an anchor) is also different compared to the first try. It seems to go further in a direction at each try to re-enter in AR mode (without refreshing the page). However, the model is (or at least seems to be ) well placed on the ground at the good height.
It’s quite similar to what I experience in my app.
Let me know if I made something wrong or missed something.
Thank you so much for the reproduction! And the wonderful explanation. Helped me understand what’s going on much faster
PR is here - [XR] reset AR camera’s transformation on XR init by RaananW · Pull Request #11723 · BabylonJS/Babylon.js (github.com) .
The quick explanation - when creating the experience helper, the xr camera’s transformation is 0 (rotation and position), but it is kept between sessions for the AR camera. so the transformation from which you exited the session is now the initial transformation. Works wonderfully for the XR features (because they don’t really care about your camera transformation), but doesn’t work for elements in “babylon” world space, as they are adjusted to the default coordinate system.
Also fixed the exiting XR state change. thanks a lot for that!
I have a last little question. I’m currently using the npm package for Babylon in version “5.0.0-alpha.22” for my web app so do you know if the package is automatically updated and the fix is already on it (and working after an npm install) ?
Or do I have to wait or do something else to have it working on my side once updated ?
the latest 5.0.0 release (i believe it’s the first beta version) includes the fix. So you will need to update to latest preview in order to use the fix
I noticed that some little fps losses are still occuring sometimes (after XR exited at least once) but it is probably related to my way of setting things up.
I think it may have to do with the transfer of camera positions and then offsetting, but it looks like AR has a different mechanism than VR and I haven’t looked at that before. Sorry, but I am on the slopes for next 3 days - will try to connect when back on Monday. Hopefully that is enough and somebody else will pick up the baton or enough clue for you to see what is happening.
