BabylonReactNative throws exception when adding anchor

I’m building a native iOS webxr experience, where we load a scene and a user is able to place the scene on the floor. Everything works in Chrome on Android :slight_smile:

Now we’re porting it to iOS, so we can show the same experience in an app clip. It seems their could be a bug when calling addAnchorPointUsingHitTestResultAsync, in rare occasions it does work but often we get an exception in:

Anchor System::Session::Frame::CreateAnchor(Pose pose, NativeTrackablePtr) const {
    return m_impl->sessionImpl.CreateAnchor(pose);
}

The exception:

sessionImpl -> com.facebook.react.JavaScript (15): EXC_BAD_ACCESS (code=1, address=0x0)

App.tsx:

/**
 * Generated with the TypeScript template
 * https://github.com/react-native-community/react-native-template-typescript
 *
 * @format
 */

import React, { useState, FunctionComponent, useEffect, useCallback } from 'react';
import { SafeAreaView, StatusBar, Button, View, Text, ViewProps, Image, Alert } from 'react-native';

import { EngineView, useEngine, EngineViewCallbacks } from '@babylonjs/react-native';
import { Scene, Vector3, ArcRotateCamera, Camera, WebXRSessionManager, SceneLoader, TransformNode, DeviceSourceManager, DeviceType, PointerInput, WebXRTrackingState, IMouseEvent, WebXRFeatureName, IWebXRHitResult, CreateTorus, Quaternion, WebXRHitTest, FreeCamera, WebXRAnchorSystem, IPointerEvent, PickingInfo, WebXRState, DirectionalLight, ShadowGenerator } from '@babylonjs/core';
import '@babylonjs/loaders';
import Slider from '@react-native-community/slider';

const EngineScreen: FunctionComponent<ViewProps> = (props: ViewProps) => {
	const defaultScale = 1;
	const enableSnapshots = false;

	const engine = useEngine();

	const [toggleView, setToggleView] = useState(false);
	const [camera, setCamera] = useState<Camera>();
	const [rootNode, setRootNode] = useState<TransformNode>();
	const [scene, setScene] = useState<Scene>();
	const [xrSession, setXrSession] = useState<WebXRSessionManager>();
	const [scale, setScale] = useState<number>(defaultScale);
	const [snapshotData, setSnapshotData] = useState<string>();
	const [engineViewCallbacks, setEngineViewCallbacks] = useState<EngineViewCallbacks>();
	const [trackingState, setTrackingState] = useState<WebXRTrackingState>();

	useEffect(() => {
		if (engine) {
			const scene = new Scene(engine);
			console.log('Hallo');
			setScene(scene);
			// scene.createDefaultCamera(true);
			// (scene.activeCamera as ArcRotateCamera).beta -= Math.PI / 8;
			var camera = new FreeCamera('webxr-camera', new Vector3(0, 1, -5), scene);
			camera.setTarget(Vector3.Zero());
			camera.minZ = 0.01;


			setCamera(scene.activeCamera!);
			scene.createDefaultLight(true);
			const rootNode = new TransformNode('Root Container', scene);
			setRootNode(rootNode);

			const deviceSourceManager = new DeviceSourceManager(engine);
			const handlePointerInput = (event: IMouseEvent) => {
				if (event.inputIndex === PointerInput.Move && event.movementX) {
					rootNode.rotate(Vector3.Down(), event.movementX * 0.005);
				};
			};

			deviceSourceManager.onDeviceConnectedObservable.add(device => {
				if (device.deviceType === DeviceType.Touch) {
					const touch = deviceSourceManager.getDeviceSource(device.deviceType, device.deviceSlot)!;
					touch.onInputChangedObservable.add(touchEvent => {
						handlePointerInput(touchEvent);
					});
				} else if (device.deviceType === DeviceType.Mouse) {
					const mouse = deviceSourceManager.getDeviceSource(device.deviceType, device.deviceSlot)!;
					mouse.onInputChangedObservable.add(mouseEvent => {
						if (mouse.getInput(PointerInput.LeftClick)) {
							handlePointerInput(mouseEvent);
						}
					});
				}
			});

			const transformContainer = new TransformNode('Transform Container', scene);
			transformContainer.parent = rootNode;
			transformContainer.scaling.scaleInPlace(0.2);
			transformContainer.position.y -= .2;

			scene.beforeRender = function () {
				transformContainer.rotate(Vector3.Up(), 0.005 * scene.getAnimationRatio());
			};

			SceneLoader.ImportMeshAsync('', 'https://raw.githubusercontent.com/KhronosGroup/glTF-Sample-Models/master/2.0/BoxAnimated/glTF-Binary/BoxAnimated.glb').then(result => {
				const mesh = result.meshes[0];
				mesh.parent = transformContainer;
			});
		}
	}, [engine]);

	useEffect(() => {
		if (rootNode) {
			rootNode.scaling = new Vector3(scale, scale, scale);
		}
	}, [rootNode, scale]);

	const startXR = () => {
		(async () => {
			console.log('Start XR');
			if (!scene || !rootNode) return;

			rootNode.getChildMeshes().forEach(m => m.isVisible = false);
			rootNode.rotationQuaternion = new Quaternion();

			const xr = await scene.createDefaultXRExperienceAsync({ disableDefaultUI: true, disableTeleportation: true });
			console.log('XR')

			const fm = xr.baseExperience.featuresManager;
			const xrTest = fm.enableFeature(WebXRFeatureName.HIT_TEST, 'latest')as WebXRHitTest;
			const xrPlanes = fm.enableFeature(WebXRFeatureName.PLANE_DETECTION, 'latest');
			const anchors = fm.enableFeature(WebXRFeatureName.ANCHOR_SYSTEM, 'latest')  as WebXRAnchorSystem;
			const xrBackgroundRemover = fm.enableFeature(WebXRFeatureName.BACKGROUND_REMOVER, 'latest');

			var isPlaced = false;

			var marker = CreateTorus(
				'marker',
				{
					diameter: 0.15,
					thickness: 0.05,
				},
				scene
			);
			marker.rotationQuaternion = new Quaternion();
			marker.isVisible = false;

			var hitTest: IWebXRHitResult;
			xrTest.onHitTestResultObservable.add((result) => {
				if (result.length) {
					marker.isVisible = !isPlaced;
					// marker.isVisible = true;
					hitTest = result[0];
					marker.position = hitTest.position;
					marker.rotationQuaternion = hitTest.rotationQuaternion;

					rootNode.position = hitTest.position;
					rootNode.rotationQuaternion = hitTest.rotationQuaternion;
				} else {
					marker.isVisible = false;
				}
			});

			if (anchors) {
				anchors.onAnchorAddedObservable.add((anchor) => {
					console.log('Anchor added');
					rootNode.getChildMeshes().forEach((m) => (m.isVisible = true));
					// model.isVisible = true;
					// model.isEnabled(true);
					anchor.attachedNode = rootNode;
					// shadowGenerator.addShadowCaster(model.attached)
				});
			}

			scene.onPointerDown = (evt: IPointerEvent, pickInfo: PickingInfo) => {
				if (hitTest && anchors && xr.baseExperience.state === WebXRState.IN_XR) {
					console.log('hittest', hitTest);
					console.log('session.inXRSession', session.inXRSession);
					anchors.addAnchorPointUsingHitTestResultAsync(hitTest);
					marker.isVisible = false;
					isPlaced = true;
				}
			};
		
		

			const session = await xr.baseExperience.enterXRAsync('immersive-ar', 'unbounded', xr.renderTarget);
			session.onXRSessionInit.addOnce(() => {
				console.log('Session Init');
			})
	})();
	}

	const onInitialized = useCallback(async (engineViewCallbacks: EngineViewCallbacks) => {
		setEngineViewCallbacks(engineViewCallbacks);
	}, [engine]);

	const onSnapshot = useCallback(async () => {
		if (engineViewCallbacks) {
			setSnapshotData('data:image/jpeg;base64,' + await engineViewCallbacks.takeSnapshot());
		}
	}, [engineViewCallbacks]);

	return (
		<>
			<View style={props.style}>
				<Button title="Start" onPress={ () => startXR() } />
				<EngineView camera={camera} onInitialized={onInitialized} displayFrameRate={true} antiAliasing={2} />
			</View>
		</>
	);
};

const App = () => {
	const [toggleScreen, setToggleScreen] = useState(false);

	return (
		<>
			<StatusBar barStyle="dark-content" />
			<SafeAreaView style={{ flex: 1, backgroundColor: 'white' }}>
				<EngineScreen style={{ flex: 1 }} />
			</SafeAreaView>
		</>
	);
};

export default App;

I mentioned it in a discussion about a problem I had starting XR on ios, I don’t think these are related. The XR issue is solved :slight_smile:

pinging @BabylonNative

Does it repro easily? Can you repro with a small sample?

I forked the BabylonReactNative repo, and configured it like this:

Started by installed dependencies

npm ci
npm run select 0.71

Then I build the ios workspace with ruby 3.3.1 using chruby

gem install cocoapods

Opened the project in xcode and build it. Which gave me two issues. After some reading it seems related to a my version of xcode (13.4) and the version of react native (0.71). Is solved in 0.74.

  1. No template named 'unary_function' in namespace 'std'; did you mean '__unary_function'?:
    Changed it to: __unary_function
  2. Error in FlipperTransportTypes:
    Added #include <functional>

I’ve only tested it for 0.71, and here the issue seems quite consistant.
After pressing the “start” button a simple “marker” is shown, which locate the floor plane. When pressing the screen the model is added at the anchor. Which results in the exception.

If it does work the first time it does crash after pressing the screen again, which tries to create a new anchor.

Please let me know if I can do anything to help. I do feel a bit guilty, submitting issues. I really love your commitment and the BabylonJS is a blast!

I remember having the __unary_function build issue but I don’t remember the details. I’ll try to repro the crash in the coming days.

1 Like

Your are a true hero!

I was able to solve the build issues. I think they are related to a combination between xcode 13.4 and react native 0.71. I tried to setup a react native project with version 0.74, and the issues was solved, but not sure how to set this up with Babylon react native (this is another topic)…

However I don’t think my problem with adding anchors throwing EXC_BAD_ACCESS is related though.

I’m working on an updated version of Babylon Native inside BRN.
My branch is here : Update BN and BJS version to 7.6.2 by CedricGuillemet · Pull Request #645 · BabylonJS/BabylonReactNative · GitHub
Can you please test it on iOS and let me know if it fixes your issue?

I testet the branch and I still get the same exception, when I try to add an anchor with a call to:

addAnchorPointUsingHitTestResultAsync

I did however find another issue. In the newest branch of BabylonNative I found this issue where one of the boxes in the glb BoxAnimated.glb is drawn inverted. Should I file this issue in the BN repo?

The blue box have a hole which matches the red box.


In the newest version og BabylonNative the inside of the blue box is drawn instead:

no need to open an issue for that. it was a known problem fixed with Support non-normalized, non-float vertex buffers and unaligned vertex buffers by bghgary · Pull Request #1382 · BabylonJS/BabylonNative · GitHub
The BRN PR has been updated this morning.

That was effective :clap:

yes but the add an anchor issue is still there :slight_smile:

Good news: I can repro…now looking for a fix!

3 Likes

It’s a bit more clear about what happens. No valid fix yet but I’ve opened a GH issue to write down my investigations.

2 Likes

Hi @dnrn
I may have a valid fix : Fix XR Frame alloc by CedricGuillemet · Pull Request #1386 · BabylonJS/BabylonNative · GitHub
Tested w/o crash on my side. Let me know how it runs

2 Likes

I works!

Just pulled and tested. I do not get the exception anymore, and is able to add anchors.
Thanks a lot!

1 Like

Hi @dnrn
Sorry to bug you again. I’ve updated the PR with a simpler fix. No issue/crash on my side. Can you please test it as well? Thanks in advance!

It works :slight_smile:

And no problem bugging me. I’m glad I can help, it’s an amazing work you do!

@dnrn really sorry, this fix takes ages to merge because of simplification, refactor and such can you please try the fix again? Fix XR Frame alloc by CedricGuillemet · Pull Request #1386 · BabylonJS/BabylonNative · GitHub

@Cedric Just testet it, and it works :slight_smile:
Thanks for your great work!