Hello All,
I am trying to pick an object using Babylon Native on iOS.
My goal is to pick the object in an AR environment but also without AR I am not able to make this work.
Have anyone succeded to pick meshes under touch in babylon native and AR?
Thank you very much.
Here’s is everything I’ve tried in the past days but nothing seems to works on Native.
None of the logs is ever written.
scene.onPointerObservable.add((pointerInfo) => {
switch (pointerInfo.type) {
case BABYLON.PointerEventTypes.POINTERDOWN:
BABYLON.Tools.Log("onPointerDown");
break;
case BABYLON.PointerEventTypes.POINTERUP:
BABYLON.Tools.Log("onPointerUo");
break;
case BABYLON.PointerEventTypes.POINTERMOVE:
BABYLON.Tools.Log("onPointerMove");
break;
}
});
scene.onPointerDown = (evt, pickInfo, type) =>
{
BABYLON.Tools.Log("onPointerDown");
if ( anchors && pickInfo) {
anchors.addAnchorPointUsingHitTestResultAsync(xrHitTestModule);
}
};
scene.onPointerPick = (evt, pickInfo) =>
{
BABYLON.Tools.Log("onPointerPick");
if ( anchors && pickInfo) {
anchors.addAnchorPointUsingHitTestResultAsync(xrHitTestModule);
}
};
scene.onPointerObservable.add( pointerInfo =>
{
BABYLON.Tools.Log("onPointerObservable");
});
Hi Peppe.Belgenio,
Welcome to Babylon! If you’re using Babylon Native by itself (as opposed to Babylon React Native), input may not be hooked up to work the way you’re expecting yet. (This is a work-in-progress.) Are you using the Babylon Native Playground app? If so, experience.js provides an example of how to manually work around rudimentary input handling short-term. Longer-term, more seamless and sophisticated input handling is coming; it just hasn’t been implemented quite yet. Will the InputManager approach demo’d in experience.js be enough to unblock you for your current scenario?
Thank you syntheticmagus for the quick answer.
Yes, I tried to workaround using a custom implementation of the InputManager but there are
still two big problems at the current stage:
The click is only detected after moving a little the finger on the screen and furthermore I would also need the clicked object under the touch .
This is what I tried but the “click” is not detected before a small drag on the screen(I guess a couple of renders) and no object is picked up nor added with “addAnchorPointUsingHitTestResultAsync”:
function CreatePickHandling(scene,anchors,xrHitTestModule)
{
var inputManager = new InputManager();
scene.onBeforeRenderObservable.add(function() {
if(anchors && xrHitTestModule && inputManager.isPointerDown)
{
anchors.addAnchorPointUsingHitTestResultAsync(xrHitTestModule);
}
});
}
Anchors and xrmodule are defined as follow:
const xrHitTestModule = xr.baseExperience.featuresManager.enableFeature(
BABYLON.WebXRFeatureName.HIT_TEST, "latest", {
offsetRay: { origin: { x: 0, y: 0, z: 0} , direction: { x: 0, y: 0, z: -1}},
entityTypes: ["mesh","plane"] });
...
const anchors = xr.baseExperience.featuresManager.enableFeature(BABYLON.WebXRFeatureName.ANCHOR_SYSTEM, "latest");
...
if (anchors) {
console.log('anchors attached');
anchors.onAnchorAddedObservable.add(anchor => {
model.isVisible = true;
anchor.attachedNode = model.clone("clone");
model.isVisible = false;
});
anchors.onAnchorRemovedObservable.add(anchor => {
console.log('disposing', anchor);
if (anchor) {
anchor.attachedNode.isVisible = false;
anchor.attachedNode.dispose();
}
});
}
I don’t have a complete answer for both issues, but at least the first one I should be able to help with. A delay between input and reaction is expected with InputManager's current implementation because it’s quite naive (really only built for debugging, we didn’t expect it to get real use). In order for input to be processed by JavaScript, information about the input must move from where the program receives it on the main thread into the JavaScript thread. The InputManager does this by naively Dispatching a setter to the JsRuntime; this setter will apply the new value once it’s run, but because it goes to the back of the work queue (on which all the rest of the JavaScript work must also run), that’s usually not for a frame or two. This delay becomes more noticeable when framerate is low.
In theory, if you’re seeing multi-frame delays, this behavior could be changed by modifying the InputManager's behavior to not use Dispatch anymore. Instead of Dispatching from the main thread, just set the InputManager's state directly, using a mutex to make the operation thread-safe. I put up a branch with a quick implementation of this which I tested locally (to commit this would require more changes, but this is sufficient to try). On Windows, I did not observe a meaningful difference between this approach and the prior one, presumably because my framerate was already quite good; but if you’re experiencing multi-frame input delays, you might try this approach out and see if it helps. If this improves things for you, I can likely turn this into a change that we could check in.
As for XR hit tests on iOS, that’s unfortunately not something I have any direct knowledge about. @ryantrem might know more. Ryan, does the code pasted above look like it should work with Babylon Native as it’s implemented today? Thanks!
If you can’t use Babylon React Native, but you are on iOS, then you should use Babylon Native’s NativeInput class. This is not used in the Playground app right now because we currently only have support for iOS, Android, and UWP (not other platforms). The InputManager you see in Playground is just for quick testing - it is a completely separate “input manager” from the Babylon.js API. NativeInput on the other hand fully routes through the Babylon.js input system so everything should work as expected, including XR input and anchors. To use NativeInput in your own custom Babylon Native app, there are just a couple steps:
- Initialize
NativeInput when other Babylon Native components are initialized, and store the NativeInput instance. For an example, here is how we do it in Babylon React Native: BabylonReactNative/BabylonNative.cpp at master · BabylonJS/BabylonReactNative (github.com)
- Handle raw input (e.g. touch input) from the
MTKView you are creating in your app and forward it to the NativeInput instance. For an example, here is how we do it in Babylon React Native: BabylonReactNative/EngineViewManager.mm at master · BabylonJS/BabylonReactNative (github.com). Note this goes through an intermediate layer (BabylonNativeInterop) which in your case you could skip (go directly to your NativeInput instance).
Again, if you use Babylon React Native, this is all set up for you automatically, but if you need to use Babylon Native directly then you’ll need to set something like this up. Later we will replace the InputManager test code with the actual NativeInput class, but probably not until NativeInput supports all the platforms generally supported by Babylon Native.