"Near dragging" a mesh in immersive-vr with and without SixDofDragBehavior

Questions
1

I’d like to let user grab a mesh and drag/turn it around in a immersive-vr (webxr) session, “near interaction” style.

User approaches the mesh, stretches out their hand (with or, if their system supports hand tracking, without a controller), pinches (selectstart event) when their hand is close enough to the mesh, and then they can move/rotate the object until they unpinch (selectend event). That’s more or less what I would like to enable.

I use an OculusBrowser for testing, and xr-near-interaction feature is enabled in the session (default WebXRDefaultExperience behavior).

Adding SixDofDragBehavior to a babylon-native (i.e. created by calling a babylon/core method at runtime) mesh does what I look for. I need that to also work with non-native meshes: models imported from GLB.

Adding SixDofDragBehavior to meshes imported from GLB does not result in “near grabbing” to work (although does result in setting mesh.isNearGrabbable = true on the meshes).

This playground works around some other SixDofDragBehavior issues (discussed there; probably unrelated to webxr and near-dragging). In an immersive-vr session we can drag all those meshes around with a ray, but we can only “near drag” the babylon-native mesh (the “crate” box covered with the wooden-looking texture) and not the ones imported from GLB.

First question: Is near-dragging expected to work when we add SixDofDragBehavior to any mesh? If yes, then … is it a bug that it doesn’t with GLB-imported meshes? If no, what other setup do I need to do for those?

Second question: Is SixDofDragBehavior necessary to enable the near-dragging of a (GLB-imported)? Are there other (simpler and more minimalistic) way(s) of setting it up? Setting mesh.isNearGrabbable = true and starting a immersive-vr session with xr-near-interaction feature enabled doesn’t seem to be enough, not even with a babylon-native mesh.

I must have missed jsdoc, documentation and forum posts that explain this clearly… Maybe @RaananW can help? :pray:

3

Yes! it is expected to work. Just as it works when dragging your mouse on the desktop. I’ll have to debug and see what happens there.

Nope! There are other ways of doing that. The gizmo is just a helper. A very simple way of doing that is to parent the mesh when the pointer down selects them, and un-parent when pointer up is triggered. setting the meshes to be near-pickable will allow you to get the pointer events trigggered on them as well, so that you can filter the actions correctly in onPointerDown

Small note - I noticed you mentioned selectstart - we don’t use these events directly, we emulate pointer events in babylon. So a pointer down will be triggered when you pinch, and a pointer up will be triggere when you let it go.

4

Thank you. I know you guys must be busy with getting the version 7.0 out, so not expecting miracles, but you @RaananW surprised me positively before, so hoping for a fix sometime soon :slight_smile:

We did that a while ago; that does what we want when a motion controller is used but unfortunately not when a “hand controller” (hand tracking) is used. More specifically, when we parent the grabbed mesh to the controller (see pseudo code of exactly what we do below), the grabbed mesh moves/rotates in sync with user’s forearm movement AND (importantly for us) in sync with user’s hand movement and rotation. That is what we want and that does happen when a (Quest) motion controller is used.

However, when controller that’s grabbed the mesh happens to be a hand, the grabbed mesh still moves in sync with user’s forearm movement (which is good) but it does NOT move/rotate in sync with user’s hand (wrist). And we need that. That is why I am onto the “near grabbing” hunt here…

Just in case; here is how we parent the meshes:

const event = ...;
const defaultXRExperience = ...;
const __root__mesh_imported_from_GLB = ...;

const controller =
    defaultXRExperience.pointerSelection.getXRControllerByPointerId(
        event.sourceEvent.pointerId);

__root__mesh_imported_from_GLB.parent = controller.pointer;

If you see something missing there :point_up_2: in order to get the mesh to move/rotate in sync with user's _hand_ (wrist) movement/rotation, can you point that out for us?..

Thank you. I see. And you also said that before. Sadly, after reading that and this I still don’t understand enough about it in order to actually use (you see me ask the basic questions like whether setting mesh.isNearGrabbable = true is enough to enable “near grabbing”, etc).

Can we make that clear once and for all? Would you mind clarifying exactly what mesh.isNearGrabbable = true does and how concretely a BSJ user can make use of it? Including

  • that small white sphere that appears (probably indicating that “near interaction proximity is achieved”) when a controller is brought near a isNearGrabbable mesh in space
  • exactly how setting isNearGrabbable affects firing those synthetic pointer events that BJS fires on meshes,
  • all other things that setting isNearGrabbable does,
  • and how one can harness that to implement useful interactions?

Huge thanks, as always :pray:

1 Like
5

First - the playground:

SixDofDragBehavior repro | Babylon.js Playground (babylonjs.com)

The issue is that the children of the root node are not marked as near-grabbable, so it did not work. Setting them all works. I will see if I can push this as a fix for 8.0. Otherwise, will be a fix right after the release.

There is an interesting issue with hand near-interactivity that causes the meshes to change their scaling every time i interact with them if near and regular pick is enabled. Again - working on a fix, but not sure we can merge it before 8.0 is released.

And to your questions :slight_smile:

Let’s sort out grabbable and pickable -

Grabbable is used when you grab an object. So - a touch is not enough, you need to be close and actually use your controller to interact with it. So, in the case of the 6DoF gizmo, picking will do nothing - i can touch the mesh with my finger/controller as much as I want - until I use the trigger or squeeze component, the gizmo will not trigger its actions.
Pickable is - events are triggered when you touch the component. Think - GUI elements like the color picker. In this case there is no need for an event (a WebXR event) to trigger a pointer event in Babylon. There also has a hover-action, which you can see very clearly with the controller (it is not available with hands). The hover event is not entirely configurable, but if anyone will ever ask, i can expose it.
Also note - when you grab, no pick events will be triggered. so you are touching until you are grabbing, and when you stop grabbing, you are (probably) still touching.

And last thing - i totally agree, there needs to be more on near-interactivity that this - Babylon.js docs. I’ll add an issue for me to expand on that.

The disc is, as you think, the near interaction point. When using controller you will also see the hover sphere, which will tell you at which point the controller interacts (because it is not so obvious with a controller like it is with a hand - a hand has a finger, but what does a controller have?..)

Both of those types of event will trigger a pointer event,

6

Thanks, this helps. Let me guess the details of what you are saying; please correct whatever I guess wrong:

  • (this you are not saying but it’s relevant) right now SixDofDragBehavior sets isNearGrabbable = true only on the mesh to which the particular SixDofDragBehavior instance is added as a behavior.
  • you will at some point land a fix that will ensure that SixDofDragBehavior, when added to mesh X, will also set isNearGrabbable = true on all child meshes (recursively) of that mesh X
  • until that’s committed/released, you propose that I and other users can manually set isNearGrabbable = true on all the child meshes when we use SixDofDragBehavior

I am assuming the fix you are thinking will be specific to SixDofDragBehavior. I wonder if this is correct?..

7

Yep, I saw that one. I also saw different wacky movement of meshes when dragged, both with a motion controller and with a hand (but wacky in different ways with these two).

I think it makes sense to let you resolve those issues you already found; after that I will be happy to test this stuff again and report any remaining (or new) issues with dragging in immersive-vr

8

100% correct! :slight_smile:

I am also working on fixing the multipointer issue. I already found a solution for the weird scaling, and am now working on fixing multipointer positioning.

1 Like
9

I appreciate you clarifying this, but this wasn’t one of the things I, personally, have been misunderstanding. That’s exactly what I thought should be happening. Some of the things I am still confused about are

  • whether just setting isNearGrabbable = true on a mesh (without adding a SixDofDragBehavior or other behaviors, register actions, do anything else special to that mesh) actually makes it “near grabbable”, i.e. whether user moving a hand/controller close to that mesh and squeezing – results in the mesh getting grabbed.
    • by now I suspect strongly that the answer to that question is negative (i.e. adding a SixDofDragBehavior is also necessary)
    • but you can probably appreciate how extremely natural it is for a user to expect that setting mesh.isNearGrabbable = true makes that mesh “near grabbable” and how therefore unexpected it is to find out that it’s not how it actually works. If you do appreciate that, and don’t see a way to change the API/naming (because public API, etc), let’s try to write about this all over documentation so more users find out before they waste lots of time.
  • literally, what changes about the mesh, its behavior, events that are fired in relation to that mesh, etc – once mesh.isNearGrabbable = true is set.
10

I get what you are saying. However, these flags are just a setters for other behaviors to decide what to do with them. The same thing goes to isPickable. If there is no callback registered, setting isPickable will mean nothing. Maybe not nothing - it means that when the predicate executed, this mesh will be selected, if some other component is looking for pickable meshes.

isNearGrabbable is used in the predicate. This predicate decides if, for this specific mesh, a pointer event will be triggered if it was grabbed, or picked. It is important to differentiate, otherwise we have no way of knowing in what case the poitner event needs to be triggered.

11

This. Exactly.
And the rest of that comment is great. Please include that in jsdoc AND deepDive pages AND anywhere else where a user might possibly land as they try to make some near interactions working.

You see, right now jsdoc for AbstractMesh#isNearGrabbable literally says Gets or sets a boolean indicating if the mesh can be near grabbed. That’s simply false and very misleading; at least as far as I’m concerned: as a BJS user I set it on a mesh, and user of my application cannot near-grab that mesh :person_shrugging:

It is great to explain that it’s just a property. Any code may choose to use it for any purpose. Certain babylon/core (“built in”) components/behaviors (good to mention/reference specific ones, but also you probably want to be clear that the list of examples is not exhaustive) happen to use that property to [… explain to do what exactly, at a lower level like firing certain pointer events in response to certain specific other things happening…] and, when certain specific other things are also done (such as adding SixDofDragBehavior to the same mesh and its children, such as doing other things – give a few more examples if possible) that results, at a higher level, in certain specific user-perceivable behaviors.

Something like that would be super helpful :pray:

1 Like
12

while we are on this… a little while ago I noticed how SixDofDragBehavior#dispose() simply sets (this._ownerNode as Mesh).isNearGrabbable = false; and I thought that is actually problematic: imagine a mesh with mesh.isNearGrabbable set to true. It sits and bothers no one (and perhaps some babylonjs/core or user’s code uses that that value to do some magic). Then we add SixDofDragBehavior to that mesh. Later we remove that behavior from that mesh. Unexpectedly, mesh.isNearGrabbable === false :person_shrugging:

1 Like
13

makes total sense. It will also make sense not to turn it on per default on all meshes, however this ship has already sailed :slight_smile:
I will fix the dispose.

I assume it’s a matter of semantics here. Maybe it should be “Gets or sets whether or not the mesh is grabbable” or something similar.

14

I’d say that isn’t any better. I guarantee you that many people will read that and think that’s all they need to set for the interaction to work for the user.

Instead I’d love you to start that jsdoc article from saying that …

… it’s just a property. Any code may choose to use it for any purpose. Certain babylon/core (“built in”) components/behaviors (good to mention/reference specific ones…

and then go on and explain what (babylonjs/core) code does use it for what specific purpose(s) and then add that in the future other babylonjs/core code may use it for certain unknown-but-approximately-known other purposes, and that the BJS users can also use it for whatever they see fit.

By the time you explain all that, people will know what it is.

I personally prefer to see much of that in jsdoc, and only more extensive examples on some other docs article (referenced from jsdoc). but that’s a matter or style and consistency…

I think you get my point; i’ll stop now :slight_smile:
Many thanks :pray:

15

:slight_smile:

This is the same as pickable. I guess a proper page on the docs would make more sense. I am working on one :slight_smile:
But I totally get what you are saying. Maybe naming was not very good with this one.