SixDofDragBehavior: different behavior with different meshes: native, GLTF, GLB

Questions
, , , ,
1

Update: initially this post included details and questions regarding the behavior in the immersive-vr (webxr) session as well as the non-immersive “canvas on screen” mode. It was later repurposed to discuss just the set of questions/issues with SixDofDragBehavior vs GLTF vs scene.useRightHandedSystem = true. A separate post was created later to discuss immersive-vr specific stuff. Now to the original post:

In this playground I add separate instances of new SixDofDragBehavior() to three meshes:

SixDofDragBehavior exhibits different behavior with all three; it seems to work “perfectly” with the babylon native box, but not with the other two.

Can someone please help explain the behaviors I am seeing and give guidance on how to go about understanding and debugging stuff (and in particular SixDofDragBehavior) with various GLTF/GLB models (with different structure of meshes, transform nodes, etc).

Note: I add SixDofDragBehavior to the __root__ mesh of the imported models. Doing otherwise produced wild behaviors that I found even harder to explain… So I stopped trying that.

Here are the behaviors I am seeing.

  • with the native babylon Box – things seem to work:
    • dragging works as expected with the ray in the normal (not the immersive-vr) mode
    • dragging works as expected with the ray in the immersive-vr mode
    • near-dragging activates and works as expected in the immersive-vr mode
  • with the BoomBox.gltf:
    • dragging works as expected with the ray in the normal mode
    • dragging works as expected with the ray in the immersive-vr mode
    • near-dragging does not activate/work in the immersive-vr mode even though mesh.isNearGrabbable is set to true as a result of adding the SixDofDragBehavior
  • model in the GLB format:
    • dragging almost works as expected with the ray in the normal mode: when grabbing the mesh to drag it, it first rotates 180 degrees (which is unexpected) and then dragging works fine until I release the mouse button
    • in the immersive-vr dragging with a ray works the same as in the non-immersive mode: the mesh rotates 180 degrees at the beginning of each dragging
    • near-dragging does not activate/work in the immersive-vr mode even though mesh.isNearGrabbable === true

How can I explain (and then amend) the behaviors I highlighted with bold above?

I suspect that the structure/contents of the GLB file is a part of the explanation. As you can see, __root__'s only child is a transform node and that in turn has two more transform nodes (for whatever reason with such a simple model…) and only one of those has a mesh child:

image
image762×766 53.9 KB

but I still have little idea where to start looking for the specific bits that influence UX, and which of those can/should be fixed in the GLB/model internals and which might need to be fixed in SixDofDragBehavior…

2

cc @Cedric

3

I suspect the scaling/rotation that converts right handed gltf to Left handed scene.
Can you try the .gltf and .glb with RightHandscene?

scene.useRightHandedSystem = true;

gltf and glb should work the same.

4

Yes. After setting scene.useRightHandedSystem = true the GLB model stopped flipping and basic functions of SixDofDragBehavior seem to work in the “canvas on screen” mode. (immersive-vr session still has issues, but let me create a separate post for that. so in this one we can concentrate on the more fundamental issue of the left- and right-handedness)

The BoomBox/BoomBox.gltf still seems to work exactly as before, i.e. setting scene.useRightHandedSystem = true didn’t mess that up either.

Elsewhere I saw other, crazier behavior artifacts when using SixDofDragBehavior with other GLB models (with various different transforms in them brought from Blender) in babylon.js apps using the default left-handed system. And those artifacts, too, disappeared when I set scene.useRightHandedSystem = true.

So this is progress. Thank you! However,…

Of course, when I set scene.useRightHandedSystem = true in the actual product, all the scenes and much of the transform/geometry logic got completely messed up. Consequences are huge.

While it is not impossible for me to just roll up my sleeves and go fix, rebuild the product, tools, scripts, etc to work with the right-handed system, I have concerns:

  • whether using SixDofDragBehavior (vs. other ways of achieving similar/good enough UX I have in mind) is worth doing all of that additional work of migrating to the right-handed system
  • I heard babylonjs team members say that stuff gets much less testing with the right-handed system; so I worry I’ll get a less stable system and will hit many more bugs (specific to the right-handed system) over time if I switch

The question is: what are we dealing with here? …

  • Is this just a bug in SixDofDragBehavior?
  • Is (ir isn’t) SixDofDragBehavior supposed to work with the imported GLTF models in the normal (for babylonjs) left-handed system regardless what transforms/internals those models have? (and is that even theoretically possible?)
  • Is SixDofDragBehavior supposed/expected to work with certain known subset of GLTF model structures/transforms/situations?
    • If so, can we clarify and document some of what is a) supported and is expected to work, b) unsupported and is expected not to work and what is c) gray area?
  • If some of what I observe should be expected to work (in the left-handed system and without jumping through any hoops) but doesn’t work due to bugs, is there bandwidth in the babylonjs team to tackle those bugs any time soon?
  • On the other hand, if the official position of babylonjs team is that SixDofDragBehavior with imported GLTF/GLB models is expected to work only if scene.useRightHandedSystem = true is set and is unsupported in the left-haded system, that would be helpful to know (and prominently document), too.

What would you say, @Cedric ?

Right. I included those two not because of their “different format” but because of the different behavior SixDofDragBehavior exhibits with them; the behavior must be dependent on the internal data/structures in the models, not on the format.

P. S. For convenience, here is a playground revision where I commented out the scene.useRightHandedSystem = true line and added a few more GB models. They (move camera back a bit, you will see “two tea boxes” model; try grabbing and moving it) demonstrate one more behavioral artifact that reproduces in the left handed system (that playground revision unchanged) but not in the right handed system (if you uncomment scene.useRightHandedSystem = true line, the artifact disappears).

5

There is something missing in SixDofDragBehavior
Babylon.js provides a way to change RH/LH coordinates system, it must work with the rest of the engine.
I’ll do my best to get it fixed in the coming days …

6

Another thing to try @sevaseva
Can you check if setting mesh parent to null (to remove upper hierarchy) and baking transforms using bakeTransformIntoVertices can fix it?

7

Thank you for prompt responses, @Cedric !

Thanks for confirming that you believe that SixDofDragBehavior is expected to just work when added to the __root__ mesh of any imported LGTF/GLB model in a babylon-standard scene with left-handed coordinate system, but as of version 6.47.0 it does not work in some situations due to a bug. Just trying to make things clear to myself and others; please correct me if I misinterpreted something.

I couldn’t find a way to interpret this sentence in a way that makes it relevant to this particular issue. I must be missing something.

Thank you! I am looking forward to testing a fix.

None of the meshes in the playground that reproduces the issue have a non-null parent.

Good point though. I am assuming that you plan/hope to come up with a fix that make it just work regardless if meshes involved have a parent or not. Once we have a fix, can you also share a set of relevant use cases that you tested or would like me/others to test (involving various parent-child relationships, transforms, geometries, etc)?

You meant bakeCurrentTransformIntoVertices, right?

I didn’t think that doing that can possibly fix a bug in SixDofDragBehavior, but…

indeed when I called bakeCurrentTransformIntoVertices [1] on the meshes before adding the behavior to them in a revision of the repro playground, the basic artifacts disappeared in the non-immersive mode. Good catch! I imagine this observation might help you localize the bug and fix it?

[1] calling it with the defailt argument value bakeIndependentlyOfChildren: boolean = true did the trick. Calling it with false predictably created visual issues with multi-mesh models and … I didn’t really test what it did to the behavioral artifacts.

P. S. Calling bakeCurrentTransformIntoVertices seems to have caused new issued with the near-interactions in that scene in the immersive-vr session, but I don’t think those likely have anything to do with SixDofDragBehavior specifically; I will probably create a separate post for that…

8

you can set parent to null with a call like mesh.setParent(null); and hierarchy will be updated.

9

Right. I just thought that if parent value is null, then first statement of TransformNode#setParent that Mesh inherits

        if (!node && !this.parent) {
            return this;
        }

will return without making any changes. So I didn’t try doing that. I must be missing something. In any case, trying that in the playground revisions that reproduce the issue is trivial, can always be done.

10

It looks like a regression. Please test the PR once the snapshot is available.

1 Like
12

thanks! testing with the snapshot in the same playground with all workarounds (useRightHandedSystem and bakeCurrentTransformIntoVertices) disabled, both mesh movement artifacts don’t seem to come up anymore in the non-immersive mode, so appears fixed at this level.

(various different artifacts in immersive-vr are still there; will follow up in the other post)