PointerDragBehavior with thin instances & sprites

Hello,

I would like to use the PointerDragBehavior with thin instances as well as sprites but neither seems to be supported. Below are the steps I’m planning to take to get a similar behaviour. I wanted to post this before I start the implementation to 1) check if anyone has a better idea to suggest or if there is an option I’m not aware of, and 2) share what I’m doing in case it’s useful to anyone else going forward.

For thin instances:

  1. add onPointerDown, onPointerUp & onPointerMove handlers to the scene to detect drag start
  2. perform thin instance picking as described in Thin Instances | Babylon.js Documentation
  3. hide instance by reordering instances buffers and reducing thinInstanceCount
  4. replace instance by clone of mesh
  5. use onPointerMove handler to position the cloned mesh to follow the pointer
  6. use onPointerUp handler to detect drag end
  7. dispose cloned mesh, update instances buffers with new position, restore thinInstanceCount, commit new position to app state

For sprites:

  1. use the same handlers to detect drag start
  2. perform sprite picking as described in Picking Sprites | Babylon.js Documentation
  3. use onPointerMove handler to position the sprite to follow the pointer
  4. use onPointerUp handler to detect drag end
  5. commit new position to app state

I’ll probably have to detach camera controls during the drag as well.

Any thoughts?
I’ll probably get to this in about a week and post back on learnings.

1 Like

Hi!
If there is really no support for this already (sorry didn’t do my homework :smiley: ) then this seems to be the most straightforward solution.

I suggest not to modify the whole thin instances buffer in point 3, just set the instance’s position somewhere offscreen, so you have to update only one instance on the gpu by calling thinInstanceSetMatrixAt or thinInstanceBufferUpdated. Same applies in point 7.

Have a look at the ActionManager as well.

Hopefully there will be no issues detecting when to detach and reattach the camera controls.

Good luck!
R.

1 Like

I was curious and created this as a proof of concept for thin insances and it is working :slight_smile: You will ofcourse have to do the math for the movement.

https://playground.babylonjs.com/#RC2IAH#15

A lot can be used from the existing mesh dragging source code.

2 Likes

@roland super cool demo! Colors are pretty trippy :exploding_head:

2 Likes

Thank you! Actually the demo was created I believe by @Deltakosh. I saw him presenting this PG in one of his YouTube videos, so the credits goes to him :hugs: I just added the dragging stuff :sunglasses:

2 Likes

Thanks for the tips and PG @roland, you’re really the thin instances overlord on here :slight_smile:. A quick question is how do you typically send an instance offscreen? Is putting it far away enough or does it also need to account for not being in the view angle of the camera?

Good point about the ActionManager but I opted out of it due to the sprites for which not everything work if I recall correctly.

For the PG, that’s very similar to what I had in mind. I might have gone with starting “drag” on the first move after the pointer is down. Any thoughts on that?

Thanks again and take care.

It depends on the size of your scene and your camera limits. I think the safest way is to put the it behind the camera. If you are using an ArcRotateCamera you can get a position in front of the camera by getFrontPosition(distance) so you can get the position behind easily. However I would save some cpu cycles just by setting x, y, z to a high value and adjust camera.maxZ so the objects far away gets clipped. Or maybe just x, it really depends on your setup, that’s is what I’m doing all the time.

x = 999999999 and you are good to go :smiley:

Try it out whether it fits your needs :slight_smile:

You’re welcome!
R,

2 Likes

Thanks for the extra tips. Will get time to thinker with this next week and report back.

Hi folks, here is the result of my tinkering with this as promised.

TL;DR: no need to calculate the position of the cloned mesh, we can add the PointerDragBehavior to it and call startDrag to “simulate” the first click that started the drag. So you basically swap the instance by a mesh with the behaviour you wanted on the instance and it takes over smoothly. I think that’s actually really elegant – thanks to the great API design here :tada:.

Here is a PG: https://playground.babylonjs.com/#RC2IAH#16. In my app, I hide the instance while dragging, then on drop I update the instance buffer to the new position, show it again and dispose the equivalent of the boxSelected mesh. The PG has a shaky cursor tracking that doesn’t happen in my app.

Now the long story not short for people who like that.

Attempt 1 - I started as recommended by @roland but quickly got overwhelmed by how much of the code from PointerDragBehavior I had to copy in order to get the smooth positioning on the xz plane. You don’t actually appreciate how much work goes into getting this to work so well in 3 lines of code until you look under the hood.

Attempt 2 - At that point I thought it might be better to modify PointerDragBehavior to get it to work on thin instances than to strip 50% of the code anyway. I started doing that and got it working (without handling rotation which I haven’t tested). I’ve posted a gist with the diff of my changes for “posterity” lol :point_right: Revisions · Babylon.js PointerDragBehavior for thin instances · GitHub. Just look at the 2nd revision.

Attempt 3 - While modifying PointerDragBehavior, I came across the comments on line 307 that read “Simulates the start of a pointer drag event on the behavior” and it got me to realise I could fake the first click and just use that on a mesh that wasn’t actually there when the click happened. The line :point_right: Babylon.js/pointerDragBehavior.ts at 40f0ba2cc8a7acbd9dbdc81492a305fa781a41bc · BabylonJS/Babylon.js · GitHub

2 Likes

Wouldn’t it be easier to do the math yourself for the posiitioning? I mean you have the mouseOffsetX and Y values so you can calculate the movement quite easily I think. Or are there any gotchas?

I thought like you but it’s not straight forward at all actually. Pointer coordinates are on the screen, not in 3D. How do you convert a camera (x,y) offset in a 3D movement that’s dependent on your camera type and position? I think it’s possible but it’s a lot of maths and you have to do it for each camera type you want to use (I have more than one).

A simpler way (and what I initially intended) is to project the pointer in the 3D space through picking and it’s easy, but then it becomes dependent on the meshes in your scene. The position immediately jumps from the initial position of the mesh. It also makes it impossible to drag where there are no meshes to hit. The PointerDragBehavior gets around this by picking against an invisible plane that’s located where your mesh is initially and given a certain normal (the xz plane in my case). I thought I’d just do that myself but there is quite a bit of subtleties to get picking on that plane working with any angle of incidence, as well as positioning the mesh so it moves smoothly and not aggressively follow the pointer.

At the end of the day, I’d rather use engine code that’s well tested against edge cases, optimised and will be maintained for me. :wink:

1 Like

I get your point :slight_smile:

However look at this PG:
https://playground.babylonjs.com/#RC2IAH#21

It’s not perfect but starts to look good :smiley: (i’m still learning and fighting 3D math) You can use lerping for smoother movement (on 144 FPS it is looking very smooth even without lerping), starting at line 121.

Yep that’s what I refer to as painful maths to get the perfect behaviour, and camera dependent. Looks cool though, admittedly less complexity than I would expect to get to this level.

Lerping is really interesting as well, I didn’t know about it! :pray:

1 Like

And how about this? No math involved :slight_smile:

https://playground.babylonjs.com/#RC2IAH#30

4 Likes

Is :star_struck: an acceptable answer? It’s really neat, I might actually lift it from you! :wink:

It’s funny because I understood how the PointerDragBehavior worked after a while reading through the code, but I wasn’t able to then abstract away the complexity and imagine rebuilding from scratch with the simplest approach possible. That’s what you did brilliantly, kudos.

2 Likes

Yep love it and am bookmarking this one too! One thing I found forgotten thou is on pointer up to use boxSelected’s position to translate the thin instance’s matrix and then disable boxSelected again (otherwise the previous block’s position will seem to reset when you select another block since the thin instance was never updated). :beers: :slightly_smiling_face:
https://playground.babylonjs.com/#RC2IAH#32

Edit: one more change to move the thin instance offscreen while selectedBox is being dragged - LOL (10000, 10000, 10000) is far enough away I guess. :man_shrugging: :sweat_smile:

3 Likes

Thanks @Blake, good to get the full example working :slight_smile:.

I realize we completely ignored the sprites part of the initial question. Once we solved the problem for thin instances with either applying the PointerDragBehavior to the cloned mesh or performing picking and positioning like @roland showed, sprite is simply the same on a temporary invisible mesh that has its location set to the sprite location. If you have spriteDragMesh.position = sprite.position, they’re actually pointing to the same vector and the sprite will move as if the pointer tracking was applied to it.

2 Likes

Hi! I didn’t forget it :sunglasses: the main problem here was to solve the dragging stuff so I didn’t bother with the thin instance itself. I am glad you like it thou :+1::sunglasses::hugs:

It’s cool to see it in full action!

I’m still playing with the cubes lol