This started out as a question, because I having a hard time getting it to keep track of pointers. So I still want to know if there’s better/more efficient ways of doing this, but anyway here’s my code:
https://www.babylonjs-playground.com/#LABFNA#2
The idea is that you can grab an object with one hand, then while holding grab it with the other hand, then not have any issues releasing either hand after that.
I think this is a clever way of solving it. I guess you will have different problems once it is not a sphere (mainly due to the parenting and rotation), but in general - this is a great way to solve this.
I would mark the last pointer that interacted with it, and in a different loop check this controller’s (or camera’s) forward way position, and set the object in this position on each frame. this way i avoid parenting and keep the object in front of the right object. and this way each pointer down decides which controller currently holds the object, so you can replace hands. You can also say that if the current holding controller triggers a pointer down on a different object (or on none), you “release” the object, which will simply stop setting its position.