I think this is a clever way of solving it. I guess you will have different problems once it is not a sphere (mainly due to the parenting and rotation), but in general - this is a great way to solve this.
I would mark the last pointer that interacted with it, and in a different loop check this controller’s (or camera’s) forward way position, and set the object in this position on each frame. this way i avoid parenting and keep the object in front of the right object. and this way each pointer down decides which controller currently holds the object, so you can replace hands. You can also say that if the current holding controller triggers a pointer down on a different object (or on none), you “release” the object, which will simply stop setting its position.