Thanks for looking into this!
Your proposed solution is sorta what I have implemented as a interim solution, it feels a little hacky but this is more or less how I imagine 6dof drag should work in XR
A more ideal solution would be a scaling scale factor that changes based on how far the target is from the user. Ill post another play ground if I get something working nicely.