I was having a similar problem in Typescript. What I’ve found the most success with is custom behaviours Use Mesh Behaviors (eg. dragging) - Babylon.js Documentation. You may have luck with the builtin behaviours being all you need.
I find this gives a really nice generic and re-usable way to add interactivity to a scene without having to extend base Babylon classes (implementing interfaces is much simpler in my opinion).
Thanks @DisownedWheat and @sebavan. I played around with behaviours and that got me half the way but then I realized gizmos, which use behaviors, do most of what I want already, and there’s an option to customize them with my own geometry, so I just made a transparent sphere larger than my object and then rotationGizmo.setCustomMesh(transparentSphere).