I would advise to not use the legacy mode (false in second param of setPivotPoint) it might be the unsupported mode for gizmo and cedric might look into it but not sure of the outcome.
Using the other mode would move the origing for scaling and rotating without inducing a translation. It also does not break gizmo https://playground.babylonjs.com/#53G1ES#1 you can uncomment the bottom part to see it.
If you wish to fully remap the origin (not only a pivot). I would advice to add a parent transform and if you can bake it, rely on bakeCurrentTransformIntoVertices… https://playground.babylonjs.com/#53G1ES#2