You can see here that setting a pivot does not change the relative position of a gizmo https://www.babylonjs-playground.com/#3RTT8P#75. The pivot has been changed (to a position shown by the red sphere) yet the gizmo is still placed at the centre of the cube (yellow sphere).
@MarianG seems to have a solution to positioning a gizmo How can I move position gizmo axis? but as I am a very slow learner his solution is hidden to me within all the code of the PG he references. Perhaps he will help out by showing how the gizmo can be position to the red sphere in the PG I gave above.