Scale behaviour using custom handles/gizmo

I have a scene with a cube and a ground. The cube is fixed to ground and can be dragged parallel to it. I want to implement some handles that does non uniform scaling.

The green handles should perform scaling(box gets scaled up in x and z)

The red handles scale either in x when the right/left red handle is dragged or gets scaled in z when front/back red handles are dragged.

I tried bbox gizmos but they give many more handles for rotation etc. This is a specialised need so how can I achieve this type of gizmo behaviour;

You could inspire yourself from how the gizmo code does ?