How about adding h, s and v getters and setters to Color3? RGB will be the actual color, and then the HSV getters / setters will to the RGB conversion.
It would make toHSV and toHSVToRef obsolete and possibly confusing though.
Yeah this is my main issue actually