Yes turning it off should not have impact on anything else than the Clear color.
That said if you rely on other effects like bloom it would impact as well cause bloom needs to have an hdr input to work accurately. Basically the values over 1 would ge the one spreading around and without hdr you can not reach those kind of values.