Unfortunately IBL Baker is proposed as a free alternative to lys but all the best results we achieved were always done through Lys by Patrick.
Also in IBL baker the mips are linearly sorted from roughness whereas by default the PBR expects log2 ramp like in the lys setup.
You could try the flag linearSpecularLOD on the env texture to see if it improves your case with ibl baker.