Are you using the prepass renderer directly or indirectly (using SSAO2, SSR, motion blur, sub-surface scattering, etc)?
It seems that there is an additional delay when using the prepass renderer, see Model load much slower with cascadedShadow and prepassRenderer - #5 by Evgeni_Popov.
I am looking into this.