I don’t see a difference on my computer between first and second PG in the time it takes to compute the intersections:
See the console log, it displays the time when you click on a layer. On my computer, it’s around 240ms.
Now, the pick itself can take time because your scene is very heavy in triangles: more than 3.3M. The system finds the picked point by scanning all triangles and looking for intersections with the mouse position. If you display only a single layer (270k triangles with only the blue layer), there are less triangles to check and it’s faster to find this picked point.
You could be interested in some options to make picking / intersecting faster: