So i was able to find the problem, with the help of @bghgary, which was the reason for the long loading time.
It turned out that Scene.markAllMaterialsAsDirty was the problem. We have in every single .gltf file a materials assets which takes time to load. One option would be to add some unique id metadata to the glTF and load the materials using a loader extension to prevent duplicates.
For now we just called
scene.blockMaterialDirtyMechanism = true;
before loading and after it completed it we call
scene.blockMaterialDirtyMechanism = false;
again and set it to false. Now the loading time is down to couple seconds.