@RaananW might have some ideas
I don’t see the babylon version you are using, but even with that - it would be great to have a project reproduction to understand what’s going on.
I am also not sure why the webpack loader is used in a vite project, but that might be something that does work? Also babel is probably not needed?
Anyhow - tl;dr - reproduction would be great, otherwise - hard to help.