So here are my guess’ as of now:
I use the dummy scene to preload images, meshes and sounds etc… Then i use some of those images for gui in my real scenes. Maybe something is causing them to not be updated to the new scene.
I also have some home grown babylonjs modules (that i hope someday i can commit to the greater babylon 
) they seem to work but throw compile errors. (Can still run with these…). The one in question being circle button, which is and extension of ellipse…
https://www.babylonjs-playground.com/#EJX0YB#1 (seems to work fine though)
It only happens on pointer up, never anywhere else, not onPointerDown. Which is very strange.