After you hide the GUI3D (and thus all it’s controls) by setting
GUI3DManager.utilityLayer.shouldRender = false;
Then still any buttons/control receive pointer observables.
See the following playground (wait 5 seconds for the instructions popup) to reproduce the problem:
https://www.babylonjs-playground.com/#2YZFA0#52
After the button disappears from the screen, click on its former location to see the button still acts.
My take on this is that when a control is not visible or rendered, that it should not receive pointer observable events.