Hey @nonexplosive!
I ended up not using baking since, like you mention, it’s not really robust. I actually figured out that you can use scene.useRightHandedSystem = true to eliminate any trouble with importing gltfs. With this, you won’t have any negative scaling.
Like this: https://playground.babylonjs.com/#TWVYMQ#16
More info here: Issue with loading mesh and colliders and adding physics - #11 by wilcoschoneveld