There it is… the ugly truth:
[Babylon.js/pointerDragBehavior.ts at master · BabylonJS/Babylon.js · GitHub] Line 321… _moveDrag(). More specific… Line 339… this._worldDragAxis.scaleToRef(dragLength, this._dragDelta);
I wish we could simply change scaleToRef… into scaleToRefWithCollisions. 
But nooooo. heh.
@jasOn, when you rotated the mesh, and the ellipsoid DIDN’T “go along”… but instead stayed un-rotated… HOW did you rotate it? With a pointer dragger of some kind? A little animation, maybe? Just curious. Puzzling. Was .showBoundingBox = true? Did you see IT rotate, properly?
There is a thing nearby… which I call “quaternion rotate”… it is a method .rotate()… but any other mesh.rotationQuaternion method of rotating a mesh… might be worth testing.
IF the ellipsoid COULD be told to rotate… WHEN the mesh rotates… then you have invented turnWithCollisions()… the sister to .moveWithCollisions(). But a really NICE-WORKING .turnWithCollisions() method… might be full of troubles… AND might need a “special” and different Engine.CollisionsEpsilon… or perhaps NOT use that setting AT ALL.
Just possibly, this is why ellipsoids don’t turn… when mesh turns. Too collision-complicated… would cause many CPU tumors. 
Sorry i just saw the message, this is a quick reply.
This PG https://www.babylonjs-playground.com/#32M3IH#16 (line 173)
Not sure if I wrote correctly. 