Hi,
A transformation matrix is applying scaling, quaternion rotation, and position. As shown in the documentation for static method Matrix.compose(). Matrix | Babylon.js Documentation
decompose is to extract the scaling, rotation, and position out from the transformation.
e.g. hitTest.transformationMatrix.decompose(undefined, b.rotationQuaternion, b.position);
It extracts the rotation and position, and since the code is not interested in retrieving the scaling. undefined is passed into the function.