You have to use compose method:
const scale = new BABYLON.Vector3(2, 2, 2);
const rot = new BABYLON.Quaternion.RotationYawPitchRoll(Math.PI/3, 0, 0);
const trans = new BABYLON.Vector3(-1, 2, 0);
var newMat = BABYLON.Matrix.Compose(scale, rot, trans);
sphere.thinInstanceSetMatrixAt(idx, newMat);
for some reason it is working in 4.2 and not in 5.0 at PG
https://playground.babylonjs.com/#217750#38
just saw a PR by @Evgeni_Popov in the topic , it maybe identical