Roughness seems to have been disabled recently (within the last week or so) in the Sandbox for KHR_materials_transmission. Perhaps Babylon.js now requires IOR to be included alongside transmission?
Test asset:
glTF-transmissionRoughness_baseColorTexture.zip (32.8 KB)
This is working in vscode using Babylon.js 5.0.0-alpha.7 and I am able to repro this in the current sandbox, so something must have broke between then and now.
See this comment from the KHR_materials_volume PR:
The LOD we are using to read from the refraction texture is now also taking into account the volume IOR (in addition to taking into account the roughness/alphaG):
float refractionAlphaG = mix(alphaG, 0.0, clamp(vRefractionInfos.y * 3.0 - 2.0, 0.0, 1.0));
float refractionLOD = getLodFromAlphaG(vRefractionMicrosurfaceInfos.x, refractionAlphaG);
Note that vRefractionInfos.y = 1 / volume_IOR.
Given the formula, if the volume IOR = 1 (which is the case with the file in the sandbox), refractionAlphaG = 0 (whatever alphaG) and refractionLOD = 0.
It seems the formula should be changed so that alphaG still factors in even when volume IOR is 0…