@biscuit, there are a few things you are tangling up in your application of the textures in you materials, so I’ll lay out a complete flow for you to follow.
The glTF spec requires Physically-Based Rendering (PBR) materials for the format, with PBR Metal-Rough being the default material model. You can have access to the PBR Spec-Gloss material model with an extension but you need to know that wherever your model is being used has implemented the Spec-Gloss extension for your model to render correctly. Babylon.js can handle either, but for compatibility’s sake I would suggest using the Metal-Rough material model as there are only a few instances where you want to introduce color into your specular contribution on dielectric materials which would require the PBR-SG material model. Most models will work well with the PBR-MR model so I would stick with that.
A good primer on the PBR workflow is the Allegorithmic PBR Guide which will give you all the information you need about how to author the specific maps for PBR materials. This has both the theory behind the rendering methods as well as practical guides to get the most out of your maps.
Now to tackle your questions directly:
- If you are exporting to glTF from the Babylon exporter, all of your materials in 3ds Max should use the Physical Material shader. The reason for that is that the exporter will need to convert any non-physcial material into a PBR material when writing out the glTF, so you should be working in that to begin with as conversions from blinn-phong to PBR-MR are approximations and not exact. Working directly with physical materials gives you direct control over what is written to file.
- The difference between a bump map and a tangent space normal map is that the bump map stores data in one channel (0-1) and tells the pixel in the texture where it sits in relation to the surface along the surface normal. The normal map stores a vector (X, Y, Z) in the RGB channels of the texture and tells the pixel in the texture what its normal direction is exactly, which can be different than the surface normal. This gives you far more control over how lighting reacts with your surface and is why you don’t see bump maps being as popular as they once were. If you need to convert bump maps into tangent space normal maps, you can use a conversion tool like CrazyBump which is free and has been around for a very long time. Though it is always best to create your normal map from scratch by baking the difference between a low-poly and high-poly mesh. Some of my favorite tools for that are from Allegorithmic, but there are others like Marmoset Toolbag, xNormal, Knald, and Quixel (though Quixel needs a copy of Photoshop to run as it works on top of Photoshop).
- If you are using a specular map, that is not a roughness map. A roughness map determines how rough a pixel is from 0 (not rough at all, reflects the environment) to 1 (completely rough and all reflections are broad with no discernible detail). A specular map determines how much light a pixel reflects and what the light color is. The specular map is paired with a gloss map that determines how sharp the specular reflection is which implies how smooth the surface is. So you will need a roughness map, which you can easily create in any image editor by laying down values between 0 and 1 for the UV islands in your model as a starting point. You will also need a metalness map to complete your PBR material which is a texture that denotes a pixel is non-metallic, or dielectric, (0) or metallic (1). There are very few areas in a metalness map that aren’t black or white and normally are transitions between metallic and dielectric like dirt on a metal shovel head. The more detail you put into the roughness map, the better your materials will look, but you have to know why you are putting in that change in roughness. A desktop will have a single roughness value, but adding detail into the roughness map can imply scuffs in the finish, water rings on the surface, scratches, etc. There should be a reason that there is a difference in roughness and that is because the materiality of surface has changed. For example, a painted surface that has been scratched reveals a different material below and thus changes roughness.
- For opacity, you need to separate meshes that need a BLEND alphamode from meshes that need an OPAQUE alphamode. You assign the meshes that have some transparency a material that uses the file alpha (in Max) and a texture with an alpha channel. The rest are assigned no transparency in their material. The reason is that if you put meshes into the transparent queue, you will suffer from Z-fighting where the renderer does not use a depth pass to determine which mesh is in front of which. Using a depth pass is more costly than loading extra meshes and textures as the depth sort is done per frame. The best method is to isolate all meshes needing transparency into one material and use a second material for the opaque meshes. Then the depth sorting that is done by the opaque meshes will usually take care of and Z-fighting that occurs with your transparency queue meshes.
- Simply inverting the specular map does not equal a roughness map. You are inverting the amount of light reflected by each pixel, but that does not equate to how rough a material is. Roughness has to do with how sharp the reflection of the environment is on the material, not how much light is reflected off of it.
Let me know if any of this doesn’t make sense or if you have more questions about anything.