@kutomer, there isn’t any automatic solution for storing the height in alpha in Max. If you are using one of our exporters, there is no composition of the final texture from the shader graph. The tool traces back to the source texture and passes that file, so any alteration done in the shader graph is ignored, which would include packing two textures into one.
The easiest ways for your artists to generate height maps would be in the authoring step. If you are using texturing tools like Substance Painter or Designer, you are normally authoring height as you create and then the final output is a normal map. You can simply set up an export to place the height in the normal alpha slot.To set up a custom export from Substance painter, you can see an article I did around the art pipeline for glTF which touches on making a custom export texture in the Substance Painter section. A complete tutorial can be found in this video from Allegorithmic.
If you are not authoring, but rather have a normal map and need to reverse engineer a height map, there are several tools that can do that.
I have a lot of Substance experience, but have not used the others. I wanted to provide several options, and the GIMP option is free, but I always go back to the Substance packages because they are so powerful. I would also give a plug for Knald becuase we use their package Lys for creating our environment maps and that software is very powerful. I haven’t tried Knald because I do most of my baking right in Substance, but I am sure the tool is very good from its reputation.
Once you have the height map, you just need to channel pack it which can be done in any image editor like Photoshop, GIMP, or texture packages like Substance. Leave the RGB channels alone and add the height to the alpha channel and make sure to save in a 24-bit file format.
Please feel free to ping me, or have your artist ping be directly, with questions you have.