Material Instances appears to be a mechanism to avoid writing shader code when you want to modify an existing material’s properties without incurring shader compilation. To do so with Unity’s Material Instances, you have to use Dynamic parameters and avoid Static material parameters.
A big thing that makes material instances special is the ability to have parameters. One equivalent in Babylon is a custom material, one that has “attributes”. Another aspect of material instances is Inheritance documented under the second section header. I don’t think Babylon has a direct equivalent to material inheritance beyond that provided by JavaScript itself.
Babylon has ShaderMaterial, CustomMaterial, PBRCustomMaterial, Node Material Custom Blocks, and Material Plugins.
CustomMaterial is “a StandardMaterial with some customizable Options” while ShaderMaterial “is really meant for raw shader code and is only a thin layer above such code.”
Here is an older forum post with options for creating an NME material that uses attributes, including Node Material Editor Custom Blocks.
Shader Material allows you to define a custom shader with parameters, though the terminology is slightly different: “The vertex data for position, normal and uv coordinates are passed to the Vertex Shader as variables of category attribute. User data can be passed to both the Vertex Shader and the Fragment Shader as variables of category uniform. Data can be passed from the Vertex Shader to the Fragment Shader with variables of category varying.”
Unity’s Material Instance parameters appear to be limited to already-defined material parameters, though you can use Material Parameter Expressions to essentially expand beyond straight “material property = parameter value” into “material property = expressionAsFunction(parameter value,…)”.
There are complications when trying to use a shader material that comprehends different values for mesh instances because instances share vertex data, and vertex data is the underlying mechanism for supplying shader material attributes. I don’t know if/how Unity’s Material Instance addresses that issue.
Mesh thinInstances can have attributes that are passed to the shader material (or, I think, custom NME block). Here an example using color.
I can’t quite decode whether Unity’s Material Instances have the equivalent of the Shader Material’s “uniform.” A uniform is a constant for all meshes using that specific instance of material, but a uniform can be modified (for all meshes using it) without shader recompilation. As opposed to Unity’s Material Instance having Static Parameters which when changed do recompile the shader.
I hope this helps! Note that I got all this information from the links above. I’m kind of a “well-read newbie,” so an expert might come along and correct some of this.