In docs, there is a SceneSerializer that serializes a scene to a .babylon file.
https://doc.babylonjs.com/features/featuresDeepDive/Exporters/Save_Babylon/
As babylon.js changes, the serialization format changes too, so is there some kind of compatiblity matrix, that documents explictly the compatiblity of the serialization format against babylon.js version?
Or, is there any documents for breaking changes of the serialization format, like for the api?
I think we never broke the file format compatibility so far 
and we do not intent to.
Any older version should and will continue to work. It means some entities/properties might be supported in different flavors in the format.
What if babylon file created by newer engine loaded by older engine? Will the scene renders as-is in this case? A file format, like gltf or fbx, is expected to be able to be exported for older versions (maybe with the cost of lacking of some optionsl features, but not breaking the rendering)
Node, it is only maintaining back compat without aiming to forward one as well
But is there anywhere that documented when forward compatiblity broken? Or maybe a compatiblity matrix, version of engine to serialize and version to parse.
nope I would say any as long as you generate with v X only use in X+ to be 100% safe.
Is there a plan to make a doc for it in future?
We should definitely add it in the doc.
Looking forward to it.