@kvasss, I can certainly help you with some of your questions. I looked at your shared files, and I didn’t see the Max file you reference in your thread images, but I am assuming you are importing the FBX and textures into Max and exporting to the babylon format. @julien-moreau looks to have you covered with your issues on preview in the editor. What I can help you with is conventions and best practices.
Babylon.js expects the OpenGL format for normal maps. The easiest way to demonstrate this is to take your FBX, and load it into Maya assigning the DX version of your normal map. I exported to glTF because this format can be used across engines for comparison. You can see below the files that I exported from Maya, note the normal is your DX version:
I opened that glTF in the sandbox before opening the normal texture that was exported in Photoshop and inverted the green channel. I then saved it out and loaded the same glTF with the edited normal texture. You can see them below, DX on the left and inverted DX, which is OpenGL convention, on the right:
The reason you can see the edges in the DX version is that your low poly geometry does not conform to the curve in the high poly geometry and therefore the normal map will “bend” the normals to correct them to represent the high-poly geometry. This is why you see the edges in the normal map, this is the correction for the surface different between the two meshes. When you invert the green channel, we are in essence flipping the angle of the normals so something embossed will become debossed and vice versa. The correction of the curve gets inverted and pinches the curve instead of smoothing it.
The other thing you are doing in your comparisons in the thread is to compare two different engines with two different files. Since you only included .babylon files in your shared files, I am guessing you did not export a glTF format which can be read by both Babylon and Sketchfab. If you are rendering a .babylon file in Babylon and then comparing with uploading an FBX and textures to Sketchfab, you are going through two different conversion processes and the comparison is not apples to apples. Even taking the same file (glTF) into Babylon and then uploading the Sketchfab gives different results:
Even the Sketchfab engine optimizes differently than Babylon. Sketchfab uses extra post process when the model is still and drops some features when the model is in motion. The extra passes make the model look better but drop the frame rate in their engine when applying every frame so the engine is only partially real-time. To test this, open the iMac Sketchfab sample from your message above and notice the back isn’t as smooth when you are rotating it as it is when the model is still and more post process is done.
But you can see from the image below, I loaded up your custom editor, the playground and compared them and the rendering looks the same:
Even the Sketchfab render from your earlier message looks to be in the same ballpark (minus lighting):
So if you stick with an OpenGL format for your normal map, you will be good. And honestly, if you don’t have animation in your file and are using Substance Painter anyway, you can export to glTF directly from painter and get a glTF 2.0 compliant file which works in Babylon perfectly. Just choose the glTF PBR Metal Roughness export setting from your texture export window:
And Painter will export a glTF and glb (packed version of glTF… like a zip) and either can be loaded into Babylon and will render correctly:
And notice that I just used your imac_DX file from painter directly and it renders correctly in Babylon when exporting to glTF. This is because Painter knows what format the normal is using in the software and the convention for glTF 2.0 compliance and will convert as necessary for you on export. Hope this helps clear up some of your questions. Please let me know if you have more.
