Incorrect display of normal maps in the sandbox, desktop and web editor

Hello. The essence of the problem is the incorrect display of the normal map in the sandbox and editor.
Here are all the materials you may need
https://drive.google.com/drive/folders/1r0mDyR8PGUk3GgIYT_yu_2QMRYFeI3Xe?usp=sharing

And some screenshots

substance-painter project-settings normals mode is direct-x

painter-export mode direct-x

3ds max shader parameters

desktop-windows editor inverted x and y?

sandbox after export from 3ds max

desktop-editor player/ all works fine

web editor and player do not work correctly

custom app works fine
http://alevg1531.beget.tech/imac.html

Hey! do you mind sharing your 3dsmax file? This way @drigax can have a look

All files on the link https://drive.google.com/drive/folders/1r0mDyR8PGUk3GgIYT_yu_2QMRYFeI3Xe?usp=sharing to google disk. I think the problem is not in the max file because in the end everything is displayed correctly in the engine but different results in different viewers suggest that the problem is in viewers and not in files and not in normal map

sandbox is using the exact same engine as you (providing you are on preivew.babylonjs.com (or preview on npm)).

Pinging @julien-moreau for the Editor

@kvasss the editor is using the version 4.0.3, I’m deploying a new version soon with bug fixes, improvements and using the lastest version of BabylonJS. I think this will fix your issue as the preview version should work fine.

but sandbox show different results it’s weird and looks like a sandbox bug if engine is same as i use for test scene. i can provide other models and textures in different formats for tests but i observe same problem with other model

correct view


format DX and in engine flipped Y

links to scene with different types of normals

chair
https://1drv.ms/u/s!AsX2zlTCVKnHkkARlg27nFEueHXl?e=YX3c4N

imac
https://1drv.ms/u/s!AsX2zlTCVKnHklMhUXL3jMZMVHyQ?e=jFv5Bo

I can’t understand what normal map convention your engine uses?
I tried both the DX and openGl formats in substance painter and both formats do not display correctly in babylon

for both models i use same DX workflow but got strange results
1)no need to invert channels

2)need to invert X channel and although the editor still does not display correctly, everything is ok in the player

accordingly, I don’t understand which convention is used. it turns out some kind of porridge

Adding @PatrickRyan for help

1 Like

@kvasss, I can certainly help you with some of your questions. I looked at your shared files, and I didn’t see the Max file you reference in your thread images, but I am assuming you are importing the FBX and textures into Max and exporting to the babylon format. @julien-moreau looks to have you covered with your issues on preview in the editor. What I can help you with is conventions and best practices.

Babylon.js expects the OpenGL format for normal maps. The easiest way to demonstrate this is to take your FBX, and load it into Maya assigning the DX version of your normal map. I exported to glTF because this format can be used across engines for comparison. You can see below the files that I exported from Maya, note the normal is your DX version:
image

I opened that glTF in the sandbox before opening the normal texture that was exported in Photoshop and inverted the green channel. I then saved it out and loaded the same glTF with the edited normal texture. You can see them below, DX on the left and inverted DX, which is OpenGL convention, on the right:

The reason you can see the edges in the DX version is that your low poly geometry does not conform to the curve in the high poly geometry and therefore the normal map will “bend” the normals to correct them to represent the high-poly geometry. This is why you see the edges in the normal map, this is the correction for the surface different between the two meshes. When you invert the green channel, we are in essence flipping the angle of the normals so something embossed will become debossed and vice versa. The correction of the curve gets inverted and pinches the curve instead of smoothing it.

The other thing you are doing in your comparisons in the thread is to compare two different engines with two different files. Since you only included .babylon files in your shared files, I am guessing you did not export a glTF format which can be read by both Babylon and Sketchfab. If you are rendering a .babylon file in Babylon and then comparing with uploading an FBX and textures to Sketchfab, you are going through two different conversion processes and the comparison is not apples to apples. Even taking the same file (glTF) into Babylon and then uploading the Sketchfab gives different results:

Even the Sketchfab engine optimizes differently than Babylon. Sketchfab uses extra post process when the model is still and drops some features when the model is in motion. The extra passes make the model look better but drop the frame rate in their engine when applying every frame so the engine is only partially real-time. To test this, open the iMac Sketchfab sample from your message above and notice the back isn’t as smooth when you are rotating it as it is when the model is still and more post process is done.

But you can see from the image below, I loaded up your custom editor, the playground and compared them and the rendering looks the same:

customEditorAndPlayground

Even the Sketchfab render from your earlier message looks to be in the same ballpark (minus lighting):

sketchFabFromForum

So if you stick with an OpenGL format for your normal map, you will be good. And honestly, if you don’t have animation in your file and are using Substance Painter anyway, you can export to glTF directly from painter and get a glTF 2.0 compliant file which works in Babylon perfectly. Just choose the glTF PBR Metal Roughness export setting from your texture export window:
image

And Painter will export a glTF and glb (packed version of glTF… like a zip) and either can be loaded into Babylon and will render correctly:

image

And notice that I just used your imac_DX file from painter directly and it renders correctly in Babylon when exporting to glTF. This is because Painter knows what format the normal is using in the software and the convention for glTF 2.0 compliance and will convert as necessary for you on export. Hope this helps clear up some of your questions. Please let me know if you have more.

2 Likes

Hello, @PatrickRyan and @julien-moreau

@PatrickRyan thanks thanks for the detailed answer.

After reading your answers, I began to check the export settings sequentially and eventually realized that the problem is that I export with smooth groups and import into the painter.
1
Texturing in painter and applied textures to the model in 3ds max, but the 3ds max model used the one that was exported to fbx not from a painter with the right smoothing groups. As a result, the normals were displayed incorrectly in engine apparently due to some kind of difference between the smoothing groups of max and painter After I set the default export settings , the problem disappeared. Now everything is displayed correctly in the sandbox and engine …


but not correctly in the editor viewer.

after exporting from the editor, everything is displayed correctly with the latest version of the engine. I conclude that the problem is in the editor viewer. as Julian wrote above

UPD
I was wrong. after exporting from the editor, the X channel is inverted same as in editor viewer


it turns out the problem is not even in the editor’s viewer but in the fact that the model in the editor undergoes some kind of transformations?

1 Like