Babylon Native not rendering HDR to Render Target Texture

Anupam_Das · September 25, 2023, 9:55pm

Hi All, Im noticing that HDR rendering isnt working in Native . See the two screenshots.
** from browser
browser

** from babylon native
native

the ABXY buttons from native is duller than what we are seeing on the browser. (post processing, material, lighting setup is identical… its the same code).
Also for post processing we are using scene.imageProcessingConfiguration NOT defaultRenderingPipeline.

There is some colored text and a transparent bubble over it which is reflecting a HDR .env. its the transparent HDR .env that adds the brightness to the ABXY buttons because of alpha blending. I dont think this has anything to do with the renderer but rather the RTT. We were using DXGI_FORMAT_R8G8B8A8_UNORM which is 8 bit so thought this just needs to be 16 or 32bpp.

github.com

BabylonJS/BabylonNativeExamples/blob/2177438e44e3f141c4b61b65cc6c33d52564a945/Apps/ConsoleApp/Win32/App.cpp#L49


      
              return d3dDevice;
          }
          
          winrt::com_ptr<ID3D11Texture2D> CreateD3DRenderTargetTexture(ID3D11Device* d3dDevice)
          {
              D3D11_TEXTURE2D_DESC desc{};
              desc.Width = WIDTH;
              desc.Height = HEIGHT;
              desc.MipLevels = 1;
              desc.ArraySize = 1;
              desc.Format = DXGI_FORMAT_R8G8B8A8_UNORM;
              desc.SampleDesc = {4, 0};
              desc.Usage = D3D11_USAGE_DEFAULT;
              desc.BindFlags = D3D11_BIND_RENDER_TARGET | D3D11_BIND_SHADER_RESOURCE;
              desc.CPUAccessFlags = 0;
              desc.MiscFlags = 0;
          
              winrt::com_ptr<ID3D11Texture2D> texture;
              winrt::check_hresult(d3dDevice->CreateTexture2D(&desc, nullptr, texture.put()));
              return texture;
          }

so i changed this to DXGI_FORMAT_R16G16B16A16_UNORM but that wasnt it. I also tried
DXGI_FORMAT_R32G32B32A16_UNORM. That resulted in a washed out looking image even though i was gamma correcting in the end like this:

github.com

BabylonJS/BabylonNativeExamples/blob/main/Apps/ConsoleApp/Win32/App.cpp#L219


      
          
                  // Tell RenderDoc to stop capturing.
                  RenderDoc::StopFrameCapture(d3dDevice.get());
          
                  // Save the texture into an PNG next to the executable.
                  auto filePath = GetModulePath() / asset.Name;
                  filePath.concat(".png");
                  std::cout << "Writing " << filePath.string() << std::endl;
          
                  // See https://github.com/Microsoft/DirectXTK/wiki/ScreenGrab#srgb-vs-linear-color-space
                  winrt::check_hresult(DirectX::SaveWICTextureToFile(d3dDeviceContext.get(), outputTexture.get(), GUID_ContainerFormatPng, filePath.c_str(), nullptr, nullptr, true));
              }
          
              return 0;
          }

Oh and I know the drill with a repro, i’ll try to get something going, but figured id post incase anything obvious stands out

Any thoughts?
@bghgary @sebavan

bghgary · September 25, 2023, 10:53pm

Can you capture a frame with RenderDoc and send it to me privately?

Anupam_Das · September 26, 2023, 6:38pm

@bghgary sent.

bghgary · September 27, 2023, 12:14am

Unfortunately, I don’t think I can figure this out from just the RenderDoc output. I tried but there are many draw calls and it’s difficult without source or something to compare with. I need to be able to compare a working version on the web that I can use Spector.js and compare with the native version with RenderDoc.

xiehangyun · September 27, 2023, 6:01am

HDRFiltering does not work on Android phones - Bugs - Babylon.js (babylonjs.com)

Environmental reflection problem of gltf model - Questions - Babylon.js (babylonjs.com)

Reading both pages, I guess this should be the same issue.

Anupam_Das · September 27, 2023, 2:52pm

thanks so much for investigating. i think ive tracked this down to a more fundamental problem with the renderer that manifests its self also on the browser side and how reflections are blended/composited. So i dont think this has anything to do with Native afterall (apologies for the oversight), so yes its also happening on the regular engine. Im putting together a PG right now that clearly illustrates the issue. this is good news as at least we can repro this in an environment we can both work in. I suspect @sebavan may have some thoughts. This thread contains hints of this issue as well. Enabling HDR in defaultPipeline disables AA

Anupam_Das · September 27, 2023, 4:49pm

@bghgary @sebavan here is the repro:

if you disable the default rendering pipeline, note that the abxy buttons render darker.
if you diable HDR rendering in the rendering pipeline, the abxy buttons render darker. as you would expect.

i assume if we dont have a rendering pipeline its using scene.imageProcessConiguration. I thought the renderer by default is in “HDR mode” … am i correct? mabey there is a flag that needs to be set to get the scene to render in HDR? I have combed the docs and cannot find a way to turn on HDR which leads me to assume it should just work.

also i have a feeling some of this is my own confusion on the difference between scene.imageProcessConiguration and DefaultRenderingPipeline. I thought they should look exactly the same except that DefaultRenderingPipeline is a post process that is applied to everything (the RTT) where as imageProcessConiguration is per material.

in the end… we want scene.imageProcessConiguration to look identical to DefaultRenderingPipeline. we also would like to use scene.imageProcessConiguration for the brunt of post processing moving forward, but need the refelctions to look like how it looks when using DefaultRenderingPipeline in HDR mode.

sebavan · September 27, 2023, 9:16pm

TL/DR it is not possible as the canvas itself does not allow hdr so far so the rendering is limited to 0…1 with 8 bits per chanel.

But in your case the difference comes from the blending, As this scene uses transparency, one is doing blending in linear and the other in gamma space resulting in the difference you observed between with and without pipeline.

Finally about why the pipeline in ldr does not work like hdr, it is because we internally rely on imageProcessing only as an optim if the pipeline is ldr

Anupam_Das · September 27, 2023, 9:46pm

got it. Looks like we have to use DefaultRenderingPipeline.
Guessing there is no way to get 8bpp to blend in linear space?
Also, just to be clear, you are saying the real limitation is that browers dont support greater than 8bpp on the canvas element? (that would make sense)

sebavan · September 27, 2023, 9:53pm

Blending in linear space would require sRGB support for the canvas back buffer I guess. HDR would be a plus but not mandatory. This is a coincidence that it works in hdr in the PG. It is only related to the extra pass done in linear space.

Topic		Replies	Views
DefaultRenderingPipeline. Glow layer support in Babylon Native Questions	17	332	February 22, 2024
Use Babylon to load HDR textures Questions	2	669	January 2, 2021
Will BabylonNative support HDR textures? Questions babylon-native	1	358	June 23, 2023
Issue with depth texture rendering in Babylon Native? Bugs babylon-native	7	980	July 20, 2022
Dark Contrast with HDR true in DefaultRenderingPipeline in camera2 viewport Questions	10	338	June 3, 2022

Babylon Native not rendering HDR to Render Target Texture

Related topics