Path-tracing in BabylonJS

[Evgeni_Popov]

Using Texture to load the .hdr as a 2D texture does work:

https://playground.babylonjs.com/#M2V0ZX#4

[erichlof]

Fantastic! This is exactly what I was hoping for! I will work on getting this in the Babylon Path Tracing Renderer as soon as I can.

I’ll let you know if I have any other questions, but hopefully I can do everything correctly now. Thanks again for your time and expertise!

[erichlof]

Hello @Evgeni_Popov ,

If it’s ok with you, I must once again lean on you for support with a loading issue I encountered.

I have already done all the infrastructure for the .js and .glsl code to be able to (theoretically) load, decode, find brightest pixel area, pass to GPU, and sample from an HDR inside our GPU path tracer. All the plumbing is there, ha.

The problem is that when viewing the final path tracing scene, the background is either black or blown out with oversaturated values - I can’t make out any kind of picture from the hdr, it must be junk lying around in GPU memory or something.

Here’s a quick link to the scene:
Non-Working HDR Environment demo (debugging, W.I.P.)

I went back to the .js source where BABYLON.Texture() is used and tried to add some console.log() debug output. It’s not even firing that part of the code. When I then type hdrTexture (which is the variable to hold the hdr texture object to be handed over to the GPU to be sampled from as a normal texture2D sampler) the browser console reports that hdrTexture is undefined. The graphical output kind of makes sense in this situation because nothing is getting passed to the GPU which I am erroneously trying to sample from in the path tracing shader.

The question becomes, why is the async () => callback not firing? Could you please take a look at the .js file where all the hdr loading action happens?

Loading section of the HDR demo

Maybe I’m missing something or doing something that happened to work with three.js, but doesn’t fit Babylon’s pattern of dealing with float textures. As you can see from my code, I used your very handy bright pixel finding routine (credit to you coming in the comments in the source code soon!) with a few minor changes just because of how I want to look at the data on the screen. But my minor changes shouldn’t have broken anything - I can’t even get the callback to fire or print out any console output at all, lol. It’s as if the hdr texture never existed.

Thanks once more for taking a look at this. We’re so close!

[sebavan]

Your callback should not be async here but it still does not explain why it does not fire I let @Evgeni_Popov check this part

[erichlof]

@sebavan
Ah, ok thank you. I must admit I don’t really have a firm grasp on async, await, etc. - I have been so deep with path tracing for the last 6 years that all these new javascript features have passed me by, ha.

[Evgeni_Popov]

In fact you do need the async keyword as we are using a await construct to get the pixel data.

Are you sure you don’t have a 404 in the browser console? It’s the only explanation I can come by to explain you don’t see the first console.log (or pathTracingScene is not ok)…

[erichlof]

[erichlof]

There are no 404 errors that I can see. Just to test, I misspelled the hdr path name on purpose, then I got the 404 right away.

This is in odd situation, because it works fine in your playground example - it’s nearly the same code. Could it be that the PG has a createScene() function that returns the scene whereas I do not? Or could it be that by the time the big hdr is loaded from the server, the time to get it into the GPU has somehow passed? I’m afraid I’m not familiar with the network inner-workings.

p.s. as you can see from my screenshot, it is 93 degrees outside in September here in Texas, lol!

[Evgeni_Popov]

It seems there was a problem with .hdr loading and/or loading notification in 4.2, if you test my PG (https://playground.babylonjs.com/#M2V0ZX#4) in 4.2 you will see it does not work.

I guess you should switch to 5.0.

[erichlof]

Yay! At some point I was going to ask you guys if I needed to upgrade to the latest version of Babylon and how often. I haven’t been keeping an eye on the main development page as much as I should. It’s good to find out that this was the problem. Going to upgrade now!..

[erichlof]

[erichlof]

Whoo hoo! That was it! As you can see, it is loading as expected.
The model not loading was due to the same issue -
I had to not only update Babylon.js to 5.0, I had to update the babylon.glTFFileLoader.min.js as well. When all that was done:

I still have some firefly kinks to work out in the shader. I remember from when I did this in the three.js renderer, this particular demo (using an hdr) was the most finnicky and sensitive in terms of suppressing hot pixels or fireflies - it’s simply because the range of luminosity is so great inside the hdr texture. For now, I turned off refractive caustics so it’s not so visually distracting, but I’m confident that soon I will be able to work out the firefly issues so we can have focused refractive hotspots from glass glTF models.

Here’s the working demo minus refractive caustics:

Working HDR demo

Again, thanks so much for helping me solve the loading issue!

p.s. it says I can’t post more than 3 times in a row on this thread, which is why I keep editing this last post, lol.

To all, I am still fiddling with the sun brightness, color bleeding, caustics, firefly suppression, etc. So the end HDR result will change for the better over the coming days - it just requires a lot of finesse and fine-tuning on my end. HDR path tracing is the hardest in terms of my custom shaders (a lot of conflicts and tradeoffs to juggle). Thanks for your patience while I work on this, but I’m so excited to have this displaying correctly now!

p.p.s.
Hey everyone,
On the new HDR Environment demo, I just now added an HDR image picker to the gui as well as an HDR exposure controller and Sun power slider!
Could a Babylon forum admin please grant me access to be able to post more than 3 times in row? I have to keep adding to this post, ha ha.

Or, since I didn’t start this thread, could maybe @PichouPichou or @sebavan or @Evgeni_Popov please post a short reply sentence below, so that it will break my 3-post-in-a-row sequence? Thanks!

[sebavan]

This is Amazing !!!

[erichlof]

Thank you! And thanks for breaking the 3-post streak - Ha! I was in limbo for a couple of days

[Deltakosh]

LOL!

@erichlof

the 3-post streak

[erichlof]

Hello Everyone,

Hopefully most of you have been able to try out the latest features of the Babylon path tracing renderer, including glTF model loading and HDR environment support. If you are having any problems, please post here and I will try to take a look and/or fix the issue.

Sorry for the week-long delay in posts from me - I have been kept busy with family commitments and my children’s school/after-school activities. If there are features that the three.js renderer has that you think would be helpful also inside the Babylon renderer, again please post here for all to see. If I can implement it, I will put it on the TODO list over on the renderer’s GitHub repo readme.

The Babylon renderer is shaping up quite nicely now and has the core features that the three.js one has, and even more features/functionality in some areas. Hopefully, some of the more experienced developers out there can take my code and add to it what they would like to see (even if it doesn’t relate to the actual path tracing). Things that come to mind are:

• A loading spinning graphic when the page’s models are being loaded and processed and shaders are being compiled

• An ability to type the path of the glTF model or HDR to load directly into the UI

• Drag-And-Drop functionality for dragging an HDR thumbnail from your computer onto the webpage, and it immediately is processed and used as the path traced background.

• Drag-And-Drop functionality for glTF models - you just drag the model’s glTF file from your computer onto the webpage and it loads it, quickly creates a BVH, and starts rendering.

• Mouse manipulation of items in the scene (like a proper 3D editor can do). For instance, click on a cube (which would require typical 2D mouse coordinate mapped/unprojected to 3D space, ray cast against bounding boxes of moveable scene objects), then drag to position the cube, or have it like I have now, where you control the mouse-selected object’s complete 3D transform with the UI sliders.

• In relation to the previous feature, when you click on an object, or select it in the UI editor panel, it actually changes the shader to put a bright white outline around the silhouette of the selected object (but it is still path traced normally). This would require some work on my end, but I recently figured out how to do it when I was inspired by MAGI’s use of thin white outlining combined with traditional ray tracing for the classic 1982 film, TRON.

• a mobile-friendly version of the demos that are supported. This means everything that doesn’t have hefty glTF models. I’m still puzzled as to why anything over the Utah Teapot triangle count crashes on mobile devices. But in any case, a mobile version with pinch gestures to zoom the camera, arrow buttons to fly around the scene (like Minecraft PE), and swiping to smoothly rotate the camera would be awesome. I have done this with the three.js version, but would probably need some help getting my custom mobile js code to work with Babylon’s camera system.

The core features of our renderer are in place - now it’s just a matter of how far we want to polish it, and how many features we want to add.

[erichlof]

I recently saw this post of on the unity forums:
iOS 15 Safari will support WebGL 2.0? - Unity Forum

And here’s the link to what they were referring to:
Safari 15 Beta Release Notes | Apple Developer Documentation

I just saw this today and it definitely made my morning! I sore point for me in the past was the fact that my demos would not work on mobile iOS devices (which seem to be kind of popular these days, lol). I personally have only ever used Android, (not because I have anything against Apple, it’s just what I started out with when I joined Verizon many moons ago, ha) and that’s what I’m comfortable with I guess.

But I would get reports from personal friends and/or GitHub issue OPs that the renderer wouldn’t work on iOS. In the past I would just chalk it up to Apple’s slow roll-out of WebGL and reduction of support for anything related to OpenGL, for that matter.

However, iOS 15 now supporting WebGL 2 out of the box opens up a whole segment of the end-user population that was previously unable to enjoy our renderer. I’m so excited for this feature, and so thankful to the iOS team for actually implementing WebGL 2 support (and even considering it something worth adding support for).

If any of you have the latest and greatest iPhones with iOS version 15 and latest Safari, would you please try out the demos and report back here if you were able to see them working on your device? Now I’m not expecting the glTF demos to work (but maybe the Teapot model will, who knows?), but it would be great if at least the first basic Babylon path tracing demo would be able to run on your device.

Thank you!

[samevision]

The demos didn’t work until iOS 15. But now everything is working well. On my iPhone XR it’s quite fast

Sometimes the page reloads if I change the model though. But the page reload happens in a lot of 3d scenes on iPhone.

[erichlof]

@samevision
That’s great news! Thank you so much for sharing your findings.

I happened to be at the shopping mall today and as I walked towards the Apple store, I was toying with the idea of going in, just to try out the GitHub repo demos on a recent iPhone. But ever since Covid started, there is now always a line of people outside the store front, all waiting to be admitted in by the store crowd control employees (only a certain number of customers are allowed to be inside together at the same time). So, unfortunately I didn’t have time to wait in line, and then there’s the embarrassing part of having to explain to the hopeful salespeople that I’m not really interested in purchasing an iPhone (I just renewed my Samsung Galaxy plan, ha), but that I am doing technology recon and project field research, lol!

So, in essence you saved me a lengthy and somewhat uncomfortable trip to the store, ha ha!

In all seriousness, that is the news I was hoping for - that with the rollout of WebGL 2 support (which all of our shaders need) from the iOS dev team, all the demos would just start magically working across Apple’s newer devices, which covers a huge part of the tech-consuming population.

Thanks again, and enjoy the demos!

[samevision]

I’m glad I saved you from waiting in line. At least a small part I could contribute to this project.