Path-tracing in BabylonJS

Not particularly sure why, but this latest version seems to hard lock my entire desktop PC (requiring it to be power cycled) within about half a second of opening that URL on Chrome. I haven’t dared experiment with it further :sweat_smile:

I have a beefy rig with an RTX 3090 and I’m running it at fullscreen 4K - I wonder if something scales wildly out of control when it has more pixels or more GPU real estate to occupy? My colleagues with lower end machines don’t seem to have the same issue. Will try to narrow down the key factors in replicating it.

Edit1: Does not crash in Firefox, but does crash in Chrome. Resolution of window makes no difference in either browser. Seems to crash after/as soon as bookshelf loads in.

2 Likes

@pcarret
Thanks for the pics - looking good!

About the denoising, could you clarify your question (or needs) - I’m afraid I didn’t quite catch what you were asking.

The denoiser is built into the path tracer, but is not as sophisticated and quick as NVIDIAs deep learning A.I. denoiser for example. If you look at games like Minecraft RTX, the speed of which the diffuse surfaces clean up is amazing (almost once an animation frame or two) Whereas with ours, it’ll clean up after a couple of seconds, or maybe a couple hundred samples.

If you have the patience, you can just let it sit for 1000 samples or so, and it should be all cleaned up pretty nicely - enough to take a screen grab and use for production.

I had a friend in Norway who was using my technology for rendering apartment interiors with complex geometry, sunlight pouring in through half shaded windows, recessed lighting - so my little denoiser was not enough for his use case. He ended up using an open source image denoiser over top mine, and the results (although not in real time for him) were equally amazing - especially on difficult stuff like mirror and glass caustics.

But please clarify what you are wanting to do in the end and I’ll see if I can help. :slight_smile:

@Valentine

Oh, that’s troubling to hear - sorry for the crashes on your 4K system. Yes this path tracing stuff is very intensive on the pixel shader. The complexity and cost does scale (linearly?) with the resolution of your monitor. For instance, on a small scene with 2 spheres and a box room, when I’m at 0.5 pixel resolution, I get 60 fps, but when I bump it up to full 1.0 pixel resolution, I get below 30 fps.

Unfortunately for path tracers, one of the curses is GPU thread divergence. On one pixel, you may have hit a diffuse surface where the ray goes in a random direction on the next bounce loop, meanwhile the very next neighbor pixel may have hit a glass sphere, and must calculate Fresnel reflectance and either reflect or refract - a totally different computation that has no correlation with the other pixel. Things only get worse on the next bounce loop, you can imagine!

So, even if you have a really good graphics card, the monte carlo-style path tracing algo will definitely challenge it. And I can’t even imagine 4K - I haven’t ever tried this code on a 4K system before! But thanks for reporting anyway. It’s good to know all the different situations there might be in the wild.

Could you maybe try reducing the resolution slider to 0.5? Or, if crashing happens before you can touch the slider, maybe change my gui default slider value in my code from starting the program at 1.0, to starting at 0.5? I can do a quick test demo for you on our GitHub repo if you need. I’m curious as to whether this would allow your system to load up the scene correctly, and what kind of framerate you would get with an RTX 3090.

Hope we can find something that works for you! :slight_smile:

I’m getting someone else on the team with a 3090 to try it as well to see what the commonality is - I would have expected simply launching it in a smaller window to solve the issue if it was down to resolution, which doesn’t seem to be the case. But yes, it crashes much too quickly to interact with the UI.

I’ve not had an issue with the path tracer previously, either, it’s only this newest version with the shelves that’s causing this crash. Generally it all runs great :slight_smile: Might be a quirk of my system, or something about the Chrome configuration specifically - again, it all works fine in Firefox.

1 Like

@Valentine
Ok, thanks for the details and updates. Please keep me posted on what happens with the other 3090s. Hope it turns out to be something on Chrome’s end that maybe they can patch in a future release.

@erichlof

About denoising, I did not know there was already a denoiser in your PathTracer.
I have waited until 800 samples, the result is better but still not clean enough especially inside the cabinets and it will take too long for todays customers, so I am just wondering if it is not mandatory to complete the PathRay after few seconds with a denoiser to have a compute time “customer acceptable”
So, my concern is quite the same as your friend in Norway. If anybody has any information about an open source denoiser that can work on a windows server with and without an NVidia graphic card, I would be glad to get the information.

1 Like

@pcarret

Apologies, I may have spoken too soon when I used the phrase, “your image will be production ready” if you simply wait for 1000 samples. Please allow me to explain further, as this applies to all reading.

Denoising is a yet-unsolved problem in computer graphics and is an area of active research for companies (like NVIDIA, OTOY, Autodesk, etc.), and academic institutions alike.

Therefore, the amount of available denoising technology and (even more helpful in my case) actual source code behind that technology is either behind a proprietary wall, too specific for a certain graphics card or single platform, not specific enough and too academic, or inside equations in someone’s university research dissertation (lol).

You can Google “How do I write a ray tracer?” and you may get some decent results and even videos on YouTube. But if you Google “How do I add denoising to an existing path tracer?” , the results will either be not specific enough for your project (as in someone’s doctoral thesis) or non existent altogether.

So my approach has been to take what is available for programs like PhotoShop and GIMP, and apply that basic image denoising tech to the path tracer - well understood and widely used 2d stuff like box blur, edge detection, and Gaussian filters - topics that you can actually learn from the internet because they’ve been around for decades.

The results are fairly good, especially on diffuse surfaces. I wish I could flip a switch and turn off denoising for our project just to demonstrate the awful distracting noise that you would see if we didn’t have denoising in place. So it’s not perfect, but it does make a huge difference- you’ll just have to trust me on that one.

That being said, there are scenes and lighting configurations where no amount of clever denoising (even proprietary professional grade) will get rid of the noise in a timely manner for an impatient end customer. This is because part of the responsibility falls on us, the content creators, to add lights and position objects in the scene in such a way that the denoiser can do its job. If you have a possible customer just dragging and dropping furniture into a path traced room with a single quad light on the ceiling (like our Cornell box), they will never be satisfied.

This is because although path tracing is a beautiful and elegant solution to photo realistic image generation, the algorithm can get tripped up on certain settings. If there are not enough lights, too many lights, not enough bounces in the ray tracer’s light bounce loop, etc, the noise just won’t go away. This why my Norwegian friend had to resort to post processing and PhotoShop style 2d image denoising techniques to clean up his million poly scenes with recessed, hidden lighting and sunlight pouring through the half shaded windows. I’ll share a link to his website in the next post so you all can see what kind of scenes he had to deal with while using the path tracer.

Lastly, the majority of the responsibility falls on my shoulders to give you guys and potentially your customers more rendering options. This is why programs like VRay, Blender Cycles, Arnold, Octane, etc. have so many darn buttons and sliders! Lol But seriously, I want to have features like a side panel GUI that let’s you change the lighting (number and power of lights) in the scene, as well as, update the number of light bounces and quality you need for your particular scene setup (MAX_BOUNCES, MAX_DIFFUSE_BOUNCES, Caustics on/off, etc). - think of any lightmap baker you might have used in the past, and you’ll have an idea. When my friend got access to these dials and settings, he was able to render scenes like yours, no problem. That being said, he did have to play around with the number of lights and light placement, light power, which is so important not just from a technical standpoint for our path tracer to succeed, but also from a visual and artistic one as well for the final consumer.

So, this was my really long way of saying yes, as the path tracer stands now, if you just place the book shelf model into the Cornell Box that only has 1 quad light on the ceiling, it will never be clean enough, and no amount of denoising will do the job. However, if we get the materials working correctly (I don’t even think the book shelf materials are being handled correctly inside my bounces loop as it stands now), the wooden textures up and running (still a w.i.p., lol), and maybe an HDRI in the background, or physical sky with room window to help light reach all the nooks and crannies of your small shelves, everything should be much better, denoising wise.

I’ll work on getting all these buttons and features into our renderer, it just takes time. :slight_smile:

-Erich

My mysterious friend in Norway has name by the way (lol) - it’s Tonny (with 2 n’s). Here’s his project website (his picture is at the bottom if you scroll down):

QiSpace apartment finder

Here’s a link to the building picker:
Apartment building chooser

If you click on either building A or building B, you can then select an actual apartment inside one of those buildings. You’ll then be taken to a gallery of path traced images (yay) from the apartment interiors and balconies.

What’s even cooler is if you select “Virtual View” button near bottom of image, you can walk through the apartment, rotate your camera with dragging, and take a 360 virtual tour.

I am proud to state that all the images on this website were made with the same path tracing technology that you have on the Babylon.js-PathTracing-Renderer GitHub repo. :slight_smile:

I must add though that I cannot take any credit for the other features on this website. What Tonny did with my technology is slick and amazing. He somehow was able to create selectable views of the apartment buildings and rooms (which is very intuitive for a customer), he then presented the gallery in a slick way, and finally the crown jewel, is that he somehow rendered and stitched together all the path traced images he made into a 360 degree view, so that customers can walk through a path traced apartment - noise free! This final touch was brilliant. My path tracing tech is open source, but Tonny did tell me that his other parts of his codebase are proprietary (and rightfully so I guess).

@pcarret I will shoot an email to him and ask about what kind of denoiser post processor he used - I’m fairly sure it was free, but I may be wrong. This collaboration was 2 years ago so I can’t remember all the details of what he was using.

Holy hell! It looks flipping great on iOS 15 on my iPad. On my iPad I’m getting 18fps! WebGL2 makes such a huge difference!

Btw, @erichlof do you twitters? I really want to tweet this out, give proper credit!

3 Likes

@jelster

That’s great news! Thanks for posting that it’s working on iPad also! Yes WebGL2 made a huge difference. I’m so glad that the iOS team made it a priority, even though they didn’t have to (because they have Metal). But when they did offer WebGL2 support, it opened up a huge segment of people on Apple’s mobile devices.

I figured that the latest iPhones could handle real time path tracing, but I am pleasantly surprised to hear that you are getting interactive framerates on your iPad!

About social media, I don’t have Twitter - it’s not a conscious choice or anything, I just haven’t gotten around to creating an account , ha. I am on Facebook however, but I haven’t been active lately because I was starting to feel mysteriously depressed - and then I read somewhere that seeing how good other people’s lives are going compared to yours can have side effects, lol. So I use it sparingly. My wife loves Instagram but I never really got into it - maybe I just don’t get it, ha.

But anyway, by all means tweet away! - I may join soon, because I feel that of all the social media, Twitter will be the one I can actually handle. :sweat_smile:

2 Likes

@jelster
Congrats, you have inspired me to take the plunge! Here’s a link to my freshly created Twitter account:

#ErichLoftis

Let the feeds begin! :grinning_face_with_smiling_eyes:

P.S. It’s humbling (and necessary at times to keep us grounded) to see that I have 0 followers, LOL

3 Likes

You now have one lol

2 Likes

Just caught that! Ha ha
Thank you for being the first, @sebavan :grinning_face_with_smiling_eyes:

2 Likes

Dang it I was too slow now I’m the second! :sunglasses::laughing:

1 Like

@jelster LOL, this is turning into the comments section of a YouTube video - 1st, 2nd, 42nd, ha ha

Seriously though, thanks for the mention (and encouragement!) and I followed you guys right back!

1 Like

How dare you compare these fine environs with the cess-pit that is a YouTube comment thread! For shame, sir! :laughing:

2 Likes

My sincerest apologies - oh the shame, the shame! :laughing:

1 Like

My other colleague tried it on his machine with a 3090 and he gets the same freeze, so seems like its reproducible :thinking: Will stick with Firefox for now, but ping me if there’s anything you’d like me to try!

1 Like

@Valentine
Ok thanks for the update - even though that’s not what I wanted to hear, maybe we can try to further isolate the issue. I’ll work on making a special demo for you that I’ll temporarily place on the GitHub repo. I’ll put the default pixel resolution way down when the app opens, and then maybe you can actually reach the slider before anything crashes. I will be interested to see at what resolution pixelRatio will cause the crash to occur, if it has anything to do with the problem at all (which it might not according to your earlier browser window reduction experiments, but it’s worth a shot).

Will let you know when I get it posted on the repo. :slight_smile:

2 Likes

@sebavan
I’m still wanting to try your suggestion of opening the bookshelf inside Babylon’s playground and seeing where the wood texture is hiding. I just haven’t found the time to start the process, but I will soon - I promise @pcarret !

2 Likes