Path Tracing with Babylon, Background and Implementation

@shimmy
I totally understand now - thank you for clarifying everything for me. I had kind of guessed that the glsl way vs the .js way should not be dependent on one another, and that the multiple sample loop should not be implemented on both files at the same time, but I just wanted to make sure I was running the same code that you had in place on your system.

Having tried both ways, I am leaning more towards your earlier GLSL-only implementation in the shader, as opposed to altering the .js ping pong code. I’ll give you some of the reasons and maybe you could see if these are valid and inline with what you would like the future version of the demos to look like, and how the end-user would be able to change in realtime the new multisampling features.

The first reason I’m leaning towards doing everything in the path tracing shader is that, like you mentioned in an earlier post, it will most likely be more performant. If we examine what is happening in a tight multisample loop inside the main() function of the shader, vs. the multisample loop around the ‘3 stages’ rendering as I named it in the commonFunctions.js file, I believe that a lot less has to go on inside the shader, vs. calling render(scene) 3 separate times (remember the 3 separate stages) in the .js code. I would imagine there is a non-zero penalty for even calling a render function under Babylon.js(or three.js) 3 times, and then piling up multiple occurrences of that inside a loop. I think that when you call render(), it has some sort of setup or communication that the library must do, even if it is a small amount. If for instance, I had a lot of objects with their own matrices, the .js way of looping could possibly add overhead, because when you call render(), I think all of the scene objects’ matrices must be updated and refreshed as dictated by the underlying library. For each animation frame, it would possibly be spinning in place and doing redundant matrix calculations.

The 2nd reason relates to the matrix updates just mentioned, in that variables across different files would have to be changed because of the multiple render calls. The spinning gold torus going too fast illustrates this issue. Going even further, if for instance I had a physics simulation controlling my scene objects (like in my Anti-Gravity Pool game), it would be tricky to dial back the physics motion integration every animation frame, if render was called many times.

By doing the loop in the shader, we can essentially leave the .js code, with its underlying matrix updating and possible physics motion integration, unchanged. Everything should, in theory, run at exactly the same rate as it used to.

So here is what I propose moving forward: how about I try adding some of the same type of variables that you did to the commonFunctions.js file and hook them up to new shader uniforms, so that end-users will have control over how many samples they want. Also I’ll put some weighting variables/uniforms in, so they can strengthen or weaken the influence of previousColor vs. pixelColor and hopefully achieve different frame blending effects like you demonstrated. Or, if they have a powerful enough graphics card that can afford a very high number of samples, they can turn frame blending off altogether.

In the GLSL shader, I’ll add a multisample loop like you did, and also I’ll change how the random seed is calculated as it steps through that loop, as you cleverly did also. Btw, thank you for showing me how to change the seed calculation with multisamples so that the random seed will indeed appear random! :smiley:

Let me work on all this for a few days and as soon as I have something working, I’ll post a Gist here for you and everyone else to see.

I hope this path forward will be inline with what you originally wanted. Looking forward to hearing your thoughts on this. I’ll be back hopefully soon with some new code! :slight_smile:

@shimmy

Hello again! Sorry for the couple day delay. I was able to put together a multi-sample feature system based on your first approach, as well as make multi-sampling work on both Dynamic and Static scenes. Instead of a Gist I went ahead and created a new GitHub repo just for this feature. In the time that we have been discussing all this, three.js changed some things on how they handle WebGL 2.0 and OpenGL ES 3 in their latest library version - you may notice some small differences in my shaders (like the absence of the Version directives at the top, or the final out_FragColor being renamed to pc_fragColor). I just decided to make a repo so there would be no file discrepancies if people tried to use it inside their editor. Plus, as a dedicated repo, you can try everything out, see the source code, see some of the changes I made, and offer feedback or maybe even make a Pull Request if I missed something.

Here’s the link to the new multi-sample demo repo

I used the same demos that you showed in your initial video share on this forum thread. I also added the GameEngine path tracer demo, since that was aimed at dynamic game-type environments, with constantly moving objects. The nice thing is that by keeping most of the sampling work in the shader under WebGL 2.0, the uniforms can be tested against in the path tracing loop - and that means that users can smoothly dial up or down the new feature parameters in real time and immediately see the results the very next animation frame. :slight_smile:

I have a couple of recommendations for all who want to try this out. Increasing samplesPerFrame will bring a lower-spec system to its knees pretty quickly. If you are on modest hardware like I am, I suggest dragging the borders of your browser down to a very small size. Then you will be able to experience 15+ samplesPerFrame in all their glory. The law of diminishing returns applies to our Monte Carlo-style path traced rendering unfortunately, so although there is a big difference between 1 and 4 samples, between 20 and 24 - not so much. But you still pay the linear framerate hit price each sample number you increase. So in the end, somewhere between 1 and 4 samples for low to mid computers, and 8 to 16 for high powered computers should be a nice tradeoff between quality vs. framerate.

The other suggestion is that I found that frameBlendingAmount at around the default 50%, or 0.5 to 0.6, works well in all scenes, all cases, moving or static camera. When I decreased it to 0.0 or 0.1, a fast camera rotation was very crisp, but man, that noise! It was just too distracting. Then at the other end of the spectrum, 0.7 to 0.85 produces pretty much what I already had in place for dynamic scenes, but you’re right - the ghosting is a problem and most gamers don’t like that effect, especially on FPS games. So again, I think a nice compromise is 0.5-0.6 previous frame blend weight.

If you guys like what I have done, I can update my original repo and every demo, whether static or dynamic, will have these new features added. I’ll have to figure out a way to disable this on mobile though, because my smartphone can barely handle 1 sample per frame on some of the heftier demos, let alone 4! :slight_smile:

Have fun! To all, I promise to be back soon with the next installment of the implementation and background posts!

1 Like

So why the focus on path tracing and not sphere marching? Sphere marching SDF’s can get this same effect but with none of the noise and rendering delay.

Hello @Pryme8

Yes sphere tracing / marching could be employed in certain areas and demos, but as I mentioned a couple of background/implementation posts back, this technique is limited in its global Illumination abilities and can actually end up being slower and less accurate than pure ray tracing / path tracing, which uses the traditional analytic ray-primitive raycast intersection functions.

I’ll try to expand on those comments a little. Sphere marching is a good alternative to ray tracing if the scene objects are either simple and mathematical in nature, like spheres, cylinders, boxes, etc., or they have some kind of twisting/ warping to these simple shapes (Shadertoy has tons of examples), or if they are fractal, like terrain, water waves, clouds, etc., where pure traditional ray tracing could not be used because there’s no analytic solution for these fractal shapes. So in fact I have done several sphere marching-only demos on my repo as you suggest, but most of those are the above mentioned fractal shapes and mostly outdoor environments. The only exception is that I employ sphere marching to efficiently trace a torus, which has a quartic solution with 4 possible roots (intersection points), and is notoriously difficult and mathematically unstable to raycast against traditionally.

Leaving the torus aside, I prefer traditional ray/path tracing when it comes to simple mathematical shapes. The intersection can be found instantly with algebra rather than waiting for the marching spheres to get close enough to the distance field. Also, with traditional ray casting, the returned rendering is pixel perfect no matter the resolution, as opposed to marching shapes, which end up having gaps and slightly warped areas around the object silhouettes if we can’t afford 500+ marching steps (and 200 is usually expensive enough, already slowing down the framerate).

Here’s the other caveat about exclusively using sphere marching: it can result in more severe GPU thread divergence which usually cuts the FPS in half. If you try out my terrain rendering demo, notice how when you point the camera straight up just at the sky, which sphere-marches clouds, you get the full framerate, 50-60fps. Likewise, try hovering right above the rocky landscape and point the camera directly down at your virtual feet, and you get full framerate again, 50-60fps on my laptop. But now, hover up in the air, (helicopter altitude) and point the camera towards the horizon, where you can see the sky on the top half of the screen and the terrain on the lower half of the screen, and notice how the framerate tanks. This is because GPU threads prefer doing the same thing as their neighbor threads, and in this last case, half are doing something completely different and ending their work at randomly different times than the other half. This is a real issue with sphere marching in general because often at any given moment, we need anywhere from 10-500 steps, which is big range and likely divergent. This is why I only use it when I can’t do pure ray tracing on a complicated fractal shape found in nature (or a torus!). :wink:

We haven’t even discussed models and meshes yet, but another major limitation to sphere marching is that it does not efficiently render full triangular polygon meshes which are the norm in 3d graphics pipelines. A sphere marching to a triangle function is even more costly than the famous optimized pure ray-triangle intersection routine from rendering literature decades ago (I think it’s the Mueller Trumbol or something - in any case I use it exclusively when rendering anything with triangles).

But if it were possible to create a distance field surrounding a traditional polygon mesh, like a tight bubble, you could sphere march it. However, due to the resulting expensive distance field equations of an arbitrarily shaped triangle mesh, it ends up having to be just an approximate shape in order to keep the frame rate real time. Once you have this approximate mesh-shaped bubble calculated (must be calculated offline too, can’t do this pre-process in real time), then you can traditionally rasterize the entire scene (which is the fastest part), then send out shadow marching spheres towards these approximate shapes surrounding the meshes. This results in fairly good shadows without the need for shadow maps, but not without less setup and effort in my opinion. I believe that in addition to shadows, some people have used this same technique to add ambient occlusion and even second- bounce GI color bleeding, because when you deal with ao and color bleeding, triangle-accurate shadow/GI gathering shapes aren’t absolutely necessary to the final output, but this does bias the renderer. The process i just described is exactly what Unreal tried in the past with limited success and mixed reviews.
To read more about this traditional rasterization plus shadow/ao technique using SDF marching, please have a look at this
Unreal Article

You’ll see that a lot of work must be done offline to get this working. Even then, you can’t do caustics (like glass refractive-focusing on diffuse surfaces) with this technique like you can with traditional ray/path tracing.

So in conclusion, sphere marching has it’s place inside the bigger picture, and I rely on it for certain situations where ray casting won’t do the job, but its drawbacks and limitations have lead me to focus on traditional ray/path tracing, especially when it comes to BVH and triangular meshes.

Having said all that, if you were wanting me to discuss the actual technique of sphere marching more (as I have with traditional ray casting) and how I did my outdoor environments, I would be happy to! :smiley:

Sorry this reply got so long. I just love talking about rendering!

3 Likes

@Necips
That was a fun and informative intro to path tracing! I love the old-timey narrator and feel. This will tie in nicely with my next 2 posts: lighting, and one on materials. Thanks for sharing! :smiley:

1 Like

Part 6 - Implementation (cont.)

So we last left off with placing objects in the scene and looking at ways of tracing the geometry and shapes that we want to have path traced. Now I thought it would be a good time to discuss lights and scene lighting as they relate to Babylon and our WebGL path tracer shader.

Good lighting is probably the most important and difficult area to get right in computer graphics, whether you are rasterizing in the traditional way, or path tracing like we are. Luckily for us, light sources fall into a handful of broad categories, and there are tried and true methods for path tracing these different types of lights.

The broad categories of lights are: Point lights, Spherical area lights, Rectangular area lights, Spot lights, Directional lights (usually the sun), and natural indoor/outdoor(mainly) sky or dome environment lighting (usually handled with either a HDRI map or a physical sky function with atmospheric scattering). You probably recognize some of these from using Babylon.js, as most engines have some notion of point, directional, spot, sky dome, sky box, and maybe even area lights(area lights are only an approximation if you have been rasterizing).

There’s good news and not so good news when it comes to path tracing these various light source categories. The good news is that, as previously mentioned, there are a lot of fast, efficient, and ground-truth-reality-correct routines already in ray tracing literature, some of which have been around since the early 1980’s. No matter how big or infinitely small your light sources are, there is a way to use them in a path tracer - often in a handful of lines of code for each source. The beauty of path tracing is that for a small amount of code that is low-complexity (compared to rasterization lighting effects and shadowing techniques), you get in return a perfect ground truth reality image, assuming you let it refine enough.

The not so good news is that in order to effectively trace a practically-infinite amount of light rays out in the scene, we need some way of randomly selecting a handful of light ray segments in a randomly chosen ray ‘path’ to work with, just like the above video’s narrator mentions. And any time you introduce the words ‘random’ or ‘randomly selected’ in computer graphics, you automatically get the noise that goes along with it.

Everyone who writes a path tracer (including me) sooner or later has to confront the lighting/shadowing noise problem. If you just implement a naive path tracer text-book style, as all of us rendering programmers have when we were starting out, you will get unbearable noise on even the most simple scenes with 1 sphere and a ground plane. Luckily there are 3 ways to lessen the noise or get around the problem altogether: denoising, taking more samples, and more clever light sampling. I rely on the latter, but I’ll briefly discuss denoising first.

Some renderers rely on A.I.-assisted denoisers to try to mitigate some of this noise that is negatively associated with path tracing. For instance, Nvidia has an excellent proprietary A.I. denoiser for its RTX system. A typical path traced extremely noisy image is the input, and the output is a believeable, fully cleaned-up rendering, at 60 fps! They somehow train the A.I. with tons of before-and-after images of what scenes should look like after they have had time to progressively refine. I still don’t quite understand how all that magic works, but maybe in the near future I can experiment with my own small non-A.I. denoiser for this project.

The 2nd tactic is just to take more samples per frame. If you’ve been following my side thread with @shimmy, then you have already seen this in action. The only issue with this approach is that the user must have a pretty recent, powerful graphics card to be able to maximize the amount of samples per frame. Since from the very start of my project I have wanted to include those people with commodity hardware and even mobile, I have focused mainly on the last option.

As mentioned, I rely on smarter, more efficient light source sampling routines to combat noise, some of which are 40 years old, and some, like in my volumetric fog scattering demo, are taken from research papers only 5 years old. Without getting too technical, basically these clever routines exploit the probabilistic nature of light sampling, so that either only 1 sample is required to get decent lighting, or the handful of samples that you do have to take really count!

BTW the reason for noise in path tracing lighting is the sampling of randomly chosen light ray paths, which was inspired by the Monte Carlo numerical integration method. If everyone wants, I can go into more depth on Monte Carlo in a future path tracing post. But for now, Monte Carlo basically boils down to ‘sample and average’. Just like a team conducting a nationwide poll, it is very inefficient and even impossible to interview every member of a population. But if we truly randomly select a good sized sample group and divide the sum of their answers by the number of citizens sampled (average), then we can be pretty confident that our results would closely align with the entire population, if we were somehow able to completely sample every single person. That is the beauty of Monte Carlo rendering. We can’t possibly sample or even count the number of light rays in a scene, but if we cleverly sample, we can still get a good unbiased answer back relatively quickly on how the scene should be lit (if we were able to poll every ray, like in the real world with our eyes!) :slight_smile:

I’ll now briefly cover each of the light categories and how they are handled by the path tracer.

Since Point lights are infinitely small, they are not found in nature. But they can be useful in certain situations. More of a relic of 80s ray tracing (the classic movie TRON comes to mind), point lights produce crisp, hard, pixel perfect shadows (again, impossible in nature). The one benefit of using point lights is that since we don’t have to sample different spots on the light source/bulb (because it is a mathematical point), there is no noise at all in the light or shadows - and that is why older renderers used them. They did not have the CPU power to take multiple paths/passes from a physical area light source.

Spherical area lights and Rectangular area lights use very similar techniques, so I’m discussing them together. These have physical properties found in reality and can lead to very realistic indoor scenes. Since we sample different spots on the surface of the sphere or rectangle from different angles, we get physically correct soft shadows and smooth lighting. Again, along with that benefit comes the baggage of sampling noise that we must battle with.

Directional lights are one of the easiest to implement as they are really just represented by their namesake - a directional vector. Usually normalized to a unit length of 1, this vector gives the direction that the energy is flowing. Since the sun is so bright and far away, its light rays are effectively parallel by the time they reach the Earth, and therefore all of the sun’s rays can be described by a single 3d direction vector. One of the benefits of outdoor lighting with directional sources is that there is much less noise since we only have a narrow band of ray samples to deal with. There is still some shadow noise as objects get farther from the ground, but it is much less evident than indoor shadow noise resulting from sampling a small, bright light bulb in a darkly painted room.

Spot lights in traditional rasterizing renderers usually have an angle and cutoff so that the triangles can be analytically lit. However, I have found that since we’re dealing with physical lights, it is better to construct a simple physical model of the spot light, and then let the ray tracer naturally do its job. By this I mean actually make a metallic open cylinder or open box in the scene and place a bright spherical light inside of the metal casing. To see this in action, please check out my BVH_Spot_Light_Source demo. Notice that you get the spotlight circle and realistic penumbra for free. Again though, with sampling that physical spot light bulb, along comes noise again.

HDRI environment images and physical sky dome environments can produce very realistic scene lighting, provided that you are careful about where and when you sample various parts of the surrounding scene dome or surrounding sphere/sky box. HDRIs are a little tricky in that you have to know the location of the sun or light sources in precise uv coordinates of the original HDRI texture image. To see this in action, please check out my HDRI_Environment demo with the Stanford Dragon in an outdoor garden environment.

The only category I have not covered (and not tried yet in my own project) is volumetric light sources. The most familiar examples would be candlelight, campfires, explosions, lava, space nebulae, etc. However, if we use ray marching like previously mentioned in earlier posts, it is possible to trace these nebulous, ever-changing sources as well. But with anything marching or stepwise, comes the potential hit to framerate, especially on lower-powered devices.

On my TODO list is to allow users to call the familiar Babylon.js light setup functions like new Directional Light, and have that translate into path tracing shader code. But for now, it’s best to just hard code the lights you need into the shader’s scene function - so you actually don’t need to call any light instantiation functions at all with Babylon. However, if you need the lights to move or rotate, or to give a direction vector for the sun, you can indeed use Babylon’s notion of an empty 3d transform (like an editor gizmo), that will update from your js code every frame and then get sent over to the path tracing shader via a uniform. This follows exactly with how objects were transformed in the scene by the host js library, as covered in the previous post. If you need examples to work from, I have separate demos dealing with each kind of light source I mentioned above.

Next time we will cover different kinds of materials and how to deal with ray paths and as always, the noise problem that comes along with sampling rough surfaces (like a diffuse wall, or frosted glass).

Till next time! :slight_smile:

3 Likes

Part 7a - Implementation (cont.)

The last part in this Implementation series of posts will be on different types of surface materials. Similar to lights in the previous post, materials have some overlap with the underlying library, Babylon.js (mainly in naming conventions and parameter names, i.e. color, roughness, opacity, etc.), but how these materials are handled in our path tracer requires further investigation.

There are almost countless types of materials found in the real world, but just like the light source types, luckily all materials end up falling into a handful of broad categories for path tracing purposes. I will discuss each broad category below, which will cover all conceivable materials. You might think that 5 or 6 categories is too limiting for describing millions of possible surfaces, but what ends up differentiating these seemingly countless materials is their parameters and numerous parameter combinations like color (that can come from a texture image), normalMap(also from a texture), metallic or not, index of refraction, roughness, clearCoat, etc.

What follows is a brief description of the main categories to which all materials belong.
Let’s start with the easiest surface to define and to path trace - an emissive material, or aka - a light! Wait I thought we covered lights already? Well yes we did, but it was more focused on their shape or form than than their actual material and emissive properties. When we have a light type in place, like a directional light or a point light, we need to give some sort of value for how much power is steadily radiating from its surface. What I failed to mention in the previous post is that with lights, you can either define their power or intensity with real world units and measurements (joules, lumens, or Watts), or it can be with relative units. Since most three.js and Babylon.js scenes have relative measurement units, like 10 instead of 10mm or 10cm, it makes sense for the lights to also give off radiation in relative units of power, like 100 or 1000. Keep in mind that these are arbitrary relative numbers, just like your 3d scene relative dimension units, that must be experimented with a little to your tastes. I know that some serious professional rendering packages offer real world light units and measurements with light bulb specifications in Watts or lumens, but to use this physical data effectively, the whole scene would need to be changed in order to accommodate a cm or meter basis. Therefore, we just simply assign a relative power to the light surface, so if 1 is your unit (which it is in my case usually), then an intensity of 10 would be 10 times as bright, an intensity of 100 would be 10 times as bright as the 10-power light, and so on.
That all aside, light surfaces are usually defined with an intensity in RGB values. So a bright reddish light would be (10, 1, 0.5) or similar. The reason I said these were simple surfaces is that these 3 rgb numbers are all we need to totally define an emissive surface’s radiance! Additionally, in path tracing when you encounter a light or a shape with an emissive material, you are done tracing for that frame! Since we are ray tracing in reverse vs. the direction found in nature, rays do not ‘bounce’ back off from a bright light once you find it. One of the best sayings in path tracing theory is “Always follow the light!”. We won’t see a thing until our rays encounter a surface with an emissive material. And once we do find a light, we’re done!

The next easiest material is the Metals category. Metal (aka specular) materials are also simply defined by an RGB specular reflectance. In the real world, the metal that comes very close to a pure white (unaltered and non-tinted) reflection is aluminum, with relative RGB reflectance of (0.99,0.99,0.98) or something like that, I can’t exactly recall. But anyway, you can give a metal surface a value of white and be close enough to the eye to look like an aluminum backed mirror. What’s interesting about metals is that they tint the outgoing ray when it has reflected off the metal surface. So if you have a blue metal sphere, any rays reflecting will be tinted blue, and therefore the entire reflected image on the sphere’s surface will be also tinted blue. That is why gold throughout history is so prized and coveted - in addition to being rare, it reflects light and tints reflected light in a fiery red/orange color spectrum, which makes its surface reflections brilliant, fiery, warm and captivating to human eyes.

In code, Metals also have the most simple 1-line ray reflectance function that uses the glsl intrinsic reflect(incomingRay, surfaceNormal) function. The angle of the outgoing ray matches that of the incoming ray, everything being oriented about the surface normal.

The next broad category has a fancy name: dielectric materials. That means that this type of material does not conduct electricity or is a poor conductor, if anything. In fact, not counting emissive materials (lights) and dismissing rarely encountered semiconductor materials, we can even further reduce the number of rendering material categories to 2! - metals and dialectrics. Metals conduct electricity, dialectrics do not. Metals reflect practically 100% of all incoming light, dialectrics reflect some, and let the remainder pass through their surfaces - all according to their fresnel reflectance and viewing angle. Metals tint their reflected scenes by multiplying the reflected rays by their surface color reflectance, dialectrics leave the reflections un-tinted, or you can think of the rays getting multiplied by pure white (1.0,1.0,1.0) which is like multiplying a number by 1.

Only a small percentage of real world materials are metals that conduct electricity.
Most materials in nature are dialectrics, or insulators (do not conduct electricity). Well-known examples of dialectrics include water, glass, plastic, skin, rubber, wood, concrete, even gasses like air, etc. - pretty much everything except metal!

When we are path tracing dialectrics, a percentage of the rays get reflected off the surface (and left un-tinted, remember), and the remaining percentage of rays must be refracted, which is a fancy word for letting the rays pass through the surface to the other side. The interesting phenomenon is that if the dialectric surface is transparent, and the rays pass through, they are colored (or tinted) by the rgb color of the surface interior! So there is a dual nature to dialectrics like murky water and colored plastic or colored glass. Imagine a perfectly smooth, shiny red plastic ball - if you look at the sides of it from a glancing angle, the reflected scene will be unaltered white, not tinted red, but left whatever color it’s going to be. Same with a still, blue lake as viewed from a glancing angle- the mountains and trees and clouds in the mirror-like reflection are not tinted blue, but remain unchanged. Now if you are able to see through the surface and the rays encounter particles in a medium that they are traveling through, like colored glass, or a murky green algae-infestesed pool, then whatever is on the other side of the surface boundary will indeed be colored, and the deeper the rays go into the medium, the more color tinted they get. Usually in ray tracing we further group dialectrics into transparent materials like clear plastic, clear glass, colored glass, gems and water, and then we place all the other thicker, more opaque mediums like skin, rubber, wood, marble, chalk, wax, etc., into an ideal diffuse group, (or subsurface scattering group if you have a very sophisticated renderer), all of which have refraction and subsurface scattering of rays just below the surface which gives them their warmth and depth, and sometimes even their apparent glow.

When it comes to tracing rays through the first group of dialectrics (water, glass, plastic, diamond, precious stones, etc.), the code gets a little more mathematically intense, as it has to first calculate the Fresnel reflectance, which gives the correct proportion of light that must be refracted (or allowed to pass and subsequently tinted) vs. the proportion of light that must be reflected and left un-tinted. This all relies on the angle that we are viewing the surface from, as well as the material’s index of refraction, or IoR for short. The IoR serves 2 purposes - first it gives the relative amount of reflected rays that will bounce off the surface like from an aluminum mirror. The higher the IoR, the more pronounced this reflection becomes at grazing angles. The second purpose is that the IoR controls how much the refracted rays will get bent when they pass through the surface boundary. The higher the number, the more the rays are bent. If you consider an expertly cut diamond, it has one of the highest index of refraction values of all clear materials found on earth, and with that high number comes bright, strong reflections at grazing angles, plus if the rays do pass through, they are so severely bent that they end up getting bounced around many times inside the diamond and even may undergo total internal reflection. This severe refraction, bright grazing reflection, and multiple total internal reflections resulting from its high IoR, gives diamonds their bright, sparkling, fiery, brilliant look.

Now we move on to the Diffuse category. Diffuse materials are often a convenient, idealistic abstraction of what really happens in nature with thick, opaque dialectrics - subsurface scattering. Going back to the red ball example, let’s instead make it a dull red, rubber ball - like a worn gym kickball. If we were to examine what happens with rays entering the ball’s surface, we would find that they bounce around randomly, interact with the microscopic paint particles in the material, and then exit somewhere different than they entered. This is true of most opaque dialectrics that have some thickness to them. However, even though it is possible, subsurface scattering is very difficult and expensive time-wise to model in a path tracer. So usually we idealize these materials into a broad Diffuse category. Instead of interacting underneath the material’s surface and progressively picking up the surface color, light rays are immediately tinted the surface color and then randomly reflected from the surface to continue on their journey through the scene. The diffuse reflection directions are chosen inside an imaginary tiny hemisphere that is oriented around the surface normal. The flat bottom of the hemisphere lies on the surface, while its top dome apex coincides with the surface normal’s tip. The surface normal is normalized and is of length 1, and so are the dimensions of the hemisphere - radius 1, and apex dome height of 1. Diffuse surfaces like chalk have many microscopic bumps and crevices, which make incoming rays reflect in seemingly random directions all over the hemisphere. Contrast this with the smooth V shape of the incoming and outgoing reflection ray when it encounters a smooth metallic surface.

In code, just like the metal surface tinting the outgoing reflected rays, so does our idealized diffuse model (but we know the real truth - rays undergo subsurface scattering underneath the material’s surface boundary, causing them to pick up the reflected color of the object). In path tracing, we follow the life of a ray as it bounces around the scene. If it hits a rough / or dull diffuse object, we must randomly select an outgoing reflection direction somewhere in the hemisphere. Oops, there are those words “randomly select” again! And by now we all know what that means: our friend, noise is back!

Luckily for us there are well-established, clever sampling strategies for ideal diffuse surfaces, so that all paths that are randomly chosen will make the most contribution to the final color and lighting of the diffuse surface. No samples are wasted.

The final category that I’d like to talk about is ClearCoat: or diffuse underneath / ClearCoat on top. 2 examples that come to mind are billiard balls and a heavily polished wood table. If you view the edges of these objects, or view their surfaces from a glancing angle, you will see a pure, un-tinted reflection. But at the same time you can see through the clear coating (polish), and if you view the surface parallel to the surface normal, it looks like a diffuse object underneath at the same time. Again with anything clear, like ClearCoat materials, you have to specify an IoR for the clear coating itself.

When path tracing ClearCoat materials, first a reflection ray is cast out, based on the viewing angle and reflective properties of the IoR. Then, since the material has an idealized Diffuse surface underneath, we must treat it just like we did with normal Diffuse materials- making the incoming ray pick a new random scattering path direction when it interacts with the microscopically rough (or subsurface) material.

Well I apologize - this part 7 post is getting way too long. I will return soon with part 7b, which will cover roughness(on metals and glass, etc.), and how it affects materials when path tracing. Also I’ll briefly touch upon material BRDF sampling and how we can sample a rough metal material or frosted glass for instance, just like we randomly sampled a spherical area light all over its surface before.

I’ll be back soon! :slight_smile:

2 Likes

Part 7b - Implementation (cont.)

Ok I’m back with the continuation of materials and how they are efficiently sampled in our path tracer!

In the previous post we discussed all the materials categories and how they were differentiated from one another and handled. In this post I’ll touch upon how material parameters like roughness can further customize the look and feel of various materials, even within the same category. Also we’ll explore more efficient sampling strategies, especially how Monte Carlo integration and even statistics can help us with both rough specular and diffuse surfaces.

Let’s start with roughness. This is a fairly universal material parameter and one that is very useful for easily changing the appearance of an otherwise too-plain or too-sterile looking material. Roughness is usually given in a normalized range between 0.0 and 1.0, 0.0 being completely smooth and 1.0 being completely rough. Physically, roughness usually refers to very small bumps/valleys and imperfections in the surface, most of which are too small to see with the unaided eye.

Let’s first imagine a thin square sheet of aluminum lying on the ground. This sheet of metal has a roughness parameter of 0.0, or perfectly smooth. An incoming ray will reflect perfectly around the surface normal. If you look into the reflected image on this piece of flat metal lying on the ground, it would operate much like a mirror, giving a perfectly crisp reflection image. But as we increase the microscopic roughness of the sheet of metal, the reflected image becomes a little more fuzzy and out of focus, but we can kind of still make out what the reflected objects are - until at last with a roughness value of 1.0, the reflected image is completely fuzzy and unrecognizable. This is because the incoming rays are reflected in more random directions due to the surface bumps and valleys associated with an increasingly rough surface. To get this working in a path tracer, I first tried (as everyone else has when they’re starting out) taking the perfect mirror reflection vector, and randomly nudging it, based on how high the roughness value was set. Although this appears to work at first, this strategy usually is accompanied with too much noise.

Let’s take a moment and jump over to the diffuse materials. Technically these surfaces always act like their roughness is set to the maximum, or 1.0. I also first tried completely randomizing the rays that were reflected from this kind of surface, only to again encounter way too much noise. I was dealing with the same problem, even worse with diffuse because it is a seemingly totally random reflection - and remember any time you introduce the word random in rendering, you have to deal with noise. There has to be a better way to reduce noise in these types of materials!

And luckily there is. Much later on in my project, I found out about Monte Carlo integration through importance sampling BRDFs. Hold on, there were a lot of new vocabulary words in that last sentence! Let’s break some of those terms down:

Monte Carlo integration basically means randomly sample and average, sample and average, over and over until you feel that you have sampled a good enough sized group out of the entire possible population. Similar techniques are used in statistics, surveys, and polls. One major requirement is that you must be unbiased about your sample group selection, otherwise the results will be biased. Each sample must be chosen in a random manner in order to have confidence that your results will match that of the entire population, if you were somehow able to sample them all.

The second term in the opening sentence, ‘Importance’ sampling, at first glance sounds like it is in direct conflict with what I just wrote! How can we decide who’s important to sample and who’s not? I thought we weren’t supposed to bias our random sample group selection? But many years ago, some clever mathematicians realized that they could use probability, namely probability densities, to more intelligently random-sample, in order to make the noisy random results converge much faster to the answer. I can go into more theory in a later post if people want, but basically if you weight the random samples according to how likely they are to produce a certain mean average value, then you can safely sample more from the sample group candidates that are likely to give an answer that is near the final average or median, and likewise safely sample less from the outliers who would be at the extreme left or extreme right of the median average and wouldn’t contribute much to the final answer anyway. For example, imagine a bell curve - if you knew that the results of a randomized and averaged experiment would look like this bell curve (and most randomized experiments end up fitting that exact shape, due to the ‘law of large numbers’ in statistics), then you would probably want to take more random samples from the area where the results curve is at its highest points - and this would be near the middle of the bell shape. Likewise, you could safely decrease the number of wasted random samples taken from the right and left extreme edges of the bell curve, because their output contribution is close to 0 and won’t matter much in the big picture, after you’ve tallied and averaged the whole sample group’s answers. Thus we can save our precious samples for the bulk of the population that really counts towards the converged answer. This is called ‘importance’ sampling because we are sampling from areas of the population that we know will contribute the most, and therefore are considered ‘important’. And the beauty of it all is that our final answer that we get back much faster, and with far less noise, with importance sampling is still 100% unbiased! This technique has major wide-ranging benefits for all areas of math and scientific research, and for CG rendering / path tracing in particular. Just to be clear though, in order to remain truly unbiased, there must be a small non-zero probability of randomly choosing an outlier on the bell shape to sample. Even if the chances are very tiny, there must be a handful of samples taken from all areas of the curve including its most remote edges, even if they don’t seem to contribute much to the averaged, converged answer at the median.

Ok enough statistics, so how does all this relate to diffuse or rough specular surfaces, you are probably wondering? Well, it turns out that the bell shape or any similar 2d probability density shape can be stretched out in 3 dimensions to accommodate any 3d shape or surface. One shape that is particularly of interest to us in path tracing is the hemisphere shape. So going back to our diffuse surface sampling, we need a reliable and non-noisy way of sending out random diffuse reflection ray sampling paths in the hemisphere above the point to be shaded and lit.

I’m sure you have at one time or another heard of Lambert’s cosine law, or Lambert diffuse, or cosine falloff, or perhaps in code ‘NdotL’ lighting. All of these terms say the same thing: the more aligned with the normal your light rays are, the more strong the lighting. The cosine of a ray in parallel with the normal is 1.0. On the other hand, the further you stray away from the normal towards the bottom edges of the hemisphere, the weaker the light becomes. The cosine of a ray that is perpendicular to the surface normal is 0.0. This cosine can be found using the dot product between the surface normal and the light ray in question. That’s the reason that simple NdotL lighting in old rasterization engines looks pretty good and convincing on diffuse surfaces! :slight_smile:

So armed with this cosine falloff weighting for diffuse surfaces, is there a way we can treat the hemisphere as sort of a bell curve and importance sample it? Yes! And this very algorithm is relied on by all major renderers even to this day. Nothing better has yet been found in scientific CG research for diffuse reflection sampling. Essentially what happens is the hemisphere of infinite ray sample choices becomes the ‘population’ in our survey or poll. And the shape of Lambert’s cosine law looks almost like a perfect 2d bell curve, with the surface normal on the graph being right at the median or middle of the bell curve - the highest point with the strongest contribution to the final answer of the amount and color of light we need to gather or survey (integrate) from the infinite ray population of the hemisphere. If you imagine a solid 3d bell made to hug the hemisphere surface, you can see where you should sample the most - the area around the surface normal’s direction! And as we get to the sides and bottom of the hemisphere, we get the outliers to the left and right extremes of the final averaged answer that don’t make much of a contribution or difference at all, because of their extreme cosine falloff. But remember, to be an unbiased renderer, we must take at least a few samples from these outlying areas, even if the probability is low that we will choose them. The probability of them being picked must be above 0.0 if they contribute anything above 0.0 at all to the final averaged answer or bell shaped curve.

This technique is known in the CG world as Cosine Weighted Importance Sampling. It drastically reduces the inherent noise that goes along with any Monte Carlo path tracer, and it is used exclusively in our path tracer and on all of my demos.

Jumping back to the metal sheet with roughness applied, is there a similar way we can importance sample the specular reflection direction choices as they relate to the amount of specular roughness? Again, yes there is! Although at the beginning we have to think a little differently about the bell curve shape as well as how it is oriented in the hemisphere. Imagine if you will the bell shape around the normal from the previous diffuse example. Now make the surface change to our flat metal sheet again with the same surface normal pointing up. Imagine an incoming light ray coming from the left side of the upper hemisphere, going downward, then it strikes the smooth metal surface, bouncing back up at a similar angle and it keeps going to the right, eventually leaving the hemisphere’s upper surface. This is a classic perfect mirror reflection. It sort of looks like a letter V and the light ray follows your pencil if you were drawing that letter V from the top. Now imagine in your mind rotating that bell shape (that was originally centered around the surface normal vector which also bisects our V). All we do is rotate that 3d density bell curve down to the right until it is exactly aligned with the outgoing mirror reflection ray, or the right side of our V, if you will. This would represent a probability density of 1.0 or full roughness on the metal. The reflected image would be extremely blurry and unrecognizable, much the same as the diffuse surface. But since it is importance sampled, it will also converge much quicker!

Finally, imagine decreasing the roughness of the metal, and we would find that the wide bell curve starts to stretch out and in doing so, it becomes more narrow and more narrow, until finally with a roughness value of 0.0 (or totally smooth metal), the very thin bell shape collapses to a line - which is the outgoing reflected vector on the right leg of our V! To be precise, when referring to specular reflection probability densities, this shape of possible ray population sample choices is called a Specular Lobe. This 3d lobe shape kind of looks like a stretched out Q-tip or a severely elongated teardrop. So the difference between diffuse and metal ray path sampling is that the bell shape to sample from when dealing with diffuse materials is centered around the surface normal vector always, and when dealing with smooth to rough metals and specular surfaces, the bell becomes more of an elongated lobe or teardrop shape that is centered around the perfect outgoing mirror direction vector only, not around the normal vector (unless of course the incoming light ray is coming straight down the normal and must be reflected back along the normal!)

A good sampling routine for arbitrarily rough specular surfaces like metals relies on the powered cosine exponent of the famous and historic Phong material / lighting model. If you look at a 3d graph of Phong reflectance (or its BRDF), its powered cosine around the mirror reflection direction resembles the lobe shape that we were seeking to efficiently importance sample from, when dealing with rough specular surfaces. As the exponent grows higher, the teardrop lobe becomes thinner and more stretched out, finally hugging the mirror reflection vector almost exactly. That is also why we see a smaller and smaller and more pronounced highlight on Phong materials as we increase the specular exponent - the reflection lobe shape and resulting highlight gets tighter and tighter, as there is less room for possible reflection rays.

You might be thinking I forgot about the last remaining vocabulary term in the above opening statement, BRDF, but I haven’t! :smiley:

The materials we discussed earlier, Lambert diffuse, Perfect mirror, Phong Specular, all can be mathematically described with their BRDFs. BRDF stands for Bidirectional Reflectance Distribution Function. Now hopefully we have the necessary information and statistics context to see where this name comes from. The letter B in the acronym stands for Bidirectional, which means that it works both ways - you can describe the directional probability of a ray exiting if you are first given its incoming direction, or if you are given its outgoing reflection direction possibility lobe shape, you can reverse time and determine the light ray’s initial incoming direction - pretty nifty! The letter R in BRDF stands for Reflectance, which just means how light bounces off materials. The letter D in BRDF stands for Distribution, which directly relates to every probabality density shape that we discussed earlier: all the way from the 3d bell shape of a Lambert material BRDF centered around the normal, to the stretched teardrop lobe possibility shape of specular materials like the Phong material BRDF, centered not around the surface normal, but rather around the perfect mirror reflection outgoing vector. Finally the F in BRDF stands for Function, which in math means you give it an input and you get a reliable output (or a good probability of possible outputs). If we give this function an input of an incoming ray, we get back the output of an outgoing reflected ray (or a possibility lobe shape of reflection ray directions, or even a whole hemisphere bell shape of possible reflected rays in diffuse). Likewise, given an output of a reflection probability density shape, you can rewind time and calculate the input to the function, which was the incoming ray. The only BRDF where this backwards rewinding of time to find the input incoming ray doesn’t work is with Lambert diffuse. The output that is given is a bell shape always centered around the surface normal, which does not help us because an incoming ray could have come from anywhere on the hemisphere and its reflection possibility distribution would still look like a bell shape centered around the normal. But this is the only BRDF input that is non determinant. The other BRDF shapes inputs can be investigated and confidently determined, much like a police ballistics expert determining the possible angle of an incoming bullet at a crime scene.

Finally, I’ll repeat the opening statement sentence: Monte Carlo integration through importance sampling of material BRDFs. Now hopefully you know: Monte Carlo integration (a bunch of sampling and averaging) through importance sampling (leveraging statistics to cleverly choose random samples, but still remaining unbiased) of material BRDFs (the 3d graph showing probability distribution of possible incoming and outgoing reflected rays for a particular material). Now we got it!

Well, another epic post, ha ha. I didn’t intend to go down the rendering theory and statistics road as much as I ended up doing, but I hope I have explained these seemingly complex ideas in more simple terms that you can visualize in your mind’s eye. If you take anything away from my lengthy post, I hope it will be a picture of what a material’s BRDF looks like. If someone ever says ‘diffuse BRDF’, just imagine a hemisphere and a 3d bell curve shape overlapping it and centered around the surface normal. Likewise, if someone says ‘specular BRDF’, just imagine a vertical normal vector bisecting a V and a stretched teardrop hugging the right leg of that V. This is all you need to understand the mouthful of words that make up the complex-sounding acronym BRDF! :smiley:

If you all would like, I can go even further down the Monte Carlo and statistics path, or discuss topics like Bidirectional path tracing (hopefully now you can guess why that sampling strategy is even possible! :smiley: ), or go into acceleration structures like BVHs, etc. I’ll kind of take a break from epic posts for now and let you guys chime in if you want to see this thread go in a certain direction.

Cheers!
-Erich

2 Likes

Hello @Necips !
I thought the silence was because everyone was tired of me rambling on and on about path tracing! Ha :smiley:

No but seriously, thank you for the compliment. And thank you for sharing your beautiful Sandbox shader! I love the Mandelbrot addition - so cool!

It’s fitting you linked to a shader with the original code coming from P_Malin. Now there is a true ‘giant’ in shader programming. Have a look at his profile (P_Malin) and his listed shaders on Shadertoy. Some of his demos are pure magic!

Back before I got into rendering, I had also stumbled upon his incredible work on glslSandbox, and in fact I had bookmarked the very shader you had worked from. What struck me besides his beautiful results, was that if you look at his source code, it is so legible and organized. Even the naming of each variable was taken into careful consideration. Someone stumbling upon his shaders who wanted to browse the source, actually has a chance of understanding what he is doing on every line and in each function. That legibility and clearness of intention is a great source of inspiration as well as his brilliant techniques!

P_Malin along with the legendary iq, PauloFalcao, TDM, and koiava have greatly influenced my development as a webgl pathtracing shader programmer. Also the path tracing port by scott ( just ‘scott’ on Shadertoy) from Kevin Beason’s C++ smallpt to Shadertoy was a great help for me getting started with my three.js renderer project.

About your question on whether I do this professionally, actually programming is a passion hobby of mine. With all my lengthy posts on the background of path tracing, I failed to mention my own background! Ha. So here is my story:

I am now 47 years old (although my line of work and hobbies allow me to ‘feel’ much younger! :smiley: ), I am a professional musician and music teacher in Texas (yee-haw!), fairly close to NASA (Houston, we have a problem), but computer games and graphics programming have always been a passion hobby of mine. I have played the piano since I was 5, as well as drumset/percussion since high school, and currently make a living playing and teaching private lessons on both instruments. But always in the background there was the love for anything computer related! To show my age even further, my first gaming console as a kid was the Atari 2600, my school had a couple of Apple IIe’s, and my first pc in junior high was the Commodore 64! But having said that, the Commodore 64 in my opinion is one of the greatest systems ever to be produced. It solidified my love for computers and helped me start my journey into tinkering with Basic code and eventually coding my own stuff.

Although my focus and path in life was primarily centered around Music, there were moments along the way that really had an effect on me in regards to computers and rendering. I remember seeing the 1982 movie TRON in theaters, and although I didn’t understand what I was seeing as a 9 year old kid at that time during the movie’s famous LightCycle sequence, I knew I wanted to do that someday, somehow, with a computer. Also our local public TV station would occasionally play classic computer animation clips from the late 1980’s and early 1990’s and I would be fascinated with how smoothly they ran and looked. Check out the channel VintageCG on YouTube for an idea on what those were like. Turns out it was just pure ray tracing for both the TRON visuals and these 80’s/90’s clips visuals that took several minutes if not hours to painstakingly render each animation frame - but now with RTX cards and such, one can do it real time on a home pc! :smiley:

You are right when you say it takes a long time to accumulate knowledge in the area of rendering. And the sad part (or fun part, depending on how you look at it) is that I don’t know everything there is to know, nor will I ever! Ha. The subject of rendering on its own is vast and can take a lifetime (i.e., ray tracing legends Eric Haines and Peter Shirley, both now working as senior engineers for NVIDIA RTX). After my years playing around with the Commodore 64, I eventually got an IBM clone pc and when these new fangled things came out called ‘graphics cards’ (ha) like the VodooFX cards, I bought books on OpenGL 1, introductory C programming, and basic Windows programming, and tried my hand at rudimentary 3d graphics and very simple games like 3d pong in OpenGL under Windows 98. If you would like to see old screenshots, inside my Github bio you’ll find a link to my old website where I shared my basic 3d games from the early 2000’s - it’s a now defunct website but it is archived thankfully by the internet ‘way-back machine’. This 3d OpenGL undertaking helped solidify some of the basic math concepts for me like vectors, 3d geometry, cameras, and matrices and such, that ultimately helped me recently ease into 3d path tracing, which uses matrices, geometry, and 3d vectors (rays) all over the place. So it has been a long and winding road of learning bit by bit for me! :slight_smile:

Well once again I have rambled on, but I just wanted to share some of my background with you all. Thankfully, I am in a unique position with my line of work in which, although the hours are weird (my work day starts at 5 p.m.) and I work every weekend and most holidays, I have enough spare time during the mornings and early afternoons to spend on my passion hobby, computer graphics. Also, I have greatly benefited from the publicly available code of others, which is why I love sharing my code and ideas freely with all who are interested.

Take care!
-Erich

7 Likes

Hi,
I haven’t implemented any of this, but like to keep up with things. I guess the rng is determining where the samples come from ?

Have you thought about trying blue noise dithered sampling, it seems like it gets pretty good results.

2 Likes

Hello!

Thank you for the heads-up about that more recent blog post! I have read some of his older posts but I wasn’t aware of this one. I just read it and am following some of the links he put at the bottom. Great resources!

About generating random samples for path tracing and RNGs, yes this is hot topic and still an active area of research, even after many years of the CS community having developed various efficient, quality RNGs.

The needs of the path tracer (or stochastic (fancy word for randomized) ray tracing) are more demanding than other general CS fields because with any average viewer’s eyes, they can see right away if the randomized pattern has clumping, gaps, or repeats in the image. This is inline with game developer’s needs (that there be no visible repeating patterns or uneven clumping, such as when populating vegetation in a landscape level) , but that is where the similarities with game devs stop. The path tracer goes much further because the rendering equation in its true form is hopelessly multidimensional and infinitely recursive. Our only hope is to use randomized Monte Carlo sampling and try to reconstruct the scene with limited information. With every bounce of a pixel’s initial ray cast out into the scene, we must have a very good non-repeating quality RNG available and it must be blazing fast (especially if we’re trying to do realtime path tracing at 60 FPS). Most importantly though, it must not be correleated with the screen pixel’s location, otherwise you’ll get ‘clumping’ and ‘gaps’ and ‘patterns’ in shadows and G.I. when it is time to pick sample directions for the secondary rays.

You’re right on the money when you suggest using blue noise to sample randomly, as blue noise combines the best of both worlds - it is regularly spaced but not ‘too’ regular (as opposed to stratified sampling), and at the same time it looks random enough to produce a little light noise without patterns, but it is not ‘too’ random (as opposed to white noise which, although gives pleasing results to the eye from afar, can have clumping artifacts and gaps in the information upon closer viewing). However, the method he is using in his blog post (the ‘Mitchell’ algo) is too slow to calculate/generate each animation frame in a shader for GPU rendering. But for C++ CPU rendering and other image manipulation applications, which is most likely his use case, it would work perfectly.

Actually I have dabbled a little with this idea of generating blue noise-type sample points on the fly for each twist and turn in a ray’s lifetime when it interacts with the scene. If you take a look here you’ll see my attempt at generating a blue noise-type spiral pattern, similar to phyllotaxis spirals found in various plants in nature. Mathematically, the function uses Fibonacci numbers and the golden angle to space out the sample points. The only issue with path tracing is that at each juncture or bounce, only one sample point can be taken, so when you go over that spot again (progressive rendering), you have to cycle through the spiral and make sure you eventually hit all the points and then wrap around again when all points have been exhausted. You can see my attempt in my source code.

In the end I abandoned this approach as it took a little while longer each frame to get the sample back, and the visual quality difference between this and a good white noise RNG (like the one I currently borrowed from the famous Inigo Quilez ‘iq’ of ShaderToy), was minimal.

However I may revisit this idea using a recent approach created by the clever team behind Quake II RTX (which does real time path tracing). Here’s a project link - take a look at the pdf paper. Instead of using Fibonnaci spirals like I did, they just have a bunch of small pre-generated blue noise tileable textures lying around in GPU memory that they quickly cycle through and sample from. The only thing holding me back at this point is how they actually extract the needed sample from the texture array - it has to be pretty large, maybe 32 or 64 small blue noise textures. I’m a little out of my element when dealing with 3d textures or texture arrays. But hopefully I’ll gain some traction soon with this idea of theirs, because I agree with your suggestion wholeheartedly and think it would benefit every aspect of my sampling.

Thanks again for the link! :slight_smile:
-Erich

3 Likes

https://playground.babylonjs.com/#JY1CH7#38

I am struggling to get the hemispheric samples from the bounce to work correctly for my monti carlo setup.

Its based off of GPU Path Tracing in Unity – Part 2 – Three Eyed Games

Which I was able to get the standard pathTracer working from part one with the addPass and all that good stuff, but for some reason like I said can not get the last bits of this to work correct. Maybe someone else knows whats going on?

2 Likes

https://playground.babylonjs.com/#XPFJ55#63

Welp its prolly the worlds worst path tracer… but here we go… I tried to have multiple traces blend each step so that the noise would clean up quicker, but Im not sure how much effect that is really having.

I think to get rid of the noise more you would have to do a bunch of path traces each frame and then add them as if they are incremental frames like it does now. Just not sure if that would actually work though as I cant really see a difference between doing the 4 passes and then doing a combine of those which then gets mixed incrementally. As opposed to just doing one pass added incrementally.

Also kinda seems like I have a trash noise distribution.

Update looks like at about 8 sub steps you start noticing a difference in the amount of noise while moving.


This is what it looks like while moving and the scene is downgraded to 0.5, I could prolly get away with 0.25 as well. But the concept will be to do a standard scene with primitives that will control the positions of the raymarched elements, then I was thinking do a single pass to identify all the elements on the screen, then like a steps on a pass to get the diffuse and reflections, third then do the “multitap” pass but only worry about grabbing all the emissive data and the identifiers of the objects again. Last pass all this info to a postProccess that will blur the emissive data to a certain amount, and then add that back on top of the diffuse and reflections depending on matching id buffers… but that is all kinda a pipedream while I dont have time right now.

8 Likes

Hello @Pryme8 !
Looking good so far! You mentioned 2 posts ago that something was not working with the hemisphere sampling and in 1 post ago you suspected the noise distribution was not optimal. You were correct and that is the culprit!

Random number generators (rng’s) are an art unto themselves and although I rely on iq’s brilliant and fast rng’s, I have no idea how they do their magic. However what I do understand is that anytime you see weird patterns or artifacts, it usually can be traced back to either something wrong with your surface normals, or you have correlated (which is non-desirable) random samples. In your case I believe it is the latter.

If you look at iq’s original rng ShaderToy example that you provided a code attribution to in your shader code https://www.shadertoy.com/view/4tXyWN, you’ll notice that he seeds the rng in a special way. This ‘seeding’ is just as important and also unfortunately just as finnicky as the rng itself to get right when doing anything Monte Carlo on the GPU. In line 23 he initializes the seed ‘p’ to the pixel location based on a 1920x1080 resolution, but you could just as well cast Babylon’s uint(resolution.x) and uint(resolution.y) values instead. But notice at the end of this line, he constantly alters the seed ‘p’ every animation frame by multiplying by the ‘frame’ number. Not sure if Babylon has a built in ‘frame’ counter, but you could use your currentSample uniform as well - it is the same type of variable.

When all this is in place, the first frame will look great with uniform noise (no repeating patterns or zig-zags or lines) - it should look like old fashioned analog TV static. But we are still not done - the sampleHemisphere functions continually call hash() many times in one frame, so having one stationary seed per GPU frame (no matter how good the seed or rng is) is not good enough. The result will be the same distracting patterns on all diffuse surfaces.

The remedy is making a global (outside any other function, usually near the top) seed variable like so:
uvec2 seed;
Then inside iq’s rng hash() function, you must add something like this on the first line:
seed += uvec2(1,2);
So:
float hash( ) // removed argument as it is now a global
{
seed += uvec2(1,2);
uvec2 q = 1103515245U * ( (seed>>1U) ^ (seed.yx ) );
uint n = 1103515245U * ( (q.x ) ^ (q.y>>3U) );
return float(n) * (1.0/float(0xffffffffU));
}
Finally in the path tracing shader’s main() function, you initialize the seed once every animation frame like so:
seed = uvec2(gl_FragCoord) + uint(resolution.x) * uint(resolution.y) * uint(currentSample); /* or better than ‘currentSample’, something like ‘frame’.*/

All these techniques slightly change the global seed on a hash function-call-to-function-call basis (which might happen dozens of times in one GPU animation frame), rather than just once every frame. Only then will you get smooth noise and faster Monte Carlo convergence (because all possible sample points are hit sooner and more efficiently).

Hope this helps your Monte Carlo code to operate smoother! Let me know if you have any other issues. Good Luck!
-Erich

1 Like

Hello Everyone! So sorry for my exceptionally long pause in posting to this topic. If you have kept up with my three.js path tracing renderer, you will see I have been very busy all these months! I didn’t realize that the Babylon PathTracing conversion project had come to a halt. Therefore recently, I have worked on and created a new dedicated GitHub repository for just this purpose! You can check it out at:
Babylon.js-PathTracing-Renderer

If you want to follow our progress, stay tuned with the Main Thread about path tracing inside Babylon.js.

As I mentioned in that forum thread, although I am experienced with using three.js, I am a beginner to Babylon.js. I have overcome some learning/conversion speedbumps when I started this recent Babylon pathtracing project, but there are more hurdles to overcome! If you take a look at my super-simple setup .js file on my new repo, maybe you can spot some places that need an experienced Babylon user’s touch, or a recommendation if you have had to solve some of the same issues (like for instance, I have to implement a ping-pong postProcess and I don’t know yet how to do it!).

Please feel free to contribute info/code so that this project can move ahead to completion!

Cheers,
-Erich

4 Likes

That looks fantastic, well done. I can’t believe how reasonable the performance is, to be honest.

1 Like

Hi Erich. Is there any progress related to the implementation of the path tracing with Babylonjs?

Hello @mathus1

Sorry I haven’t been as active over here at Babylon.js as I would have liked. I have indeed made recent progress on the visual quality of the three.js version of my path tracing renderer. And progress there means progress here as well, as soon as I reflect the changes and improvements to the custom shaders in the Babylon path tracing renderer.

That being said, you may have read elsewhere on this forum that the glTF loading and geometry/material handling is far from complete. I can only load and correctly render a single glTF model at a time, and that model can only have 1 over-arching texture for the whole model (although we do support seperate diffuse, normal, metallic and roughness maps from the PBR-style workflow, but they must all cover the entire model using the same UV coordinate unwrapping, like the Damaged space Helmet model example).

Once I pull in the new path tracing changes here to Babylon, I’ll be pretty much satisfied with the visual quality and rendering speed. But on both the three.js and Babylon.js versions, the glTF loading and scene handling has a long way to go before it becomes useful to a more general user base.

I am hoping that someone can come along and help me implement the glTF geometry/material loading and handling, because although I am very comfortable with the rendering graphical side of things, model and data parsing is one of my coding weaknesses. Im trying to understand it better and improve on that front, but it’s more slow-going than the ray tracing area is for me.

-Erich

3 Likes

So you want to implement the GLTF loader into your code?. I guess I have done that in my app
Check it out at: http://3dhouseplanner.com/
you can drag and drop the 3D models to the scene to import it.
I can give you the code snippet related to that part

That’s a cool app! Thanks for the loading code sharing offer, but my difficulties lie not so much in the loading, but how to take all of that loaded data and turn every triangle and every material (and possible texture) associated with each triangle into a representation that is ray tracing friendly. Leaving out materials and just focusing on geometry for instance, all of the triangles of all of the different possible components (each piece of furniture, each wall of the room, etc) need to be fused together into a single monolithic model. This is for acceleration structure building purposes when ray casting against all of those triangles. If this is not done, then an approach like NVIDIA’S is required, where each component (piece of furniture for example) gets its own personal acceleration structure for its own set of triangles. But then, an over-arching top level acceleration structure over all of these mini acceleration structures must be constructed.

When adding in materials, if the 1st option is chosen, we have to be very careful in retaining some sort of lookup id for each and every triangle, and then load all of the textures onto the GPU where they be can looked up on a per-triangle texture unit ID basis. I have the beginnings of this in the glTF model viewer demo over at my Three.js-PathTracing-Renderer repo, but it is too fragile and non-robust for general use.

If the 2nd option is chosen, the texturing id problem might be a little easier (unless a piece of furniture has more than one material texture!), but the problem arises of building and updating a double acceleration structure (TLAS and BLAS, Top-level acc structure and Bottom-level acceleration structure). I can currently do 1 bottom level fairly well for any kind of single model with single overarching texture applied to the model as a whole. But i have yet to attempt gathering all of these up into a top level structure that has pointers to their materials and such. It’s challenging no matter what path you choose, ha.

As mentioned, I’ve made some progress in both areas, but im still learning how to integrate all of the seperate pieces of knowledge into a solution that handles arbitrary scenes and glTF models.

-Erich