Hello again!
Here’s part 2 of the posts about finding a 3d sun direction vector in the scene when the background and sun came from a 2d HDR image. Wouldn’t it be nice if we didn’t have to open up every single HDR image in PhotoShop/GIMP to hunt around for the Sun disk’s brightest pixels with our mouse? Well, the following second method (which actually just replaces step 1 in the previous post) is more robust in that you should be able to load any outdoor HDR image with the Sun visible (or even partly visible behind light cloud cover) and it will automatically return the brightest pixel coordinates that correspond with the Sun’s disk center. This ability comes at the expense of being slightly more complicated, but hopefully I can walk everyone through the process so that the steps will be more clear.
By the way, since this entire post takes the place of step 1 in the preceding post (the step where you have to go inside PhotoShop/GIMP, etc), means that when you’re done with this part, you can just continue from step 2 through 5 in the above method and you’ll have everything automated.
So from a global overview, this is what we’d like to do: First, load in an arbitrary outdoor HDR with arbitrary resolution, then find the Sun natural light source in the image (which could have been placed anywhere, from the original photographer’s camera setup), then return this brightest (or one of the brightest if they’re in a group) pixel’s x and y coordinates. Then as mentioned, we continue on from step 2 in the proceeding post and we’re good to go.
In WebGL when we’re reading or storing image pixel data, it most often has to be to and from a JavaScript flat array, such as:
const data_rgba = new Uint8Array( 4 * imgWidth * imgHeight );
We replace the rgba above with rgbe and that’s pretty much what we’re dealing with. Notice that it’s the (image dimensions * 4) because we have to take each pixel and spread its 4 channels among 4 unique array elements. So what we like to think of as pixel0.rgbe, pixel1.rgbe, pixel2.rgbe (which would be initially pixel[0,1,2] in the image), becomes a spread out flat list of numbers:
p0.r, p0.g, p0.b, p0.e, p1.r, p1.g, p1.b, p1.e, p2.r, p2.g, p2.b, p2.e, and so forth, which is represented in typed array form as pixel_data[0,1,2,3, 4,5,6,7, 8,9,10,11, etc.]. Note that in the original array we had 3 elements for 3 pixels in the original HDR image, and in the 2nd Javascript flat array, we have 12 elements which is 3 pixels * 4 channels. This will come back around soon.
As mentioned in the previous post, the RGB data doesn’t really help us find the Sun because there may be snow or white clouds everywhere in the scene and so most pixels would be 255,255,255 including the Sun’s pixels. The key is every 4th array element: the E channel, or exponent. If we can somehow iterate over every pixel’s E channel and keep a running record for the highest exponent we have found so far, by the end we will have successfully located the brightest pixel in the arbitrary image. And since the Sun is the brightest thing by far in an outdoor HDR image, the highest exponent will max out near or at the very center of the Sun disk.
So if the data is laid out like p0[0],p0[1],p0[2],p0[3],etc. for each pixel’s p0[r],p0[g],p0[b],p0[e], channels etc., then we’re only concerned with index [3], which is the pixel’s E channel. Then we just add 4 to this running index on each loop iteration as we scan the whole HDR image. Here’s some code from my three.js version:
hdrTexture = hdrLoader.load( hdrPath, function ( texture, textureData )
{
texture.encoding = THREE.LinearEncoding;
texture.minFilter = THREE.LinearFilter;
texture.magFilter = THREE.LinearFilter;
texture.generateMipmaps = false;
texture.flipY = false;
hdrImgWidth = texture.image.width;
hdrImgHeight = texture.image.height;
hdrImgData = texture.image.data;
// initialize record variables
highestExponent = -Infinity;
highestIndex = 0;
for (let i = 3; i < hdrImgData.length; i += 4) // every 4th element, starting at [3]
{
if (hdrImgData[i] >= highestExponent)
{
// record the new winner
highestExponent = hdrImgData[i];
highestIndex = i;
}
}
console.log("highestIndex: " + highestIndex); // for debug
}
Above we start at pixel data index [3] which is the beginning pixel’s E channel, and then just add 4 to this index on every loop, searching each and every pixel for the highest exponent as we go. Once the last pixel is reached, by the end we should have a flat array index that corresponds to the winning brightest pixel. Some of you may have noticed a shortcoming in my algo; this will indeed find the brightest pixel, but the winning brightest pixel stops being recorded at the edge of the disk when the pixels return back to normal sky with less brightness as we scan from left to right. To find the true center (and maybe I will add this, or someone else can take a shot at it for me!) you have to remember when you reached the first winning bright pixel, the last winning bright pixel, then take the average of those 2 values to get the exact center of the winning group of brightest pixels. This is straightforward in 1D as we scan from left to right, but what complicates matters is that we have to do this going up and down and find that average as well. Therefore I simply left my above loop in place and it seems to do ok with the HDRs that I have tested so far. Where it would fail is if we somehow had an alien sky HDR from another planet view, and the HDR had 2 or more Suns. This algo would only find and sample the brightest light source when path tracing.
So from here on, it’s a matter of getting this desired flat array brightest index number into something more meaningful and useful, namely an HDR image x,y coordinate corresponding to the exact Sun disk pixel. Once more, here’s the high overview plan:
- loop through every 4th element in the large flat array of hdr rgbe pixel data, so [3] and onwards, adding 4 to each loop index - [3],[7],[11],[15],etc. Find and record winning highest exponent, E channel.
- take this winning exponent flat array index number and subtract 3 from it to get back to the r channel, or [0]'th place of each pixel - that way we are back on the boundary of every 4 numbers of each pixel. Our result will now be guaranteed to be divisible by 4 (because of how it was spread and laid out for WebGL in the first place, initially multiplying by 4)
- Divide this index number by 4 - now we’ll be back in range of the original 2048x1024(or whatever) resolution. This will still be large number because it is still 1d flat, but at least it is a little smaller than it was before! Its range will be anywhere from: 0 to (width*height)
- Now to make this correct-range flat number into something useful, we have to map this large number back into the 2D space of the HDR image (width location, height location), or (x,y) which needs to be in the range: (0 to width-1, 0 to height-1) . Note: the ‘-1’ reflects the fact that our data is always 0-based. Through trial and error and some experimenting (ha) I came up with the following formulas for getting the final x and y coordinates:
X = highestIndex modulo with imgWidth
Y = highestIndex divided by imgWidth, then Floored to get the integer portion
At the end of all this, we will have the same thing we had in step 1 of the previous post: an exact X,Y pixel location corresponding to the brightest part of the HDR image.
Here’s some code that follows the above outlined steps
1. /* As printed above: the 'for (let i = 3...)' loop which results in 'highestIndex' */
2. highestIndex -= 3; // get back to the 0 boundary of every possible pixel
3. highestIndex /= 4; // guaranteed to be evenly divisible by 4, by design
4. brightestPixelX = highestIndex % hdrImgWidth; // range: 0 to imgWidth-1
brightestPixelY = Math.floor(highestIndex / hdrImgWidth); // range: 0 to imgHeight-1, essentially this is
//how many times that the large number has exceeded/wrapped the imgWidth x dimension.
We have now automatically completed step 1 in the first post, avoiding opening up an image manipulation program and hunting around the image with a mouse to try and find the brightest pixels. Now we can just continue on as normal with steps 2-5 from my earlier post and we should have a quick and robust way to find the Sun in 3D from a 2D HDR image!
If you look at all the math, it’s not that involved, especially for the CPU that won’t even hardly blink during this whole affair. But just to get the algo steps right in my mind, man it wasn’t pretty! Lol. At one point I was scribbling numbers from 0-100 in a square 2d matrix on a little scrap of paper like a madman, to see how a 1d list of numbers gets mapped to a 2d square array with a width boundary and height dimensions (ha). But I eventually understood why step 4 has to do what it does.
Now a caveat with all this is that this method will only work for a single, dominant light source, like the Sun outdoors. To scan an indoor HDR image and return multiple bright areas coming from human-made artificial lights will require a more sophisticated approach. So that’s still a big TODO. But hopefully this method will at least let us efficiently render and sample all outdoor scenes with Sun visible, and even if you’re not doing path tracing, I hope that this technique will assist you in locating the light source in your general 3D Babylon scene if you are using an HDR image.
If there’s any confusion about these HDR posts (there were a lot of words back there, ha ha!), please feel free to post here on this thread. I will do my best to clarify anything that was not clear, either algo-wise, code-wise, or both.
Best of luck and happy HDR rendering!
-Erich