Path-tracing in BabylonJS

Questions
249

Major breakthrough!

For years I was unable to run my BVH path tracers on mobile - they wouldn’t even compile, or would just crash after a few seconds, even on my Samsung Galaxy S21.

A recent Twitter post by Garrett Johnson caught my eye because he was having similar problems when developing his BVH system. We got into a discussion about it, chalked it up to lack of precision, and I was ready to give up. Not Garrett however, he is a robotics expert at NASA and he knows all about low level stuff and machine precision. He couldn’t let this go, because mobile devices should have as much floating point precision as desktops, as reported. Yet there was still a discrepancy.

Many posts later and a multiple-expert discussion later (Ken Russell, Romain Guy from Google, someone who knows mobile chips, etc…) it was discovered that for some reason (which I can’t quite fathom), the designers of Adreno mobile chips treat all members of structs{} as mediump or lowp precision, even if I specifically put highp precision at the beginning of my shaders! I have structs all over the place, stuff like Ray{vec3 origin; vec3 direction;} Intersection{ vec3 hitPos, etc.}, BVHNode{vec4 data, etc.}.

So the problem was that when traversing the very precise BVH, my mobile devices were unable to even start because floating point round-off errors were flying all over the place! All this was happening unbeknownst to me because I thought everything was in highp precision, like I had instructed on every shader - not so!

As soon as I got rid of all my structs, and unpacked them into seperate variables, like vec3 rayOrigin instead of ray.origin, everything magically started to work! It was such a joy to be able to see the Stanford Bunny (30,000+ triangles) path traced at 60 fps,… on my phone!

So I just wanted to pass this gold nugget on to you all for all your future GPU projects that might have to be run on mobile. Use individual vec2, vec3, vec4 variables, instead of grouping them together into structs in your shaders. It will be an inconvenience for me (and you possibly) to go back and change all my code now, but the result is well worth it!

2 Likes
250

Also, none of this would have been possible (or I wouldn’t have seen it for months), if I hadn’t joined Twitter - so a big thank you to @jelster for inspiring me to join! Thank you Josh!

We will all reap the benefits of this discovery. I am going to start retroactively changing my code snippets that have structs in them, and hopefully soon, three.js and Babylon.js will have high quality, real time path tracing of scenes with hundreds of thousands of triangles,…on mobile devices! :smiley:

4 Likes
251

w000000000000t do you have a pointer to the adreno thread ??? I guess it might explain some of the issues we had for years on those :slight_smile: cc @Evgeni_Popov and @Deltakosh

4 Likes
252

Sure thing! Twitter’s fragmented thread system is a little hard to nail down for this topic, because he was talking with different people, but here is what I can gather in one page:
https://twitter.com/garrettkjohnson?t=TG79hg2jbYbqbHTDJaFjOw&s=09

If you scroll through his recent posts, he talks all about it. The one with the mobile shader precision profiling app (black screen with gray text and gradient bars) was a real eye-opener for me!

I wonder if iPhones and iPads have this problem? Or is it just devices that use Adreno chips?

2 Likes
253

This mobile bug had been haunting me for several years - and then in a span of a couple of days of Garret’s excellent detective work, and 30 minutes of refactoring on my part, it has been fixed! Ha ha, I guess that’s the way programming goes sometimes :stuck_out_tongue_winking_eye:

P.S. to all iPhone and iPad users; were you always able to see the BVH demos (both on my three.js renderer and our Babylon.js renderer), or did they crash for you too? If they didn’t work in the past, hopefully this discovery will fix things for you all as well!

3 Likes
254

The issues with iOS not being able to see the demo were mostly due to safari/iOS not supporting webgl2, since then it’s been working pretty well AFAICT!

Thanks for the shout out too - always happy to help amplify stuff and people doing great things! :sunglasses:

3 Likes
255

Oh and by the way, I haven’t forgotten about the glTF texture loading issue on the bookcase model provided by @pcarret . Even though I’m taking a small moment to celebrate squashing a nasty multi-years-long mobile GPU bug (that wasn’t even my fault!), this texture loading issue is still at the top of my priorities list!

2 Likes
256

Hey Everyone!

I’ve been hard at work - going back and changing every precision critical structure inside all of the fragment shaders to instead be seperate variables with similar names. For instance, struct Ray {vec3 origin; vec3 direction} (ray.origin and ray.direction) have to become vec3 rayOrigin and vec3 rayDirection, respectively. Also, intersection records (intersection.normal, intersection.uv, intersection.rayT), had to become seperate variables as well. Then last but not least, the BVH’s struct BoxNode{vec4 data0, vec4 data1} had to be separated out into vec4 BoxNodeData0, BoxNodeData1, respectively.

I never knew how many darn structures I had in my fragment shaders, lol! I went through the entire three.js renderer with its 34 demos, and changed every single occurrence. Once that was done, I went through our entire Babylon.js renderer as well, (thankfully it only has 6 demos, ha!). It was a lot of dull, mind numbing syntactic changes to the whole codebase, but the payoff is well worth it!

Screenshot_20211111-154312_Edge
Screenshot_20211111-154312_Edge2400×1080 172 KB

I can finally see the Bunny :rabbit: on my Samsung phone, yay!

This ‘structure-into-seperate-variables’ repo-wide change has opened up glTF model path tracing to the Android mobile space as well. And now that Apple iOS recently added WebGL2 support for iPhones and iPads, we should have nearly complete support for any device with a browser, no matter what operating system the end user is on! :smiley:

14 Likes
257

Hey buddy! Just checking how you’re doing :smiley:

2 Likes
258

@Deltakosh
Hello David! Thanks for checking in - I’m doing ok!

Sorry it’s been quiet around here on this thread lately, but I actually have been updating the Babylon.js PathTracing Renderer over the last few weeks and months.

I mainly use my three.js version for testing new ideas, but as soon as I make a new discovery, or improve on an existing technique, I go right to the Babylon.js version and make the updates there as well. :slight_smile:

Also, I’ve been known to go down rabbit holes from time to time, ha ha! This particular rabbit hole I’m currently down is all about efficiently intersecting rays with quadrics of various types. Normally we see quadrics as the usual spheres, cylinders, cones, paraboloids, etc., but recently I came across this really nifty older research paper from 2004: Link, has downloadable PDF of the Paper

It describes an efficient method to render any quadric shape imaginable on the GPU. They pack the coefficients into a 4x4 matrix, which fits nicely into GPU memory as a mat4().

I’ve always been fascinated by ray tracing different mathematical shapes and non-triangular geometry - and quadrics are no exception!

Anyway, that’s what I’ve been up to recently. I hope you are doing well also, David! BTW, I’ve been enjoying your lovely artwork posts on Twitter! :smiley:

3 Likes
259

I’d like to incorporate this path-tracing into my app. Is it already possible to use babylonjs to render and denoise a snapshot or video in the same way that blender does? Is there an example of this in a playground? If not, when can we expect this feature to be implemented in babylonjs?

260

Hi @erichlof, really nice to have some news from you. I wish you to continue having fun playing with these raytracing methods ! Again I don’t understand much but it seems really fascinating.
Did you see on Twitter where I tagged you what the team of brioXR made using path-tracing rendering? It would be really interesting to match and compare your methodologies and algorithms for tackling this huge challenge.

@mathus1 I think we are several to wish the same thing haha. For now, Eric is developing this new path tracing engine with the help of the BabylonJS team on Github
You can try to implement it I guess looking at the code. Just note that it doesn’t use Typescript (yet) and that this is still a work in progress.

261

Why that if I may? (feel free to ignore me ;))

2 Likes
262

You’re going to make me blush :smiley:

1 Like
263

@Deltakosh
It’s not that one is faster iteration over the other - both have really quick iteration times for making changes to the code and then having it refresh the browser window. It’s just that I have more code memorized from the three.js path tracer version (since it has been in development for 7+ years now), so if a new experiment doesn’t work or is giving errors, I can usually quickly eliminate possible suspects in the codebase. Then I’m able to locate and fix the problem within a shorter timeframe - in order to get back to the new experimenting.

Having said that, working on the BJS version is directly influencing how I approach the three.js codebase now, especially when refactoring or adding new features. This ‘second’ time around of making a path tracer for BJS has toughened-up my algos and helped to solidify my general approach to rendering. :slight_smile:

1 Like
264

Thanks for the answer! That makes total sense :smiley:

1 Like
265

Hey guys!
Just wanted to give you all a sneak peek at my previously-mentioned experiment (rabbit hole, lol) of rendering any arbitrary 3D quadric shape on the GPU. I put the files up on the three.js path tracer repo in order to share it with you all here, but it isn’t currently accessible from the main three.js GitHub repo landing page, like my other more-established demos.

Real-time Path Tracing arbitrary Quadrics Demo

This demo implements the technique outlined in the previously mentioned paper from 2004, Ray Tracing Arbitrary Objects on the GPU, by Wood, et al.
I’m surprised I hadn’t seen this paper before a couple of weeks ago. It is a short, little gem of a paper that uses a novel technique of storing and doing multiplications of all the possible Quadric coefficients (A-P) inside a neat 4x4 matrix (mat4 in GLSL), which plays nice with the GPU. I had some help implementing this paper in GLSL, when I came across this ShaderToy shader.

My W.I.P. version allows you to fine-tune all 16 coefficients with sliders, as well as demonstrating some classic presets (like hyperboloid). I’m currently working on adding a transform matrix for the shape as a whole, so users will be able to rotate, translate, and uniformly/non-uniformly scale the shape in real time, after they have dialed in the desired slider settings.

You may notice that although there may seem like an infinity of possible configurations (A-P), only certain configurations/settings of the A-P coefficients give a truly solid shape. I’m not sure why yet, but when I mess around with some particular slider coefficient combos, the resulting quadric shape develops thin cracks and even large semi-transparent portions which let you incorrectly see what’s behind the shape. Only the solid surfaces are useful/interesting to me, but there are so many to explore with 16 possible coefficients, it should keep me busy for a while! :smiley:

Enjoy!
-Erich

9 Likes