Post-mortem analysis of a Babylon project

I was waiting to post this when my code was already done, but with this great Golden Paths for Babylon.js! that was posted today, I decided to post this a little early since it might become one of the golden paths.

So I wrote a SCUBA dive simulator using Babylon. The simulator is ALMOST finished, but there’s still some work to add fish schools and fix a couple of issues. I’ll post it to the projects category once it’s done. It’s a pet project that wants to bring visibility to a beautiful and historically relevant marine park, with this website, that deserves to be better known and visited. Read about it there. Babylon is used all through the website. Pretty much every page has a BJS component, from the landing page ocean effect to 3D models of fish.

There’s a whole post in that site about the code and how everything was implemented for those curious about it, which includes a lot of Babylon stuff – and that can perhaps be part of one the golden paths mentioned here.

But I wanted to make a post-mortem post specific about the experience of using Babylon for that project. What worked and what didn’t. This post is basically a list of “IMHO” (in my humble opinion, or not so humble), by the way: it’s a personal tale and comparison with my experience of building 3D stuff over many years.

First impressions

This project started a while ago, and the pandemic paused it – which is why I made a few posts and contributions last year, then disappeared for a while.

First of all, let me start with the reason I picked BabylonJS over ThreeJS: animations didn’t work with ThreeJS when I was doing a post-processing multi-pass rendering for caustics. Perhaps this bug was fixed after all this time, but essentially the second pass with a different material wouldn’t apply bones. But back then, after writing some ThreeJS code for the project and thinking this bug was not trivial to fix (details are fuzzy; I think I posted an issue at the time in their GH) I decided to give BabylonJS a test and things worked well. I had a previous experience with BabylonJS, but loooong ago.

Well, it turned out that multi-pass rendering was not so easy to do in BJS too, but I got a lot of help in the forum and any issues I had were immediately fixed. This is one thing I want to say, and in bold: the community here is awesome. I’ve contributed to several OS projects over the years and this is by far the most welcoming one. Bugs are immediately taken care, maintainers apparently never sleep and are always answering questions here in the forum, and any code contributions are welcome and carefully reviewed with all the help you need to fix the problems and a bit more. Thank you, particularly to @Evgeni_Popov (who not only is my Personal Babylon Support Person, but also constant amazing help with my PRs) but also to @sebavan and @deltakosh. You all rock. Thanks for the patience and help.

For someone who used a ton of 3D engines over the years, Babylon is a breeze. The documentation is excellent and the playground makes things easy to test for a new comer. Plenty of practical examples, and the forum works very well as a dynamic form of documentation. Technically, Babylon also does something that I really value in frameworks: it mostly stays out of your way, and if you need to get over it to do something unexpected or low level you can almost always find a way to do it. I didn’t need to get down to WebGL code, but I got around almost everything I needed – with one exception that I’ll mention below.

The first shock I had with Babylon was, however, its default performance. The same scene from ThreeJS, naively converted to BabylonJS, was much slower. I know, docs go over basic optimizations pretty well, but it ends in a lot of freeze calls all over the place. I missed having a “by default everything is frozen” setting. I understand the choice, but it certainly results in people thinking “how slow” and never reading the docs to understand why. There are lots of internal tools to get around the main performance issues (thin instances, SPS, etc) that were very helpful, but it takes some diving to find them (like realizing, oh, there are thin instances and not only instances). Somehow the optimizer never did a significant job for me, so sprinkling the code with freezes and merges was part of the job. But what I really miss is a more high-level API, something like mesh.completelyStatic(), which would freeze everything for me: mesh, material, normals, matrices.

Once I got around I was back to the usual “JS has a crappy multithread model and this is the root of all slowness” problem that I was used to. I’m glad that WebGPU will apparently help this and it’s finally coming into production in browsers soon. But I was always bumping into “can’t do that on the CPU because it’ll be too slow”, and adding code that suddenly dropped my frame rate all through the project, always wondering if it was a missed freeze call or a more serious problem. This meant finding some creative ways to handle things and constantly doing checks (and cursing Firefox, which slows down after several page reloads, leaking who knows what). This is very boring: finding out why there are too many draw calls and what could have been frozen and wasn’t. The new performance profiler seems like a giant step to improve this, particularly if it can help pinpoint the bottlenecks at a higher level so it’s easy to fix them. I’m looking forward to it, because profiling and optimization are normally boring, but BJS doesn’t help much. My go-to way to find bottlenecks is commenting large chunks of code and seeing what slows things down the most. And my phone has much different bottlenecks from my desktop, which makes this painful. A better model to handle platforms is also something I miss: my phone is GPU bound, my desktop CPU bound. As I optimize things I’m starting to consider LOD that is tied to the GPU performance, for example – so annoying to implement.

Still on the performance issues, I was happy to contribute with the Vertex Animation Texture code, though it was only possible from the PoC @raggar orignally wrote and the EXTENSIVE code review from maintainers, particularly @Evgeni_Popov, who was way more relevant than I for that PR. So I can barely take any credit for it. Thank you so much, again!

The limitation I mentioned above was “how to patch shaders”. You have a material that you want to change slightly, but BJS didn’t provide a way to modify its shader code. I ended up doing a multipass RTT (and bothering everyone so much here that I felt obligated to write the doc page about it), but I mentioned this issue in the VAT discussion and it was very well received. It led to the material plugin manager code which is coming soon. I love that PR, but again the credits should go to the maintainers who wrote the core of that change. I’m just happy that my little dive project resulted in some improvements to BJS that I think are relevant, but they only happened because the maintainers are so awesome. I’m pretty sure they just bang their heads on the wall whenever they see another one of my posts or PRs :wink:

Web and native development

This is of course another difference between using a web 3D engine and something like Unreal or Unity: the amount of technology is lagging a bit – and of course, the limitations of size, money, hardware, browser limitations, number of developers and architecture are huge. Comparing these projects is apples to oranges. And it’s not an issue if you are doing something simple instead of a AAA game. The convenience of running on a web page in my opinion trumps the difficulties. But there’s a line where a smaller project could benefit from being web based and things are much harder here.

One thing I’d very much like to have more of are plugins and assets. Finding code for the ocean was hard, for example. The old ocean post-process code was removed from BabylonJS for license issues, but it was pretty much the only realistic code around. Even so, the original shader was only for above the water. I ended up changing it considerably and making it work for underwater too. (BTW that new ocean for WebGPU looks awesome!) Anyway, it’s all part of the job, but it took me a few extra days of work to make the ocean work properly, implement the underwater changes, etc, particularly because the original shader code is quite cryptic. So I was almost writing it from scratch, from fixing small issues and implementing the underwater area, and it still doesn’t work properly if you are very close to the water line, which requires handling the ray marching in a better way than the original implementation – and there’s no reference implementation of a ray marching shader for BJS, for example, which would have been helpful.

So here’s one suggestion: I’d like to recommend to this project creates a standard way to write plugins, perhaps with a template plugin repo, already handling all the nasty JS stuff and making it easy to test with the PG, and somewhere in the main site listing the plugins. Perhaps the same thing for materials: the NME is nice but there’s no catalog of interesting materials.

I can make a good comparison because a friend of mine is building a similar to project in complexity to mine, but he’s using Unity. Besides the asset store helping a lot, another difference is that he relied heavily on the GUI to build his scene. I’m very comfortable with the IDE and have been writing CG code for years now, and I don’t miss the GUI: the inspector is plenty to debug and make minor real time changes. The editor looks great, but I found it after I was already coding and it was not clear how to use my custom code that handled all the performance gotchas with the editor. The inspector was very helpful, but I ended doing some work on Blender. It’s been ages since I last did any 3D modeling, and I never used Blender extensively even then. So any task that involved positioning assets or checking textures involved opening Blender, scratching my head to figure out how the hell I do something trivial etc. I am sure to a lot of people an integrated graphical editor is essential.

Things that were harder than I expected

That “almost” I mentioned above is one important problem in BabylonJS that is getting fixed: standard and PBR materials are not easy to change, because the shaders are fixed. This means that if you need a simple change to the shader, you are in bad luck. I could have added caustics to basic materials, but I had to to a separate pass for them and post process to merge it. This would have let me avoid the caustic multi-pass, and would have made the VAT implementation much easier. I’m so happy that it got picked up and will be released soon.

Another relevant issue was debugging shaders. Man, it’s 2021 and it is still awful to do. I know the whole CPU/GPU/communication blah, and this isn’t a BJS specific problem, but boy do I miss a print() or step debugger sometimes. Painting pixels to debug things is so horrible. I may have missed some interesting tools here, but what I’d like: 1) a print()-like-helper, that perhaps saved data to a texture. I would be happy to select a single pixel to get data out from. I don’t think that’s very hard to do, and my approach would be to allocate a texture and just dumping characters there. I know there’s no string in GLSL, but I’d be perfectly happy to just print the basic GLSL types and get them back in order and printed to the console. A print(vec4) would have been SO USEFUL, and I was too lazy to setup output textures to do it myself. 2) a step-by-step GLSL simulator. glsl-simulator was written and abandoned, which is a shame, because it’s pretty much what I’d like to get.

Performance optimization was also painful, like I wrote before. My dev machine is pretty good, so 60fps in it could very well translate to 15fps on my phone, which is not too shaby either. I wrote code in a way that made it easy to comment parts and quickly see what was slowing the app. It’s painful. Little things were always causing issues (add thin instances on mobile, sudden drop to 30fps, why? GPU bound? Or the CPU animation code? Let’s sit down for yet another profiling session). I am looking forward to that performance profiler that is coming with 5.0.

Still, finding out what is slowing down the GPU is awful. I’m thankful that mobile debugging is such a breeze these days with remote debugging, because otherwise it’d be par with debugging microcontrollers using a single LED. The GPU is still a blackbox. SpectorJS is usually too low level to help with scene optimization, so my go-to technique was really “comment/test/loop”. Also, I was surprised to get the GPU as the bottleneck in this application, something that happened on mobile, which is not my usual development target. I was expecting to perhaps hit a memory limit, but be CPU bound in pretty much any platform with a 3D chip. Is it my doing something silly? Have I pushed too many polygons, or is it textures?

Most of the development time in this project was spent on things that were not trivial to do (like how do I render caustics with a multi pass) and performance optimization. But a lot was also spent in small annoying things. From webpack stuff to animating boids (had to write my own code for that, too). It’s a very simple project in its complexity, and when I began it I didn’t expect to spend so much time optimizing things. I am not blaming BJS for this; JS and WebGL are much to blamed. I wouldn’t have had half of the performance issues if I were developing a native application.

But I want to finish saying that all in all my experience with Babylon was very smooth. I’d definitely use it for future projects. I’m particularly looking forward to WebGPU coming to production browsers (even though the basic performance won’t be better, from what I read) and WebXR. Hopefully I’ll get a chance to work on new projects using them.

That’s it. Thanks to those who read this long novel!

11 Likes

Thanks a lot for this write-up! I’m pretty sure @syntheticmagus will find some ideas useful for his upcoming golden paths

Some comments:

  • PBR and Standard are static shaders (a bit less now thanks to you and the material plugins) but NME is quite the opposite. It is super flexible and clearly a good tool to look at if you want to get more control. NME can repro PBR or standard material for instance.
  • Regarding the make everything frozen, we have the ultimate combo with scene.freezeAllActiveMeshes() which should trigger the best performance in one high level call ;). Furthermore in that mode, WebGPU will kick ass because it is directly optimized for it

Finally one ask for you: Please help us improve the documentation. You have an invaluable experience with Babylon.js and some of your pain point could have been solved by either a golden path or some additional documentations.

4 Likes

I know about it, but I don’t need to freeze everything some times. The usual scene has N fixed objects but M that are not so fixed. That’s why a mesh.isCompletelyStatic() & friends might be helpful. It’s clear to the dev, and I believe can be used by the engine to optimize and merge static meshes into a single call, for example. As a dev it’s not really clear what things I should freeze: should I call freezeNormals()? What if my vertex shader changes the mesh? Perhaps we could have a single mesh.optimize() which gets as a parameter a list of higher level flags. It’s a single point of optimization and it makes it clear which flags we have to optimize. I could try to propose something but I’m afraid to send another PR proposing this that will turn into @Evgeni_Popov spending the rest of his week writing code for me :wink:

Finally one ask for you: Please help us improve the documentation.

Absolutely, I’ll do that.

2 Likes

As you stated, this is not possible. What if the shader or the current user needs some stuff to move or not? We cannot have millions of entry points for that (We already have too much).
There is no magic bullet because every scene is different. Are you slow because too much draw calls? too complex shaders? etc… All of this is not something we can easily track through one single API call (we would have done it else of course ;))

mesh.freeze() for instance will allow the system to skip building the mesh world matrix and list of instances. scene.freezeAllActiveMeshes will freeze all meshes and will not run the frustum clipping anymore. material.freeze() will lock the shader

As you can see there no way we can provide a simpler approach just because the problem is not simple

2 Likes

Oh yes, I’m sorry if I wrote in a way that made it seem like an easy problem. I am very aware it’s not.

What I meant is that as an user, the current freeze etc calls are not very easy to understand. Examples: freezeNormals seems useful when the IDE is autocompleting, but when you read the docs say it’s only useful for parametric meshes. Is doNotSyncBoundingInfo necessary if I called freezeWorldMatrix? Does material.freeze() means that if I change a texture used as uniform it will not update? All these things can be answered by reading the docs/code/testing them, but it makes harder to optimize a scene.

Second, they are scattered. If there was a single call like mesh.optimize(optimizations: OptimizationInterface) it’d much simpler to find all the possible optimizations one can call – even if it’s pure syntactic sugar. And perhaps these options would help BJS to do internal optimizations, because they could provide semantic information about what the user wants.

2 Likes

I do not believe we should entertain the idea that the complex task of optimizing a scene would be possible without reading the doc (unfortunately :frowning: )

I’m interested to see how you would organize that mesh.optimize :slight_smile: So if you feel like doing a PR or even a draft, it could be great to discuss (And I would be happy to be convinced this will help)

1 Like

Not without reading the doc :slight_smile: But making the calls a bit simpler. Here’s some actual code I wrote:

            mesh.doNotSyncBoundingInfo = true;
            if (mesh.material) {
              mesh.material.freeze();
            }
            mesh.alwaysSelectAsActiveMesh = true;
            mesh.cullingStrategy = BABYLON.AbstractMesh.CULLINGSTRATEGY_OPTIMISTIC_INCLUSION;
            mesh.convertToUnIndexedMesh();
            mesh.freezeNormals();
            mesh.freezeWorldMatrix();
            mesh.freeze();
            mesh.bakeCurrentTransformIntoVertices();
            this.sunLight.includedOnlyMeshes.push(mesh);

Do I even need all those calls? Is there any other call that could improve performance? Hard to say.

What I’d like as a user is something like this, very rough draft just to explain the idea:

interface OptimizationInterface {
  /**
   * this is something like freezeWorld + freezeNormals + material.freeze + doNotSyncBoundingInfo
   */
  completelyStatic: boolean;

  /**
   * Set to true if your mesh doesn't translate/scale/rotate
   */
  freezeWorldMatrix: boolean;

  freezeNormals: boolean;

  alwaysActive: boolean;

  material: {
    freeze: boolean;
  },

  animation: {
    /**
     * This pre-bakes the animations into VAT.
     */
    bake: boolean;
  }

};

class Mesh {
   async function optimize (o: OptimizationInterface): Promise<void> {}
}

Why would I like that?

  1. so I have a single optimization point per mesh, like mesh.optimize({completelyStatic: true}); instead of 10 separate calls. Pure sugar, but nice.
  2. so I can see at a single glance all the optimizations I can do instead of browsing the docs and reading all the methods of the API to see if there’s something I could use. I still would need to read, but it’s concentrated now and my IDE autocomplete will help me.
  3. BJS maintainers know much more than I do and perhaps created a new freezeMagicStuff method last week. My code will automatically call it if I use completelyStatic: true, even if I don’t know it exists.
  4. It also possibly signals to BABYLON.SceneOptimizer that it can merge this mesh with others to reduce the draw calls. Perhaps this interface could have semantic information about the desired optimization that is useful to the optimizer and currently not available to it.
2 Likes

FreezeNormals is not needed:)

But I have one big point here: you should not shoot in the dark. You should profile your code and see where the CPU is spent and then optimize accordingly

That being said your point 2 is valid even though some “optimization” like the culling strategy or the alwaysSelect are heavily situational. If you are gpu bound they will kill your perf (likely on mobile). BakecurrentMatrix is not an optimization. It simply does a transfer of your world matrix to the vertices.

This is why I stand by my first point here: you should not shoot in the dark.

I believe that we need to do a better job at explaining all these features (hence why I need your help:)) and at bringing more tools that can help understand your cou/gpu budget

1 Like

Something ai think could be useful would be to describe far more what every function you mentioned does (and clearly state if it is an optimization and when to use it)

Cc @PirateJC

1 Like

I try to always update that doc with everything perf related:

This is your one stop shop and this is maybe where we need to improve (probably by @see it in the code to begin with)

3 Likes

Cc @sebavan annd @RaananW fyi

Interesting post, but I went to those links to your project and they all say ‘site not found’.

‘site not found’

I don’t know what Github did, but fixed.

I try to always update that doc with everything perf related:

Yes, and I keep reading that page – it’s a great resource. But perhaps code could get automatic performance improvements with a higher level API, instead of the developer reading that page to find something new.

While new docs would certainly help (and I’ll try to send some PRs soon), I still maintain that as a user I’d like a call that unified the optimization APIs. I am no strange to profiling (see Animation performance drop by translation/rotation, which you have helped me before, thanks again!. In fact that PR for the material plugins is in part because I’m expecting it to be much more efficient than a separate RTT pass), but the problem is that it’s not always easy, as a user not profoundly familiar with the API, to know how to solve a bottleneck with BJS. And I’m not claiming that it should be easy – optimization is hard by nature, and abstractions always pay a performance price.

And if you are GPU bound, profiling is actually hard. Spector.js is low level, and translating it to a sensible optimization on your high level code is not trivial.

I just feel that the API could be easier to peruse. Some things that optimize are flags, other are methods. I know freezeNormals is useless, but I thought bakeCurrentTransformIntoVertices would save a transformation. And even with much improved documentation (and BJS has pretty decent docs IMHO) it’s still hard to find all the calls by the very nature of code. I maintain that a mesh.optimize call would be very helpful and more easily translate a “my bottleneck is this” to “this is what I should do”. It would give devs a single point of optimization with IDE auto-complete, so the options to optimize are easy to find. Then you can read the docs to get details. But look at the difference between a mesh.optimize({completelyStatic: true}) to what is necessary now, something like 4 calls and 3 flags?

Besides, I still think that the SceneOptimization classes could benefit from the high level information too. If you look at MergeMeshesOptimization, it sensibly avoids merging if there’s a skeleton. But maybe the mesh has a skeleton but it’s not being animated – as a dev I’d only know it’s not being merged if I read the Optimization code, and documentation about it would be as long as the code itself. Or TextureOptimization, which could get info per mesh or material as to how much texture could be downsized.

Still, only my two cents! It’s very easy to write a long post here saying how things should be :wink:

I understand your point but let me try to say what I said differently: It is almost impossible for an automatic function to determine what kind of bottleneck you are currently facing. Like I said, this requires profiling. We hope that the performance monitor will help but I don’t know how to create an API that will automagically determine that you are CPU bound and that the culprit is x or y.

Then if your ask is just to simplify the API by replacing some calls with one function that takes parameters, I would not be opposed to it but I do not see how it helps because you will still need to understand what each parameter does and when to use it. Hence the need to read the doc and profile to avoid shooting in the dark.

Yes, I’m asking for a simpler, higher level API. Not a magic and impossible automatic optimization system :wink: It helps because everything is concentrated: all calls that are directly related to optimization are in a single .optimize() call, even if it’s syntactic sugar for what’s present in the current code.

Look at it this way: you wrote Optimizing Your Scene | Babylon.js Documentation because it made sense to have a single place in the docs that mentioned relevant optimization procedures. If the same text was spread around all the other pages it’d be much harder to find it, right? So making it a single place in the code would make it much simpler to find optimization points, read their documentation and call them. Even the API docs would be easier to read, because a single interface would be able to handle most of the optimizations (it wouldn’t be able to replace copies for thin instances, of course). And instead of calling 6 functions and 3 flags for every completely static mesh, devs can call just .optimize({fullyStatic: true}).

I’m sorry if I couldn’t make myself clearer. I’m completely out of time to send a PR for this right now and would rather devote time to write doc for the material plugins, but perhaps in a couple of weeks I can send something more concrete for you to evaluate and see if it makes sense or if I just complain too much :slight_smile:

2 Likes

No worries at all. It is super cool to discuss about it :smiley:

And it is also super fine to disagree sometimes. But I’ll be pleased to be proven wrong so when you have time to send a PR

3 Likes

I like this idea too - and agree it would be worthwhile to be able to read through the documentation of each optimization in the code completion for the options param, as well as being able to find all the options in one place in the API doc. :slightly_smiling_face:

1 Like

As long as it does not incur a double maintenance and code duplication when adding/maintaining optims I am fine.

1 Like

You know, this is in fact something that could perfectly work as an addon to Babylon (like mentioned in that golden path discussion). It doesn’t even have to be in the main tree.

This is why having a standard add-on template, with all the build/TS/test infrastructure/publish instructions that can be easily forked would be so neat. I fully understand not having an asset store, but this repo template would be neat – it would be an easy way to publish new materials, material plugins or extensions like what’s being discussed here without bloating BJS itself (and if it’s a really useful one, it can be merged). And if the PG could import these extensions easily, like JSFiddle allows external JS… wow.

I’m a bit too busy to build this repo now, but if someone wants to do it with me I’ll find time.

2 Likes