Babylon.js Benchmark Suite

*This is more of a suggestion for addition to the bjs main page rather than the repo. Sorry for formatting, there’s only so much I can do with generic BBcode :wink:

There are literally hundreds of posts by devs looking at BJS 3d engine for their applications needs. They are either on the fence or unsure if BJS can meet their specs/needs completely. There are also cases where devs might want to check if their target device(s) can support BJS/webgl functionalities with hard data/comparisons.

Currently, there are a few ways to determine if a device is up to webgl specs.

  • WebGL Conformance testing.
    Pros: Highly detailed, exhaustive.
    Cons: Takes a long time on older machines to complete all tests. Highly technical lingo.

  • Browse the docs and visit the PGs
    Pros: Fast, informative, educational, visually confirmative
    Cons: Examples are generally simplified and showcases intended feature. Not application directed. No performance comparisons.

  • Community showcases
    Pros: A huge variety of applications with different levels of BJS integration and usage.
    Cons: Not maintained by BJS team, not necessarily industry standard. No comparisons.

Devs have a vague idea of the full capability of BJS on their target device.

Proposed Solution
The idea is similar to a consumer looking to buy a TV/graphic card/PC/ this days. For a start, we’d naturally look at its performance and then compare prices :wink: . In the PC world, we live and die on BENCHMARKS!

Consider this graphic card benchmark from Eurogamer

Pros: Explicit, data-centric, application tested, clear, concise, convincing.
Cons: It take time, effort and :moneybag: to track and maintain benchmarks ! :open_mouth:

In a similar vein, I propose the Babylon.js Benchmark Suite!

BJS is a swiss army knife, middlewares are usually so. However, 3D web applications tend to fall into a few categories. Most of which can be easily supported by MS native industry available products. I propose benchmarks for applications in the realm of

  1. Gaming (strategy/rpg): BJS benchmark scene from Age of Empires 2.
    Showcase particles, instances, animation, environment, pathfnding, GUI etc

  2. Gaming (racing): BJS benchmark scene from Forza
    Showcase physics, collision, particles, pbr, GUI etc

  3. Gaming (fps): BJS benchmark scene from Halo
    Showcase character animation, particles, realtime performance, pbr, GUI etc

  4. Architecture (exterior): BJS benchmark scene from (insert some MS building in some country)
    Showcase pbr materials, lighting and shadows, reflections, environment, etc

  5. Architecture (interior): BJS benchmark scene from (insert some MS office)
    Showcase pbr materials, lighting and shadows, caustics, refraction, AR, VR etc

  6. Product Visualization (hard surface): BJS benchmark scene of an automobile, complete with the ability for a user to view individual components.
    Showcase pbr materials, lighting and shadows, sheen, reflections, refraction, AR, VR etc

  7. Product Visualization (organic): BJS benchmark scene of clothing. (erm, i have no idea…I don’t do product vis :wink: )
    Showcase pbr materials, lighting and shadows, physics, GUI, AR, VR etc

These benchmark scenes MUST be industry standard, able to push the mid-high tier capabilities of the BJS engine. They are not single feature only, but can standalone in an exhibition. They are tested against commonly available devices: desktops, laptops, hps, tablets etc with clear and concise performance results quantified terms of OS, browser, gpu, mem consumption, frame rates etc,

Pros: Devs can easily check how their device fares against a standard BJS coded application tested scene. Useful for BJS to showcase their full capabilities with industry standard applications. Useful goto site that allows for focused development and discussion. No more vague performance concerns. Also works as advertising for MS products.
Cons: Need money, need dedicated team, need maintainers.

That will be the problem :slight_smile: