Shado - 1 million instances, 20k+ rendering, auto VAT, full custom properties, dynamic WebAssembly

Hello hello everyone, I am emerging from the depths – I was pretty active awhile back working on a game and supporting frameworks and got sidetracked with work and then RL, and somehow after our new child was born recently I’ve had enough time to collect the various parts that had supported that and publish a release version of the package extension for BJS core I’ve been using. In the meantime the pre-release version is something I’ve been able to integrate into my work and will be using the package for future-forward 2d/3d visualizations for high scalability and high performance rendering.

Introducing Shado ( https://www.npmjs.com/package/@knervous/shado ) major release, available for public use and maintenance / requests.

Shado is a full plugin extension for BabylonJS. I’ve been working on this from the ground up in different ways for a few years tackling different problems separately. It is not for every use case, and requires to stay inside its framework to be useful to the full extent. Shado is right for you if:

  1. You are rendering hundreds / thousands of actors of the same entity type that differ in ways not currently exposed via BJS core (thininstances support base attributes like location, rotation, scale, rudimentary animation controls), and would benefit from concepts like many (50-100k) entities “defined and live in runtime” but not rendered at any given moment.
  2. You are willing to dive into a more technically-complex solution in a trade for high performance scaling (comfortable with writing some custom shader logic and reading / understanding assemblyscript - not a hard requirement but streamlines dev)
  3. You are looking for full-stack messaging solutions that scale well beyond existing frameworks like protobuf, flatbuffers, capnp etc. This is an advanced feature to sync backends with a shared message format that reduces deference timing and GC alloc that “solves” a lot of network backend scalability for higher level language backends like TypeScript/JS in nodejs for a backend high throughput server like an MMO server.

Before going through the whole spiel, a picture’s worth a thousand words:

51k instances, all unique animations and skins and held items (sword, mace, spear, etc.) running at 60fps while 11k of them are rendered on screen. The AssemblyScript/WASM culling reducer is a sidecar layer for the underlying memory structure that gives a very fast hot loop scan and trading 2ms / tick for 50k instances can effectively radius cull our rendered instances while still keeping them alive in memory.

Here’s the PG and here’s what it’s doing:

  1. Has a set of .glb’s that back the set of demo models - complete with textures / animations etc.
  2. Bakes them to VAT (a new format, potentially 4x smaller via dual quaternion VAT baking and float16) and creates a Shado wrapper
  3. Updates UI panel to allow for inspection and loading more models, including stock BJS PG assets or bring your own glb to see how it works - change armor, held items, name, etc.

How I got here, a walk down memory lane:

While working on the game port a few years back, I found some areas I wanted to really hone in on.

  1. Animation / instancing - VAT as a substitution model: Optimizing VAT Baking to near-instant and using Float16 for 50% memory savings - #9 by knervous
  2. Basis texture arrays - This is the current bake model native in the framework, creating a texture array atlas and mapping to textures for a baked entity - Support for RawTexture2DArray with compression from basis textures - #3 by Deltakosh
  3. Unified memory usage, field definitions. The start of Shado - conceptualized a “define once, use everywhere” model of C-like structs where we can use the same object from TS/JS in code (actor.armor = 2;) as in shader language code (auto generated now in Shado) and in WebAssembly by sharing memory. That is now how Shado works, one memory for each group of actor types used across all three domains. JavaScript/TypeScript object mapping in GPU Shader structs for complex binding
  4. Nameplates. I noticed a bottleneck in current MSDF is scaling unique msdf instances - in practice in my game, entity names are unique most of the time. I can’t have nameplates.forEach(render) running in the hot loop with 500+ NPCs. Nameplates are now baked in as a sidecar in Shado that require only 1 complete additional draw call and allow modifying names in runtime (try in the PG!): 10k+ Instances with MSDF Text Nameplates and inline WASM compilation
  5. Particle system integration and layered sockets/particles across a scene - This is next up once it surfaces in a use case in my game. Will be fun to scale out gpu particle systems to o(1), these are some rudimentary proofs of concept from a long time ago GPUParticleSystem syncing with VAT animations on mesh instances

The npm library should be well-documented and ready to plug in use cases. The repo is a subdirectory in my game repo which is in rough dev right now: GitHub - knervous/eqrequiem: EQ Requiem · GitHub - subdir shader-object

My WIP game I’m churning through is https://eqrequiem.com - a legacy project being overhauled to transition to a new game with this framework.

Glad to be back among the babylonians and hopeful I can appear and contribute accordingly in future weeks, months, years. I see much has changed and can’t wait to dive into Lite and get a lite package for this as well!

Cheers,
Knervous

9 Likes

This could be used for Babylon - Total War! Super impressive :slight_smile:

3 Likes

Really good performance!
I think I will use some of your ideas for my Lite Instancer.

There are already some experiments with massive VAT scenes like Unarmed VAT Arena Crowd but I am sure that I’ll be able to rise a perf greatly due to your findings.

Thank you!

2 Likes

Your recent post was inspiration to get this over the finish line! Great work yourself, spent a good bit going through your impl. Good news is I believe we are solving separate problems, would be happy to collab on any common efforts or chat about arch!

2 Likes

This is insane !!!

1 Like

Man! This is amazing. really. I will reserve some time today to go over the examples in more detail

CC @PirateJC for visibility

1 Like

Thanks @labris for some good feedback via PMs :slight_smile:

Been iterating on the next minor version 1.1.0 with lots of performance improvements. Notes are in the changelog, here is 140k+ entities in memory and 11k+ on screen rendered with different animations and texture sets

And support for lite now as well!

3 Likes

I’ve been trying to get something similar for BATs animations. I was able to do something similar with VATs, but BATS has been a different story. I was able to condense 300+ skeletal animations into a single BATs image. It’s a huge file, but you only need it once for every player. It’s the only method I think will work for getting Unreal 2k4 multi part player character assets working.

If you have these models hosted anywhere would love to take a look!

Probably final pass for awhile, went through last optimizations, now have over 1 million instances hot in memory and 20k rendering at 60~ FPS!

Initial PG still works and targets latest shado

4 Likes

kinda… There is a whole MMO worth of assets… I’ll DM you with a link.

There are issues in my unreal material parser. AI broke some things that i need to fix (i haven’t had time)

1 Like

Been doing work on this in the background while supporting my current game dev and spitballing with @reddozen – There is a nice use case for the following:

  1. Actors are built up of many possible variants of submeshes (different arms/legs/hands) depending on configuration or in-game equipment
  2. Need to scale out to N instances while retaining each instance’s unique combination without prebaking configured modes, i.e. letting it scale horizontally forever

Came up with a concept for this in the latest npm release via shado - a concept of a shado container instance that can add any number of submeshes and share across all instances, also including different tints

This fit perfectly in the step I took recently to incorporate open licensed art assets from the Ryzom game which uses this exact paradigm: A human model has a base, then depending on equipement, swaps out arms, hands, feet etc.

Here is a demo scaling it out to 5k uniquely animating and configured instances (also with tint) rendering at 60fps. 10k hits about 50fps on my machine

Recently doing some repo migration work and will have the latest npm release synced with a public github (moved to private for my game assets)

Other parts of this including further optimizing the dual quaternion VAT baking using a different contiguous schema and zst compression - end result is what would originally be a 10mb VAT from core Babylon bake compresses to 1mb

8 Likes

This is insane :slight_smile:

1 Like

Finalized this round and published the latest package at @knervous/shado 1.6.0 and published the repo again locally at GitHub - knervous/shado · GitHub

Added a texture mask and more variation to the sandbox demo which is still pretty comfortable on an older phone.

One very nice QOL included in the latest shado is a dev cli or scripting API from an idea I had – I think it could be ported to generally be Babylon specific: use Google’s dawn library which is a cpp layer over webgpu to provide a headless node instance with a real adapter in order to render a real scene with a full BJS webgpu engine. It has massively sped up visual validation being able to write small template scripts to spin up a one time png output for any glb or a complex custom render sequence. Lots of other potential use cases too like streaming.

3 Likes

Was pondering making this its own thread but don’t want to muddy up the forum with content creep posts :slight_smile:

Followup on the last reply I made here about the headless WebGPU rendering, it dawned on me to take this a step further from generating stillframes to video writing and streaming would be not a huge lift and has quite a few nice use cases. Babylon already has some nice video capture tools built in, but requires the browser to run them. I just pushed a package update that includes an extension CLI @knervous/shado/video via npx or a scripting entrypoint that adds a few pieces of functionality:

  1. Pass in a glb/gltf/custom setup script via npx shado-video --input model.glb --out spin.mp4 --seconds 6 --materials and it will render 6 seconds of video (as fast as possible rendering all the frames, usually 1-2ms per frame) and save it to disk as mp4 (gif also supported)
  2. Rendering a BJS scene headlessly on a server and streaming it to a browser client or clients who are connected. This would be very useful for a multi-user application or demo that has deterministic rendering and predefined state on a server and can broadcast a standard H.264 encoded video stream for any type of client to connect to. I know I will be incorporating this into scaling and access tier gates on the job.

More documentation on this here shado/VIDEO_CAPTURE.md at 04d2eb1edcb5a2f67e7e437601189667cfbd1d38 · knervous/shado · GitHub

Here’s a rendered fox generated headlessly:

fox

I also did some looking around after the fact and found there is already a headless renderer using Dawn the same way for three which was a bit disappointing, obviously not a novel solution, but as far as I can tell, this is the first plugin for BabylonJS and it is pretty thorough.

1.7.0 released here to get started!

https://www.npmjs.com/package/@knervous/shado

3 Likes

Gosh, I have so many questions and so little time :slight_smile:

So, if I get this correctly, we can render a headless scene (or just a single frame of it) on the server and send it streaming to the client. Is there a way to get a single image as well? This could simplify visual-testings. I guess I can pipe the ffmpeg video output and export a single image, just wondering if I can do it straight with the framework instead.
Does it work with any BJS scene? Or is it strictly for scenes running for Shado?
(The one question I can answer myself - Am i too lazy to read the full docs? yes…)

Haha yeah documentation is slowly becoming fodder for the robots nowadays anyway :wink:

To answer directly yes, headless nodejs server (also in CI) can render a still image or stream them at a desired frame rate to disk or live. Entrypoints are generic so no requirement to use shado, just decided to keep the tool in this package. Can be a bjs scene file, custom js script or gltf model file.

Caveat is it’s webgpu only, but if I’m picking up on a hint, it could replace playwright in CI for that test band.

Part of the dep chain is the ffmpeg npm package but flagged optional since it’s a big binary payloads. There are other options out there I toyed with like rs-napi webcodecs but in the end they do a static link to ffmpeg anyway, but is faster at encoding decoding.

Would enjoy a real use case to throw it at and see if it meets spec.

2 Likes

not a hint! a very direct ask :slight_smile:

Most Ci agents already have ffmpeg installed, so that’s a major plus. I will run experiments this week. WebGPU is perfect - Babylon-lite is pure webgpu.

4 Likes

Nice! Let me know if anything about this ends up a headache or not working. Just published a new version 1.8.0 exposing adapter for lite, was already an optional peer dependency. Also an option for bring-your-own ffmpeg. Updated docs in the repo:

import { createLitePreviewSession } from '@knervous/shado/devtools';

const session = await createLitePreviewSession({ width: 640, height: 480, decodeImage });
await session.newScene({});
const container = await session.loadGlb(bytes);
const camera = await session.frameCamera({ zoom: 2 });
const source = await createSessionFrameSource(session, {
  width: 640, height: 480, camera, format: 'yuv420p',
  onFrame: (t) => { clip.currentTime = t % clip.duration; },
});

And ffmpeg from path yaml

env:
  SHADO_FFMPEG_FROM_PATH: '1'
1 Like

you’re awesome :slight_smile:
I’ll keep you updated