JavaScript/TypeScript object mapping in GPU Shader structs for complex binding

Demos and projects
1

Hi all,

Something that popped into my head awhile back while preparing to do some extension work on existing shader code was how binding data/buffers on the CPU–TS/JS side to the GPU level can be a hassle and there’s a lot of maintenance and manual updating going on while defining uniforms and updating on bind.
Some inspiration for this framework comes from buffer message protocols like protobuf, flatbuffers, capnp. I wanted to be able to create a binding between the two languages (JS/TS and GLSL) to create an object in JS/TS easily and have it be represented/accessible/automatically updated in the GPU side to avoid manually managing each uniform and track indices/strides/”types”

I’ve been working on this for about a week and still have a lot to do in terms of naming/ergonomics/testing. The WebGPU side has some kinks to work out as well, but this is the PG I am working from

The gist of it is being able to define objects in TS/JS like this and bind them to a ShaderMaterial automatically. This could be extended to NME likely as well. Wanted to share this and see if anyone has any thoughts on usefulness and direction.

// TS version - direct binding with decorators

// @gpuStruct({ name: 'TestStruct' })
// class TestStruct extends GPURecord {
//   constructor(engine: BABYLON.Engine) {
//     super(engine);
//   }
//   @field(1, 'vec4') color!: Float32Array; // live view (Float32Array)
//   @field(2, 'f32') lifetime!: number; // scalar
//   @field(3, { varOf: 'f32' }) times!: number[]; // array
// }


// JS compatible version with schema builder

class ChildInstance extends GPURecord {

static schema = new GPUStructSchemaBuilder('ChildInstance')
.registerField(1, 'id', 'u32')
.registerField(2, 'color', 'f32')
.build();

constructor(engine) {
 super(engine);
}
}

class TestStruct extends GPURecord {
  static schema = new GPUStructSchemaBuilder('TestStruct')
  .registerField(1, 'color', 'vec4')
  .registerField(2, 'lifetime', 'f32')
  .registerField(3, 'times', { varOf: 'f32' })
  .registerField(4, 'instances', { varOf: { structOf: ChildInstance } })
  .build();

constructor(engine) {
  super(engine);
}

}

Will be continuing work here and providing updates as they come

2 Likes
2

Been chipping away at this endeavor while on vacation and honing in on what I’m trying to accomplish.. It appears StorageBuffer backing for the WebGPU side would need additional changes to “plug and play” into GLSL that’s translated but still would work if written all in WGSL. For now everything is backed with a one-segment arena, one array buffer, that has a view on top representing all the internal types including variable sized arrays, nested structs.

Here is a demo of using it to sit on top of Mesh thin instances and create a stronger “handle” on instances that include any additional metadata you’d want to define for the instance and not just the mat4 translation. This is just one use case but something I’m leaning towards being a good candidate for extension since the thin instance logic can be applied to other parts of the framework like particles, textrenderers, potentially lights with pbr

Here is a PG piggybacking off thin instances

And here is a PG implementing the instance draw call manually with the backing transform data used in the vertex shader

3 Likes