This came up in an experiment with using a shader to do some work on multiple textures, and using the output of the shader as a texture for another material.
The basics are as follows:
Load N number of textures, and once they are loaded, convert them to a RawTexture2DArray
Use the RawTexture2DArray as the uniform for the ProceduralTexture
Use the output of the shader as the texture input on another material
This is pretty simple, but I am noticing a few quirks that I’d like to understand (pardon me if some of these are dumb!)
Why do I need to wait for texture ready before I can read the pixels? Is there a faster way if that’s all I need? Like perhaps loading the images as buffers using fetch?
My tdArray input can’t be disposed without breaking the shader. If I am just creating a new texture from the output of the shader, isn’t there a way to stop that from having to re-render so I can dispose all the inputs? After all, it should be static, right?
Is this optimized or is there a better way to accomplish this? What do you all think of this approach?
Your example is telling the engine to load 2 textures into the gpu memory from a disk or network resource.. in this case network resource (url). So, it has to download those images first, then upload them to the gpu and correct the render format, if they’re not already in the render format desired. That’s 2 main reasons you have to wait. The 3rd is because when you execute readPixels you’re downloading the gpu texture back into a cpu side buffer for your code to process those pixels. You can control this behavior with the constructor, or load the textures with an image loader yourself and use the contructor with those image buffers for more control. See here: Babylon.js docs and here Babylon.js docs
When a shader is given a uniform that data is usually deemed as in use in most engines and the underlying api (webgl, webgl2, webgpu, etc). If you dispose of the array it may unbind the shader uniform causing the error because the resource is no longer available, or incorrect. Plenty of ways to cause this including context loss. If you loose your rendering context those textures must be recreated in the gpu in most cases.
There’s many ways to optimize this but up front, you can load the images and grab the data from the browser yourself but this can be a little more complicated because now you’re on the line for making sure it’s in the correct render format expected or desired. Plus taking into account things that you would have to anyway, like if the textures are in linear color or premultiplied alpha, etc. Depends how far you want to take that. Sure it saves a round trip from cpu to gpu but it puts more load on you. You can also preprocess your images and store the desired output as a single load and specify the data in the RawTexture2DArray constructor like you are but that would make it require loading that entire data array from the disk or network. It’s all trade-offs on what you want to manage or deal with as you go.
Can you explain this further? Is it possible to use the Babylon Texture loader to only parse the image file (KTX, PNG, etc) and return the pixels without the round trip to the GPU?
For example, I can do that using fetch and canvas but I’m wondering if this is possible with compressed texture assets.
If you load the data via html, either it or you will need to decode the compression. There are limits to what html interfaces decompress and it fails quickly because the formats are very specialized.
When I say you can control the behavior, you can hand the constructor a prepared data buffer and tell it not to delete / release that buffer on dispose, etc. This leaves you with more control but more memory hanging around.
You can load the image data with fetch, but if it’s a format like KTX2, ASTC, even cases of DDS will need a decoder prior to reading the buffer raw. There should be a KTX2 decoder in WASM but that’s one more dependency and hard requirement that gets pulled in.
You can also use babylon to create the canvas as well if you like but it’s not going to end-run around any of these issues. i.e.
If you want to know how to retrieve the possible compression formats in use / available you can query it like this, Be sure to check the console output when you run it.
With current, I don’t feel like your solution is THAT bad. You can always use scene.executeWhenReady given a function to execute if you don’t want to await on the process, and instead show a loading / processing status until it’s ready.
Will you need to modify the layers in the texture array or do anything except blend them once? If all you need to do is blend them once, setup a bake to texture and reuse that rtt.
Yeah, that’s the wrinkle. If all we needed was one variation we could bake them into one KTX file, but since we’ll have to support multiple variations (V^P) it could get out of hand quickly. And these sorts of textures (cast shadows) can bleed outside the UV islands, it seems like it might work to just blend in a shader on the fly. This seems to work pretty quickly, but I’m wondering if there’s a way to optimize it.
So I was looking into storing multiple images in a 2d texture array awhile back for the use case of a compressed spritesheet.. Lo and behold compressed textures in rawtexture2darray aren’t supported yet and I started another thread about it here:
If you’re okay with doing some preprocessing with your images, this might be helpful. I ended up using basis encoding with a node script. Basis supported 2d texture arrays so it ends up with one file (one network call) to fetch all the images. There is a binary wrapper in Node with npx basisu. Here is the script I use to create atlases:
And here is a lot of the work tied together to combine loading basis files in runtime and either completely decompressing to flat RGBA (worst case scenario) or using the available compression based on engine caps and uploading compressed textures to the GPU. Then you can upload as a sampler2DArray and do your work in the shader. There’s a lot more going on in this PG but I hoisted all the relevant code to the top, lmk if this is something that might fit your case.
Correct, they aren’t. In order to get use of the compressed textures well with shaders you would have to swap to an array of texture2d instead of using a sampler2DArray/texture2dArray and if supporting webgl/webgl2 you would be limited to somewhere between 8 and 16 textures in those as it is. Midrange mobile’s don’t like using 16, so the target use and audience matters. Otherwise if it’s webgpu only, other paths open up.
You can use an MRT to render the textures into a texture2dArray but that depends again on the hardware.. The cpu path that they have already is the perfect fallback mechanism for no MRT support. So this really becomes a matter of hardware targets and use-case to optimize it specifically.
To optimize scene load, you can always generate an empty texture2darray to bind up front while the processing is taking place in the background, then when finished onBeforeRenderObserver can be attached to for the procedural texture to rebind the texture2darray to the final data. i.e.
if (scene.getEngine().getCaps().texture2DArray) {
console.info("TEXTURE_2D_ARRAY is supported!");
} else {
console.warn("TEXTURE_2D_ARRAY is not supported, falling back");
}
Clarification: When I said between 8 and 16 I meant as an array of texture2d..not a texture2darray. number of samplers matters.
I wrote up a PG for the original demo using an implementation for compressed GPU textures. I have a few config variables at the top to add in an extra 2 and 64 textures into the original basis file. The 66 texture 2d array seems to load on my phone, I’m not too savvy with all the limitations of 2d texture array in difference browsers in hardware though. The texture arrays I’m using in my game as spritesheets have many, many textures (200+) and seem to be loading for most people who have used it so far.
When I say 8 to 16, I’m not talking about texture2darray, I was highlighting some limitations of using an array of 2d textures. Same reason in cases where even with an MRT you could end up in in a chunked processing mode where you can only process 4 at a time on the gpu in some cases.
@Alex2 check out that playground and use that example. There’s a lot of good info there on the use of the basis format over the KVX2.. basis is the same compression as KVX2, it’s just not khronos format and .basis might end up being the wider used one. For compressed texture support that’s more than enough.
Thanks to everyone for jumping in (I was trying not to engage in work this weekend ). It’s a fun topic!
The .basis example doesn’t seem to load for me in Firefox (it says the transcoder does not support this format), but loads fine in Brave (and I assume Chrome). In general, the ideal approach would be to use Babylon to load a single KTX2 file with a texture array that we pre-bake, and we could generate a lookup table when we encode it to figure out which layer is which, but we don’t know right now how many variations there might be, so there could be two dozen layers in the file even though we only need to display a handful at any given time. Additionally, it appears that Babylon does not currently support loading KTX2 files with texture arrays, so that may be a problem long-term. Am I correct that .basis was donated to Khronos and isn’t actively maintained anymore?
In general, it looks like all browsers support WebGL2, and I can use the texture2dArray if I construct it myself, so it seems like the main question I have is whether or not it’s faster to load PNG files and read their pixels or load KTX2 files (using a custom loader) and do the CPU=>GPU=>CPU transfer.
Considering that the loaded KTX2 files take up GPU memory and the PNG pixels can be discarded, that seems like it might work. So far it looks like PNG => pixels is every slightly faster to load.
I guess I need to prove that with some real benchmarks. Right now that’s just my eyeballs telling me. Stay tuned!
EDIT: Playground with just lots of duplicated maps.
Seems like we can load and process these textures (at 256px) in about 20ms, and if I ramp that up to about 20 textures (cached in the browser), it takes about 100ms. Might be good enough!
And to throw a little bit more into the mix, merging the images using Sharp or Rust is actually slower than the shader loading PNG files, although Rust is really fast blending smaller files like the example Babylon textures in the Playground. Once we start using larger textures, Rust is slightly slower and then you’d still have network time and GPU upload time.
Great job on that example!
). It’s a fun topic!