Path-tracing in BabylonJS

Hello everyone,

Once again I am asking for seasoned Babylon users’ advice about loading models, in particular a glTF loading-related problem.

I recently received an issue over on the GitHub repo for our Babylon PathTracing Renderer, concerning a failure to load a user’s particular glTF model.

Here’s a link to the issue

In order to reproduce and demonstrate this bug/error most clearly, I went ahead and created a debug version of our glTF path tracing demo (which now temporarily lives in the same GitHub repo) that tries to load in his exact model (which I also got from him and placed into our ‘models’ folder). If you check the browser console while running this debug demo, you should get the same error message:

glTF Debugging demo

As previously mentioned, one of my many programming knowledge weaknesses is glTF loading and glTF data processing. So I don’t really understand what the error message means. I’ll be glad to work on the loading code part of our renderer to make it more robust and generalized for use-cases like this, but I don’t know where to start, ha.

Any info or guidance (or even ‘multiple-glTF loading / merging into 1 model’ source code) would be much appreciated. Thank you all in advance! :slight_smile:

1 Like

The main thing is you can not really merge geometry of incompatible meshes :slight_smile: Basically the attributes information needs to be compatible to be mergeable which does not look like it is the case in your scene.

Why can you not keep them separated ?

1 Like

To All, (especially @sebavan thank you for your quick reply)

Warning: long post coming up, ha. But I feel the need to explain the whole issue so we can try to fix it or at least make a work-around.

Just to bring everyone up to speed who might be reading this thread at this point, I received an issue on our GitHub repo saying a glTF was failing to load. The console had an error message from Babylon that I didn’t quite understand. Thank you @sebavan for telling me what that error message means - I think I follow you now! :slight_smile:

Sorry if you all know this info already, but basically all 3D models (glTF models in our case) are built up from various triangles, each triangle having 3 vertices, and each of those vertices having some sort of info about them that makes them unique. This vertex info is called ‘attributes’ (historically named I believe from the early days of OpenGL?). These attributes (or vertex ‘details’) usually come in the form of ‘positions’ (where is this vertex located in space?), ‘normals’ (which way is the normal vector pointing for this vertex?), colors(what rgb color is this vertex?), and textureUV (what are the texture coordinates for this vertex when sampling the triangle’s material texture, if any?). There are other attributes that might or might not be present among different models, which brings us to this issue.

Basically the error that GitHub issue OP was getting was the result of feeding the geometry merging routine with 2 separate gltf’s (not usually a problem) but that had different sets of info, or attributes, associated with each model. Maybe one had a complete set of UV attributes, while the other did not even have any UV attributes listed at all - in other words ‘attributes.uv == undefined’ or something similar.

So when the merge function gets to this point, it doesn’t know how to handle the seemingly incompatible lists of attributes for the 2 different models. Understandably, you can’t expect Babylon to magically add a list of uv attributes to the model that didn’t have them from the modeling program exporter in the first place. Therefore, as it stands with Babylon (and now three.js has followed suit - I just checked), the merge utility function throws an error and halts program execution altogether.

Here’s the spot in the merge function: source

I agree that there should be some kind of warning in the console if the attribute names for all models don’t line up exactly, but I sort of disagree with both libraries (Babylon and Three) throwing an error and refusing to go further with program execution (basically a crash).

For instance, a year ago I was able to load a TRON tank gltf model that was composed of 5 or so different components. Just for reference, I went back just now and tried with the current version of three.js and its handy function ‘bufferGeometryUtils.merge(geometryList)’ . Keep in mind that this used to load just fine - here’s what I got:

Then I simply went inside the three.js buffer geometry utilities source code and commented out the ‘throw error’ part, basically saying ‘it’s ok, I know what I’m doing, just continue with merging anyway, lol’ and voila:

A gaudy pink (my color choice) metal TRON tank (lol) but it works and looks great! So even when the attributes for the various gltf’s did not match, it doesn’t appear to harm the library or the user’s program when execution just keeps moving along.

So, I did a similar test with Babylon - I disabled the line where it says ‘throw error(…’ if the attributes don’t quite match, and the issue OP’s composite model loads just fine:

Just for reference, here is a pic of the online glTF viewer with the same data:

You’ll notice though that the diffuse albedo color and wood textures are not present in our pathtracing renderer, even though they are defined inside the gltf file. This is another side issue that I will have to address: namely, when the textures are actual external files, everything is fine, but when the textures are somehow expressed as binary data Inside the single gltf file itself, my loading scheme can’t handle it. I’m so glad this OP gave me this example and brought this to my attention, because it’s exposing so many flaws in my code (lol!). :grinning_face_with_smiling_eyes:

So I guess the question is, if Babylon does in fact spot discrepancies between the attribute lists of 2 different geometries that it is trying to merge, what do we want to happen? If an error absolutely must be thrown, then we must tell the OP and other users, ‘make sure your modeling program exports similar data for all the components of your model that you want to later merge.’
Or we could soften the blow with a warning from Babylon in the console, instead of an error and terminating execution altogether - but I don’t think this will be a popular solution.
Or the last option, and needing most effort, would be to write a similar preemptive check that Babylon’s merge function does, and if we detect attribute differences, intercept it by giving ghost attributes to geometry that are missing these attributes, so that all models match. This doesn’t seem very tractable though.

Sorry, for the lengthy post, but I wanted everyone to be on the same page as to what is happening and the decisions we have to make.

Regarding why I must merge the geometries in the first place (@sebavan ), rather than just leaving them separate geometries, is because the way I have my triangle BVH acceleration structure set up is that it expects a flat list of all the triangles in the model or scene. The vertex array indices must increase as you go from triangle to triangle, until all have been listed. The BVH takes this complete list, and builds an efficient, balanced binary tree of bounding boxes (with matching increasing integer ID’s to their triangle counterparts, box[0], box[1], box[2], etc…) for fast and efficient ray casting inside the GPU shader.

Now it is possible that in the future we could just leave everything separate and have multiple gltf models in the scene, each with their own transforms, each with their own material lists, etc., sort of like users specify for typical WebGL rasterized Babylon scenes currently. It would be great to have 100’s of different models floating around - but each of those models needs its own BVH tree. And then all those smaller trees need to be grouped into an overarching scene tree - so essentially a 2-tiered BVH system - a series of bottom level BVHs and a large top level BVH. This is NVIDIAs proprietary solution by the way. I made some successful experiments on small scales with my Sentinel game remake (that has dozens of models moving around a path traced scene in real time), but I’m not quite there yet for true, generalized Babylon scenes that might have an arbitrary number of models and triangles that a typical user might want to add. For the time being, everything must be squashed down to a big, flat list of triangles and their associated vertex data.

Thoughts and comments are welcome as always. Thanks for listening to my rantings, ha! Hopefully we can find a solution that’ll suit most parties. :smiley:

-Erich

1 Like

I guess 2 things are mixed up here. the need for a BVH tree requiring only indices and positions (hence why you mostly can disregard incompatible attributes ) and the rendering infos requiring all the necessary attributes to render correctly.

In your case I wonder if you could manually extract/copy vertices data with getVerticesData(“position”) and getIndices() for all your meshes and merge them in a separate buffer in order to create your bvh structure ?

3 Likes

Sorry for the delay. When I return from work, I’ll respond to your suggestions. Thanks! :wink:

Edit:
Ok I’m back!

Those are good observations, @sebavan - yes I think I could do something like what you suggest.

I’ll break down what the BVH structure builder would typically need from the models in terms of attributes:

  1. positions - this is a must have, as the BVH builder places tight AABBs around each triangle using min and max vertex position extents of each triangle.
  2. normals - this is also a must have, if smooth interpolated shading from triangle to triangle is desired. Otherwise, when raytraced, the model will appear faceted (flat shading).
  3. uvs - this is also a must have if the model contains any type of material textures, which most models do nowadays.
  4. colors - if the model does not contain any textures and instead has vertex colors, than I guess this would be a must have.
  5. tangents? - don’t know if we ever would need these for ray tracing purposes, and admittedly I don’t quite understand tangent space and how these operate, or what this kind of attribute would be used for.
    The rest, like matricesIndices and matricesWeights, I’m not so sure we need at this point. I’m assuming those are related to skeletal bone data? If so, in the future, it would be really cool to be able to path trace a gltf model with skeletal animation in real time.

As a fun side note, here’s a Brigade path tracer demo from 2013 running on multiple GPUs that path traces a 3D human model in real time with a skeletal animation (I think the bone dance animation is widely available as ‘Samba dance’ with Mixamo, or something like that):

Path tracing a model with skeletal animation
Even though it’s kind of creepy at times when they mess with the human vs. buildings scaling (lol), this is amazing (especially for the year 2013) and something I would love to learn how to do! Plus, this old video has got the perfect soundtrack. I say, “good choice!” to whoever selected the song.

Anyway, back on topic - so if I could at least get my own version of the attributes that I need, I may be able to merge them. btw, I would indeed need them merged, because as I mentioned the integer IDs have to match from AABB data texture, over to Triangle lookup texture. For instance, if leaf node box aabb[1729] from the AABB_DataTexture is hit, then the GPU path tracing shader quickly accesses triangleData[1729] from the Triangle_DataTexture to get this leaf triangle’s vertex data, including the above listed attributes info. Having matching uint IDs for everything makes it very fast and efficient to look up the data inside the shader’s tight path tracing loop.

The last piece of the puzzle would be resolving differences between attribute lists of different model components. A traditional ‘merge’ would not be possible in this situation (like the OP’s wooden shelf scene), so we would need to essentially make our own vertex lists where we can ensure that they all either have ‘positions’, ‘normals’, and ‘uvs’, or just ‘positions’ and ‘normals’ for models without material textures. Only then would a Babylon merge be guaranteed to be successful.

Also as a side issue, does anyone know how we can detect when there is a material texture that is actually defined inside the single gltf file itself?

For instance, when I open up the OP’s wooden shelves model in the online gltf viewer, a wood grain texture appears on the model, even though there are no separate texture files associated with the model. When I scroll through the file itself, it looks as though the wood texture is actually packed into a long string of binary text.

Since I didn’t know this was possible with gltf (learning more as we go, ha), my code only was looking for dedicated, separate texture files that happen to be in the same directory as the model. But I guess I need to do more checks. Also, the internal texture needs to be given to and read by the gpu path tracing shader (texture2d sampler?) when intersecting its BVH data structure. Can anyone help in this area?

Thank you!

I would advise you to create your own merge function by dupplicating the babylon one from our code and adapting to your need so you get exactly what you want :slight_smile:

About the textures and such, instead of parsing everything manually why not using the babylon gltf loader and reading all the data back from the babylon materials? everything would then be ready for consumption.

2 Likes

Ok that sounds like a good plan. I will work on creating a custom merge function that is parallel to the library and not conflicting - I’ll also name it mergeGeometryCustom() or mergeGeoForPathTracing() or something like that. That way, if the 2 or more geometries don’t match in terms of attribute names or numbers of attributes, we can avoid the error. In the background, I’ll gather the necessary vertex info from both (or all) gltf components to use in the data textures for the GPU later on.

About detecting material textures, yes that’s a great suggestion. I’ll just read the material data from what the Babylon gltf loader found included with the model upon loading. I suppose it could handle any arbitrary configuration of material data (textures, binary strings, vertex colors and no textures, etc.) so it would be best to defer to what the Babylon loader says! :slight_smile:

Thanks again for the clarifications and suggestions!

1 Like

This might be a silly question, but where does the wood grain in the issue OP’s shelf model come from? I can’t seem to locate it when I console log pathTracedMesh.material after it has been loaded into my scene.

Could it be that I was incorrect to assume that the texture is specified as a .png or .jpg inside the single gltf model file, and rather, it is just using vertex colors with a lot of triangles covering even the flat parts of the shelf (where typically you would have just 2 large triangles against each other making a rectangle).

If it’s the latter case, I could easily work on supporting vertex colors for the path tracer (which is currently not implemented).

It might be in the bump texture ?

Well, when I check the bump texture, it reports back null. Somehow, this model is bypassing all my checks that I would do to see if the model has any material info.

I noticed it has 3546 triangles, which seems like overkill for a simple wooden shelf model like this. Where are all the triangles going? lol

Edit: nope I am wrong again:

Looks like all the triangles are going towards the blue joint things (can’t remember from wood working what these are called? ha), and the Opaque text printed on the boards. As expected for a flat surface, the flat rectangular sides of the shelf only have 2 large triangles, so there goes my vertex color theory. However, vertex colors are being used for the blue things (according to console log of material useVertexColors flag, so eventually I will have to add support for those as well.

So it would definitely be in albedo or metallicRoughness :frowning:

@erichlof You are right about the “hidden” triangles. About these drillings : they are called “hardware”. Usually it is either metal (minifix) or wood small shapes (dowels)
Tou can try this asset later for your tests : https://raw.githubusercontent.com/pcarret/assets/master/alk/62921215.gltf

image

1 Like

Thank you for the clarifications, and the extra geometry to try! :smiley:

I’m curious, how did you specify that wood grain originally? Was it a texture image that you applied in your modeling program? And when you exported the gltf, did it somehow pack that .jpg or .png wood grain picture into a string of binary text at the bottom of your gltf file?

@sebavan The fact that this is able to show up in a viewer means that the GPU is receiving the wood color texture data… somehow! This is a fun puzzle :grinning_face_with_smiling_eyes:

1 Like

@erichlof :

If you open the gltf text file, you can notice it was generated by Kronos Tool COLLADA2GLTF
I do not know the internal of this tool
Originially the file is a Collada one with a separed texture/image directory with all files and their subpath referenced in original dae file.
The COLLADA2GLTF tool put all textures in binary form encapsulated in gltf.
This is great coz everyone can send the file by itself to someone else, no texture are missing (ok the weight is +++)

    "asset": {
        "generator": "COLLADA2GLTF",
1 Like

@pcarret
Ahh ok that makes a little more sense now. Thanks. Good news is that the model geometry seems to be loading into our path tracer just fine:

Never-mind my abysmal framerate (I have an old, weak laptop with integrated graphics, lol). But anyway, the model seems to load in perfectly, it is just missing the wood texture data, and the screen saver on the monitor (which also was a texture I’m assuming?) . Once we figure out where it is hiding inside of Babylon’s engine, I can start working on making everything render properly. :slight_smile:

1 Like

You should simply load the model in the sandbox and use the inspector to see where it is coming from.

1 Like

@sebavan Ok will try that! :slight_smile:

1 Like

To All,

I have updated the GitHub repo to include the model that @pcarret just sent over. Everyone should be able to view it on their computer now. And if you want to see the source code, you can take I look at what I did. btw, I added my temporary fix to load the geometry, which consists of a simple ‘continue’ statement instead of the ‘throw error’ statement inside both Babylon.js and Babylon.max.js so this can work:

BookCase demo

An interesting note: you are looking at no less than 151 gltf meshes path traced inside the browser in real time! - Hence my requirement to combine meshes for the BVH acceleration structure builder. But nonetheless, I think we can be proud so far of this gltf multiple component loading and path tracing ability! :smiley:

4 Likes

@erichlof Thanks for this workaround
Is denoising a mandatory postprocess technique to use after PathTracer ?

4 Likes