Is babylon caching to indexedDB works as expected?

[TooCalm]

Hey Guys,
I see in babylon docs that babylon saves to indexedDB only .babylon files and materials.

Optimizing Using Cached Resources | Babylon.js Documentation

But google chrome docs says any files can be stored to indexedDB.

Working with IndexedDB  |  Web  |  Google Developers

I assume this feature was developed when there were some restrictions or smth. Could you please double check if those restrictions is still exists. Why babylon can’t save .glb files to cache.

Second thing here is that i can’t even force babylon to cache .babylon files. The only thing which babylon caches is materials with glb files.

But here is third thing that when i refresh page, babylon caches material again(and everytime) and indexedDB memory grows everytime i refresh page. so memory leak here(I see in browser how indexedDB have new line in “textures” folder with the same material)

recap
glb files. Babylon caches material every refresh. It doesn’t cache glb. Memory leak here
babylon files. Babylon doesn’t cache at all.

For me it looks like this feature is completely broken.

Could you please check is caching works as expected?
Do you have plans to add support for another files?
Do we have some updates on this feature in 5.0 release?

For loading mesh i used
Babylon 4.2.0
TypeScript
BABYLON.SceneLoader.LoadAssetContainerAsync method

Thanks

[Deltakosh]

I recently fixed the textures cache adding all the blob data from each gltf
That is why you see a “leak”

We are planning to completely rewrite the IndexedDB code after 5.0. The code is old and was done long time. We will obviously let it support all our formats (including glb, etc…)

If you have a repro where .babylon are not cached I’m interested because it should work (you have to turn it on though)

[Deltakosh]

Modernize IndexedDB · Issue #11816 · BabylonJS/Babylon.js (github.com)

[TooCalm]

I’ve tried to find an example and i’m even more confused.
First of all both examples from Optimizing Using Cached Resources | Babylon.js Documentation
at the bottom of page:

" Samples
Most of the samples on our website mainpage are configured to use offline for their scene and textures. For instance, you can try the Espilit scene. The scene is described in espilit.babylon and the associated manifest file is espilit.babylon.manifest .

One of the scenes is configured to only cache the texture. It’s the “The Car” scene. It’s because the JSON file, TheCar.babylon , is more than 93 MB. IE11 and Chrome can’t store a big file like that into their DB, so decision was made to avoid trying to cache it."

are not caching anything.
I thought that I found an example but it looks like feature is not enabled on them because i don’t see any manifest file requests

[TooCalm]

I checked about 20 examples from here Babylon.js Demos and some examples from playground and none of them are using cache(none of them have manifest file for the meshes uploaded). so i don’t see how i can provide an example.

Also what exactly are doing this engine property?

Engine | Babylon.js Documentation

and how it works with combination of
BABYLON.Database.IDBStorageEnabled = true;

[Deltakosh]

cc @PirateJC to fix the doc

[Deltakosh]

We turned that off long time ago as it was crashing Chrome (older implementation, I bet it should work better now). We forgot to update the doc. Sorry about that.

[Deltakosh]

Yeah I know it could be a bit confusing at first so bear with me

engine.enableOfflineSupport is off by default. You can turn it on to enable the system to use an offline provider. By default we provide one offline provider: the IndexedDB one.

BABYLON.Database.IDBStorageEnabled = true will turn on and off the IndexedDB one. Meaning that in that case the system will try to use another one if any.

Let’s have a look at an example:
prova | Babylon.js Playground (babylonjs-playground.com)

You can see that the system is trying to fetch the manifest:

By default the system will provide you with a Database class which alway checks the manifest first but we can force it to not do that and always cache resources:

BABYLON.Engine.OfflineProviderFactory = (urlToScene, callbackManifestChecked, disableManifestCheck) => {
    return new BABYLON.Database(urlToScene, callbackManifestChecked, true);
};

With this update, we can create a demo in the PG where the system will cache the .babylon file:
prova | Babylon.js Playground (babylonjs-playground.com)

And as you can see, the file is now cached and loaded from IDB:

Ok now let’s use the same trick with a gltf file:
glTF Loader Demo | Babylon.js Playground (babylonjs.com)

And it works
Data is stored and retrieved from IDB as expected:

[Deltakosh]

That being said I agree that our doc is terrible on that topic

[TooCalm]

Thank you for your answer.

So my problem was that i didn’t enable engine.enableOfflineSupport = true
Looks like it is must have property for feature.

I’ve tried to enable it in my project and got next results:

I added your code which overrides offline factory to disable manifest checks

.babylon files. engine.enableOfflineSupport = false; IDBStorageEnabled = false;
result: nothing cached. it’s expected because feature is disabled

.babylon files. engine.enableOfflineSupport = true; IDBStorageEnabled = false;
result: nothing cached. I assume it’s expected because the only offline provider indexedDB is disabled.

.babylon files. engine.enableOfflineSupport = false; IDBStorageEnabled = true;
result: nothing cached. I assume it’s expected because engine feature is disabled. Such combination can be in case i have 2 engines one with disabled feature and another with enabled.

.babylon files. engine.enableOfflineSupport = true; IDBStorageEnabled = true;
result: everything is cached. I have 10 meshes and all of them was cached. when i refresh page, I don’t see any requests for those meshes in network tab. expected result.

At the moment everything looks pretty good.

But now i will remove code which overrides offline factory. Results are strange:

.babylon files. engine.enableOfflineSupport = false; IDBStorageEnabled = false;
result: nothing cached. no manifest checks. it’s expected because feature is disabled

.babylon files. engine.enableOfflineSupport = true; IDBStorageEnabled = false;
result: nothing cached. no manifest checks. it’s expected because the only offline provider is disabled

.babylon files. engine.enableOfflineSupport = false; IDBStorageEnabled = true;
result: nothing cached. no manifest checks. it’s expected because feature is disabled. Such combination can be in case i have 2 engines one with disabled feature and another with enabled.

.babylon files. engine.enableOfflineSupport = true; IDBStorageEnabled = true;
result: only last loaded mesh was saved to cache(???). All meshes was manifest checked. After page refresh cached mesh was loaded from DB.

Do you have any thoughts?

I used next code:

  private static fieldTypes = ['balanceField', 'balanceXField', 'forestField', 'forestXField', 'waterField', 'waterXField',
    'mountainField', 'mountainXField', 'sandField', 'sandXField'];



    this.battleFieldMeshContainers = new Map<string, BABYLON.AssetContainer>();
    const meshPromises = [];
    /* Loading of all fields meshes */
    AppComponent.fieldTypes.forEach(fieldType => {
      const promise = BABYLON.SceneLoader.LoadAssetContainerAsync(AppComponent.fieldsRootUrl,
        fieldType + '.' + this.fileType, this._battleFieldScene);
      meshPromises.push(promise);
      promise.then(container => {
        this.battleFieldMeshContainers.set(fieldType, container);
      });
    });

I’m loading meshes like that with idea to load all meshes and then to build battlefield with data which came from server(it can be different each time). UI can take meshes from map and clone them and put in right coordinates.

I wonder maybe manifest checks doesn’t combinable with async loading

Also I have question:
I found that babylon examples are using babylonbinarymeshdata format. Babylon docs doesn’t have any info about it. How can i get such binary format? Can babylon cache it? I assume no because it is binary like glb

Thanks

[Deltakosh]

Regarding Binary data, we provide a tool to produce them:
Exporters/Tools/ConvertToBinary at master · BabylonJS/Exporters (github.com)

But I would recommend using glb instead (more features like texture compression or mesh compression with DRACO are supported)

for your other question, I will need a repro in the PG to help

[TooCalm]

Unfortunately I can’t provide pg example because I don’t know any meshes that have manifest file available in PG

[sebavan]

You can follow this guide to store external assets for the playground Using External Assets In the Playground | Babylon.js Documentation

[TooCalm]

Hey guys,
Thanks to Sebavan’s link I’ve created PG
This is initial state with four different mesh cases:

Babylon.js Playground

test.gltf mesh exported to gltf. texture is inside it
test+tex.gltf mesh exported to gltf. texture as seperate file.
babylontest.babylon mesh exported to babylon. texture as a separate file
test+tex.glb mesh exported to gltf. texture as seperate file.

Now try different combinations of meshes order + offlinefactory code enabled/disabled. In example where all meshes and offline factory hack enabled second/third etc run of code will not trigger loading meshes from server in 100% tries but with disabled code everytime I run code i’m getting different results

first run all meshes is loaded from server, but version folder in db contains not all files

second run files was loaded from server except last! That what i mentioned in previous post.

Moreover GLB was loaded from DB! (I should mention that with disabled offline factory hack it’s always loaded from DB)

Unfortunately i can’t reproduce it everytime. I think it’s reproducable in 70% cases. Sometimes it’s loaded everything from DB but next run after successful is wrong
I think the issue here is asynch calls.

Could you please check?

[TooCalm]

Hey @Deltakosh, did you have a chance to check my PG above?

[Deltakosh]

nope let me see if @RaananW can help

[RaananW]

Seems like if you disable the manifest check - glTF Loader Demo | Babylon.js Playground (babylonjs.com) , caching works as expected.
I’m checking now what’s wrong with the manifest inspection

[RaananW]

Ok! did some investigating.
The Database code currently in use is expecting a single scene file to be loaded. as you are loading 4 consecutive files, a new database object is creating per request (not indexeddb, but the babylon Database class). But since the older ones are overwritten by the new created database object only the last one is actually being cached.
The architecture makes a lot of sense for a .babylon file, as it is an entire scene in a single file, but doesn’t work well if we want to load a few glTF files.

@Deltakosh - I believe this should be a part of the revamp. Another solution would be to make the scene.offlineProvider an array instead of a single file, but I am not very happy about this solution. My solution for now is to not use the manifest (setting engine.disableManifestCheck = true;) in a code that loads more than one file, and pay attention to that issue when modernizing the Database object.

[Yudong]

It works for me, but how to update the resources.
Update the manifest.json doesn’t work.

[Deltakosh]

Updating the manifest should suffice. Do you have a repro somewhere that I can look at?