Loading Assets From Memory

Good afternoon, Babylon.js community!

I apologize in advance, I’m using auto-translate.

I wanted to ask a question about delivering resources to the end user, on the client.
I’ve organized the delivery of the model from my server and unpacked it with JSZip, unfortunately my hopes of rendering it simply didn’t come true :slight_smile:

I read a lot of documentation again, but apparently I don’t understand how to put the model ( it’s an object with files in memory ) on the scene(
I will attach everything that is required of me, I don’t even know what yet, except for the component - which acted as a test subject for me in this research ( I hope I won’t be stoned for the code - I was just trying to get results).

import {
    Scene,
    WebGPUEngine,
    Vector3,
    SceneLoader,
    FilesInputStore,
    AssetsManager,
} from '@babylonjs/core';
import Menu from '../root';
import { StaticCamera } from '../MenuProviders/Cameras/StaticCamera';
import { DevKit } from '../../DevelopEntity/DevKit';
import '@babylonjs/loaders/glTF';
import { PF_HemisphericLight } from '../MenuProviders/Light/HemisphericLight';

export class Main {
    /**
     * Движок.
     */
    private engine;
    /**
     * Канвас
     */
    private viewport;
    /**
     * Родитель
     */
    private root;
    /**
     * Ресурсы для построения сцены
     */
    private resources;

    /**
     * TODO: Надо описать
     * @param engine
     * @param viewport
     * @param root
     */
    constructor(
        engine: WebGPUEngine,
        viewport: HTMLCanvasElement,
        resources: any,
        root: Menu,
    ) {
        this.engine = engine;
        this.viewport = viewport;
        this.resources = resources;
        this.root = root;
    }

    scene() {
        const scene = new Scene(this.engine);

        new DevKit(
            { scene },
            {
                scene: scene,
                viewport: this.viewport,
                target: Vector3.Zero(),
            },
        );


        // CODE_CODE_CODE_CODE_CODE_CODE_CODE

        // console.log(this.resources.files); // тут файлы которые я распаковал с помощью JSZip
        // Подготовка файлов для загрузки
        const files = this.resources.files;
        const filesToLoad: { [key: string]: File } = {};

        let url = '';
        // Используем Object.keys для перебора всех файлов
        Object.keys(files).forEach((fileName) => {
            console.log(fileName);
            const file = files[fileName];
            const blob = new Blob([file], { type: 'application/octet-stream' });

            if(fileName.includes('.gltf')) {
                url = URL.createObjectURL(blob);
            } 
            const fileObj = new File([blob], file.name);
            filesToLoad[file.name] = fileObj;
        });

        const rootURL = url;
        const fileName = 'adamHead.gltf';

        SceneLoader.Append(rootURL, fileName, scene);

        // console.log(this.resources.files); // тут файлы которые я распаковал с помощью JSZip
        // // Подготовка файлов для загрузки
        // const files = this.resources.files;
        // const assetArrayBuffer = Tools.LoadFileAsync(files['adamHead.gltf'], true);
        // console.log(assetArrayBuffer);

        // Локальная мастер модель
        // Загружаем модель в сцену
        // const rootURL = 'adam/';
        // const fileName = 'adamHead.gltf';

        // SceneLoader.Append(rootURL, fileName, scene);

        // CODE_CODE_CODE_CODE_CODE_CODE

        /**
         * Камера
         */
        new StaticCamera(
            scene,
            'menu__main_camera',
            this.viewport,
            new Vector3(0, 0, -10),
        );

        /**
         * Свет
         */
        const mainLigth = new PF_HemisphericLight(
            'menu__main_spot_ligth',
            scene,
            new Vector3(-4.6, 4.1, -14.9)
        );
        mainLigth.setIntensity(0.2);
        /**
         * TODO: возможно стоит разобраться с clone() ради оптимизации
         */
        const mainLigth_Two = new PF_HemisphericLight(
            'menu__main_spot_ligth_two',
            scene,
            new Vector3(4.6, 4.1, -14.9)
        );
        mainLigth_Two.setIntensity(0.2);

        return scene;
    }
}

Thanks in advance to everyone who brought my problem to my attention!

Probably this section may help - Babylon.js docs

1 Like

Hi!

This works:

const data = await fetch(`${baseUrl}/${filename}`)
const blob = await data.blob()

if (blob) {
  // import from blob to scene
  const file = new File([blob], filename, { type: 'application/octet-stream' })
  const imported = await SceneLoader.ImportMeshAsync('', '', file, scene)
  return imported
}
1 Like

Sorry for the delayed response, I’ve tried this before(
But still, I’m always grateful for trying to help! Thank you!

I’m trying to do that, but while I’m trying, I wanted to clarify.
I have a .glTF model that is not a single file, but a folder that has resources for the model, I’m trying to figure out - how do I load them too ? When I did it through the public folder, directly from the client - then I understand how it is achieved, but over the network there are problems with understanding it.

update
So far all I get is this
SyntaxError: “[object Object]” is not valid JSON

but it’s probably my problems with the code base, I must have left something out - I’m trying to fix it)

Unfortunately, no results yet(
As soon as there is an answer to the question - I will definitely post it here!

Error at the time of writing - SyntaxError: Unexpected token ‘o’, “[object Obj”… is not valid JSON

code:

import {
    Scene,
    WebGPUEngine,
    Vector3,
    SceneLoader,
    FilesInputStore,
    AssetsManager,
    Tools,
} from '@babylonjs/core';
import Menu from '../root';
import { StaticCamera } from '../MenuProviders/Cameras/StaticCamera';
import { DevKit } from '../../DevelopEntity/DevKit';
import '@babylonjs/loaders/glTF';
import { PF_HemisphericLight } from '../MenuProviders/Light/HemisphericLight';

export class Main {
    /**
     * Движок.
     */
    private engine;
    /**
     * Канвас
     */
    private viewport;
    /**
     * Родитель
     */
    private root;
    /**
     * Ресурсы для построения сцены
     */
    private resources;

    /**
     * TODO: Надо описать
     * @param engine
     * @param viewport
     * @param root
     */
    constructor(
        engine: WebGPUEngine,
        viewport: HTMLCanvasElement,
        resources: any,
        root: Menu,
    ) {
        this.engine = engine;
        this.viewport = viewport;
        this.resources = resources;
        this.root = root;
    }

    async scene() {
        const scene = new Scene(this.engine);

        new DevKit(
            { scene },
            {
                scene: scene,
                viewport: this.viewport,
                target: Vector3.Zero(),
            },
        );


        // CODE_CODE_CODE_CODE_CODE_CODE_CODE

        // console.log(this.resources.files); // тут файлы которые я распаковал с помощью JSZip
        // Подготовка файлов для загрузки
        // const files = this.resources.files;
        // const filesToLoad: { [key: string]: File } = {};

        // let mainFile = null;
        // // Используем Object.keys для перебора всех файлов
        // Object.keys(files).forEach((fileName) => {
        //     let file: File | null = null;
        //     const resource = files[fileName];
        //     console.log(resource);
        //     const blob = new Blob([resource], { type: 'application/octet-stream' });
        //     console.log(blob);
        //     if(fileName.includes('.gltf')) {
        //         mainFile = new File([blob], fileName, { type: 'application/octet-stream' })
        //     } else {
        //         file = new File([blob], fileName, { type: 'application/octet-stream' })
        //     }

        //     if(file) {
        //         const fileObj = new File([blob], file.name);
        //         filesToLoad[file.name] = fileObj;
        //     }
        // });

        // if(mainFile) {
        //     // await SceneLoader.ImportMeshAsync('test', '', mainFile, scene, null, '.gltf');
        // }

        // const rootURL = url;
        // const fileName = 'adamHead.gltf';

        // SceneLoader.Append(rootURL, fileName, scene);

        // console.log(this.resources.files); // тут файлы которые я распаковал с помощью JSZip
        // // Подготовка файлов для загрузки
        // const files = this.resources.files;
        // const assetArrayBuffer = Tools.LoadFileAsync(files['adamHead.gltf'], true);
        // console.log(assetArrayBuffer);

        // Локальная мастер модель
        // Загружаем модель в сцену
        // const rootURL = 'adam/';
        // const fileName = 'adamHead.gltf';

        // SceneLoader.Append(rootURL, fileName, scene);

        // CODE_CODE_CODE_CODE_CODE_CODE


        console.log(this.resources.files);
        const resArray: any[] = [];
        Object.keys(this.resources.files).map(res => {
            resArray.push(this.resources.files[res]);
        });

        // console.log(resArray);

        const blob = new Blob(resArray);
        const file = new File([blob], 'adamHead.gltf', { type: 'application/octet-stream' })
        console.log(file);

        if(file) {
            await SceneLoader.ImportMeshAsync('', '', file, scene)
        }

        /**
         * Камера
         */
        new StaticCamera(
            scene,
            'menu__main_camera',
            this.viewport,
            new Vector3(0, 0, -10),
        );

        /**
         * Свет
         */
        const mainLigth = new PF_HemisphericLight(
            'menu__main_spot_ligth',
            scene,
            new Vector3(-4.6, 4.1, -14.9)
        );
        mainLigth.setIntensity(0.2);
        /**
         * TODO: возможно стоит разобраться с clone() ради оптимизации
         */
        const mainLigth_Two = new PF_HemisphericLight(
            'menu__main_spot_ligth_two',
            scene,
            new Vector3(4.6, 4.1, -14.9)
        );
        mainLigth_Two.setIntensity(0.2);

        return scene;
    }
}

I get resources this way:

 static LOAD_GENERAL_KIT(type: 'models', callback: Function) {
        // TODO: реализовать интерфейс доступа к indexedDB
        fetch(APID.kits.general[type].param(type), {
            method: 'GET',
            credentials: 'include',
        }).then((resp_zip) => {
            resp_zip.blob().then((models) => {
                if(models.size < 1) {
                    return null;
                }

                const zip = new JSZip();
                zip.loadAsync(models).then(des => {
                    callback(des);
                });
            });
        });
    }

have you done a console log on the argument past back from the promise here? what do you see when you log “des”

2 Likes


This is the output from JSZip, simpler is console.log(des) )

Now I can see what are you up to. You want to load the gtlf and get all the resources linked to it loaded automatically like if they were stored on a filesystem. This won’t work.

The simplest solution here would be to use glb (binary format) instead of gltf (text format). The glb file will contain every asset you would need to display the model correctly. You could then drop JSZip hence you’ll end up only with one glb file which you can easily load with the SceneLoader.

glb is the preferred format anyway.


Second option ( switch to glb instead :smiley: ) is to create an API endpoint for unziping a given zip file on the server. So you could call const rootUrl = await api.unzip('adam.zip'). The server unzips the zip to a directory and returns {rootUrl, sceneFilename}. You can then SceneLoader.AppendAsync(rootUrl, sceneFilename).

3 Likes

Yes as @roland says just use glb. Using 3rd party zip solutions is also not needed , all browsers support gzip natively, but you need to handle that on the server side, which is also pretty easy in todays world using modern build tools and packages. Unpacking is done automatically on the client side.

Sometimes there is a need to have textures as external, I do this in some of my apps, but I still use glb and I have coded a export pipeline in blender that creates all the assets and textures and a json file with all paths. So the app only cares to load the json and get the paths for the assets and textures at runtime. This way, I can completely change the model structure, amount of materials and textures and even where they are stored and I don’t have to change a single line of code in my app.

I don’t know much about what you are building and the expected changes over its lifetime but it’s always good to be flexible like this in some way or another… don’t hard code asset file names unless your scope is small or you are prototyping etc…

3 Likes

Thank you so much, for the great response! I will do some research in that direction.
I was thinking of creating an endpoint to retrieve the model, and its assets by reference to storage on my server. The problem is that I want to try to store the models in indexedDB, after the first load on the client)

I’ll take .glb on board!

1 Like

Thanks for the clarification! By my design, I’d like to save the assets in indexedDB, so I can’t take advantage of the link upload)

But I will try glb!
Thank you very much!

P.S. - I’ll be sure to note the solution ( help message from roland ), after I rewrite this part in my place, and give a comment. I will try to close the issue and the topic as soon as possible.

2 Likes

Basically you can use any data source you want until you can convert the data into a format supported by babylon.js. Let it be a mesh, texture, etc. File, url, blob, array view to mention some of them and until you are ok to load and assign the resources manually. (as @shaderbytes stated)

Since your zip file can be easily transformed to a glb and you can let the SceneLoader do all the dirty job I wouldn’t opt for a different solution.

1 Like

Sorry) Auto-translate twists the meaning of your post a bit, but I think after multiple readings I probably get the point.

I need to convert my resources from glTF to glb, and also send them in an archive, after that I can load them on the stage ?
This is what I am trying to do now, maybe I misunderstood you.
Please, if you have the ability and willingness to convey your thought in a language more accessible to a foreigner, I would be incredibly grateful!

So far I’m trying to do what I described above

What I suggest:

  1. import your gltf to Blender
  2. export as glb
  3. import the mesh using `SceneLoader``

I’ll gladly help you to create the glb file if you want and I can help you with creating a solution which will load the zip, save it in indexedDB and load the assets from there.

All I need is the zip file with your assets. You can send it to roland@babylonjs.xyz.

1 Like

I couldn’t have dreamed of such an offer! I am immensely grateful for something like this!!!
Nevertheless, I don’t want to waste your time, I will do my best to try to solve this situation myself, and I will send here the source code and the solution. I will try my best to solve my work stuff and close this issue today! So that if someone else has the same need - he can refer to this topic.
If after my attempts I fail, I will describe the problem in detail and send everything that is required of me.
Thank you very much again!

2 Likes

I don’t believe you are old enough to get this meme neverthless here it is:

The meme is based on the movie BRAVEHEART. The main character was fighting to the end never looking back.

You are brave to fight yourself through too! :muscle: Don’t forget, he was backed by fellow friends so if you start to lose steam do not hesitate to ask for help! We’re here to help you!

Yes, you could and you can!

If you’ll use IndexedDB I can recommend you to use Dexie.

Some recommendations based on your source code provided:

  1. Don’t use the Function type. Use callback: (data: JSZip) => void
  2. Use async/await instead of then
  3. You can shorten your code from:
export class Main {
    /**
     * Движок.
     */
    private engine;
    /**
     * Канвас
     */
    private viewport;
    /**
     * Родитель
     */
    private root;
    /**
     * Ресурсы для построения сцены
     */
    private resources;

    /**
     * TODO: Надо описать
     * @param engine
     * @param viewport
     * @param root
     */
    constructor(
        engine: WebGPUEngine,
        viewport: HTMLCanvasElement,
        resources: any,
        root: Menu,
    ) {
        this.engine = engine;
        this.viewport = viewport;
        this.resources = resources;
        this.root = root;
    }
}

to this (prefix your private class member with a _ for better readibility - this is optional):

export class Main {
    constructor(
        private _engine: WebGPUEngine,
        private _viewport: HTMLCanvasElement,
        private _resources: any,
        private _root: Menu,
    ) {}
}
  1. Don’t use any if you can type your parameters precisely. It’s never worth to be lazy to create an interface or type or lookup the existing one :smiley: For example:
interface Resources {
  files: string[]
}
...
   private _resources: Resources;
...

Good luck and keep us informed on your progress!

1 Like

After 3 days, I was able to figure out how to draw a model sticking to the strategy I had in mind!
This is the free model I used in my research in glb format!
Such a small detail - made my day, I am insanely excited and ready to move forward! :blush:


@labris @shaderbytes @roland

My fervent thanks to @labris @shaderbytes, thank you so much for helping me along the way, and a special thanks to @roland, I have no words to convey all the appreciation for his support and help, thank you!

As promised I will attach an explanation, I will note in advance that there are experts here who can give more specific and better recommendations, I will personally change the structure myself according to the message from @roland. I just hope you can use this as a reference:

Server:
I use Nest.js in my project, but the main thing is that I just give the file to glb.
I’ve stopped using JSZip for now, but maybe I’ll come back to it later:

@Controller('kit')
export class FileController {
  @Get('main_models')
  @Header('Content-Type', 'application/zip, application/octet-stream')
  getFile(
    @Param() param,
    @Session() session: FastifySession<PFSessionType>,
  ): StreamableFile {
    const nakama_session: NakamaSession = session.get('game') as any;
    if (!nakama_session && !nakama_session?.token) {
      Logger.log('not autorized');
      return null;
    }

    Logger.log(param);
    Logger.log('load files');

    const file = createReadStream(
      join(
        process.cwd(),
        PathToResources.general.models + 'test_healthshot.glb',
      ),
    );
    return new StreamableFile(file);
  }
}

The directory structure on the server looks like this:

But the best part will be on the client and I’m amazed at how easy it is to use Babylon.js, ( I guess I was trying to complicate what people smarter than me were doing :smile: )

This is a request to retrieve a file:

static LOAD_GENERAL_KIT(type: 'models', callback: Function) {
        // TODO: реализовать интерфейс доступа к indexedDB
        fetch(APID.kits.general[type].param(type), {
            method: 'GET',
            credentials: 'include',
        }).then((response) => {
            response.blob().then((resource) => {
                callback(resource);
            });
        });
    }

In this case, I haven’t managed to implement storage in iDB yet, but if you’re interested, I can post the code that works with it after a short time, but the topic of discussion was originally different and I can’t wait to share this victory, forgive me))) ( Dexie is already in my package.json )

And, when you set up resource delivery correctly, Babylon will render what you give it very kindly.

import {
    Scene,
    WebGPUEngine,
    Vector3,
    SceneLoader,
} from '@babylonjs/core';
import Menu from '../root';
import { StaticCamera } from '../MenuProviders/Cameras/StaticCamera';
import { DevKit } from '../../DevelopEntity/DevKit';
import '@babylonjs/loaders/glTF';
import { PF_HemisphericLight } from '../MenuProviders/Light/HemisphericLight';

export class Main {
    /**
     * Движок.
     */
    private engine;
    /**
     * Канвас
     */
    private viewport;
    /**
     * Родитель
     */
    private root;
    /**
     * Ресурсы для построения сцены
     */
    private resources;

    /**
     * TODO: Надо описать
     * @param engine
     * @param viewport
     * @param root
     */
    constructor(
        engine: WebGPUEngine,
        viewport: HTMLCanvasElement,
        resources: any,
        root: Menu,
    ) {
        this.engine = engine;
        this.viewport = viewport;
        this.resources = resources;
        this.root = root;
    }

    async scene() {
        const scene = new Scene(this.engine);

        new DevKit(
            { scene },
            {
                scene: scene,
                viewport: this.viewport,
                target: Vector3.Zero(),
            },
        );

        const file = new File([this.resources], 'master_model.glb', { type: 'application/octet-stream' })
        await SceneLoader.ImportMeshAsync('', '', file, scene);

        /**
         * Камера
         */
        new StaticCamera(
            scene,
            'menu__main_camera',
            this.viewport,
            new Vector3(0, 0, -10),
        );

        /**
         * Свет
         */
        const mainLigth = new PF_HemisphericLight(
            'menu__main_spot_ligth',
            scene,
            new Vector3(-4.6, 4.1, -14.9),
        );
        mainLigth.setIntensity(0.2);
        /**
         * TODO: возможно стоит разобраться с clone() ради оптимизации
         */
        const mainLigth_Two = new PF_HemisphericLight(
            'menu__main_spot_ligth_two',
            scene,
            new Vector3(4.6, 4.1, -14.9),
        );
        mainLigth_Two.setIntensity(0.2);

        return scene;
    }
}

Once again, I thank those people who have helped me gratuitously in achieving this result! I hope I can help someone else as much as you do!

3 Likes

I’m glad youe made it :slight_smile:
Nest.js is cool, I’m writing a new controller in it just right now :slight_smile:

Have a nice day bro!

2 Likes