I need to do a different kind of asset loading for a react native application. What I propose is adding extensibility to BabylonJS to allow developers to add their own or augment existing asset loaders.
In current state, I would need to hoist the FileTools static convenience methods. What I would like to do is add extension (open closed) to allow developers to substitute. It seems at least for where DOM Image is used to load assets that Scene is available everywhere (except for 2D Image) and we could make scene an optional constructor param perhaps. So, my proposal is to add to SceneOptions an optional IAssetLoader to override asset loading.
I havenāt fully investigated, but I think IAssetLoader could basically follow FileTools method signatures, unless you are ready to switch to Promises (all the native code uses async/await, so I need to wrap as sync). Iāve seen how videos and sounds are loaded and that will require some extra thinking. I did some tests on the weekend getting images loaded to textures from local app disk (ie: bundled assets/CameraRoll) using react-native utilities.
I realize we have overrides to load ādata:ā¦ā and I can work around and add custom code. I want to add this though not for a custom application, but for a library - otherwise I could just hard-code things and use existing static methods like āTexture.LoadFromDataStringā, whereas I would like to use new Texture(require('./localAsset/file.jpg'), ...) etc.
So, just want to start a discussion to see if this would be a welcome addition before working on a PR and also interested to learn more how BabylonJS Native project intends to allow Textures/Sounds/Videos to load from native filesystem/camera/etc. in case I have missed out the proper way to integrate native files with BabylonJS.
Our current plan for loading files is to override BABYLON.WebRequest. Accessing local files via file:/// syntax and access app package files using a TBD url scheme (e.g. app:///).
As for React Native, we are already actively working on this integration.
Once all of this is done, all file access behaves like normal (i.e. FileTools will work as is without modification). The only difference is use file:/// to access local files by absolute or relative path and app:/// or something similar to access app package files.
@bghgary that does help and glad to hear it is being actively worked on. The main site lists react-native as not supported or implemented. Are you using the GLView from expo-gl or something else for canvas? I have similar questions related to how to move pan and gesture events as BabylonJS input. If you can share a branch or anything would be appreciated. Thanks.
@bghgary It feels like we have gone down 2 different paths and I still need the customization from the original comment
Right now I am using the WebGLRenderingContext provided by GLView. From expo site: āGLView that acts as OpenGL ES render target and gives GL context object implementing WebGL 2.0 specificationā. If I want to continue going that way then I would still need the extensibility from my original comment - although I think there are a lot of edge cases that I will have issues with. I am just beginning my investigation recently, but that component looks standard for even for non-Expo react-native projects.
I read the Inputs article thanks for the link. I added the initial support for 2 of the VR devices and did some work on the GUI keyboard, so have already worked with the existing code somewhat. I am really interested to see where what you are building goes - the Profile part looks like what the spec for webxr does. I just want to send through events like the PEP polyfill - nothing with other devices currently and from that article was not possible to see if that could be done? In other words, I donāt need so much another level of abstraction as a way that I can send through pan/gesture events directly. If you look at the existing camera inputs they are āattachedā instead of being sent pointer events.
I feel like in both cases they are missing inversion of control to allow the extensibility that I am discussing. I want to intercept asset loading (similar to existing CorsBehavior) and flow through pointer events using my library of choice.
Regarding customizability, have you checked out the āpluginā architecture we introduced to the project a few weeks ago? With that architecture, virtually all functionality is split into modular plugin-style components that can be easily swapped out and customized independently. For example, Babylon Nativeās provided Console polyfill isnāt always necessary, and in some cases itās harmful (React Native provides its own of these already), so itās very easy to include/exclude or to swap out with something else.
The kind of customization I think youāre talking about, which I believe boils down to polyfilling new capabilities in addition to the ones Babylon Native has already targeted, should be especially easy to add. All you should have to do is make a new folder following the pattern in the Plugins directory, make sure that subdirectory is added at the correct point during CMake configuration so that the names of its dependencies are defined (for something that pretty much only depends on JsRuntime and napi, you could just add it to the Plugins directory itself), then link to it and initialize it within your consuming app to make the new types and behaviors available in JavaScript. Will that work for the customization you have in mind?
(Note: The plugins change is still new, and not everything has been adapted to use the new paradigm correctly yet. Please excuse the plugins with weird initialization patterns [like Console] and things like that; those should be fixed soon.)
As for whether or not your customization belongs in Babylon Native itself, I think thatāll depend on how broadly applicable it is. If you decide to go down the plugin route, then since plugins are quite modular and can be pretty easily transplanted within the file structure, I donāt think it matters much where you put it at first. You could just put it in the Plugins folder directly for simplicityās sake, and then it could be pretty easily moved elsewhere later (even to a separate project) if that turns out to be the right thing to do.
So, in sum, our new plugin-based architecture is specifically designed to make customization, especially of the sort that I think youāre talking about, as easy as possible. We havenāt had a chance to implement or write down all the patterns weāve been thinking about on that front yet, but let me know if youād like to discuss further!
@syntheticmagus the work you have done looks very promising. I want to extend BabylonJS to react-native cross platform with javascript only and not through the low level calls. It now makes sense how you are overriding the Request, while I am hoping to extend with purely javascript (and even non-native). I havenāt done any c++ for a while, so I think Iāll just sit back for a bit and see where this goes. I have written a lot of hardware device drivers, so can see what you are building, but I want to stick for this project with javascript, so I can easily integrate and swap out existing NPM libraries - like the react-native Pan/Gesture and asset libraries.
Thought I would round this topic with my assumptions, questions. I had already assumed that a default path, or explicit file url would grab from somewhere in the app structure. Am also assuming that you could grab files whether they be assets, or say a 3kb JSON file with the directions to perform something using http or https. Anything but a .js file, right?
If this is running on the Android platform, there are 3 types of storage App, Data, & Cache. Data can be deleted with the CLEAR DATA button. While this also deletes Cache, Cache also has a separate CLEAR CACHE button.
I would hope that any export / textures / JS files which are bundled could be placed in the App area of the distribution, so that they could not be deleted. What url pattern would be used to get files stored in each of the 3 types?
If the answer to my from http assumption is Yes, is there a way or could there be to put the stuff into storage? Actually, even stuff generated like preferences would be great to store. Is it possible to make a plug-in which puts it somewhere, preferably in Data?
@JCPalmer for react-native āexpo-assetā takes care of that. You can require('./asset/texture/file.jpg') files that are deployed with your app and access it directly, but for BabylonJS you currently need to take that asset (download and resolve it) and only then was I able to get it to load with Texture.LoadFromDataString(). It is that part of the process that I want to hijack. I did not do a lot of testing, but did run into some webgl errors about origin when applying textures.
On the other hand any external files like new Texture('https://playground..../', ...) load automatically. BabylonJS creates a DOM image attaches a src and corsBehavior and then when loaded attaches to the texture. That does work right now in Expo with no changes. From what I understand the BabylonNative project does not have DOM access and will override WebRequest behavior instead and use their own scheme convention.
The device OS itself does the caching, so what happens is the asset itself knows if it is downloaded already and will check the device cache (to avoid multiple downloads and see if cache is cleared), but the babylonjs engine also maintains a texture cache.
I am still in learning mode, so not 100% positive about all those comments.
Hey @brianzinn, you can take a look at what I did here, few months ago.
Unfotunatelly I lost the main project, but atleast I learned a lot about RN and babylonjs too
This is still on my todo list, and Iām planning a 2.0 version on which Iām already working on my sparetime.
It has import for textures and gltf files and interaction for rotation and zoom. Maybe it is not the best and the clearest code,ā¦ Iām sorry for this Cheers!
Thatās great to see others interested @MarianG !! I see you ended up basically with the same issues as me. You needed to resort to static methods for loading app textures and had to hand-code Device inputs over to specific camera properties.
What I would prefer is an inversion of control where we can say that a pointer event occurred and publish an event Without coupling. For loading assets to have ability to handle asset (URL) loading requests ourselves instead of static hard coded in framework Tools.
Those two changes would allow me to integrate native easier. Iām not writing the code for my own app, but instead trying to make it easier for developers to integrate BabylonJS and also declaratively when possible - otherwise I could follow some of your code patterns. I am interested to work with you on your v2 and maybe we can share progress and tricks. I have an iPhone and Pixel for testing.
I added custom props handling yesterday, so maybe can get local assets loading, but itās a hack. Would like to work towards a clean native integration I need to do more testing!!
One of the best parts of a system like Babylon is the ability to write UI with modern webdev frameworks. Which are arguably the most efficient ways to write extensive UI.
I really think there is a lot of value in either embedding a virtual dom based ui system compatible with modern MVC frameworks, or having a Babylon plugin for react native.
@waverider404 Iām not totally sure you can use G++ to compile. Perhaps @Cedric will know. Iām not a big fan of Xcode either, but it doesnāt crash for me.
Just researched online and results say itās possible
Please note that you also need to install Clang if you want to create native executables with Swift. Clang is not needed to run the swift command interactively or to run a .swift file. Interestingly, installing GCC (including g++) and creating symlink clang++ to g++ does allow swiftrc to build an executable"
Please excuse the plugins with weird initialization patterns [like 
We havenāt had a chance to implement or write down all the patterns weāve been thinking about on that front yet, but let me know if youād like to discuss further!
Cheers!