Running scenes from comand line or Magic Leap (Exokit)

Demos and projects
1

I was / am looking at alternatives for creating WebXR enabled apps. It is my understanding that Magic leap has a browser as well as a port of Firefox that is WebXR capable, but I want an app.

Not to discuss all the options of doing this in this topic, but there is a nodeJS way, once you fork Node.JS to make it work on Magic Leap. The project ExoKit not only allows you to embed WebGL / WebXR scenes on Magic Leap, but also on a desktop.

In a series of edits, I am going to see what I can and cannot get to run on a Windows command line. If something cannot be gotten to run from the desktop (not WebXR related), I am going to assume it would also fail on a device.

I am going to do this in installments. This is to make the pieces all in the first post, not separated by any comments (welcomed). Much better for looking at in future.

Hello world / basic scene / STD Materials

scene: https://palmer-jc.github.io/scenes/QueuedInterpolation/mesh_parent/index.html

exokit
exokit.JPG749×751 36.2 KB

First thing to note, is any console log messages are written into the shell window. Exports from my Blender Javascript exporter have the option to be quite verbose. The command was

exokit https://palmer-jc.github.io/scenes/QueuedInterpolation/mesh_parent/index.html

The console log was:

Babylon.js engine (v3.3.0) launched                                                                                                                   
BJS - [15:23:43]: Queued Interpolation Timeline Control system initialized, version: 1.1.0 Beta                                                       
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Ground completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                            
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus.Bottom_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                 
BJS - [15:23:43]: defined mesh: Gus.Middle_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                 
BJS - [15:23:43]: defined mesh: Gus.Top_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: defined mesh: Gus.Mouth completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                         
BJS - [15:23:43]: defined mesh: Gus.Eyes completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                          
BJS - [15:23:43]: defined mesh: Gus completed:  0.02, geometry:  0.00, skey:  0.00 secs                                                               
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_1.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_1.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_1.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_1.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_1.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_1 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_2.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_2.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_2.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_2.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_2.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_2 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_3.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_3.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_3.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_3.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_3.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_3 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:44]: Texture Load delay:  1.00 secs                                                                                                      
BJS - [15:23:44]: Unable to compile effect:                                                                                                           
BJS - [15:23:44]: Uniforms:  world, mBones, viewProjection, diffuseMatrix, lightData, depthValues, biasAndScale, morphTargetInfluences, diffuseSampler
BJS - [15:23:44]: Attributes:  position                                                                                                               
BJS - [15:23:44]: Error: 0(36) : error C1008: undefined variable "unique_fragColor"

There seems to be a fragment shader compile error, for the ground mesh. If you look at the browser version from the link above, you will see that the ground is accepting shadows. It is generally accepted that shadows are not going to work on AR head sets, so it looks like we are still good.

There are also some command line switches to help with development. For this scene, uncap FPS & performance logging were used (-up). Here was about 10 seconds worth of data at full screen of a 30" (2560 x 1600) & a nVidia GTX 1050. Clearly GPU must be in use. My CPU could not keep up this rate at this resolution. Wish these columns were documented though:

59 FPS | 818ms idle | 1ms wait | 0ms prepare | 2ms events | 0ms media | 187ms user | 11ms submit  
59 FPS | 832ms idle | 1ms wait | 0ms prepare | 4ms events | 0ms media | 178ms user | 8ms submit   
59 FPS | 854ms idle | 2ms wait | 0ms prepare | 1ms events | 0ms media | 154ms user | 10ms submit  
59 FPS | 819ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 189ms user | 8ms submit   
59 FPS | 864ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 146ms user | 8ms submit   
59 FPS | 840ms idle | 0ms wait | 0ms prepare | 3ms events | 0ms media | 165ms user | 10ms submit  
59 FPS | 824ms idle | 0ms wait | 0ms prepare | 0ms events | 2ms media | 185ms user | 7ms submit   
58 FPS | 840ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 183ms user | 3ms submit   
63 FPS | 796ms idle | 2ms wait | 0ms prepare | 3ms events | 0ms media | 140ms user | 7ms submit   
59 FPS | 810ms idle | 0ms wait | 0ms prepare | 31ms events | 0ms media | 168ms user | 10ms submit 
66 FPS | 735ms idle | 3ms wait | 0ms prepare | 6ms events | 0ms media | 150ms user | 22ms submit

WebGL Version

PBR / WebGL 2 / sky box scene.

Babylon.JS has supported both WebGL 1 & 2, since version 3.0. The 4.0 preview version writes to the console in the first line both the BJS version & the version of WebGL the “browser” reports. I am getting:
Babylon.js v4.0.0-alpha.21 - WebGL1. I do not know if this is causing any errors, but looks like it is being assumed that shaders are WebGL 1, and being transpiled to WebGL 2. I think ThreeJS is also WebGL2, so think this npm module make be in need of some TLC.

SkyBox & Environment Textures

When you have a scene with an environment texture, and your all your materials are PBR, then all your shaders fail with the following message. I think I have enough to open some issues on the repo.

BJS - [12:21:52]: Error: 0(2020) : error C7614: OpenGL/ES doesn't allow use of reserved word textureCubeLodEXT
0(2020) : error C7506: OpenGL does not define the global function textureCubeLodEXT

PBR Materials

Did get PBR shaders to compile & run, but without an environment texture. Also, the unique_fragColor shows up here too.

Web Audio

GUI

scene: https://www.babylonjs-playground.com/frame.htm#WZZDNR#7

It seems fonts are very grainy & button labels are not even shown. It is still using webgl1. Not sure if that has anything to do with this though. No errors encountered.

exokit%20fonts
exokit fonts.JPG1282×1056 124 KB

If this is not solve-able, I could pull out my old Dialog code. There fonts are actual meshes. It is a little grainier in exokit, but it works.

scene: https://palmer-jc.github.io/scenes/dialog/index.html

exokit%20dialog
exokit dialog.JPG1282×1056 91.3 KB

2 Likes
2

I know that @trevordev is close to ExoKit team

3

Yep, I really like what ExoKit is doing but its definitely a bit rough around the edges at the moment. They’ve been pretty helpful in the past when I’ve filed issues here: Issues · webmixedreality/exokit · GitHub

4

Yes, I read through the list already. I also had an issue yesterday. I had version of Python on my system, which this node-gyp module needed to be older. I got things sorted (5 hours of cursing), but when it actually compiled, things weren’t right. This was due to not having a clean run right from the beginning.

They processed the issue in under an hour. They did not actually had to change anything, but the responses were like in 20 mins.

Edit: Just found your issue, linking it here all most as a note to myself. I left the vrdisplayactivate part out my status, but I had already seen it in docs.

All tried to run some of they’re examples. I never get the ‘running!’ console message, but I get a scene. Do not know anything about threeJS. This code never checks if supported. I’ll have to use a UniversalCamera when helper._supported is false.

5

As an update, I have switched to preview version of BJS, coded a scene using WebXRHelper ( a universal camera as fallback), and put it with PBR materials, a skybox / environment texture, and shadows.

I do not think shadows for PBR are important, but wanted to see what would happen. It failed. See edited first post for more.

The first noticed was that webgl1 was being reported on the console. As the backend is really OpenGL 3.0 ES, thought that odd.

Searched through exokit issues, and found one for the enhancement to WebGL2, and commented. Got a response and a question of where BJS decides which webgl to use.

Looking closer at that the list of open issues, the deleteQuery check is tripping this up. I am guessing this test was chosen, since there is no WebGL extension. It could also be being used, but I did not find any beginQuery calls.

These seem to be the enumerated list of WebGL2 features still needed to be exposed:

Seems they are using their own native code for Magic Leap & GLFW for desktops. Probably need to solve these twice.

I am guessing in order to do environment textures, the 3D textures one absolutely needs to be done for PBR. Does PBR even make sense without an environment texture?

What about the others, are they in use in BJS?

6

I updated the GUI section. Anyone describe what an actual button is, so I might file a issue?

1 Like
7

I have not filed an issue for GUI yet. What has been bothering me is how the Dialog system scene was also grainy. Especially when the Use System Camera is not clicked, when even the boxes are sketchy. There are no fonts involved.

I tried to check the hardwareScalingLevel. It is the same as when running from a browser. That is 1. I changed it to both 0.5 & 2, but no visual change.

Is there anything else, in the api or gl which can change the quality of render?

8

Could aliasing be causing that issue? You could try using the anti-aliasing options found in this pg: https://playground.babylonjs.com/#Y3C0HQ#146

9

Good try, but almost made it worse. Think i am going to have to let this issue go for now.

10

Well exokit is now running WebGL2 (after a bug fix PR) & BJS is now using it once Query was supported (not yet in master).

If I rerun my tests, I am now getting no errors. I am now getting shadows & PBR is now working. Performance is much improved from earlier, though I am not convinced this is due to WebGL2. OpenGL 3.0 ES always being used regardless of whether WebGL1 shaders are being translated or not.

Though the translation could be deficient, or “native” features of 2.0 could be in the shader from BJS which are not in WebGL1 code, or both. I do not have static system to compare against as there has been a performance refactoring of the render process in the interim. (The kind of high class problem that I can live with.

I am glad I snapped some of the performance logging above from before for the STD materials test. Basically before, the bottom line was about 59 FPS. Here is what I get now for the exact same scene (running BJS 3.3):

362 FPS | 5ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 57ms user | 213ms submit
360 FPS | 1ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 68ms user | 207ms submit
356 FPS | 1ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 63ms user | 216ms submit
345 FPS | 2ms idle | 0ms wait | 0ms prepare | 3ms events | 0ms media | 58ms user | 227ms submit
351 FPS | 2ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 70ms user | 211ms submit
357 FPS | 4ms idle | 0ms wait | 0ms prepare | 0ms events | 0ms media | 66ms user | 210ms submit
360 FPS | 2ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 67ms user | 208ms submit
355 FPS | 2ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 60ms user | 219ms submit
358 FPS | 3ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 63ms user | 211ms submit
362 FPS | 1ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 59ms user | 214ms submit
358 FPS | 2ms idle | 0ms wait | 0ms prepare | 0ms events | 0ms media | 62ms user | 215ms submit
3

I am pretty really sure the FPS is a not a real number, but take it that many more meshes could be added to the scene with no “real” drop off. Wish I had more detail on the meaning of the columns.

Now the question part: What additional WebGL2 features are the most valuable to add in addition? Adding Query was filled with many “thrills & spills”, but I now know much more about how to wrapper these calls. I wish to avoid my obsession for completion & just do everything without regard for what is being used or planned.

11

The more important are documented here:
https://doc.babylonjs.com/features/webgl2