Running scenes from comand line or Magic Leap (Exokit)


#1

I was / am looking at alternatives for creating WebXR enabled apps. It is my understanding that Magic leap has a browser as well as a port of Firefox that is WebXR capable, but I want an app.

Not to discuss all the options of doing this in this topic, but there is a nodeJS way, once you fork Node.JS to make it work on Magic Leap. The project ExoKit not only allows you to embed WebGL / WebXR scenes on Magic Leap, but also on a desktop.

In a series of edits, I am going to see what I can and cannot get to run on a Windows command line. If something cannot be gotten to run from the desktop (not WebXR related), I am going to assume it would also fail on a device.

I am going to do this in installments. This is to make the pieces all in the first post, not separated by any comments (welcomed). Much better for looking at in future.


Hello world / basic scene / STD Materials

scene: https://palmer-jc.github.io/scenes/QueuedInterpolation/mesh_parent/index.html

First thing to note, is any console log messages are written into the shell window. Exports from my Blender Javascript exporter have the option to be quite verbose. The command was

exokit https://palmer-jc.github.io/scenes/QueuedInterpolation/mesh_parent/index.html

The console log was:

Babylon.js engine (v3.3.0) launched                                                                                                                   
BJS - [15:23:43]: Queued Interpolation Timeline Control system initialized, version: 1.1.0 Beta                                                       
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Ground completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                            
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus.Bottom_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                 
BJS - [15:23:43]: defined mesh: Gus.Middle_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                 
BJS - [15:23:43]: defined mesh: Gus.Top_Button completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: defined mesh: Gus.Mouth completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                         
BJS - [15:23:43]: defined mesh: Gus.Eyes completed:  0.00, geometry:  0.00, skey:  0.00 secs                                                          
BJS - [15:23:43]: defined mesh: Gus completed:  0.02, geometry:  0.00, skey:  0.00 secs                                                               
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_1.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_1.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_1.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_1.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_1.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_1 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_2.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_2.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_2.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_2.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_2.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_2 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:43]: mesh_parent.defineMaterials completed:  0.00 secs                                                                                   
BJS - [15:23:43]: defined mesh: Gus_3.Bottom_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_3.Middle_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                      
BJS - [15:23:43]: defined mesh: Gus_3.Top_Button (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                         
BJS - [15:23:43]: defined mesh: Gus_3.Mouth (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                              
BJS - [15:23:43]: defined mesh: Gus_3.Eyes (cloned) completed:  0.00, geometry:  0.00, skey:  0.00 secs                                               
BJS - [15:23:43]: defined mesh: Gus_3 (cloned) completed:  0.01, geometry:  0.00, skey:  0.00 secs                                                    
BJS - [15:23:44]: Texture Load delay:  1.00 secs                                                                                                      
BJS - [15:23:44]: Unable to compile effect:                                                                                                           
BJS - [15:23:44]: Uniforms:  world, mBones, viewProjection, diffuseMatrix, lightData, depthValues, biasAndScale, morphTargetInfluences, diffuseSampler
BJS - [15:23:44]: Attributes:  position                                                                                                               
BJS - [15:23:44]: Error: 0(36) : error C1008: undefined variable "unique_fragColor"                                                                   

There seems to be a fragment shader compile error, for the ground mesh. If you look at the browser version from the link above, you will see that the ground is accepting shadows. It is generally accepted that shadows are not going to work on AR head sets, so it looks like we are still good.

There are also some command line switches to help with development. For this scene, uncap FPS & performance logging were used (-up). Here was about 10 seconds worth of data at full screen of a 30" (2560 x 1600) & a nVidia GTX 1050. Clearly GPU must be in use. My CPU could not keep up this rate at this resolution. Wish these columns were documented though:

59 FPS | 818ms idle | 1ms wait | 0ms prepare | 2ms events | 0ms media | 187ms user | 11ms submit  
59 FPS | 832ms idle | 1ms wait | 0ms prepare | 4ms events | 0ms media | 178ms user | 8ms submit   
59 FPS | 854ms idle | 2ms wait | 0ms prepare | 1ms events | 0ms media | 154ms user | 10ms submit  
59 FPS | 819ms idle | 0ms wait | 0ms prepare | 2ms events | 0ms media | 189ms user | 8ms submit   
59 FPS | 864ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 146ms user | 8ms submit   
59 FPS | 840ms idle | 0ms wait | 0ms prepare | 3ms events | 0ms media | 165ms user | 10ms submit  
59 FPS | 824ms idle | 0ms wait | 0ms prepare | 0ms events | 2ms media | 185ms user | 7ms submit   
58 FPS | 840ms idle | 0ms wait | 0ms prepare | 1ms events | 0ms media | 183ms user | 3ms submit   
63 FPS | 796ms idle | 2ms wait | 0ms prepare | 3ms events | 0ms media | 140ms user | 7ms submit   
59 FPS | 810ms idle | 0ms wait | 0ms prepare | 31ms events | 0ms media | 168ms user | 10ms submit 
66 FPS | 735ms idle | 3ms wait | 0ms prepare | 6ms events | 0ms media | 150ms user | 22ms submit  

WebGL Version

PBR / WebGL 2 / sky box scene.

Babylon.JS has supported both WebGL 1 & 2, since version 3.0. The 4.0 preview version writes to the console in the first line both the BJS version & the version of WebGL the “browser” reports. I am getting:
Babylon.js v4.0.0-alpha.21 - WebGL1. I do not know if this is causing any errors, but looks like it is being assumed that shaders are WebGL 1, and being transpiled to WebGL 2. I think ThreeJS is also WebGL2, so think this npm module make be in need of some TLC.


SkyBox & Environment Textures

When you have a scene with an environment texture, and your all your materials are PBR, then all your shaders fail with the following message. I think I have enough to open some issues on the repo.

BJS - [12:21:52]: Error: 0(2020) : error C7614: OpenGL/ES doesn't allow use of reserved word textureCubeLodEXT
0(2020) : error C7506: OpenGL does not define the global function textureCubeLodEXT

PBR Materials

Did get PBR shaders to compile & run, but without an environment texture. Also, the unique_fragColor shows up here too.


Web Audio


GUI


#2

I know that @trevordev is close to ExoKit team


#3

Yep, I really like what ExoKit is doing but its definitely a bit rough around the edges at the moment. They’ve been pretty helpful in the past when I’ve filed issues here: Issues · webmixedreality/exokit · GitHub


#4

Yes, I read through the list already. I also had an issue yesterday. I had version of Python on my system, which this node-gyp module needed to be older. I got things sorted (5 hours of cursing), but when it actually compiled, things weren’t right. This was due to not having a clean run right from the beginning.

They processed the issue in under an hour. They did not actually had to change anything, but the responses were like in 20 mins.

Edit: Just found your issue, linking it here all most as a note to myself. I left the vrdisplayactivate part out my status, but I had already seen it in docs.

All tried to run some of they’re examples. I never get the ‘running!’ console message, but I get a scene. Do not know anything about threeJS. This code never checks if supported. I’ll have to use a UniversalCamera when helper._supported is false.


#5

As an update, I have switched to preview version of BJS, coded a scene using WebXRHelper ( a universal camera as fallback), and put it with PBR materials, a skybox / environment texture, and shadows.

I do not think shadows for PBR are important, but wanted to see what would happen. It failed. See edited first post for more.

The first noticed was that webgl1 was being reported on the console. As the backend is really OpenGL 3.0 ES, thought that odd.

Searched through exokit issues, and found one for the enhancement to WebGL2, and commented. Got a response and a question of where BJS decides which webgl to use.

Looking closer at that the list of open issues, the deleteQuery check is tripping this up. I am guessing this test was chosen, since there is no WebGL extension. It could also be being used, but I did not find any beginQuery calls.

These seem to be the enumerated list of WebGL2 features still needed to be exposed:

Seems they are using their own native code for Magic Leap & GLFW for desktops. Probably need to solve these twice.

I am guessing in order to do environment textures, the 3D textures one absolutely needs to be done for PBR. Does PBR even make sense without an environment texture?

What about the others, are they in use in BJS?