The BabylonJS Scene Explorer and Inspector in realtime in 3D with two way event binding

Demos and projects
1

Hi! :vulcan_salute:
I am into debugging nowadays and had an idea. Can we render the Scene Explorer and the Inspector into a texture and use it in 3D and can we establish a two way event binding?

It seems we can! Please watch this 1 minute long introductory video :smiley:

Green cursor - my windows desktop cursor
Gray cursor - virtual cursor (when you hover over a 3d panel)
Glows - every element selected on the scene in the 3d panel have it’s bro in the HTML version and it gets highlited
You can see it chaning how I move between the HTML part and the scene just right over a panel.
And you can see my tiny debugger in the right upper corner doing it’s job very well :blush:

Guys, is this something cool or was I just wasting my time again? I’ve learned new stuff though. Is/was someone trying to render and two way bind HTML into the scene? I usually found out that someone did the stuff I try to do 5 years ago… :smiley: :smiley: :smiley:

Thanks for watching! Ideas are welcome!

R.

EDIT: added links in a post below

7 Likes
2

@roland it looks like black magic to me !!!

I wonder if we could leverage this in XR @RaananW ?

3 Likes
3

@roland

Totally unrelated btw.
You use vue right? I noticed this
GitHub - troisjs/trois: ✨ ThreeJS + VueJS 3 + ViteJS ⚡ . I have only played around with vue, but could we port that logic over to babylon? The trois scenes hot update like with react-babylon and r3f and code is way way simpler

Edit:
I did some sleuthing, i was wrong about createApp being the reconciler equivalent, its actually createRenderer , and the hot updates in trois come from defining an hmr key here:
.trois/Scene.ts at master · troisjs/trois · GitHub

An example implementation of the createRenderer api that i was confusing for createApp can be seen here vue-next/index.ts at master · vuejs/vue-next · GitHub , which is vue’s dom implementation.

This is actually even simpler than i thought. It might actually be better than r3f / react-babylon in a portability sense.

edit again:
Ok i didnt make things up originally, there is actually a reconciler using createRenderer.
.trois-renderer/index.ts at main · troisjs/trois-renderer · GitHub

Also thinking… maybe just implement hmr in babylon instead.

4

Maximize your browser. It is even better to go full screen. You can switch to full screen anytime later. When asked choose the screen with your browser:

image
image1919×1079 92.6 KB

https://demos.babylonjs.xyz/scene-explorer-3d/#/

Click Share and enjoy! :vulcan_salute:

Being a prototype only the click events are propagated to the HTML page. The screen offsets are hardcoded so they expect the exact setup you can see on the screen (no embeded mode) and should be run on a FULL HD screen. It runs on 4k as well but the texures are set to 1080px height so they will be cropped.

The textures in BabylonJS are grabbed from the media source and stored as RawTextures so cool stuff can be made. This way we can easily dispaly even the Developer tool windows and watch the console.log in realtime. Or you can transfer your Visual Studio code window and debug with your VR headset put on! Theoretically! :rofl:

Repo: GitHub - RolandCsibrei/babylonjs-3d-scene-explorer-and-inspector

IR BABOON!

EDIT:
The Baboon forgot to tell you that he had to put the whole /src folder into .eslintignore because of one missing type and he refused to find a better solution.

EDIT2: I tried to do the same using html2canvas so no permissions are required because it doesn’t capture the thing but it renders and it worked very good but it would need a PWA with aggressive caching because it downloads all the Scene Explorer and Inspector assets at every render.

1 Like
5

Hey @jeremy-coleman !
I am using Quasar to build my prototypes and it comes with Vue so that’s why but yeah I am a Vue fanboy! :smiley:

Yes, Troi seems to be portable Two cool ideas here, Troi itself and to port it :slight_smile:

6

Only to find all the @types was black magic but I’m still missing one so I did something unforgivable, I put /src into .eslintignore… Game over! :smiley:
However it can be compiled.

7

image

1 Like
8

This is me when it ran correctly for the first time

9

I wonder if you could rely on VideoTexture instead ?

10

I wonder too because I didn’t get to that page in the Documentation yet :smiley: :smiley: :smiley:

1 Like
11

Ok, I’ve checked it. The question here is whether we can connect this to it:

12

Yup through a videoElement in the page. basically create a videoElement set its source to the stream and use the element for your texture :slight_smile:

13

I knew, you have guys things for everything already! :smiley:
Thank you! I will try it!

EDIT: This way I could get rid of the missing ImageCapture type definition error! :vulcan_salute:

14

@sebavan, thanks for the tip with the VideoTexture, it is working! Actually all the frame grabbing code was trashed by this change :smiley: But I can’t get rid of this :frowning:
image

This is the with the original frame capturing code:
image

Any idea? Changing the sampling mode doesn’t help.
Thanks!
R.

15

try changing the wrap mode of the texture to clamp :slight_smile:

1 Like
16

image
image523×515 68 KB

Thank you! :smiley_cat:

2 Likes
17

I think I will give it a try next month :vulcan_salute: !