Gaussian Splatting in Babylon.js

Has anyone seen an implementation of G splats in Babylon?

For reference, here is a webGL viewer - https://gsplat.tech/

Luma.ai and Poly.cam have similar options on their websites.

Here is an implementation in three.js, although the online demo did not load for me - GitHub - mkkellogg/GaussianSplats3D: Three.js-based implementation of 3D Gaussian splatting

It would be very cool if you could use a g splat inside of a Babylon scene.

Thanks!

7 Likes

I’ve started an implementation some weeks ago. It was basically code from gsplat.tech with calls to Babylon api for mesh/instancing/shaders/… for an easier integration.
I still have some bugs so not great for a public release.

16 Likes

Ship it!!! :smiling_face::smiling_face:

Seriously though, excited to see any progress on that!

-Gabe

6 Likes

Super excited for this! Thanks @Cedric for giving this a shot!

1 Like

Why is Gaussian Splatting a big deal? I come across it very often on Twitter these days. I’m curious - is it practical to use?

1 Like

Technically it’s practical, there are several optimizations that can go in place like managing draw calls, i like to think of gaussian splatting as a form of advanced particle rendering, guassian being a single unit of particle can become complex through splitting or cloning and the info each guassian hold…blahblah lot of technical stuffs there but yeah gaussian splatting is kinda part of the future of 3D art I guess, was experimenting with it a couple days ago

3 Likes

Its an improved nerf impl that also has very realistic and somewhat efficient global illumination. It lets you transform a video or image to a virtual world, which obviously has many use cases.

4 Likes

hi @SonnyC56 - this implementation using three.js looks easier to follow to me, so may be easier to port.
dylanebert/gsplat.js: JavaScript Gaussian Splatting library. (github.com)

3 Likes

really excited to see the release!

Better late than never

There is a camera Y inversion I can’t find a fix for! :frowning: I’ll look again in the coming days.
Thanks @PirateJC for the capture!

I tried to keep the code as short as possible. Let me know if you find some bad math.

Happy hacking!!!

14 Likes

An update to the playground with proper camera and the ability to mix ‘classic’ scenes with geometry

19 Likes

Great work!

I would like to use your code in my application.
Could you provide the license for the code?

It’s now part of Babylon.js as an experimental feature

Things will change like API method names, parameters. be warned :slight_smile:

5 Likes

@Cedric is the best !!!

3 Likes

Hey guys!

Thanks for the amazing work. Would love to share that I just integrated it into Nowhere.io!

Here is a link to a room where you can meet and chat around the splat - Welcome to NOWHERE

I have not had enough time to do a deep dive yet, but I noticed that the performance increases dramatically and it looks much better when you switch to a different camera in our product. You can test this out by clicking the camera icon on the bottom right toolbar and entering either selfie or drone mode.

I am not sure what causes this discrepancy here, but the main (less performant) camera is a UniversalCamera, the selfie camera is an ArcRotateCamera and the drone camera is a UniversalCamera.

I can do a deeper dive next week, but some differences I quickly spotted that we do to the less performant main camera vs the drone one (both universal) is that the main camera has these settings applied:

    camera.fovMode = UniversalCamera.FOVMODE_VERTICAL_FIXED;
    camera.fov = Angle.FromDegrees(Client.PLAYER_FOV).radians();
    camera.minZ = 0.2;
    camera.maxZ = 50000;
    camera.inputs.removeByType('FreeCameraKeyboardMoveInput'); // Only allow free look
    camera.inputs.removeByType('FreeCameraTouchInput'); // We have our custom one
    camera.inputs.add(new TouchDeviceInput(camera));
    camera.inputs.removeByType('FreeCameraGamepadInput');
    Babylon.fixCursorDragOutWindow(camera);

Oh, and the main camera is a parent of the player mesh, while the selfie and drone cameras are children of the player mesh.

One thing that would be super nice to have would be the conversion of .ply to .splat so you can download right from luma and import into Babylon. I used this jsfiddle, Edit fiddle - JSFiddle - Code Playground, to do it, so it should be able to be handled locally by Babylon. Once converting with this tool I was able to load my own splats though.

The last thing to note was that the performance on my end was not as good as what is delivered on sites like luma or polycam, I am sure this will improve over time I just thought I would report my findings here.

Hopefully, this info helps you with development! Super excited about this, thanks again!! We will be doing a more complete implementation of splats in our platform at some point soon!!

4 Likes

Yes he is! :heart_eyes:

1 Like

Next in my small roadmap is to allow loading .ply and saving .splat.

5 Likes

I can’t run the PG, there is an error;

I’ve changed the method name. Try with LoadFileAsync

It works now - https://playground.babylonjs.com/#Y54RPQ#2

1 Like