Gaussian Splatting in Babylon.js

Completed Features
21

The correct method name is without the first capital letter.

loadFileAsync
1 Like
22

It seems like this is limited to very small files, is there any future updates that will allow for larger .ply and splats to be used?

It is also very non performant as soon as you move the camera, I was going to dive into the code but a few pointers on what to look at would be nice.

All and all great work! Just trying to use this in a more production like setting now and am having some issues but it is very promising.

Are there any plans for webGPU support with this?

23

We have no specific roadmap for now. As usual, we are driving our development from inputs and needs of the community.
There is a .ply import. Do you have use-cases youā€™d like to see addressed?

24

If we do any normal sized (100+ mbs) spat it seems to tank performance quite a bit. Also seems like when you move the camera even with the 10mb example the fps drops nearly in half.

Here is a link to some example splats.

https://file.io/s0v8rfcrSVhv

Pretty much all of these but the gardens scene will crash the webGL context. Playcanvas is able to render them but then it seems that some of the splats do not get their affine matrix applied and appear as blue dots, which is probably because the buffers are too large.

But sites like https://poly.cam/ can render them no problem and at a high fps. Just wondering if there is a way to get our to work with the larger files. Im assuming its because they are using a wasm solution.

Also we probably need webGPU support!

25

Hello @Cedric , greeeeeeat gaussian splatting.

When I tried to load a large dataset with 2.5 million splats, I found the usage of thin instances might be a performance bottleneck, I tried to remove the sorting worker completely (ignore the rendering result mistake), but the fps is still relatively low.

Maybe we can use a single mesh containing all splat quads? This may take larger memory but will have better performance?

Or do you have any other idea to improve the performance of thin instances? I would like to have a try.

Thanks~

P.S. Hereā€™s a threejs implementation with instancing geometry, very fast.

1 Like
26

I need to discuss that with @sebavan

28

Hi Cedric, I have found one of the reasons of performance issue and have an improved version.

The idea is to avoid update instance matrix (covA, covB, centers, colors) after sorting, so the instance buffer could be set to static (dynamic is slow).

Gaussian info (covA, covB, centers, colors) are used as sampler2Ds, and a single splatIndex attribute is used to represent sorted index which is dynamic. In my case, the fps of 2.5 million truck.splat could be raised from 15 to 60.

I have submit a PR here. Could you review it?@Cedric @sebavan

2 Likes
29

Sure! Can you please fix linting/formatting issues so the CI will produce a snapshot we can test?

30

Love the update !!!

34

cc @Cedric, but we may not support this file format.

35

Indeed, we donā€™t support that format.
Do you have some infos on its adoption/format/usage?

37

Can this be supported on XR? @Cedric you seem to know if its possible or even supported, i tried it on an AR scene trying to place a splat on a surface but it wouldnt render

38

I donā€™t see why it would not. can you repro in a PG so I can take a look?

39

Yes just give me a minute to change the splat for something more SFW

40

here you go, is quite straightforward but i tested it in android and ios chrome and safari and didnt render anything, maybe im doing something wrong in the marker replacement?

41

I got the Halo GS to display on top of the camera view on my Android device. Then it vanished after a few seconds and I couldnā€™t get it back.
Iā€™m not really familiar with XR but Im sure @RaananW has some good tips.

42

actually, none of the demos are working, in any of my devices, and i remember them working on my phone on chrome like a couple of weeks ago, maybe is a new unreported bug? i cant seem to do any type of hit testing demo or mesh placement demo. ( Even without the splat, just the regular demo in the documentation:

43

Nevermind, i just tested everything again and indeed it does work, must have been an issue with the environment i was testing it in, maybe it wasnt detecting any surfaces and hence not rendering anything.

but can confirm, it did load the Halo GS, performance wise was a bit hard but i expected that, i intend to use small splat anyways so yay! it works

Ill test in VR during the weekend also and report back if it works too

2 Likes