The Future of GS in BBL

[yamaciller]

Hi,

as a more broader question, what is the plan for Gaussian Splatting in Babylon for the (close) future.

We have been developing a prototype of a project for quite a while now with BBL.
Just recently we discovered (when implementing low resolution GSs), that the implementation of GS is decently different to other viewers and (atm) lags of visual quality (I guess its the compression?)
Is there a motivation to further develop the GS implementation and also bring in/improve different features?

Thx Werner We love BBL (btw) :).

[Cedric]

Yes, GS impl will get more work and features in the coming weeks/months.

Do you have a specific quality improvement in mind?

[yamaciller]

Sound great, cedric.
Atm we are confronted to get better visual quality from the splats using a consistent workflow from creating to presenting. ( I think I already flagged that, and I got some replies in a different thread).( GS color/detail differences in different viewers - #5 by Cedric )

If there would be options regarding the compression (even the .ply files look kind of lossy) and the splat formats in re of filesize and perfomance.
We were working with .spz which gave good filesizes and decent visuals when using high amount of splats (which counters the filesize and FPS of course).
SOG would be great as it seems the best option atm quality-size-performance.

In fact we discovered that visual quality issue, while we are now versioning (desktop/mobile) and using higres/lowres/very lowres versions of the GS data.

The compression/quantization seems to show up pretty heavily in a low amount splat, which doesnt seem to be a big issue with a lot of data (which of course is controversial to the data/filesizes).

Which plans do you have?
We are serioulsy considering other alternatives for that particular project (which is a pitty as we have put in quite a lot of time into BBL and this as a core demand would be a main decider).

In fact, we are using more than just GS features in the project, so we put in work in developing all of the other features. (custom shader et.al), so we would really want to stick to BBL.

Is there a particular GS roadmap, etas etc. I could plan onto?

We are basically starting production right now and have a 2 months production phase, for the delivery in December.

If there is a promising outlook onto the improvements for the GS, we could just keep going with the project and swap then the GSs with the improved/different formats (e.g.).

Best. Werner

[Cedric]

Regarding production pipeline, if lossless .PLY doesn’t meet your quality requirements, I bet .SPZ will not either as it’s PLY + quantization.

We’ve used SPZ in production (with SH) and it looked really good. Also because the capture was very, very high quality.

I still plan to do SOGS asap, we can discuss that.

In the mean time, if you have .PLY/.SPZ you want to share, we can discuss their rendering quality.

Except SOGS are there other features you need?

[yamaciller]

@Cedric

Regarding production pipeline, if lossless .PLY doesn’t meet your quality requirements, I bet .SPZ will not either as it’s PLY + quantization.

Well the first visual differences are obvious, but maybe we do something very wrong>

Thats an uncompressed .ply file (screenshot).

@Cedric

We’ve used SPZ in production (with SH) and it looked really good. Also because the capture was very, very high quality.

Same experience. Not too much visual differences at high GS count.

@Cedric

In the mean time, if you have .PLY/.SPZ you want to share, we can discuss their rendering quality.

I sent u an PM!

@Cedric

Except SOGS are there other features you need?

SOG was a lucky try. It meets the same visuals as ply or spz (in other than BBL) but filesize is smaller than ply. So that is a good alternative to spz. we would evaluate as long as there are no (major) visual differences.

Low res (100k GS) visual comparison

Hig res (3M GS) visual comparison

So basically the core of the issue is the visual difference. If thats out of the way, then we can think of spz or sog or ply (formats for different versions).

It would be great to find the source of that and why it is so different.

[Cedric]

How do you compare quality? I don’t have confidence in my eyes when it’s a question of rendering quality and visual fidelity.

[yamaciller]

Well thats a good point.

I do trust mainly my eyes and I am confident to say, that there is something, lets say not equal in the (visual) comaprison above.

The data set is the same and there are 3 (as supersplat using PC) different visualizer. 2(3) look the same one is obviously different.

So why is that, where is the source of that diffference coming from, and how and if it is fixable.

I take the first dataset ( this is the set which comes out of Postshot) as the reference.

I even did a test with brush, and it looks basically as the postshot output (it cannot be compared when i would compared it from further back - the colmap data aka sparse point cloud and cameras as the algorithm for generating the splat has different parameters).

But even in brush if I import the ply while …. > look at the screenshot, its the same as the others.

We suspect that there must be a difference in how the splats are decoded?
Is it the color? Is there some different compression happening? Rotations or SHs?

You do know more about that I assume, and we wanna find that difference and see if that can be fixed

[yamaciller]

… and another viz here. (for vis comparison)
(source> GitHub - kishimisu/Gaussian-Splatting-WebGL: 3D Gaussian Splatting Renderer for WebGL )

[Cedric]

Looks like SOG is replacing SOGS : PlayCanvas Open Sources SOG: The WebP of Gaussian Splatting | PlayCanvas Blog

[Cedric]

Update to add SOGS/SOG should take a few days. I’ll keep you updated.

[yamaciller]

Wow too cool, Cedric.
Actually unconcioulsy I meant the SOG not the SOGS (lol about the naming), as we were testing PC and the compression tool as well their player (and the engine).

As far as I understood, SOGS technique does require a specific trainig (?).
So we did a quick test with the SOG and that was quite successful.
But we would be soo happy not to switch engine because of that single feature.

If that is possible and we can get the same visuals, then thats literally a project savior!

Looking so much forward to that!

Thank you! Best Werner

[yamaciller]

Is it also possible, to address the current implemetation regarding compression?

It seems that there is some compression happening, and could that be the source of the visual differences?

Or would the SOG implementation replace the current anyway?

[Cedric]

SOG file import will not replace any implementation of existing formats. It will be an addition.

[HelmutB]

Hello Cedric,

I’m Helmut and I am working with Werner together on this project.

First of all thank you for working on Babylon. It is a pleasure to work with it, it is very intuitive and in many aspects it meets the sweet spot in abstraction and usability for us (I am a big fan of the PBRCustomMaterial).

As Werner mentioned we are also chasing a rendering quality issue of PLY files (please see the GScompare1 image). The left hand side image shows the 100k PLY file rendered in Babylon. The most visible difference is in the sand around the shoe (in comparison to the other viewers). With higher splat count density there is no visible difference in the viewers anymore.

We are considering using a similar low splat count (and density) for our mobile devices version (for performance reasons mainly) - so the rendering issue might become problematic for us.

When we render the 100k PLY with the pure WebGL implementation from GitHub - kishimisu/Gaussian-Splatting-WebGL: 3D Gaussian Splatting Renderer for WebGL (we have used that internally for testing some things with the shaders), the sand looks as it should. Now the thing is that the pure WebGL viewer doesn’t use any compression/quantization at all, everything is 32 bit floating point.

Therefore my guess would be that it is coming from some sort of quantization, which is also applied to the data from the PLY file. I have spotted two sources, but I don’t know if they are relevant, if the code is indeed executed.

The first one is the static _GetSplat() method (Babylon.js/packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingMesh.ts at master · BabylonJS/Babylon.js · GitHub) of the GaussianSplattingMesh class, which writes the rotations (and also the colors) into a Uint8ClampedArray. The colors are going directly into a texture - but the rotations need to be converted into the Covariances anyway, before they are written into the textures, is this correct?

The other one is the internal format of the color texture, selected in the _updateTexture() method ( Babylon.js/packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingMesh.ts at master · BabylonJS/Babylon.js · GitHub ). It seems to me that 8 bit are used - I could imagine that this might not be sufficient for encoding the opacity in some cases (which is - I believe - in the alpha channel of that texture).

(On a side node, we also tried to render the splat into a 16bit offscreen texture and indeed one can notice a tiny difference but it doesn’t fix the rendering issue.)

Many thanks for your help.

Helmut

[Cedric]

Depending on the format, there is quantization. And for a same format like spz, quantization depends on the file format version. SPZ in v3 format has better rotation quantization precision.

There is no compression without quantization at some points. Compressing .ply to .zip is the only way (AFAIK) to get a smaller file without losing precision.

SPZ quantizes pretty much everything: 24bits for position component, 8bits exponential per scale component. 10bits per quaternion component for v3 …

SOG(S) is quantized as well with 16bits per position component for example. It may sound low but in practice, it seems enough. I have no doubts that comparing it to uncompressed .PLY might give visible artifacts.

Every format I’ve seen use 8bits precision for alpha. It’s a very important data for visual quality, so better not touch it. But same as others, there will be a difference with uncompressed float.

Covariances values are converted to half-float (16bits floats) when stored into a GPU texture.

At the end, it will be like .jpg: what is the acceptable loss that is not visible enough for a file size ratio. I’m sure it’s possible to slightly increase rendering fidelity with tweaks here in there in the engine but there will always be some loss compared to raw .ply.

[leo_h]

Looking forward to this as well for some research projects - I’m hoping to get BabylonJS mentioned in a published paper or two

[Cedric]

I’ll be very happy to ship you some Champagne when it happens

[HelmutB]

@Cedric

Every format I’ve seen use 8bits precision for alpha.

Hmm. Maybe rendering issue is coming from the quantized rotations then.

In the _makeSplat() method (see Babylon.js/packages/dev/core/src/Meshes/GaussianSplatting/gaussianSplattingMesh.ts at master · BabylonJS/Babylon.js · GitHub ) the covariances are calculated this way:

        quaternion.set(
            (uBuffer[32 * index + 28 + 1] - 127.5) / 127.5,
            (uBuffer[32 * index + 28 + 2] - 127.5) / 127.5,
            (uBuffer[32 * index + 28 + 3] - 127.5) / 127.5,
            -(uBuffer[32 * index + 28 + 0] - 127.5) / 127.5
        );
        quaternion.toRotationMatrix(matrixRotation);

        Matrix.ScalingToRef(fBuffer[8*index+3+0]*2,...);

        const m = matrixRotation.multiplyToRef(matrixScale,TmpVectors.Matrix[0]).m;

        const covariances = this._tmpCovariances;
        covariances[0] = m[0] * m[0] + m[1] * m[1] + m[2] * m[2];
        covariances[1] = m[0] * m[4] + m[1] * m[5] + m[2] * m[6];
        covariances[2] = m[0] * m[8] + m[1] * m[9] + m[2] * m[10];
        covariances[3] = m[4] * m[4] + m[5] * m[5] + m[6] * m[6];
        covariances[4] = m[4] * m[8] + m[5] * m[9] + m[6] * m[10];
        covariances[5] = m[8] * m[8] + m[9] * m[9] + m[10] * m[10];

so it seems that the quaternions are reconstructed from the 8 bit representation before the covariance matrix is constructed.

Beside that, the 8 bit alpha this is the only sort of quantization I have found.

Thanks!

Helmut

[Cedric]

SOG(S) in progress here : GS SOG by CedricGuillemet · Pull Request #17212 · BabylonJS/Babylon.js · GitHub

I could test PLY converted to SOG. I’m working on SH support.

[Cedric]

SOG PR in review now. Please test the snapshot Playground & Sandbox.

GS SOG