Light interacting with Gaussians in Gaussian Splatting for generating shadows

Questions
, ,
1

I imported .splat and .ply file using ImportMeshAsync() and loaded a mesh using Gaussian Splatting method.

Now since, I want to generate shadow for the Gaussian Mesh, I tried to add ShadowGenerator() to the gaussian mesh and created a simple surface (a plane) below it to receive a shadow. But the gaussian mesh is not able to generate/cast shadow on the ground.

Then I read about the ShadowGenerator Class that it only interacts with standard geometry and materials, and NOT with particle systems and point clouds.

Question: Is there any way (or workaround) to make Gaussian meshes interact with the light and cast shadows?

2

There is no easy way to have GS shadows or light interaction.
This is 2 different problems. I think the shadow is doable. It’s “just” rendering into a render target. No need to sort splats. But Shadows don’t support transparency (or is there already some support for that @Evgeni_Popov ?) so an alpha test is needed at some point.
Light interaction is more problematic because splat color is diffuse + lighting baked. SH can help with the specular but this won’t be enough.

edit: I’ve opened an issue to keep track : Gaussian Splatting Updates · Issue #16671 · BabylonJS/Babylon.js · GitHub

3

Yes, I guess we would need to take into account the alpha value before deciding if a depth value should stay in the shadow map or not, meaning we would need to first render the scene from the point of view of the light to record a map with the alpha values. Also, I don’t know if it’s currently possible to render a splat from different point of view: we would need an additional worker thread to render from the point of view of the light.

4

Worker thread is only needed to sort splats back-to-front. For shadows, we only need opacity accumulation so no sorting is needed for the light point of view.

1 Like