Gaussian Splatting | Picking a point

Questions
1

Is there a way to select a point of the splat? i imagine for example a way to insert a hotspot. is collision with splat points handled?

As usual…thanks a lot!!

2

cc @ryantrem as I guess some work was done for hotspots and I wonder if it could be reused here ?

3

This seems to work:

scene.onPointerObservable.add((pointerInfo) => {
    if (pointerInfo.type === BABYLON.PointerEventTypes.POINTERPICK) {
        const pickResult = scene.pick(scene.pointerX, scene.pointerY);

        if (pickResult.hit) {
            console.log("Hit point:", pickResult.pickedPoint);
            console.log("Hit mesh:", pickResult.pickedMesh);
        }
    }
});

Gaussian Splatting | Babylon.js Playground

ChatGPT

Yes, selecting a point (or splat) from a Gaussian Splatting Mesh in Babylon.js requires interaction techniques that depend on how the Gaussian Splatting Mesh is implemented. Here’s a step-by-step outline to achieve this:

1. Determine the Rendering Context

Gaussian Splatting Meshes typically render points or splats as Gaussian blobs in 3D space. The interaction logic will depend on whether the splats are:

  • True mesh vertices
  • GPU-rendered point clouds
  • Custom shader-driven entities

2. Raycasting to Identify Points

Babylon.js supports raycasting for mesh selection. If the splats are implemented as part of a mesh, you can use Babylon’s Scene.pick or Scene.pickWithRay methods to detect which part of the mesh (or which vertex) the user is interacting with.

Example Code for Ray Picking:

// Create a pointer observable for mouse clicks
scene.onPointerObservable.add((pointerInfo) => {
    if (pointerInfo.type === BABYLON.PointerEventTypes.POINTERPICK) {
        const pickResult = scene.pick(scene.pointerX, scene.pointerY);

        if (pickResult.hit) {
            console.log("Hit point:", pickResult.pickedPoint);
            console.log("Hit mesh:", pickResult.pickedMesh);
        }
    }
});
  • If pickResult.pickedMesh is the Gaussian Splatting Mesh, use additional data structures (like vertex buffers or custom shaders) to identify the specific splat.

3. Handling GPU-Rendered Point Clouds

If the splats are GPU-rendered without explicit mesh geometry (e.g., using custom shaders or textures), you may need to implement:

  • Custom picking logic: Render splats to an offscreen texture with unique IDs for each splat. Read the texture’s color under the mouse cursor to identify the splat.
  • Mouse-to-World Projection: Convert the mouse position to a 3D ray and manually check against splat positions.

Example for Manual Picking:

const ray = scene.createPickingRay(scene.pointerX, scene.pointerY, BABYLON.Matrix.Identity(), camera);

// Iterate over splats' positions
splatPositions.forEach((position, index) => {
    const distance = BABYLON.Vector3.Distance(ray.origin, position);
    if (distance < splatSize) {
        console.log("Selected splat index:", index);
    }
});

4. Using GPU Selection

For high-performance applications:

  1. Assign each splat a unique color ID in a shader.
  2. Render the scene to a selection buffer.
  3. Read the color at the mouse’s position in the selection buffer to identify the splat.

Shader Selection Example:

  • Vertex Shader: Pass splat ID as color.
  • Fragment Shader: Render the splat with its ID color.
  • Read buffer with engine.readPixels to get the ID.

5. Debugging and Visualization

To visualize splats for debugging, you can add bounding spheres or small markers to each splat position, making it easier to see what is selected.

Example:

splatPositions.forEach((position) => {
    const sphere = BABYLON.MeshBuilder.CreateSphere("marker", { diameter: 0.1 }, scene);
    sphere.position = position;
    sphere.material = new BABYLON.StandardMaterial("markerMat", scene);
    sphere.material.diffuseColor = new BABYLON.Color3(1, 0, 0); // Red
});

Key Considerations

  • Performance: Large numbers of splats may slow down raycasting or GPU selection.
  • Precision: Gaussian splats may have a radius or size. Ensure your selection logic accounts for this.
  • Custom Shaders: If Gaussian Splatting is implemented in shaders, you may need to modify the shader code to expose splat information for interaction.

If you provide more details about your Gaussian Splatting implementation, I can tailor the solution further!

4

Thank you!

I’ll try it :slight_smile:

5

Gaussian Splatting mesh has a bounding volume information. So 1st level of picking is possible (you can check the picking with the Inspector and you can pick GS). But getting more precision like picking individual splat is more difficult because of the amount of them. A CPU solution involves computing individual volume and testing each one. This can be daunting when your GS is made of million of individual splats. But because each splat is semi transparent, using a GPU solution with 2D buffer (rasterizing each individual splat with some ID per splat instead of a color) may not return perfect solution.

ChatGPU explains it better with far more details :smiley:

splatPositions.forEach((position) => {
    const sphere = BABYLON.MeshBuilder.CreateSphere("marker", { diameter: 0.1 }, scene);

Valid in theory, not in practice :smiley:

1 Like