Splat rendering issues with right handed system and container loading

Bugs
1

I noticed some issues when rendering gsplats and gsplat depth using scene.useRightHandedSystem.

  • scene.useRightHandedSystem = false (default behavior):

    • everything works: loading by container or importMeshAsync, rendering both rgb and depth

  • scene.useRightHandedSystem = true:

    • loading with container: splat does not show at all (rgb or depth)

    • loading with importMeshAsync: rgb works, depth does not work

Reference playground below. You can play with the booleans at the top.

var createScene = async function () {

const rightHandedSystem = false;

const showDepth = true;

const useContainer = false;



BABYLON.Effect.ShadersStore\["depthDisplayVertexShader"\] = \`

    precision highp float;

    

    attribute vec3 position;

    attribute vec2 uv;

    

    uniform mat4 worldViewProjection;

    

    varying vec2 vUV;

    

    void main(void) {

    gl_Position = worldViewProjection \* vec4(position, 1.0);

    vUV = uv;

    }

\`;



BABYLON.Effect.ShadersStore\["depthDisplayFragmentShader"\] = \`

    precision highp float;

    

    varying vec2 vUV;

    

    uniform sampler2D depthSampler;

    uniform float cameraNear;

    uniform float cameraFar;

    

    void main(void) {

    // Read depth value from the depth texture

    float stored_depth = texture2D(depthSampler, vUV).r;

    

    // Babylon's DepthRenderer stores: stored_depth = (z + minZ) / (minZ + maxZ)

    // Recover the linear depth z:

    float linear_depth = stored_depth \* (cameraNear + cameraFar) - cameraNear;

    

    // Normalize to \[0, 1\] range over \[near, far\]

    float depth_normalized = (linear_depth - cameraNear) / (cameraFar - cameraNear);

    depth_normalized = clamp(depth_normalized, 0.0, 1.0);

    

    // Invert so closer objects are brighter

    float inverted = 1.0 - depth_normalized;

    

    // Apply stronger gamma to darken and increase contrast

    // Higher exponent = darker overall, more contrast

    float enhanced = pow(inverted, 2.5);

    

    gl_FragColor = vec4(enhanced, enhanced, enhanced, 1.0);

    }

\`;



// This creates a basic Babylon Scene object (non-mesh)

var scene = new BABYLON.Scene(engine);

scene.useRightHandedSystem = rightHandedSystem;



// This creates and positions a free camera (non-mesh)

var camera = new BABYLON.ArcRotateCamera("camera1", -1, 1, 3, new BABYLON.Vector3(0, 0, 0), scene);

camera.minZ = 0.1;

camera.maxZ = 10.0;



// This targets the camera to scene origin

camera.setTarget(BABYLON.Vector3.Zero());



// This attaches the camera to the canvas

camera.attachControl(canvas, true);



// This creates a light, aiming 0,1,0 - to the sky (non-mesh)

var light = new BABYLON.HemisphericLight("light", new BABYLON.Vector3(0, 1, 0), scene);



const source = "https://raw.githubusercontent.com/raymondyfei/SharedTestModels/main/a_cute_cat_float32.ply"

var splat;

if (useContainer) {

    var container = await BABYLON.LoadAssetContainerAsync(source, scene);

    var instances = container.instantiateModelsToScene((sourceName) => {

            return sourceName;

        },

        true,

        { doNotInstantiate: true }

    );

    splat = instances.rootNodes\[0\] as BABYLON.TransformNode;

} else {

    var importResult = await BABYLON.ImportMeshAsync("https://raw.githubusercontent.com/raymondyfei/SharedTestModels/main/a_cute_cat_float32.ply", scene);

    splat = importResult.meshes\[0\];

}



splat.rotation.x = -Math.PI / 2;



var depthRenderer = scene.enableDepthRenderer(camera);

depthRenderer.forceDepthWriteTransparentMeshes = true;

depthRenderer.alphaBlendedDepth = true;



var engine = scene.getEngine();

var aspectRatio = engine.getAspectRatio(camera);



// Calculate the size needed to fill viewport at the given distance

const quadDistance = 2.0;



// For perspective camera, calculate visible size at distance

// visible_height = 2 \* tan(fov/2) \* distance

const fov = camera.fov; // Field of view in radians

const visibleHeight = 2 \* Math.tan(fov / 2) \* quadDistance;

const visibleWidth = visibleHeight \* aspectRatio;



if (showDepth) {

    var depthQuad = BABYLON.MeshBuilder.CreatePlane("depthQuad", { 

    width: visibleWidth,

    height: visibleHeight,

    sideOrientation: BABYLON.Mesh.DOUBLESIDE 

    }, scene);



    depthQuad.position = new BABYLON.Vector3(0, 0, quadDistance);

    depthQuad.parent = camera;

    depthQuad.renderingGroupId = 3; // Render on top of everything



    const depthTexture = depthRenderer.getDepthMap();

    const depthMat = new BABYLON.ShaderMaterial(

    "depthQuadMat",

    scene,

    {

        vertex: "depthDisplay",

        fragment: "depthDisplay",

    },

    {

        attributes: \["position", "uv"\],

        uniforms: \["worldViewProjection", "cameraNear", "cameraFar"\],

        samplers: \["depthSampler"\]

    }

    );

    depthMat.setTexture("depthSampler", depthTexture);

    depthMat.setFloat("cameraNear", camera.minZ);

    depthMat.setFloat("cameraFar", camera.maxZ);

    depthMat.backFaceCulling = false;

    depthQuad.material = depthMat;



    depthRenderer.getDepthMap().renderList = \[splat\];

}

return scene;

};

export default createScene

3

Please bear with us as @Cedric is in vacation for a bit

1 Like
4

Hi @haskha

Please provide a working PG and paste its link here. It will make debugging much more easier :slight_smile:

5

Hi @Cedric,

I have tried but when saving I get the error “Unable to save your code. It may be too long.”

Sorry the code in my initial post was broken by copy-paste. I have pasted the code here instead, you should be able to copy into a typescript playground babylon-playground - Pastebin.com

6
7

Thank you!

You can see how if we play with the booleans at the top it fails as I described in the first post.

rightHandedSystem false → everything works, while rightHandedSystem true has issues with loading from container.

8

Yes, I can see the issue. I’ll take a closer look asap.

9

problem with depth renderer and right handedness is not linked to the gaussian splatting. It’s the custom renderer and full screen quad that are not good.

check this sample for reference/

In the PG,

depthQuad.position = new BABYLON.Vector3(0, 0, quadDistance); should be depthQuad.position = new BABYLON.Vector3(0, 0, -quadDistance);for right handed scene. this will display the fullscreen quad.

I can repro the issue with the container and I’m looking for a fix.

1 Like
10
1 Like