Cascaded Shadow Map are not properly projected with camera.freezeProjectionMatrix

Questions
1

Hi,

it seems CascadedShadowMap does not work with a custom projection matrix for a camera.
The playground shows stripes, when I set a custom projection matrix.
The debug layers seem to work properly (this however is not the case in my own environment)

image
image1596×1218 123 KB

I am trying to implement them within a maplibre layer, which transforms the projection matrix of the camera. Add a 3D model with babylon.js - MapLibre GL JS

Going to summon @Evgeni_Popov since he has been of great help in the past for CSM. Not sure how easy this one is…

Kind regards!

2

This is really tricky indeed as well as definitely an edge case. I hope it is an easy one for @Evgeni_Popov but if not, would you want to contribute smthg ?

1 Like
3

I think I would need some assistance on this. Matrices are :exploding_head:

4

This line is wrong:

camera.freezeProjectionMatrix(tempCamera.getWorldMatrix().multiply(tempCamera.getProjectionMatrix(true)))

You should pass the projection matrix only, so:

camera.freezeProjectionMatrix(tempCamera.getProjectionMatrix(true))

Fixed PG:

1 Like
5

are something we have to master sooner or later :smiley:

6

Ah the multiplication with world matrix was on purpose, since otherwise projection matrix of camera and tempCamera should be equal and that is why your playground works, while mine does not.

If you look at the integration into maplibre, it does the following:

// Calculate world matrix
    const worldMatrix = BABYLON.Matrix.Compose(
        new BABYLON.Vector3(worldScale, worldScale, worldScale),
        BABYLON.Quaternion.FromEulerAngles(
            worldRotate[0],
            worldRotate[1],
            worldRotate[2]
        ),
        new BABYLON.Vector3(
            worldOriginMercator.x,
            worldOriginMercator.y,
            worldOriginMercator.z
        )
    );
.....
 render (gl, matrix) {
            const cameraMatrix = BABYLON.Matrix.FromArray(matrix);

            // world-view-projection matrix
            const wvpMatrix = worldMatrix.multiply(cameraMatrix);

            this.camera.freezeProjectionMatrix(wvpMatrix);

            this.scene.render(false);
            this.map.triggerRepaint();
        }

I just wanted to simulate the bevhiour with just a simple matrix multiplication. View Matrix left out…

are something we have to master sooner or later :smiley:

i know :weary:

7

Maplibre configures Babylon in a special mode, to be able to interoperate correctly:

  • they create a basic Camera, not a FreeCamera or ArcRotateCamera
  • they call scene.render(false), meaning camera.update() is not called
  • thanks to this, they are able to trick the system regarding the view/projection matrices!
    • camera.viewMatrix is equal to the identity matrix and never changes
    • camera.projectionMatrix is the world*view*projection matrix calculated by worldMatrix.multiply(cameraMatrix);

Because the final transformation matrix of the camera is calculated by the scene by doing this.setTransformMatrix(activeCamera.getViewMatrix(), activeCamera.getProjectionMatrix(force));, and setTransformMatrix is simply multiplying both matrices, you get the right result.

But, in reality, the projection matrix they set in the camera is not the projection matrix, it is the world*view*projection matrix!

CSM needs that a proper view matrix exists to work correctly. I don’t know if Maplibre is able to provide a separate view and projection matrices, instead of giving a view*projection combined matrix…

8

Thanks for your reply!
I understand how maplibre is misusing the projection matrix as by applying transforms to it. I was surprised to see, that this works, since the geometries all stay close to 0,0,0 and there are no jittery floating point imprecisions.

So what is missing is the correct view matrix? Since maplibre is only setting the projection matrix and not the view matrix? Maybe there is a chance then to get it to work by setting _computedViewMatrix.
Or I just need to rotate and position the camera, but I think this won’t be as easy.

I realised GreasedLine has a similiar - maybe the same - issue.

9

CSM itself does work, the shadow maps are generated correctly (we only need the final transformation matrix of the camera, not view/projection independently).

What fails is the rendering of meshes with receiveShadows = true: that’s where the view matrix is used, because we must transform the current vertex position into the camera view space. So, I’m afraid that this view matrix is required for CSM to work…

10

Yes you are right it is most probably the rendering within the shader. A normal shadow generator does not need the view matrix?
But wouldn’t it be enough if I then set the computedViewMatrix of the camera to a different value? Then this view matrix should be used in the rendering process.
I tried setting _computedViewMatrix in my plaground. But this does not change anything. I am not sure if it is because it will be recomputed at some point.

11

You can try to override Camera._getViewMatrix, as the result of this function is assigned to _computedViewMatrix. However, for getTransformationMatrix to work correctly, you must make sure that projectionMatrix only contains a projection and nothing else:

public getTransformationMatrix(): Matrix {
    this._computedViewMatrix.multiplyToRef(this._projectionMatrix, this._transformMatrix);
    return this._transformMatrix;
}
12

I just realized that ArcRotateCamera and TargetCamera implement different _getViewMatrix functions, probably need to have a closer look.
Hope I can come back with a solution soon.

13

calculating the projection matrix as in maplibre helped me at the end to get it done! now i have a proper babylon camera again without hiccups because of a weird projection matrix :slight_smile:
here is the part where it is calculated - it is mainly copy and paste:

1 Like
14

Congrats!

1 Like