Check `isInFrustum` but with a percentage of the camera view

Hi all,
I’m trying to find a way to detect if my BoundingInfo is inside a percentage of my camera view (for the sake of the example, say 20% of the height and 40% of the width)… I’m trying to retro-engineering how isInFrustm works, but I’m quite new to BJS and a total newby to camera/frustum/matrix stuff… So this is what I’ve got so far:

  1. I’ve started from the standard way to check if mesh is in frustum (I didn’t find any existing thread/resource about my specific need), and found that it works perfectly well with BoundingInfo too:

var frustumPlanes = BABYLON.Frustum.GetPlanes(camera.getTransformMatrix());
mesh.isInFrustum(frustumPlanes)
  1. Which made me look at the source code of Camera.getTransformMatrix(), and then thought that all I needed to do was to do something like this:
class Camera() {
    // Readaptation of `getTransformMatrix` method, temporary name
    public getAdaptedTransformMatrix(widthPercentage: number, heightPercentage: number): Matrix {
        // Only the construction of `_projectionMatrix` seems to be influenced by FoV values, so I guessed I could keep `_computedViewMatrix` as it is
        return this._computedViewMatrix.multiply(this.getAdaptedProjectionMatrix(widthPercentage, heightPercentage))
    }
}
  1. So here is my try to get a modified projection matrix:
class Camera {
    // Readaptation of `getProjectionMatrix`, temporary name
    public getAdaptedProjectionMatrix(widthPercentage: number, heightPercentage: number): Matrix {
        let result: Matrix = new Matrix();

        // I've removed a bunch of `camera._cache` logic since I don't think I need it

        // Matrix
        var engine = camera.getEngine();
        var scene = camera.getScene();
        if (camera.mode === Camera.PERSPECTIVE_CAMERA) {

            if (camera.minZ <= 0) {
                camera.minZ = 0.1;
            }

            const reverseDepth = engine.useReverseDepthBuffer;
            let getProjectionMatrix: (fov: number, aspect: number, znear: number, zfar: number, result: Matrix, isVerticalFovFixed: boolean) => void;
            if (scene.useRightHandedSystem) {
                getProjectionMatrix = reverseDepth ? Matrix.PerspectiveFovReverseRHToRef : Matrix.PerspectiveFovRHToRef;
            } else {
                getProjectionMatrix = reverseDepth ? Matrix.PerspectiveFovReverseLHToRef : Matrix.PerspectiveFovLHToRef;
            }

            getProjectionMatrix(camera.fov * widthPercentage, // Apply percentage here
                engine.getAspectRatio(camera) * (widthPercentage / heightPercentage),  // Apply percentage here
                camera.minZ,
                camera.maxZ,
                result,
                camera.fovMode === Camera.FOVMODE_VERTICAL_FIXED);
        } else {
            var halfWidth = engine.getRenderWidth() * widthPercentage / 2.0; // Apply percentage here
            var halfHeight = engine.getRenderHeight() * heightPercentage / 2.0; // Apply percentage here
            if (scene.useRightHandedSystem) {
                Matrix.OrthoOffCenterRHToRef(camera.orthoLeft ?? -halfWidth,
                    camera.orthoRight ?? halfWidth,
                    camera.orthoBottom ?? -halfHeight,
                    camera.orthoTop ?? halfHeight,
                    camera.minZ,
                    camera.maxZ,
                    result);
            } else {
                Matrix.OrthoOffCenterLHToRef(camera.orthoLeft ?? -halfWidth,
                    camera.orthoRight ?? halfWidth,
                    camera.orthoBottom ?? -halfHeight,
                    camera.orthoTop ?? halfHeight,
                    camera.minZ,
                    camera.maxZ,
                    result);
            }
        }

        return result;
    }
}

Unluckily when I now use my custom getAdaptedTransformMatrix(0.4, 0.2) method, the isInFrustum check behaves as if I gave it the “classic” camera transformMatrix instead (meaning that it returns true as soon as my boundingInfo is in my screen, and I do not need to have it near the center of my screen as I would expect)… Any idea why?

P.s: I couldn’t add more than one empty line between phrases, any idea on how I could improve the formatting of this post?

1 Like

Maybe you could use a second camera with your percentage baked in so you simply use the isInFrustrum function of this camera ?

1 Like

It’s definitely an idea worth exploring, thanks! :slight_smile:

The only thing is that I may have up to 20 BoundingInfo to check each frame, and each one of them may require a unique set of width/height percentages… So I would need to rescale the “dummy camera” ~20 times each frame, or have as many dummies cameras as of distinct width/height pairs (so up to 20 cameras)… But I think I will start thinking about performance once I get something that works ^^

1 Like

Hi again,
I’ve made some tries with a clone camera and modifying it’s viewport (see this playground test), but it seems that it is not the proper way to go about it…

The only interesting resource I could find on how to resize a camera is this one, so I wonder if @RaananW may have some clues on how to achieve what am I looking for? :crossed_fingers:
Especially since the project I am working on uses a XR camera, and you seem to know how I could create an appropiate projection matrix (own deduction from your answer) :slight_smile:

@RaananW will definitely be able to have a look right after the break next week.

I have no idea what you are talking about :wink:

Nice question! Gonna be interesting.

To calculate the projection matrix you can use Matrix.PerspectiveFovLHToRef , which will be the way i would recommend you do that as well. The frustum is built using 6 planes ( Viewing frustum - Wikipedia) You will need them calculated correctly otherwise it will not work.

A question before - will isCompletelyInFrustum work in your case? or do you really want a different camera definition per object?

Hi RaananW!
Sorry if I wasn’t explicit enough, I will try to better explain what I meant :slight_smile:

About my “deduction from your answer”, I was referring to thes part of the phrase where you said:

fov is being ignored, since we are using the projection matrix directly. If you want to do it , you will need to calculate the projection matrix yourself

Which led me think you knew what camera projection matrix, and matrices in general, are (and how to work with them)… Which I think it’s the knowledge I’m missing to achieve what I’m looking for :slight_smile:

I’ve then updated my playground experience with Matrix.PerspectiveFovLHToRef as you suggested, but it seems that I’m still not doing it correctedly since it doesn’t behave as expected…

Here is the goal of the experience: I want to turn my debug area (which has the width/height percentage I want to use for my isInFrustum checks) to green only if the sphere is in the center of the screen area.
It’s similar to StarWars spaceship combat scenes actually, where you can fire only if the enemy is 'locked-on" on the center of the aim helper ^^

And about your other questions:

  • isCompletelyInFrustum , no I do not need the object to be completely inside if just one tiny bit is inside then I want to return true
  • what I want is different frustum planes per object, based on the size of the frustum that each object will require to be detected (in our StarWars example, some spaceships are “more easy/difficult” to lock-on than others)… Using a different camera for each one of them is a suggestion from sebavan, which I thought it could work if we can easily change the width/height of each camera… But, as shown in my playground example, I then dispose the new camera right after having created it since what I truly need is just the view/projection matrices to create the frustum planes

Was just a bad joke on my side, i understood what you meant. Sorry… :slight_smile:

I totally get what you are trying to achieve. And am trying to think of the best way for you to do it. One way would be to use the original camera and changes its fov:

Check isInFrustum but with a percentage of the camera view | Babylon.js Playground

Notice that

  1. it is not exact (but can be calculated)
  2. fov is camera dependant and if you have different width and height, you will need to calculate width planes and height planes (like i am doing here)

Another way would be to have an invisible mesh that has the camera as its parent. a very long mesh that will emulate the frustum. This way you can check if other meshes intersec with this mesh (instead of planes intersection).

You can also try “moving” the planes of the original camera to a new location, but i find it to be the last resort.

2 Likes

Don’t worry, it’s my bad for not having understood the joke ^^’

The updated playground that you shared really comes near what I am trying to achieve, that so cool !! :smiley:

But, as you said, it is not precise and I guess it comes from the fact that you did not use the widthPercentage and heightPercentage variables that I have defined at the top of the code…

Btw, what is the “magic” beneath these lines ?

camera.fov / 8 // Used when getting the widthPlanes
camera.fov / 16 // Used when getting the heightPlanes

// And afterwards we copy some values of the widthPlanes into the heightPlanes
heightPlanes[2] = widthPlanes[2];
heightPlanes[3] = widthPlanes[3];

It feels as if getting these parameters right can lead to a precise result, but after having played a bit with them (especially the first two divisions) I didn’t manage to find the proper way to do it…

If you do not have much time to explain (which I totally understand, and I’m already really grateful for the overall support that you guys of BabylonJS already provided me) maybe you have some interesting resources you can point me to? :slight_smile:

It all comes down to the way fov is being used to calculate the projection matrix.

this line:

let t = 1.0 / (Math.tan(fov * 0.5));

from the function calculating it is the one you will need to pay attention to. What you want is t to be 40% of what it was with camera.fov .

Thinking about it, you might just… well… create the projection matrix yourself instead of manipulating the fov :slight_smile: I have no idea why i didn’t come up with this earlier, would be much simpler:

Check isInFrustum but with a percentage of the camera view | Babylon.js Playground

Something you need to notice in this example - the “inFrustum” function is using the bounding sphere. which is, in case of a sphere, larger than the actual sphere! I rendered a new sphere (sphere2) just to show the size of the bounding sphere. This way you understand why it feels a bit off.

I hope this helps!!

4 Likes

That works wonderfully, thanks a lot RaananW !! :bowing_man:‍♂

And it also seems to me that it’s way more efficient than the other possibilities we were exploring :ok_hand:

It’s probably a noob question, and I should probabably make a dedicated thread in the forum for it, but how comes that isInFrustum uses the bounding sphere instead of the bounding box of a mesh for its checks? (the bounding box already ensures that the mesh in completely inside it, doing the check with a sphere that inglobes that box seems a little overkill and counter-intuitive to me… I get that performance wise is way simpler this way, but isn’t there a way to tell which “precision degree” we want to use?)

1 Like

You can set the default strategy. This is the signature of the isInFrustum function of the bounding info.

public isInFrustum(frustumPlanes: Array<DeepImmutable<Plane>>, strategy: number = Constants.MESHES_CULLINGSTRATEGY_STANDARD): boolean

Thses are the different strategies:

/** Default culling strategy : this is an exclusion test and it's the more accurate.
     *  Test order :
     *  Is the bounding sphere outside the frustum ?
     *  If not, are the bounding box vertices outside the frustum ?
     *  It not, then the cullable object is in the frustum.
     */
    public static readonly MESHES_CULLINGSTRATEGY_STANDARD = 0;
    /** Culling strategy : Bounding Sphere Only.
     *  This is an exclusion test. It's faster than the standard strategy because the bounding box is not tested.
     *  It's also less accurate than the standard because some not visible objects can still be selected.
     *  Test : is the bounding sphere outside the frustum ?
     *  If not, then the cullable object is in the frustum.
     */
    public static readonly MESHES_CULLINGSTRATEGY_BOUNDINGSPHERE_ONLY = 1;
    /** Culling strategy : Optimistic Inclusion.
     *  This in an inclusion test first, then the standard exclusion test.
     *  This can be faster when a cullable object is expected to be almost always in the camera frustum.
     *  This could also be a little slower than the standard test when the tested object center is not the frustum but one of its bounding box vertex is still inside.
     *  Anyway, it's as accurate as the standard strategy.
     *  Test :
     *  Is the cullable object bounding sphere center in the frustum ?
     *  If not, apply the default culling strategy.
     */
    public static readonly MESHES_CULLINGSTRATEGY_OPTIMISTIC_INCLUSION = 2;
    /** Culling strategy : Optimistic Inclusion then Bounding Sphere Only.
     *  This in an inclusion test first, then the bounding sphere only exclusion test.
     *  This can be the fastest test when a cullable object is expected to be almost always in the camera frustum.
     *  This could also be a little slower than the BoundingSphereOnly strategy when the tested object center is not in the frustum but its bounding sphere still intersects it.
     *  It's less accurate than the standard strategy and as accurate as the BoundingSphereOnly strategy.
     *  Test :
     *  Is the cullable object bounding sphere center in the frustum ?
     *  If not, apply the Bounding Sphere Only strategy. No Bounding Box is tested here.
     */
    public static readonly MESHES_CULLINGSTRATEGY_OPTIMISTIC_INCLUSION_THEN_BSPHERE_ONLY = 3;

Having said that, you can just test against the boundingBox instead of using the bounding info, if it is enough for you.

2 Likes

Oh my bad, I should have looked into it a bit more… Thanks for your patience and explanations !!

2 Likes

Sorry to resurrect this thread, but this Playground doesn’t work in Typescript and I was wondering if you know a solution.

Specifically, you can’t do:

const m = new Matrix();
    m.copyFrom(defaultCamera.getProjectionMatrix());
    m.m[0] /= 0.2;
    m.m[5] /= 0.2;

because m.m is a DeepImmutableObject

1 Like

Figured it out, type safe version:

const m = new Matrix();
m.copyFrom(defaultCamera.getProjectionMatrix());
m.multiplyAtIndex(0, 5);
m.multiplyAtIndex(5, 5);
2 Likes