Camera bugs related to delta time

Hi, Babylon team!

A few weeks ago I reported an issue with the camera related to inertia:

Thank you all who helped implement a temporary solution!

Since then, I’ve continued to work on the camera experience for my company and I’ve found a few more issues that I’d like to report here. It was mentioned in the other forum post that the Babylon team has a camera overhaul planned. I want to make sure that these issues are included in that, and are hopefully fixed.

1. The camera applies inertia without respect to delta time.

Take a look at the checkInputs method of the arc rotate camera (used for reference; issue exists for all camera types).

This code is meant to apply inertia over time. The issue is that “over time” is how fast the checkInputs method is called. At 240 FPS, the inertia will dissipate four times faster than 60 FPS.

The solution is to include delta time:

const deltaTimeScale = this.getEngine().getDeltaTime() / Frequency;
const inertia = this.inertia ** deltaTimeScale;
this.inertialAlphaOffset *= inertia;
this.inertialBetaOffset *= inertia;
this.inertialRadiusOffset *= inertia;

(the double star ** is intentional)

2. The camera inputs don’t respect delta time

As far as I can tell, this affects all the camera input classes. I’ll use ArcRotateCameraPointersInput to illustrate though. The onTouch method in particular.

The onTouch method adds to the camera inertia every time it’s called. This is then applied in the camera checkInputs method. The problem here is that onTouch is called by the browser (per event), while checkInputs is called by Babylon (per frame). These operations can easily happen out of sync with one another. Say, the browser is measuring pointer events at 60 FPS and the render engine is running at 30 FPS. Ultimately, this causes the camera to move at different speeds depending on frame rate when inertia is on.

The solution is to include delta time:

public override onTouch(point: Nullable<PointerTouch>, offsetX: number, offsetY: number): void {
    const deltaTimeScale = this.camera.getEngine().getDeltaTime() / Frequency;
    if (this.panningSensibility !== 0 && ((this._ctrlKey && this.camera._useCtrlForPanning) || this._isPanClick)) {
        this.camera.inertialPanningX += -offsetX / this.panningSensibility * deltaTimeScale;
        this.camera.inertialPanningY += offsetY / this.panningSensibility * deltaTimeScale;
    } else {
        this.camera.inertialAlphaOffset -= offsetX / this.angularSensibilityX * deltaTimeScale;
        this.camera.inertialBetaOffset -= offsetY / this.angularSensibilityY * deltaTimeScale;
    }
}

3. The camera inertia cutoff doesn’t scale with delta time

This is the issue that I originally reported in this post. I’m putting it here so that it is included with the others since they all, in part, relate to each other.


You may have noticed the Frequency constant used in the above solutions. This is defined as follows:

const Frequency = 1000 / 60;

While not strictly necessary to fix the above issues (delta time is the only necessary part), without it all the numbers would be scaled. This would cause everyone’s existing projects to behave very different. By including this frequency, inertia will behave exactly like it does at 60 FPS regardless of the actual FPS.


I can put together some playgrounds with the above solutions tomorrow. I’ve already implemented these solutions for my company’s project though. You can see it in action here:

cc @georgie

thank you! @amoebachant and I will discuss when we are back in office next week! Good timing as we were discussing this topic yesterday :slight_smile:

Hi.

I’ll add my 5 cents.

The AutoRotateBehavior is also driven by checking inputs via camera.onAfterCheckInputsObservable

It kinda does respect delta time though. Although not via Engine.getDeltaTime() but with its own this._lastFrameTime = now

However, when scene rendering is suspended by Engine.stopRenderLoop (e.g. in some visibility checking), camera continues to check inputs and the delta time accumulates. When rendering is resumed after awhile, camera jumps to unpredictable position.

Disabling camera does not help: disabled cameras still check inputs and run their behaviors.
Detaching controls also does not help, because the last frame time is recorded.

Workaround on app-level is to add additional handler like this:

camera.onEnabledStateChangedObservable.add(() => {
    if (camera.isEnabled()) {
        camera.useAutoRotationBehavior = myoptions.autoSpin;
        camera.attachControl(); 
    } else { 
        camera.useAutoRotationBehavior = false;
        camera.detachControl();
    }
});

And the visibility checker should enable/disable cameras (and hope that camera won’t change in suspended scene)

Hey @reedsyllas and @qwiglydee, thanks for reaching out! As @georgie mentioned, we’ve been talking about this area recently, and your observations and suggestions are very helpful. We’d like to address these issues with a fresh approach that also simplifies the camera motion APIs at the same time, and these thoughts will be very helpful as we dig into that. You can track the work using this issue and please feel free to leave any other feedback or ideas you have there.

Thanks again!

@amoebachant

Any news about the progress?

Camera inertia needs upgrading.

Hi @a_bolog the new Geospatial Camera uses a different approach for inertia and velocity. All of the calculations for movement / speed live in the CameraMovement class – which takes the velocity from the previous frame and uses it for the next frame (applying inertial decay). This approach should be framerate agnostic. And the units for the speed and movement vars are more intentionally defined (ex: zoomSpeed: Desired coordinate unit movement per input pixel when zooming) vs arbitrary numbers.

We’d like to test out this system and if it meets our needs for fixing bugs related to delta time, we can port the other cameras to using the CameraMovement system.

You can test out the camera using this playground, and read more about the camera in the docs
Geospatial Camera | Babylon.js Documentation. The docstrings in the code also provide a bunch of details about the units used for movement / speed variables.

Please test out the GeospatialCamera and let us know if this approach would solve the concerns!

Hey @georgie , that looks great, the inertia is consistent across multiple framerates.

But how can we use this inertia with FreeCamera and FPS moving with arrow keys?

@a_bolog great! I will work on introducing the camera movement system to the other cameras and let you know once available

@a_bolog this PR has the change, however I’m going to wait to merge it until after the Babylon 9 release (as it touches critical camera code)
Introduce framerate-independent movement to cameras by georginahalpern · Pull Request #18030 · BabylonJS/Babylon.js

You can test using the snapshot link from the PR and if you notice any issues I can address at checkin time!
Babylon.js Playground

@georgie Thanks, that’s great. I’ve noticed a little bug, the inertia doesn’t work when enterPointerLock().
Check this out: https://playground.babylonjs.com/?snapshot=refs/pull/18030/merge#YN2Y2C#1

And another thing, when moving with arrows in FreeCamera, the speed inertia is significantly slower when 15FPS vs 120FPS.

Also moving diagonally is a bit strange. When walking forward and then moving left right, is somehow overwriting the inertia.

thanks for testing! will look into the above and perform further testing on the branch as well

I explored this some more and would like to update everyone on my progress.

Fairly recently I watched a video by Freya Holmér that speaks directly about the issue at hand here. She dives much more into the math side of things than I have. Link here: https://www.youtube.com/watch?v=LSNQuFEDOyQ.

The relevant point for this discussion is that my solution for inertia using ** (Math.pow), while perfectly valid, is a slower solution than the one using Math.exp. This is mentioned in the video here.

Therefore, the following code:

const Frequency = 1000 / 60;
const deltaTimeScale = this.getEngine().getDeltaTime() / Frequency;
const inertia = this.inertia ** deltaTimeScale;

this.inertialAlphaOffset *= inertia;
this.inertialBetaOffset *= inertia;
this.inertialRadiusOffset *= inertia;

could be written faster this way:

const Decay = 16 / 1000;
const decayAmount = Math.exp(-Decay * this.getEngine().getDeltaTime());

this.inertialAlphaOffset *= decayAmount;
this.inertialBetaOffset *= decayAmount;
this.inertialRadiusOffset *= decayAmount;

However, it produces some questions:

Where did this.inertia go? And what is the decay constant?

In this system, ‘inertia’ is replaced by ‘decay’. Inertia is the amount of momentum to keep each step. In contrast, decay is the amount of momentum to keep after 1 second. Decay doesn’t care about the number of steps that happened, only about the elapsed time. This makes decay a frame rate independent factor.

Why is decay set to 16 / 1000?

This is where my understanding of Math.exp starts to break down a little. Freya comments that a decay factor between 1 and 25 is the “useful range”. I can confirm that this range works well for the ArcRotateCamera’s alpha, beta, and radius. A value of 1 takes a while to stop, while 25 stops almost instantly.

As for why there’s a division by 1000: the decay and delta time units must match. Since this.getEngine().getDeltaTime() returns milliseconds, I’m dividing the decay factor by 1000 to match the units up.

I would encourage the BabylonJS developers to replace the concept of inertia/damping in favor of decay or even half-life which are superior descriptors. For example, I know that a half-life of 250ms means that my camera will move half as fast as it did after 250ms. What does an inertia of 95% mean? When will my camera stop? I’d have to know the frame rate.

I’m confident that it’s possible to convert between the inertia/damping, decay, and half-life factors. If I’m right, we can use the most performant one under the hood for calculations (probably decay), but expose the easiest to understand one to users (half-life seems the simplest). I’ll try to figure out the math for this soon.


The last thing I’d like to bring up is:

Should we switch from ‘delta change’ to ‘absolute positioning’?

At the moment, inertialAlphaOffset is added to alpha each frame. inertialAlphaOffset is set by the CameraInputs (on pointer drag, for example). A discrepancy can happen if the CameraInput writes to inertialAlphaOffset at a different speed than the BabylonJS engine reads it.

For example, the engine is running at 30 FPS, but the browser reports mouse events at 60 FPS. This causes the mouse delta-change stored in inertialAlphaOffset to be half of what the camera controller expects. So, even if our camera controller has the correct math, it’s being told that the mouse is moving half as fast as it actually is. This again introduces a frame rate dependent issue.

This is the issue I described in point #2 at the top of this forum. I wrote that the fix is to include the engine’s delta time in the event handler of the pointer. However, an alternative option would be to replace inertialAlphaOffset with a new value I will call alphaTarget.

Instead of using all of this fancy math to approach inertialAlphaOffset towards 0, we would approach alpha towards alphaTarget. The pointer event listener can write to alphaTarget as fast as it wants. Each engine update, alpha is simply stepped toward alphaTarget. It doesn’t matter what happened to alphaTarget since the last update. It just steps towards it. This greatly simplifies the implementation of the pointer/wheel event handlers, since they don’t need to worry about delta time.

There is another advantage. If a user wants to update the camera’s position smoothly, they won’t have to figure out all this math again. Instead, they would simply write to the alphaTarget and the controller would take care of the rest (by approaching alpha towards the new alphaTarget). If the user wants it to be instant, they can write the value to alpha also.

There is one more advantage to this, addressing the epsilon/cutoff check I described in point #3. Since we know where the camera is going, we can simply use Math.abs(camera.alpha - camera.alphaTarget) < (Epsilon * camera.speed). Notice how delta time is not necessary anymore.


To summarize, I believe that if Babylon accepted all of these changes:

  1. Math.exp would provide a performant, smooth, and frame rate independent method to update the camera position.
  2. Using a factor like half-life allows users to precisely and intuitively control their camera’s smoothness.
  3. Using alphaTarget simplifies the camera’s input handlers. It allows users to update their camera position instantly (by writing to alpha & alphaTarget) or smoothly (by just writing to alphaTarget). And finally, it simplifies the epsilon/cutoff check in the camera controller.

I’d like to put together a playground showcasing all of this, but it’s quite an undertaking due to how cameras are currently structured in Babylon. I’ll still try to do it at some point.

I went ahead and found the math for converting between the different factors.

ln(2) / half_life = decay_constant

ln(2) / decay_constant = half_life

-frames_per_second * ln(inertia) = decay_constant

With that we can define the following:

const LOG_2 = Math.log(2);

// Get decay constant from half-life.
function decayConstantFromHalfLife(halfLife) {
  return LOG_2 / halfLife;
}

// Get decay constant from inertia @ fps.
function decayConstantFromInertia(inertia, fps) {
  return -fps * Math.log(inertia);
}

// Calculate the decay after duration `deltaTime`.
function expDecay(start, end, decay, deltaTime) {
    return end + (start - end) * Math.exp(-decay / deltaTime);
}

These can be used within the camera’s _checkInput like so:

const inertia = 0.95;

// This can be calculated once (when inertia changes).
// 60 is used so that it will feel the same as 60 FPS inertia did.
// In other words, 60 FPS users won't feel a difference with this system vs the current system.
const decayInMs = decayConstantFromInertia(inertia, 60) / 1000;

// Approach `this.inertialAlphaOffset` towards 0.
this.inertialAlphaOffset = expDecay(this.inertialAlphaOffset, 0, decayInMs, this.getEngine().getDeltaTime());

Or, if we use half-life and alphaTarget, it would look like:

// Half the speed every 1/4 second.
const halfLife = 1/4;

const decayInMs = decayConstantFromHalfLife(halfLife) / 1000;

this.alpha = expDecay(this.alpha, this.alphaTarget, decayInMs, this.getEngine().getDeltaTime());

cc @amoebachant

@georgie

Any updates?

This looks very similar to ExponentialEasing. Is there any value generalizing to an Easing function? What about if a selectable EasingFunction can be applied to this and other camera controls?

Hey all! Status update: framerate-independent inertia recently landed across the Target, Free, Fly, Geospatial, and ArcRotate cameras. That work introduced a dedicated CameraMovement class that owns the physics layer — velocity, inertial decay, speed, and per-frame delta computation — It gives us one isolated place to evolve the math without touching the public camera API, so new behavior can be added backward-compatibly.

You can test here Babylon.js Playground (with version 9.12.1)

@reedsyllas regarding your points

  • We calibrate decay against a 60 Hz reference using the inertia Math.pow approach you describe. Math.exp(-decay * dt) is the marginally faster equivalent, however switching means remapping the user-facing constant to keep the feel identical. Could be worth revisiting now that it’s a localized change inside CameraMovement.
  • Agreed that while half-life may be a more intuitive concept, inertia / panningInertia are public API we must keep. However CameraMovement is the seam to add half-life or decay as an additional knob mapping onto the same internal decay without breaking backwards compat.
  • The input layer still works in accumulated deltas, so the event-rate vs frame-rate mismatch is real. The absolute alphaTarget model is appealing, while a bigger change (external code reads/writes inertialAlphaOffset etc.), but CameraMovement lets us explore it as an alternative mode rather than a rewrite.

Please feel free to open a PR building on CameraMovement to enable any of the above, thanks for raising this!