Cant Get Havok Simulation to Run at the Same Rate Per Client

I have tried multiple different ways to get a Havok simulation to run consistently between clients. I have the delta being applied to all the forces and have even tried dynamic time scaling on the simulation.

This thread: Ensure that the forces being applied with Havok are the same for different clients without relying on deltaTime? was posted about it in the past but for some reason I can not get things to run consistent.

I was hoping people could take a quick glance at the sections of code responsible for the application of the forces and delta time and see if they notice a problem with the setup.

Input Handling:

private _handleInputs(delta: number) {
    this._isBreaking = false;
    // if (this._player.state !== GoKartPlayerStates.RACING) {
    //   return;
    // }
    if (this._inputs.forward && !this._inputs.back) {
      if (this._forwardForce >= 0) {
        this._forwardForce = Math.min(
          this._forwardForce + this._accelerationForce * delta,
          this._maxForwardForce,
        );
      } else {
        this._forwardForce *= this._breakingFactor;
        this._isBreaking = true;
      }
    } else if (!this._inputs.forward && this._inputs.back) {
      if (this._forwardForce <= 0) {
        this._forwardForce = Math.max(
          this._forwardForce - this._accelerationForce * delta,
          this._maxForwardForce * -0.5,
        );
      } else {
        this._forwardForce *= this._breakingFactor;
        this._isBreaking = true;
      }
    } else {
      this._forwardForce *= this._decelerationFactor;
    }
    if (Math.abs(this._forwardForce) < this._eps) {
      this._forwardForce = 0;
    }
    if (this._inputs.left && !this._inputs.right) {
      this._turningForce = Math.max(
        this._turningForce - this._turningAcceleration * delta,
        -this._maxTurningForce,
      );
    } else if (!this._inputs.left && this._inputs.right) {
      this._turningForce = Math.min(
        this._turningForce + this._turningAcceleration * delta,
        this._maxTurningForce,
      );
    } else {
      this._turningForce *= this._turningDecelerationFactor;
    }
    if (Math.abs(this._turningForce) < this._eps * 0.1) {
      this._turningForce = 0;
    }
  }

Force Application:

private _handleForces(delta) {
    this._isSkidding = false;
    if (this._isGrounded) {
      const vel = this.body.getLinearVelocity();
      const nVel = vel.clone().normalize();
      const nSpeed = vel.length() / this._maxSpeed;
      let turnDampen = Math.max(0, Math.min(1, nSpeed));
      if (turnDampen > 0.5) {
        turnDampen = Math.pow(Math.sin(turnDampen * 3.14), 1.4) + 0.5;
        turnDampen = Math.min(1, turnDampen);
      } else if (turnDampen > 0.0125) {
        turnDampen = Math.pow(Math.sin(turnDampen * 3.14), 0.5);
      } else {
        turnDampen = 0;
      }

      const dragFactor =
        nVel.length() === 0
          ? 0
          : 1.0 - Math.abs(Vector3.Dot(nVel, this.chassis.forward));
      if (dragFactor > 0.005 / nSpeed) {
        this._isSkidding = true;
      }

      this.body.setLinearDamping(Scalar.Lerp(5, 15, dragFactor * dragFactor));

      const alignQuat = Quaternion.FromLookDirectionLH(
        this.chassis.forward.scale(-1),
        this._rayCasts.center.results.hitNormalWorld,
      );

      this.chassis.rotationQuaternion = Quaternion.Slerp(
        // @ts-expect-error is always set
        this.chassis.rotationQuaternion,
        alignQuat,
        10 * delta,
      );
      const dir = Math.sign(this.currentVelZ);
      this.chassis.rotate(
        this.chassis.up,
        this._turningForce * delta * turnDampen * dir,
        Space.LOCAL,
      );

      const force = this.chassis.forward.scale(this._forwardForce);
      this.body.applyForce(force, this.chassis.position);

      const newVel = this.body.getLinearVelocity();
      if (newVel.length() > this._maxSpeed) {
        this.body.setLinearVelocity(newVel.normalize().scale(this._maxSpeed));
      }
    } else {
      this._isSkidding = false;
      this.body.setLinearDamping(0);
      const alignQuat = Quaternion.FromLookDirectionLH(
        this.chassis.forward.scale(-1),
        this._rayCasts.centerAirborn.results.hitNormalWorld,
      );
      this.chassis.rotationQuaternion = Quaternion.Slerp(
        // @ts-expect-error is always set
        this.chassis.rotationQuaternion,
        alignQuat,
        6 * delta,
      );
    }
  }

I know its hard to debug when just looking at code, but I’m hoping someone sees something sticking out.

Would using the fact I’m using deltaSeconds vs deltaTime be the reason this is not working? If I switch if over to deltaTime instead the whole system freaks out.

It’s been a (very) long time since I worked on network code but I think some principle still applies.
You can’t, with best effort possible, make sure that with same preconditions you’ll get same results.
Things are as complex and crazy as processors might not have the same result with float because of FPU implementation.

Latest game netcode feature mecanisms like rollback to hide latency

It means you’ll have to keep some physics states.

1 Like

One of those tracks elapsed time since engine started rendering the other tracks time between individual frames.

Could be worth checking that you’re using the proper delta figure. Maybe you are passing in render (frame) delta but really should be passing the physics delta? It’s not clear if you’re using deterministic lockstep either but for sure try that to see what differences pop up.

I thought the physics delta was the same as the scene delta? How do I grab the physics delta?

I don’t see anywhere in the docs where you can grab the physics delta.

Sorry, haven’t been at my computer yet where I can look up an example for what I’m thinking - if you’re using deterministic lockstep then the physics tick rate is set independently of the render tick rate

2 Likes

Definitely curious about this!

I’ll put together a PG to demonstrate more of the time delta stuff differences tomorrow, but the simplest means of getting the physics delta is just using the lockstep rate you pass into the engine constructor, ex: if my physics step is 0.2, the physics engine will update every 200ms, or 5/s. That’s obv too slow for your purposes, but in this simple example your physics delta is a constant 200ms*

*of course, in non-deterministic scenarios the refresh is variable, and there are also cases where the engine may be called at an interval greater than the lockstep, but I’ll hopefully have some time to put some code up in the morning

Ok here are some details and (ed) some observations:

There are a set of methods on Engine (not the physics engine) that expose different time values, and while I’m not entirely sure if these are wrappers around the physics plugin’s properties/methods, they are worth knowing:

  • engine.getLockstepMaxSteps()
  • engine.isDeterministicLockStep
  • engine.getTimeStep()
  • engine.startTime

The default is 1/60 (e.g., 60 updates per second) but again most of these are only applicable when using deterministic lockstep. If you aren’t, then there’s no way to guarantee two clients will run at the same rate due to differences in frame rates and floating point-arithmetic, as Cedric mentioned earlier.

While on the Havok plugin (and this is likely more relevant to you) there are:

  • hk.executeStep(delta, [bodies])
  • hk.setTimeStep(delta);
  • hk.getTimeStep();

The executeStep function is important if you want to perform forward- or backwards- prediction of a simulation’s state (e.g., to see whether a bullet intersects a player in the next n frames) or if you want manual control over when physics runs.

Also, the constructor parameter for the Havok plugin useDeltaForWorldStep can be set to true (it is false by default obv). TBH, I’m not entirely sure what exactly that parameter does, but my assumption is that it has a similar effect of turning on deterministic lockstep. The docs here seem to be lacking a bit since I also couldn’t find a description for the plugin’s configuration settings and such. Flagging this for @PirateJC

EDIT: I just realized that Ieft out some notes on the code you posted upthread –

  • have you tried using applyImpulse instead of applyForce? It seems counterintuitive, but I also haven’t done extensive testing on Havok between the two
  • Going back to your orig. question, deltaSeconds and deltaTime might be scaled differently - ms versus s for instance. Divide deltaTime by 1000 and it should match deltaSeconds (assuming that you’re not comparing startTime ofc)

HTH!

I thought the useDeltaForWorldStep was default true, so maybe that was the step I have been missing.

Otherwise I was already utilizing delta on everything and thought I was using deltaSeconds correctly which from the sounds of it I was. Already knew about the prediction thing.

If it ends up being that useDeltaForWorldStep true is the only thing I was missing Im going to cry…

Thanks for the input.

1 Like

Just an additional warning as the information we can find on the internet indicates Havok is a stateful engine and it’s not so easy to rollback and replay. Citing the doc for Havok for Unity:

How does Havok Physics for Unity work with networking use cases since it’s not stateless?

Havok Physics for Unity is a deterministic but stateful engine. This implies that a copy of a physics world will not simulate identically to the original world unless all of the internal simulation caches are also copied. Therefore, for networking use cases that depend on deterministic simulation of a “rolled back” physics world, you should rather use the Unity Physics simulation since it is stateless.

I’m not aware whether the babylon version is different but I doubt so.

2 Likes

Looking at the code, useDeltaForWorldStep defaults to true.
When it’s set to true, the delta is taken from the render loop and when it’s set to false the fixed timestep is being used.

What’s unclear for the second case is that the physics engine’s steps are still controlled by the Scene render loop anyway: moving up the callstack here and here and here. Wait, not if deterministic lockstep is enabled.
So it looks, as stated in the doc for animations, that deterministic lockstep is the way to go (setting useDeltaForWorldStep to false is not enough), but you will also lock the framerate for your animations.

I personally dodged all this potential mess and used the trusted external loop library (with clear render/physics separation) I have been using for years instead of babylon’s engine renderLoop.

1 Like

Interesting findings and nice sleuthing there SerialF!

I guess I got the meaning of the useWorldForDelta inverted then, and I certainly didn’t know about the dependency on the animation lockstep, but in retrospect it makes more sense. Feels a bit clunky though, so I can see why you’d want to use an external render loop

So with this it added a ton of jitter to the simulation, is there a way to smooth that out?

When I’ve had that happen to me in the past, it’s been because I’ve been mis- timing or mis- applying custom forces in collaboration with the physics engine.

Have you tried putting your update logic into the Scene.onBeforeStepObservable or alternatively the onAfterStepObservable? Note that ofc only fires when deterministic lockstep is enabled. Otherwise, I would check that you’re applying your logic pre- render (I’m sure it is, but sometimes things. get messed up without noticing)

yeah its for sure because of the substeps, I ended up doing a onAfterRender update for the position but that did not fully fix it. Probably because I was pointing to the physicsObject.position and not doing set or clone from that. Now that I have thought about it more Im pretty sure that is it and will go back to fix that!

Hmmm actually that did not fix it, there is still jitter interesting.

I have a solution that uses Lerping to smooth it out, but its more of a work around.

I just Thought of something else to try - after you make your changes to the position and such, recalculate the mesh’s’ world matrix and/or add a 0-duration setTimeout. Alternatively, see if there are methods for synchronizing the parameters to the physics sim.

HTH!