Large frame rate drop in scene on a hosted webpage versus running on localhost server

[arcman7]

The problem:
When I deploy my project code to a colyseus hosted web page:
https://g7r7sz.colyseus.dev/client/index.html
the frame rate drops significantly from the frame rate that I have when running the exact same code on a localhost development server. The only difference on the live deployed version is that it get’s minified by whatever minification process the npx module utilizes.
I initialized the project like so:
npx create-react-app my-app
I create a production build using:
npm run build
Then I just push that compiled code to colyseus for deployment. I’m having a brain fart moment here trying to guess at why this framerate drop occurs.

Here’s a screen shot from localhost:

Here’s a screen shot from the live demo:

One thing that I noticed is that the mesh selection takes far longer on the live demo than on localhost, but I have no idea why, and that doesn’t explain the full difference in frame rate.

[bghgary]

Maybe capture a performance trace for both cases and see if anything stands out?

[jeremy-coleman]

i did a quick profile and a mousemove is taking 6ms, plus a 50 ms function call inside every raf which seems to originate in your player update animations and ultimately in the then engine raf callback. I took a look at your player updates and its not really doing much except for calling the bt transforms.

check this out first

Performance hit with scene pick and mousemove

This is done by the system to see if some meshes are pickable and need to trigger their actions (onPointerMove). This is the predicate used to select if a mesh needs an action: Babylon.js/scene.inputManager.ts at master · BabylonJS/Babylon.js (github.com) So if you have a lot of meshes in your scene the system will go through all of them on every pointer move. I will introduce an option on the scene to skip that feature entirely if you do not need it

tldr: try doing scene.skipPointerMovePicking = true

if that doesnt work, take a look at the player updates:

_beforeRenderUpdate(): void {
this._updateFromControls()
this._animatePlayer()
}
^^^^
(this is what’s calling into thin engine and calling whatever is running 50ms per raf)

i’d check this first. also, i think you’re calling this twice in each input impl. like for keyboard, line 664 and again on 767.

  this._kinematicController.playerStep(
      this.game.physicsWorld,
      this._deltaTime / 1000,
    )
    
    tran = this._kinematicController.getGhostObject().getWorldTransform()
    tranPos = tran.getOrigin()

so, either the kinematicController.playerstep or the world transform. could be the world transform if you’re using a bt heightmap for that huge ass terrain? not sure how all that hooks up.

[arcman7]

@jeremy-coleman This is fantastic, these are great places to start looking. I guess I got a little too caught up on viewing this as a problem of localhost vs live deployment. If your suggestions do lead to either an improvement or full on results in equal framerates then hypothetically there’s some race conditions here causing certain async functions to be called at the wrong times or multiple times, and these race conditions for whatever reason do no occur on localhost. Does this seem reasonable to you?

I’ll start trying out your suggestions and let you know!

[jeremy-coleman]

Not a race condition, but it could be network related since its the player update functions, like if you’re awaiting a state update or ack inside the raf / scene beforeRender calls. Maybe try to rule that out by running only the client locally and connect to the hosted backend (if thats not already the case). I checked that before doing the profile with a control f for “await” but didn’t see anything sus , but i also didnt really see much network related stuff … little hard to find stuff in devtools lol.

[arcman7]

To your point, the only network related stuff get’s setup in an on event call back, and even then it’s just a rough implementation of storing other players previous position such that their mesh will be animated (Vector3.lerp) to their next position. So everything else is just running as fast as the javascript runtime event loop will allow. Also, that part of my code is not well organized as of yet so I’m not surprised you didn’t find it lol.

I’ll have some updates for you tomorrow, it’s currently 1:23 am here; night!

[Evgeni_Popov]

Are you able to test by deploying a non minified version to the server? I recall I also noticed some significant differences in perf between a minified and non minified version when doing some testings some time ago (can’t remember what I was testing, though…).

Also, are you able to test on different browsers?

[arcman7]

@Evgeni_Popov I assume by different browsers you mean a non-chromium browser such as Firefox?

Firefox live deployment:

Firefox localhost:

It’s better, but still a large difference. I’m skeptical about the GPU frame time being zero ms .

[Evgeni_Popov]

Maybe Firefox is not able to get the GPU time correctly.

I think the biggest problem in your first screenshots is that GPU time is around 11ms in local and 18ms in production. You should check with Spector that you are really doing the same rendering in both cases, you should not get a so massive difference (actually you should also get around 11ms on the production server).

[jeremy-coleman]

I think that’s correct, GPU memory consumption is super low

It’s not rendering perf, its definitely something synchronous running inside the render loop related to player / input updates. IMO, what you should do is comment out those functions i linked above 1 at a time and try to find what exactly is causing the problem. Considering what Mr. Popov said, does the server send the client networking related code? like inject the bit packing schema into the client or something

[Charge]

Just to confirm, are you comparing running the production build locally to the production build remotely?
I ask because my own react app with Babylon ran much slower after building the production version because the dev version (run with yarn start for me) was configured to support the last 2 versions of chrome in the package.json but the production build was set to support a lot of different browsers.
This caused the biggest slowdown when using web workers.
Like I say, this may not be your issue, but it may be worth changing package.json to support the same browsers for both development and production

[arcman7]

@Charge Yes; I will try that!

@jeremy-coleman I took some perf logs, but it still doesn’t reveal anything for me. I can see that there is a javascript function being called that takes a lot of the time between frames, but I’ve tried stepping through the code to figure out where, and I’m still not having any luck. I’ll keep you posted if I find anything note worthy

[arcman7]

@Charge right now it’s looking like you nailed the cause of the problem; in my package.json file I changed it from:

        "browserslist": {
          "production": [
            ">0.2%",
            "not dead",
            "not op_mini all"
          ],
          "development": [
            "last 1 chrome version",
            "last 1 firefox version",
            "last 1 safari version"
          ]
        }

to:

  "browserslist": {
    "production": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ],
    "development": [
      ">0.2%",
      "not dead",
      "not op_mini all"
    ]
  }

The framerate performance on localhost now looks identical:

I’m going to revert and change both dev and prod builds to:

        "browserslist": {
          "production": [
            "last 1 chrome version",
            "last 1 firefox version",
            "last 1 safari version"
          ],
          "development": [
            "last 1 chrome version",
            "last 1 firefox version",
            "last 1 safari version"
          ]
        }

Given that this game is still just in a prototype phase I do not see the value in trying to support most browsers. Even it were at some sort of initial release I think it’s fair to ask users to use certain browsers so that they get substantially better framerates on a free game.

EDIT: Wow. That did it, the live demo is now performing the same as localhost:

Thanks everyone for the helpful tips, and thank you @charge letting me know about the targeted browser support having such an impact on performance!

[bitbybit-dev]

Wow, I had this issue for a while now on two react projects and this finally solved it. I did not have time to dig deeper into the causes and thus kept those projects on Babylon v5.21.0 as it seemed to still work with old react settings after deploying to prod. While I am happy that this is finally resolved on my projects I still wonder what happened during 5.21.0 path to 5.22.0 which makes this package.json setting relevant. At least in my projects v5.22.0 began those huge mystical frame drops. Few months ago I was digging in the change logs, but didn’t see anything special in 5.22.0… Anyway, THANK YOU GUYS for giving a solution!

[arcman7]

I’m with you there, I will always wonder what caused it, but I can’t justify a ton of time investigating it atm. Super glad this helped you!

[inteja]

@bitbybit-dev

I still wonder what happened during 5.21.0 path to 5.22.0 which makes this package.json setting relevant

This is interesting. It may be unrelated but we’ve noticed some significant new performance issues in our Babylon React Native app in recent weeks. We’re assuming it’s something we’ve done but this indicates there might have been an issue in the upgrade to 5.22 and beyond?

CC @BabylonNative

[sebavan]

cc @RaananW could it fit with the change of target ???

[bitbybit-dev]

I suppose if you downgrade to 5.21 and problem goes away + upgrade to 5.22 and it comes back, it’s a good chance it’s somehow related All versions above 5.21 that I tested had the same issue on production build in our projects. I must say performance monitor in chrome didn’t show much… I’m also no expert in inspecting what happens while frames are being rendered. I remember back then not being able to formulate my issue well for others to reproduce so I did not report it at all as I thought it might even be reacts fault or smth…

[RaananW]

changing the target should not hurt performance (more improve TBH), but the downgrade might be the reason behind it. I am not sure how the downgrade works under the hood, but that might be the issue here. Especially if it changes when you change the browserslist…

[Jack]

Hello, I am experiencing the same issue as arcman7: upgrading to version 5.22.0 or higher (I have tested up to 5.40.0) causes severe slowdowns in my scene. I should mention that my scene is complex and has many textures, and downgrading to a version below 5.22.0 fixes the slowdowns.
Changing the browsers list has no effect for me.

I was wondering if any new information has been found on this issue?

Thank you in advance.