Upcoming BabylonJS-powered MMORPG Closed Beta! [heroboundgame.com]

Demos and projects
, , ,
1

After a year of full-time development, it is time for the first semi-public release of HeroBound: The Epic Web-Based MMORPG!

We’re looking for anyone who might be interested in participating in the closed beta of HeroBound (Nov. 1 - TBD) to help provide feedback for the game’s ongoing development. If the game looks appealing to you, or even if you’re just a fellow developer looking to learn about the technical details behind implementing a browser-based MMO, we’d love for you to join us.

Edit: Previous issue with the form not validating the length of the responses correctly has been fixed.

For those interested in some of the technical aspects (since I’ve been absolutely terrible at sharing the process regularly on the dev blog), here’s the brief overview:

The game client is built using BabylonJS 7 and React, while the back-end is primarily Bun-powered Typescript. There are, however, several import factors in the game’s architecture that have allowed us to hit nearly 50,000 active player instances on a single server (utilizing under 50% of the server’s CPU), and several thousand rendered players in a client’s immediate region. This was done by utilizing every bit of performance optimization provided by Babylon for rendering (thin instances, baked animations, etc), creating custom web-assembly modules for handling heavy processes and data structures such as pathfinding and priority queues, and not least of all, taking full advantage of binary web socket streaming.

Side note: The µws library blows away any other TCP implementation, and on Bun, the performance benchmarks in a single thread were nearly the same as a multi-threaded C++ server.

As the goal has always been accessibility, there were also some graphical and mechanical tradeoffs made in favor of performance. One important rendering optimization was having the player (and some NPC) models synchronize their animations, so that they can all share a single animated skeleton. While not aesthetically pleasing, this was required to be able to attach equipment meshes to the animated skeleton, without needing a separate skeleton instance for each animated entity. Another solution would be to bake equipment animations, and perhaps that’s something that will be revisited in the future, but it was too cumbersome a solution this early on when the equipment models are frequently added and changed. Additionally, an important mechanical tradeoff was to represent the underlying game world as a 2D grid, which loses some immersion and opportunity for fun features, but gains an incredible performance boost to things like pathfinding. As it stands, the NBA* pathfinding WASM module is so fast, that generating a 64-cell path around dynamic obstacles is 900x faster on the client than the most optimized navigation mesh solution, and on the server, the average pathfinding operation takes less than 10ns!

That’s 0.00000001 second!

This does not necessarily mean that the game is unable to accomidate 3D features (which is already not the case), but it means that unless there is a large architectural change, the game cannot support native 3D elements like 3D paths or obstacles, and instead have to be simulated from a 2D origin.

And while we’re on the subject of the server, let’s talk about the scalable network. There are two primary servers that provide us with the full MMO network: The World Manager, which communicates with the authentication server and synchronizes data between world instances, and the World, of which there can be an infinite number of instances. A World instance, as you might expect, holds its own copy of the game world. Each 2D grid of the world (of which there can be several representing separate isolated areas of the game world) is split into 32x32 sectors. Initially, a World instance is responsible for the entire game world, and under load, begins to divide each grid into its own managed instance, which can further divide down to a single sector (although this is highly unlikely to ever occur). When a player first joins, his or her authentication token is sent from the world to the manager for validation, is provided the persistent player data in response, and is then subscribed to each sector’s channel, within a 2-sector radius. If that channel happens to be offloaded to another instance, the channel is relayed from the primary instance. And likewise, when the player moves to a new sector, their sector subscriptions are updated, and stale entities dropped, only ever maintaining data (and rendering) that 5x5 sector radius.

Admittedly, because RAM is cheap (and amazingly fast on Apple ARM machines – used by our servers), and the back-end is optimized to cache everything to keep CPU usage exceptionally low, the server instances had to be artificially restricted just to test the autoscaling feature, because even at 50,000 simulated, networked players, the initial, single world instance never hit the threshold required to subdivide.

The process has been LONG, and it’s still so far from anything resembling a “complete game,” but it’s slowly getting there. Having to build so many features from scratch due to either not being available or not being performant enough has been extremely time-consuming, but also an invaluable learning experience, even for a seasoned developer. From the ECS engine, to the world builder, to the entity and animation editor, to the custom binary serialization/deserialization module, so much heavy lifting has already been completed. (…If only it felt that way :sweat_smile:).

I’d be happy to go further into any details, so if you have a question, don’t hesitate to ask. Hoping to see some new faces come November. :slight_smile:

13 Likes
2

This looks great! The fact that you’ve been developing this for under a year is super impressive considering everything present in the information and the game trailer content. Speaking of which, you should definitely put your game trailer at the top of this post in my opinion!

(utilizing under 50% of the server’s CPU)

Impressive numbers, what are the server specs? And is the provider cheaper than Linode? XD

One important rendering optimization was having the player (and some NPC) models synchronize their animations, so that they can all share a single animated skeleton.

I’d love to learn more as I’m currently trying to optimize the rendering performance in my own game. When all characters are on screen there’s 600 active bones which is my current bottle neck. I’ll also give a shot at baking VATs in game (as you’ve implied) but your current system sounds like it has some flexibility. The character animations in your trailer were satisfactory for sharing a skeleton.

1 Like
3

Thanks for the kind words, @cyborg_ean!

Impressive numbers, what are the server specs?

Interestingly enough, the game server chosen for the beta and initial launch is a Mac Mini. It sounds strange, especially since there is no headless version of MacOS (anymore), but because of the way the server is implemented (using lots of RAM and focusing on single-thread performance), it runs MUCH faster on Apple’s ARM architecture than even the best commercial x64 server hardware, due almost entirely to the unbeatable access speeds of its shared memory.

Clearly this option wouldn’t work for everyone, but already being equipped with the necessary network bandwidth to host thousands of connected users, and Apple’s 0% financing, it made more sense to just buy a mac mini outright for the initial launch than to pay to host it elsewhere, for now, at least. Obviously this won’t scale, but if there are too many players for this one server to handle, or enough interest from other locations in the world that I need to host servers offsite, that’s not a bad problem to have! It also won’t be a very hard one to solve since the CI/CD tools are already in place.

[Edit: Forgot to mention that another consideration that lead to this choice was that the low power consumption. Only uses around 40w on average – with a max of 185w. So very cheap to run.]

It’s also worth noting that we’re using Cloudflare to relay connections, so we don’t have to worry about a ddos attack taking down the server. At most, they’ll be able to take down one relay node, causing anyone else who happens to be connected through the same node to experience a 5-10 second hiccup as they reconnect through a different one. This is another benefit of the mechanical tradeoff of using point-and-click grid movement – the slight increase in ping from using commercial ddos protection isn’t even noticeable. If the game had free movement with WASD keys, for example, which would already feel sluggish since the web does not support UDP sockets (yet), adding the additional proxy could be a game-killer.

I’ll also give a shot at baking VATs in game

Absolutely do this (but don’t actually bake them in game, bake them externally and import the baked animations). The complexity only comes into play if you need to attach to bones of the skeleton. Which is still doable, but just requires a bit more thought to implement properly. This thread was very helpful for getting started with the VAT baking: Baked Texture Animations with Animation Groups? - #38 by Evgeni_Popov

If you run into the same issues I did when it comes to managing baked animations (playing in sequence, stopping when it’s complete, etc.), find me in the HeroBound discord and I can help you out.

1 Like
4

This sounds so epic! :open_mouth:

1 Like
5

WoW. That looks pretty amazing especially considering the ‘just’ 1Y development :open_mouth:

I don’t want to show ‘selfish’ but I’d be very interested to hear a little more about your pathfinding solution.
I’m sorry to ask but “What’s the NBA pathfinding WASM module”? I’m currently looking into a base for a suitable pathfinding solution for a TD game and, your numbers look impressive. Could you tell me a bit more about it? No rush, I’m still just in the design thinking process :sweat_smile:
Again, congrats for this huge making. I cannot commit to the beta at this moment but rest assured I’ll be keeping an eye on this :child: :face_with_monocle: :grin:. Meanwhile, have a great weekend :sunglasses:

1 Like
6

Don’t apologize; it’s a good question. So, essentially, I had implemented an NBA* pathfinder in javascript years ago, but for this project, it was re-implemented in Zig and compiled to wasm. (NBA* = New Bidirectional A-star; WASM = Web Assembly). NBA* works very similar to the common A* algorithm, but instead of working from one end to the other, it starts crawling out from both ends of the desired path and looks for intersections.

However, for a tower-defense game, you may be fine using any pre-made A* pathfinder, and just tweaking it as needed. Provided you can get away with just running the pathfinder any time a player updates his or her map, updating available enemy paths instead of running the pathfinder for each enemy, pathfinding shouldn’t be a bottleneck for you. You’ll want something that supports tile weights so your pathfinder takes into account things like traps that slow down enemies when calculating the “shortest” route. This might look like a 2D array of numbers (number[][]) where 0 means the tile is blocked, 1 is open, and 0.5 might be where the player has put down sand to slow down enemies. This way your enemies aren’t too easy and predictable always taking the path with the fewest amount of cells to traverse, regardless of traps and slow-downs. Additionally, you might want to eventually write your own scoring function so you can account for things like how well a cell is defended, so your enemies will be smart enough to take a longer route to avoid defenses.

This does all make the assumption that your game operates on a 2D grid. If not, you will need to implement your pathfinder on a graph, and it will take a little more work on your part to build that graph from your world (A* works perfectly well for graphs too). But if this is the case, definitely try out the RecastJS plugin for navigation mesh generation and compare the performance, because it may be a lot simpler to use without being too burdensome for your game: Babylon.js docs

Anyway, I’m not sure exactly how versed you are on the subject, so some of this might be way too basic for you, but in case any beginners are interested in pathfinding, here are some good resources.

Introduction to A-star pathfinding:

A typical javascript A-star implementation using a binary heap:

The EUR research paper on bi-directional A-star:

Hope this helps!

3 Likes
7

Looks really impressive!
Could You elaborate a little bit more topic related to world instances? I’d like to understand how it works. I’m a little bit confused. So far i have two questions related to it. maybe it will help me ask more questions :stuck_out_tongue:

How world manager sychronize data between world instances? What data needs to be sychronized?

How did You manage to synchronize data and perform interactions between entities which are in separate world instances (in neighbour sectors)?

Do You plan to use histeriesis for sector radius subscription?

1 Like
8

Currently, the only data that persists between the server instances are the player entities (e.g., player location, status, inventory, “bank” inventory, etc). A user can log out of one world, and log in on another, preserving their state (more like Runescape rather than WoW).

The process of data synchronization is fairly simple: the instance itself will cache its own world state periodically, so if there is some unforeseen catastrophic failure, the world state can be still restored to the last save. Additionally, when a player is disconnected (or the server shuts down) the entire player data blob is sent back to the world manager to be stored in the database.

GREAT question! The interactions from the player are always sent to the primary world instance, which bubbles the event down to the affected sectors (whether they are managed locally or offloaded to independent servers). To use a basic example, let’s say a user is in one sector (A) and clicks to attack an entity that resides in an adjacent sector (B), and the server has subdivided to the point that each sector is running on its own instance (worst-case scenario). The Attack interaction is dispatched to the server with the target ID and the ID of the player who requested it (among other data to ensure the request is authentic). For most interactions, the case of needing the state from two separate instances to validate an action is handled by the event receiver (primary world instance) assigning an ID to the interaction event and indicating that it requires another check to be valid. Then the event dispatcher (which in this case is both the dispatcher of Sector A and the dispatcher for Sector B) notify and wait on each other before dispatching the update reduced from the interaction to their respective subscribers, assuming both report success.

However, for movement, specifically, we cheat a little bit by always handling the pathfinding on the primary instance, and relying on sector instances to send path-weight updates back to the main thread any time they change so its pathfinding grids† are always in sync. Once the path is generated, the update is sent down and broadcasted from the player’s sector. Likewise, when a player transitions between sectors, simple add/drop events are propagated down to the next and previous sectors.

†A pathfinding grid in our implementation is literally just a large buffer / Uint8Array that uses pre-calculated bit shifts to index as a two-dimensional array. This is one of the reasons the pathfinding operations are so much faster than anything else on the web. Doing this: grid[(y << SHIFT) | x] is significantly faster than doing this: grid[y][x] whether in javascript or web assembly. In JavaScript, you can safely use this method for grid sizes of up to 32k x 32k (although that would be a 1 GiB array if each cell is only 1 byte). To put this in perspective, the entire game world of Runescape (RS2 from back in the day, at least) was under 2k x 2k tiles, which in our implementation would translate to a 4 MiB pathfinding buffer – so it’s hardly a concern.

As far as the actual technical implementation, all of the server instances communicate to each other through binary TCP sockets, just as the client communicates with the primary instance (for global updates) and each sector instance.

I’m not really sure if I understand your question correctly (and had to actually google Hysteresis in the context of CS, because I previously only associated it with EMag :sweat_smile:), but if the question is about how the sector subscriptions are calculated and implemented, it’s just a very simple iteration from sX-2,sY-2 to sX+2,sY+2 initially, and then transitions are managed by unsubscribing from the out-of-range 5, subscribing to the newly in-range 5. (Each of which may require initializing new websocket connections if the server has actually subdivided to that extent, but as of now, such a case is highly unlikely).

Hopefully that makes sense?

9

Thanks a lot for a detailed answer! I need some time to digest it :slight_smile:

Regarding histeriesis as I understand player has visibility of 5x5 = 25 sectors. But imagine situation that some entity is on the edge of sector and while moving it can leave/enter sector several times (even in one second) this will cause overhead of removing entity and replicating whole data several times (of course it can be optimised eg by comparing hashes of visual state of entity).
The idea of histeresis is to avoid this on/off situations by adding additional subscription ‘layer’. Entity will subscribe for entities entering sectors in ±2 range and unsubscribe to entities with ± 3 range.

1 Like
10

Ah, yeah, that’s a good point. Another way this case could be handled is to debounce the drop sector and drop entity updates so there is some tolerance.

We’ll have to do some more testing to see if this becomes an issue. So far, even with a densely-packed region, the sector updates have been lightning fast thanks to all of the data transfers being packed binary, instead of stringified JSON. Large network payloads also don’t affect the FPS very much since the socket client runs in a web worker (separate thread).

11

Many thanks for all the details and for vulgarizing this part for my small brain :brain: :sweat_smile: :rofl:
Currently, ‘incidentally’ as I should say, the astar from brgrins is actually also my base :smiley:
And my making/logic is (I suppose) similar to yours (in the sense that it is also based on a 2D grid and cells (which I believe to be appropriate for a TD game that’s based on a ‘dynamic’ maze - towers can be build to ‘block’ enemy path and force them to move around - while always attempting to take the shorter path). Clearly, I don’t quite have the same load than you have, with the same number of requests for pathfinding. Still, I believe I will have at the average about 15-30 player units and anything between 15 to 30 enemies (per wave). Knowing that a wave can be called-in early or start while the previous is not yet all dead, I believe there could be something like 80 units on the battlefield at a time. So, when I include all of the context/gameplay that will also have ‘melee’ units moved around on the battlefield, it still makes for quite a heavy number of requests for a number of different pathfinding. Since I’m a total ‘noob’ with this (and even whilst I’m attempting to create just a pilot/proto… this gets me a bit worried :sweat_smile: :joy: So I’m trying to feed myself with all I can collect (and understand :grin:) before making a choice for this part that will obviously be a core of the game design.
For this, I cannot thank you enough :pray: for your input and all the details: THANKS A LOT :hugs: and until we chat next, have an awesome day :sunglasses:

EDIT:

WoW. In fine, I might not be the worst of noobs :stuck_out_tongue_closed_eyes: :joy: That’s exactly the logic I have at the moment (incidentally :laughing:) However, I’m still thinking of adding another level of ‘weight’ that could be set on tower placement to early evaluate just the neighbooring cells, to check just quickly on the neighbooring cells whether the path may be blocked or remains open (without making yet another call for pathfinding). Would you say this is a ‘reasonable’ approach?

1 Like
12

Nice, gonna steal this :money_mouth_face:

1 Like
13

Definitely keep this piece in mind. For a tower defense, all of your units should be able to use the same pathfinding result, which only needs to change if the grid changes in a way that would affect the paths. So rather than running pathfinding for each enemy, just run pathfinding whenever the player places a tower (for example), and have all the enemies use that path. Or if there are multiple spawn points, one for each spawn point.

If there really is a legitimate case for needing separate paths because certain kinds of enemies use different scoring methods, then you still only need to generate the additional path for that kind of enemy to share, and not for every instance of the enemy.

The only time I can think of that multiple paths would really be needed would be when the first path is cut off while enemies are traversing it. In this situation, I would probably just generate up to 4 additional paths, one that starts at each adjacent cell that the newly blocked cell touches (if its open) going to the destination, and then updating the enemy paths to be a slice of one of those paths starting at their current cell, assuming their current cell is no longer a part of the primary path (from its spawn point).

You may begin to consider optimization tradeoffs like we made for HeroBound, where it may prevent certain features that would be nice to have, but allows you to optimize the game a lot more. Some examples of these might be only having a single enemy spawn point, or not allowing the user to place new towers during active waves. But if you definitely don’t want to go that direction, and your design has necessitates enough pathfinding calls to become a burden on the game loop, you will probably want to move the finding to web assembly, and perhaps even a worker thread.

If I understand correctly, and that you mean you will only re-run pathfinding if a tower is placed in a cell that currently exists in the enemy path, then yes, I think that’s a reasonable optimization. I don’t think it requires an additional weight value though.

1 Like
14

Wow. Thanks again. I will need some time to digest and evaluate all of this :dizzy_face: :sweat_smile:

However, the way you’re breaking this down for me is of GREAT HELP. Thanks again :pray:

Yes, indeed. That’s my current scenario.

It isn’t about the enemy here. It was more about player ‘melee’ units. They can be placed on the battlefield and have a certain range from their spawning position. But you are right here (again) these units do not need a pathfinding. Since it’s based on cells (and each cell already has identifiers to whether it is walkable/buildable, etc…) they do not need the pathfinder. As they are simply moving towards the enemy when the enemy comes in range - and returning to the origin position when no enemy is in range).

Yes, thanks. I’m still working my head around this part. I suppose I will eventually start running just the base to better understand the misses :grin: I’m more of a designer and a very ‘visual’ person :joy:

No, it doesn’t. Of course, it doesn’t. I realized this just a few seconds after typing it :sweat_smile: Stupid me :joy:

EDIT: Don’t reply to this post of mine. I think I’ve already overtaken this thread enough with my own little thingy :face_with_hand_over_mouth: If I have more questions for you, I’ll ping them from the thread I created for this purpose. Sorry for creating so much ‘noise’ in the showcasing of your demo :pray:

1 Like
15

Honestly, don’t worry about it. This stuff is WAY more enjoyable than the icky promotional posts I’ve been doing over the weekend. If self promotion wasn’t absolutely necessary to get the word out about independent games, I would never do it. But I’d like to continue working on this game full time, which requires getting some revenue generated next year – or finding a really good argument that convinces my wife that a house is just unnecessary luxury. :sweat_smile:

2 Likes
16

haha. She may be right in a way. You need a shelter of some sort… a tent, a cave… and thinking of it, I’m not too sure about getting broadband Internet in a cave :rofl: :joy:

1 Like
17

Quite impressive work!!!

2 Likes