Multiplayer llama world

A boutique digital agency in LA asked if I could help out with some performance problems they'd encountered on the time-sensitive, high-stakes, secret client project they were building for a large tech company. Covid had disrupted the client's annual all-hands meeting, so they'd been commissioned to build an online version where company staff could join in live, communicate with each other in virtual rooms ("Worlds"), watch C-suite presentations together on virtual in-world screens, and chat about them afterwards. All while manifesting in this custom 3D world as ... multi-coloured llamas. Who was I to argue?

The application was at a fairly late stage of development, with some lovely 3D animation work, as well as what looked like a very nicely organised, well-structured and neatly-written node.js backend application communicating with front-end clients over a websocket connection. Deployment process was solid, everything was in Kubernetes on AWS and managed very capably in Terraform.

It all worked just fine - until they hit around 100 concurrent connected users. Everything slowed down after around 80, and pretty much fell apart near 100. Problem was, they needed to handle 6,000. Hence the agency devs needed urgent help on this kind of performance profile, as they only had a few weeks to make it work.

So I set about investigating how the application logic and comms worked, including what was in the database, what in memory, how the app flow worked, what messaging was being done, etc. Once I had a clear picture of all this, I built a testing tool which simulated the entire process of llama avatars spawning, moving, interacting in and leaving the virtual world, and allowed us to gradually increase the number of participants until we started to see problems.

This showed up a few immediate obvious problems: some DB indexes could be tweaked, some redundant messages could be removed - but none of these was going to be a game-changer.

The main issue was fan-out of server-to-llama and inter-llama message notifications. Every time a llama did anything, it notified the server, which then notified all the other llamas, so of course the performance requirements scaled exponentially with the number of llamas.

On top of that, every time any client sent any message, the server was receiving it and then "broadcasting" it by sending it on to every other connected client in a tight loop. The application wasn't taking into account the explosion of CPU and I/O work necessary beyond a certain level, and the single-threaded node.js app was falling over under the volume of messaging, even on some of the larger pod types available.

The notion of re-writing the app to split it into multiple instances and somehow synchronise data between them at this stage in the game was understandably panic-inducing. So we needed a way to jack up the performance without going back to the drawing-board, by whatever means necessary. I knew we needed to get more CPU grunt in there, but the existing architecture made vertical scaling the only option, and taking that much further was looking difficult.

We discussed various approaches based not only on the compute/deploy resources available but also, importantly, the time and availability of key people. Given that there was still a fair amount of feature work to be completed in the application build, we decided to try to maximize the difference I could make in part by using my work to minimize the effort required from other team members.

To do this I proposed writing a simple websocket proxy that would run in sidecar containers for each World, taking most of the processing load away from the app server:

• The proxy would handle the establishment and maintenance of all the downstream websocket connections, manage all direct inter-llama messages in a given instance, and only communicate critical information (such as entries, exits, statuses and interactions) upstream to the node.js server.
• That, along with its writer, would then be freed up to handle database/session management and further feature development.
• Each proxy instance would maintain its client's upstream websocket connections to the node.js server, listening out for specific messages for that client as well as World-wide messages the server wanted to "broadcast" to all clients, and handling downstream comms in all these cases.
• Rather than trying to shove through every outgoing message ad-hoc, the proxy would rationalise the messages by storing them in a memory-based data table and running a "fetcher" process. The fetcher would retrieve batches at a configurable/tunable rate and send them together, significantly reducing the numbers of calls.
• I'd write the proxy service in Erlang/OTP, as it has such a good async I/O story, meaning it can shovel large amounts of data in and out of large numbers of sockets really efficiently, and so is much better equipped to handle lots of really fast messaging.
• Erlang's runtime (the BEAM) can also share larger data blobs efficiently between concurrent processes, passing references to socket handlers so there's a lot less data copying going on - so the batching mechanism meant we could shovel a lot of the data out to the clients with a lot less virtual blood, sweat and tears than the node.js app had had to expend.
• Most importantly, unlike node.js, the Erlang platform could also take advantage of multi-core CPUs automatically. This meant we could get a lot more execution cycles into the mix, and so scale the performance a lot further without needing to re-architect the node.js app for multiple instances.

I worked with the agency's backend developer to rationalise and streamline the message flows, and put together the simplest possible version of the proxy service I could, using cowboy as the downstream websocket server, gun as the upstream client, syn to manage client grouping and notification and jiffy for blazing fast JSON encoding & decoding of messages.

Time spent upfront investigating and diagnosing the problems paid off when the core of the proxy only took a few days, with the main "broadcast" code hot-path moved to the new service, leaving unhandled messages passed through to the upstream for the time being. The effect was drastic:

The performance anxiety was removed, allowing the node.js developer to focus on adding the remaining features using the new comms pattern, while I tweaked and tuned the system by gradually intercepting more and more of the message types in the proxy, and working with dev-ops on the proxy deployment story. We fine-tuned the batching interval to balance fluid-feeling UX and optimal batch size, and ended up doing what I call a "whoompf" (batch send to all interested parties) every 20ms.

The system was now able to support the full 6,000 users necessary across all the Worlds, and the llama event was a roaring (clucking? bleating?) success. Erlang to the rescue once more.