RSC: Dream

When the team at Artists & Engineers got in touch to see if I'd be able to help out on an interactive online experience for the Royal Shakespeare Company, I tried to play it cool for at least 5 seconds. Not sure how good a job I did of it.

The project was Dream, a large-scale R&D experiment led by  Marshmallow Laser Feast for the Audience of the Future Consortium, digging in detail into how live theatrical work might develop online - especially in the context of a COVID world where people had to experience it from their homes.

It was a highly creative and extremely ambitious piece, in which actors in motion-capture suits contributed position data to an Unreal Engine 3D scene, which took in and incorporated interaction data from a live audience representing thousands of fireflies in the scene, finally rendering and streaming it out as HD video to ... that same live audience.

The project had already been developing for a while and a lot of the 3D work was in progress already; the time had come to start working out how the audience interaction would work, both creatively and technically, and the team were very sensibly looking at using pre-built components to glue together.

I helped to assess a series of commercial systems which would synchronize the audience data with the 3D camera activity and the HD stream, but while a couple seemed very promising and certainly capable, none did exactly what we needed so we decided to ... do it ourselves.

I knew the success of the interaction would not only be heavily sensitive to data latency and throughput considerations, but it would also involve a lot of creative iteration. So, working in close collaboration with A&E and AllSeeingEye, I built in parallel a data-streaming system - connecting the studio-side Unreal cluster with audience-client-side WebSocket code - and a stateful load-testing system allowing us to generate large volumes of meaningful robot clients, setting up and tearing down thousands of live WebSocket connections ad hoc and sending randomised but realistic client data across them to keep a constant eye on how the server would perform.

This meant that the creatives, the TD, the 3D team, the front-end developer and I could iterate together. We now had a test harness in place which would allow us to try out new features and functional changes and seeing what kind of impact each would have on the overall performance of the system, using realistic audience numbers.

The target was to be able to hit 2000 live users per performance. Initially we were looking at sending live position data for all the fireflies, and the 3D scene was running at 60fps, so there could potentially be a lot of data. We needed to send the 3D camera positions and an aggregate of all the firefly positions to all the clients, so fan-out was not going to be our friend.

I built the interaction server and the load test server in Erlang/OTP (notably using Cowboy HTTP/WS server, Gun HTTP/WS client and syn process group manager), and implemented a dynamic batching mechanism to constrain the outgoing data figures.

I also created a web-based UI for the load-generation system. This enabled quick tweaking of firefly "flock size" and randomisation parameters for the flock, and with some simple system monitoring tools we could see exactly the efffect of the changes made in the tool.

Erlang's automatic multi-core scaling meant that I could reliably scale the server performance pretty high simply by bumping up the number of cores, and wouldn't need to worry about scaling the service horizontally. I still wanted to make sure we got the best performance out of the available resource though, so I implemented all the web client/server WebSocket conversations as Protocol Buffer messages, using a native protobufs implementation on the server side and auto-generating a Javascript comms lib for the client-side as part of the build process. I also set up a comprehensive CI/CD pipeline which allowed team members to build and re-deploy the system independently during development, using Bitbucket CI tools and AWS Code Deploy.

Running the systems on large AWS multicore VMs, I managed to get upwards of 3000 clients running quite happily at a full 30fps in both directions, and was able to tweak the performance of the overall system by modifying the send rates for the camera frames on the Unreal side, the firefly positions on the client side, and the batching system on the interaction server, all on the fly while running thousands of live clients. Although the final result didn't require such a high level of throughput, this meant that we could freely experiment creatively throughout development, and maintain a high level of confidence that changes we made would work both in terms of functionality and performance.

Of course there were whole other aspects of the system required for a live, paid, ticketed audience event. It being also a timed event with a "waiting room" meant that we'd be dealing with a large number of almost simultaneous connections to the interaction service, as all the audience for a given performance tried to join at the allotted time.

I worked with AOTF's marketing and ticketing partner FX Digital to integrate the "magic link" and "virtual lobby" systems they'd created to authenticate & authorize audience members, and I implemented a queueing system in the interaction server to ensure the WebSocket connections were held safely while the server transacted with FX's ticketing API. I wrote code to manage upstream HTTP/2 connections dynamically, pipelining authorization requests across as few connections as possible, and helped FX in their load-testing to ensure their cloud deployment was tuned to handle the necessary volume of API requests.

Although it wasn't used in the final implementation, I also built on request a recording system which could store and play back camera and audience positions in real time, using Erlang's built-in soft-real-time database Mnesia.

Finally,  I implemented a dynamically reverse-proxied redundant server setup (using OpenResty) to fail clients over to a backup interaction server in the event of catastrophic AWS host failure, built AWS AMIs for the load test tool using Packer, and deployed the whole system to AWS using Terraform, including a CloudFront CDN distribution, to serve all the static web assets and handle SSL termination.

Overall this was one of the most creatively and technically satisfying projects I've worked on. Thanks, RSC, and thanks A&E for pulling me in!

Press/media coverage in New York Times, The Guardian.