An online radio station that allows visitors to create complete songs and scheduled shout-outs, just by typing in encouraging messages for their friends.

Working with TBWA\DotDotDash out of NYC, I architected and tech-directed the entire system for this AI-driven campaign site for LG. I also wrote a custom back-end API using postgREST, and designed & implemented a complex off-line processing chain to generate the songs using AWS Lambda functions.

The processing chain was orchestrated by a sequence of PostgreSQL functions, each triggered by user inputs or outputs from peer processes, and kicking off the appropriate Lambda functions to build a song and its "album cover".

Each step carried out a single task, from content moderation through prompt, lyrics, audio and image generation, and finally copyright filtering, storing its output back to the database - which then triggered the next step.

The Lambda functions were written in node.js and communicated variously with 3rd-party AI services to create the songs and the artwork. These services included OpenAI and LlamaGuard on Replicate for UGC moderation and sanitisation, Controlla Audio for song generation, Adobe Firefly for images, and ACR Cloud for audio content fingerprinting.

As the services were all using AI, each step in the chain would take an unpredictable amount of time, and the overall delivery time for a user's song could be between 60-90 or even 120 seconds. This meant that the entire processing flow needed to be completely asynchronous. To this end I designed the system to work with a series of event-based actions and timeouts, rather than standard synchronous API calls, in order to be able to keep the user experience responsive and the back-end processing resources lightweight.

The complexity and timing of the song processing also meant that we had a fairly involved chain of actions, with a variety of possible execution paths. These all needed to be carefully monitored and tracked, to give us detailed observability of the processing flow during development, enable extensive integration and performance testing flows for the full API and 3rd-party services, and to help with any troubleshooting in production. We also needed to be able to  generate large volumes of song requests in order to understand the scaling profile of the Lambda processing system.

To that end, I created a load/performance-testing and live monitoring tool, using Erlang/OTP for rapid development, very low latency and response times. The tool spawned large numbers of concurrent API clients via its web-based UI, using randomised user data to request a song generation for each one, and reported low-level information about all the steps taking place in the course of generating song's audio and imagery.

Every PostgreSQL trigger in the API database sent lightweight NOTIFY messages to any listening monitor tools, containing metadata about each event. The monitor tool would then fan out these notifications in turn to any listening UI clients, and ultimately back to the mock API clients so that they'd be able to complete their mock user journeys appropriately. This simulated realistic load for the song generation system, while providing real-time updates on the progress of individual and aggregate song generations via the monitor UI.

Showing all the currently-processing songs in a grid provided a really quick overview as to the overall health of the system and its partner services, as well as allowing a drilled-down view into very specific details of what was working out and what was having problems.

This helped out enormously in development as well as providing a very detailed view for our LG deployment partners, giving them confidence about system visibility and immediate feedback on system performance as they rolled out the software into their regulated AWS environment.

The site's front-end and middleware were written in Next.js, deployed via AWS Cloudfront CDN and Fargate serverless compute, and talked via an OpenResty/nginx proxy to the postgREST API. PostgreSQL functions (running in an AWS RDS database) invoked Lambda functions asynchronously using the aws_lambda extension (provided by AWS ), which communicated in turn with the various 3rd-party service APIs. Assets were all stored in AWS S3, also served behind a Cloudfront distribution.

I directed a 3-strong tech team of front-end developers and dev-ops, which swelled to 6 at some of the busiest parts of the project, as requirements inevitably changed in the face of a static deadline. We managed all app code & assets using GitHub & AWS CloudFormation for our internal CI/CD and converted the CloudFormation templates to Terraform for delivery to the deployment partner.

Here's a demo video of the load-test & monitoring tool, which shows more detail about how it works. As often, developing a comprehensive testing tool in tandem with the system itself allowed for really intricate performance measurements, along with effective integration/regression testing during development, deployment, and into production.

For the 2024 All-Star weekend, NBA commissioned ASB GlassFloor to create a series of interactive games for game stars to play on their game-certified LED floor at the Indiana Pacers' Gainbridge Fieldhouse.

Players carried multiple tracking beacons including BlackTrax and Kinexon, all of which transmitted position data to be collected and aggregated in close-to-real time, and streamed over the network to an Unreal Engine app which would compute and render the game graphics for the huge display.

I was asked to design and implement a data system to handle all the incoming tracking data via various UDP app protocols, aggregating it into a common data format, storing it for fast time-based retrieval, providing access, recording and playback APIs for use by game logic apps to enable dynamic instant replays.

After some initial proof-of-concept work, prototyping and benchmarking, I settled on "the simplest possible architecture that would work": a Redis pub-sub server and client and a MariaDB (MySQL) database, with node.js as a data-fusing & routing engine.

The node.js app received BlackTrax, Kinexon, Unreal and Mixon data over various RTTrPM and OSC connections, packaging and publishing them to a Redis channel, providing a constant stream of current player positions. Two separate subscribed clients fetched new data on the channel, the first forwarding directly to game logic code, the second storing position frames to the database as part of specified game segments when requested.

A PlaybackEngine  provided a mechanism for game logic to look up, select and request specific segments to be played back, at which point the engine would fetch the relevant data frames and re-transmit them via the live channel, so that app code could integrate instant replays into gameplay.

As the project was on a tight deadline and there were a lot of unknowns relating to game content, much of the work needed to be done in parallel, and as close to the actual live environment as possible. Significant differences in behaviour between some of the data-stream simulator software and the real streaming hardware also meant that we needed to be testing code against the production rig as much as possible. This in turn meant firstly going to the ASB HQ near Munich, and then to a colossal warehouse outside Indiana where the full system was installed for development and testing in the weeks leading up to the game - both of which were in at least -5ºC temperatures!

I wrapped up and handed over my work on the data service engine a week or two prior to the event, at which point the rest of the team pressed ahead finishing and polishing the game components for a successful and much-lauded display in Indianapolis on the live weekend.

Since the successful reception of the Prevalence Simulator, I've continued to provide cloud consulting and development services for the NTD Modelling Consortium (NTDMC) via the epidemiology faculty at Oxford University's Big Data Institute (BDI). A big part of my work for the modelling team involves using tools and processes I've developed to run large numbers of their data simulations in parallel, across many CPU cores on powerful cloud computers. Which is what's happening in the image: lots of simulations running on lots of CPUs.

Funded by the Gates Foundation, the NTDMC works on projecting effective approaches for eradicating a selection of neglected tropical diseases. With the help of Research Software Engineers (RSEs) at the BDI, epidemiology researchers maintain and develop open-source software implementations of their agent-based statistical models to simulate multiple treatment scenarios for a variety of parasite-borne diseases until 2040. The scenarios usually include mass drug administration (MDA) programs, and currently model worm and human-host life-cycles in sub-Saharan Africa, with future efforts potentially looking to regions of India.

Working with colleagues from OpenCulture, I've been helping with the development, orchestration and large-scale execution of the software models; managing the result data sets and coordinating with technical and scientific members of the BDI and other NTDMC partners at other universities; and shepherding the data all the way through to OpenCulture's visualisations for the NTDMC's research paper outputs. As part of the development work I've liaised with, and helped hire, RSE staff at the BDI, as well as advising epidemiology researchers on software structure and modularisation.

Any one "run" of a particular model carries out a number (usually 200) of simulations, starting from a set of seed data pre-computed by other epidemiological tools. Each run uses a specific set of parameters for a scenario (e.g. MDA treatment type & frequency, provision of protective bed nets, adherence to the program in a given population), and calculates outcomes for that scenario based on averages of all the simulation results.

I organise per-disease simulation runs and data management. Starting from the basic tooling I initially put in place for pre-computing the Prevalence Simulator's result sets, I've extended and automated the approach, so that we can quickly and straightforwardly run large numbers of scenario simulations across multiple diseases, on many CPU cores across many virtual machines at a time, on an ad-hoc basis.

This may not be Google-scale "big data", but it does generate tens to hundreds of gigabytes, across thousands of output files, for each of many simulation runs, all of which need to be managed, transmitted to partners, and post-processed for incorporation into intuitive visualisations of the outcomes. That’s perhaps not enough to justify using enterprise-style data management products (or the associated price tags), but it definitely is enough to need some kind of structure and organisation. As the processing schedule is intertwined with the outputs of sometimes unpredictable research work, we have very bursty processing patterns: no processing might be needed for days or weeks, but then many thousands of simulations for a number of diseases will need to be quickly turned round all together.

The models work at the level of a given implementation unit (IU, an administrative designation usually representing a region of a country). Running 200 simulations for a given disease/scenario/IU combination can often be completed in a few minutes, though this can vary greatly depending on the complexity of the model, the extent to which a given disease is endemic in a particular IU, and other complicating factors. Providing statistically meaningful numbers of simulations  requires running multiple scenarios for each disease across thousands of implementation units; scenarios and even modelling approaches can vary as a result of new field data or organisational change, requiring new sets of simulation runs across large geographical regions.

The software models are written variously in Python and  C++, and are generally CPU-bound: i.e. they take up all the time of the CPU cores they're using at a given point. Some of the models are single-threaded and some use Python's 'multiprocessing' library to split the work across multiple cores. I use a combination of cloud virtual machine (VM) automation and Linux tooling to parallelise the tasks across worker processes as necessary according to their execution model and then further across multiple VMs.

I run all this on Google Cloud Platform. This gives us low-cost, easily managed data storage that can be organised hierarchically by disease and by scenario and shared with NTDMC partners. It also gives us a lot of flexibility in accessing processing power: we can quickly spin up large, multi-processor VMs, billed by the second, and switch them off immediately when the processing’s done. I've organised big enough usage quotas with GCP that we can access the processing power we need to run large jobs at will: the largest so far has been twelve 128-core machines running 1536 simulations at a time to project forward a range of epionchocerciasis treatments across 19 countries.

I've also independently created and presented a proof-of-concept for running agent-based worm life-cycle models in Erlang/OTP. Following my instinct about the work required to parallelise the existing model runs, my OpenCulture colleague and I identified and planned a potential new approach by which we could run simulations across much larger populations, with multiple diseases simultaneously. This wouldn't just be for efficiency purposes, though that's a key factor, but also to help account for the presence of (and interaction between) multiple parasites in a given human host. I wrote the Erlang POC to demonstrate how this could work.

My rationale was that the existing models' implementation in Python and C++ has meant that while they can achieve very good sequential performance, single-threaded code has to be managed to use all the resource of a multi-core VM, scaling even multi-threaded code beyond the limits of a single machine will involve complex distribution mechanisms in code or tooling, and organising large simulation runs involves potentially brittle orchestration across multiple VMs (or containers, pods, etc).

Conversely, modelling worm life-cycles using Erlang's lightweight concurrency primitives means firstly that the BEAM scheduler (sometimes with a bit of hinting) can seamlessly spread the work across all the cores in a large multi-core machine; and secondly that the simulations can be scaled across multiple such machines straightforwardly using Erlang's built-in distribution & clustering mechanism, with host and worm processes in a life-cycle system running on an Erlang cluster executing on separate machines while participating as peers in the overall simulation.

The prototype runs on a single machine so far, but clearly demonstrates the ability to run multi-disease simulations extremely efficiently across 128 cores in an environment providing powerful clustering tools:

The presentation was well received, and I'm hopeful that once the current schedule has been completed there may be opportunities to further this work in partnership with NTDMC research scientists.

A boutique digital agency in LA asked if I could help out with some performance problems they'd encountered on the time-sensitive, high-stakes, secret client project they were building for a large tech company. Covid had disrupted the client's annual all-hands meeting, so they'd been commissioned to build an online version where company staff could join in live, communicate with each other in virtual rooms ("Worlds"), watch C-suite presentations together on virtual in-world screens, and chat about them afterwards. All while manifesting in this custom 3D world as ... multi-coloured llamas. Who was I to argue?

The application was at a fairly late stage of development, with some lovely 3D animation work, as well as what looked like a very nicely organised, well-structured and neatly-written node.js backend application communicating with front-end clients over a websocket connection. Deployment process was solid, everything was in Kubernetes on AWS and managed very capably in Terraform.

It all worked just fine - until they hit around 100 concurrent connected users. Everything slowed down after around 80, and pretty much fell apart near 100. Problem was, they needed to handle 6,000. Hence the agency devs needed urgent help on this kind of performance profile, as they only had a few weeks to make it work.

So I set about investigating how the application logic and comms worked, including what was in the database, what in memory, how the app flow worked, what messaging was being done, etc. Once I had a clear picture of all this, I built a testing tool which simulated the entire process of llama avatars spawning, moving, interacting in and leaving the virtual world, and allowed us to gradually increase the number of participants until we started to see problems.

This showed up a few immediate obvious problems: some DB indexes could be tweaked, some redundant messages could be removed - but none of these was going to be a game-changer.

The main issue was fan-out of server-to-llama and inter-llama message notifications. Every time a llama did anything, it notified the server, which then notified all the other llamas, so of course the performance requirements scaled exponentially with the number of llamas.

On top of that, every time any client sent any message, the server was receiving it and then "broadcasting" it by sending it on to every other connected client in a tight loop. The application wasn't taking into account the explosion of CPU and I/O work necessary beyond a certain level, and the single-threaded node.js app was falling over under the volume of messaging, even on some of the larger pod types available.

The notion of re-writing the app to split it into multiple instances and somehow synchronise data between them at this stage in the game was understandably panic-inducing. So we needed a way to jack up the performance without going back to the drawing-board, by whatever means necessary. I knew we needed to get more CPU grunt in there, but the existing architecture made vertical scaling the only option, and taking that much further was looking difficult.

We discussed various approaches based not only on the compute/deploy resources available but also, importantly, the time and availability of key people. Given that there was still a fair amount of feature work to be completed in the application build, we decided to try to maximize the difference I could make in part by using my work to minimize the effort required from other team members.

To do this I proposed writing a simple websocket proxy that would run in sidecar containers for each World, taking most of the processing load away from the app server:

• The proxy would handle the establishment and maintenance of all the downstream websocket connections, manage all direct inter-llama messages in a given instance, and only communicate critical information (such as entries, exits, statuses and interactions) upstream to the node.js server.
• That, along with its writer, would then be freed up to handle database/session management and further feature development.
• Each proxy instance would maintain its client's upstream websocket connections to the node.js server, listening out for specific messages for that client as well as World-wide messages the server wanted to "broadcast" to all clients, and handling downstream comms in all these cases.
• Rather than trying to shove through every outgoing message ad-hoc, the proxy would rationalise the messages by storing them in a memory-based data table and running a "fetcher" process. The fetcher would retrieve batches at a configurable/tunable rate and send them together, significantly reducing the numbers of calls.
• I'd write the proxy service in Erlang/OTP, as it has such a good async I/O story, meaning it can shovel large amounts of data in and out of large numbers of sockets really efficiently, and so is much better equipped to handle lots of really fast messaging.
• Erlang's runtime (the BEAM) can also share larger data blobs efficiently between concurrent processes, passing references to socket handlers so there's a lot less data copying going on - so the batching mechanism meant we could shovel a lot of the data out to the clients with a lot less virtual blood, sweat and tears than the node.js app had had to expend.
• Most importantly, unlike node.js, the Erlang platform could also take advantage of multi-core CPUs automatically. This meant we could get a lot more execution cycles into the mix, and so scale the performance a lot further without needing to re-architect the node.js app for multiple instances.

I worked with the agency's backend developer to rationalise and streamline the message flows, and put together the simplest possible version of the proxy service I could, using cowboy as the downstream websocket server, gun as the upstream client, syn to manage client grouping and notification and jiffy for blazing fast JSON encoding & decoding of messages.

Time spent upfront investigating and diagnosing the problems paid off when the core of the proxy only took a few days, with the main "broadcast" code hot-path moved to the new service, leaving unhandled messages passed through to the upstream for the time being. The effect was drastic:

The performance anxiety was removed, allowing the node.js developer to focus on adding the remaining features using the new comms pattern, while I tweaked and tuned the system by gradually intercepting more and more of the message types in the proxy, and working with dev-ops on the proxy deployment story. We fine-tuned the batching interval to balance fluid-feeling UX and optimal batch size, and ended up doing what I call a "whoompf" (batch send to all interested parties) every 20ms.

The system was now able to support the full 6,000 users necessary across all the Worlds, and the llama event was a roaring (clucking? bleating?) success. Erlang to the rescue once more.

When the team at Artists & Engineers got in touch to see if I'd be able to help out on an interactive online experience for the Royal Shakespeare Company, I tried to play it cool for at least 5 seconds. Not sure how good a job I did of it.

The project was Dream, a large-scale R&D experiment led by  Marshmallow Laser Feast for the Audience of the Future Consortium, digging in detail into how live theatrical work might develop online - especially in the context of a COVID world where people had to experience it from their homes.

It was a highly creative and extremely ambitious piece, in which actors in motion-capture suits contributed position data to an Unreal Engine 3D scene, which took in and incorporated interaction data from a live audience representing thousands of fireflies in the scene, finally rendering and streaming it out as HD video to ... that same live audience.

The project had already been developing for a while and a lot of the 3D work was in progress already; the time had come to start working out how the audience interaction would work, both creatively and technically, and the team were very sensibly looking at using pre-built components to glue together.

I helped to assess a series of commercial systems which would synchronize the audience data with the 3D camera activity and the HD stream, but while a couple seemed very promising and certainly capable, none did exactly what we needed so we decided to ... do it ourselves.

I knew the success of the interaction would not only be heavily sensitive to data latency and throughput considerations, but it would also involve a lot of creative iteration. So, working in close collaboration with A&E and AllSeeingEye, I built in parallel a data-streaming system - connecting the studio-side Unreal cluster with audience-client-side WebSocket code - and a stateful load-testing system allowing us to generate large volumes of meaningful robot clients, setting up and tearing down thousands of live WebSocket connections ad hoc and sending randomised but realistic client data across them to keep a constant eye on how the server would perform.

This meant that the creatives, the TD, the 3D team, the front-end developer and I could iterate together. We now had a test harness in place which would allow us to try out new features and functional changes and seeing what kind of impact each would have on the overall performance of the system, using realistic audience numbers.

The target was to be able to hit 2000 live users per performance. Initially we were looking at sending live position data for all the fireflies, and the 3D scene was running at 60fps, so there could potentially be a lot of data. We needed to send the 3D camera positions and an aggregate of all the firefly positions to all the clients, so fan-out was not going to be our friend.

I built the interaction server and the load test server in Erlang/OTP (notably using Cowboy HTTP/WS server, Gun HTTP/WS client and syn process group manager), and implemented a dynamic batching mechanism to constrain the outgoing data figures.

I also created a web-based UI for the load-generation system. This enabled quick tweaking of firefly "flock size" and randomisation parameters for the flock, and with some simple system monitoring tools we could see exactly the efffect of the changes made in the tool.

Erlang's automatic multi-core scaling meant that I could reliably scale the server performance pretty high simply by bumping up the number of cores, and wouldn't need to worry about scaling the service horizontally. I still wanted to make sure we got the best performance out of the available resource though, so I implemented all the web client/server WebSocket conversations as Protocol Buffer messages, using a native protobufs implementation on the server side and auto-generating a Javascript comms lib for the client-side as part of the build process. I also set up a comprehensive CI/CD pipeline which allowed team members to build and re-deploy the system independently during development, using Bitbucket CI tools and AWS Code Deploy.

Running the systems on large AWS multicore VMs, I managed to get upwards of 3000 clients running quite happily at a full 30fps in both directions, and was able to tweak the performance of the overall system by modifying the send rates for the camera frames on the Unreal side, the firefly positions on the client side, and the batching system on the interaction server, all on the fly while running thousands of live clients. Although the final result didn't require such a high level of throughput, this meant that we could freely experiment creatively throughout development, and maintain a high level of confidence that changes we made would work both in terms of functionality and performance.

Of course there were whole other aspects of the system required for a live, paid, ticketed audience event. It being also a timed event with a "waiting room" meant that we'd be dealing with a large number of almost simultaneous connections to the interaction service, as all the audience for a given performance tried to join at the allotted time.

I worked with AOTF's marketing and ticketing partner FX Digital to integrate the "magic link" and "virtual lobby" systems they'd created to authenticate & authorize audience members, and I implemented a queueing system in the interaction server to ensure the WebSocket connections were held safely while the server transacted with FX's ticketing API. I wrote code to manage upstream HTTP/2 connections dynamically, pipelining authorization requests across as few connections as possible, and helped FX in their load-testing to ensure their cloud deployment was tuned to handle the necessary volume of API requests.

Although it wasn't used in the final implementation, I also built on request a recording system which could store and play back camera and audience positions in real time, using Erlang's built-in soft-real-time database Mnesia.

Finally,  I implemented a dynamically reverse-proxied redundant server setup (using OpenResty) to fail clients over to a backup interaction server in the event of catastrophic AWS host failure, built AWS AMIs for the load test tool using Packer, and deployed the whole system to AWS using Terraform, including a CloudFront CDN distribution, to serve all the static web assets and handle SSL termination.

Overall this was one of the most creatively and technically satisfying projects I've worked on. Thanks, RSC, and thanks A&E for pulling me in!

Press/media coverage in New York Times, The Guardian.

I collaborated with colleagues from Open Culture and partners at the NTD Modelling Consortium who were creating an epidemiological simulator for use by various health organisations including the WHO. The simulator was organised at Oxford University and funded by the Gates Foundation. My part of the project was to help boost client-side performance, and integrate large-scale server-side model computation with the UI application.

The project had started as a data-viz prototype demonstrating the output of the Oxford University research team's predictive models, furnishing answers to the question of what happens to the spread of various sub-Saharan diseases in the face of treatment disruptions caused by funding problems and COVID. Users could tweak a wide variety of parameters such as mosquito net coverage, treatment frequency, drug regimen adherence and others across a period of years to get a fine-grained understanding of how funding issues could have direct real-world effects on the progress of treatments for these curable diseases.

As the project developed, from a client-side-only prototype covering a single disease (Lymphatic Filariasis) into a larger application providing analysis for several related strains of Neglected Tropical Diseases, I came on board to help speed up and re-structure some of the state management code, and to integrate the necessary larger datasets and more complex, performance-intensive statistical models.

To start with I rewrote sections of the ReactJS application, improving performance of client-side session storage and reorganising some of the data access patterns. Then, as the brief expanded to take in various types of soil-transmitted helminth it became clear that some of the models would run too slowly to be effective in a purely JS implementation and needed to be converted into native code.

The research team had models running in various languages, ranging from R to C++, and there was work underway to port the core models into Python. I looked at various approaches to getting them integrated with the simulator, including using Pyodide, a WebAssembly implementation of the Python scientific stack, but while promising it had too many intermediate steps slowing down the performance, and so I turned to doing the processing on the server side.

Some of the Python models using numpy for the underlying math processing were sufficiently snappy to run as a back-end API, so I built a Flask API, allowing the simulator UI code to call out to the model with the appropriate parameter values, and deployed it as Docker containers to Google Cloud Run.

Other models were still far too slow to work as part of a UI system, taking up to several minutes to run even smaller simulations. The code was CPU-bound, but rewriting the models as GPU code was beyond the project scope and team experience, so I worked with NTD's research scientists to run thousands of sets of parameter permutations so that we could pre-calculate all the outputs for whatever parameters a simluator user might select.

The team pre-computed a series of prequisite historical data and relevant parameter combinations on the university's parallel compute clusters, which I then took and ran against the live models on a 1000-CPU compute-intensive VM setup I organised with Google Cloud Platform. I negotiated access to this large number of compute-optimised machines with the GCP technical sales team, explaining the nature, motivation and funding of the project, and the reason for needing so many fast CPUs, and eventually got through enough layers of tech support to be able to get access to as many CPU-heavy VMs as we needed.

Having carried out many tens of thousands of runs of the models, we had all the data we needed to pre-populate the results for all the possible UI permutations, so I stored it all in Google Cloud Storage, and integrated with the simulator such that it would call out to the appropriate model for the selected disease, either running the model directly, calling out to a cloud API, or downloading a pre-computed result as necessary. I also implemented a storage-cache layer such that each run of the cloud API would add its output to the pre-computed result store, meaning the next access would be immediate and not require any back-end API usage.

Just before COVID hit the world, Nexus Studios asked me to help out with a 5G-powered, AR-enhanced experience for Changdeok Palace, a UNESCO World Heritage site in South Korea.

The project took advantage of the latest Google AR technologies (ARCore and Cloud Anchors) to enable a rich mobile experience for both physical and virtual visitors to the site, bringing the location's history to life by conjuring AR versions of traditional Korean folklore characters.

Visitors to the site contributed their own personalised content in the form of virtual kites, saved into the scene using Cloud Anchors and stored in a remote backend API.

As this was such a rich experience it required a large amount of volumetric 3D data to be transmitted to the users' phones, but WiFI and phone availability at the site are limited. SKTel, the project's Korean telco co-sponsor, stepped in to provide 5G access, helping users get much faster downloads and hence much snappier response times in the application.

I worked with Nexus's 3D team to design and build the cloud API on Google Cloud Platform infrastructure, and co-ordinated with SKTel's network teams, implementing a Mobile Edge Cache system providing a site-local, 5G-optimised data store, with assets smart-mirrored to edge servers, keeping them close to the end-user devices. This way SKTel 5G users would have the most streamlined experience, with data rates much faster than most WiFi.

I wrote the backend API in node.js from scratch in fairly short order, running on Google Cloud Run and using the Firestore Native Database, enabling realtime data updates during complex 3D development phases, and ensuring that the 247,000 virtual users who downloaded the app could enjoy the site experience remotely with similar performance to the 5G users at the real-world location.

I set up CI/CD using CloudBuild in multiple test and production environments so that the 3D team could continue to tweak and iterate as necessary. COVID-19 happened at exactly the right time to mess up the installation plans, but I kept in touch with Nexus as they worked tirelessly over that time to ensure a smooth delivery later in the year when physical access was possible again.

My friends at Rare Volume in NYC invited me to handle the tech implementation and installation for this self-contained interactive app. Commissioned by their client Samsung, it was to be installed at the (then under-construction) Samsung KX space at Coal Drops Yard in London's Kings Cross.

Showcasing the flagship Samsung Galaxy S10 phone, the installation would enable users to take a selfie with a provided S10 device, and incorporate it as the centrepiece of a London-themed collage on a Samsung 4K LED, using the S10 as a touch interface.

I put together a 3-person team, consisting of a an ArtRabbit pal writing React Native UI, a friendly graphics wizard to write the big-screen display app, and me, to design, build and deploy the software solution. We created a custom kiosk-mode Android app which sent picture data, real-time path controls and various user information to a Windows Cinder app, which creates an appropriate colour scheme according to the selfie picture content and generates collage-style "tears" in the on-screen image, revealing sections of prepared imagery in real-time to fit with the scheme.

Once users have created their collage, the system prints out a copy on high-quality photographic paper and optionally sends them an electronic version to their email.

I architected the overall system, wrote the React Native networking code, created a client-testing harness in node.js providing a feature-complete mock back-end, integrated the Cinder app with Samsung's CRM APIs using a Python interface, and project-managed the development and delivery of the software. Alongside colleagues from Rare Volume I installed it on site, troubleshooting all technical aspects of the deployment, and working with them and the Samsung clients to manage several rounds of creative and functional changes to best fit the surroundings.

Google's NYC-based Hardware Marketing team came up with a really nice concept for a series of automated dynamic tableaux, showcasing various features of their product lineup at MWC 2019 in Barcelona.

The displays included products ranging from Pixel phones and Slate tablets to Nest home automation lines, each of which should display custom content on-device to highlight various product features out of the normal working context - in a physical installation with constraints making it impossible to use the normal software.

Synchronising multiple streams of activity, on multiple OSes and different products with physical display mechanisms and lighting systems, was proving challenging. So the team asked me to help co-ordinate the various systems and come up with solutions for reliably getting the right content to the right displays, on the right devices, at the right times.

Consulting with the creative, product and engineering teams to work out exactly what we could and couldn't do with the various devices, I quickly established that there was no way of getting custom OS builds for the Home Hub devices in the time available.

So I settled instead on a web-based display system using Google Cast to send content to the Home Hubs, and an event-based messaging system using open Internet protocols to communicate with the products and physical display mechanisms.

Included in the installation were:

• 5 Pixel phones running a complete web-based mock-up of the Lens UI, mounted on a rotating carousel synced with the control server via UDP, "recognising" a pre-planned sequence of commercial products
• A custom USB control mechanism to handle automated state changes on the Nest thermostat (with an SSH fallback in case the physical constraints made wiring impossible), synchronising lighting in the diorama by sending control messages to a Pharos lighting system
• More UDP messaging with a revolving segment of a front-door controlling what to display via custom live streaming video from a display-mounted camera, Cast-ed to HTML content on a Home Hub video screen
• Another Pixel mounted on a moving platform, alternately connecting and disconnecting to a wireless charger, sync'ed via the control server to know when to display appropriate content depending on whether it was "charging" or not
• A Home Hub displaying animated segments of the Home Hub UI as though in stop-motion animation Cast'ed to the device and then sequenced via Websocket in time with the Pharos, to give the impression of a calendar day's events taking place
• Marvel Universe content streaming to a Pixel Slate in an animatronic miniature movie theater

I took responsibility for the end-to-end design, build and delivery of all the software across the installation. I sourced and organised resource for front-end UI programming support, collaborated with the the hardware and lighting production companies, designed an overall system with a central control server synchronising everything, and got cracking.

Time was short so I wrote the control server in Erlang/OTP, which made it a lot quicker to get to a system that received and passed messages to and from the devices using the necessary mixture of UDP streams, Websocket and USB connections, and Web content.

I spec'd and sourced redundant hardware servers to run and serve the control software using Docker, built a test rig of the system at the production company's workshop in London, flew to Barcelona with the Google and production company teams, and installed and set up all the devices and software in situ at MWC.

Finally, I stayed on-site and kept an eye on all the systems through til the end of the event. Fortunately things went well enough that the last few days were pretty relaxed. Boring, even.

Musical Forest is a WebVR Experiment by Google Creative Lab using “co-presence” - the synchronization of multiple people in a virtual space - to allow people to play music together in VR. All the players’ experiences are sync’d in real-time using WebSocket connections.

A couple of years after I stopped working there full-time, Google Creative Lab in NY decided to get the band back together and asked me to create custom co-presence server software, enabling the participants to see each other’s movements and share the musical experience in VR. I worked with the team at Google to make the smoothest connections possible and to ensure that the system would support tens of thousands of simultaneous users around the world.

I designed and built the entire server system, writing the websocket server in node.js (on request! 😬) along with custom load testing software in Erlang. The system used Cloud Pub/Sub and Cloud Datastore for client and horizontal server scaling synchronization, and I deployed it to Google’s global cloud platform using Docker and Kubernetes on Google Container Engine, testing performance up to 7,500 concurrent users per region across USA, Europe and Asia.

Press/media coverage in The FWA, Thirst4Beats, Android Headlines.

ArtRabbit is a unique web + app guide to the contemporary art scene, connecting thousands of art spaces, exhibitions and events to artists, art professionals, collectors, students and art-interested people alike.

The site runs on an open-submission model, enabling independent artists, curators and vendors to publicise their events, and working worldwide, but focusing on London, Berlin, NY and LA.

I co-conceived and co-created the iOS & Android app along with a variety of new features and interactions on the site.

I build and manage backend systems for the site and app, including graph database design, programming and optimisation with Neo4J; REST API design; development and deployment setups with Vagrant and Google Cloud Platform; client app development using Redux/Sagas for React Native; server side programming in Lua/OpenResty, PHP and nginx; a custom high-performance caching image server and … pretty much anything else that comes up.

As part of a global Android brand campaign, we created a cross-media digital campaign enabling users to create their own personalized Android characters on web or mobile, and have them take part in huge, real-time, interactive games on the biggest LED display in the world, in New York’s Times Square.

Users made their individual characters using the native Android app or on the website, sharing them to our custom AppEngine CMS and asset store.

Games and character galleries were displayed on the Times Square board via Javascript/WebGL applications running from custom, local application servers. These co-ordinated with the upstream apps to retrieve remotely-generated Android characters from the asset store, and receive new ones created by visitors to the live installation. T-shirt and postcard stations also connected in and printed out personalised physical keepsakes for visitors to the event.

I directed the technology from inception to delivery, designing the overall architecture, choosing and managing specialist development vendors, managing and negotiating the project’s progress through Google software processes, integrating our development efforts across all platforms, through to installation and launch on the giant board.

The timeline was extremely short - 2.5 months from start to launch. I designed and constructed the tech team from carefully picked partners, organised them into logical units to attack different components in parallel, worked with Google security teams, physical event and network management partners to get the on-site equipment set up, and even dug into diagnosing and providing fixes for low-level code issues in the Android app in the small hours before launch.

Finally, as a core part of the launch team I was on site for the entire duration of the event and made sure the tech side ran as smoothly as possible. We got some great coverage and a huge amount of user feedback.

Press/media: The Verge, Mashable, Gizmodo

Using the power of Google Chromecast, up to 6 friends can compete to answer questions from their phones and tablets as the game plays out on the big screen. Questions are all generated by asking Google’s Knowledge Graph about popular search terms and constructing quirky, unexpected trivia questions by traversing the relationships stored in the graph.

I pitched our original idea to Google’s NY engineering team, and worked with them and our external application vendor to integrate the questions and answers with the native Android/iOS and App Engine applications.

Players use native Android and iOS apps to launch the game on their Chromecast devices, which loads all the questions and the game server from an App Engine app. Gameplay, messaging and synchronisation happens via inter-device comms between the apps and the Chromecast using the Cast SDK.

I co-ordinated between our development partner, the Knowledge Graph team and the Google internal Android team to complete, publish and launch the app. We showed off a custom version the app running on a large screen at Google I/O 2014 in San Francisco where it was one of the most popular stands in the show.

Press/media: The FWA, OMGChrome

The launch of the Google Cast SDK and Chromecast means anyone can develop applications that take advantage of the biggest screen in the house - the TV.

We got excited about using the TV as a place for people to collaborate, so we started tinkering with ways for people to play with photos using their phones and tablets - like an interactive slideshow. The experiment became Photowall for Chromecast, a new Chrome Experiment that lets your friends collaborate and display images in real time on the TV.

I came up with the overall system concept and design, managed development between Google internal processes and our external vendor and delivered the overall project with my team-mates at Creative Lab. After the launch I wrote a Google Developers Blog post about the system, and took part in a live-streamed video interview for Google's Developer Advocate team.

Users launch Photowalls using a dedicated Android & iOS app, decorating and sending their photos to the wall via a desktop and mobile web interface to an App Engine application. Communication between browsers in the same "rooms" happens over a secure websocket connection, served from a dynamically-provisioned fleet of Compute Engine VMs. If the user chooses, the system processes all the photos from the wall into a video and uploads it as a private keepsake in the user's YouTube account, once the wall has been closed.

Press/media coverage in Engadget, The Verge, The Next Web.

W+K created a "second-screen" experience for Coke fans to enjoy during the 2012 Superbowl spot. Working with specialist partners Framestore and MLB Video, we created a way for fans to share the big game with Coke's signature polar bears. Animated and streamed to secondary devices, and controlled using a custom X-Box-based marionette system, they responded live to game events - and even competitors' advertising.

I directed the development and technical production teams in building social content channels, integrating with the streaming system, and negotiating with Coke's IT architecture board to build the system using J2EE/JBoss/Oracle. We deployed and delivered it using their preferred hosting environment and CDN (Savvis/Akamai).

Throughout the game 9 million consumers across various platforms checked in on the bears, and spent an unprecedented average dwell time of 28 minutes with the stream playing.

Press/media coverage in AdAge, Campaign, Fab Awards, Business Insider.