LG: Radio Optimism

An online radio station that allows visitors to create complete songs and scheduled shout-outs, just by typing in encouraging messages for their friends.

Working with TBWA\DotDotDash out of NYC, I architected and tech-directed the entire system for this AI-driven campaign site for LG. I also wrote a custom back-end API using postgREST, and designed & implemented a complex off-line processing chain to generate the songs using AWS Lambda functions.

The processing chain was orchestrated by a sequence of PostgreSQL functions, each triggered by user inputs or outputs from peer processes, and kicking off the appropriate Lambda functions to build a song and its "album cover".

Each step carried out a single task, from content moderation through prompt, lyrics, audio and image generation, and finally copyright filtering, storing its output back to the database - which then triggered the next step.

The Lambda functions were written in node.js and communicated variously with 3rd-party AI services to create the songs and the artwork. These services included OpenAI and LlamaGuard on Replicate for UGC moderation and sanitisation, Controlla Audio for song generation, Adobe Firefly for images, and ACR Cloud for audio content fingerprinting.

As the services were all using AI, each step in the chain would take an unpredictable amount of time, and the overall delivery time for a user's song could be between 60-90 or even 120 seconds. This meant that the entire processing flow needed to be completely asynchronous. To this end I designed the system to work with a series of event-based actions and timeouts, rather than standard synchronous API calls, in order to be able to keep the user experience responsive and the back-end processing resources lightweight.

The complexity and timing of the song processing also meant that we had a fairly involved chain of actions, with a variety of possible execution paths. These all needed to be carefully monitored and tracked, to give us detailed observability of the processing flow during development, enable extensive integration and performance testing flows for the full API and 3rd-party services, and to help with any troubleshooting in production. We also needed to be able to  generate large volumes of song requests in order to understand the scaling profile of the Lambda processing system.

To that end, I created a load/performance-testing and live monitoring tool, using Erlang/OTP for rapid development, very low latency and response times. The tool spawned large numbers of concurrent API clients via its web-based UI, using randomised user data to request a song generation for each one, and reported low-level information about all the steps taking place in the course of generating song's audio and imagery.

Every PostgreSQL trigger in the API database sent lightweight NOTIFY messages to any listening monitor tools, containing metadata about each event. The monitor tool would then fan out these notifications in turn to any listening UI clients, and ultimately back to the mock API clients so that they'd be able to complete their mock user journeys appropriately. This simulated realistic load for the song generation system, while providing real-time updates on the progress of individual and aggregate song generations via the monitor UI.

Showing all the currently-processing songs in a grid provided a really quick overview as to the overall health of the system and its partner services, as well as allowing a drilled-down view into very specific details of what was working out and what was having problems.

This helped out enormously in development as well as providing a very detailed view for our LG deployment partners, giving them confidence about system visibility and immediate feedback on system performance as they rolled out the software into their regulated AWS environment.

The site's front-end and middleware were written in Next.js, deployed via AWS Cloudfront CDN and Fargate serverless compute, and talked via an OpenResty/nginx proxy to the postgREST API. PostgreSQL functions (running in an AWS RDS database) invoked Lambda functions asynchronously using the aws_lambda extension (provided by AWS ), which communicated in turn with the various 3rd-party service APIs. Assets were all stored in AWS S3, also served behind a Cloudfront distribution.

I directed a 3-strong tech team of front-end developers and dev-ops, which swelled to 6 at some of the busiest parts of the project, as requirements inevitably changed in the face of a static deadline. We managed all app code & assets using GitHub & AWS CloudFormation for our internal CI/CD and converted the CloudFormation templates to Terraform for delivery to the deployment partner.

Here's a demo video of the load-test & monitoring tool, which shows more detail about how it works. As often, developing a comprehensive testing tool in tandem with the system itself allowed for really intricate performance measurements, along with effective integration/regression testing during development, deployment, and into production.