Parallel Processing Pipeline

Since the successful reception of the Prevalence Simulator, I've continued to provide cloud consulting and development services for the NTD Modelling Consortium (NTDMC) via the epidemiology faculty at Oxford University's Big Data Institute (BDI). A big part of my work for the modelling team involves using tools and processes I've developed to run large numbers of their data simulations in parallel, across many CPU cores on powerful cloud computers. Which is what's happening in the image: lots of simulations running on lots of CPUs.

Funded by the Gates Foundation, the NTDMC works on projecting effective approaches for eradicating a selection of neglected tropical diseases. With the help of Research Software Engineers (RSEs) at the BDI, epidemiology researchers maintain and develop open-source software implementations of their agent-based statistical models to simulate multiple treatment scenarios for a variety of parasite-borne diseases until 2040. The scenarios usually include mass drug administration (MDA) programs, and currently model worm and human-host life-cycles in sub-Saharan Africa, with future efforts potentially looking to regions of India.

Working with colleagues from OpenCulture, I've been helping with the development, orchestration and large-scale execution of the software models; managing the result data sets and coordinating with technical and scientific members of the BDI and other NTDMC partners at other universities; and shepherding the data all the way through to OpenCulture's visualisations for the NTDMC's research paper outputs. As part of the development work I've liaised with, and helped hire, RSE staff at the BDI, as well as advising epidemiology researchers on software structure and modularisation.

Any one "run" of a particular model carries out a number (usually 200) of simulations, starting from a set of seed data pre-computed by other epidemiological tools. Each run uses a specific set of parameters for a scenario (e.g. MDA treatment type & frequency, provision of protective bed nets, adherence to the program in a given population), and calculates outcomes for that scenario based on averages of all the simulation results.

I organise per-disease simulation runs and data management. Starting from the basic tooling I initially put in place for pre-computing the Prevalence Simulator's result sets, I've extended and automated the approach, so that we can quickly and straightforwardly run large numbers of scenario simulations across multiple diseases, on many CPU cores across many virtual machines at a time, on an ad-hoc basis.

This may not be Google-scale "big data", but it does generate tens to hundreds of gigabytes, across thousands of output files, for each of many simulation runs, all of which need to be managed, transmitted to partners, and post-processed for incorporation into intuitive visualisations of the outcomes. That’s perhaps not enough to justify using enterprise-style data management products (or the associated price tags), but it definitely is enough to need some kind of structure and organisation. As the processing schedule is intertwined with the outputs of sometimes unpredictable research work, we have very bursty processing patterns: no processing might be needed for days or weeks, but then many thousands of simulations for a number of diseases will need to be quickly turned round all together.

The models work at the level of a given implementation unit (IU, an administrative designation usually representing a region of a country). Running 200 simulations for a given disease/scenario/IU combination can often be completed in a few minutes, though this can vary greatly depending on the complexity of the model, the extent to which a given disease is endemic in a particular IU, and other complicating factors. Providing statistically meaningful numbers of simulations  requires running multiple scenarios for each disease across thousands of implementation units; scenarios and even modelling approaches can vary as a result of new field data or organisational change, requiring new sets of simulation runs across large geographical regions.

The software models are written variously in Python and  C++, and are generally CPU-bound: i.e. they take up all the time of the CPU cores they're using at a given point. Some of the models are single-threaded and some use Python's 'multiprocessing' library to split the work across multiple cores. I use a combination of cloud virtual machine (VM) automation and Linux tooling to parallelise the tasks across worker processes as necessary according to their execution model and then further across multiple VMs.

I run all this on Google Cloud Platform. This gives us low-cost, easily managed data storage that can be organised hierarchically by disease and by scenario and shared with NTDMC partners. It also gives us a lot of flexibility in accessing processing power: we can quickly spin up large, multi-processor VMs, billed by the second, and switch them off immediately when the processing’s done. I've organised big enough usage quotas with GCP that we can access the processing power we need to run large jobs at will: the largest so far has been twelve 128-core machines running 1536 simulations at a time to project forward a range of epionchocerciasis treatments across 19 countries.

I've also independently created and presented a proof-of-concept for running agent-based worm life-cycle models in Erlang/OTP. Following my instinct about the work required to parallelise the existing model runs, my OpenCulture colleague and I identified and planned a potential new approach by which we could run simulations across much larger populations, with multiple diseases simultaneously. This wouldn't just be for efficiency purposes, though that's a key factor, but also to help account for the presence of (and interaction between) multiple parasites in a given human host. I wrote the Erlang POC to demonstrate how this could work.

My rationale was that the existing models' implementation in Python and C++ has meant that while they can achieve very good sequential performance, single-threaded code has to be managed to use all the resource of a multi-core VM, scaling even multi-threaded code beyond the limits of a single machine will involve complex distribution mechanisms in code or tooling, and organising large simulation runs involves potentially brittle orchestration across multiple VMs (or containers, pods, etc).

Conversely, modelling worm life-cycles using Erlang's lightweight concurrency primitives means firstly that the BEAM scheduler (sometimes with a bit of hinting) can seamlessly spread the work across all the cores in a large multi-core machine; and secondly that the simulations can be scaled across multiple such machines straightforwardly using Erlang's built-in distribution & clustering mechanism, with host and worm processes in a life-cycle system running on an Erlang cluster executing on separate machines while participating as peers in the overall simulation.

The prototype runs on a single machine so far, but clearly demonstrates the ability to run multi-disease simulations extremely efficiently across 128 cores in an environment providing powerful clustering tools:

The presentation was well received, and I'm hopeful that once the current schedule has been completed there may be opportunities to further this work in partnership with NTDMC research scientists.