NTD Prevalence Simulator

I collaborated with colleagues from Open Culture and partners at the NTD Modelling Consortium who were creating an epidemiological simulator for use by various health organisations including the WHO. The simulator was organised at Oxford University and funded by the Gates Foundation. My part of the project was to help boost client-side performance, and integrate large-scale server-side model computation with the UI application.

The project had started as a data-viz prototype demonstrating the output of the Oxford University research team's predictive models, furnishing answers to the question of what happens to the spread of various sub-Saharan diseases in the face of treatment disruptions caused by funding problems and COVID. Users could tweak a wide variety of parameters such as mosquito net coverage, treatment frequency, drug regimen adherence and others across a period of years to get a fine-grained understanding of how funding issues could have direct real-world effects on the progress of treatments for these curable diseases.

As the project developed, from a client-side-only prototype covering a single disease (Lymphatic Filariasis) into a larger application providing analysis for several related strains of Neglected Tropical Diseases, I came on board to help speed up and re-structure some of the state management code, and to integrate the necessary larger datasets and more complex, performance-intensive statistical models.

To start with I rewrote sections of the ReactJS application, improving performance of client-side session storage and reorganising some of the data access patterns. Then, as the brief expanded to take in various types of soil-transmitted helminth it became clear that some of the models would run too slowly to be effective in a purely JS implementation and needed to be converted into native code.

The research team had models running in various languages, ranging from R to C++, and there was work underway to port the core models into Python. I looked at various approaches to getting them integrated with the simulator, including using Pyodide, a WebAssembly implementation of the Python scientific stack, but while promising it had too many intermediate steps slowing down the performance, and so I turned to doing the processing on the server side.

Some of the Python models using numpy for the underlying math processing were sufficiently snappy to run as a back-end API, so I built a Flask API, allowing the simulator UI code to call out to the model with the appropriate parameter values, and deployed it as Docker containers to Google Cloud Run.

Other models were still far too slow to work as part of a UI system, taking up to several minutes to run even smaller simulations. The code was CPU-bound, but rewriting the models as GPU code was beyond the project scope and team experience, so I worked with NTD's research scientists to run thousands of sets of parameter permutations so that we could pre-calculate all the outputs for whatever parameters a simluator user might select.

The team pre-computed a series of prequisite historical data and relevant parameter combinations on the university's parallel compute clusters, which I then took and ran against the live models on a 1000-CPU compute-intensive VM setup I organised with Google Cloud Platform. I negotiated access to this large number of compute-optimised machines with the GCP technical sales team, explaining the nature, motivation and funding of the project, and the reason for needing so many fast CPUs, and eventually got through enough layers of tech support to be able to get access to as many CPU-heavy VMs as we needed.

Having carried out many tens of thousands of runs of the models, we had all the data we needed to pre-populate the results for all the possible UI permutations, so I stored it all in Google Cloud Storage, and integrated with the simulator such that it would call out to the appropriate model for the selected disease, either running the model directly, calling out to a cloud API, or downloading a pre-computed result as necessary. I also implemented a storage-cache layer such that each run of the cloud API would add its output to the pre-computed result store, meaning the next access would be immediate and not require any back-end API usage.