Data Quality

e

Between 2018 and 2019, I worked as a consultant at Shoklo Malaria Research Unit, a medical research centre on the Thailand-Myanmar border. The centre operated three regional health clinics providing medical services for people from Myanmar, including displaced migrants. Alongside medical care, the research centre conducted leading research on malaria.

Many of the data management systems used databases built in Microsoft Access. This was the preferred tool because it allowed flexibility and could be customised within the available IT skills. However, data validation in Microsoft Access was challenging, which lead to some medical data being entered that was outside normal ranges, indicating a data quality issue. I ran the OpenValid project to address this problem. 

OpenValid is a module I developed for Shoklo Malaria Research Unit that allowed medical staff to set the standards for data validation across all databases. It was written natively in VBA and was designed so it could easily be implemented on all MS Access database.

Implementing the module across all databases involved gathering the initial settings from doctors and medical staff, carrying out capacity building with the IT team to ensure the system could be locally maintained, and training the data entry staff in the data validation prompts.

After the system had been installed and was operational, there was a reduction in the number of data quality issues, improving the data quality used by the research teams.

Working at the SMRU headquarters was an exciting time. The team is highly motivated to provide medical treatment for a vulnerable population, providing anti-natal care for the community, and to build rigorous research toward the eradication malaria. I am proud that my time there contributed to the data quality that the teams could use for their work.

Data Collection

In 2018, I worked as a consultant on a project with a regional education authority in Myanmar who oversaw the provision of primary and secondary education for 180,000 students across 1,700 schools.

The schools were remote, across hundreds of small villages in a region with unreliable transport infrastructure, and many were only accessible by foot. Mobile phone signal was unreliable in these regions.

The education authority carried out regular data collection cycles, where the education outcomes of students were reported on paper forms. I was brought in as a consultant to develop and implement a data management system whereby the data collection process could be digitized. The purpose of the project was to provide reporting for donors to show impact, while also enabling the education authority to easily see the impact of their work.

This project involved many difficult technical, infrastructural, and logistic challenges. It involved building IT capacity within the education authority so that the system could be maintained. From a technical perspective, the data entry process designed to work reliably on a set of 100 thin-client laptops running on unreliable power infrastructure.

A key aspect of the implementation of the system was in the need for extensive periods of training and capacity building. This was key to ensuring that they system could sustain after my project runaway was complete. For the training process, I used a train-the-trainers approach, where I trained a set of people within the education authority who, in turn, trained the data entry staff and data collection team. I choose this approach because of language barriers and cultural barriers, and to ensure that the process of training could continue after I exited the project.

Data Integration

Shoklo Malaria Research Unit is a medical research centre on the Thailand-Myanmar border that specialises in research for the eradication of malaria. The unit’s Entymology Department maintains a large catalogue of mosquitos caught in the wild, a dataset that is used to study the size and behaviour of the mosquito population. 

The data for this catalogue was stored across a large number of Excel spreadsheets that had been customised to fit within the department’s data workflow. However, this created data silos, where data could not be easily analysed easily. Over the years that the cataloguing had been ongoing, the data model had gradually changed, so the structure of many of these Excel spreadsheets was subtly different.  

In this project, I worked on building an Extract Transfer Load (ETL) process that the head of the Entymology department could use to integrate the catalogue. The project involved integrating legacy datasets into a centralised dataset that could be used for medical research. Together, we designed the data mapping for the legacy datasets. I developed and implemented the ETL process using Talend, an ETL tool with open-source features and a visual drag-and-drop interface that the head of department could maintain themselves.