A script to set up a local Docker instance of DHIS2 seeded with the demo database and with analytics tables populated for the DHIS2 dashboard.

Quick summary:

• What it does: Installs Docker if needed, pulls DHIS2 2.43, spins up a local instance seeded with the Sierra Leone demo database, and (optionally) populates the analytics tables so the dashboard has real data to show.
• What it doesn’t do: It’s not a production deployment tool, doesn’t work with other DHIS2 versions or demo databases out of the box, and doesn’t touch any live or shared environments.
• Who it’s for: Anyone implementing, evaluating, or demoing DHIS2 who needs a disposable local instance. Consultants, implementers, trainers, etc.
• Runs on: Ubuntu (tested on Ubuntu with apt and Docker). Not built for macOS or Windows as-is.
• Available at: https://github.com/KatherineWyers/dhis2-docker-local-setup

Before you can test a DHIS2 configuration or walk someone through a dashboard, you’ve got to spin up an instance first. Install Docker if it’s not there already, pull the DHIS2 image, seed the database, and then, if you want the dashboard to display, you need to populate the analytics tables. This isn’t a DHIS2 problem. Any system with this much going on has the same setup headaches. But it’s still a lot of steps for something you can end up doing multiple times over a project, and it can eat a whole afternoon that you didn’t have.

This script handles all of this in one pass, on an Ubuntu machine. It checks for and installs Docker if it’s needed. It pulls DHIS2 version 2.43, spins up a local instance seeded with the Sierra Leone demo database, and gives you the choice of running full analytics, just the last two years, or skipping analytics altogether if you’d rather trigger it manually later. Either way, it’s meant strictly for local development, not something you’d point at production.

I built this because I wanted to quickly be able to test a change before it touches a live environment, or to quickly show a client what their org unit hierarchy or data model would actually look like without needing access to their infrastructure. Having the analytics tables populates and ready to use makes a big difference, because a dashboard is much easier to discuss when there’s real data there to demo. That all saves a lot of hassle when demoing and running training sessions.

It’s not a big script, but it’s pretty much my approach to working. If I see the same problem come up a few times, I usually just build a script to automate it because it’s a good way of explaining the system. Some goes for the bigger projects too. Civil registration linking to health data, whether it’s a DHIS2 or OpenEMIS system, whatever is needed.

The script is open source and available here: https://github.com/KatherineWyers/dhis2-docker-local-setup. If you’re mid-implementation and hitting the same setup friction, or you just want to see what the system would look like before you commit to it, just reach out and we can figure out what it’d take to adapt the script to your context.

Introduction

You have a social enterprise. Over the last few years you’ve seen it grow and develop and you’ve met the people whose lives you’re impacting. You can feel it maturing and you want to help it move forward. To do that, you know that data is important. It helps you show the impact that you’re having, helps you plan out the best use of your funding and helps you communicate to your staff and your donors about your successes and where you want to go next.

How do you manage that data? It needs to be collected and stored, cleaned, and then needs to be analysed. This all needs to happen in the most efficient way, because the less time spent working with data, the more time your enterprise has to grow.

I’d like to show you how I think about data-management, and share a few tips you can use to improve your data-management processes. I’ve worked in data-systems design for the past twelve year, providing support for many different kinds of humanitarian, medical and commercial organisations. These tips will help you reduce the time your enterprise spends working with data, so you can have a greater impact with your time.

The main goal of a data-system is to lighten the work-load for your organisation. It is not to show off a fancy new computer system. So, I favour a balanced discussion about pros and cons, and only swap a pen-and-paper process for a computerised process if they will dramatically improve your organisation’s efficiency. Computers are great when they’re used correctly, but they come with their own challenges and can become a barrier to sustainability if they are not implemented with enough foresight.

Visualizing your data-pipeline

When I’m discussing data-management, I find it helps to visualize the whole system so that we can discuss individual stages. This will help you find the stages that are using up the most of your time, and lets you focus on making improvements to those.

Think of your data-management as a pipeline with a series of stages. On the left, we have the data, swimming out there in the real world waiting to be collected by anyone who want it. Then, over on the right, we have the end-result of the process. This is the presentation of the data for internal or external stakeholders.

I’m deliberately keeping this diagram very generic. This pipeline can apply to you whether you do all your data-collection manually, or if some stages have been automated. Some of the stages are manual and labour-intensive, some are automated using computers. Generally the manual-process and computer-processes are interchangeable, and there are pros and cons to choosing one or the other.

Quick Tip: Computers don’t work any magic. All they do is speed up the processes you already have. If your paper-based data-processes aren’t working correctly, then a computer will only help you do the wrong thing really fast! Fix the manual-process first. Then set up a computer to speed up the process. 

Why do we collect data? 

Before we start reviewing each individual stage of the data-pipeline, and look for the problems you might be facing, let’s start by first looking at the reasons why you are collecting all this data in the first place. There are two reasons to collect data

1. To give you and your staff information so you can carry out your day-to-day tasks
2. Produce reports for internal strategic planning, and for liaising with external stakeholder such as donors or authorities

Internal: Day-to-day tasks

What information do you and your staff need so they can carry out their daily tasks? Make a list of the tasks they do and the information they need while they’re doing the task.

For example, if you run a refugee teacher-training project, your teacher-trainers will need to know the school’s address, the teacher’s name and background, the notes and comments from the last training-session with that teacher. If you run a medical research centre, your researchers might need a subset of the full research-dataset that they can quickly run their analysis.

Internal & External: Strategy, planning and reporting

Ask yourself ‘how do we want to show our successes?’. What information best shows the way the great work that you’re doing, and the positive impact that your organisation is having? What reports do our funders need? What reports did your donors ask for in the past, but no longer need? Make a list of all the reports, both internal and external, that you need.

Your Tasks

1. List the information that you and your staff need for day-to-day tasks
2. List the reports that you need to produce to show your impact and liaise with external stakeholder

Looking at each stage in the pipeline

Let’s take a look at each stage in the data-pipeline. I’ll describe each stage, show some issues that tend to appear at that stage, and share some techniques to help you improve that stage of the pipeline.

Think critically about your current system

As you go through this process, I want you to think critically about the data that’s being collected and how it’s being processed. We’re looking to streamline your data-process so that you get the maximum out of your time.

• Is this data really necessary? How does it support the internal and external needs we saw earlier?
• Are all those external reports really needed?
• Is any data being collected that’s unnecessary?
• Is there data that you don’t collect yet, but it would be very beneficial to you if you did?
• What types of data in your database need to be cleaned? What stage in your pipeline might be allowing in the dirty data?

It takes a lot of time to collect and process data. The less time spent in data-management, the more time you’ll have to carry out the real work of helping people. It’s not always easy to be critical about what’s being collected and what could be collected. Try to stay detached from that for the moment. Just be a passive observer, noting down what you see in your organisation. Then, later, you can make a plan for which changes are worth going after, and which aren’t a priority.

The real world

Description

Out there in the real-world, your users are creating data. Whenever they change address, buy something new, make a meal, go to school. Every time someone does anything, there’s some potential information about that person.

Common problems

One of the major problems is that there are so many different pieces of information you can collect. Most of the information out there is completely irrelevant to your organisation. It’s easy to err on the side of caution, and collect as much information as you can, and then worry later about whether it’s needed. This creates problems, because too much data puts unnecessary demands on your time without helping you show your impact.

A lot of the data in the real world is unreliable. In many cultures, a date of birth is not used, and a person’s age is only an estimate.

Improvements

1. Only collect the data that you need. Remember to focus on the internal and external needs that your organisation has. All the data you collect needs to help you in some way.
2. You might come across some data that would be valuable for you when showing your impact. Note this down.
3. Be aware of the context of your data. For example, is a date of birth a reliable value in the culture?

Getting the raw data inside your organisation

Description

Some of the raw data from the real-world has been collected and is now inside your organisation. This is often stored on paper forms that were hand-written by your users, or during observations by your staff. Those papers might be in a pile on someone’s desk, or in an Excel file sent by a user.

Common problems

The main issues arise here because of the workflow. This stage can quickly become complex when different types of data is collected from different places by different people. Hand-written forms can be unclear.

If some fields/questions are often answered incorrectly, it’s likely that the question on the form is unclear.  If your organisation operates in a multi-lingual environment, this is a common issue you may be facing, as the question may be interpreted differently due to varying levels of understanding of the language.

Improvements

1. If you’re using paper forms, try to limit the number of fields on the form to the data you strictly need.
2. Read through the responses in your paper forms. Review the questions in the forms, and the written responses, to see whether they are unclear/ambiguous. If there are things written in the margins (there will be!) then it’s a signal that the fields in the form aren’t adequate. If one of the fields in rarely/never being written in, find out if it’s still necessary.

Raw data entered into a storage system (Data-entry)

Description

During this stage, the raw data is moved into long-term storage. There are many different options for how to store your data. If you’re going with a computer-based solution, you can choose to enter it into a spreadsheet, or into a database-system. Entering the data can be done using a custom application, or an off-the-shelf web-based system.

Common issues

The user interface is not clear.

Mismatches between workflow and user interface. Mismatches can cause the computer-system to get in the way of the workflow, which causes frustration. Leads to people finding ways to work around the system in order to carry out their tasks.

Computer-based data-entry needs to correctly validate the data. If the upper and lower boundaries of the data aren’t set, then it’s human nature that eventually some data will be entered incorrectly and it’ll be allowed into your database. 

Improvements

1. When you’re optimizing your data-processes, you’ll learn a lot by being a passive observer while the staff carry out data-entry. Even if you think the user-interface on your database is crystal-clear, I guarantee you’ll have some surprises when you watch a few people using it!
2. Add validation to the system. Check the upper and lower boundaries of the data and make sure that the computer validates the data so that data can’t be stored that isn’t within sensible ranges

Raw data stored

Description

At this stage, the raw data has been entered into long-term storage and it’s sitting there quietly waiting for you to use it. The data can be digital or paper-based.

Common Problems

Data security: The major issues with resting data is data-security. How easy would it be for someone to get access to the data? Is it possible for someone to accidentally share the data?

Backups: Is the data backed up correctly? One of the major objections I have to paper-based storage of data is that it’s not easy to back it up. Things go wrong with paper-forms. Damp, fire, flooding… even just that the pencil marks fade over time… paper-based forms are volatile and need to be carefully stored. On the other hand, backups is one of the areas where computers win. It’s so easy and quick to do, it’s far more likely that the data will be backed up regularly on a computer system.

Improvements

1. Store your data securely. Find out who has access to the data, and put policies in place to ensure it’s not shared accidentally.
2. Make sure that your data is being backed up regularly, and that those backups are stored somewhere safe, in a different building to the main system.

Data retrieval and analysis

Description

At this stage, you take the raw data that you’re storing and assemble it together to present it. You’ll do some calculations to analyse the data. You might list out all the items in your data. You might get a percentage of all the raw data, or select a subset of the data from a particular month or year.

Common problems

While this task can be done manually, it is far more efficient to program a computer to do this. Computer-systems are a perfect solution either, and problems can arise when you need to produce a new report. In the real-world, change is inevitable. If an donor has a new reporting-requirement, something new in the M&E reports, a new metric they need to track, your organisation will need to be able to program the computer to carry out the calculation. You need to avoid the situation where you have a system that works great for a few months, but that you can’t adapt. As the reporting requirements changed, you need to be able to implement those changes. Otherwise you’ll find that whenever there’s a new report, you’ll have to revert back to the slow, tedious task of doing manual calculations.

Improvements

1. Make sure that the systems you use to do the calculations is something that your organisation can adapt as needed. Ideally, this would be someone within your organisation, possible with support from someone outside with further expertise.
2. Whenever a new report is requested, I find that there’s a knee-jerk reaction to just calculate it manually. This should be avoided. It’s much better for your organisation to have someone spend that time learning how to implement the new calculation as a formula in code or in Excel. Preferably, that person is a staff-member who can be tasked with maintaining the formulas in your data-system.

Report produced and used

Description

The results of the analysis are compiled together and presented for use. This can be the information that’s needed internally for the day-to-day operations. It can also be for strategic planning at management level, and for quarterly M&E reports for donors and other external stakeholders.

Common problems

The main problems I’ve seen here revolve around the number of reports that are being produced. As I said earlier on, the whole purpose of the data-system is to be able to produce information so that you and your staff can work better. Often, there are reports that were needed in the past, but they aren’t used anymore. When you find a report that isn’t used anymore, it’s best to remove it from the system. Maintaining those calculations will take time, so avoid having reports being generated that aren’t needed. If you’re worried that you might need the report in the future, you can keep collecting the data into your database, and just set up the calculations in the future when you need the report again.

Improvements

1. Ask yourself whether all the reports are needed. If you find a report that’s no longer needed, remove it.
2. When a new report is needed, encourage your staff to implement it in code or in and Excel formula. Avoid the knee-jerk reaction of doing it manually. It will grow the skills of your staff and will make them more confident in the long-run.

Conclusion

We’ve looked at the data-pipeline, and I hope you’re starting to see how this idea can help you improve your own data-processes. Each stage can be manual or computerized, and the choice between one or the other should be based on a discussion of the pros and cons. Computers won’t always be the best solution for each stage. Sometimes it helps to automate a manual process using a computer. Sometimes it even makes sense to take an automated process and make it manual.

Manual processes tend to work well in the early stages of the pipeline. The early stages of the pipeline are more involved in the real-world where workflows change quickly and where data-collection systems need to be able to adapt to change. As the data moves down the pipeline, it becomes more and more useful for it to be digitized so that it can be easily backed up, analysed and presented as a report. Of course, the more you rely on digitized data, the more skills your organisation will need so that the changes in the real-world can be implemented in the software and the databases.

The key is to get the most out of the time that you and your staff have. If a process is taking a lot of someone’s time, and it’s clear to you that the process is helping you have an impact, then see if you can make it more efficient. Each stage in the pipeline has different things that can be optimized and improved. Look for the bottlenecks, and see if it’s an easy fix. Your enterprise will become more streamlined, and you’ll have more impact with the same amount of time and effort.

What kind of bottlenecks do you see in the data-pipeline at your organisation? What can you do to make them more efficient. Let me know in the comments below.

This page contains

• Introduction to OpenEMIS features
• How to install OpenEMIS with Docker
• The three parts of the OpenEMIS Docker application
• How to configure OpenEMIS for the first time

If you already have an installation of OpenEMIS up and running and you just want to configure it, skip to the section titled Configuring OpenEMIS.

What is OpenEMIS?

OpenEMIS is an open-source data-management software package for education management. It was developed with support from UNESCO. It is a tool that be used to build sustainable education data-management systems. It can be implemented in many different ways, and this enables it to be configured for various different contexts, in both the global North and the global South. The software has many useful features that make life easier for system administrators and education authorities.

Bulk importing of data

The bulk-importer lets you import student and teacher data from an Excel spreadsheet or a csv file. This enables you to quickly migrate data from an old system without needing to manually type the data.

Managing user permissions for administrators, schools and donors

With the User/Group/Role module, the administrator can easily manage access permissions for each user. This enables them to grant access to specific schools, and specific types of student information. This can let them make school-statistics available to donors, while keeping personal student information only available to school managers.

The software is open-source

Since the community edition of the software is open-source, it is free to install and use. This helps to make the system financially accessible for low-resource projects.

Insufficient documentation for OpenEMIS configuration

The problem I found with it was that it’s very difficult to configure for first use. The project has a website https://support.openemis.org/core/en/home-en where you can find documentation explaining how to work with the software once it’s installed and configured. The documentation is quite good for this, but I found that there isn’t enough documentation explaining how to install the system, and how to configure it so that it is ready for you to enter school data and student data. If it’s not configured correctly, it gives cryptic 404 error messages, which quickly gets very frustrating.

OpenEMIS as part of a sustainable data-management system

I recently built a sustainable data-management platform for a regional education authority in Thailand. Change is inevitable in software systems and information systems, so a sustainable system must be able to be adapted and modified easily without relying on external support. The goal for this system was to create a secure online system that could then be handed over to the local IT team, who would maintain it and change it as the education-system changes.

In order to maintain the system, the local IT team needed to easily see how the platform was configured. To help make it easier for them to maintain, I wrote documentation so they could see the steps for how to configure OpenEMIS. The documentation is below. Hopefully it will be helpful for another low-resource education project and it’ll help more people to collect high-quality data and make it easier for them to show the impact.

How to install OpenEMIS

Warning: The following steps will install the OpenEMIS system on a local development server using port 80. To use the system on a production server, you will also need to configure an SSL certificate to encrypt the data.

For this project, I carried out the OpenEMIS installation using a Docker container available through Docker hub. This is an official release maintained by the OpenEMIS team, so it is likely that the project will continue to be maintained for the long-term.

Install Docker

Docker is a technology that lets you run an application inside your computer. It makes life a lot easier for developers because you can set server configurations in one file and quickly get a project up and running. Once you have docker installed, you can use it to install OpenEMIS.

The OpenEMIS Docker project has instructions for how to install Docker on your computer/server, and how to use Docker to get the OpenEMIS app: https://hub.docker.com/r/openemis/core

With Docker installed, rundocker pull openemis/core This will pull the OpenEMIS application from the docker-hub repository and install it on your server. You can see the application running by typing docker-compose up -d and visiting http://localhost:8082 By default, the OpenEMIS docker container runs on port 8082.

To stop the application, type docker-compose down

You can change the port to port 80 by editing the <project-directory>/core/docker-compose.yaml file. You should only use port 80 for a development server. When you deploy to a production server, you will need to use port 443 and an SSL certificate before deploying OpenEMIS to a production server.

You can quickly set the application to listen on port 80 by changing the docker-compose.yaml to the following (you’ll need to turn off Apache2 if it’s installed, so that it’s not listening on port 80):

ports:

    - "80:80"

A better way to set it up is to make an Apache virtual host with port forwarding, forwarding port 80 to port 8082, the default port for docker.

First, install mod_proxy and other requirements

sudo a2enmod proxy

sudo a2enmod proxy_http

sudo a2enmod proxy_balancer

sudo a2enmod lbmethod_byrequests

Next, update the virtual host to forward port 80 to port 8082.

#/etc/apache2/sites-available/openemis.conf

<VirtualHost *:80>

    ServerAdmin webmaster@localhost

    ProxyPreserveHost On

    ProxyPass / http://127.0.0.1:8082/

    ProxyPassReverse / http://127.0.0.1:8082/

    <Directory /var/www/html/>

        Options Indexes FollowSymLinks

        AllowOverride All

        Require all granted

    </Directory>

    ErrorLog ${APACHE_LOG_DIR}/error.log

    CustomLog ${APACHE_LOG_DIR}/access.log combined

    <IfModule mod_dir.c>

        DirectoryIndex index.php index.pl index.cgi index.html index.xhtml index.htm

    </IfModule>

</VirtualHost>

The docker-compose.yaml can stay at its default, listening on port 8082.

#/var/www/openemis/core/docker-compose.yaml

ports:

    - "8082:80"

The OpenEMIS Docker application

There are three parts to the OpenEMIS docker project.

1. The OpenEMIS PHP application
   1. This is where the code of the application is stored. You access it by visiting http://localhost and logging in. When you log in for the first time, you can use the default administrator login account.
   1. Username: admin
   1. Password: demo
2. MySQL database
   1. The MySQL database stores the data for your application. It will store the schools, students, and system configurations
3. phpMyAdmin
   1. This is an application that lets you connect to your MySQL database using your web-browser. You can use it by visiting http://localhost:8083 and logging in. When you log in for the first time, you can use the default administrator login account.
   1. Username: admin
   1. Password: demo

Need More Advice for your OpenEMIS Implementation?

I’m available to discuss your setup and to help you get your system working smoothly. Get in touch to send me a message, or click here to schedule a call with me to discuss your system.