Assistant Scientist at Johns Hopkins Bloomberg School of Public Health, Dr Amnesty LeFevre, talks about the importance of mHealth monitoring and evaluation (M&E) and how South Africa has become a global leader in implementing national mHealth projects since the launch of MomConnect.

Dr LeFevre will be chairing a Round Table at eHealthALIVE2016 and talking to Ministries of Health from across the SADC region about lessons learned and strategies going forward. Get your ticket now and be part of this transformational event.

Tell us about your background in healthcare.

I started working in the field of public health over 15 years ago, initially doing volunteer work in Ecuador vaccinating monkeys, cats and dogs against rabies and later in working on Chagas disease prevention in rural Bolivia. While in University I trained as a paramedic and spent breaks working in the only small medical clinic providing health services to the town I’m from in rural Texas.  After finishing a master’s degree in public health at Johns Hopkins, I took a job with WHO in the Health Systems division of their Cambodia country office. I eventually returned to Johns Hopkins, took a faculty position in 2004, finished a PhD, and never stopped wanting to do research on how to improve health systems.

From designing large scale programmes in Myanmar; evaluating community based delivery of maternal and new-born care in Bangladesh, Pakistan and Tanzania; to measuring maternal health in Afghanistan, I have really been very privileged to work with amazing people on a wide array of programme evaluations in Asia and Africa. In the last five years, I have broadened my focus to look at the introduction of and evaluation of digital health technologies, including mHealth, on improving the delivery and utilisation of health services. As the monitoring and evaluation technical lead for the Johns Hopkins University Global mHealth Initiative, I’ve been fortunate enough to work with colleagues to provide monitoring and evaluation technical support to a number of programmes, including the UN Foundation and WHO’s Innovation Working Group (IWG) Catalytic mHealth Grants Program. It’s this interface between technology, service delivery and utilisation that I’m most excited about.

Do you think it was your start in emergency medicine that made you realise the need for information systems and quality data?

My initial training as a paramedic was important, but I think it was really during my time with WHO conducting a study on health information systems that I first began to see just how much time healthcare providers spend collecting data. In Cambodia and in every country I’ve worked in since, providers undertake this time intensive and laborious process of filling out paper-based registers with data that is of minimal to no use in the day-to-day delivery of services. It’s just shocking to me what a missed opportunity that is and yet with the increased use of technology in the health sector, we have the potential to change that. Not only can we improve how and what data are collected but we can improve the timeliness and use of data to enhance the quality and continuity of care provided. The further ability to foster a direct line of communication with users using mobile phones means that we can too have the opportunity to empower patients by putting information in their hands about their right to health services and how to improve their personal health. Digital health is a game changer in the health industry, from both the supply and demand side.

You mentioned the need to focus on maternal health. Tell us about some of the challenges you’ve encountered in this area and what do we already know from M&E in maternal and child health?

When I first joined Johns Hopkins I worked on a series of community-based maternal, new-born and child health services that used frontline health workers to address gaps in access to quality care. All of these programmes were in high disease burdened regions and focused primarily on new-borns because that’s the stage of infancy where the majority of deaths occur and yet utilisation of services is so low. While these programmes provided basic health information to mothers – viewing them as a gateway to the child – they too missed critical opportunities for identifying and preventing maternal infections. I think the direction we need to be moving in is one that not only recognises the need for providing integrated family healthcare (starting not just with the new-born but with the mother) but also takes steps towards addressing individuals’ full health and well-being.

While the situation here in South Africa is a bit of an anomaly for the region in that there’s a strong degree of utilisation of and continuity of care from pregnancy through to delivery care, as with most other countries in the region, care seeking during the post-partum period is limited. In Pakistan and Bangladesh, we’ve just finished evaluating an algorithm that would have a frontline health worker expand the traditional package of new-born care assessments to include postpartum infection screening for mothers. In Cote d’Ivoire, I’m working with Jhpiego to expand integrated care for persons living with HIV to also include screening for other chronic diseases including TB, diabetes and hypertension. It’s exciting to think about pushing the boundaries on what care can optimally be provided for individuals and their family units on the frontlines, particularly in areas where the facility-based services are limited and/or healthcare providers are overwhelmed.

What else do we know about South Africa’s use of eHealth?

South Africa is a pioneer for the rest of the region in showcasing how health systems can function well. That does not mean that there aren’t problems, but it means that there’s a lot that the rest of sub-Saharan Africa can learn – both in terms of how to optimise healthcare and, most importantly, how to integrate technology into health services. That integration is happening better and more efficiently in SA than anywhere else in the world that I’ve seen.

For example, MomConnect is a global first as a large-scale national mHealth project. There are others but none in the region at this scale or with the same level support from government. The back-end system and infrastructure around it were designed by the best in the world (also based here in South Africa) and a lot of thought went into planning how other health programmes could be integrated into it. While I hope MomConnect is the first of many programmes to come, the national level leadership, thinking and support around technology is something that I think is a lighthouse for the region in terms of what can be done and how we can best engage with technology.

So what are some of those lessons – tell us about the barriers to scaling eHealth solutions and how it can be overcome.

One big problem is that a lot of eHealth projects globally try to follow the traditional trajectory of starting with a small pilot phase and then scaling from the ground up. That might make sense for new technologies where people are still trying to determine the feasibility and usability. However, as the technology stabilises and increases in maturity from pilot to demonstration and scale up, it’s critical that the interoperability layer be designed from the outset to accommodate that and consider integration with other programmes at different levels of the health system. If that interoperability layer cannot accommodate scale nor outward expansion or integration with other programmes, scaling up is a non-starter.

Two other critical impediments to scaling and sustaining eHealth interventions have been 1. Robust evidence on their effectiveness and value for money; and 2. Buy-in and leadership from key stakeholders, including the government.

Throughout the last decade, we’ve seen a proliferation of mHealth programmes – over 600 in this region alone—and yet the evidence on the effectiveness of these in achieving improvements in utilisation, service delivery, or even health outcomes is limited. As a community we need to be pushing for more rigorous evaluations.

Finally, and perhaps most importantly, implementation needs to occur hand in hand with key stakeholders, including governments, and ideally occur as part of a larger vision for eHealth rather than fragmented programmes. My hope for the region is that we start to see more countries develop 3-5 year eHealth plans which paint a vision of how technology is going to be used in the health sector and guide innovation. Ideally these would include consideration of both the creation of an interoperability layer and how different technologies, and in what order of priority, are going to be integrated into that. That doesn’t mean that you have to start with all the answers, it just means looking for the service deficits and gaps, and how to prioritise technology use to address those gaps.

Walk us through the critical points and milestones of evaluating an eHealth system.

We’ve just finished developing an M&E Workbook for WHO, and as part of that we’ve stipulated stages of maturity and stages of evaluation.

A stage of maturity is where technology sits within a particular life cycle, starting with a pre-prototype phase all the way through to when a solution is ready for delivery at scale. For each stage of maturity, there is a corresponding stage of evaluation that includes assessments to determine feasibility/usability, efficacy, effectiveness, or assess implementation factors.

Depending on where your programme sits in a stage of maturity or evaluation, evidence needs and thus M&E activities may differ. For programmes in a feasibility/usability stage you’re looking at the functionality, stability and performance of the system. More mature programmes that have gone through pilot phase, but are not quite ready to scale, might be implemented as part of a ‘demonstration’ stage of maturity and ‘effectiveness’ stage of evaluation. For these programmes, M&E activities might focus on measuring not only outcomes but processes around the technological and behavioural performance of the programme: fixing bugs within the system; ensuring that the messages go where you want them to; looking at how users engage with the technology; what the technological barriers are to using it; and what the human barriers are to using it.

Why is transparency so important when it comes to implementing eHealth on any scale?

In health systems research we use the term “programme drift” which relates to the deviations from the idealised protocols as to how a project should run, and how those deviations contribute to deficits in health service delivery. What’s important to remember is that deviations from plans are bound to happen in real-world scenarios. For eHealth programmes, to date, few have really published data on how well the technology performs and really does what you say it will do. If you have a messaging programme, do registered users actually receive and access the messages they’re supposed to? Then do they adopt the intended behaviours or practices?

We just finished evaluating a programme in the region where only 5% of intended voice messages reached and were listened to by users in the postpartum period. The breakdown was not in users not listening to messages, but rather in messages not being received for a variety factors. Had we not looked at how the technology and the behavioural performance intersect, we might have overestimated actual exposure to the programme.  In the case of this programme, the implementing partner was tremendously supportive and transparent – they really drove the M&E process. Even when the findings highlighted challenges with the technology platform itself, there has been no question that we’d still proceed with publishing the findings, and trying to promote similar discourse for other programmes. I give them a lot of credit for that, particularly given the investment into the technology and programme, and the pressure to promote its use elsewhere.

As a community, we need to be more open and transparent about how well our systems are working so we can all learn from it. Unless we as a research programme community start to push one another to publish data we have no mechanism to hold people accountable to their claims and drive the adoption of eHealth forward.

For more information contact, like us on Facebook or tweet us @eHealthNewsZA.