2 key factors to consider in development programmes

I examine, in my never-ending thesis, the political-economy of adopting and implementing a large global health program, the Affordable Medicines Facility – malaria or the “AMFm”. This program was designed at the global level, meaning largely in Washington, DC and Geneva, with tweaking workshops in assorted African capitals. Global actors invited select sub-Saharan African countries to apply to pilot the AMFm for two years before any decision would be made to continue, modify, scale-up, or terminate the program. One key point I make is that implementing stakeholders see pilot experiments with uncertain follow-up plans as risky: they take time and effort to set-up and they often have unclear lines of accountability, presenting risk to personal, organizational, and even national reputations. This can lead to stakeholder resistance to being involved in experimental pilots.

It should be noted from the outset that it was not fully clear what role the evidence from the pilot would play in the board’s decision or how the evidence would be interpreted. As I highlight below, this lack of clarity helped to foster feelings of risk as well as a resistance among some of the national-level stakeholders about participating in the pilot. Several critics have noted that the scale and scope and requisite new systems and relationships involved in the AMFm disqualify it from being considered a ‘pilot,’ though I use that term for continuity with most other AMFm-related writing.

In my research, my focus is on the national and sub-national processes of deciding to participate in the initial pilot (‘phase I’) stage, focusing specifically on Ghana. Besides being notable for the project scale and resources mobilized, one thing that stood out about this project is that there was a reasonable amount of resistance to piloting this program among stakeholders in several of the invited countries. I have been lucky and grateful that a set of key informants in Ghana, as well as my committee and other reviewers, have been willing to converse openly with me over several years as I have tried to untangle the reasons behind the support and resistance and to try to get the story ‘right’.

From the global perspective, the set-up of the global health pilot experiment was a paragon of planning for evidence-informed decision-making: pilot first, develop benchmarks for success and commission an independent evaluation (a well-monitored before and after comparison) — and make decisions later.

In my work, through a grounded qualitative analysis, I distil the variety of reasons for supporting and resisting Ghana’s participation in the AMFm pilot to three main types: those related to direct policy goals (in this case, increasing access to malaria medication and lowering malaria mortality), indirect policy goals (indirect insofar as they are not the explicit goals of the policy in question, such as employment and economic growth), and finally those related to risk and reputation (individual, organizational, and national). I take the latter as my main focus for the rest of this post.

A key question, on which I have been pushed, is the extent to which resistance to participation, which meant resisting an unprecedented volume of highly subsidized, high-quality anti-malarial treatments entering both the public and the private sector, emerges from the idea of the AMFm versus the idea of piloting the AMFm with uncertain follow-up plans.

Some issues, such as threats to both direct and indirect policy goals, often related to the AMFm mechanism itself, including the focus on malaria prevention rather than treatment as well as broader goals related to national pride and the support of local businesses. The idea of the AMFm itself, as well as its symbolism as a harbinger of approaches (such as market-based approaches) to global health, provoked both support and resistance.

However, some sources of resistance stemmed more directly from the piloting process itself. By evidence-informed design, the global fund gave “no assurance to continue [the AMFm] in the long-term,” in order to ensure that the evaluation of the pilot would shape their decisions. At the national level, this uncertainty proved troubling, as many local stakeholders felt it posed national, organizational, and personal risks for policy goals and reputations. Words like ‘vilification‘ and ‘chastisement‘ and ‘bitter‘ came up during key informant interviews. In a point of opposing objectives, some stakeholders may have supported the pilot if they knew the program would not be terminated (even if modified), whereas global actors wanted the pilot to see if the evidence suggested the program should (not) be terminated.

Pilot-specific concerns related to uncertainties around the sunk investments of time in setting up the needed systems and relationships, which have an uncertain life expectancy. Also, for a stakeholder trying to decide whether to support or resist a pilot, it doesn’t help when the reputational and other pay-offs from supporting it are uncertain and may only materialize should the pilot prove successful and be carried to the next stage.

A final but absolutely key set of concerns for anyone considering working with policy champions is what, precisely, the decision to continue would hinge upon. Would failure to meet benchmarks be taken as a failure of the mechanism and concept? A failure of national implementation capacity and managerial efforts in Ghana (in the face of a key donor)? A failure of individual efforts and initiatives in Ghana?

Without clarity on these questions about how accountability and blame would be distributed, national stakeholders were understandably nervous and sometimes resistant (passively or actively) to Ghana’s application to be a phase I pilot country. To paraphrase one key informant’s articulation of a common view, Phase I of the AMFm should have been an experiment on how to continue, not whether to continue, the initiative.

How does this fit in with our ideas of ideal evidence-informed decision-making about programs and policies?  This experience raises some important questions when we talk about wanting policy champions and wanting to generate rigorous evidence about those policies. Assuming that the policies and programs under study adhere to one of thedefinitions of equipoise, the results from a rigorous evaluation could go either way.

What risks does the local champion(s) of a policy face in visibly supporting a policy?
Is clear accountability established for evaluation outcomes?
Are there built-in buffers for the personal and political reputation of champions and supporters in the evaluation design?

The more we discuss to the policy relevance and uptake of evaluation evidence, and how early stakeholder buy-in is a sine qua non of these outcomes, the more we need to think about the political economy of pilots and those stepping up to support policies and the (impact) evaluation of them. Do they exist in a learning environment where glitches and null results are considered part of the process? Can evaluations help to elucidate design and implementation failures in a way that has clear lines of accountability among the ‘ideas’ people, the champions, the managers, and the implementers? These issues need to be taken seriously if we expect government officials and implementing agents to engage in pilot research to help decide the best way to move a program or policy forward (including not moving it forward at all).

This article was first published by the World Bank’s People, Spaces, Deliberation blog. Publication does not imply endorsement of views by the World Economic Forum.

To keep up with the Agenda subscribe to our weekly newsletter.

Author: Heather Lanthorn is currently completing her SD in Global Health and Population and also working as an Evaluation Specialist at the International Initiative for Impact Evaluation (3ie).

Image: A girl selling apples by the roadside waits for customers just outside the Angolan city of Lubango, January 15, 2010. REUTERS/Finbarr O’Reilly.