Blog
Identifying 'What Works' in Foreign Aid

24 September 2013

Rachel M. Gisselquist and Miguel Niño-Zarazúa

UNU-WIDER's ReCom programme is centred around four core questions: what works, what could work, what is scalable, and what is transferrable in foreign aid? One of the first challenges in addressing these questions is a methodological one: what is the best way to identify the policies that 'work'? This question has obvious links to another major debate in development studies over the use of experimental methods, and in particular, of randomized controlled trials (RCTs). While some researchers have argued that RCTs are the best means of determining what works in development policy (e.g., Banerje 2007; Druckman et al. 2006), others raise a number of strong concerns (see, e.g., Deaton 2009; Ravallion 2009).

In a previous WIDERAngle, we focused on the contributions and limits of experimental methods in the study of government performance, building on preliminary findings from UNU-WIDER's project on this topic. In short, we argued that governance-related topics highlight many of the major challenges for experimental approaches, including the ability to address 'big questions' and 'big theories', to examine medium- and long-term processes, to generalize beyond the experimental site, and to reasonably and ethically manipulate key variables of interest—see Gisselquist, Niño-Zarazúa, and Sajuría (forthcoming), and Gisselquist and Niño-Zarazúa (2013).

UNU-WIDER's on-going project on 'Experimental and Non-Experimental Methods in the Study of Government Performance: Contributions and Limits' now comprises a collection of seven studies—all of which are, or will be shortly, published as UNU-WIDER working papers—which together outline an alternative middle ground in the extensive debate over experimental methods. Collectively, these studies interrogate how we come to know what we know with particular reference to questions and testable hypotheses relative to the observed variation in government performance across developing (and also developed) countries. The elegance of findings from experimental research, we suggest, has had a tendency to promote method-driven, rather than question- and theory-driven, research. In other words, it has encouraged research that asks the questions that can be addressed under very specific conditions, rather than research that begins with major theories, questions and hypotheses, and then considers which methods are most appropriate for addressing and testing them. Collectively, the set of studies under the present UNU-WIDER project, tries to turn this around.

Two of the studies in this collection address specifically the use of experimental methods in the study of government performance. Our paper, which frames the collection, draws on a review of the governance literature and a systematic review of experimental and quasi-experimental studies on governance to discuss the major contributions and limits of experimental research in this area. Armando Barrientos and Juan M. Villa's study explores the political economy of the use of experimental and non-experimental methods in the evaluation of anti-poverty transfer programmes in Latin America and sub-Saharan Africa.

The other five papers in the collection focus on how researchers can address some of the key challenges to the use of experimental methods identified in these two studies by both (1) improving experimental approaches, and (2) using non-experimental methods and combining experimental and non-experimental techniques. Two papers focus on the first option, tackling two major challenges to experimental approaches: Fernando Martel García and Leonard Wantchekon address the important issue of external validity, using the structural causal language of directed acyclic graphs (DAGs) to argue that external validity should be not so much a matter of whether an experiment has been replicated in multiple contexts, but rather whether it addresses generalizable theories. Macartan Humphreys explores the challenging topic of ethics in field experimentation, providing a detailed discussion and critique of various approaches to informed consent which might help to experimental researchers in the development of ethical guidelines.

Moving beyond experimental research, three papers discuss non-experimental and 'mixed' methods: Rajeev Dehejia provides a synthesis of non-experimental methods and compares them with experimental methods in terms of internal and external validity; Michael Bratton explores what can and cannot be learned from public opinion surveys and how experiments may complement survey-based approaches, drawing in particular on analysis of the Afrobarometer surveys; and Kate Baldwin and Rikhil Bhavnani explore what has been and can be learned from 'ancillary' experiments, which use existing experimental data along with newly-collected data to address new research questions.

More broadly, this collection underscores that although experimental research has much to add to the study of development policy and foreign aid more specifically, there are also clear limits to what can be learned from experimental approaches and that multiple methods are necessary. In keeping with these findings, the ReCom programme draws on a variety of approaches to addressing its core questions, including experimental methods, quasi-experimental analysis, observational and survey research, comparative historical analysis, and interviews and fieldwork.

Rachel M. Gisselquist and Miguel Niño-Zarazúa are both UNU-WIDER Research Fellows.
 

References

Banerjee, Abhijit Vinayak (2007). 'Making Aid Work'. In Abhijit Vinayak Banerjee (ed.) Making Aid Work, . Cambridge, MA: MIT Press.

Deaton, Angus (2009). 'Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development'.  Princeton University

Druckman, James N., Donald P. Green, James H. Kuklinski, and Arthur Lupia (2006). The Growth and Development of Experimental Research in Political Science. American Political Science Review, 100(04): 627-635.

Gisselquist, Rachel M., and Miguel Niño-Zarazua (2013). 'What can experiments tell us about how to improve governance?' WIDER Working Paper No. 2013/077. Helsinki: UNU-WIDER.

Gisselquist, Rachel, Miguel Niño-Zarazúa, and Javier Sajuría (forthcoming). 'Improving Government Performance in Developing Countries. A Systematic Review'. Helsinki: UNU-WIDER.

Ravallion, Martin (2009). 'Should the Randomistas Rule?'. Economists' Voice, 6(2).
 

WIDERAngle newsletter
September 2013
ISSN 1238-9544

http://creativecommons.org/