Quantcast
home > international > Technical Information > Criteria and Process > Program Evaluation Criteria

Criteria for evaluating programs

Print
Email

This page outlines the major criteria we use to evaluate different types of programs in international aid. We examine particularly promising programs in more depth than others, and have a preference for charities that exclusively or primarily focus on these programs.

In brief, we consider programs to be promising when they are associated with past demonstrated success in improving people's lives - i.e., have a track record, and/or when they are recommended by experts who explicitly seek to compare programs and identify the most promising ones.

Table of Contents

Track record

We have a strong preference for programs that have had demonstrable past success in improving lives - a preference we believe is particularly appropriate for individual donors seeking to help people they ultimately know very little about. The evidence we consider falls into two categories:

  • "Micro" evidence refers to evidence from relatively contained, small-scale studies of a program's effects. We prefer to rely on randomized, controlled evaluations (more on this approach on the Poverty Action Lab website and in Duflo and Kremer 2008); we use the term "high quality" to refer to evaluations that use this basic approach. When necessary, we also sometimes utilize non-randomized evaluations, but generally focus more on "high quality" studies.

    A well-designed "micro" study can leave very little doubt about a program's effects at a particular time and place. On the other hand, it leaves questions about how the results of a small, carefully executed program would translate on a larger scale - in new environments, often with less oversight, and with other concerns as well. For example, a study may find that people with loans earn more than people without loans, but this could be a simple matter of competitive advantage of those already receiving loans - it may be that extending loans to everyone would not make an area better off on net.1

  • "Macro" evidence refers to evidence from programs carried out on a large scale (regional, national, or multinational) without separating people into "treatment groups" and "control groups." "Macro" evidence inherently does not come from carefully controlled studies, and so it can always potentially be wrongly attributing impact to a program - for example, one might observe that child mortality fell sharply following the introduction of a vaccination program, when in fact other factors (such as generally improving standards of living) had more to do with the decline. However, we consider a "macro" story to be an important indicator that a program can work on a large scale.

We have the highest confidence in programs that are supported by both types of evidence - programs that have been rigorously demonstrated to be effective on a small scale, and that have been associated (even if loosely) with larger-scale successes.

Below are sources we've found particularly useful in assessing programs' track records.

Cochrane Library

The Cochrane Library2 publishes reviews of the evidence for healthcare interventions, focusing on high-quality "micro" evaluations (as defined above). We have found that its reports generally review a large number of studies and are very clear about the findings, strengths and weaknesses of these studies. For health programs, when there are often many high-quality evaluations available, we therefore use Cochrane as our main source of information on "micro" evidence when possible.

J-PAL and IPA

The Abdul Latif Jameel Poverty Action Lab (J-PAL) and Innovators for Poverty Action (IPA) are groups that focus on the use of randomized (what we call "high quality") evaluations of developing-world anti-poverty programs. We believe that their lists of working papers,3 which include many evaluations that were not directly carried out by the organizations themselves, represent a significant proportion of the available high-quality "micro" evidence on relevant non-health-centered programs.

Millions Saved: Proven Successes in Global Health

Millions Saved: Proven Successes in Global Health (cited here as Levine et al. 2004) is a publication by the Center for Global Development with 20 case studies on large-scale success stories; it has been cited multiple times as an example of success stories in international aid.4 It is not a complete list,5 and we have not relied on it exclusively, but it is the only compilation of large-scale success stories we have found (in global health or in other areas) that is relatively clear about its information sources and about its criteria.6 We have done some vetting of its conclusions and feel that it is generally a fairly reliable source.7 It is our primary source for "macro" evidence.

Recommendations from comparative analyses

Since we seek to identify the most promising programs, we rely on experts who explicitly seek to compare interventions - using reasonably clear, consistent, and transparent criteria, and without the aim of advocating for a particular charity. We have essentially only found one suitable source (as the authors for the below three publications overlap heavily). We welcome referrals to any suitable sources we may have missed.

Disease Control Priorities Report

Disease Control Priorities in Developing Countries, 2nd edition is a publication produced for the World Bank (and in collaboration with the World Health Organization) by over 300 contributors.8 It provides information on a variety of health programs, discussing how they're implemented, potential risks and concerns, and estimated cost-effectiveness.

This report often gives references on the effectiveness of an intervention, but in our experience the quality of such references (as defined above) varies widely, and the references given are rarely as comprehensive or clearly described as they are in Cochrane reviews. We therefore use this publication more as a source of general information and expert opinion than as a clear indicator of a program's track record.

Copenhagen Consensus

We use the work of the 2008 Copenhagen Consensus, a panel of experts that explicitly aimed to answer the question: "Imagine you had $75bn to donate to worthwhile causes. What would you do, and where should we start?"9

The Copenhagen Consensus lists 30 ranked areas to focus on.10 Its scope is broader than ours, covering a variety of topics. We read the papers that were directly relevant to our focus areas11 and flagged endorsed programs.

We have found that the Copenhagen Consensus analysis tends to focus heavily on cost-effectiveness estimates, with little discussion of the differentials in track records between different programs.

Copenhagen Consensus 2008 challenge paper on diseases

The paper on diseases for the 2008 Copenhagen Consensus is authored by Dean T. Jamison, Prabat Jha, and David Bloom. It explicitly aims12 to build on the work of the Disease Control Priorities Report (about which more above). It presents its own list of ranked priorities, which differ (in rank order) from the Copenhagen Consensus master list,13 as well as several unranked priorities. We have noted its conclusions separately.

Like the Copenhagen Consensus, Jamison, Jha, and Bloom (2008) focuses largely on cost-effectiveness analysis; it mentions but does not compare the track records of interventions.

Sources

  1. 1.

    For more on this idea, see Rodrik 2009 and Deaton 2009.

  2. 2.

    Cochrane Collaboration, "Cochrane Reviews."

  3. 3.

    • Poverty Action Lab, "Publications."
    • Innovations for Poverty Action, "Publications."

  4. 4.

    Examples: Moss, Pettersson, and van de Walle 2006, Pg 5; Easterly 2009, Pg 407.

  5. 5.

    See Center for Global Development, "Frequently Asked Questions."

  6. 6.

    Center for Global Development, "What Is Success?"

  7. 7.

    Our notes from vetting this publication were sent to our public e-mail list, see GiveWell, "Vetting 'Millions Saved'."

    See also GiveWell, "Analysis of a Success Story: Implementation of the DOTS Strategy in China," for a more detailed analysis of one case study.

  8. 8.

    Jamison et al. 2006, Pgs iv, xxv-xxxiv.

  9. 9.

    Copenhagen Consensus Center, "The Experts."
    Quote from Copenhagen Consensus Center, "The Basic Idea."

  10. 10.

    Copenhagen Consensus Center, "Copenhagen Consensus 2008."

  11. 11.

    The directly relevant papers from this project are: Horton, Alderman, and Rivera 2008; Jamison, Jha, and Bloom 2008; King, Klasen, and Porter 2007; and Whittington et al. 2008.

  12. 12.

    See Jamison, Jha, and Bloom 2008, Pg 3.

  13. 13.

    Jamison, Jha, and Bloom 2008, Pg 51. For the Copenhagen Consensus master list, see Copenhagen Consensus Center, "Copenhagen Consensus 2008." To give one example of a difference in priority, Jamison, Jha, and Bloom (2008) ranks tuberculosis treatment first, while the master list ranks it below three of the other interventions listed in Jamison, Jha, and Bloom (2008) (expanded immunization coverage; heart attack acute management; malaria prevention and treatment).

Powered by Olark

Clicky Web Analytics