Published on GiveWell (http://givewell.org)

Home > Criteria for evaluating programs

Criteria for evaluating programs

This page outlines the major criteria we use to evaluate different types of programs in international aid. We examine particularly promising programs in more depth than others, and have a preference for charities that exclusively or primarily focus on these programs. In brief, we consider programs to be promising when they are associated with past demonstrated success in improving people's lives - i.e., have a track record, and/or when they are recommended by experts who explicitly seek to compare programs and identify the most promising ones.
Table of contents 

Track record

We have a strong preference for programs that have had demonstrable past success in improving lives - a preference we believe is particularly appropriate for individual donors seeking to help people they ultimately know very little about. The evidence we consider falls into two categories: We have the highest confidence in programs that are supported by both types of evidence - programs that have been rigorously demonstrated to be effective on a small scale, and that have been associated (even if loosely) with larger-scale successes.
Below are sources we've found particularly useful in assessing programs' track records.

Cochrane Library

The Cochrane Library2 publishes reviews of the evidence for healthcare interventions, focusing on high-quality "micro" evaluations (as defined above). We have found that its reports generally review a large number of studies and are very clear about the findings, strengths and weaknesses of these studies. For health programs, when there are often many high-quality evaluations available, we therefore use Cochrane as our main source of information on "micro" evidence when possible.

J-PAL and IPA

The Abdul Latif Jameel Poverty Action Lab (J-PAL) and Innovators for Poverty Action (IPA) are groups that focus on the use of randomized (what we call "high quality") evaluations of developing-world anti-poverty programs. We believe that their lists of working papers,3 which include many evaluations that were not directly carried out by the organizations themselves, represent a significant proportion of the available high-quality "micro" evidence on relevant non-health-centered programs.

Millions Saved: Proven Successes in Global Health

Millions Saved: Proven Successes in Global Health (cited here as Levine et al. 2004) is a publication by the Center for Global Development with 20 case studies on large-scale success stories; it has been cited multiple times as an example of success stories in international aid.4 It is not a complete list,5 and we have not relied on it exclusively, but it is the only compilation of large-scale success stories we have found (in global health or in other areas) that is relatively clear about its information sources and about its criteria.6 We have done some vetting of its conclusions and feel that it is generally a fairly reliable source.7 It is our primary source for "macro" evidence.

Recommendations from comparative analyses

Since we seek to identify the most promising programs, we rely on experts who explicitly seek to compare interventions - using reasonably clear, consistent, and transparent criteria, and without the aim of advocating for a particular charity. We have essentially only found one suitable source (as the authors for the below three publications overlap heavily). We welcome referrals to any suitable sources we may have missed.

Disease Control Priorities Report

Disease Control Priorities in Developing Countries, 2nd edition is a publication produced for the World Bank (and in collaboration with the World Health Organization) by over 300 contributors.8 It provides information on a variety of health programs, discussing how they're implemented, potential risks and concerns, and estimated cost-effectiveness. This report often gives references on the effectiveness of an intervention, but in our experience the quality of such references (as defined above) varies widely, and the references given are rarely as comprehensive or clearly described as they are in Cochrane reviews. We therefore use this publication more as a source of general information and expert opinion than as a clear indicator of a program's track record. DCP2"); ?>

Copenhagen Consensus

We use the work of the 2008 Copenhagen Consensus, a panel of experts that explicitly aimed to answer the question: "Imagine you had $75bn to donate to worthwhile causes. What would you do, and where should we start?"9 The Copenhagen Consensus lists 30 ranked areas to focus on.10 Its scope is broader than ours, covering a variety of topics. We read the papers that were directly relevant to our focus areas11 and flagged endorsed programs. We have found that the Copenhagen Consensus analysis tends to focus heavily on cost-effectiveness estimates, with little discussion of the differentials in track records between different programs.

Copenhagen Consensus 2008 challenge paper on diseases

The paper on diseases for the 2008 Copenhagen Consensus is authored by Dean T. Jamison, Prabat Jha, and David Bloom. It explicitly aims12 to build on the work of the Disease Control Priorities Report (about which more above). It presents its own list of ranked priorities, which differ (in rank order) from the Copenhagen Consensus master list,13 as well as several unranked priorities. We have noted its conclusions separately. Like the Copenhagen Consensus, Jamison, Jha, and Bloom (2008) focuses largely on cost-effectiveness analysis; it mentions but does not compare the track records of interventions.

Sources

Creative Commons License
This work is licensed under a Creative Commons Attribution-Noncommercial-Share Alike 3.0 United States License.
GiveWell, aka The Clear Fund (a tax-exempt 501(c)(3) public charity)
320 7th Ave #232 - Brooklyn, NY 11215

About        Site Map        Contact       

Source URL (retrieved on 02/10/2012 - 08:23): http://givewell.org/international/technical/criteria/program-evaluation