We seek charities that are "cost-effective" in the sense of changing lives as much as possible for as little money as possible. There are many limitations to our cost-effectiveness estimates, and we do not assess charities only – or primarily – based on their cost-effectiveness. However, because different approaches to helping people can have extremely different costs – and because we want donors to have some sense of what they're "getting" for their donations – we include cost-effectiveness as one of our criteria for a strong charity.

We feel that the following are important points to keep in mind when considering cost-effectiveness. (The rest of this page elaborates on these points.)

• Charities frequently cite misleading and overly optimistic figures for cost-effectiveness.
• Our cost-effectiveness estimates include all costs (including administrative costs) and generally look at the cost per life or life-year changed (death averted, year of blindness averted, etc.) However, there are many ways in which they do not account for all possible costs and benefits of a program.
• Because of the many limitations of cost-effectiveness estimates, we consider only extremely large differences when comparing charities.
• We consider anything under $1,000 per "significant life change" to be excellent cost-effectiveness. We consider anything over $10,000 per "significant life change" to be excessive for the cause of international aid, as it implies more than an order of magnitude higher costs than the strongest programs.

Charities frequently cite misleading cost-effectiveness figures

In The Life You Can Save, Peter Singer discusses the fact that many common claims about cost-effectiveness are misleading. We quote at length from the book, with his permission. (Note that our excerpt does not include footnotes, which are in the original.)1

Organizations often put out figures suggesting that lives can be saved for very small amounts of money. WHO, for example, estimates that many of the 3 million people who die annually from diarrhea or its complications can be saved by an extraordinarily simple recipe for oral rehydration therapy: a large pinch of salt and a fistful of sugar dissolved in a jug of clean water. This lifesaving remedy can be assembled for a few cents, if only people know about it. UNICEF estimates that the hundreds of thousands of children who still die of measles each year could be saved by a vaccine costing less than $1 a dose. And Nothing But Nets, an organization conceived by American sportswriter Rick Reilly and supported by the National Basketball Association, provides anti-mosquito bed nets to protect children in Africa from malaria, which kills a million children a year. In its literature, Nothing But Nets mentions that a $10 net can save a life: "If you give $100 to Nothing But Nets, you've saved ten lives."

[GiveWell] found similar gaps in the information on the effect of immunizing children against measles. Not every child immunized would have come down with the disease, and most who do get it, recover, so to find the cost per life saved, we must multiply the cost of the vaccine by the number of children to whom it needs to be given in order to reach a child who would have died without it. And oral rehydration treatment for diarrhea may cost only a few cents, but it costs money to get it to each home and village so that it will be available when a child needs it, and to educate families in how to use it.

The cost-effectiveness figures we use

Generally speaking, the cost-effectiveness estimates we use – most of which are taken from the Disease Control Priorities in Developing Countries report2 - have a fairly consistent set of strengths and limitations. We discuss these strengths and limitations in general, then give several examples of cost-effectiveness estimates to illustrate them.

Strengths compared to commonly cited figures:

As discussed above, many commonly cited figures are misleading because they (a) account for only a portion of costs (for example, citing the cost of oral rehydration treatment but not the cost of delivering it); and/or (b) cite "cost per item delivered" figures as opposed to "cost per life changed" figures (for example, equating bed nets delivered with deaths averted, even though there are likely many bed nets delivered for each death averted).

The cost-effectiveness estimates we use avoid these problems:

• Estimates are always given in terms of life impact, and specify what sort of life impact can be expected. The unit commonly used by the Disease Control Priorities in Developing Countries report is the disability-adjusted life-year (DALY), a standardized health-related metric (see our discussion of the DALY metric). We generally provide conversions from this metric into units such as "years of blindness averted," "deaths averted," etc.
• Estimates include all direct costs associated with interventions, from all involved funders. Thus, planning costs, management costs, distribution costs, etc. are accounted for.
• Estimates are generally based on actual costs and actual impact – measured with varying degrees of precision – from past projects (i.e., they are not pure projections, although they generally involve many assumptions).

Limitations

The estimates we use do not capture all considerations for cost-effectiveness. In particular:

• Estimates are often overly optimistic, as they are based on particularly well-studied (and likely particularly well-executed) programs and often make optimistic assumptions such as "the program will be continued over time." Examples at this blog post.
• Because estimates are based on total costs (from all funders), it is not necessarily true that they cleanly capture the effect of your donation. We address this issue – whether your donation is likely to lead to more of the activity in question, or merely to replace others' funds – separately, under the heading of "Room for more funding".
• Estimates do not include indirect costs of projects (an issue acknowledged by the Disease Control Priorities in Developing Countries report3), including (a) costs to beneficiaries (travel, waiting, etc.); (b) costs to local systems and personnel as a result of accommodating the program.

• Estimates consider only direct impact, and do not attempt to incorporate the many ways in which a program may affect people beyond (or as a result of) its immediate impact on health/income. This issue is discussed specifically as it relates to GiveWell in a recent paper by Leif Wenar.4 Some examples of possible indirect impact of a charitable program:
  • Programs generally employ local labor. On the one hand, this could create additional benefits by resulting in increased incomes for locals; on the other, it could create costs by diverting people from other productive activities. A charity that pays above-market salaries to highly skilled locals risks pulling them away from other needed work; this issue is considered further at our discussion of "Room for more funds".
  • International charities often work closely with governments (and/or send funds directly to them, as the Global Fund to fight AIDS, Tuberculosis and Malaria, among others, does). The services provided by charities (particularly health care) often supplement what could be considered the proper role of the government. Therefore, there is generally some risk that a charity will interfere with government plans and/or decrease government accountability (by increasing its resources and taking on its responsibilities). There is also generally some hope that a charity's work will help to improve the government's operations beyond the immediate impact of the project.

  A March 2010 blog post discusses this issue in the context of a specific intervention (deworming).

• Estimates are generally based on extremely limited information and are therefore extremely rough.
  • Data on the burden of disease – generally necessary in order to estimate the impact of a health intervention – is frequently estimated based on very limited data.5

  • Estimates are based on a limited number of studies of particular programs in particular regions; extrapolating results to formally similar programs in different environments, carried out at different scale, or with different personnel, adds significant uncertainty.6

  • Estimating cost-effectiveness sometimes requires making assumptions about unobservable aspects of people's behavior – for an example, see our cost-effectiveness estimate for condom distribution, which requires many assumptions about users' sexual behavior.

Examples of cost-effectiveness estimates

Most of our cost-effectiveness estimates are taken from the Disease Control Priorities in Developing Countries (2nd Edition) report,7 and we generally cite these estimates without going into the details of the calculation. Here we give examples of how cost-effectiveness calculations are carried out, to illustrate the points above about the strengths and limitations of these estimates.

Insecticide-treated nets (ITNs) for preventing malaria

The Disease Control Priorities in Developing Countries report is relatively vague about how it produces final estimates of cost-effectiveness for ITN distribution, but it appears that it is essentially adding new information and sophistication to the approach used in Goodman 2000,8 which is relatively explicit.

Goodman, Coleman, and Mills (2000) estimated cost data using cost analysis from previous trials of ITN distribution, which was supplemented with additional independent data on likely input costs under different conditions.9 Costs included "the cost of the insecticide, staff, sensitization and awareness campaign, transport, other overheads, and community time."10 Protective effects of ITNs were estimated based on a review of highly rigorous studies of past ITN distribution programs.11 Compliance and retreatment rates were also based on results from past trials, with some adjustment for the fact that "Retreatment rates under programme conditions are likely to be much lower than under trial conditions."12

The Disease Control Priorities in Developing Countries report's final estimate is $5-31 per DALY averted, which we equate to $182-$1126 per death averted (each death averted is also associated with ~320 less severe malaria episodes averted - more at our report on insecticide-treated nets). This estimate is in the same range as several independent estimates:

• An analysis of the impact of a large-scale ITN distribution program in The Gambia (discussed and compared to the Disease Control Priorities in Developing Countries report's estimate here), putting costs around $600 per death averted.
• Population Services International's estimate of its own ITN distribution program's cost-effectiveness ($12.38-$18.82 per DALY averted) as well as our estimate of the same program's cost-effectiveness ($600-$2,400 per death averted). Details here.

Vaccinations

How the Disease Control Priorities in Developing Countries report carried out its estimate: costs of existing vaccination programs were taken from 102 studies of total costs for past vaccination programs.13 Cost calculations included "labor, vaccines, supplies, transportation, communication, training, maintenance, and overhead and included the annualized value of equipment, vehicles, and building space."14 Data on how many children were immunized came from survey data.15 The report then adjusted these figures to account for current vaccine prices and the specific activities that would be involved in expanding immunization programs beyond their current coverage.16

The "cost per fully immunized child" derived from this exercise was then converted to a "cost per death averted," based on the expected impact of the antigens in the immunization program in question.17 (The report is not specific about what data it used to project the impact of vaccines, but data is available - from highly rigorous studies - for the vaccines in question.18 The result was an estimate of the cost per death averted ranging from $169 (sub-Saharan Africa) to $1,754 (Europe and Central Asia).19

An unconnected assessment of a multi-country immunization program in sub-Saharan Africa - also attempting to incorporate all direct costs - estimated that expanding routine measles immunization coverage cost $2.50 per "year of healthy life gained."20 A simple conversion calculation21 equates this figure to about $75 per child death averted. The program is not strictly comparable to the program that the Disease Control Priorities in Developing Countries report estimated cost-effectiveness for, and as a "success story" may have better-than-usual cost-effectiveness, but the figures still appear to be in the same range.

How cost-effective is cost-effective?

As stated above, we feel that common claims of donors' ability to save a life for a few dollars, or even a few cents, are generally overly optimistic. We list the most promising developing-world direct-aid programs we know of, with links to detailed write-ups including cost-effectiveness estimates (note that most of these programs were identified using sources that focus heavily on cost-effectiveness).

Based on our knowledge of these programs, we feel that a program can be considered extremely cost-effective if estimates put it in the range of $100 per disability-adjusted life-year (DALY) averted, $50 per life-year significantly changed, or $1000 per life significantly changed.

Because of the imprecision of cost-effectiveness estimates, we do not distinguish between programs within this range unless they are directly comparable (for example, similar in form but using different drugs). We consider any program within this range to be extremely cost-effective, and feel that decisions between charities running cost-effective programs should be made on the basis of confidence in their impact rather than on the basis of further differences in cost-effectiveness estimates.

However, we believe a donor should prefer programs in this range to those significantly outside it, on a cost-effectiveness basis.

Sources

• GiveWell. Website:
  • Program report: Distribution of insecticide-treated nets (ITNs) to prevent malaria
  • Program report: Expanding immunization coverage for children
  • In-depth health program reviews
  • Interpreting the DALY metric
  • Guide to "room for more funding" analysis
  • Charity review: PSI (2007-08)
• Goodman, Catherine, Paul Coleman, and Anne Mills. 2000. Economic analysis of malaria control in sub-Saharan Africa (PDF). Geneva: Global Forum for Health Research.
• Jamison, Dean T. et al., eds. 2006. Disease control priorities in developing countries (2nd Edition) (PDF). New York: Oxford University Press.
• Levine, Ruth. 2007. Case 17: Eliminating measles in southern Africa (PDF). In Millions saved: Proven successes in global health. Washington D.C.: Center for Global Development.
• Lopez, Alan D. et al., eds. 2006. Global burden of disease and risk factors (PDF). New York: Oxford University Press.
• Singer, Peter. 2009. The life you can save: Acting now to end world poverty. New York: Random House.
• Wenar, Leif. Forthcoming. Poverty is no pond: The challenges of individual action. We reviewed a draft copy of this paper.

1. 1.

   Singer 2009, Pgs 86-87.

2. 2.

   Jamison et al. 2006.

3. 3.

   "Direct and indirect costs should be distinguished, and choices should be made about which, if any, of the latter to include. In addition to the direct costs to the health system of producing an intervention, the U.S. Public Health Service guidelines (Gold and others 1996) recommend including the indirect costs to patients and their families of consuming it. This recommendation means, in particular, the value of time needed for travel, waiting, and undergoing medical tests and procedures, or the value of time used in caregiving, as well as any income forgone during treatment. Externalities, or costs imposed on third parties, such as on the school system or the environment, should also be included. The analyses in this volume generally exclude such costs and report only the direct costs of delivering interventions, partly because published analyses seldom include the various indirect costs, and they are harder to estimate.
   …
   Not only the characteristics of the interventions themselves, but also the capacity to deliver interventions greatly affect cost- effectiveness across many activities. In a complete analysis, each intervention is characterized by how demanding it is of managerial or institutional capacity. This element is difficult to measure directly, but authors often provide at least an intuitive description of how easy or hard delivery of an intervention is or what factors facilitate or impede its implementation.Where capacity to deliver several interventions together is important, authors deal explicitly with the issue, as in the chapters on health facilities (chapters 64–66), resources (chapters 71–72), service management (chapter 73), and whole packages of interventions (chapters 56 and 63)." Jamison et al. 2006, Pg 280.

4. 4.

   Wenar 2009.

5. 5.

   "Estimating prevalence and incidence is usually much harder than estimating mortality. Data collection, when done, is often limited in terms of both time and geographical area and problems of case definition abound. Not surprisingly, data are frequently incomplete, and when available, their validity may be in doubt. In particular, given differences in the way the data for incidence, prevalence, and mortality are collected, it is almost inevitable that observations are internally inconsistent. For example, when a cohort study misses more incident cases than deaths, the observed incidence will be too small to account for the observed mortality.

   To address such issues, the GBD studies have exploited two kinds of knowledge. First, disease characteristics, such as remission, case fatality rates, and duration, may be relatively constant across countries and known from studies in some populations, from clinical studies, or from expert knowledge. Supplementing observed data with expert knowledge may help to overcome a lack of data. Second, because the various epidemiological variables are causally linked by a disease process, a disease model that explicitly describes these causal pathways allows us to infer missing data if existing data are sufficient to do so." Lopez et al. 2006, Pg 74. Global Burden of Disease and Risk Factors (GBD) is the companion volume to the Disease Control Priorities in Developing Countries report.

6. 6.
   • "What would improve the kind of estimates and conclusions reported in this volume? Most crucially, more and better data are needed in low- and middle-income countries to reduce reliance on extrapolation from high-income countries and on expert judgments. The need for information starts, in some cases, with better estimates of incidence and prevalence, but even where the epidemiology is well known, data on coverage and outcomes of existing interventions are scarce. Evidence of what it would cost to change coverage of existing interventions or add new interventions, and with what results, is particularly scarce and depends heavily on assumptions. This situation is sometimes true even for activities that have been conducted widely for many years and have been extensively analyzed, notably the EPI (chapter 20)." Jamison et al. 2006, Pg 284.
   • "Because costs of inputs differ among regions, intervention costs vary even if effectiveness does not—and there are often reasons why the same intervention is more effective in one place than another. Such variation means that a single estimate of incremental or average cost-effectiveness of an intervention is not universally applicable. All estimates should ideally be local, and regional values capture only part of the real variation. For example, the average cost per DALY of chemotherapy for active or contagious tuberculosis, in the absence of HIV and AIDS, is US$15, but that figure varies from US$6 to US$31 across regions, and such wide variation is common in many chapters. Whenever the estimates of cost-effectiveness in different chapters use the same input prices, their results are comparable within a given region. Analyses that draw on published estimates for price or unit cost information are necessarily less comparable across interventions and introduce another element of variation, even in the same locale. Still more variation arises when costs or outcomes are extrapolated from one country or region to another." Jamison et al. 2006, Pgs 282-3.

7. 7.

   Jamison et al. 2006.

8. 8.

   "Goodman, Coleman, and Mills’s (2000) study represents the most thorough attempt to compare the cost-effectiveness of a wide range of malaria control interventions. …

   The following analysis incorporates new knowledge on the effects of interventions and on their costs for a low-income, Sub-Saharan African population living in an area of high, stable transmission. The modeling draws on a wide range of sources on the costs and effects of each intervention, extrapolated across settings and operational conditions. The approach allows for changing cost-effectiveness over time, for example, as resistance to antimalarial drugs or insecticides increases. To address problems of uncertainty in relation to many of the parameters, we used probabilistic sensitivity analysis, which allows for multivariate uncertainty by assigning ranges rather than point estimates to input variables. We assumed that cost and effectiveness input variables follow uniform triangular or normal continuous probability distributions (Mulligan, Morel, and Mills 2005).

   We consider the cost-effectiveness of a limited subset of interventions: ITNs, IRS, IPT during pregnancy, and patient management with a change of the first-line drug. We include costs to the provider and the community and incremental out-of-pocket expenses for households, but because of major valuation and measurement problems, we do not include the indirect costs of patients’ time to seek care and of the lost productivity resulting from morbidity. We consider only gross costs for all interventions except patient management, given the uncertainty inherent in estimating savings.

   Insecticide-Treated Nets. We based our analysis of ITNs on the delivery mechanism used in the WHO Special Programme for Research and Training in Tropical Diseases (WHO/TDR) trials, where householders, community health workers, and program staff worked together to treat the nets. In relation to insecticide, we considered permethrin and deltamethrin. Deltamethrin is effective for a year; thus, re-treatment is annual. Permethrin lasts for six months; thus, we assumed two treatments per year if the transmission season is longer than six months. The activities undertaken were the training of staff and community health workers, a campaign to inform the community about the intervention, the procurement and transport of the insecticide and nets, and the initial treatment and the re-treatment of the nets. We calculated cost-effectiveness for each intervention for two scenarios: one whereby nets were distributed to households and the second whereby treatment was arranged for existing nets. We drew estimates of the effectiveness of ITNS from a recent meta-analysis of WHO/TDR-sponsored trials conducted in Sub- Saharan Africa (Lengeler 2004). We adjusted the key parameter and effectiveness estimates to account for the proportion of children sleeping and not sleeping under a recently treated net." Jamison et al. 2006, Pgs 422-3.

9. 9.

   "The cost per child for the purchase, distribution and annual treatment of ITNs was calculated using estimates of input parameters drawn from the economic evaluations that accompanied the WHO/TDR trials (20-23) , supplemented by additional technical and economic data from other published and unpublished sources and expert consultation. The costing included the cost of the insecticide, staff, sensitization and awareness campaign, transport, other overheads, and community time." Goodman, Coleman, and Mills 2000, Pg 32. More detail follows on the sources used to estimate costs.

10. 10.

    Goodman, Coleman, and Mills 2000, Pg 32.

11. 11.

    "Estimates of the effectiveness of ITNs were drawn from the Cochrane meta-analysis of WHO/TDR trials conducted in SSA(17) . Children aged 1–59 months sleeping under ITNs experienced a significant reduction in all cause mortality of 19% (95% CI 14%–24%) and a reduction in clinical episodes of 46% (95% CI 41%–51%). The data on the reductions in mortality are based on the results of the meta-analysis for the five trials that included mortality as an outcome. The confidence intervals for the reductions in morbidity and mortality are slightly underestimated as they are not adjusted for group randomization. However, adjusting the confidence intervals would not change the significance of the effectiveness estimates (C. Lengeler, personal communication). The proportional reductions in the incidence of severe malaria and the prevalence of malaria associated anaemia were set to equal the reduction in clinical episodes.

    Effectiveness was assumed to be the same for the “Treatment and Nets” scenario and the “Insecticide Treatment only” scenario, and was assumed not to vary with the length of the transmission season, as in the meta-analysis all scenarios were combined. The two insecticides considered were assumed to be of equal efficacy, although all the WHO/TDR trials used permethrin. The effectiveness of nets treated with deltamethrin and permethrin has never been directly compared in epidemiological terms, but the entomological evidence clearly suggests that deltamethrin is at least as effective as permethrin(18)." Goodman, Coleman, and Mills 2000, Pgs 31-32. The Cochrane review mentioned in this quote is also discussed at our overview of evidence regarding ITNs.

12. 12.

    "The effectiveness estimates were adjusted to account for non-compliance. A fully compliant child was defined as one whose net had recently been treated and who slept under the net. In practice, households may not re-treat their nets, and children may not sleep under them if they sleep outside in hot weather, the nets are used for other family members, or the nets are taken away, destroyed or sold. The average compliance of the WHO/TDR trials from which the effectiveness estimates were taken was 65% (4-7) . Retreatment rates under programme conditions are likely to be much lower than under trial conditions, and were given a range between 20% and 80% (19) , while between 50% and 97% of children were assumed to use the net correctly (4-7) . These two estimates were multiplied together to calculate actual compliance. A linear relationship was assumed between compliance and effectiveness, such that zero compliance results in zero effectiveness, and 65% compliance results in the reductions in mortality and morbidity found in the meta-analysis. The effectiveness results from the meta-analysis were then multiplied by the ratio of actual compliance to trial compliance to estimate effectiveness under a programme situation. All effectiveness input parameters are listed in Table 3.2." Goodman, Coleman, and Mills 2000, Pg 32.

13. 13.

    "Our literature review found 102 estimates of total and unit immunization program costs from 27 countries between 1979 and 2003 for different immunization delivery strategies (Berman and others 1991;1 Beutels 1998, 2001; Brenzel 2005; Brenzel and Claquin 1994; Brinsmead, Hill, and Walker 2004; Creese 1986; Creese and Domínguez-Ugá 1987; Domínguez- Ugá 1988; Edmunds and others 2000; Griffiths and others 2004; Levin and others 2001; Pegurri, Fox-Rushby, and Walker 2005; Robertson and others 1992; Soucat and others 1997; Steinglass, Brenzel, and Percy 1993). All costs were converted to 2001 U.S. dollar equivalents. Because total and unit costs are related to population size, table 20.4 reports population-weighted results only. National immunization program refers to total national costs for all strategies." Jamison et al. 2006, Pg 399.

14. 14.

    Jamison et al. 2006, Pg 399.

15. 15.

    "The number of FICs [fully immunized children] in these studies was measured using community-based sample surveys (Henderson and Sundaresen 1982)." Jamison et al. 2006, Pg 399.

16. 16.

    "WHO (2004) estimates that in 2001, 30 million children were inadequately immunized with DTP. Achieving higher coverage rates by improving access for remote populations, accelerating immunization delivery strategies, and introducing new vaccines will mean increasing the level of investment (Batt, Fox- Rushby, and Castillo-Riquelme 2004).

    We estimated the costs of scaling up EPI coverage for a hypothetical population of 1 million in each region between 2002 and 2011. Costs were reported in 2001 dollars, and a 3 percent discount rate was applied. Brenzel (2005) provides details on the methods. The costs of scaling up coverage are based on vaccine and delivery costs per dose. We derived vaccine costs from the unit price of each vaccine (provided by WHO, UNICEF, and the Vaccine Fund); wastage rates for vaccines and injection supplies by strategy; the required injection supplies; and the number of doses per FIC. A 2-percent adjustment was made for inflation. We used data on the cost per FIC generated earlier to derive delivery costs per dose by strategy and region by subtracting the costs of vaccines, injection supplies, and fixed costs.

    Fixed costs were excluded from the scaling-up exercise because they were assumed to remain constant during the projection period." Jamison et al. 2006, Pg 401.

17. 17.

    "A proxy for the total number of deaths averted is the sum of the individual deaths averted for each antigen in the traditional EPI [expanded program on immunization]. This figure may overestimate the actual number of deaths averted by fully immunizing children and therefore underestimate the cost per death averted. However, the values estimated by region appear to support previously reported estimates, and direct estimation of deaths averted was impossible given data and model limitations." Jamison et al. 2006, Pg 408 (note 6).

18. 18.

    See our discussion of the immunization program in question.

19. 19.

    Jamison et al. 2006, Pg 403, Table 20.6.

20. 20.

    "The majority of the funding for the measles initiative came from national budgets. An estimate of the total cost of the program is $26.4 million, with the average cost per immunized child at $1.10. The cost of increasing routine coverage from 50 percent to 80 percent has been estimated at about $2.50 per year of healthy life gained, making measles immunization an extremely cost-effective intervention." Levine 2007, Case 17, Pg 1.

21. 21.

    Lopez et al. 2006, Pg 402, Table 5.1 equates the death of a 0-5-year-old to ~30 years of life lost (using 3% discounting and no age weights, as the Disease Control Priorities in Developing Countries report does, Jamison et al. 2006, Pg 29: "These are labeled DALYs (r,k) with the first number indicating the discount rate in percent per year and the second indicating whether uniform or nonuniform age weights were used. The most widely reported variant on the DALY is the DALY (3,1)— that is, one that uses a 3 percent discount rate and nonuniform age weighting. This chapter and the companion volume report DALYs (3,0)—that is, with discounting but uniform age weighting.").