You are here

2020 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2020. For past versions of our CEA, see this page.

Version 1 — Published September 11, 2020

Link to the cost-effectiveness analysis (CEA) file: 2020 CEA — Version 1

For the impact of each of the following changes on our cost-effectiveness estimates, see this spreadsheet.

Change 1: Corrected errors in our cost per seasonal malaria chemoprevention (SMC) treatment estimates for Malaria Consortium

We found and corrected two errors in our 2019 cost per SMC treatment estimate for Malaria Consortium.1 The corrections led to our cost per SMC treatment estimate for Chad changing from $8.77 to $8.17 and our estimate for Nigeria changing from $7.70 to $7.79.2

Change 2: Updated data on population distributions by age for our CEA of the Against Malaria Foundation (AMF)

Our CEA for AMF relies on estimates of the distribution of populations by age for our calculations of "Mortality reduction in children under 5," "Mortality reduction in individuals 5+ years old," and "Income increases — ages 14 and under."3 We previously used multiple sources for data on population distributions by age.4

For consistency with our CEA of Malaria Consortium's SMC program, we now use data from the Institute for Health Metrics and Evaluation’s Global Burden of Disease Study 2017 Population Estimates for all countries in our CEA for AMF.5

Change 3: Corrected an error in our leverage and funging adjustments for Helen Keller International (HKI)

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. Our leverage and funging adjustments are intended to account for the impact of these spending changes. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

We noticed and corrected a formula error in our leverage and funging adjustment calculations in our CEA for HKI's vitamin A supplementation program:

Change 4: Corrected an error in our estimates of AMF's marginal spending by country

We noticed that the estimates of the proportion of marginal donations that we expect to be allocated to each country in our CEA for AMF did not match the values we calculated in our 2019 room for more funding analysis for AMF. We have corrected the error.6

Change 5: Updated our leverage and funging calculations for AMF

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

Our leverage and funging adjustments for AMF are calculated on a non-public spreadsheet.7 We made the same changes described in Change 2 and Change 4 above in our AMF leverage and funging spreadsheet.

Change 6: Corrected an error in our cost breakdown in our CEA of HKI

We noticed that some of the values in our cost breakdown in our CEA of HKI (the proportion of total costs covered by HKI, other philanthropic actors, and domestic governments) did not match the values calculated in our 2019 cost per supplement analysis for HKI. We have corrected the error.8

Change 7: Corrected errors in our leverage and funging adjustments and country weighting methods

We noticed and corrected a set of errors related to the relative weights countries receive in our overall cost-effectiveness estimates. These errors were caused by mismatches in spending totals between the main section of each CEA and the leverage and funging sections.

Background

Our top charities often work in partnership with other organizations or governments.9 Our top charities' spending may lead other organizations or governments to spend more on programs supported by our top charities than they otherwise would have ("leverage") or less than they otherwise would have ("funging"). Our leverage and funging adjustments are intended to account for the impact of these spending changes. For a full introduction to GiveWell's views on leverage and funging, see this blog post.

Our CEA also includes country-specific cost-effectiveness and overall cost-effectiveness estimates for each top charity.10 Overall cost-effectiveness estimates account for the percentage of marginal donations we expect each top charity to allocate to programs in each country. For AMF in our 2020 CEA — version 1, this means that our cost-effectiveness estimate for the Democratic Republic of the Congo (where we expect AMF to allocate 68% of marginal donations) has more influence on our overall cost-effectiveness estimate than Guinea (where we expect AMF to allocate 7% of marginal donations).

Errors we identified

We noticed mismatches in our 2019 CEA — version 6 between the spending totals reported in the core CEA models and in the leverage and funging sections. For Malaria Consortium, for example, we list the (arbitrary) total spending on the program in the main CEA section at $100,000. But in the leverage and funging section, we state that the overall spending total is ~$110,000. These mismatches lead to incorrect weighting by country in our overall cost-effectiveness estimates in cases where our top charity does not cover the same proportion of total costs in each country (details in footnote).11

These types of errors applied to all of our top charities (excluding GiveDirectly, for which we do not estimate country-specific cost-effectiveness).12

How we corrected the errors

We have addressed these errors with the following changes:

  • We have added a "Total amount spent on the program by all contributors" row to the core model of our CEAs for each top charity (in order to match the spending total listed in the leverage and funging section).13
  • We have edited all calculations of "units of value per $100,000 donation" to "units of value per dollar spent."
    • We needed to make this change because we are no longer assuming that $100,000 is spent in total on each program in the core CEA model for each top charity. We instead assume that our top charity is spending $100,000, and that other actors spend proportionally to their contributions to total costs.14

Change 8: Corrected an error in our cost per net estimates for AMF

We noticed and corrected an error in our cost per net analysis spreadsheet for AMF. This spreadsheet is not public because we do not have permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on. See our review of AMF for more details.

Correcting the error led to slight changes (less than $0.01) in our cost per net estimates for Papua New Guinea and Nigeria.

Change 9: Corrected an error in importing data for our CEA of AMF

Our CEA for AMF relies on data on malaria prevalence and mortality by country from the Institute for Health Metrics and Evaluation's Global Burden of Disease (GBD). We noticed and corrected an error in importing data from GBD for Nigeria in our CEA of AMF.15

Change 10: Reordered adjustment calculations

Our CEAs include a core model for each top charity and three additional types of adjustments:16

  • Downside adjustments: Adjustments to account for the risk of "wastage" (e.g., double treatment with vitamin A supplements or deworming tablets), quality of monitoring and evaluation, and confidence in funds being used for their intended purpose.17
  • Adjustments for excluded effects: Adjustments to account for our best guesses about the impact of intervention-level factors excluded from our core CEA model.18
  • Leverage and funging adjustments: Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

We previously made adjustments for leverage and funging first, followed by downside adjustments and adjustments for excluded effects.19 Following this change, we implement downside adjustments first, followed by adjustments for excluded effects and leverage and funging adjustments.20

We reordered these calculations in order to more easily calculate "cost per life saved" estimates in our CEAs of AMF, HKI, and Malaria Consortium (details in footnote).21

Change 11: Updated development effects estimate for vitamin A supplementation programs

Some health programs for children may positively impact development, leading to lasting increases in their productivity and earnings in adulthood ("development effects").22

Our updated development effects estimate for HKI's vitamin A supplementation programs is based on a comparison of the evidence base for factors that may be correlated with development effects (such as impacts on cognition, child growth, and anemia) for both seasonal malaria chemoprevention (SMC) and vitamin A supplementation. See this document for more details (the method we previously used to estimate development effects for vitamin A supplementation programs is discussed here.)23

Change 12: Updated external validity adjustment for vitamin A supplementation programs

We use an external validity adjustment in our CEA for HKI's vitamin A supplementation program to account for differences between the populations studied in randomized controlled trials (RCTs) of vitamin A supplementation programs and populations reached by HKI's programs today.24 Our 2020 external validity adjustment calculations are in this spreadsheet (our previous version from 2019 is available here.)

Our main updates in 2020 to our external validity adjustment calculations for HKI are as follows:

  • Our 2019 external validity spreadsheet compared cause-specific child mortality rates between populations in 1990 in countries where HKI works and populations in those countries today.25 Our 2020 analysis instead compares cause-specific mortality rates between populations studied in RCTs of vitamin A supplementation included in Imdad et al. 2017, a Cochrane Systematic Review, and targeted populations in countries where HKI works today, which we believe is a more appropriate adjustment.26
  • Our 2019 external validity spreadsheet used cause-specific mortality data for children under five years of age.27 We updated our 2020 external validity spreadsheet to use mortality data for children aged 6 months to 59 months, the target population for vitamin A supplementation programs.28
  • Our 2019 external validity calculations included adjustments for both "changes in VAD prevalence" and "changes in vitamin A-susceptible child mortality."29 In 2020, we added a rough additional adjustment to account for possible non-independence of these two factors.30

Compare the archived version of our CEA preceding this change here to the version following this change here.

Change 13: Updated AMF's marginal spending by country

We updated our estimates of the distribution of AMF's marginal spending across countries.31 Our updates were based on our 2020 room for more funding analysis for AMF.32

Change 14: Updated AMF's cost per net

In 2019 and 2020, we updated our analysis of AMF's cost per net but did not publish an updated version of the spreadsheet with the data we used and our calculations because we have not received permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on.33

In 2019, our overall cost per net estimate for AMF was $4.59. Following updates in 2020, our overall cost per net estimate is $4.95. For updated overall and country-level cost per net estimates, see the version of our CEA following this change here.

As part of our 2020 cost per net update for AMF, we also updated our estimates of the proportion of nets AMF plans to purchase that incorporate piperonyl butoxide (PBO), which impacted the insecticide resistance adjustment in our CEA.34

Change 15: Updated our estimate of equivalent coverage years per net

Our CEA for AMF relies on evidence from randomized controlled trials (RCTs) using conventionally treated nets (CTNs) to calculate the effectiveness of nets at averting child mortality.35 However, the CTNs used in these trials differ in key ways from the long-lasting insecticide-treated nets (LLINs) used today.36

Using field durability monitoring data, we've evaluated the physical survival of today's LLINs and roughly compared it to the expected decay of CTNs in terms of equivalent coverage years.37 "Equivalent coverage years" refers to the average duration for which we estimate an LLIN from an AMF distribution offers protection equivalent to conventionally-treated nets (CTNs) re-treated every 6 mos as used in RCTs, over the 36-month interval between net distributions. Our current best guess is that an LLIN from an AMF distribution confers 2.11 equivalent coverage years.38

Prior to our analysis this year, we estimated the lifespan of nets using the decay model described on this page.

Change 16: Updated malaria mortality and malaria case incidence estimates for AMF

Our assessment of the effectiveness of insecticide-treated net distributions relies heavily on Lengeler 2004, a Cochrane Systematic Review of randomized controlled trials (RCTs), as discussed in our intervention report here.

In 2020, we reviewed Pryce, Richardson, and Lengeler 2018, an updated version of Lengeler 2004.39 Our review of Pryce, Richardson, and Lengeler 2018 led us to update our malaria mortality and malaria case incidence estimates in our CEA of AMF.40

Change 17: Corrected an error in our adjustments for excluded effects for HKI

Our CEAs for each top charity include core models and a set of additional adjustments outside the core models (see above).

We noticed an inconsistency in our adjustments for excluded effects—we included an adjustment for "investment of income increases" for other organizations, but did not include the adjustment for HKI. We have now included this adjustment for HKI as well.41