This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2020. For past versions of our CEA, see this page.

- Version 1 — Published September 11, 2020
- Change 1: Corrected errors in our cost per seasonal malaria chemoprevention (SMC) treatment estimates for Malaria Consortium
- Change 2: Updated data on population distributions by age for our CEA of the Against Malaria Foundation (AMF)
- Change 3: Corrected an error in our leverage and funging adjustments for Helen Keller International (HKI)
- Change 4: Corrected an error in our estimates of AMF's marginal spending by country
- Change 5: Updated our leverage and funging calculations for AMF
- Change 6: Corrected an error in our cost breakdown in our CEA of HKI
- Change 7: Corrected errors in our leverage and funging adjustments and country weighting methods
- Change 8: Corrected an error in our cost per net estimates for AMF
- Change 9: Corrected an error in importing data for our CEA of AMF
- Change 10: Reordered adjustment calculations
- Change 11: Updated development effects estimate for vitamin A supplementation programs
- Change 12: Updated external validity adjustment for vitamin A supplementation programs
- Change 13: Updated AMF's marginal spending by country
- Change 14: Updated AMF's cost per net
- Change 15: Updated our estimate of equivalent coverage years per net
- Change 16: Updated malaria mortality and malaria case incidence estimates for AMF
- Change 17: Corrected an error in our adjustments for excluded effects for HKI

## Version 1 — Published September 11, 2020

Link to the cost-effectiveness analysis (CEA) file: 2020 CEA — Version 1

For the impact of each of the following changes on our cost-effectiveness estimates, see this spreadsheet.

### Change 1: Corrected errors in our cost per seasonal malaria chemoprevention (SMC) treatment estimates for Malaria Consortium

We found and corrected two errors in our 2019 cost per SMC treatment estimate for Malaria Consortium.1 The corrections led to our cost per SMC treatment estimate for Chad changing from $8.77 to $8.17 and our estimate for Nigeria changing from $7.70 to $7.79.2

### Change 2: Updated data on population distributions by age for our CEA of the Against Malaria Foundation (AMF)

Our CEA for AMF relies on estimates of the distribution of populations by age for our calculations of "Mortality reduction in children under 5," "Mortality reduction in individuals 5+ years old," and "Income increases — ages 14 and under."3 We previously used multiple sources for data on population distributions by age.4

For consistency with our CEA of Malaria Consortium's SMC program, we now use data from the Institute for Health Metrics and Evaluation’s Global Burden of Disease Study 2017 Population Estimates for all countries in our CEA for AMF.5

### Change 3: Corrected an error in our leverage and funging adjustments for Helen Keller International (HKI)

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. Our leverage and funging adjustments are intended to account for the impact of these spending changes. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

We noticed and corrected a formula error in our leverage and funging adjustment calculations in our CEA for HKI's vitamin A supplementation program:

- Our CEA for HKI previously used different estimates of the counterfactual value of spending by domestic governments in our overall and country-specific leverage and funging calculations, which was an error. See our 2019 GiveWell Cost-Effectiveness Analysis — Version 6 (public).
- We have corrected the error here.

### Change 4: Corrected an error in our estimates of AMF's marginal spending by country

We noticed that the estimates of the proportion of marginal donations that we expect to be allocated to each country in our CEA for AMF did not match the values we calculated in our 2019 room for more funding analysis for AMF. We have corrected the error.6

### Change 5: Updated our leverage and funging calculations for AMF

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

Our leverage and funging adjustments for AMF are calculated on a non-public spreadsheet.7 We made the same changes described in Change 2 and Change 4 above in our AMF leverage and funging spreadsheet.

### Change 6: Corrected an error in our cost breakdown in our CEA of HKI

We noticed that some of the values in our cost breakdown in our CEA of HKI (the proportion of total costs covered by HKI, other philanthropic actors, and domestic governments) did not match the values calculated in our 2019 cost per supplement analysis for HKI. We have corrected the error.8

### Change 7: Corrected errors in our leverage and funging adjustments and country weighting methods

We noticed and corrected a set of errors related to the relative weights countries receive in our overall cost-effectiveness estimates. These errors were caused by mismatches in spending totals between the main section of each CEA and the leverage and funging sections.

#### Background

Our top charities often work in partnership with other organizations or governments.9 Our top charities' spending may lead other organizations or governments to spend more on programs supported by our top charities than they otherwise would have ("leverage") or less than they otherwise would have ("funging"). Our leverage and funging adjustments are intended to account for the impact of these spending changes. For a full introduction to GiveWell's views on leverage and funging, see this blog post.

Our CEA also includes country-specific cost-effectiveness and overall cost-effectiveness estimates for each top charity.10 Overall cost-effectiveness estimates account for the percentage of marginal donations we expect each top charity to allocate to programs in each country. For AMF in our 2020 CEA — version 1, this means that our cost-effectiveness estimate for the Democratic Republic of the Congo (where we expect AMF to allocate 68% of marginal donations) has more influence on our overall cost-effectiveness estimate than Guinea (where we expect AMF to allocate 7% of marginal donations).

#### Errors we identified

We noticed mismatches in our 2019 CEA — version 6 between the spending totals reported in the core CEA models and in the leverage and funging sections. For Malaria Consortium, for example, we list the (arbitrary) total spending on the program in the main CEA section at $100,000. But in the leverage and funging section, we state that the overall spending total is ~$110,000. These mismatches lead to incorrect weighting by country in our overall cost-effectiveness estimates in cases where our top charity does not cover the same proportion of total costs in each country (details in footnote).11

These types of errors applied to all of our top charities (excluding GiveDirectly, for which we do not estimate country-specific cost-effectiveness).12

#### How we corrected the errors

We have addressed these errors with the following changes:

- We have added a "Total amount spent on the program by all contributors" row to the core model of our CEAs for each top charity (in order to match the spending total listed in the leverage and funging section).13
- We have edited all calculations of "units of value per $100,000 donation" to "units of value per dollar spent."
- We needed to make this change because we are no longer assuming that $100,000 is spent in total on each program in the core CEA model for each top charity. We instead assume that our top charity is spending $100,000, and that other actors spend proportionally to their contributions to total costs.14

### Change 8: Corrected an error in our cost per net estimates for AMF

We noticed and corrected an error in our cost per net analysis spreadsheet for AMF. This spreadsheet is not public because we do not have permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on. See our review of AMF for more details.

Correcting the error led to slight changes (less than $0.01) in our cost per net estimates for Papua New Guinea and Nigeria.

### Change 9: Corrected an error in importing data for our CEA of AMF

Our CEA for AMF relies on data on malaria prevalence and mortality by country from the Institute for Health Metrics and Evaluation's Global Burden of Disease (GBD). We noticed and corrected an error in importing data from GBD for Nigeria in our CEA of AMF.15

### Change 10: Reordered adjustment calculations

Our CEAs include a core model for each top charity and three additional types of adjustments:16

**Downside adjustments:**Adjustments to account for the risk of "wastage" (e.g., double treatment with vitamin A supplements or deworming tablets), quality of monitoring and evaluation, and confidence in funds being used for their intended purpose.17**Adjustments for excluded effects:**Adjustments to account for our best guesses about the impact of intervention-level factors excluded from our core CEA model.18**Leverage and funging adjustments:**Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

We previously made adjustments for leverage and funging first, followed by downside adjustments and adjustments for excluded effects.19 Following this change, we implement downside adjustments first, followed by adjustments for excluded effects and leverage and funging adjustments.20

We reordered these calculations in order to more easily calculate "cost per life saved" estimates in our CEAs of AMF, HKI, and Malaria Consortium (details in footnote).21

### Change 11: Updated development effects estimate for vitamin A supplementation programs

Some health programs for children may positively impact development, leading to lasting increases in their productivity and earnings in adulthood ("development effects").22

Our updated development effects estimate for HKI's vitamin A supplementation programs is based on a comparison of the evidence base for factors that may be correlated with development effects (such as impacts on cognition, child growth, and anemia) for both seasonal malaria chemoprevention (SMC) and vitamin A supplementation. See this document for more details (the method we previously used to estimate development effects for vitamin A supplementation programs is discussed here.)23

### Change 12: Updated external validity adjustment for vitamin A supplementation programs

We use an external validity adjustment in our CEA for HKI's vitamin A supplementation program to account for differences between the populations studied in randomized controlled trials (RCTs) of vitamin A supplementation programs and populations reached by HKI's programs today.24 Our 2020 external validity adjustment calculations are in this spreadsheet (our previous version from 2019 is available here.)

Our main updates in 2020 to our external validity adjustment calculations for HKI are as follows:

- Our 2019 external validity spreadsheet compared cause-specific child mortality rates between populations in 1990 in countries where HKI works and populations in those countries today.25 Our 2020 analysis instead compares cause-specific mortality rates between populations studied in RCTs of vitamin A supplementation included in Imdad et al. 2017, a Cochrane Systematic Review, and targeted populations in countries where HKI works today, which we believe is a more appropriate adjustment.26
- Our 2019 external validity spreadsheet used cause-specific mortality data for children under five years of age.27 We updated our 2020 external validity spreadsheet to use mortality data for children aged 6 months to 59 months, the target population for vitamin A supplementation programs.28
- Our 2019 external validity calculations included adjustments for both "changes in VAD prevalence" and "changes in vitamin A-susceptible child mortality."29 In 2020, we added a rough additional adjustment to account for possible non-independence of these two factors.30

Compare the archived version of our CEA preceding this change here to the version following this change here.

### Change 13: Updated AMF's marginal spending by country

We updated our estimates of the distribution of AMF's marginal spending across countries.31 Our updates were based on our 2020 room for more funding analysis for AMF.32

### Change 14: Updated AMF's cost per net

In 2019 and 2020, we updated our analysis of AMF's cost per net but did not publish an updated version of the spreadsheet with the data we used and our calculations because we have not received permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on.33

In 2019, our overall cost per net estimate for AMF was $4.59. Following updates in 2020, our overall cost per net estimate is $4.95. For updated overall and country-level cost per net estimates, see the version of our CEA following this change here.

As part of our 2020 cost per net update for AMF, we also updated our estimates of the proportion of nets AMF plans to purchase that incorporate piperonyl butoxide (PBO), which impacted the insecticide resistance adjustment in our CEA.34

### Change 15: Updated our estimate of equivalent coverage years per net

Our CEA for AMF relies on evidence from randomized controlled trials (RCTs) using conventionally treated nets (CTNs) to calculate the effectiveness of nets at averting child mortality.35 However, the CTNs used in these trials differ in key ways from the long-lasting insecticide-treated nets (LLINs) used today.36

Using field durability monitoring data, we've evaluated the physical survival of today's LLINs and roughly compared it to the expected decay of CTNs in terms of equivalent coverage years.37 "Equivalent coverage years" refers to the average duration for which we estimate an LLIN from an AMF distribution offers protection equivalent to conventionally-treated nets (CTNs) re-treated every 6 mos as used in RCTs, over the 36-month interval between net distributions. Our current best guess is that an LLIN from an AMF distribution confers 2.11 equivalent coverage years.38

Prior to our analysis this year, we estimated the lifespan of nets using the decay model described on this page.

### Change 16: Updated malaria mortality and malaria case incidence estimates for AMF

Our assessment of the effectiveness of insecticide-treated net distributions relies heavily on Lengeler 2004, a Cochrane Systematic Review of randomized controlled trials (RCTs), as discussed in our intervention report here.

In 2020, we reviewed Pryce, Richardson, and Lengeler 2018, an updated version of Lengeler 2004.39 Our review of Pryce, Richardson, and Lengeler 2018 led us to update our malaria mortality and malaria case incidence estimates in our CEA of AMF.40

### Change 17: Corrected an error in our adjustments for excluded effects for HKI

Our CEAs for each top charity include core models and a set of additional adjustments outside the core models (see above).

We noticed an inconsistency in our adjustments for excluded effects—we included an adjustment for "investment of income increases" for other organizations, but did not include the adjustment for HKI. We have now included this adjustment for HKI as well.41

- 1.
- One error was in the cell references in the formula used in this cell.
- The second error was in the cell references in the formula used in these cells.

- 2.
Compare our 2019 GiveWell Cost-Effectiveness Analysis — Version 6 (public), "SMC" sheet, Cost per equivalent child treated with all 4 cycles of SMC for Chad and Nigeria,

to 2020 GiveWell cost-effectiveness analysis — version 0 [change 1 archive], "Malaria Consortium" sheet, Cost per equivalent child treated with all 4 cycles of SMC for Chad and Nigeria. - 3.
See 2020 GiveWell cost-effectiveness analysis — version 0 [change 2 archive], "AMF" sheet.

- 4.
- We previously used data from Demographic and Health Surveys (DHS), Malaria Indicator Surveys (MIS) and the Institute for Health Metrics and Evaluation’s Global Burden of Disease.
- See 2019 GiveWell Cost-Effectiveness Analysis — Version 6 (public), "Bednets" sheet, "Age distribution of net coverage" section.

- 5.
See 2020 GiveWell cost-effectiveness analysis — version 0 [change 2 archive], "AMF" sheet, "Age distribution of net coverage" section and "Malaria Consortium" sheet, "Income increases — ages 14 and under" section.

- 6.
- 7.
We have calculated our leverage and funging adjustments for AMF on a non-public spreadsheet because we do not have permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis, and Malaria, which our calculations rely on. See our review of AMF for more details.

- 8.
Compare the archived version of our CEA preceding this change here to the version following this change here.

- 9.
For example, see our review of AMF for a discussion of the roles played by AMF and its partners in net distributions.

- 10.
See the sheets for each top charity in 2020 GiveWell cost-effectiveness analysis — version 1. The drop-down settings on the "Results" sheet can also be used to toggle between overall and country-specific cost-effectiveness estimates. To use drop-down settings, make a copy of the CEA linked above using "File," then "Make a copy."

- 11.
Using Malaria Consortium as an example:

- The calculations in the core CEA model for Malaria Consortium use our total cost per child treated estimates (including costs covered by actors other than Malaria Consortium). See our CEA preceding this change here.
- The CEA also uses our expectations of Malaria Consortium's spending by country.
- Because Malaria Consortium does not cover the same proportion of total costs in each country, using our expectations of Malaria Consortium's spending by country and total cost per treatment estimates led to errors in our calculations of the number of children expected to be treated by country. Compare the version of the CEA preceding this change here to the version following this change here.

- 12.
See 2020 GiveWell cost-effectiveness analysis — version 1, "GiveDirectly" sheet.

- 13.
Compare the top charity sheets in the version of our CEA preceding this change to the version of our CEA following this change. For Malaria Consortium, for example, this row was added.

- 14.
Compare the top charity sheets in the version of our CEA preceding this change to the version of our CEA following this change.

- 15.
Compare the archived version of our CEA preceding this change here to the version following this change here.

- 16.
Our 2020 GiveWell cost-effectiveness analysis — version 1 reports the following cost-effectiveness results for each top charity, found here (to toggle among these choices, make a copy of the CEA file using "File," then "Make a copy"):

- Core model only
- Include downside adjustments
- Include downside adjustments and adjustments for excluded effects
- Include leverage/funging and all other adjustments

- 17.
See here for an example of how we apply these adjustments in our CEA for Malaria Consortium.

- 18.
- 19.
See the archived copy of the CEA preceding this change here.

- 20.
See the archived copy of the CEA following this change, here.

- 21.
- Our cost-effectiveness analyses of AMF, HKI, and Malaria Consortium include multiple types of benefits, including deaths averted and long-term income increases for some program participants. See our "AMF," "HKI," and "Malaria Consortium" sheets in 2020 GiveWell cost-effectiveness analysis — version 1 for more details.
- We sometimes want to report "cost per life saved" estimates for these organizations, which exclude other types of benefits of their programs.
- We have found these calculations simpler to implement when leverage and funging adjustments are applied after downside adjustments and adjustments for excluded effects. That is because our cost per life saved estimates use all downside adjustments in our model, but only account for selected adjustments for excluded effects and adjustments for funging (but not leverage). For more details, see the calculations and cell notes for Malaria Consortium here. The same calculations are implemented at the bottom of our "HKI" and "AMF" sheets.

- 22.
For example, we discuss potential development impacts of mass deworming programs in this section of our intervention report.

- 23.
- See this section of our cost-effectiveness analysis of HKI for our updated development effects estimate.
- For comparison, see the archived version of our cost-effectiveness analysis preceding this change here.

- 24.
See our review of HKI and our intervention report on vitamin A supplementation for further discussion on RCTs of vitamin A supplementation.

- 25.
See our 2019 HKI VAS - information for country-level cost-effectiveness spreadsheet, "External validity by country" sheet.

- 26.
See our 2020 HKI VAS - information for country-level cost-effectiveness spreadsheet, "External validity by country" sheet.

- 27.
See our 2019 HKI VAS - information for country-level cost-effectiveness spreadsheet, "External validity by country" sheet.

- 28.
- See our 2020 HKI VAS - information for country-level cost-effectiveness spreadsheet, "External validity by country" sheet.
- See our review of HKI for information on the ages of children targeted for VAS programs.

- 29.
See our 2019 HKI VAS - information for country-level cost-effectiveness spreadsheet, "External validity by country" sheet, this row.

- 30.
See our 2020 external validity spreadsheet, "External validity by country" sheet, this row.

- 31.
Compare the archived version of the CEA following this change here to the version preceding this change here.

- 32.
As of the date of the publication of this changelog, our 2020 room for more funding analysis for AMF is not yet published. See our most up-to-date version of our AMF room for more funding analysis in this section of our review of AMF.

- 33.
See this section of our review of AMF for additional information.

- 34.
- We discuss insecticide resistance in depth in this report. PBO nets are discussed in this section.
- Our updated insecticide resistance adjustment following this change is available here, and our adjustment preceding the change is available here.
- We do not have permission from AMF to share the specific proportions of PBO nets it plans to purchase by country.

- 35.
- See our intervention report on mass distribution of long-lasting insecticide-treated nets (LLINs) for discussion of these RCTs.
- We incorporate our estimate of the mortality reduction effect of net distributions, based on a meta-analysis of RCTs, in our CEA of AMF here.

- 36.
See the "What is LLIN distribution and how does it target malaria?" and "What is the evidence regarding the general effectiveness of LLIN distributions?" sections of our intervention report on mass distribution of long-lasting insecticide-treated nets (LLINs) for more details.

- 37.
- Our analysis is available in this spreadsheet.
- As of the date of the publication of this changelog, we have not published a full report on our equivalent coverage years analysis.

- 38.
- For all countries except Democratic Republic of the Congo, the value of 2.11 comes from this calculation. In our review of AMF, we note that "the data we have seen from AMF's distributions in DRC provide some evidence that LLINs decayed considerably more quickly than expected." We roughly adjust our LLIN lifespan estimate for DRC downwards by 10% to account for this difference.

- 39.
See the "History" section of Pryce, Richardson, and Lengeler 2018, Pg 65.

- 40.
Compare the "AMF" sheet in the archived version of the CEA preceding this change here to the highlighted cells in the version of the CEA following this change here.

- 41.
See the "Inclusion/exclusion" sheet on the version of the CEA preceding this change here compared to the version following this change here.