You are here

2021 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2021. For past versions of our CEA, see this page.

Table of Contents

Version 1 — Published May 5, 2021

Link to the cost-effectiveness analysis (CEA) file: 2021 CEA — version 1

See this spreadsheet for the impact of each of the changes below on our cost-effectiveness estimates.

Change 1: Updated Global Burden of Disease (GBD) estimates in our CEA of the Against Malaria Foundation (AMF)

Our CEA of AMF relies on country-level estimates of malaria prevalence, child mortality rates, and population distributions by age from the Institute of Health Metrics and Evaluation (IHME)'s Global Burden of Disease (GBD) project.1

We previously relied on the 2017 version of the project (GBD 2017), but in October 2020, IHME released an updated version, GBD 2019.2 We have incorporated updated GBD 2019 estimates into our CEA of AMF.3

Change 2: Updated GBD estimates in our CEA of Malaria Consortium

We also incorporated updated GBD 2019 estimates for malaria prevalence, child mortality rates, and population distributions by age into our CEA of Malaria Consortium (see change 1 above).4

Change 3: Corrected an error in our leverage and funging adjustment for AMF

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

Our leverage and funging adjustments for AMF are calculated on a non-public spreadsheet. We have not made this spreadsheet publicly available because we have not received permission to publish country-specific cost estimates we received from the Global Fund to Fight AIDS, Tuberculosis and Malaria, which our calculations rely on.5

We noticed that the discount rate on the separate AMF leverage and funging spreadsheet had not been updated along with the discount rate in the main CEA in 2020 (see our 2020 CEA changelog here). We have corrected the error.6

Change 4: Updated percentage of program costs covered by different organizations for our CEA of Helen Keller International in Cameroon

In December 2020, Helen Keller International provided us with updated information on the proportions of costs covered by different organizations for a proposed vitamin A supplementation (VAS) program in Cameroon.7 We have incorporated this updated information into our CEA of Helen Keller International.8

Change 5: Updated Helen Keller International CEA with GBD 2019 mortality rate estimates

Our CEA of Helen Keller International also relies on GBD data (see change 1 and change 2 above). We have updated the mortality rate estimate inputs in our Helen Keller International CEA from GBD 2017 to GBD 2019.9

Our CEA of Helen Keller International also relies on estimates of the prevalence of vitamin A deficiency (VAD) from GBD, but we have not yet updated these estimates from GBD 2017 to GBD 2019.10 We have not yet updated the VAD prevalence estimates because we do not have a clear understanding of the changes in IHME's modeling assumptions and data sources for VAD between GBD 2017 and GBD 2019—we plan to seek more information from IHME this year about our remaining questions.11

Change 6: Added our CEA of New Incentives

In November 2020, we added New Incentives to our list of top charities.

Our original CEA of New Incentives is here. We have adapted the format of the New Incentives CEA and added it to the workbook with our other top charity CEAs (see the version of the CEA following this change here).12

Change 7: Updated AMF cost per net and insecticide resistance adjustment for the Democratic Republic of the Congo

Insecticide resistance is a threat to the effectiveness of long-lasting insecticide-treated nets (LLINs) in preventing malaria.13 We believe that there is some evidence that a newer type of LLIN incorporating piperonyl butoxide ("PBO nets") provide additional protection in areas where mosquitoes have developed insecticide resistance.14

In January 2021, AMF shared updated information with us on the proportion of PBO nets it planned to purchase for upcoming LLIN distributions in the Democratic Republic of the Congo (DRC).15 We have updated our CEA of AMF to account for the greater cost of PBO nets and their expected effectiveness in DRC.16

Change 8: Updated additional external validity adjustment for Helen Keller International in Kenya

For our 2020 CEA — version 2, we added an additional external validity adjustment for Helen Keller International in Kenya to account for relatively low baseline mortality rates and a micronutrient survey finding low prevalence of vitamin A deficiency (see our 2020 changelog here).

After further investigation in 2021, we increased the size of our downward adjustment to Helen Keller International's expected cost-effectiveness in Kenya. We describe our reasoning for our updated adjustment in this document.17

Change 9: Updated equivalent coverage years estimate for LLINs in DRC

We expect that LLINs have limited lifespans—over time, the insecticide on the nets may wear off or the nets may acquire holes.18 In countries other than DRC, we estimate that AMF's LLINs provide 2.11 years of effective protection during each three-year distribution cycle (relative to the effectiveness of the bed nets used in the randomized controlled trials we rely on for our estimate of the impact of bed nets on child mortality).19

We have previously applied a 10% downwards adjustment (to 1.90 equivalent coverage years) for AMF's bed net distributions in DRC, relative to its distributions in other countries.20 That adjustment was based on our rough impression that post-distribution monitoring (PDM) results we had seen from AMF's distributions in DRC indicated shorter LLIN lifespans than data we had seen from other countries.21

After further investigation, we have updated our estimate of the durability of LLINs in DRC. Based on PDM results for AMF distribution in Equateur province, we now expect that nets in DRC have around 17% lower durability (1.74 equivalent coverage years) than nets AMF distributes in other countries—see our full updated analysis in this spreadsheet.22

Change 10: Split large funding gaps in DRC for AMF and in Nigeria for Malaria Consortium into separate cost-effectiveness estimates

Our CEAs generally provide country-level cost-effectiveness estimates.23 With this change, we have split up our cost-effectiveness modeling for AMF in DRC into two groups (the first ~$20 million of AMF's funding gap and the last ~$20 million) and we have split our model of Malaria Consortium in Nigeria into two groups (the first ~$15 million of the funding gap and the last ~$10 million).24 We have split our modeling of these large funding gaps in order to vary some of the inputs of our CEA between filling the first and last parts of each gap. These inputs are varied in change 11 and change 16 below.

Change 11: Edited adjustments for effects excluded from our core model for AMF

For each of our top charities, we use a core cost-effectiveness model along with a set of additional adjustments.25 We use one of these types of adjustments, "adjustments for effects excluded from our core model," to account for our best guesses of the impact of intervention-level factors we have not analyzed in depth.26

For AMF, we have added a new adjustment for an effect excluded from our core CEA ("Adjustment for program impact being to move distributions closer together") and updated our estimates for a previous adjustment ("Marginal funding goes to lower priority areas").

Adjustment for program impact being to move distributions closer together

Our CEA for AMF is structured to model the impact of LLIN distributions occurring at three-year intervals.27 Our underlying assumption for most countries where AMF works is that, without funding from AMF, distributions would still happen on three-year intervals but fewer LLINs would be distributed, leading to lower coverage levels.

Our understanding is that in DRC and Nigeria, additional funding for LLIN distributions may lead to reductions in the amount of time between distributions (e.g., moving from a four-year spacing to the recommended three-year spacing), rather than increasing coverage for campaigns that occur every three years.28 Without an additional adjustment, we expect that our CEA would overestimate cost-effectiveness of funding used to reduce the amount of time between distributions.29

To address this issue, we have added an "adjustment for program impact being to move distributions closer together" for AMF in DRC and Nigeria to our list of adjustments for effects excluded from our core model.30 The values for this adjustment are calculated in this spreadsheet.

For Nigeria, we estimate that it is 35% less cost-effective to shift bed net distributions from occurring every 3.75 years to every three years, compared to the cost-effectiveness of moving from no bed net distributions to distributions every three years.31 This calculation is based on estimating how many incremental bed net coverage years are gained by shifting from distributions every 3.75 years to distributions every three years, relying on assumptions on how quickly bed nets in Nigeria wear out after distribution.32

Using similar calculation methods for DRC, we estimate that it is 41% less cost-effective to decrease the amount of time between distributions than it would be to move from no bed net distributions to distributions every three years.33

Adjustment for marginal funding going to lower priority areas

One of the adjustments for effects excluded from our core AMF CEA is an adjustment for marginal funding for LLIN distributions being allocated to lower priority areas.34 If there is not enough funding available to fully reach entire populations with LLIN distributions, our guess is that some countries will make prioritization decisions to exclude some lower priority areas (e.g., areas with lower malaria prevalence), and that providing additional funding for LLIN distributions in effect allows these areas to be covered.35

As discussed above, our understanding is that in Nigeria and DRC governments respond to limited funding availability by increasing the spacing between LLIN distributions. Because the governments of Nigeria and DRC are using spacing between LLIN distributions to triage resources for malaria programs, our guess is that these countries implement less prioritization by geographic areas. We have accordingly reduced our adjustment for the impact of marginal funding going to lower priority areas in Nigeria and DRC.36

Change 12: Edited adjustments for effects excluded from our core model for Malaria Consortium

We also use an adjustment for marginal funding going to lower priority areas for Malaria Consortium (see our discussion of this adjustment for AMF above).37 We previously used a single estimate for this adjustment across all of the countries in which Malaria Consortium operates SMC programs, but we have now created country-specific adjustments.38 We describe our reasoning for each of our country-specific adjustments in our Malaria Consortium CEA spreadsheet.39

Change 13: Removed "Results" sheet

Our past CEAs have included a "Results" sheet, displaying country-specific and overall cost-effectiveness estimates for each charity.40 To simplify our presentation of cost-effectiveness estimates, we have removed the "Results" sheet from this version of the CEA—cost-effectiveness results remain available in each top charity's sheet and on the "Cost-effectiveness by funding gap" sheet.41

Change 14: Added Nigeria to our CEA of AMF

In early 2021, AMF told us that it could use additional funding to support LLIN distributions in a set of states in Nigeria where the President's Malaria Initiative (PMI) works.42 AMF expected that it would use additional funding to support LLIN distributions in Akwa Ibom and Bauchi State in 2021, and that there would be additional funding gaps for LLIN distributions in Nigeria in 2023.43

We have added a column to our CEA of AMF with cost-effectiveness estimates for Nigeria.44

Change 15: Updated pre-existing nets adjustment for AMF

We include an adjustment in our CEA of AMF to account for our expectation that some proportion of individuals receiving LLINs from AMF distributions already have functional LLINs.45 Our previous adjustment was based on rough guesses on the percentage of individuals with pre-existing nets and the proportion of "extra" nets that are eventually used.46

We have completed additional analysis in this spreadsheet to update our pre-existing nets adjustment.47 Our updated analysis includes separate adjustments for distributions on a three-year cycle, distributions moved from a four-year cycle to a three-year cycle with additional funding, and DRC.48

Change 16: Updated leverage and funging adjustments for AMF and Malaria Consortium

We estimate leverage and funging adjustments for our CEAs, as discussed above.

We updated our estimates of the probability that AMF's support of LLIN distributions in DRC crowds out funding from the Global Fund or PMI (see our calculations in this spreadsheet).49 We also updated our estimates of the probability that Malaria Consortium's support of SMC programs in Nigeria crowds out funding from the Global Fund or PMI (see our estimates in this spreadsheet).50

Change 17: Added adjustment for seasonal malaria chemoprevention (SMC) distributions with five cycles

Malaria Consortium's SMC programs typically deliver four "cycles" of SMC per year. Each cycle includes a four-day distribution period and lasts 28 days, after which a new cycle starts.51 In late 2020, we learned that Malaria Consortium planned to support a fifth SMC cycle per year going forward in some countries.52

Our best guess is that marginal spending on a fifth SMC cycle is less cost-effective than spending on the previous four cycles. We added an adjustment to our list of excluded effects to account for our best guess at the size of this effect.53

Change 18: January and February 2021 funding allocation

In January and February 2021, we granted or recommended grants of $71.8 million to our top charities. Our cost-effectiveness estimates for these grants are available in the "Cost-effectiveness by funding gap" sheet here.

Change 19: Remove "overall" cost-effectiveness estimates for AMF, Malaria Consortium, Helen Keller International, Sightsavers, END Fund, and Deworm the World

Our cost-effectiveness analyses for our top charities that operate in multiple countries have previously included country-specific cost-effectiveness estimates and overall cost-effectiveness estimates.54 These overall cost-effectiveness estimates are weighted averages calculated based on how much marginal funding we expected each top charity to spend by country.55

We have decided to remove these estimates for AMF, Deworm the World, END Fund, Sightsavers, Malaria Consortium, and Helen Keller International because they add additional complexity to our analyses and because we do not expect to use them in our funding allocation decisions—instead, we expect to focus our allocation decisions for these organizations on specific country-level funding gaps.56 We have not removed our overall cost-effectiveness estimate for SCI Foundation, because we expect to continue to use our overall cost-effectiveness estimate for SCI Foundation for allocation decisions going forward.57

In order to remove the overall cost-effectiveness estimates discussed above, we have also made the following additional changes:

  • Our leverage and funging adjustments (discussed above) rely on estimates of the cost-effectiveness of what governments and other organizations would spend funding on if they were not contributing to programs implemented by our top charities.58 Our estimate of the "counterfactual value of donated deworming drug costs" was previously calculated as being 10% of AMF's overall cost-effectiveness59 and would fluctuate as our estimate of AMF's overall cost-effectiveness increased or decreased. We now anchor our cost-effectiveness estimate for the counterfactual value of donated deworming drug costs at a fixed value equal to 10% of AMF's overall cost-effectiveness, as estimated in the previous published version of our CEA, 2020 GiveWell cost-effectiveness analysis — version 2.60
  • Some health programs for children may positively impact development, leading to lasting increases in their productivity and earnings in adulthood ("development effects").61 Our estimates of the development effects of Helen Keller International's and New Incentives' programs were previously based on our overall development effects estimate for Malaria Consortium's SMC programs62 and would fluctuate as this parameter increased or decreased. We now instead anchor our estimates of the development effects of Helen Keller International's and New Incentives' programs at a fixed value equal to our overall development effects estimate for Malaria Consortium's SMC programs, as estimated in the previous published version of our CEA, 2020 GiveWell cost-effectiveness analysis — version 2.63

Change 20: Update to "Cost-effectiveness by funding gap" sheet

We have updated the "Cost-effectiveness by funding gap" sheet to reflect our understanding of funding gaps remaining after taking our January and February 2021 funding allocation into account.64 We have also updated our notes on our prioritization of each remaining funding gap.65

  • 1.

    See highlighted sections of the version of the CEA following this change here.

  • 2.
    • "Published in The Lancet in October 2020, GBD 2019 provides for the first time an independent estimation of population, for each of 204 countries and territories and the globe, using a standardized, replicable approach, as well as a comprehensive update on fertility and migration. GBD 2019 incorporates major data additions and improvements, and methodological refinements." GBD 2019 Resources page
    • Compare the version of the CEA preceding this change here to the version following this change here.

  • 3.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 4.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 5.

    See this section of our review of AMF for additional information.

  • 6.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 7.

    See details in this spreadsheet.

  • 8.

    These changes did not have an impact on our cost-effectiveness estimates for Helen Keller International in Cameroon. Compare the version of the CEA preceding this change here to the version following this change here.

  • 9.
    • Compare the version of the CEA preceding this change here to the version following this change here.
    • There are changes to two rows in our Helen Keller International CEA: row 33 ("Baseline deaths per 1,000 child-years for children aged 6- to 59-months") and row 37 ("External validity adjustment — VAS"). See links in bullet point above.
    • The "External validity adjustment — VAS" row changed because mortality rate estimates are an input into our external validity calculations for Helen Keller International. Compare the updated version of our external validity spreadsheet preceding this change here to the version following this change here.

  • 10.
    • Compare the updated version of our external validity spreadsheet preceding this change here to the version following this change here. In both versions, we use GBD 2017 data on vitamin A deficiency prevalence (see "External validity by country" sheet).
    • Compare the external validity row of the CEA preceding this change here to the version following this change here.

  • 11.

  • 12.

    There is a small difference between the cost-effectiveness estimate in our original CEA (10.8 times as cost-effective as cash transfers) and the adapted version (10.7 times as cost-effective as cash transfers). This difference is caused by a change in our method for calculating the "Adjustment for all-cause mortality effect" between the two versions. In the original CEA we manually adjusted the all-cause mortality adjustment factor in order to find the adjustment that leads to an estimate of 0.5 deaths indirectly averted for each death directly averted (see this sheet). In the adapted version, we used a goal-seek function rather than making manual adjustments, which resulted in a slightly different adjustment factor (see here).

  • 13.

    See our full report on insecticide resistance and malaria control here.

  • 14.

    See this section of our report on insecticide resistance and malaria control for our review of the evidence.

  • 15.

    Rob Mather, AMF CEO, email to GiveWell January 12, 2021 (unpublished)

  • 16.
    • AMF has told us that the PBO nets it purchases are more expensive than standard LLINs. Rob Mather, AMF CEO, email to GiveWell January 12, 2021 (unpublished)
    • Compare the version of the CEA preceding this change here to the version following this change here.

  • 17.
    • Our reasoning for the previous version of this adjustment is available in this document.
    • Compare the version of the CEA preceding this change here to the version following this change here.

  • 18.

    For a full discussion, see our Estimating Equivalent Coverage Years for Long-Lasting Insecticide-Treated Nets (LLINs) report.

  • 19.

    See this section of our report on estimating effective coverage years for long-lasting insecticide-treated nets.

  • 20.

    See the version of the CEA preceding this change here.

  • 21.

    See this section of our 2019 review of AMF.

  • 22.

  • 23.

    See any of the top charity sheets in the version of the CEA preceding this change here.

  • 24.

    Compare the AMF and Malaria Consortium sheets in the version of the CEA preceding this change here to the version following this change here.

  • 25.

    See this 2020 changelog entry for a description of all three types of additional adjustments.

  • 26.
    • See this 2020 changelog entry for a description of all three types of additional adjustments
    • See the "Adjustments for excluded effects" sheet in the version of the CEA following this change here.

  • 27.

    See this row in the version of the CEA preceding this change.

  • 28.
    • "The benefits of being able to commit now to funding nets that will be distributed in DRC in 2022 and 2023 are significant. First, it allows our co-funding partner, the Global Fund, not to scale back its allocation of funding in the first two years of the three year funding cycle. Without confirmation of our funding, reprogramming would be necessary, including scaling back of interventions and delayed distributions. This would be required to spread funds evenly and avoid a significant lack of funds for the final year of the funding cycle. Any resulting delays to distributions mean that households have to wait longer than three years to receive new nets. In the third year, nets wear out and so delays extend the period during which households are less well protected which can lead to malaria resurging." AMF agrees to fund a further 16.4 million nets for distribution in DRC in 2023
    • AMF told us in February 2021 that it expected that the impact of additional funding for LLIN distributions in Nigeria would be to reduce the amount of time between distributions from four years to three years. See this spreadsheet.

  • 29.
    • Our CEA for AMF includes an estimate of the number of effective coverage years an LLIN provides (2.11 years for most countries and 1.74 years in DRC, relative to the coverage years of nets used in randomized controlled trials of bed nets). See here in the version of the CEA preceding this change.
    • This is an estimate of the average LLIN lifespan, but we expect that some LLINs would last for longer periods. In our spreadsheet modeling equivalent coverage years here, we estimate that the relative net coverage years in year following a distribution has decayed due to physical damage and insecticide loss, but it has not been driven down to zero. (We have not explicitly modeled the equivalent coverage years we would expect from remaining nets in the fourth year following a distribution.) See our report on estimating equivalent coverage years for LLINs for full details on our analysis.
    • Due to the remaining effectiveness of some LLINs after three years after a distribution, we expect that funding reducing the amount of time between distributions from four years to three years would have less of an impact on LLIN coverage (and be less cost-effective) in a particular area than funding that increased the number of LLINs distributed during three-year distribution cycles.

  • 30.
    • See "Adjustment for effects excluded from our core model" section of our CEA of AMF in the version of the CEA following this change here.
    • We incorporate this adjustment for our CEA of AMF in Nigeria when we add it to our model in change 14.

  • 31.

    See our calculation here

  • 32.

    See our incremental cost per net coverage year calculations here and our net decay assumptions for Nigeria here.

  • 33.

    See our DRC calculations on this sheet and our DRC net decay assumptions here.

  • 34.

    See here in the version of the CEA preceding this change.

  • 35.
    • See here in the version of the CEA preceding this change.
    • For further discussion of sub-national prioritization of malaria interventions, see this blog post.

  • 36.
    • Compare the version of the CEA preceding this change here to the version following this change here
    • We incorporate this adjustment for our CEA of AMF in Nigeria when we add it to our model in change 14.

  • 37.

    See here in the version of the CEA following this change.

  • 38.

    Our previous estimate was a 3% reduction in cost-effectiveness and our updated country-specific estimates range from 5% to 15%. Compare the version of the CEA preceding this change here to the version following this change here.

  • 39.

    See the version of our CEA following this change here.

  • 40.

    See the version of the CEA preceding this change here.

  • 41.

    See the version of our CEA following this change here.

  • 42.

    See this spreadsheet.

  • 43.

    See this spreadsheet.

  • 44.

    See the version of the CEA following this change here.

  • 45.

    See the version of the CEA preceding this change here.

  • 46.

    See the version of the CEA preceding this change here.

  • 47.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 48.

    See this cell for our three-year cycle adjustment, this cell for our adjustment moving distributions from a four-year cycle to a three-year cycle, and this cell for our DRC adjustment.

  • 49.
    • Compare the version of the CEA preceding this change here to the version following this change here.
    • Note that, as mentioned above, we are not able to publish our full leverage and funging adjustments for AMF.

  • 50.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 51.

    See this section of our review of Malaria Consortium for more details.

  • 52.

    See cells on Burkina Faso 2021, Chad 2022, and Nigeria 2022 in the "Assumptions" table in our 2020 room for more funding analysis for Malaria Consortium, "Source: 2020-2023 projections" sheet here.

  • 53.
    • Compare the "Adjustments for effects excluded from our core model" section for Malaria Consortium in the version of the CEA preceding this change here to the version following this change here.
    • See our calculations for this adjustment in this spreadsheet.

  • 54.

    See the AMF, Deworm the World, END Fund, SCI Foundation, Sightsavers, Malaria Consortium, and Helen Keller International sheets in the version of the CEA preceding this change here.

  • 55.

    See the AMF, Deworm the World, END Fund, SCI Foundation, Sightsavers, Malaria Consortium, and Helen Keller International sheets in the version of the CEA preceding this change here, "Percentage of funding to be allocated to each country with marginal donations" row.

  • 56.

    See cost-effectiveness by country-level funding gaps in the version of the CEA following this change here. Also see that overall cost-effectiveness estimates have been removed from the sheets for AMF, Deworm the World, END Fund, Sightsavers, Malaria Consortium, and Helen Keller International.

  • 57.

    See our cost-effectiveness estimate for SCI Foundation in the version of the CEA following this change .

  • 58.

    See the "Counterfactual non-philanthropic" sheet in the version of the CEA following this change here. The estimates on the "Counterfactual non-philanthropic sheet" are used in the leverage and funging sections on each top charity's CEA sheet.

  • 59.

    See here in the version of the CEA preceding this change.

  • 60.

    See the "Counterfactual non-philanthropic" sheet in the version of the CEA following this change here.

  • 61.

    For example, we discuss potential development impacts of mass deworming programs in this section of our intervention report.

  • 62.

    See this section of our Helen Keller International sheet and this section of our New Incentives sheet in the version of the CEA preceding this change.

  • 63.

    See this section of our sheet on New Incentives and this section of our sheet on Helen Keller International in the version of the CEA following this change.

  • 64.

    Compare the version of the CEA preceding this change here to the version following this change here.

  • 65.

    Compare the version of the CEA preceding this change here to the version following this change here.