You are here

2019 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2019. Below each change, we share a table indicating how the change impacted our cost-effectiveness estimates for each charity. (If a charity doesn't appear in the table, it was not impacted by the change.) For past versions of our CEA, see this page.

Version 5 — Published August 7, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 5

Change 1: Restructured to calculate country-level cost-effectiveness estimates and to use an outcomes-per-dollar structure

Background

Every year, we use our cost-effectiveness model as one of the primary inputs into our decisions about where to recommend that Good Ventures allocate funding among our top charities.

In past years, the structure of the model has made it difficult to use, and therefore more prone to error. A few problems we ran into last year included:

  1. The only way to look at the cost-effectiveness of individual programs was to make a separate copy of the model for each program.
  2. The main cost-effectiveness model didn't necessarily give the same result as the individual charity programs (due to different weightings), which made it more difficult to vet for errors.
  3. For the parameters for which we have country-specific data (e.g. baseline mortality rates), we were aggregating the country-specific data into a single figure, weighted by e.g. the proportion of the charity's funding that was spent in each country over the past few years. The country weightings for different parameters were calculated on separate spreadsheets, which made it possible for us to use different country weights for different parameters for the same charity. In putting together this update, we found that we had in fact used different country weights for the cost per treatment and deworming intensity adjustment parameters for Deworm the World.
  4. It was difficult to track the total good outcomes we expected to accomplish with a given grant.

Description of changes

To address these problems, we made two major changes to the structure of the model:

  1. Adding country-specific information to columns of intervention sheets: We are now using the columns in the intervention sheets to calculate the cost-effectiveness of programs by country (or by state, in some cases), addressing the first three problems listed above.
  2. Moving from a dollars-per-outcome structure to an outcomes-per-dollar structure: Previously, the CEA was structured to calculate the cost per outcome (for example, the cost per death averted). We restructured the model to start by inputting an arbitrary donation amount and modeling how that donation would be split across countries, and what outcomes would be achieved in each country with that amount of funding (e.g. how many deaths would be averted in each country, as well as in total). This allows us to consistently and easily model the total good outcomes that we expect a given grant to achieve, addressing the fourth problem listed above.

We also made some formatting changes to make the model easier to read, as well as making a few minor changes to address errors or areas of improvement that we found.

Impact of these changes on bottom-line cost-effectiveness estimates

Since this was primarily a structural update, we did not expect significant changes to the bottom line. These changes were mainly due to errors or inconsistencies that we identified and corrected.

The overall changes in cost-effectiveness between version 4 and version 5 are depicted in the table below. We include additional tables below for a breakdown of the impact of individual changes.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Deworm the World 19.2x 17.5x -8.9%
Schistosomiasis Control Initiative 9.8x 11.1x 12.5%
Sightsavers 10.3x 10.2x -0.6%
Against Malaria Foundation 6.4x 7.4x 14.2%
Malaria Consortium 8.5x 8.5x 0.0%
Helen Keller International 5.8x 5.4x -7.6%
The END Fund 5.8x 6.1x 5.2%

Total change to the bottom line that resulted from the structural updates (rather than the other changes listed below)



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Sightsavers 10.3x 10.2x -1.0%
Against Malaria Foundation 6.4x 6.7 4.7%
Malaria Consortium 8.5x 8.4 -1.2%
Helen Keller International 5.8x 5.9 1.7%

Moved from a single overall insecticide resistance adjustment to country-specific adjustments



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Sightsavers 10.2x 10.3x 1.0%
Against Malaria Foundation 6.7x 6.6x -1.5%

Reduced the expected lifespan of an LLIN in DRC by 10%

The data we have seen from AMF's distributions in DRC provide some evidence that LLINs decayed considerably more quickly than expected. We added an adjustment of 10% to account for this.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Schistosomiasis Control Initiative 9.8x 9.9x 1.0%
Against Malaria Foundation 6.6x 6.5x -1.5%

Moved from an overall estimate of the proportion of deworming going to children in SCI's programs to country-specific estimates

Since we don't have good data on the proportion of deworming treatments going to children in the END Fund's programs, we guess that the proportion is roughly similar to that in SCI's programs. Accordingly, changing the estimate for SCI also changed our estimate of the END Fund's cost-effectiveness.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Schistosomiasis Control Initiative 9.9x 11.2x 13.1%
The END Fund 5.8x 6.1x 5.2%

Updated estimates of malaria prevalence in AMF context (Age <5)

We noticed that our estimates of malaria prevalence for children under age 5 in the countries where AMF works were out of date, so we updated this to use more recent data.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Schistosomiasis Control Initiative 11.2x 11.1x -0.9%
Sightsavers 10.3x 10.2x -1.0%
Against Malaria Foundation 6.5x 6.8x 4.6%

Updated estimates of malaria prevalence in AMF context (Age 5-14)

We noticed that our estimates of malaria prevalence for children aged 5-14 in the countries where AMF works were out of date, so we updated this to use more recent data.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Against Malaria Foundation 6.7x 7.0x 2.9%

Removed a 5% wastage adjustment for AMF

We removed this adjustment for consistency with our other charities. We plan to add this adjustment to the spreadsheet that we use at the end of the year to help with our decisions about what funding allocation to recommend to Good Ventures.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Against Malaria Foundation 7.0x 7.4x 5.7%
Malaria Consortium 8.4x 8.5x 1.2%

Updated country weightings for Deworm the World

We noticed that for Deworm the World, we were previously using two different sets of country weights for different parameters. Specifically, the calculations of a) an overall cost per treatment and b) an overall deworming intensity adjustment both weighted countries based on the proportion of treatments that went to each country, but were looking at different time periods. We chose one of these time periods to use for both parameters.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Deworm the World 19.2x 17.5x -8.9%

Updated the proportion of total benefits attributed to development effects for HKI

We were previously benchmarking our estimate of HKI's development effects off of our estimate of the development effects of deworming. We moved to using the median of staff estimates of the percentage of total benefits of vitamin A supplementation that come from development effects, which changed this percentage from 31% to 24%.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Helen Keller International 5.9x 5.4x -8.5%

Version 4 — Published May 29, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 4

Change 1: Fixed an error in which moral weights were being used in the "Leverage/funging" sheet

It was brought to our attention that in some places in the "Leverage/funging" sheet, our calculations incorporated a single staff member's moral weights rather than our staff aggregate moral weights. We fixed the relevant calculations.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Deworm the World 19.3x 19.2x -0.3%
Schistosomiasis Control Initiative 9.9x 9.8x -0.6%
Sightsavers 10.3x 10.3x -0.4%
Against Malaria Foundation 6.4x 6.4x 0.3%
Malaria Consortium 8.5x 8.5x -0.7%
Helen Keller International 5.9x 5.8x -0.9%
The END Fund 5.8x 5.8x -0.5%

Change 2: Added a parameter to the "Inclusion/exclusion" sheet

We added "Serious adverse events due to side effects of SMC treatment" to the "Inclusion/exclusion" sheet as a parameter that we have not included in the CEA. For more detail, see row 51 here.

Version 3 — Published March 21, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 3

Change 1: Fixed errors in calculation of insecticide resistance adjustment

We found and fixed some errors in the way we were calculating our estimate of the reduction in the effectiveness of anti-malarial bednets due to insecticide resistance.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Deworm the World 19.3x 19.3x -0.1%
Schistosomiasis Control Initiative 10.0x 9.9x -1.0%
Sightsavers 10.4x 10.3x -0.6%
Against Malaria Foundation 5.6x 6.4x 15.5%
Malaria Consortium 8.5x 8.5x 1.1%
The END Fund 5.8x 5.8x -0.3%

Version 2 — Published January 25, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 2

Change 1: Moved from aggregating outputs of staff values to aggregating inputs

For some of the most uncertain and debatable parameters in our cost-effectiveness estimates, we attempt to use the wisdom of the crowds to reduce bias. That is, we allow multiple staff members to input their own best guesses, and then aggregate those answers to arrive at a central estimate. In this version of the CEA, we made a change to the way we aggregate staff estimates.

Previously, for each of our charities, each staff member would calculate their own cost-effectiveness results based on their own inputs. We took a median of these results to arrive at the final output of our CEA. We changed this to instead aggregate each input and feed these aggregate inputs through the model to arrive at the final output of our CEA.

The aggregated inputs can be seen on the User inputs and Moral weights sheets.

The primary advantages we see to this structure are:

  • It is easier to track how changes in staff inputs (including adding and removing staff members from the model) lead to changes in our final results, particularly when we make several changes simultaneously.
  • We believe there are theoretical reasons to aggregate staff inputs rather than aggregating outputs of the model, though we are not certain we fully understand every argument for this. In brief, this is because when aggregating outputs, we are taking into account information about who contributed which inputs (which in the majority of cases we believe is irrelevant) and disregarding other information that may be relevant. For example, if a staff member had a particularly low estimate for the value of preventing deaths relative to increasing consumption, it is likely that they would be towards the lower end of estimates for the cost-effectiveness of life-saving interventions, such as distributing anti-malaria bed nets. Their empirical estimates would therefore be disregarded, because they would be unlikely to influence which value is taken as the median for a given parameter.
  • We think that by making it clearer which values are being fed into the model, this structure makes it easier for others to productively disagree with the values we are using.

The primary disadvantages we see to this structure are:

  • As noted above, we are not taking into account any relevant information that may be communicated by which staff member contributed which inputs (for example, in cases where it would make sense for someone with a low value for one parameter to also have a low value for another parameter). For the most part we believe that the values staff use for each parameter should be independent of the values they use for others, but there may be a small number of cases where we are disregarding relevant information.
  • It is not clear to us what is the best way to aggregate moral weights. The primary difficulty comes from the fact that staff use different units to represent their moral weights, and we are therefore aggregating the ratios between staff values for different outcomes. When aggregating more than two ratios, the aggregate result changes depending on which outcome we normalize our results to (i.e. the outcome to which we arbitrarily assign a value of 1). To mitigate the effect of this arbitrary decision, we have normalized the moral weights to three different outcomes and taken the mean of the results of the three outcomes. We are not confident that this is the theoretically best approach, but have not prioritized further work as we believe it is unlikely to materially change our conclusions.

      Our bottom-line cost-effectiveness estimates for each charity changed slightly as a result of the new structure.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 19.1x 19.3x 1.3%
      Schistosomiasis Control Initiative 9.6x 10.0x 4.3%
      Sightsavers 10.1x 10.4x 2.8%
      Against Malaria Foundation 5.8x 5.6x -3.8%
      Malaria Consortium 9.0x 8.5x -6.6%
      Helen Keller International 6.1x 5.9x -4.5%
      The END Fund 5.7x 5.8x 2.3%

      Version 1 — Published January 3, 2019

      Link to the cost-effectiveness analysis file: 2019 CEA — Version 1

      Change 1: Removed suggested inputs from "User inputs" sheet

      We removed the suggested values from the "User inputs" sheet because these values no longer represent our best guesses. The median result of the CEA was not affected by this change.

      Change 2: Removed two former staff members

      We removed two former staff members from the cost-effectiveness model.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 18.3x 19.0x 3.9%
      Schistosomiasis Control Initiative 9.4x 9.6x 2.1%
      Sightsavers 9.8x 10.1x 3.3%
      The END Fund 5.5x 5.7x 3.9%

      Change 3: Fixed a small error in malaria development effects calculation

      We found and fixed a small error in the way we were calculating malaria development effects.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 19.0x 19.1x 0.0%
      Schistosomiasis Control Initiative 9.6x 9.6x 0.2%
      Sightsavers 10.1x 10.1x 0.1%
      Against Malaria Foundation 5.9x 5.8x -2.4%
      Malaria Consortium 9.3x 9.0x -3.0%
      The END Fund 5.7x 5.7x 0.1%