You are here

2019 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2019. Below each change, we share a table indicating how the change impacted our cost-effectiveness estimates for each charity. (If a charity doesn't appear in the table, it was not impacted by the change.) For past versions of our CEA, see this page.

Version 3 — Published March 21, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 3

Change 1: Fixed errors in calculation of insecticide resistance adjustment

We found and fixed some errors in the way we were calculating our estimate of the reduction in the effectiveness of anti-malarial bednets due to insecticide resistance.



Charity
Median [Charity]
vs. cash before
Median [Charity]
vs. cash after
Percent
change
Deworm the World 19.3x 19.3x -0.1%
Schistosomiasis Control Initiative 10.0x 9.9x -1.0%
Sightsavers 10.4x 10.3x -0.6%
Against Malaria Foundation 5.6x 6.4x 15.5%
Malaria Consortium 8.5x 8.5x 1.1%
The END Fund 5.8x 5.8x -0.3%

Version 2 — Published January 25, 2019

Link to the cost-effectiveness analysis file: 2019 CEA — Version 2

Change 1: Moved from aggregating outputs of staff values to aggregating inputs

For some of the most uncertain and debatable parameters in our cost-effectiveness estimates, we attempt to use the wisdom of the crowds to reduce bias. That is, we allow multiple staff members to input their own best guesses, and then aggregate those answers to arrive at a central estimate. In this version of the CEA, we made a change to the way we aggregate staff estimates.

Previously, for each of our charities, each staff member would calculate their own cost-effectiveness results based on their own inputs. We took a median of these results to arrive at the final output of our CEA. We changed this to instead aggregate each input and feed these aggregate inputs through the model to arrive at the final output of our CEA.

The aggregated inputs can be seen on the User inputs and Moral weights sheets.

The primary advantages we see to this structure are:

  • It is easier to track how changes in staff inputs (including adding and removing staff members from the model) lead to changes in our final results, particularly when we make several changes simultaneously.
  • We believe there are theoretical reasons to aggregate staff inputs rather than aggregating outputs of the model, though we are not certain we fully understand every argument for this. In brief, this is because when aggregating outputs, we are taking into account information about who contributed which inputs (which in the majority of cases we believe is irrelevant) and disregarding other information that may be relevant. For example, if a staff member had a particularly low estimate for the value of preventing deaths relative to increasing consumption, it is likely that they would be towards the lower end of estimates for the cost-effectiveness of life-saving interventions, such as distributing anti-malaria bed nets. Their empirical estimates would therefore be disregarded, because they would be unlikely to influence which value is taken as the median for a given parameter.
  • We think that by making it clearer which values are being fed into the model, this structure makes it easier for others to productively disagree with the values we are using.

The primary disadvantages we see to this structure are:

  • As noted above, we are not taking into account any relevant information that may be communicated by which staff member contributed which inputs (for example, in cases where it would make sense for someone with a low value for one parameter to also have a low value for another parameter). For the most part we believe that the values staff use for each parameter should be independent of the values they use for others, but there may be a small number of cases where we are disregarding relevant information.
  • It is not clear to us what is the best way to aggregate moral weights. The primary difficulty comes from the fact that staff use different units to represent their moral weights, and we are therefore aggregating the ratios between staff values for different outcomes. When aggregating more than two ratios, the aggregate result changes depending on which outcome we normalize our results to (i.e. the outcome to which we arbitrarily assign a value of 1). To mitigate the effect of this arbitrary decision, we have normalized the moral weights to three different outcomes and taken the mean of the results of the three outcomes. We are not confident that this is the theoretically best approach, but have not prioritized further work as we believe it is unlikely to materially change our conclusions.

      Our bottom-line cost-effectiveness estimates for each charity changed slightly as a result of the new structure.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 19.1x 19.3x 1.3%
      Schistosomiasis Control Initiative 9.6x 10.0x 4.3%
      Sightsavers 10.1x 10.4x 2.8%
      Against Malaria Foundation 5.8x 5.6x -3.8%
      Malaria Consortium 9.0x 8.5x -6.6%
      Helen Keller International 6.1x 5.9x -4.5%
      The END Fund 5.7x 5.8x 2.3%

      Version 1 — Published January 3, 2019

      Link to the cost-effectiveness analysis file: 2019 CEA — Version 1

      Change 1: Removed suggested inputs from "User inputs" sheet

      We removed the suggested values from the "User inputs" sheet because these values no longer represent our best guesses. The median result of the CEA was not affected by this change.

      Change 2: Removed two former staff members

      We removed two former staff members from the cost-effectiveness model.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 18.3x 19.0x 3.9%
      Schistosomiasis Control Initiative 9.4x 9.6x 2.1%
      Sightsavers 9.8x 10.1x 3.3%
      The END Fund 5.5x 5.7x 3.9%

      Change 3: Fixed a small error in malaria development effects calculation

      We found and fixed a small error in the way we were calculating malaria development effects.



      Charity
      Median [Charity]
      vs. cash before
      Median [Charity]
      vs. cash after
      Percent
      change
      Deworm the World 19.0x 19.1x 0.0%
      Schistosomiasis Control Initiative 9.6x 9.6x 0.2%
      Sightsavers 10.1x 10.1x 0.1%
      Against Malaria Foundation 5.9x 5.8x -2.4%
      Malaria Consortium 9.3x 9.0x -3.0%
      The END Fund 5.7x 5.7x 0.1%