2023 Cost-Effectiveness Analysis Changelog

This page provides details about changes that were made to our cost-effectiveness analysis (CEA) in 2023. Each changelog entry represents our understanding at the time the change was made. For past versions of our CEA, see this page.

Table of Contents

Version 1 — Published January 13, 2023

Link to the cost-effectiveness analysis (CEA) file: 2023 CEA — version 1

See this spreadsheet for the impact of each of the changes below on our cost-effectiveness estimates.

Change 1: Added additional Nigerian states to the New Incentives CEA and separated out cost-effectiveness estimates by state.

Our CEA previously calculated a single cost-effectiveness estimate for New Incentives' conditional cash transfer program in Nigeria by averaging together vaccination and disease burden estimates for the three states New Incentives originally operated in: Jigawa, Katsina, and Zamfara. New Incentives has now expanded its program into areas within a couple other Nigerian states1 and is planning to expand to even more states in the future.2 In order to more accurately estimate the cost-effectiveness of New Incentives' program in different areas, we have now updated our CEA to calculate separate cost-effectiveness estimates for each state and incorporated additional vaccination and disease burden data specific to current and potential expansion states.3

Change 2: Updated the counterfactual vaccination coverage rate in the New Incentives CEA

In our CEA, we model the benefits of New Incentives' program in terms of "counterfactually vaccinated" infants, meaning we only count the benefits to vaccinated infants who wouldn't have been vaccinated in the program's absence. In order to make this calculation, we estimate the proportion of infants enrolled in New Incentives' program who would have been vaccinated whether the program existed or not. How we set this estimate can have a significant effect on cost-effectiveness, since the costs of vaccinating these infants are still included in the CEA, even though the benefits aren't.

In the previous version of the CEA, we based this estimate on the endline vaccination rate for the BCG vaccine among the control group of the randomized controlled trial (RCT) of New Incentives' program. After adjusting for self-reporting bias, the results of the RCT imply that nearly half (48%) of enrolled infants would have received BCG vaccinations in the absence of the program.4

New Incentives has begun conducting rapid assessments of baseline vaccination coverage before expanding to new areas, and we have now seen the results of the earliest of these rapid assessments. After adjusting for self-reporting bias, the rapid assessments imply lower average baseline BCG vaccination coverage (34%) than the results of the RCT.5 We expect vaccination coverage in New Incentives' current areas of operation and future expansion areas to be more similar to the areas assessed in the rapid assessments than the areas where the RCT was conducted. However, we also think data from the RCT is likely to be higher quality than data from the rapid assessments.6 We have accordingly decided to put 35% weight on the results of the RCT and 65% weight on the results of the rapid assessments, resulting in a final counterfactual vaccination coverage estimate of 39%.7

Change 3: Updated the cost per infant estimates in the New Incentives CEA

We updated our cost per infant analysis for New Incentives to incorporate program data from September 2021 to July 2022. This resulted in a slight decrease to our estimate of New Incentives' cost per enrolled infant.8

We also added an adjustment to our estimates of spending incurred by the Nigerian government and Gavi, an international organization that primarily supports vaccination programs in low-income countries, to account for some vaccination costs being fixed costs of the immunization platform. We very roughly guess that fixed costs make up about 30% of total costs and have accordingly reduced our estimates of the government's and Gavi's costs per counterfactually vaccinated infant by 30%.9 We are highly uncertain about the appropriate value for this adjustment and may investigate this question further in the future.

Change 4: Updated the proportion of PBO nets purchased for South Sudan in the cost per net and insecticide resistance analyses for the Against Malaria Foundation (AMF) CEA

For some distributions of long-lasting insecticide-treated nets (LLINs), a portion of the nets AMF purchases are treated with a piperonyl butoxide (PBO) synergist in addition to standard insecticide (pyrethroid). These "PBO nets" are more expensive than standard nets, but there is evidence that PBO improves the effectiveness of nets in areas where a significant proportion of mosquitoes are resistant to standard insecticide.

In a previous changelog entry, we described our work creating a CEA for LLIN distributions in South Sudan, including our work estimating an appropriate insecticide resistance adjustment for this country. Based on the data we'd seen on insecticide resistance rates in South Sudan and our initial understanding of AMF's plans, we initially expected that all of the nets purchased by AMF for a distribution in South Sudan would be PBO nets. This assumption increased our cost per net estimate for the program and decreased the size of our insecticide resistance adjustment, which estimates the reduction in LLIN efficacy caused by local insecticide resistance.10

However, we have since learned that about 80% of the LLINs needed for the upcoming 2023 distributions in South Sudan are standard nets that were ordered prior to AMF's involvement in planning the campaign. AMF has told us that it will backfill the funding for those standard nets and that it plans to purchase PBO nets to make up the remaining 20% of nets needed for the campaign. We have updated our cost per net estimate and insecticide resistance adjustment in our South Sudan CEA to incorporate the lower proportion of PBO nets. We also incorporated some additional cost information we received from AMF into our cost per net estimate. These changes decreased our cost per net estimate for South Sudan slightly and increased our insecticide resistance adjustment for South Sudan substantially, leading to an overall decrease in cost-effectiveness.11

Change 5: Updated the leverage and funging and campaign spacing adjustments in the Against Malaria Foundation (AMF) CEA for South Sudan

We learned additional information about the funding landscape for LLIN campaigns in South Sudan and how AMF's contributions will impact those campaigns. This led us to update two adjustments in our AMF CEA for South Sudan: our leverage and funging adjustment and our adjustment for the impact of the program being to reduce the amount of time between LLIN distributions.

Leverage and funging

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

As part of our leverage and funging adjustment calculations, we estimate the probability of several scenarios that might occur in the absence of philanthropic support for a particular program (e.g., "government costs would replace philanthropic costs" or "distributions would go unfunded"). In our AMF CEA, one of the scenarios we include is that the Global Fund to Fight AIDS, Tuberculosis and Malaria (the Global Fund) or the President's Malaria Initiative (PMI) would fund the program in the absence of AMF's support. We had previously estimated that there was a 30% chance the Global Fund would replace philanthropic costs for a distribution in South Sudan (PMI does not provide funding to South Sudan). We have since gotten feedback from AMF that, given the time-sensitivity of this funding gap, it was very unlikely that the Global Fund would reallocate funding in time to fill it. Based on this feedback, we have decreased our estimate to 15%.12

Campaign spacing

For most countries, our AMF CEA estimates the cost-effectiveness of philanthropic funding causing people to receive new LLINs who otherwise would not have received LLINs. In situations where we believe the true impact of philanthropic funding is to allow a distribution to occur sooner than it would have otherwise, we apply an adjustment to account for the difference in cost-effectiveness between these two scenarios. Based on information we received from AMF and conversations we had with other stakeholders, we expect that the impact of providing funding for LLIN campaigns in South Sudan would be to allow them to take place an average of 10 months sooner than they would have otherwise, moving them from an average interval of 45 months between campaigns to an average of 35 months between campaigns.13 We estimate that this is 37% less cost-effective than the scenario modeled in the unadjusted AMF CEA,14 and have accordingly added a 37% downward adjustment to the CEA.15

Change 6: Updated the counterfactual value of spending by Gavi in the New Incentives CEA

Our top charities' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

In order to estimate the impact that leverage and funging have on the cost-effectiveness of a program, we need to estimate how cost-effectively other actors would hypothetically spend their money if they didn't spend it on the program, which we refer to as the "counterfactual value" of their spending. One of the key benefits we model in our CEA of New Incentives' conditional cash transfers for vaccinations program is leveraging funding from Gavi, an international organization that primarily supports vaccination programs in low-income countries. We previously assumed that the counterfactual value of Gavi's spending is equal to the counterfactual value of the Global Fund's spending, which we estimate to be about 38% as cost-effective as an Against Malaria Foundation (AMF)-funded LLIN distribution in the DRC.16

Conceptually, this estimate was based on the idea that our estimate of the counterfactual value of Gavi's spending should approximate the cost-effectiveness of other programs Gavi supports, assuming that Gavi funds that would be leveraged by New Incentives' program would otherwise be spent on other programs within Gavi's portfolio. However, after investigating this issue in more depth, we concluded that Gavi has consistently been able to raise sufficient funding to cover its entire portfolio of programs, and we expect it will continue to be able to do so in the future. Therefore, it makes more sense to conceptualize the counterfactual value of Gavi's spending as how cost-effectively Gavi's donors, primarily the Bill and Melinda Gates Foundation and high-income country governments, would otherwise spend the funds that would be leveraged by New Incentives' program.

Upon investigating the cost-effectiveness of spending by Gavi's donors, we revised our estimate of the counterfactual value of Gavi's spending downward by more than half, from 0.0167 units of value per dollar to 0.007 units of value per dollar, which increased the overall cost-effectiveness of New Incentives' program.17

Change 7: Added ten additional Nigerian states to the New Incentives CEA

We learned that New Incentives is considering expanding its conditional cash transfer program to ten additional Nigerian states. We have added those states to our New Incentives CEA.18

  • 1
    • We recommended grants to fund expansions of New Incentives' program into new areas in August 2021, January 2022, and May 2022. New Incentives has told us that some of the expansion areas funded by the May 2022 grant are located in the states of Bauchi and Sokoto.
    • “This means we can now start operations in Sokoto once the baseline rapid assessments are completed (assuming coverage rates are low). We have submitted a draft MoU for expansion and collaboration to Bauchi State and have requested some members of the Bauchi State Health Research Ethics Committee (BASHREC) to review our draft application for rapid assessments. While we will continue to expand within current states and start expansion in Sokoto, going to Bauchi should help us minimize the proportion of LGAs with a likelihood of network shutdown or security issues, and also help us reduce vaccine supply and stakeholder risks since Bauchi is in the North East Zone of Nigeria. Bauchi was selected after reviewing states for low immunization rates, high Under-5 mortality (IHME), security, stakeholder support, measles incidence (2016), and other factors.” New Incentives, Program update to GiveWell, December 2021 [unpublished].

  • 2

    We recommended a grant in November 2022 that New Incentives expects to use to expand its program into six additional states in Nigeria. More information on this grant will be published in a forthcoming grant page.

  • 3

    See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 4

    See the version of the CEA preceding this change here.

  • 5

    See the results of the rapid assessments and our calculations here.

  • 6

    For more detail, see this write-up.

  • 7

    See our calculations here. See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 8

    See the previous version of our cost per infant analysis here and the updated version of this analysis here.

  • 9

    See our calculations here. See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 10

    We had previously set the cost per net estimate in South Sudan equal to the cost per net estimate in DRC, partially on the basis of our expectation that AMF would purchase a similarly large proportion of PBO nets for campaigns in South Sudan as in DRC. See our previous cost per net estimate for South Sudan here. See the previous version of our insecticide resistance adjustment calculations here.

  • 11

    See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 12

    See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 13

    See more detail in this spreadsheet

  • 14

    See our calculations here.

  • 15

    See the version of the CEA preceding this change here and the version of the CEA following this change here.

  • 16

    See the version of the CEA preceding this change here.

  • 17

    See our full write-up on the counterfactual value of Gavi spending here and our calculations here. See the version of the CEA following this change here.

  • 18

    See the version of the CEA preceding this change here and the version of the CEA following this change here.