New Incentives (Conditional Cash Transfers to Increase Infant Vaccination)

Published: December 2021; Last updated: September 2022

Previous versions of this page:

• May 2021 version
• November 2020 version

Does it work?

The primary benefit of New Incentives' program is reduction in mortality for vaccinated infants until the age of five. (More)

Other benefits of New Incentives' program that we incorporate into our overall estimate of the program's impact are:

• Reduction in mortality beyond the age of five. Reductions in mortality from vaccine-preventable disease also occur for vaccinated people after the age of five. (More)
• Development benefits. Early-life health improvements from vaccines might lead to increases in income when vaccinated children become adults. (More)
• Consumption benefits. Cash transfers received by caregivers may improve well-being through increased consumption. (More)

We have also attempted to account for:

• Additional benefits (e.g., morbidity effects, benefits of herd immunity) and negative or offsetting impacts (e.g., crowding out of New Incentives by other vaccination promotion programs, inflation). Rather than explicitly modeling these, we have applied percentage adjustments based on our best guesses. (More)
• Downside adjustments. We adjust for risk of wastage, the quality of monitoring and evaluation, and our level of confidence in funds being used for their intended purpose. (More)

What are the effects on child mortality?

We view reducing child mortality by improving the uptake of vaccines that prevent diseases that cause mortality as the primary benefit of New Incentives' program. We estimate that New Incentives' program averts 4.9 child deaths per 1,000 infants in areas where New Incentives operates, based on the following factors:1

• The effect of New Incentives' program on vaccination rates. We estimate the program leads to a 22 percentage point increase in the use of (directly and indirectly) incentivized vaccines in three states in North West Nigeria. This estimate is based on a GiveWell-funded randomized controlled trial (RCT) of New Incentives' program by IDinsight, to which we apply adjustments for vaccines with multiple doses and attenuation due to self-reporting bias. (More)
• The prevalence of vaccine-preventable disease and mortality in areas where New Incentives operates. We estimate that the probability of death for an unvaccinated child under 5 from diseases that could be prevented by one of the vaccines incentivized by New Incentives is 3.0%. This estimate is based primarily on data from the Institute for Health Metrics and Evaluation (IHME), to which we apply a number of adjustments. (More)
• The effect of increased vaccination on prevalence of vaccine-preventable disease. We estimate that the vaccines incentivized by New Incentives reduce recipients' risk of vaccine-preventable disease by 76%, based on meta-analyses of vaccines' effects on vaccine-preventable diseases and adjustments for (a) concerns about potentially lower vaccine efficacy in low-income countries and (b) "non-specific" effects on mortality from diseases other than those directly targeted by vaccines. (More)

We discuss each of these factors in more detail below.

Does New Incentives' program lead to increased vaccination?

IDinsight, funded by a GiveWell Incubation Grant, conducted an RCT to measure the impact of New Incentives' program on vaccination rates in three states in North West Nigeria.2 The RCT found that New Incentives' program had high uptake and led to significant increases in the use of several vaccines incentivized by the program. Households served by clinics where New Incentives was providing CCTs for vaccinations had substantially higher self-reported vaccination, compared to control clinics. We view the RCT as high-quality, and its findings are consistent with similar studies.3

The RCT finds a 14 to 21 percentage point increase in the study's three pre-specified primary outcomes — i.e. self-report of having received (1) the BCG vaccine, (2) the measles vaccine, and (3) any of the three doses of PENTA vaccine. Relative to control households, households in the catchment area for treatment clinics were:

• 16 percentage points more likely to report receiving the BCG vaccination (95% CI 0.12-0.21, 63% control group mean),4
• 21 percentage points more likely to report receiving any of the three doses of PENTA (95% CI 0.16-0.26, 54% control group mean),5 and
• 14 percentage points more likely to report any dose of measles (95% CI 0.10-0.18, control group mean 59%).6

We average the effects on each of these outcomes to generate an overall effect size estimate of 17 percentage points.7

We then apply the following adjustments to this initial estimate:

• A roughly 20% upward adjustment (from 17 percentage points to 20 percentage points) to account for self-report bias (see below).
• A 13% upward adjustment to account for vaccines with multiple doses, which we believe, on net, leads to an increase in the effective number of infants achieving full vaccination (see below).
• A 4% downward adjustment toward our prior of a 16 percentage point increase in vaccination rates (see below).

With these adjustments incorporated, our overall best guess is that New Incentives' program increased vaccination rates in the treatment group by 22 percentage points.

Overall, we view the evidence for the main effect on vaccination rates as reasonably strong. Factors that give us confidence in this result include:

• GiveWell has followed this RCT closely and believes that it meets high quality standards.
• These findings are consistent with our pre-registered prior for the effect size, based on trials of similar incentives for vaccination programs (see below).
• These findings are from a pre-registered RCT.
• The computer code used to statistically analyze the data was checked by an external auditor.8
• The increase in treatment clinic vaccinations over and above increases in control clinic vaccinations occurs in clinic data as well as self-report data.9
• The findings from IDinsight’s baseline report of New Incentives are consistent with the plausibility of immunization incentives as a driver of observed behavior change in the areas where New Incentives operates.10

Our main remaining concerns include:

• We remain uncertain about the appropriate adjustments for vaccines with multiple doses (see below) and self-reporting bias (see below), which have a substantial impact on our estimate of New Incentives' impact and cost-effectiveness.
  
• The self-reported data in the RCT show a large and surprising increase in vaccination rates in control clinics from baseline to endline. At baseline in 2017, vaccination coverage rates in control areas were roughly 15%-25% across vaccines; at endline, they were around 60%.11 While this increase in counterfactual coverage appears in self-reported data, clinic administrative records in control areas do not show any increase in vaccination volume since baseline.12 We are very unsure how to explain this discrepancy (though we do expect administrative recordkeeping to be somewhat better in treatment areas than in control areas).

• GiveWell recommended the funding for the IDinsight RCT, and New Incentives is a GiveWell Incubation Grant recipient. As a result, there may be an incentive for GiveWell to review the results favorably.

Self-reporting bias

The primary outcome New Incentives measures is caregiver self-report of vaccinations received by infants. Caregivers might misreport infant vaccinations, potentially leading to both false negatives and false positives in the data that could cause us to either over- or under-estimate treatment effects (see this page for more information). Self-reporting bias may also affect our estimate of vaccination coverage in control areas, which is a key input for our cost analysis.

Our best guess is that self-report bias causes an overestimate of control group vaccination coverage in the data and, overall, leads to an underestimate of the effect of New Incentives' program on vaccination rates.

Comparing self-report data to BCG scarring data

In addition to measuring self-reported vaccination, IDinsight reports on the effect of the program on:13

• child health cards (CHCs),
• clinic immunization records, and
• BCG scarring.

Each of these measures has limitations. For the purposes of attempting to correct for self-reporting bias, we have chosen to focus on BCG scarring.

The BCG scar rate among the control group is 41%, and the treatment effect on scars is 22 percentage points, suggesting a treatment scar rate of 63%.14 We make adjustments to this result to account for (1) some individuals who are vaccinated not developing scars, and (2) enumerators potentially failing to identify all BCG scars.

We assume, based on scarring rates in the literature, that 90% of infants in the control group who received BCG developed a scar.15 Based on data we received from New Incentives, we assume that 97% of infants in the treatment group who received BCG developed a scar.16 We also assume that (a) 90% of infants who receive BCG and develop a scar have their scar detected by IDinsight enumerators, and (b) 95% of infants who do not develop a scar are correctly identified by enumerators.17 This implies a control group BCG rate of 48% and a treatment group rate of 69% — i.e., a treatment effect of roughly 20 percentage points. This treatment effect of a 20 percentage point increase in vaccination is roughly in line with the increases in self-reported vaccination for the study's three primary outcomes — i.e., receiving the BCG vaccine, the measles vaccine, and any dose of the PENTA vaccine. For comparison, the treatment effect on BCG vaccination based on self-report data is a 16 percentage point increase over a control vaccination rate of 63% (see the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Increase in vaccination rates due to New Incentives" section).

The following table summarizes (1) self-report data across the RCT's three primary outcomes, (2) BCG scar data, and (3) our best guess of actual BCG vaccination rates.18

Outcome	Control	Treatment Effect	Control + Treatment Effect
Self-report, any dose of PENTA	0.54	0.21	0.75
Self-report, measles	0.59	0.14	0.73
Self-report, BCG	0.63	0.16	0.79
BCG scar	0.41	0.22	0.63
BCG guess	0.48	0.20	0.69

If our best guess for control group coverage (48%) is accurate, this implies that self-report of BCG is biased upward in both the control group (63% using self-reported BCG vs. 48% using best guess true BCG) and the treatment group (79% using self-reported BCG vs. 69% using best guess true BCG), but the bias in the treatment group is less.

We expect control group self-report to bias coverage estimates upward, because we expect a significant number of caregivers who did not vaccinate their infants to report that they did. In the treatment group, we expect this effect to cause less upward bias because, even if the percentage of caregivers who didn't vaccinate but falsely report vaccinating is the same (or higher) than in the control group, we expect the absolute number of false reports of vaccination to be smaller since a smaller proportion of infants were, in fact, not vaccinated.

Based on the above considerations, our best guess is that accounting for self-reporting bias leads to a roughly 20% increase in the effect of the program.19

Remaining uncertainties

Our key remaining uncertainties about accounting for self-reporting bias include:

• For both treatment and control groups, we assume the coverage rates for all vaccines are similar — i.e., differences are due to either statistical error or different levels of measurement error across different vaccines. This seems like a plausible assumption and puts more weight on BCG estimates, for which we have a clearer understanding of the biases present. However, an alternative approach (which we have not used in our estimates) would be to make best guesses about the sensitivity (i.e., the degree to which self-report correctly identifies infants that have been vaccinated) and specificity (i.e., the degree to which self-report correctly identifies infants that have not been vaccinated) of self-report data for each vaccine independently and make adjustments accordingly.20
• It might be worthwhile to consider effects across vaccines in more detail. For example, we believe self-report sensitivity is probably lower for other vaccines than for BCG, which would imply a smaller downward adjustment in control coverage.21 BCG has the highest control coverage (63%, compared to 59% and 54% for measles and PENTA, respectively), which is consistent with our belief, though we do not know by how much sensitivity varies across vaccines.
• We are unsure about our guesses for the rates of BCG scarring and of enumerator error in identifying BCG scars.
• This 20 percentage point increase is higher than our prior (see below), but previous studies (which informed our prior) may not explicitly account for self-reporting bias.

Vaccines with multiple doses

Several of the vaccines we model in our cost-effectiveness analysis require multiple doses. The current schedule includes PENTA vaccine (which includes the DTP and HiB vaccines) and PCV, both of which require three doses.22 Our cost-effectiveness analysis also includes rotavirus vaccine. Rotavirus was not part of the routine immunization schedule at the time of the RCT. We include it in the cost-effectiveness analysis because we anticipate it will be incorporated in the next few years.23 We assume the two doses of rotavirus would be administered at the same time as the first and second doses of PENTA and PCV.24

One of the primary metrics by which we evaluate the effect of New Incentives' program is whether caregivers in the RCT reported their children receiving any dose of the PENTA vaccine (see above). We also use this data to infer effects on receipt of any dose of PCV and anticipated effects on receipt of the rotavirus vaccine, since those vaccines are administered (or, in the case of rotavirus vaccine, anticipated to be administered) at the same time as PENTA. However, we base our estimate of New Incentives' mortality effects on meta-analyses that focus on disease reduction resulting from increases in rates of children receiving all recommended doses of a vaccine, rather than any dose. In order to account for this methodological difference, we make the following adjustments to our estimate of the effect of New Incentives' program:

• A downward adjustment to account for the fact that a portion of the children who received vaccinations as a result of New Incentives' program likely received fewer than the recommended number of doses, which is associated with decreased efficacy.25
• An upward adjustment to account for the fact that a portion of the children who would have received at least one dose of the PENTA vaccine in the absence of the program likely received additional doses as a result of the program, which is associated with increased efficacy.26

See here for our methods and calculations. Taking both adjustments into consideration, our best guess is that accounting for vaccines with multiple doses leads to an increase in effect of roughly 17% for PCV, DTP, and HiB27 and a 9% increase in effect for rotavirus,28 , for an overall increase in effect on vaccination rates of 13%.29

Spillover effects

It's possible that some caregivers in control clinic catchment areas went to clinics in treatment areas to receive incentives from New Incentives. This type of positive spillover effect would cause us to underestimate the effect of New Incentives' program, since those infants would be counted as vaccinated but are in the control group.

However, because IDinsight did not find any evidence of spillovers in the control group,30 we do not include any adjustment for spillover effects. (We do estimate there were substantial numbers of infants enrolled who were outside treatment clinic catchments but not in control clinic catchments. We discuss this further below.)

Evidence from other studies and adjustment toward skeptical prior

Before seeing the results from the RCT, we did a literature review of other programs providing incentives for vaccinations. Based on that review, we predicted that New Incentives' program would increase vaccination rates by 16 percentage points. (See our literature review and prior for program impact.)

This prior falls within the range of the reported effect sizes on primary outcomes in IDinsight's analysis (14 to 21 percentage point increases across vaccines).31 However, after accounting for self-report bias (see above), we estimate a treatment effect of 20 percentage points for New Incentives' program, which is roughly 30% higher than this prior.

To adjust toward this skeptical prior, we put 80% weight on the IDinsight RCT effect size (adjusted for self-report bias and vaccines with multiple doses) and 20% weight on our prior. We put a higher weight on the RCT because (a) it focuses on the same program in the same setting (whereas our prior was based on merely similar programs in different settings), and (b) it has undergone an independent code audit and a larger amount of scrutiny from GiveWell staff. Putting 80% weight on the observed 20 percentage point effect and 20% on the prior of a 16 percentage point effect leads to a weighted effect of 19.6 percentage points, or a downward adjustment factor of 0.96 (see the most recent version of our calculations cost-effectiveness analysis, "New Incentives" sheet, "Increase in vaccination rates due to New Incentives" section, "Adjustment toward skeptical prior" row).

Comparison to pre-analysis plan

Before receiving the results of the RCT, we drafted a pre-analysis plan that described how we would interpret the RCT results in our cost-effectiveness analysis. We split the factors we expected to consider into those related to internal validity and those related to external validity.

Internal validity. Regarding internal validity, we planned to focus primarily on bias from self-reports, along with several additional factors.32 A comparison of our pre-analysis plan and what we ended up considering is below:

• Self-report. In our cost-effectiveness analysis, we made adjustments for self-report bias, described here. As discussed in the pre-analysis plan, we conducted cross-checks with other measures and decided to focus primarily on BCG scars, since we have the most confidence in this as an alternative measure. We also made an adjustment to put some weight on our prior for the program (see here), as specified in the pre-analysis plan. We did not explore changes in variables related to social desirability or evidence of difficulty with caregiver recall, which were two additional considerations mentioned in the pre-analysis plan.
• Additional factors. Among the additional factors we planned to consider, we did consider treatment spillovers, alternative measles campaigns, and study quality. Factors specified in the pre-analysis plan that we did not prioritize review of or make adjustments for are: (a) the plausibility of immunization incentives as a driver of observed behavior change and (b) analysis of partially treated areas.

External validity. Regarding external validity, we planned to consider several factors. We ended up not making adjustments for several of these factors because our funding will support New Incentives' expansion in the same states in which it operated during the RCT, which reduces concerns about the RCT's external validity. However, we did adjust for several factors that might change over time within the same states between the RCT and implementation. A comparison of our pre-analysis plan and what we ended up considering is below:

• Baseline coverage. We include an adjustment for the potential "crowding out" of New Incentives due to counterfactual vaccination coverage changing over time (see below).
• Presence of competing vaccination campaigns. We would guess that some of the potential increase in counterfactual coverage and crowding out of New Incentives is due to other vaccination programs. We also account for some chance that another organization would have set up a CCT program similar to New Incentives' program in the absence of our funding New Incentives (see below).
• Introduction of additional vaccines. Our CEA incorporates benefits from rotavirus vaccine, which we guess will be introduced in 2021 but which was not available during the RCT.33
• Inflation-adjusted transfer size. We include an adjustment for the risk of inflation over time reducing the effectiveness of the current nominal (i.e., not inflation-adjusted) cash transfer amount (see below).
• Other factors. We did not prioritize review of the following factors, since we guess that they are less likely to be important, given that New Incentives is scaling up in the same states in which it operated during the RCT: poverty rates, cultural acceptability, delivery of routine immunization services, geographic access, supply-side issues, program outreach capacity and intensity, catchment security, New Incentives’ program implementation, vaccine effectiveness, disease environment, and disease spillovers.

What is the prevalence of vaccine-preventable disease and mortality in areas where New Incentives operates?

Our best guess is that the probability of death for an unvaccinated child under five from a disease that can be prevented by one of the vaccines incentivized by New Incentives' program is 3.0%.34 This estimate is primarily based on the Institute for Health Metrics and Evaluation (IHME)'s data on child mortality from vaccine-preventable diseases in Nigeria.35 We make adjustments to account for:

• The proportion of deaths from vaccine-preventable diseases caused by specific pathogens (i.e., disease etiology) (More)
• Deaths occurring before vaccines are administered (More)
• The share of children in the mortality data who received vaccines (More)
• A higher mortality rate in North West Nigeria for unvaccinated children due to worse health overall (More)

Adjustment for disease etiology and pre-vaccination deaths

The diseases targeted by the vaccines that New Incentives directly incentivizes are:

Disease	Vaccine
Lower respiratory infection caused by S. pneumoniae	PCV
Meningitis caused by S. pneumoniae	PCV
Lower respiratory infection caused by H. influenzae type b	HiB vaccine
Meningitis caused by H. influenzae type b	HiB vaccine
Diphtheria	DTP vaccine
Tetanus	DTP vaccine
Pertussis	DTP vaccine
Measles	Measles vaccine
Tuberculosis	BCG vaccine
Hepatitis B	Hepatitis B vaccine

New Incentives also indirectly incentivizes (or plans to indirectly incentivize) vaccines for the following diseases:

Disease	Vaccine
Diarrhea caused by rotavirus	Rotavirus vaccine
Polio	Oral polio vaccine
Yellow fever	Yellow fever vaccine

We take our initial estimate of the probability of death for children under 5 from vaccine-preventable diseases (LRTI, pneumococcal meningitis, pertussis, diphtheria, tetanus, HiB meningitis, measles, and tuberculosis) from IHME data. We adjust this initial estimate as follows:

• We remove deaths that occur before vaccines are administered from the data (for example, we estimate that 14% of under-5 measles deaths occur before the age of 9 months, the age at which the measles vaccine is provided, so we apply an adjustment factor of 0.86).36
• We adjust the probability of death from specific diseases to account for the etiological fraction of different pathogens — i.e., the share of LRTI deaths due to S. pneumoniae and HiB and the share of diarrhea deaths due to rotavirus. We've decided not to use IHME's vaccine etiology data, but we are highly uncertain about the appropriate adjustments to use. (See the cell notes here for further discussion.)
• We exclude child mortality from hepatitis B and yellow fever, since IHME data show a very low probability of death for children from these diseases in Nigeria, and polio, which we also expect to have a very low probability of child mortality in Nigeria today.37

Summing across diseases, with these adjustments incorporated, we find a probability of death for children under 5 in Nigeria from vaccine-preventable diseases of 1.65%.38

Mortality rates in North West Nigeria vs. Nigeria as a whole

We estimate that the probability of death for children under five due to vaccine-preventable disease is 19% higher in North West Nigeria, where New Incentives operates, than in Nigeria as a whole, due to factors such as overall health, prevalence of disease-causing pathogens, and access to healthcare.

To arrive at this estimate, we compared all-cause under-five mortality, diarrhea mortality, and lower respiratory tract infection (LRTI) mortality (accounting for differences in vaccination rates) in the three North West Nigerian states where New Incentives operates to rates in Nigeria as a whole:

• All-cause under-five mortality. We find all-cause under-5 mortality is roughly 50% higher in North West Nigeria than in Nigeria as a whole.39 All-cause under-5 mortality differences include both the effect of higher vaccination rates in Nigeria as a whole than in North West Nigeria, as well as any other effects (e.g., overall health and access to healthcare, which affects susceptibility to death from infection by vaccine-preventable disease). We view this as an upper bound on the difference between the probability of death for children under five due to vaccine-preventable disease in North West Nigeria vs. Nigeria as a whole.
• Diarrhea mortality. We find diarrhea mortality is roughly 35% higher in North West Nigeria than in Nigeria as a whole.40 The rotavirus vaccination (which protects against some forms of diarrhea) is not yet available in North West Nigeria, though we expect it to become available starting next year.
• LRTI mortality. We find that LRTI mortality, accounting for higher vaccination rates in Nigeria as a whole vs. North West Nigeria, is roughly 3% higher in North West Nigeria than in Nigeria as a whole.41 This is particularly relevant because deaths from LRTI prevented by vaccines account for the plurality of the effect of New Incentives' program.

We average the differences in mortality rates for diarrhea and LRTI to generate an estimate of overall difference in probability of death from vaccine-preventable disease in North West Nigeria vs. Nigeria as a whole of 19%, which is consistent with our treatment of the 50% difference in all-cause mortality as an upper bound (see here for calculations).

Incorporating this 19% adjustment, we find that the probability of death under five years of age due to vaccine-preventable diseases is 2.0%.42

Adjustment for proportion of vaccinated children in IHME data

Baseline mortality rates from IHME necessarily include some vaccinated and some unvaccinated infants. We adjust baseline mortality rates to reflect mortality among only unvaccinated infants as follows:

• We estimate that vaccination coverage in Nigeria was 50% across vaccines over the period from 2013 through 2019. This is based on coverage of the BCG, DTP, measles, HiB, PCV, and rotavirus vaccines from UNICEF/WHO data.43
• We adjust these coverage estimates to account for partial vaccination (i.e., children receiving the first dose but not all three doses of PCV).44

Applying the formula in this footnote,45 we estimate the probability of death in North West Nigeria from vaccine-preventable disease for unvaccinated children is 3.0%.46 We have not reviewed the calculations behind UNICEF/WHO data, and it's possible that we would refine these estimates after further investigation.47

Remaining uncertainties

We have fairly high confidence that vaccine-preventable diseases are a significant cause of mortality in North West Nigeria. However, we have some uncertainties about the exact magnitudes, given uncertainties about some of our assumptions:

• Concerns about cause of death data. Our calculations rely heavily on cause-of-death data, and we're highly uncertain about how accurate these data are at correctly attributing death to a particular disease.48 Our calculations also rely on etiological data to estimate, for example, the share of diarrheal disease deaths caused by rotavirus and the share of deaths from LRTI or meningitis caused by S. pneumoniae and HiB. We have not vetted the methods used to produce cause-of-death or etiological data. In particular, we're highly uncertain how much weight to put on the findings of the PERCH study, which finds much lower etiological fractions of pneumonia due to bacteria like S. pneumoniae and HiB.49
• Subnational adjustments for North West Nigeria. We're uncertain how well cause-of-death data describe the three states in North West Nigeria where New Incentives operates. It's possible the prevalence rates of specific diseases or etiologies are significantly different from national figures. It's also possible the overall child mortality rate is much different than we've estimated, including for diseases like hepatitis B, polio, and yellow fever, which we've estimated have a very low probability of death for children in settings where New Incentives operates.
• Baseline vaccination coverage calculations. We haven't reviewed the calculations underlying the estimates of vaccination rates in Nigeria and have not made adjustments to account for, for example, self-reporting bias in these data. If the coverage figure we use is an overestimate, then the probability of death for unvaccinated individuals will be lower than we've estimated.

Does increased vaccination lead to reductions in disease?

The extent to which New Incentives' program leads to reductions in mortality depends on the extent to which the vaccines incentivized by New Incentives lead to reductions in the diseases those vaccines target. We estimate that the vaccines incentivized (either directly or indirectly) by New Incentives reduce recipients' likelihood of contracting the diseases targeted by 76%.

This estimate is primarily based on meta-analyses of the effects of vaccines incentivized by New Incentives. A weighted average of vaccines' efficacy against targeted diseases and those diseases' contribution to child mortality in Nigeria yields an overall efficacy of 67%. (More)

We apply adjustments to this initial estimate to account for:

• All-cause mortality effects. We apply a 33% upward adjustment to account for observed "all-cause" mortality effects — i.e., it appears that vaccines have mortality benefits beyond their impacts on the targeted diseases. We believe the impacts on all-cause mortality are likely due to (a) "non-specific" effects of vaccines — i.e., vaccines may make children healthier and less susceptible to other infections or death due to diseases other than those targeted by the vaccines themselves — and/or (b) the proportion of deaths caused by vaccine-preventable diseases being higher than estimated. However, for simplicity, we've modeled these benefits as an increase in the efficacy of vaccines against vaccine-preventable disease. This adjustment corresponds to 0.5 deaths indirectly averted for each death directly averted. (More)
• Concerns about vaccine efficacy in low-income countries. We adjust estimates downward by 19% to account for concerns about vaccine efficacy in low-income countries and Nigeria in particular. (More)
• Imperfect coverage. We include a 95% coverage adjustment to account for less than 100% coverage of all doses of vaccines in trials. This is a rough guess.

In aggregate, these adjustments imply a vaccine efficacy of 76%.50

We have a moderate level of confidence in these estimates. Some of our concerns are:

• We have not thoroughly vetted the meta-analyses we use to quantify vaccine efficacy, and it's possible that further review would lead us to make adjustments to these values.
• We are highly uncertain whether the sizes of our adjustments for concerns about vaccine coverage, vaccine efficacy, and non-specific effects are appropriate.

Meta-analyses of vaccines' effects on disease for children under five

The vaccines we have included in our cost-effectiveness analysis, along with their target diseases, doses, and efficacy against targeted diseases are summarized in the following table.51

Vaccine	Diseases targeted	Doses	Vaccine efficacy52
PCV	Protects against pneumococcal bacteria.53 Our CEA models effects on (i) LRTI associated with S. pneumoniae and (ii) pneumococcal meningitis.	3 doses: at 6 weeks, 10 weeks, and 14 weeks. Co-administered with PENTA.54	0.58 (95% CI 0.29-0.75) for all serotypes-IPD (invasive pneumococcal disease)55
DTP	DTP is a combination vaccine used against diphtheria, tetanus, and pertussis (whooping cough). It is also known as DTaP or DTwP.56 Our CEA models effects on diphtheria, tetanus, and pertussis.	3 doses: at 6 weeks, 10 weeks, and 14 weeks. Part of PENTA.57	0.84 (95% CI 0.81-0.87) for acellular vaccines and 0.94 (95% CI 0.88-0.97) for whole-cell vaccines for clinical case definition of pertussis, based on combinations of vaccines containing pertussis vaccines.58 We focus on the effect on pertussis, since it accounts for more than 80% of probability of death across diphtheria, tetanus, and pertussis.59
HiB	Protects against Haemophilus influenzae type b (HiB), a bacteria responsible for severe pneumonia, meningitis, and other invasive diseases almost exclusively in children aged less than 5 years.60 Our CEA models effects on (i) HiB-caused LRTI and (ii) HiB-caused meningitis.	3 doses: at 6 weeks, 10 weeks and 14 weeks. Part of PENTA.61	0.82 (95% CI 0.73-0.87) for invasive HiB disease.62
Measles	Measles	1 dose at 9 months.63	0.85 (95% CI 0.83-0.87) for measles.64
BCG	Tuberculosis65	1 dose (at birth, or as close as possible)66	0.85 (95% CI 0.69-0.92) for meningeal and/or miliary tuberculosis,67 which are the life-threatening forms of tuberculosis.68
Rotavirus	Protects against rotavirus, which is one cause of diarrhea.69	Indirectly incentivized. 2 doses (6 weeks and 10 weeks), co-administered with the first 2 doses of PENTA and PCV70	0.46 (95% CI 0.29-0.59) for severe rotavirus diarrhea in sub-Saharan Africa.71

For our estimates of vaccine efficacy, we rely on meta-analyses from the Lives Saved Tool (LiST), except for BCG (which is not included in LiST) and HiB. For BCG, we use a meta-analysis by Mangtani et al. 2014. For HiB, we use the meta-analysis by Thumburu et al. 2015, which provides an intention-to-treat (ITT) analysis rather than a per-protocol analysis.72

Overall, vaccine efficacy in these meta-analyses ranges from 46% (for rotavirus) to 85% (for measles and BCG).73 For our cost-effectiveness model, we estimate an overall vaccine efficacy of 66%, produced by taking a weighted average of vaccine efficacies, weighted based on the estimated share of under-five mortality in Nigeria accounted for by the disease targeted by a given vaccine.74

We largely take these vaccine efficacies at face value and have not thoroughly vetted the meta-analyses or individual studies included in them. Our key remaining uncertainties related to these meta-analyses include:

• Internal validity. It is possible we should include downward adjustments to account for internal validity concerns about the effect sizes reported in the meta-analyses. We have not currently done so, because the effect sizes are almost all based on RCTs, and our impression is that the efficacy of vaccines is well-established. The effect size for measles is based on a combination of RCTs and observational studies;75 there is also other evidence consistent with the effect size reported from serological studies of measles.76 Our impression is also that measles has been reduced in many countries that have had good vaccination coverage.77
• External validity. Our impression is that the meta-analyses above include a combination of results from low- and non-low-income countries, and there may be reasons to expect smaller effects in low-income countries. It might be appropriate for us to weight studies in low-income countries more heavily, since those settings likely resemble the setting of New Incentives' program more closely. Additionally, many of the studies referenced above were conducted decades ago, and it's possible that the results' validity may not extend to current settings. We discuss how we account for external validity below.

Our other remaining questions include:

• How closely do recipients' age, dosage, and other features of the interventions included in the meta-analyses match those of New Incentives' program?
• Do the effect sizes incorporate practical challenges in implementing vaccine programs in the field (e.g., maintaining the cold chain)?
• How would incorporating any information on coverage of vaccines in ITT estimates influence vaccine efficacy? We include a 95% coverage adjustment in the CEA,78 but this is a rough guess.79
• How does efficacy vary with vaccine "type"? Our understanding is that vaccines can vary along various dimensions (e.g., "valence" for PCV, "acellular" or "whole-cell" for pertussis) and that efficacy may vary with these characteristics.
• Are there interactions between vaccines that would shift our cost-effectiveness estimate?
• How important is serotype replacement?80

It's possible that further investigation of these questions would lead us to adjust our best guess on vaccine efficacy.

Vaccine efficacy in Nigeria

The meta-analyses above include studies from both low-income and non-low-income countries, and we're uncertain about the extent to which effects vary across settings. It's possible that vaccine efficacy may be lower in low-income countries. This may be due to issues with maintaining the cold chain, which might cause vaccines to be inactive by the time they're received by children, or other problems preventing immunity from occurring.

To estimate the size of this effect, we've mainly considered:

• Results from biomarkers studies of measles immunity in Nigeria (More)
• Seroconversion studies of measles immunity in Nigeria (More)
• Meta-analyses of vaccine effectiveness in low-income settings specifically (More)

Overall, these factors lead us to make an adjustment of 0.81 (i.e., a reduction of 19%) to the vaccine efficacies from the meta-analyses described above. This adjustment reduces our estimate of vaccine efficacy across vaccines from 66%, based on meta-analyses, to 54%.81 For measles alone, this implies an efficacy of 69% (down from 85% reported in the meta-analysis we use), which is roughly in line with seroconversion studies and a meta-analysis (Uzicanin and Zimmerman 2011) based on studies specifically from Africa.82 However, we are highly uncertain about this adjustment. We view the biomarkers results in particular as somewhat concerning, and we are continuing to investigate this issue. We may decide to conduct follow-up research to understand biomarkers for infants vaccinated through New Incentives' program. We also have not deeply reviewed seroconversion studies or meta-analyses that separate effects by low-income vs. non-low-income countries, and we have not fully explored all potential explanations for low vaccine efficacy.

Biomarkers studies in Nigeria

IDinsight conducted two operational pilots to test the feasibility of using oral fluid biomarkers to validate caregiver-reported vaccination status. Both of these pilots found low agreement between the oral fluid test and caregiver-reported vaccination status, as well as administrative records (e.g., child health cards).

We view these biomarkers results as a negative update on measles vaccine efficacy in our cost-effectiveness analysis. We are highly uncertain how much to adjust based on these results, and our best guess is that the appropriate discount factor is between 0.3 and 1 (i.e., a 0% to 70% reduction in the overall effect size of New Incentives' program). This range corresponds to the following scenarios:

• 0.3 (i.e., 70% reduction in efficacy): This adjustment assumes that the biomarkers result is explained entirely by low vaccine efficacy and that we should expect the drivers of low vaccine efficacy to be present in New Incentives' implementation for all vaccines.
• 1 (i.e., 0% reduction in effect): This adjustment assumes that the biomarkers result is explained solely by testing inaccuracy or that we expect any drivers of low vaccine efficacy to be resolved in New Incentives' implementation for all vaccines.

We're not sure what's driving the poor biomarkers results. Experts that GiveWell and IDinsight spoke to point to several possible explanations.83 One commonly suggested explanation was that low vaccine efficacy can occur if the cold chain is not maintained. We are unsure whether this applies only to the measles vaccine or to all vaccines.

To produce an adjustment to account for these biomarkers results, we "pool" the following factors:

1. Our best guesses as to what is driving biomarkers results for the measles vaccine and how this should affect our expectations across vaccines — in particular, whether the results are driven by either (i) low vaccine efficacy (and we should expect the drivers of low vaccine efficacy to be present in New Incentives' implementation for all vaccines), or (ii) testing inaccuracy (or that we should expect whatever is driving low vaccine efficacy to be resolved in New Incentives' implementation for all vaccines).84
2. External studies of measles vaccine efficacy (including measles meta-analyses and biomarkers studies using blood tests).85
3. Our guess about the extent to which poor measles vaccine efficacy would imply problems with other vaccines.86

See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for lower vaccine efficacy in Nigeria" subsection. These guesses and probability weights are very rough and should be viewed as a first pass.

Seroconversion studies

Some older studies from Nigeria (though not North West Nigeria) find somewhat low levels of antibodies following measles vaccination and suggest a vaccine efficacy of approximately 70%:

• In a sample of 286 children in Ilorin, Nigeria, Fowotade et al. 2015 find 68.6% of infants developed protective antibody titres.87 The authors attribute this to low vaccine potency and find that three out of six vaccine vials tested had virus titers above WHO-recommended cutoffs.88 They attribute this to lapses in the cold chain, though they do not provide any formal tests or analysis to support this.89 We find this somewhat surprising, since the authors also note that the vaccines were maintained in cold chains after they received them from the State EPI Unit.90
• We superficially reviewed Adu et al. 1992 and Omilabu et al. 1999. Both of these studies also find lower than expected antibody levels,91 attributed to low vaccine efficacy,92 and speculate that lapses in the cold chain are the cause (though they do not formally test this explanation).93

Meta-analyses of vaccines' effectiveness in low-income countries

Our impression, based on a shallow review of meta-analyses for measles, PCV, and BCG, is that vaccine efficacy may be lower in low-income countries:

• A different meta-analysis of measles vaccines than that used by LiST (which we rely on for our main analysis of vaccine efficacy) suggests lower efficacy against measles in low-income settings.94
• A meta-analysis of the effects of the BCG vaccine notes that the BCG vaccine is less effective closer to the equator.95 It is not clear to us whether this result applies to BCG vaccine administered at birth and to the BCG vaccine's effects on fatal forms of tuberculosis.96
• The two African trials in a meta-analysis of the effects of PCV have smaller effects than trials in other settings, though the difference does not appear to be statistically significant, and it is not clear what is causing this difference.97

All-cause mortality effects

We originally modeled the effect of vaccines by multiplying the probability of death from vaccine-preventable diseases for unvaccinated children (see here) by the reduction in that disease from meta-analyses of vaccine efficacy (see here), which suggests an all-cause mortality relative risk across all vaccines incentivized by New Incentives of roughly 0.88.98

However, two meta-analyses we looked at — Higgins et al. 2016 and Lucero et al. 2009 — find all-cause mortality effects from vaccines that are substantially larger than this estimate.99

In our CEA, we estimate that measles, tuberculosis, and lower respiratory infection or meningitis caused by S. pneumoniae each account for less than 10% of child deaths in Nigeria.100 However, these diseases would need to account for 31%, 35%, and 19% of child deaths, respectively, in order for vaccinations against those diseases to lead to the all-cause mortality effects observed in the above meta-analyses.101 Combining the all-cause mortality effects of the measles vaccine, BCG vaccine, and PCV from Higgins et al. 2016 and Lucero et al. 2009 suggests an overall all-cause mortality relative risk of 0.46102 across those diseases; this would be even lower if we incorporated other vaccines. A rough guess is that, overall, these meta-analyses suggest an all-cause mortality relative risk from vaccination of roughly 0.4.

Overall, this leads us to adjust our estimate of the effect size of vaccines upward. Our best guess is that vaccination prevents 0.5 deaths indirectly (i.e., via indirect effects on non-directly vaccine-preventable diseases) for every death prevented directly (i.e., for directly vaccine-preventable diseases). We model this as a 33% increase in vaccine efficacy. However, we have very limited data with which to estimate the magnitude of this effect, and our current approach is a rough first attempt. We believe it is highly likely that we would update this parameter based on further investigation.103

Explanations for this observation that would lead us to put more weight on our estimates based on current disease prevalence and less weight on the all-cause mortality estimates include:

• All-cause mortality effect point estimates may be incorrect. The 95% confidence intervals in Higgins et al. 2016 and Lucero et al. 2009 include a risk reduction on all-cause mortality of 0%. This would mean we should put more weight on our estimates based on current disease prevalence.
• The proportion of child deaths caused by vaccine-preventable disease may have changed over time. When these studies were conducted, measles, tuberculosis, and pneumonia due to S. pneumoniae may have been responsible for a larger share of child deaths than they currently are. This would mean we should give more weight to our estimates based on current disease prevalence.
• Overall health may have improved over time. Access to healthcare, nutritional status, and access to prevention may be greater now than when the all-cause mortality trials were conducted, leading to smaller non-specific effects.
• Simply adding all-cause mortality effects likely "double counts" non-specific effects. For instance, presumably, some of the non-specific effects of BCG on all-cause mortality are due to BCG reducing risk of death from measles and pneumonia, so adding the effects of measles vaccine and PCV to that impact would count those beneficial effects twice.
• Characteristics of infants in trials may not be representative. Two of the BCG trials were limited to a sub-sample of low-birthweight infants, for whom we would guess this effect is larger than for the average infant in North West Nigeria today.
• There is high uncertainty about the mechanisms for non-specific effects. Our impression is that the mechanisms driving non-specific effects are poorly understood, which leads us to discount them slightly.
• There is evidence of a negative effect of DTP on mortality. The negative effect of DTP on mortality suggests that we ought to downweight non-specific effects, though this is based on observational studies with high risk of bias.104

Explanations that would lead us to put less weight on our estimates based on current disease prevalence and more weight on all-cause mortality estimates include:

• IHME disease prevalence and etiology data may be incorrect. It's possible that the IHME data underestimate the proportion of deaths that are due to vaccine-preventable diseases.
• "Non-specific" effects of vaccines. Vaccines may have effects on diseases other than the ones they target by generally strengthening infants' health — i.e., by preventing infants from contracting an illness that would have increased their likelihood of dying from some other cause. Being vaccinated for measles, for example, may lower the probability of death from malaria or other infections.

Our best guess is that vaccines avert 0.5 deaths indirectly (i.e., deaths due to non directly vaccine-preventable diseases) for each vaccine-preventable disease death averted, which is in line with our CEA for seasonal malaria chemoprevention (SMC).105

There are two main lines of reasoning behind this estimate.

The first is based on benchmarking against SMC:

• Our SMC CEA also makes an adjustment for non-specific effects of preventing malaria.106
• We would guess that non-specific effects would be similar for vaccines because the age profile of the beneficiaries is the same (children under 5) and the mechanism seems similar (i.e., beneficial health effects reduce the likelihood of death from other causes).
• We'd guess these effects might be somewhat lower for vaccines, since all of these vaccines together are preventing a slightly larger share of overall deaths than SMC (and so there are fewer deaths "available" to prevent indirectly). However, vaccines also prevent a wider range of diseases. As a result, an adjustment of 0.5 indirect deaths averted per vaccine-preventable disease death averted seems appropriate to us.

The second is based on putting some weight on the 0.4 relative risk implied by all-cause mortality trials from Higgins et al. 2016 and Lucero et al. 2009 and some on the 0.88 relative risk implied by our initial approach:

• We put some weight on the 0.4 relative risk from all-cause mortality trials from Higgins et al. 2016 and Lucero et al. 2009 (discussed above), since (a) we think it's plausible there are non-specific vaccine effects, and (b) we have high uncertainty about the IHME data on cause of death.
• However, we put more weight on the 0.88 estimate from our original analysis,107 given the wide confidence interval on the all-cause mortality effect and concerns that (a) the proportion of child deaths caused by vaccine-preventable disease may have changed over time, (b) overall health may have improved over time since the all-cause mortality studies were conducted, and (c) simply adding all-cause mortality effects would likely "double count" non-specific effects.
• We haven't assigned specific weights to these estimates but think it's reasonable to expect all-cause mortality effects to be higher than what we estimate based solely on disease prevalence and vaccine efficacy against those diseases but likely significantly less than implied by taking all-cause mortality point estimates at face value.

Remaining uncertainties

We have a moderate level of confidence in the above estimates. Our key areas of uncertainty are:

• How much should we discount for uncertainties about vaccine efficacy in Nigeria and low-income countries more broadly (for instance, the findings of the biomarkers study discussed above)? We currently apply a downward adjustment of approximately 20% for these uncertainties.
• How accurate are the adjustments made in our analysis for self-reporting bias, vaccines with multiple doses, and all-cause mortality effects? The values we have used for these parameters are speculative and could meaningfully affect our cost-effectiveness estimate if altered.
• How reliable are cause of death data, especially for deaths due to specific pathogens (e.g., pneumonia caused by S. pneumoniae)?
• We've chosen to use a relatively simple cost-effectiveness model108 of the benefits of increased vaccination compared to some other models used in the literature. It is possible our model leaves out key features or additional positive or negative effects that would substantially affect our cost-effectiveness estimate (e.g., disease transmission dynamics, herd immunity). We incorporate some such factors in the "Adjustments for effects excluded from our core model" section of our cost-effectiveness analysis, but only as very rough guesses.109
• How should we interpret the unexpected increase in control vaccination coverage observed from baseline to endline (see above) in the RCT? If this increase is driven by other vaccination promotion programs, those programs might "crowd out" New Incentives in the future.
• How does New Incentives' program affect caregivers outside of the catchment of treatment clinics?
• How would New Incentives' impact change if it were to scale up?

What are the benefits beyond child mortality?

In addition to effects on child mortality, we also model benefits of New Incentives' program occurring through:

• Reductions in mortality from vaccine-preventable disease in vaccinated people after the age of five. (More)
• "Development benefits," i.e., increases in income later in life due to early-life health improvements. (More)
• Consumption benefits from the cash transfers themselves. (More)

Effects on deaths above age five

Risk of vaccine-preventable diseases also occurs at ages beyond five. Vaccines may confer immunity beyond the age of five, reducing risk of mortality as vaccinated infants age.

We think it is reasonable to expect that vaccines would lead to reductions in mortality from vaccine-preventable disease at ages above five, but because most of the burden of vaccine-preventable disease above the age of five occurs at ages 50 or older, we have higher uncertainty about the size of this effect, given (a) the lack of long-term data on vaccine effectiveness and (b) uncertainty about what the prevalence of vaccine-preventable diseases will be when infants who are vaccinated now reach age 50. Consequently, we apply a large discount to these effects.

The probability of death from vaccine-preventable diseases (unadjusted for vaccination) in Nigeria is 0.11% from age 5-14, 0.89% from age 15-49 and 4.08% from age 50-74, compared to 1.65% for ages 0-5.110 This would suggest that roughly 75% of the benefits of vaccination accrue to individuals older than 5, with most of those (roughly 60%) occurring after the age of 50.111

We discount impacts on mortality occurring at older ages for the following reasons:

• We expect health to improve over time, suggesting mortality rates from these diseases will likely be lower when vaccinated infants reach these age groups.
• Vaccination rates in infancy were likely lower for individuals who are now in older age groups, so the probability of death for unvaccinated individuals is probably a lower share than what would be implied by directly comparing the figures above.
• Vaccine efficacy may diminish over time for some vaccines. The strongest evidence of this effect we found was for BCG, which protects against tuberculosis, the largest driver of mortality in older age groups (15-49 and 50-74) among the vaccine-preventable diseases included in our model.
• We discount to account for the possibility that fundamental changes occur that render the program ineffective in the future.

See the most recent version of our cost-effectiveness model, "New Incentives" sheet, "Mortality reduction for individuals older than 5" section for our calculations.

Development benefits

We would guess that vaccination in infancy may have "development benefits" — i.e., early-life health improvements from vaccines might lead to improvements in income when vaccinated children become adults. While we have not reviewed direct evidence for development benefits from vaccination in infancy, we have included these benefits to be consistent with our cost-effectiveness evaluations of other interventions that improve health early in life. See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Development benefits" section for our calculations.

Consumption effects

Cash transfers received by caregivers who vaccinate their infants may also improve well-being through increased consumption. We have not observed direct consumption effects of cash transfers; instead, we model these based on the size of the cash incentives provided by New Incentives and estimates of average household income among recipients. We haven't deeply vetted these estimates, since we think they're unlikely to substantially affect our cost-effectiveness estimates.

See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Consumption benefits" section for our calculations.

Additional benefits and negative or offsetting impacts

There are several benefits, as well as negative or offsetting impacts, of New Incentives' program that we have not modeled explicitly in the main CEA but that we do incorporate into our final cost-effectiveness estimate as rough guesses. Rather than explicitly modeling these, we have applied percentage adjustments based on our best guesses.112

Additional benefits

There are several potential benefits of vaccines that we've excluded from our main cost-effectiveness analysis. These are included in the "Adjustments for excluded effects" section of the cost-effectiveness analysis:113

• Lower likelihood of recipients infecting others. People who are vaccinated may be less likely to spread infection to others, creating a positive spillover effect. The size of this effect will depend on (1) how much vaccination reduces the recipient's likelihood of acquiring the disease and spreading it (rather than merely reducing their likelihood of dying from the disease themselves), and (2) the proportion of the population that is unvaccinated (the higher this proportion, the greater the benefit would be). This effect is included in more complicated models of vaccine efficacy, and it is possible that ignoring this effect leads us to underestimate vaccine efficacy, especially for diseases with high transmissibility (e.g., measles). We have not yet sought to model these effects, since doing so would add a substantial degree of complexity to our model, but we believe this factor may be worth further investigation, since it is a key component of vaccines' impact.114
• Herd immunity. High overall vaccination levels in treatment groups (close to 80%) suggest that it is possible that New Incentives' program could lead at least some areas to achieve herd immunity as the program rolls out over a three-year period or in specific areas where total vaccination rates have moved above 85%-90%. We have not yet sought to model these effects, since doing so would add a substantial degree of complexity to our model, but we believe this factor may be worth further investigation, since it is a key component of vaccines' impact.
• Benefits of additional indirectly incentivized vaccines beyond rotavirus. We've excluded any effects of vaccination on polio, yellow fever, hepatitis B, and meningitis A. It is possible that incorporating these could lead to additional benefits, though our impression is that these benefits are likely to be small.115
• Morbidity effects. We have not explicitly modeled impacts on morbidity from vaccine-preventable diseases.116
• Reduced outbreak risk. Vaccines could reduce the risk of disease outbreaks (e.g., measles), which could increase cost-effectiveness.
• COVID-19 may make New Incentives' program more effective. It seems plausible that COVID-19 could increase the effectiveness of cash transfer programs at increasing vaccination (e.g., because COVID-19 has reduced incomes or made it more costly for caregivers to bring their infants in for immunizations), though this is speculative.
• Treatment costs/economic losses averted from prevention. By preventing diseases, vaccines may also prevent caregivers from incurring costs of procuring treatment for these diseases (e.g., lost wages, transportation costs, out-of-pocket medical expenditures).
• Improved timeliness of vaccination. New Incentives may also make it more likely that vaccines are provided at the correct ages, which may increase the number of deaths they avert.117
• Increased clinic utilization. New Incentives may encourage more clinic visits, which could have spillover effects (e.g., improved uptake of other interventions). IDinsight did not find large effects on clinic utilization, so we have set this effect to zero.118
• Investment of income increases. We have included this factor to be consistent with our CEAs for other interventions that increase consumption.

Potential negative or offsetting impacts

There are several potential negative or offsetting impacts of vaccines that we've excluded from our main cost-effectiveness analysis. These are included in the "Adjustments for excluded effects" section of the cost-effectiveness analysis:119

• Crowding out of New Incentives. High levels of vaccination at the endline of the IDinsight RCT raise questions about whether New Incentives' program could be crowded out by other vaccine promotion efforts that push vaccination rates close to 100%. This possibility is partially accounted for by our leveraging and funging adjustment (see below), but if there are other programs that would be likely to cause vaccination rates to increase over the next year, this could diminish the impact of New Incentives.
• COVID-19 may reduce New Incentives' effectiveness. It seems possible to us that cash incentives might be less effective during the COVID-19 pandemic, if (for example) healthcare staff capacity is diverted from vaccinations, caregivers are less likely to respond to cash incentives due to concerns about infection, or vaccine supply chains are disrupted. It's also possible that the risk (to caregivers or infants) of COVID-19 exposure from traveling to clinics to receive vaccinations could offset the mortality benefits of vaccination to some extent.
• Vaccine-derived polio outbreaks. The oral polio vaccine may cause outbreaks of polio and lead to paralysis.120
• Inflation. If inflation occurs, New Incentives may need to raise the amount of its cash incentives to maintain the same effect size.
• Serotype replacement. Vaccination may lead to higher prevalence of non-vaccinated serotypes of the diseases targeted, which could offset some of the beneficial effects of increased vaccination.

Additionally, a meta-analysis by Higgins et al. 2016 finds the DTP vaccine increases all-cause mortality by 38%, though this is from observational studies that the authors rate as having a high risk of bias.121 We do not include this factor in the "Adjustments for excluded effects” sheet, since we incorporate it into our consideration of all-cause mortality effects (see above).

Downside adjustments

We also adjust for risk of wastage, the quality of monitoring and evaluation, and our level of confidence in funds being used for their intended purpose. On net, we guess these reduce the cost-effectiveness of New Incentives' program by 7%.122

How cost-effective is it?

Our cost-effectiveness analysis for New Incentives incorporate our estimates of the costs of the program to New Incentives and other actors and our estimates of the program's impact on child mortality and other benefits described above.

For information on the costs of New Incentives' program, see our New Incentives charity page. For full details on our cost-effectiveness estimates for New Incentives' program, see the most recent version of our cost-effectiveness analysis, "New Incentives" sheet.

Note that there are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Leveraging and funging

New Incentives' spending may lead other organizations or governments to spend more ("leverage") or less ("funging") on programs implemented by our top charities than they otherwise would have. For a full introduction to our approach to leverage and funging adjustments, see this blog post.

Our cost-effectiveness analysis for New Incentives includes adjustments for leverage and funging.123 These adjustments incorporate our estimates of the value of counterfactual uses of spending by Gavi and the Nigerian government and our estimate of the likelihood that the Nigerian government would fund a similar program in New Incentives' absence.124

Our key remaining uncertainties about this approach are:

• What are the appropriate values for counterfactual uses of funding by the Nigerian government and Gavi? We've chosen our current estimates in part to maintain consistency with our other CEAs.
• What is the likelihood that the Nigerian government would have funded a different program like New Incentives' in the absence of New Incentives itself? We currently assume a probability of 10%, but this is speculative and uncertain. If this probability were higher, it would decrease our estimate of New Incentives' cost-effectiveness.

Our process

Since our cost-effectiveness calculations form an important part of our decision-making process when considering whether to recommend an organization as a top charity, several research staff members reviewed and provided feedback on a draft of the model. Their written reviews are much more informal than the content we typically publish, but we are sharing them below to give a window into an important part of our research process.

See the following links for informal reviews of our cost-effectiveness analysis by:

• Stephan Guyenet
• Andrew Martin
• James Snowden
• Caitlin McGugan
• Anonymous

This work represents a more in-depth review process than what we have generally used in the past. We intend to use a similar approach going forward, spending proportionally more research time vetting conclusions that (a) are more likely to impact our funding recommendations and (b) are more likely to change as a result of being reviewed by additional researchers.

Other

Research ethics inquiry into IDinsight paper on operational pilot study to determine optimal incentive sizes for measles vaccinations

On June 29, 2020, IDinsight informed us of concerns regarding its paper on a pilot study it conducted of New Incentives' immunization program. The pilot study, also supported by GiveWell Incubation Grant funding, is separate from the main RCT discussed here. The inquiry into the paper is limited to IDinsight's practices and to the pilot study. We take these concerns about IDinsight's research ethics seriously, but we do not believe they should impact our assessment of New Incentives as an organization. Additional details are in this footnote.125

Next steps

We may update our assessment, based on the following information:

• Changes in baseline vaccination rates in Nigeria. In particular, this could update our "crowd out" adjustment in the "Adjustments for excluded effects" section of our CEA.126
• Follow-up research on measles biomarkers. This could update our "adjustment for vaccine efficacy in Nigeria."127
• Changes in the program's effectiveness with scale-up.

Sources

Document	Source
3ie, Quality Assurance of IDinsight's Evaluation of New Incentives	Source
Adu et al. 1992	Source (archive)
Beaubien, "Polio is making a comeback," 2019	Source (archive)
Cutts et al. 2005	Source
Dhanawade et al. 2015	Source (archive)
Feikin et al. 2016, Annex 10A	Source (archive)
Fowotade et al. 2015	Source
Fulton et al. 2016	Source
GiveWell, "2021 GiveWell cost-effectiveness analysis — version 3"	Source
GiveWell, "New Incentives CEA supplemental information [June 2021] (public)"	Source
Higgins et al. 2016	Source (archive)
IDinsight, Impact Evaluation of New Incentives, Final Report	Source
IDinsight, New Incentives Evaluation Baseline Report, 2019	Source
IDinsight, New Incentives Evaluation, Pre-Analysis Plan	Source (archive)
IHME, GBD Results Tool	Source (archive)
Ismail et al. 2020	Source (archive)
Klugman et al. 2003	Source
Lamberti et al. 2016	Source
Lucero et al. 2009	Source
Mangtani et al. 2014	Source
New Incentives, Responses to 13-Feb-2020 Questions from GiveWell	Source
Nymark et al. 2017	Source (archive)
Omilabu et al. 1999	Source (archive)
PERCH 2019	Source (archive)
Pietrantonj et al. 2020	Source
Pimpin et al. 2013	Source (archive)
Plotkin et al. 2018	Source
Polio Eradication Initiative, Nigeria country page	Source (archive)
Polio Eradication Initiative, Polio-Free Countries	Source (archive)
Sudfeld et al. 2010	Source (archive)
The Lives Saved Tool (LiST)	Source (archive)
Thumburu et al. 2015	Source
Uzicanin and Zimmerman 2011	Source (archive)
Walker and Black 2011	Source (archive)
Weinberger, Malley, and Lipsitch 2011	Source
WHO and UNICEF estimates of immunization coverage, Nigeria	Source (archive)
WHO, Biologicals, BCG (Tuberculosis)	Source (archive)
WHO, Immunization, Vaccines and Biologicals, Hib	Source (archive)
WHO, Immunological Basis for Immunization Series, Measles, 2020	Source (archive)
WHO, Information Sheet, Observed rate of vaccine reactions, DPT, 2014	Source (archive)
WHO, Poliomyelitis: Vaccine derived polio, 2017	Source (archive)
WHO, Summary of WHO Position Papers, Recommended Routine Immunizations for Children, 2020	Source (archive)

• 1

  1,000 (people in cohort) x 0.03 (probability of death without vaccination) x 0.75 (reduction in risk of vaccine-preventable disease) x 0.22 (percentage point increase in vaccine uptake) ≈ 4.9 deaths averted. See the list of factors in the main text following this footnote number.

• 2

  See IDinsight, Impact Evaluation of New Incentives, Final Report for the results of this RCT.

• 3

  See evidence from other studies here.

• 4

  IDinsight, Impact Evaluation of New Incentives, Final Report, Table 3, pg. 23.

• 5
  • IDinsight, Impact Evaluation of New Incentives, Final Report, Table 3, pg. 23.
  • This is similar to effects found for any dose of PCV (22 percentage point increase, 95% CI 0.18-0.27, 50% control mean), which is administered at the same time as PENTA. See IDinsight, Impact Evaluation of New Incentives, Final Report, Table 6, pg. 27.
  • Our understanding based on viewing the survey questions (not published) is that "Penta 1" refers to receiving any dose of the Penta vaccine.

• 6

  IDinsight, Impact Evaluation of New Incentives, Final Report, Table 3, pg. 23.

• 7

  An alternative method would be to treat the treatment effects for each vaccine as separate estimates. One rationale for this would be that there might be reasons to expect effects of New Incentives to differ across vaccines. We decided to use an average for a few reasons:

  • It seems reasonable to view the observed effect sizes for different vaccines as "noisy" estimates of the same underlying effect.
  • Applying the same effect across vaccines simplifies our model, and we would guess that separating the effect by vaccine would lead to similar results.
  • In adjusting for self-report bias, we have used the observed effects on BCG scarring (see below). Translating this adjustment to other vaccines introduces additional complications.

• 8

  From the independent review by the International Initiative for Impact Evaluation (3ie): "Overall, I found that IDinsight was receptive to suggestions, made major adjustments between baseline and endline that substantially improved the design, was diligent in documenting the Stata code, and their work and assessment are a valid assessment of the New Incentives Conditional Cash Transfer program. While there are remaining questions about the change in coverage (greatly improved) in control areas, and the poor accuracy of the biomarkers and its implications for the efficacy of the measles vaccine in Nigeria, we do not believe that these questions change the interpretation and conclusions of the study. The New Incentives program substantially and significantly increased immunizations rates." 3ie, Quality Assurance of IDinsight's Evaluation of New Incentives, p. 3.

• 9

  IDinsight, Impact Evaluation of New Incentives, Final Report, Figures 16-18, pg. 65-66.

• 10

  “Programmatically, New Incentives’ theory of change continues to look promising. First, the sources of vaccinations measured align with the program. Outside of the national measles campaigns, almost all infants receive vaccinations from the sources New Incentives’ program will cover: health facilities and health facility outreach. Second, most caregivers cited lack of knowledge or ambivalence and relatively few caregivers cited socio-cultural reasons or mistrust and fear, as reasons for not vaccinating. It seems likely that an incentive, coupled with awareness raising activities, can overcome these stated reasons for not vaccinating. Finally, New Incentives’ program appears to be unique. While small incentives are relatively common, incentives worth more than 500 Naira are rare and only two caregivers received cash in our sample.” IDinsight, New Incentives Evaluation Baseline Report, 2019, p. 71.

• 11

  "Across the study area, coverage in the control group was substantially higher at endline than at baseline. This was true for each of the primary study vaccines, with the largest difference for the Measles vaccine: 17.8% of children at baseline versus 57.2% of children at endline had reportedly received Measles 1 vaccine (Table 16). We did not expect a change of this magnitude." IDinsight, Impact Evaluation of New Incentives, Final Report, p. 50.

• 12

  "Changes in the questionnaire or enumerator technique could have led to increased recording of vaccinations via the survey even if true vaccination coverage had not changed. This could result from either improved recall or increased social desirability, or both. This hypothesis could explain why we do not see similar increases in clinic tally sheets, which record relatively stable control-group vaccination volumes between baseline and endline (see Appendix I)." IDinsight, Impact Evaluation of New Incentives, Final Report, p. 51.

• 13

  "We looked at coverage according to each data source we had: self-reports, cards, CIRs [clinic immunization registers], and BCG vaccine scar (for BCG vaccine only). " IDinsight, Impact Evaluation of New Incentives, Final Report, p. 64

• 14

  See IDinsight, Impact Evaluation of New Incentives, Final Report, Table 6, pg. 27.

• 15

  We have not done a thorough review of the rate of scarring for newborns who receive BCG vaccine in settings like those where New Incentives operates. Our best guess of 90% is based on a superficial review of the literature.

  Dhanawade et al. 2015 report a scar rate of approximately 90%, based on a sample of Indian infants: "Sixty-four (91.4%) had a visible scar at 12 weeks post vaccination representing a scar failure rate of 8.6%."

  They note this rate is comparable to other studies in India but that studies in other countries have found varying scar rates: “In the present study, scar failure rate was 8.6%. This was comparable to other Indian studies on term infants by Rani (10%) and Lakhar (6.1%). Higher scar failure rate (55%) has been reported in low birth weight babies from India. A study from Pakistan reported scar failure rate of 19.6%, whereas the studies from LIMA Peru, Nigeria, and Brazil showed much lower scar failure rate of 1.4%, 3.7% and 3.1%, respectively.”

• 16

  We don’t have evidence concerning the drivers of higher-than-expected scar rates among infants enrolled in New Incentives. We believe this higher-than-expected rate of scarring in the treatment group may be due to some infants in the treatment group receiving BCG more than once, either due to errors (e.g., unclear child health cards leading clinic staff to believe that an infant had not yet received BCG when they in fact had) or repeat enrollment, though this is speculative (see more discussion on our top charity page for New Incentives).

• 17

  We think it is possible that enumerators misidentify scars that aren't present, though we expect this to be less common than enumerators failing to identify scars that are present.

• 18
  • For self-report data on the RCT's three primary outcomes, see IDinsight, Impact Evaluation of New Incentives, Final Report, Table 3, pg. 23.
  • For BCG scar data, see IDinsight, Impact Evaluation of New Incentives, Final Report, Table 6, pg. 27.
  • We define the rate in the treatment group as the rate in the control group plus the treatment effect, which is the treatment coefficient from a multivariate regression controlling for covariates.

• 19

  See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Increase in vaccination rates due to New Incentives" section, "Adjustment for self-reporting bias" row.

• 20
  • We've compared this approach to simulations provided by IDinsight that estimate the effect of New Incentives on BCG using IDinsight's staff's best guesses on specificity and sensitivity of BCG self-report in control and treatment groups.
    IDinsight staff generated their best guesses for ranges of BCG self-report sensitivity and specificity, based on comparing self-report data to CHCs, clinic records, and BCG scars. Based on these ranges, they estimate a range of effects on "true” BCG coverage from 12 to 41 percentage points. See this spreadsheet, cells R28:R29. Our best guess of 20 percentage points falls slightly below the middle of that range.

• 21Because BCG is first in the schedule and leaves a scar, we would guess recall would be better and, hence, sensitivity would be higher.
  
• 22IDinsight, Impact Evaluation of New Incentives, Final Report, Table 1, pg. 11.
  
• 23

  New Incentives told us that it expects the rotavirus vaccine to be introduced in the states where it works in 2022. New Incentives, conversation with GiveWell, April 13, 2021 (unpublished)

• 24IDinsight, New Incentives Evaluation Baseline Report, 2019, Table 1, p. 15.
  
• 25Cases like these wouldn't be included in the RCT's reported effect size, because children would have been counted as receiving "any dose" of the PENTA vaccine, regardless of whether they received one, two, or three doses.
• 26Cases like these wouldn't be included in the RCT's reported effect size, because these children would have been counted as receiving "any dose" of the PENTA vaccine in both the treatment and control groups.
• 27

  See this spreadsheet, section "3 doses," row "Adjustment relative to 'any' effect."

• 28

  See this spreadsheet, section "2 doses," row "Adjustment relative to 'any' effect."

• 29

  We take a weighted average across vaccines, weighting the increase in effect due to vaccines with multiple doses (17% for PCV, DTP, and HiB, and 9% for rotavirus) by the percentage of all vaccine-preventable disease targeted by the vaccine. See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Increase in vaccination due to New Incentives" section, "Overall adjustment for vaccines with multiple doses" row .

• 30

  "There is no substantial evidence of children in control areas receiving the incentive. We found a negligible number of control households that had either a CHC with a NI-ABAE stamp (n = 2, 0.1%) or the caregiver reported receiving cash incentives for vaccination (n = 9, 0.4%). We did not find any control caregivers with a NI-ABAE card. Accordingly, by these three measures, there is no evidence that a meaningful number of infants from control areas were enrolled in the program." IDinsight, Impact Evaluation of New Incentives, Final Report, p. 47.

• 31

  IDinsight, Impact Evaluation of New Incentives, Final Report, Table 3, pg. 23.

• 32

  "GiveWell believes that inaccuracy in self-reports of infant vaccination is likely to be the most important internal validity concern for this program. Inaccurate self-reports could ultimately lead to either upward or downward bias in RCT impact estimates, which GiveWell would like to account for in the internal validity adjustment for this program. While it is likely that self-reports are imperfect, on balance GiveWell has chosen self-reports of vaccination status as the primary outcome measure for this study because we believe that self-reports have fewer weaknesses than alternative impact measures. GiveWell will consider several pieces of evidence when evaluating the final accuracy of self-reported impact estimates. These additional validation exercises, while imperfect, are similar to those conducted in past work, and our impression is that past work did not find serious discrepancies between self-reports and other measures of vaccination status." GiveWell's Pre-Analysis Plan for the New Incentives RCT: Addendum on Internal and External Validity

• 33

  See this spreadsheet, "Probability of death" sheet, row "% of vaccinated infants in years 2021-2023 covered by rotavirus."
  

• 34

  We discuss the prevalence of vaccine-preventable diseases among people older than five below.

• 35

  See our summary of this data here.

• 36

  See more detail here.

• 37
  • See IHME data here.
  • The last reported case of wild polio virus in Nigeria was in 2016. Polio Eradication Initiative, Polio-Free Countries.
  • Cases of vaccine-derived polio have been reported in recent years in Nigeria, but we expect that these case numbers are small enough that the average probability of child mortality from polio remains very low.
    • "17 cases of circulating vaccine-derived poliovirus type 2 (cVDPV2) were reported; four in Bauchi, two each in Borno and Gombe, three each in Jigawa and Katsina and one each in Kano, Taraba and Yobe. There are now 169 cases reported in 2021. There were eight cases reported in 2020." Polio Eradication Initiative, Nigeria country page

• 38

  See this spreadsheet, "Probability of death" sheet, row "Total," column "Under 5, adjusted for etiological fraction and % after recommended age of vaccination."

• 39

  See this spreadsheet, "Main" sheet, section "All-cause mortality," row "Under 5 mortality per 1,000 live births."

• 40

  See this spreadsheet, "Main" sheet, section "Diarrhea mortality," row "Diarrhea mortality per 1,000 people."

• 41

  See this spreadsheet, "Main" sheet, section "Lower respiratory infection mortality," row "SP LRTI unvaccinated."

• 42
  • 1.6% (probability of death from vaccine-preventable disease for unvaccinated children in Nigeria) * 1.19 (adjustment for higher mortality in North West Nigeria) ≈ 2.0%
  • This calculation is used in our most recent cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for all-cause mortality effect" subsection).

• 43

  See our calculations here, based on data from WHO and UNICEF estimates of immunization coverage, Nigeria.

• 44

  See this spreadsheet, "Baseline vaccination coverage" sheet, rows 13-23, particularly the note in cell E13.

• 45

  We use the following formula for each vaccine:

  • Probability of death = (Percent vaccinated x relative risk x Probability of death without vaccine) + (Percent unvaccinated x Probability of death without vaccine)

  This implies:

  • Probability of death without vaccine = Probability of death / ( (Percent vaccinated x relative risk) + Percent unvaccinated) where relative risk = 1 - vaccine effectiveness.

  Relative risk for vaccines is calculated based on the meta-analyses for vaccines discussed below.

• 46
  • See the most recent copy of our cost-effectiveness analysis here, "New Incentives" sheet, row "Probability of death from vaccine-preventable diseases for unvaccinated children under 5 in North West Nigeria."
  • We have some remaining uncertainty about this estimate. We assume the 2019 probability of death given in IHME's data is for any child who was under 5 in 2019, which would reflect individuals receiving vaccinations in the past 5 years. If this probability of death is for children born in 2019, then it might be more appropriate for us to use 2019 vaccination rates only. We also do not know the exact method IHME uses to estimate these figures.

• 47

  See this spreadsheet.

• 48

  We've compared data from IHME to data from the World Health Organization (WHO) and found the total share of vaccine-preventable deaths is roughly similar, though there are differences when looking at specific diseases.

• 49

  See this write-up for further discussion of how we ought to weight the findings of the PERCH study in our estimates of the proportion of pneumonia deaths that are caused by S. pneumoniae and HiB.

• 50

  0.67 (weighted average of vaccine efficacy and disease mortality contribution) x 1.33 (adjustment for non-specific effects) x 0.81 (adjustment for vaccine efficacy concerns) / 0.95 (coverage adjustment) = 0.76

• 51
  • We exclude hepatitis B (directly incentivized) and yellow fever (indirectly incentivized) because IHME data show a very low probability of death from these diseases in Nigeria. See IHME data here.
  • We exclude the oral polio vaccine because we expect that there is a very low probability of death from polio in Nigeria today.
    • The last reported case of wild polio virus in Nigeria was in 2016. Polio Eradication Initiative, Polio-Free Countries.
    • Cases of vaccine-derived polio have been reported in recent years in Nigeria, but we expect that these case numbers are small enough that the average probability of child mortality from polio remains very low.
      • "17 cases of circulating vaccine-derived poliovirus type 2 (cVDPV2) were reported; four in Bauchi, two each in Borno and Gombe, three each in Jigawa and Katsina and one each in Kano, Taraba and Yobe. There are now 169 cases reported in 2021. There were eight cases reported in 2020." Polio Eradication Initiative, Nigeria country page

• 52

  These efficacy numbers correspond to 1 minus the relative risk.

• 53

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 54

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 55

  "Pooled vaccine efficacy (VE) for VT‐IPD was 80% (95% confidence interval (CI) 58% to 90%, P < 0.0001); all serotypes‐IPD, 58% (95% CI 29% to 75%, P = 0.001); World Health Organization X‐ray defined pneumonia was 27% (95% CI 15% to 36%, P < 0.0001); clinical pneumonia, 6% (95% CI 2% to 9%, P = 0.0006); and all‐cause mortality, 11% (95% CI ‐1% to 21%, P = 0.08). Analysis involving HIV‐1 positive children had similar findings." Lucero et al. 2009, Pg. 2.

• 56

  See WHO, Information Sheet, Observed rate of vaccine reactions, DPT, 2014.

• 57

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 58
  • Fulton et al. 2016:
    • "Study outcomes were required to be based on the current WHO definition of (1) "typical" pertussis (>14 days of cough with at least one of the following: paroxysmal cough, inspiratory whoop, or posttussive vomiting, in addition to laboratory confirmation); or (2) "severe" pertussis (>21 days of paroxysmal cough with laboratory confirmation of Bordetella pertussis infection, or epidemiological linkage). Studies using less stringent clinical criteria were also included if their laboratory criteria provided a high level of confidence for pertussis infection (eg, positive culture or polymerase chain reaction assay for B. pertussis)." Pg. 1101.
    • "Regional Databases, with no date restrictions, for English-language studies using the following search terms: pertussis, whooping cough, DTwP, DTaP, vaccine, efficacy, morbidity, and mortality." Pg. 1101.
    • "Meta-analysis of the 2 aP vaccine efficacy studies generated a random-effects pooled vaccine efficacy of 84% (95% confidence interval [CI], 81%–87%; P<.00001; Figure 2)." Pg. 1107.
    • We use the effect of the acellular pertussis vaccine in the cost-effectiveness analysis, since it is based on 2 RCTs and is the effect used in the Lives Saved Tool (LiST); see Table 3, Pg. 1103.
  • WHO does not seem to take a stance on using acellular or whole-cell vaccines, though we have not reviewed in depth; see WHO, Summary of WHO Position Papers, Recommended Routine Immunizations for Children, 2020.

• 59

  See calculations here.

• 60

  WHO, Immunization, Vaccines and Biologicals, Hib

• 61

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 62

  "Nine randomized studies were included in the analysis. Pooled vaccine efficacy using a fixed effects model against confirmed invasive Hib disease following the 3, 2 and 1 primary dose schedule were 82% [95% confidence interval (CI) 73-87], 79% (95% CI 54–90) and 65% (95% CI 23–84), respectively, and the overall efficacy was 80% (95% CI 72–85)." Thumburu et al. 2015, p. 31.

• 63

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 64

  See Figure 1, Sudfeld et al. 2010, p. i50.

• 65
  • "Tuberculosis (TB) kills or debilitates more adults aged between 15 and 59 years than any other disease in the world. TB is caused by Mycobacterium tuberculosis, an intracellular pathogen of the genus Mycobacterium that includes some 55 species, half of which may cause disease in humans." WHO, Biologicals, BCG (Tuberculosis)
  • "Diseases prevented: Disseminated disease and meningitis caused by M. tuberculosis." Feikin et al. 2016, Annex 10A, p. 4

• 66

  IDinsight, New Incentives Evaluation, Pre-Analysis Plan, p. 2.

• 67

  Mangtani et al. 2014, Figure 5, p. 478.

• 68

  "Two forms of TB are life threatening: disseminated or miliar disease, and meningitis." Plotkin et al. 2018, p. 1098.

• 69

  "Diarrhea remains the second leading cause of death around the world for children under 5 years of age. Because the majority of diarrhea deaths occur in the low and middle-income countries, the etiologic agents responsible for diarrhea deaths among young children are unknown. Using hospitalization data as a best estimate of severe diarrheal disease and a proxy for diarrhea mortality, it has been estimated that rotavirus may be responsible for up to 39% of child deaths, the majority of which occur in low and middle income countries." Walker and Black 2011, p. 1.

• 70See IDinsight, New Incentives Evaluation Baseline Report, 2019, Table 1, p. 15.
• 71

  See Lamberti et al. 2016, Table 1.

• 72
  • An "intention-to-treat" (ITT) analysis includes all RCT subjects originally allocated to receive the intervention, whereas a "per-protocol" analysis is a comparison of treatment groups that includes only subjects who in fact completed the intervention.
  • The meta-analyses referenced in LiST include a combination of ITT and per-protocol effects. Where possible, we've used ITT effects.

• 73

  Note that some vaccines target specific serotypes of bacteria: for example, PCV is protective against some serotypes of S. pneumoniae, and we use the effect of PCV on all serotypes (not just those targeted by PCV), since we use deaths attributed to all cases of SP-caused LRTI (not only those caused by vaccine serotypes).

• 74

  See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Overall unadjusted vaccine efficacy for vaccine-preventable disease from meta-analyses" row.

• 75

  A Cochrane review finds an effect of 95% for a single dose at age 9 months to 15 years, though this is also based on cohort studies and includes a much broader age range.

• 76

  "The proportion of children who develop protective antibody levels following measles vaccination depends on the presence of inhibitory maternal antibodies and the immunological maturity of the vaccine recipient, as well as on the dose and strain of the vaccine virus as described in detail below. In general, some 85−90% of children develop protective antibody levels when given one dose of MCV at 9 months of age, and 90−95% respond when first vaccinated at 12 months of age. Median MCV effectiveness (i.e. protection from disease) following a single dose of MCV administered at 9−11 months of age was 84% (interquartile range [IQR] 72–95%) across several studies, and increased to 92.5% (IQR 84.8–97%) among children first vaccinated at 12 months or older. Thus most, but not all, children are protected following a single dose of MCV." WHO, Immunological Basis for Immunization Series, Measles, 2020, p. 12.

• 77

  In our CEA, a downward adjustment in the measles vaccine's effect size would also be offset by a weaker downward adjustment for biomarkers. (If the measles vaccine's effect size were lower, that would bring it closer to what was observed in the biomarkers study.) See below.

• 78

  See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for coverage in trials" row.

• 79

  The study we use for HiB efficacy (Thumburu et al. 2015) uses ITT effects. However, other studies use a combination of ITT and per-protocol effects. The appropriate adjustment would also take into account efficacy achieved through partial vaccination.

• 80

  "Vaccination with the protein-polysaccharide conjugate vaccine, PCV7, has significantly reduced the burden of pneumococcal disease in populations where it is in widespread use and has had an important public health benefit. This vaccine targets only 7 of the more than 92 pneumococcal serotypes, and there have been concerns that the non-vaccine serotypes (NVTs) could increase in prevalence and reduce the benefits of vaccination." Weinberger, Malley, and Lipsitch 2011, p. 1

• 81

  0.66 x 0.81 ≈ 0.54

• 82

  "When MCV1 was administered at age 9–11 months, the median reported VE was 77.0% (IQR 68.0%–91.0%); by WHO region, the median MCV1 VE point estimates ranged from 73.0% in AFR to 96.0% in EUR." Uzicanin and Zimmerman 2011, p. S135.

• 83

  These possible explanations include:

  • The test failed to detect immunity, potentially due to (a) children exhibiting a different immune response, (b) misreporting on cards, (c) the test's accuracy being lower than reported in literature due to publication bias, (d) children being vaccinated too early, or (e) the test being administered too soon after vaccination.
  • Vaccines failed to provide immunity, either because (a) the vaccines lost their potency because they got too hot (i.e., the cold chain was compromised), (b) the vaccines expired, (c) the vaccines were fake or poorly manufactured, or (d) the vaccines were incorrectly administered.

• 84

  Our best guess is that there is roughly a 70% chance of (i) and a 30% chance of (ii).

• 85

  When pooling studies, we put (1) 10% weight on the meta-analysis we use for the main effect of measles vaccine in the CEA, (2) 30% weight on biomarkers studies from Nigeria using more reliable seroconversion tests, (3) 30% weight on the meta-analysis with an Africa-specific effect, and (4) 30% weight on the biomarkers pilot results.

• 86

  We have assumed 75% of the adjustment to measles applies to other vaccines, but this is a rough guess.

• 87

  "This study was designed to assess the seroconversion rate of measles vaccine among infants receiving measles immunization in Ilorin, Nigeria. . . . Only 286 (71.5%) of the vaccines returned to give post-vaccination samples. All the infants screened had low pre-vaccination measles antibody titers. Thirty one (8.0%) of the infants had measles prior to vaccination. The seroconversion pattern showed that 196 (68.6%) of the infants developed protective antibody titers." Fowotade et al. 2015, Abstract

• 88

  "Low seroconversion rate reported in this study was due to low vaccine potency. The titers of vaccines with low potency ranged between log10(-1.0)-log10(-2.25) TCID/per dose. This was beside other non specific antiviral substances exhibited virus neutralizing activity. Only 3 (50%) of the 6 vaccine vials tested had virus titers of log10(-3.25) to log10(-3.5), which fell above the cut-off point recommended by the World Health Organization for measles vaccines." Fowotade et al. 2015, Abstract.

• 89

  "The loss in vaccine virus titers in our centre may be attributed partly to repeated thawing and freezing due to erratic power supply and the absence of standby power generating set serving the immunization clinic where the vaccines are stored. Another contributory factor is the absence of a refrigerator thermometer to ensure that the correct storage temperature is maintained. The low potency found has been the usual trend in potency studies carried out by various researchers in Nigeria. This finding is however different from vaccine studies carried out in other parts of the world. Techatharyat et al. in Thailand and Saha et al. in India reported measles vaccine potency test results of 100.0% and 95.0% respectively. Other adverse factors such as poor handling by vaccinators, existence of chains of salesmen, lack of good storage system for vaccines and difficulty in maintaining a cold chain system have also been suggested to be responsible for the loss in potency of vaccines used in Nigeria and other African countries." Fowotade et al. 2015, Discussion.

• 90

  "These vaccines were obtained from United Nations International Children’s Emergency Fund (UNICEF), through the State EPI Unit and were stored in the freezer until required for immunization. These lyophilized Ruvax vaccines were maintained in cold chains and reconstituted each time per vial according to manufacturer’s instruction." Fowotade et al. 2015, Materials and Methods.

• 91
  • "A total of 150 (52.8%) of the 284 children who received measles vaccine returned for post-vaccination screening, and of these, 82 (54.7%) seroconverted (titres, 10-320) following vaccination; 68 (45.3%) did not seroconvert (Table 1)." Adu et al. 1992, p. 458.
  • "Only 85 (41.87%) of the vaccines reported back for the post-vaccination follow-up screening. The seroconversion pattern showed that 51(60%) had potent antibody titres ranging from 1:40 to 1:1280, while the remaining 34 (40%) had a low antibody titres between < 1:20 and 1:20." Omilabu et al. 1999, Abstract.

• 92
  • "The immune response stimulated by each vaccine was directly dependent on its titre (Table 2). The poor response of the vaccinees in this study is attributable to the low vaccine titres." Adu et al. 1992, p. 459.
  • "The vaccine potency test showed that only 1 (7.14%) of the 14 vaccine vials collected at these centres had virus titre of 3.5 Log while the remaining 13 (92.86%) had virus titres lower than 3.0 Log: the recommended human dose by the World Health Organisation (WHO) for measles vaccine." Omilabu et al. 1999, Abstract.

• 93
  • "The results indicate that the vaccines were not stable at 45 °C, since those vaccines kept at this temperature for 3 days had titres that were 2 log₁₀ units less than at the start of the test. In contrast, vaccines kept at 37 °C or -70 °C for 3 days still retained their original titres (Table 3)." Adu et al. 1992, p. 458.
  • "The loss of vaccine virus titres in Nigeria and other African countries have been attributed to the adverse environmental factors, poor handling by vaccinators, existence of chains of salesman, lack of good storage system for vaccines, insensitivity to expiry date of vaccines and/or administration of expired vaccine. Only Palm-Avenue, of the three EPI centres, had 1 (25%) of 4 vials of measles vaccine used being potent with titre range of 2.5-3.5Log per human dose. This particular centre is the only one with adequate storage facilities, including a stand-by power generating set to augment the erratic public electric supply in the state. Little wonder then, that 22 (43.14%) of the 51 vaccines who had protective measles HI-antibody titre range of 1:40 to 1:1280 were from this centre." Omilabu et al. 1999, Discussion.

• 94
  • Uzicanin and Zimmerman 2011 find a median vaccine efficacy for individuals who were 9-11 months old at age of first dose of 77%, with vaccine efficacy ranging from 73% in the WHO Africa region to 96% in the WHO European region (Table 2, p. 145).
  • "Results of this literature review suggest that the VE of MCV1 administered at 9–11 months of age is generally lower than 85%, which is the usual expected rate of immune response after vaccination at that age." Uzicanin and Zimmerman 2011, p. 135.
  • "Generally, the reasons related to low VE estimates can be grouped into 3 broad categories, including (1) issues related to study methods; (2) program-related factors, such as appropriate vaccine storage, handling, and administration; and (3) host-related factors, most notably, age at vaccination." Uzicanin and Zimmerman 2011, p. 144.

• 95

  "In general, the protective effect of BCG vaccination was either absent or low in studies conducted close to the equator, whereas there was reasonably consistent evidence of good protection observed in studies conducted at latitudes exceeding 40°. Relatively high protection was observed in studies (Saskatchewan Infants and MRC) conducted above 50° latitude: rate ratio 0.22 (95% CI 0.16 to 0.30), corresponding to a VE of 78% (95% CI 70% to 84%). Latitude explained a substantial amount of the between-study variation in the protective effect of BCG vaccination." Pimpin et al. 2013, p. 24.

• 96

  "Because of the evidence that BCG protects against miliary and meningeal tuberculosis, in developing countries BCG vaccination is recommended at birth (or first contact with health services), taking into account HIV status. Our systematic review suggests that BCG also confers protection against pulmonary disease, the greatest burden from tuberculosis, when administered both in infancy and at school age, providing that children are not already infected with M tuberculosis or sensitised to other mycobacterial infections. Protection against pulmonary disease was seen in the Bombay Infants trial suggesting that, even close to the equator, if BCG is administered prior to exposure to tuberculosis and environmental mycobacteria it can provide significant protection. Further evidence of protection in populations close to the equator from BCG given before infection would strengthen these findings." Mangtani et al. 2014, p. 479.

• 97

  The meta-analysis from Lucero et al. 2009 includes two African trials:

  • Cutts et al. 2005 in Gambia: RR of 0.57 [0.38, 0.85].
  • Klugman et al. 2003 in South Africa: 0.58 [0.26, 1.31].

  The overall effect is 0.42 [95% CI 0.25, 0.71]. Effects are lower in the American Indian trial (O'Brien 2003) (0.48), Gambia (Cutts 2005), South Africa (Klugman 2003), and the Philippines (Lucero 2009) (1.00), and highest in Finland (Eskola 2001 and Klipi 2003) and the US (Black 2000) (0.11-0.33). (See Figure 4, p. 15). It is not clear to us why these effects vary. The commentary does not explicitly mention changes varying by distance from equator or lower impact in low- and middle-income countries.

• 98

  See our original cost-effectiveness analysis for New Incentives here.

• 99
  • Higgins et al. 2016 find:
    • BCG decreases all-cause mortality by 30% in clinical trials: "BCG vaccine was associated with a reduction in all cause mortality: the average relative risks were 0.70 (95% confidence interval 0.49 to 1.01) from five clinical trials." Pg. 1.
    • The measles vaccine decreases all-cause mortality 26% in clinical trials: "Receipt of standard titre MCV was associated with a reduction in all cause mortality (relative risks 0.74, [CI] 0.51 to 1.07) from four clinical trials." Pg. 1.
    • The DTP vaccine increases all-cause mortality by 38%, though this is from observational studies that the authors rate as having a high risk of bias: "Receipt of DTP (almost always with oral polio vaccine) was associated with a possible increase in all cause mortality on average (relative risk 1.38, [CI] 0.92 to 2.08) from 10 studies at high risk of bias."
  • Lucero et al. 2009 find that PCV decreases all-cause mortality by 11%: "Pooled vaccine efficacy (VE) for VT‐IPD was 80% (95% confidence interval (CI) 58% to 90%, P < 0.0001); all serotypes‐IPD, 58% (95% CI 29% to 75%, P = 0.001); World Health Organization X‐ray defined pneumonia was 27% (95% CI 15% to 36%, P < 0.0001); clinical pneumonia, 6% (95% CI 2% to 9%, P = 0.0006); and all‐cause mortality, 11% (95% CI ‐1% to 21%, P = 0.08). Analysis involving HIV‐1 positive children had similar findings." Pg. 2.

• 100
  • The "Probability of death" sheet in our supplemental spreadsheet reports that that the probability of death for children under 5 from measles in Nigeria is 0.08%, 0.10% for tuberculosis, 0.86% caused by lower respiratory infections caused by S. pneumoniae, and 0.10% caused by Pneumococcal meningitis.
  • IHME's all-cause probability of death for children under-5 in Nigeria for 2019 is 10% (see here), so each of the causes listed above accounts for less than 10% of all-cause mortality in that age group.

• 101

  See our rough calculations here.

• 102
  • Higgins et al. 2016 find:
    • BCG decreases all-cause mortality by 30% in clinical trials: "BCG vaccine was associated with a reduction in all cause mortality: the average relative risks were 0.70 (95% confidence interval 0.49 to 1.01) from five clinical trials." Pg. 1.
    • The measles vaccine decreases all-cause mortality 26% in clinical trials: "Receipt of standard titre MCV was associated with a reduction in all cause mortality (relative risks 0.74, [CI] 0.51 to 1.07) from four clinical trials." Pg. 1.
    • The DTP vaccine increases all-cause mortality by 38%, though this is from observational studies that the authors rate as having a high risk of bias: "Receipt of DTP (almost always with oral polio vaccine) was associated with a possible increase in all cause mortality on average (relative risk 1.38, [CI] 0.92 to 2.08) from 10 studies at high risk of bias."
  • Lucero et al. 2009 find that PCV decreases all-cause mortality by 11%: "Pooled vaccine efficacy (VE) for VT‐IPD was 80% (95% confidence interval (CI) 58% to 90%, P < 0.0001); all serotypes‐IPD, 58% (95% CI 29% to 75%, P = 0.001); World Health Organization X‐ray defined pneumonia was 27% (95% CI 15% to 36%, P < 0.0001); clinical pneumonia, 6% (95% CI 2% to 9%, P = 0.0006); and all‐cause mortality, 11% (95% CI ‐1% to 21%, P = 0.08). Analysis involving HIV‐1 positive children had similar findings." Pg. 2.
  • 0.74 (for measles) x 0.70 (for BCG) x 0.89 (for PCV) = 0.46
  • See the totals ​​​​here.

• 103

  See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for all-cause mortality effect" subsection.

• 104

  "Most studies indicated that receipt of DTP was associated with higher mortality, and three individual results had 95% confidence intervals that excluded no effect (one lower mortality, two higher mortality). The average relative risk was 1.38 (0.92 to 2.08) among these 10 studies, all assessed as being at high risk of bias." Higgins et al. 2016, p. 4.

• 105
  • See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for all-cause mortality effect" subsection. Even though we guess this effect is driven by non-specific effects (or higher than estimated proportion of deaths caused by vaccine-preventable diseases), we've modeled this as an increase in efficacy of vaccines against vaccine-preventable disease for simplicity.
  • See also the "Malaria Consortium" sheet in our most recent CEA, "​​Ratio of indirect malaria deaths per direct malaria death" row.

• 106

  See the most recent version of our cost-effectiveness analysis, "Malaria Consortium" sheet. In that CEA, we state: "Empirical research suggests that malaria control interventions often have a larger effect on all-cause mortality than would be expected exclusively from declines in malaria-specific mortality (see excerpts below). We're highly uncertain about the exact value of this input, but we have spoken with malaria experts who told us that it is widely accepted there are roughly 0.5-1 indirect malaria deaths for every direct malaria death."

• 107

  See the calculation in the original version of our New Incentives CEA here.

• 108

  See the most recent version of our cost-effectiveness model, "New Incentives" sheet.

• 109

  See the most recent version of our cost-effectiveness model, "Adjustments for excluded effects" sheet, "New Incentives" section, and "New Incentives" sheet, "Adjustments for effects excluded from our core model – New Incentives" section.

• 110

  See this spreadsheet, "Probability of death" sheet, row "Total."

• 111

  See calculations here.

• 112

  See the most recent version of our cost-effectiveness model, "Adjustments for excluded effects" sheet, "New Incentives" section, and "New Incentives" sheet, "Adjustments for effects excluded from our core model – New Incentives" section.

• 113

  See the most recent version of our cost-effectiveness model, "Adjustments for excluded effects" sheet, "New Incentives" section, and "New Incentives" sheet, "Adjustments for effects excluded from our core model – New Incentives" section.

• 114

  "Vaccination confers both direct and indirect effects. The direct effect implies protection against disease in vaccinated individuals. Indirect protection is when susceptible individuals avoid infection because the people who surround them are immunized. The magnitude of indirect effects is a function of transmissibility of the infectious agent, population mixing patterns, distribution of vaccine, and distribution of immunity in the population. 'Herd immunity' refers to population-scale immunity." Nymark et al. 2017.

• 115
  • Yellow fever and acute hepatitis B have a much lower DALY burden for children under five (53 and 69 per 100,000, respectively), relative to other diseases prevented by vaccines (see this spreadsheet, "Percent of effect from YLDs" sheet, cells J3, and J12.
  • For polio, meningitis A, mumps, rubella, and varicella, our impression is that the number of cases are low, but we haven’t attempted to quantify the burden. In our cost-effectiveness analysis, we include a rough guess for additional benefits from effects on these diseases.

• 116

  We believe the excluded morbidity effects are likely to be small (no more than 5% of the total DALY burden). This is based on the finding that only a small percent of DALYs is due to years lost to disability (YLDs) for children under 5 (less than 2% of DALYs across all diseases). Rates are higher for older age groups, but the absolute number of DALYs due to YLDs remains relatively low. Excluding these morbidity effects seems consistent with our decision to exclude morbidity from our CEAs of other interventions primarily aimed at reducing mortality (see here for further analysis).

• 117
  • "The incentive had the largest impact on the timeliness of the Measles 1 vaccine. Children in the treatment group who received the Measles 1 vaccine were 33 percentage points (95% CI: 28, 38; p-value < 0.001) more likely to have received the vaccination within one month of the recommended age (9 months) compared to those who received the Measles1 vaccine in the control group. Measles 1 vaccinations were also more likely to be timely in the treatment group than in the control group when using a cutoff of within 2 weeks of the recommended age." IDinsight, Impact Evaluation of New Incentives, Final Report, p. 30
  • The measles vaccine is recommended at 9 months in high transmission settings to balance higher efficacy with age (which suggests vaccinating later) with high risk of measles for those unvaccinated (which suggests vaccinating sooner).
  • “The age at vaccination is one of the most important determinants of the immune response to MCV, with older infants usually showing better responses than younger infants (Figure 4). The optimal age for measles vaccination is determined by consideration of the age-dependent increase in seroconversion rates following measles vaccination and the average age of infection. In regions of intense MV transmission, the average age of infection is low and the optimal strategy is to vaccinate against measles at as young an age as possible (usually 9 months of age, Figure 4). By contrast, in settings where MV transmission has been reduced, the age of administration of the first dose of MCV can be increased to 12 months or older. Antibody responses to MCV increase with age up to around 15 months because of the declining levels of inhibitory maternal antibodies and decreasing immaturity of the immune system. This immaturity of the immune system in neonates and very young infants includes a limited B-cell repertoire and inefficient mechanisms of antigen presentation and T-lymphocyte help. The recommended age at vaccination must balance the risk of primary vaccine failure, which decreases with age, against the risk of MV infection prior to vaccination, which increases with age.” WHO, Immunological Basis for Immunization Series, Measles, 2020, p.13.
  • As a result, vaccinating too soon may risk lowering vaccine efficacy while vaccinating too late may risk measles infection. We have not sought to vet this claim or quantify this effect, and we include a small adjustment for improved timeliness in our cost-effectiveness analysis.

• 118

  See IDinsight, Impact Evaluation of New Incentives, Final Report, Table 13, p. 36: "Outcome: Ever visited clinic… Adjusted OLS results: 0.05 ([95% CI] 0.03, 0.08)"

• 119

  See the most recent version of our cost-effectiveness model, "Adjustments for excluded effects" sheet, "New Incentives" section, and "New Incentives" sheet, "Adjustments for effects excluded from our core model – New Incentives" section.

• 120
  • See here.
  • New Incentives may offset this risk through its effect on inactivated polio vaccine (IPV), which may lower risk of outbreak from oral polio vaccines: "Circulating VDPVs occur when routine or supplementary immunization activities (SIAs) are poorly conducted and a population is left susceptible to poliovirus, whether from vaccine-derived or wild poliovirus. Hence, the problem is not with the vaccine itself, but low vaccination coverage. If a population is fully immunized, they will be protected against both vaccine-derived and wild polioviruses." WHO, Poliomyelitis: Vaccine derived polio, 2017.

• 121

  "Receipt of DTP (almost always with oral polio vaccine) was associated with a possible increase in all cause mortality on average (relative risk 1.38, 0.92 to 2.08) from 10 studies at high risk of bias." @HIggins et al. 2016@, pg. 1.

• 122

  See the most recent version of our cost-effectiveness model, "New Incentives" sheet, "Downside Adjustments - New Incentives" section.

• 123

  See the most recent version of our cost-effectiveness model, "New Incentives" sheet, "Leverage/Funging adjustment" section.

• 124

  See the most recent version of our cost-effectiveness model, "New Incentives" sheet, "Leverage/Funging adjustment" section.

• 125

  We retain a high degree of trust in New Incentives and the RCT results that inform our recommendation.

  The questions relate to IDinsight, an independent organization. The inquiry from the National Health Research Ethics Committee, Nigeria (NHREC) centers on questions about whether IDinsight had proper ethical approvals prior to starting the pilot study and whether IDinsight appropriately acknowledged its local collaborators. The NHREC has not yet issued a formal response.

  The inquiry does not relate to any aspect of the main RCT that informs our understanding of the program's impact. IDinsight told us that the main RCT received full ethical approvals and permissions in advance of data collection, and it shared the signed approvals it received from the NHREC. The questions do not relate to New Incentives' operation of the program. We thus do not believe that these concerns should impact our assessment of New Incentives.

  Details on what happened and steps we and IDinsight are taking are here.

• 126

  See the most recent version of our cost-effectiveness analysis, "Adjustments for Excluded Effects" sheet, "New Incentives" section, "Crowding out of New Incentives" row.

• 127

  See the most recent version of our cost-effectiveness analysis, "New Incentives" sheet, "Effect of vaccination on disease incidence" section, "Adjustment for lower vaccine efficacy in Nigeria" section.