Vitamin A Supplementation

Published: April 2024 (Previous versions of this page: November 2018 version, November 2017 version)

Basics

Vitamin A deficiency (VAD) is a common condition in low and middle income countries that can lead to blindness, increased susceptibility to infection, and death.

Vitamin A supplementation (VAS) involves getting 6-59 month old children to swallow a small amount of fluid containing vitamin A, delivered via a single-use capsule. GiveWell currently funds Helen Keller International and Nutrition International, who provide funding and technical support to governments to deliver VAS via mass distribution campaigns. (More) We think these campaigns increase the number of children who receive VAS and reduce child mortality.

How cost-effective is it?

As of February 2024, we estimate that it costs between ~$1,000 and ~$8,500 (varying by location) to avert a death through VAS campaigns in locations where GiveWell supports VAS. This equates to being 9 to 59 times as effective as spending on unconditional cash transfers (GiveWell’s benchmark for comparing different programs).

We think VAS is cost-effective because:

• Child mortality is high in locations where GiveWell funds VAS. We estimate that the chance that a 6-59 month old child will die from any cause each year is approximately 1% to 2% per year, depending on the location. We rely primarily on all-cause and disease-specific mortality estimates from the Global Burden of Disease (GBD) Model. (More)

• We think VAS reduces child mortality. We estimate that receiving VAS reduces child mortality by about 4% to 12%, depending on location. The starting point for this figure is an estimate from a meta-analysis of randomized controlled trials (RCTs) that VAS reduces mortality by 24%. (More) We apply two main adjustments to this:
  • A -25% adjustment to account for possible weaknesses in the underlying RCTs. (More)
  • A -41% to -79% adjustment (varying by location) to account for improved child health since the VAS trials were conducted. This incorporates estimates about the share of VAS-susceptible causes of death (primarily diarrhea and measles) and contemporary rates of vitamin A deficiency, compared to when the trials were conducted. (More)

• It is very cheap to reach children with VAS. We estimate that it costs approximately $1 per capsule, or $2 per year, to deliver VAS to a child (More). This is cheaper than most other mass distribution programs we have seen (e.g., approximately $5 to $6 per child per year for seasonal malaria chemoprevention). Intuitively, reasons why we think VAS is cheap include the capsules being donated, and because it can be co-delivered alongside other health interventions such as vaccination campaigns.

• Without GiveWell funding, fewer children would receive VAS. That’s because we think:
  • It’s unlikely that a high proportion of children would be reached without mass distribution campaigns. In most countries where GiveWell funds VAS campaigns, children can also access VAS through other sources (e.g., through routine immunization appointments). Our current best guess is that 25% of children would receive VAS through other sources besides campaigns, although this is based on limited data. (More)
  • Other funders are unlikely to replace GiveWell’s funding for campaigns. Both Helen Keller International and Nutrition International work in locations where we do not believe there is sufficient funding from other sources to support the whole country with regular, high-quality VAS campaigns. We currently estimate that there is a 20% to 50% chance (varying by location) that other funders, primarily UNICEF, would replace GiveWell’s funding for VAS in our absence. Our adjustment to account for this lowers cost-effectiveness by 18% to 46% depending on the location. This is high relative to some other programs we fund, but we think VAS is still cost-effective even after accounting for this. (More)

• We think VAS probably has additional benefits beyond reducing child mortality. These include:
  • Income benefits. We think that VAS probably increases children’s incomes in later life. Other child health programs we have investigated (malaria and deworming) have found evidence that averted illness in childhood leads to increased income and consumption in later life. We would guess that VAS has similar benefits, but we have not found any studies directly investigating this question. In total, we estimate that these effects account for approximately 20% of the total modeled benefit of VAS. (More)
  • Other supplemental adjustments. We think that VAS leads to several other possible benefits, including reduced morbidity from some illnesses (especially diarrhea and measles), improved vision, and averted costs that would have been spent on treatment. We incorporate these benefits as rough percentage best guesses, leading to an upwards adjustment of 67%. (More)

We quantify these arguments using a cost-effectiveness analysis, which allows us to compare across different programs. Here is a sketch, using estimates for one country, Guinea, as an example.

We’ve also considered other perspectives that might not be captured explicitly in these cost-effectiveness estimates (e.g., whether experts see VAS as a good investment). Overall, we’ve spent less time thinking about these questions than we have on our main cost-effectiveness analysis, but the perspectives we’ve considered somewhat lower our confidence that VAS is highly cost-effective. For example, some experts argue that vitamin A supplementation is no longer effective in modern contexts and should be discontinued. (More)

How could we be wrong?

Although our best estimate is that VAS is highly cost-effective, we see the evidence for it as more uncertain than for GiveWell’s other top charities. Our biggest open questions are:

• What was the true impact of VAS at the time the studies we rely on were conducted? Our analysis of the impact of VAS relies on a Cochrane meta-analysis of VAS RCTs (more) and a -25% adjustment to account for ways the meta-analysis estimate could be biased upwards (more). But we have a number of uncertainties about this:
  • We have received some expert feedback that the meta-analysis shows evidence of substantial publication bias, implying a smaller effect size on mortality than reported. Our internal validity adjustment intends to account for some possibility of publication bias, but we don’t use any statistical methods to estimate this and we have not systematically investigated this question. It’s possible that doing so would lead to a larger downward adjustment. (More)
  • Studies in the meta-analysis vary widely in their estimate of the impact of VAS on mortality (more). We’re unsure how to explain this variation, and the factors that we currently assume are most important in our analysis for determining the effectiveness of VAS across locations (vitamin A deficiency rates and diarrhea and measles rates) do not appear to explain it. (More)
  • The main finding in the VAS meta-analysis is sensitive to the choice of analysis (either 12% or 24% reduced mortality overall depending on the analytical approach). We use the larger of these two estimates (24%). This is a judgment call that we have thought through extensively, but if we are wrong it could imply we are significantly overestimating cost-effectiveness. (More)
  • The largest and one of the most recent trials of VAS found an effect size that is small and not statistically significant, and we’re not sure what is driving this. (More)

• How effective is VAS in modern contexts? The main studies we rely on were primarily conducted in the 1980s and 1990s, when the infectious disease landscape was different and child health was significantly worse than today. It’s likely that VAS delivered today would result in a significantly smaller reduction in mortality. We attempt to account for this with a -41% to -79% adjustment, but we’re uncertain about this for a number of reasons:
  • We don’t have a strong understanding of what mediates the impact of VAS on mortality, meaning we’re unsure what factors to adjust for. (More)
  • Our analysis is very sensitive to vitamin A deficiency rates today, but our estimates are based on information we have low confidence in (10 to 20 year old surveys of deficiency, updated for change over time, and modeled estimates from the Global Burden of Disease Project whose methodology we do not fully understand) (more). Since these surveys were conducted, many countries have introduced vitamin A fortification programs, and we’re unsure how effective these have been at reducing deficiency rates. Our understanding is also that measurement of vitamin A deficiency is sensitive to the choice of test used, reducing our confidence in our estimates (more).
  • We’re unsure how changes to health environments in the future will affect VAS. Our biggest uncertainty is whether scale-up of another child health program, azithromycin distribution, will avert deaths that would have been averted by VAS, implying the additional benefit of delivering VAS could be lower. (More)

• How many additional children receive VAS as a result of GiveWell funding? We estimate that GiveWell funding substantially increases VAS coverage, but several aspects of the evidence we rely on are limited:
  • We have some concerns about the representativeness and comprehensiveness of the coverage surveys we use to estimate the proportion of children reached in GiveWell-funded campaigns. We account for this with a -17% adjustment, but this is a rough guess (more). We have begun the process of cross-checking this data against independent sources, but have not yet finalized or published this work (more).
  • Our estimates of the proportion of children who would be reached with VAS through sources other than the campaigns we fund are based on very limited data. (More)

• Should we fund other ways to reduce vitamin A deficiency? To date, GiveWell has only funded VAS via mass campaigns. We haven’t explored other funding opportunities (e.g., vitamin A fortification programs) in detail. (More)

• How reliable are the mortality estimates we rely on? Our cost-effectiveness analysis uses estimates of mortality from measles, diarrhea and all causes from the Global Burden of Disease (GBD) Project. These estimates are based on a number of modeling assumptions that we do not understand in detail, and we have substantial uncertainty about them. (More)

• How accurate was our analysis of VAS in hindsight? GiveWell’s cost-effectiveness analyses are "forward-looking" and aim to predict the impact a program will have at the time we make a grant decision. We have paid less attention to backwards checks to understand how accurate our predictions were. We have begun an analysis to understand how our best guesses about VAS cost-effectiveness have changed over time, but (as of February 2024) haven’t yet finalized or published this work. (More)

• Are there drawbacks to VAS campaigns we’re missing? We think that VAS is one of the most cost-effective programs donors can support. However, our understanding is that there is relatively little investment in VAS from other global health funders and we think there is more scientific controversy around VAS than for GiveWell’s other top recommended charities. This concern is mitigated because VAS is recommended by the WHO, and many national governments in low income countries have VAS programs in place. (More)

Contents

• 1

  Xerophthalmia (dry eyes), is "the leading cause of preventable childhood blindness" WHO, Global prevalence of vitamin A deficiency in populations at risk 1995–2005, 2009, p 1.

• 2

  "Vitamin A deficiency (VAD) impairs body functions and may cause death. Adverse health consequences may also include xerophthalmia (dry eyes), susceptibility to infection, stunting and anaemia (Sommer 1996; Rice 2004)." Imdad et al. 2010, p. 9.

• 3

  "Low vitamin A intake during nutritionally demanding periods in life, such as infancy, childhood, pregnancy and lactation, greatly raises the risk of health consequences, or vitamin A deficiency disorders (VADD)." WHO, Global prevalence of vitamin A deficiency in populations at risk 1995–2005, 2009, p. 1.

• 4

  "Chronic VAD may develop when animal sources and fortified foods are limited, as in diets that rely heavily on vegetables and fruits (Ramakrishnan 2002)." Imdad et al. 2010, p. 9.

• 5

  "In settings where vitamin A deficiency is a public health problem, vitamin A supplementation is recommended in infants and children 6–59 months of age as a public health intervention to reduce child morbidity and mortality (strong recommendation)." p. 1.
  See p. 5 of the WHO guidelines for the 4-6 month recommended schedule.
  WHO, Guideline: vitamin A supplementation in infants and children 6-59 months of age, 2011, pp. 1, 5.

• 6

  "Mass distribution campaigns are the main delivery mechanism for VAS. These campaigns are organized at least every 6 months…"
  "Because mass campaigns take place only every 4 to 6 months, children who reach the age of 6 months between two campaigns, may have to wait several months before they get their first dose of Vitamin A despite being the most vulnerable age group.
  "To remedy this, HKI is working closely with country-level health sector experts to add a contact point in national immunization calendars – at 6 months, when no other vaccination is scheduled.
  "Additionally, HKI supports routine facility-based and outreach delivery of vitamin A for all children under 5 in countries where stronger health systems offer sufficient access to quality services. Few countries are ready for this approach and these still need to develop social mobilization actions to create demand to match the capacity to offer services." Helen Keller International, VAS overview brochure, p. 2.

• 7

  This spreadsheet lists the distribution methods and co-delivered interventions for VAS mass distribution campaigns that Helen Keller supported with GiveWell-directed funding in 2018 through 2021.

• 8

  This understanding is based on many conversations with Helen Keller and other VAS stakeholders over time.

• 9

  In January 2022, GiveWell recommended a $9m grant to Nutrition International to support VAS campaigns in Chad. This was the first VAS grant we had made to a grantee other than Helen Keller International. The grant is summarized here.

• 10

  We focus on these locations because they are the locations that GiveWell actively supported in our most recent main grant cycle, as of February 2024 (see our grant pages for Helen Keller’s program here and Nutrition International’s VAS program in Chad here). Some columns in our cost-effectiveness analysis are for countries (or Nigerian states) that GiveWell has either considered for previous grants but not yet funded (e.g., Uganda), or funded in the past but discontinued funding because we estimate that VAS in that country was below our cost-effectiveness bar (e.g., Kenya). We think that the range used here is the best reflection of the value of VAS in locations supported by GiveWell donors.

• 11

  We used the following process:

  • Two GiveWell staff members familiar with our research on VAS gave confidence intervals for the parameters in the Simple CEA sheet of our cost-effectiveness analysis. The intervals were for Helen Keller International's program in Guinea (the example program we discuss in this report). These intervals address the following question: "For each parameter, provide an upper and lower bound for the range of values that you think has a 50% probability of containing the true value (i.e., a 25% probability the true value is lower than the range, and a 25% probability the true value is higher)."
    • Where we felt that it was difficult to form an intuitive judgment about a parameter (e.g., because it comprised several separate parameters), we gave intervals for each individual parameter.
  • One of the two GiveWell staff members compared both sets of intervals and decided upon a final interval for each parameter, using their subjective judgment.
  • We applied the intervals used for Guinea to other countries as follows:
    • We first calculated both the relative differences and absolute differences between our best guesses and our 25th and 75th percentile values for each parameter. For example, if our best guess value for a parameter was 50% with 25th and 75th percentile values of 25% and 75%, then the relative differences would be -50% and +50%, and the absolute differences would be -25 percentage points and +25 percentage points.
    • We assigned confidence intervals to each country that matched the widths of our confidence intervals for Guinea, putting some weight on the relative difference between the confidence interval bounds and the best guess and some weight on the absolute difference. The higher our best guess value for a parameter in a country was compared to our best guess for Guinea, the more weight we put on the absolute difference between the confidence interval bounds and our best guess. The purpose of this method was to avoid assigning extremely wide confidence intervals to parameters in countries where our best guess was much higher than our best guess for Guinea. In some limited cases, we manually adjusted estimates where the intervals seemed implausible, compared to our best guess for Guinea.
  • Finally, we used monte carlo simulations to transform the confidence intervals for individual parameters into an aggregated confidence interval for each parameter. These are the intervals that appear in the summary table of this page, and the Guinea columns in the Sensitivity Analysis sheet of our cost-effectiveness analysis.

• 12

  $1m is an arbitrary amount that we use to quantify the benefits of the program in the rest of our analysis. See this row in our cost-effectiveness analysis.

• 13

  Note: We focus on our estimates for Helen Keller here rather than Nutrition International. Our estimates of the cost per supplement delivered for Nutrition International’s programs are based either on projected future budgets or on adapted versions of our cost per supplement analysis for Helen Keller, and we have less confidence in them than our estimates for Helen Keller. For more detail, see this section of our cost-effectiveness analysis.

• 14

  See this row of our cost-effectiveness analysis. Note: This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Madagascar, Mali, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba. See this section for our reasoning.

• 15

  Applying this adjustment, the number of additional children we estimate will receive VAS as a result of Helen Keller's program ranges from ~413,000 (in Guinea) to ~760,000 in Mali. See this row in our cost-effectiveness analysis.

• 16

  See this row in our cost-effectiveness analysis.

• 17

  See these rows in our cost-effectiveness analysis.

• 18

  See this row in our cost-effectiveness analysis. The per round cost ($0.91 in Guinea), presented in the table above ($0.77 + $0.14), is the annual cost divided by 2.

• 19

  See this column in our cost per supplement analysis.

• 20

  The % of target children receiving VAS.

• 21

  See this row in our analysis of Helen Keller’s monitoring.

• 22

  Our understanding is that workers implementing these campaigns may know in advance which campaign rounds / locations will be surveyed. If that is the case, we would guess that workers would be more incentivized to ensure high coverage in surveyed campaigns than non-surveyed campaigns (which have less oversight). This could result in overall coverage estimates being biased upwards. See this section of our review of Helen Keller’s VAS program for more details on this concern.

• 23

  We would expect these figures to be inflated somewhat by methodological features of the campaigns (e.g., social desirability bias for a self-reported outcome like whether a child has received VAS). We account for this separately in our analysis with a downward adjustment for monitoring and evaluation. This is discussed in this section of the report.

• 24

  Note - this figure does not appear directly in our analysis, because we only present the number of children reached in each individual campaign. We obtain this figure by dividing 6,003,796 (the total number of supplements we estimate were delivered, here) by 7,677,689 (the sum of the total number of children targeted in this column).
  

• 25

  See these rows in our cost-effectiveness analysis for this figure and this section below for more details on our reasoning.

• 26

  Helen Keller’s spending includes both funding in the form of grants to national governments to pay for VAS campaigns ("Sub Agreements") and spending by Helen Keller itself on programs ("Activities").
  Helen Keller’s summaries of its spending also include some funding for "Regional & Global support" or "Management" costs. Our understanding is that these costs are central support costs (e.g., staff time at Helen Keller’s head or regional offices) which are not attributable to any single country. We divide these costs proportionately between each country in our analysis in proportion to that country’s share of total program spending. See the "HKI costs" section of each country sheet in our cost per supplement analysis.

• 27

  See the "Total" column for each year’s breakdown of spending in this sheet.

• 28

  See this cell in our cost per supplement analysis.

• 29

  Key assumptions:

  • National-level financial contributions are allocated proportionally to Helen Keller areas: Helen Keller typically supports campaigns in designated focus areas in each country, but our understanding is that some other actors’ funding is designated for the whole country. Where this is the case, we include this funding in our analysis, allocated proportionally to the size of targeted populations in Helen Keller-supported areas and other areas.
    • For example, Helen Keller has told us that the Burkina Faso Ministry of Health made financial contributions of $455,834 to the first VAS campaign of 2019. Our understanding is that this spending from the Ministry of Health provides payments to Community Health Workers who distribute vitamin A supplements. We assume that this funding was not allocated only for Helen Keller-supported areas, and allocate it in our analysis proportionally to the size of targeted populations in Helen Keller-supported areas and other areas. See this section of our cost per supplement analysis explanatory notes for further details.
  • We include an estimate of indirect costs for implementing organizations, but not funders: Our understanding is that some NGOs contributing funding to Helen Keller-supported campaigns are best understood as "implementers" (i.e., they have an in-country office and play a role in delivering campaigns), whereas others are best understood as "funders" (i.e., they only provide financial contributions, and do not play a role in delivering campaigns). We would guess that implementers incur additional costs beyond its grants directly to national governments (e.g., costs for staff time). For these organizations (e.g., the WHO in Guinea, Niger and Mali), we therefore typically include a rough estimate of indirect costs, which we do not include for funders (e.g., the Global Fund in Burkina Faso). Our specific method for calculating indirect costs varies somewhat by organization. Our assumptions for these estimates are highly uncertain. See our cost per supplement analysis explanatory notes for detailed reasoning by country.
  • We do not (with some exceptions) include costs incurred by UNICEF: Our understanding is that in the countries where Helen Keller supports VAS campaigns, the other main NGO supporting these campaigns is UNICEF. Typically, Helen Keller and UNICEF both provide support at the national level and also have designated focus regions where each organization supports campaign delivery (see this section of our separate report on Helen Keller’s program for more information). Because in most cases we are estimating the costs of campaigns in Helen-Keller supported regions only, we largely exclude estimates of UNICEF’s costs from our analysis. We make an exception and include UNICEF spending for the first campaign round in Niger in 2019 and the first campaign round in Koulikoro and Sikasso regions in Mali in 2020 (see the detailed reasoning for Niger and Mali respectively in our explanatory notes).

  See our detailed explanatory notes about our cost per supplement analysis for further information by country.

• 30

  Our understanding is that WHO provided costs for polio campaigns in Guinea in 2018 and 2019 and VAS was co-delivered alongside these campaigns (more discussion here). We estimate that WHO incurred ~$1.9m on these campaigns in Guinea in 2018-19, and that 44% of these costs were incurred in locations supported by Helen Keller. This implies a cost of $0.14 per capsule delivered ($1.9m x 44%) / ~6m = $0.14. See this row in our analysis.

• 31
  • "Micronutrient Initiative (MI) (the name of the organization changed ~1 month ago to Nutrition International or NI) is only active in 4 of the 13 countries were HKI is operational, however MI provides the needed number of vitamin A capsules to all countries where HKI works. MI’s role essentially takes place at the national level, providing technical and policy guidance to governments. In most cases, MI delivers the vitamin A capsules to UNICEF, who organizes their management with the national government and ensures that they reach the field." Helen Keller International, HKI country-level technical support related to vitamin A supplementation (unpublished).
  • "We have been a global leader in vitamin A supplementation since our inception 30 years ago, supporting governments to integrate VAS into existing health platforms and ensuring children are not missed with lifesaving VAS. We have led in setting the manufacturing standards and providing vitamin A capsules to eligible countries through the in-kind donation program – implemented with UNICEF, and with support from the Government of Canada – procuring and donating more than 10 billion capsules since 1997." Nutrition International, "Strengthening health systems to deliver lifesaving vitamin A," 2022. Accessed May 2023.

• 32

  For our calculations, see the "Supplement costs" section of each country sheet in our cost per supplement analysis. We assume the cost to procure and distribute each capsule is the same across countries.

• 33

  See this row of our cost per supplement analysis.

• 34

  See footnote 88 of our separate report on Unlimit Health, the deworming organization which ran the study, for a detailed summary of our method in calculating this 30% figure.

• 35

  Including these costs, we estimate that each child reached costs $1.39 in Guinea. Excluding these costs, we estimate a cost of $0.91. See this section in our cost per supplement analysis.

• 36

  Our reasoning for this assumption is that, if Helen Keller spends more on VAS campaigns in a given country, the number of capsules required for the program and the amount of government time dedicated to these campaigns is likely to increase in response. Our best guess is that this is not true for other NGOs’ spending on Helen Keller-supported campaigns. We assume that these resources would have been spent on VAS in any case regardless of the size of the Helen Keller VAS program.

• 37

  In some (but not all) of the countries where GiveWell funds VAS, VAS is delivered as part of the country’s routine immunization schedule at routine healthcare appointments. See Nigeria’s routine childhood immunization schedule here for an example.

• 38

  See this row in our cost-effectiveness analysis for the 25% estimate. In Madagascar, we use 44% for this estimate (see GiveWell, Baseline VAS coverage in 6 regions in Madagascar proposed by Helen Keller International, 2023).

• 39

  Note: this is separate from our estimates of whether GiveWell funding for VAS would be replaced by another actor in our absence. We account for this possibility separately in another part of our analysis.

• 40

  "Vitamin A supplements in Kenya are administered to children aged 6-59 months through various delivery approaches. Throughout the year, children can access VAS in primary health care facilities, but this routine coverage only accounts for around 20 percent of children." Helen Keller International, Room for More Funding Report, 2021, p. 21.

• 41

  These survey figures are unpublished, because we have not yet asked Helen Keller for permission to publish the names of these countries.

• 42

  "Vitamin A supplements in Kenya are administered to children aged 6-59 months through various delivery approaches. Throughout the year, children can access VAS in primary health care facilities, but this routine coverage only accounts for around 20 percent of children, essentially children below 12 months, as many caregivers do not bring their children to the health facilities after the end of the immunization contact points at one year of age." Helen Keller International, Room for More Funding Report, 2021, p. 21.

• 43

  See our analysis in GiveWell, Baseline VAS coverage in 6 regions in Madagascar proposed by Helen Keller International, 2023 .

• 44

  We see the Madagascar data as providing a relatively clean estimate of routine VAS coverage, because our understanding is that no VAS campaigns took place in the period immediately before the survey (in 2021) in Madagascar.
  "Starting in 2019, MCHW was phased out, and VAS was integrated into routine health services…Throughout 2020 and 2021, no significant VAS activities took place in the country." Helen Keller International, VAS scoping visit - Madagascar, October 2022, p. 2.
  We have not seen equivalent data for other countries. While other Demographic and Health Surveys also measure VAS coverage, we think that these will be capturing coverage through campaigns as well as routine services, and it is hard to separate the two out.
  In 2023, GiveWell commissioned Rethink Priorities to investigate what other data sources could be used to inform our estimates of VAS coverage in the absence of GiveWell campaigns. The main finding from the research was that limited data was available to answer this question, and it was challenging to separate VAS coverage achieved through routine services from coverage achieved in campaigns in the sources available. Because we remain uncertain about this question, we have not updated our estimates. We plan to spend more time on this question in the future. A summary of the Rethink Priorities work is available here.

• 45

  See these rows in our cost-effectiveness analysis.

• 46

  Our analysis estimates the number of children reached using the following method:

  • First, we estimate how many children are reached with VAS with GiveWell funding, based on target population estimates and Helen Keller coverage surveys.
  • Second, we roughly guess how many children would have been reached with VAS from other sources regardless of Helen Keller’s funding.
  • We use these two estimates to produce an overall estimate of the number of children reached as a result of GiveWell VAS funding.
  • Note that this method is distinct from directly measuring the impact of GiveWell funding on VAS coverage. For example, we could conduct baseline VAS surveys before funding VAS in a given location, then surveys after funding an initial campaign, and use these to estimate the increase in the proportion of children receiving VAS.

• 47

  See this section of our separate report on Helen Keller’s program for more detail.

• 48

  In 2017, we created a number of case studies to understand what impact Helen Keller funding had in countries where it provided funding for VAS between 2013 and 2016. We concluded that there was some evidence that its funding enabled campaigns to occur and some evidence of increasing coverage rates, although the strength of the evidence varied by country. We haven’t conducted further work on this question since 2017.

• 49

  Our reasoning differs somewhat country-to-country:

  • Côte d’Ivoire: We think our cost per supplement estimate is implausibly low ($0.53) and we have previously heard from Helen Keller that some costs may have been omitted from the information they shared with us.
  • Kenya: We think our cost per supplement estimate is low ($0.78), and we have low confidence in this estimate because it relies on two coverage surveys, one in 2019 and one in 2020, that may not be representative of all Helen Keller-supported areas in Kenya.
  • DRC: We think our cost per supplement estimate is implausibly low ($0.58), especially considering that Helen Keller’s program in DRC is relatively new, and we would have expected start-up costs to be high.
  • Nigeria: We think our cost per supplement estimate is low ($0.72), and Helen Keller didn't report spending by any other actors in Helen Keller-supported regions for these campaigns, other than a small amount of spending by Saving One Million Lives in 2020. We would guess that some other actors’ spending may have been excluded from the totals we received.

  See this section of our cost per supplement explanatory notes onwards for further information.

• 50

  Helen Keller has informed us:

  • In Côte d’Ivoire the government uses a different model with less training for distributors than in other locations.
  • In DRC, there is a lot of pressure from other actors to keep costs low. Helen Keller also attributed low costs to a highly experienced VAS team in DRC.
  • In Nigeria, VAS campaigns use a fixed point rather than a door-to-door model, which is likely to keep costs low.

  Helen Keller also agreed that costs were likely to be lower in Kenya than other countries, although we are uncertain what specific factors contributed to this.
  Source: Helen Keller conversations with GiveWell, October 28, 2022 and January 26, 2023 (unpublished).

• 51

  See this section of our cost per supplement analysis.

• 52

  See this section of our cost-effectiveness analysis.

• 53

  See this row of our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Chad, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller), and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 54

  See this row of our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Chad, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller), and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 55

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Chad, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller), and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 56

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 57

  Cochrane, formerly known as the Cochrane Collaboration, is a not-for-profit research organization which synthesizes health research in systematic reviews.
  "Cochrane is an international network with headquarters in the UK, a registered not-for-profit organization, and a member of the UK National Council for Voluntary Organizations.
  …There are now over 7,500 Cochrane Systematic Reviews which we publish in the Cochrane Library. We also play a key role in developing new methods in evidence synthesis." Cochrane, "About us", accessed February 17th 2023.

• 58

  Note: An updated version of the same meta-analysis, Imdad et al. 2022, has been published since we conducted this analysis. This update did not identify any new studies meeting the inclusion criteria and the main findings were unchanged. For simplicity, we continue to cite Imdad et al. 2017 throughout this report.

• 59

  Two of the trials included, Herrera et al. 1992 and Stansfield 1993, were quasi-randomized rather than truly randomized.
  "We excluded quasi-RCTs with the exception of Herrera 1992 and Stansfield 1993; we made this decision post hoc (Differences between protocol and review). Given the design of the interventions and the placebos as well as steps to blind those administering the sequence, we do not think these studies are meaningfully different from RCTs. Herrera 1992 assigned participants alternately by household, while Stansfield 1993 used a random starting point and alternating distribution of red or green pills. Lack of a truly random sequence was not related to other sources of bias (for example, performance bias) because individuals delivering the capsules had no ongoing contact with participants, and the manufacturer (Roche) held the code until the study was completed." See Imdad et al. 2017, pp. 7-8.

• 60

  "We identified 47 studies (4 of which are new to this review), involving approximately 1,223,856 children. Studies took place in 19 countries: 30 (63%) in Asia, 16 of these in India; 8 (17%) in Africa; 7 (15%) in Latin America, and 2 (4%) in Australia. About one-third of the studies were in urban/periurban settings, and half were in rural settings; the remaining studies did not clearly report settings. Most of the studies included equal numbers of girls and boys and lasted about a year. The included studies were at variable overall risk of bias; however, evidence for the primary outcome was at low risk of bias. A meta-analysis for all-cause mortality included 19 trials (1,202,382 children)." Imdad et al. 2017, abstract.

• 61

  This range depends on whether a fixed-effects (12% mortality reduction) or random-effects (24% reduction) analysis is used. We use the random-effects finding as the main input in our cost-effectiveness analysis See this section for further details on each method and our reasoning for using the random-effects analysis.
  "At longest followup, there was a 12% observed reduction in the risk of all-cause mortality for vitamin A compared with control using a fixed-effect model (risk ratio (RR) 0.88, 95% confidence interval (CI) 0.83 to 0.93; high-quality evidence). This result was sensitive to choice of model, and a random-effects meta-analysis showed a different summary estimate (24% reduction: RR 0.76, 95% CI 0.66 to 0.88); however, the confidence intervals overlapped with that of the fixed-effect model." Imdad et al. 2017, abstract.

• 62

  See this row in our cost-effectiveness analysis.

• 63

  "We identified 47 studies (4 of which are new to this review), involving approximately 1,223,856 children. Studies took place in 19 countries: 30 (63%) in Asia, 16 of these in India; 8 (17%) in Africa; 7 (15%) in Latin America, and 2 (4%) in Australia. About one-third of the studies were in urban/periurban settings, and half were in rural settings; the remaining studies did not clearly report settings. Most of the studies included equal numbers of girls and boys and lasted about a year. The included studies were at variable overall risk of bias; however, evidence for the primary outcome was at low risk of bias. A meta-analysis for all-cause mortality included 19 trials (1,202,382 children)." Imdad et al. 2017, abstract.

• 64

  See this document for our summary of each study’s follow-up period. Note that the follow-up window was unclear for one study (Vijayaraghavan et al. (1990)).

• 65

  "At longest followup, there was a 12% observed reduction in the risk of all-cause mortality for vitamin A compared with control using a fixed-effect model (risk ratio (RR) 0.88, 95% confidence interval (CI) 0.83 to 0.93; high-quality evidence). This result was sensitive to choice of model, and a random-effects meta-analysis showed a different summary estimate (24% reduction: RR 0.76, 95% CI 0.66 to 0.88); however, the confidence intervals overlapped with that of the fixed-effect model." Imdad et al. 2017, abstract.

• 66

  "Nine trials reported mortality due to diarrhoea and showed a 12% overall reduction for VAS (RR 0.88, 95% CI 0.79 to 0.98; 1,098,538 participants; high-quality evidence)." Imdad et al. 2017, abstract.

• 67

  "Six studies reported a lower risk of mortality due to measles, but the effect was not statistically significant (RR 0.88, 95% CI 0.69 to 1.11; 1,088,261 participants." Imdad et al. 2017, p. 18.

• 68

  "Lower respiratory tract infection Nine studies did not show any significant difference between the intervention and placebo group (RR 0.98, 95% CI 0.86 to 1.12; 1,098,538 participants." Imdad et al. 2017, p. 18.

• 69

  The confidence intervals, particularly for measles, included a wide range of possible effects:

  • Diarrhea mortality: the confidence interval ranged from a 2% to a 21% mortality decrease.
  • Measles: the confidence interval ranged from an 11% increase to a 31% decrease.
  • Lower respiratory tract infection: the confidence interval ranged from an 12% increase to a 14% decrease.

  See the footnotes immediately above for citations.
  In addition, Imdad et al. 2017 notes that secondary outcomes reported in trials are more likely to be affected by selective reporting bias because studies might only report secondary outcome data if they find positive results.
  "Most of the trials in the review included multiple outcome measures, and positive results are more likely to be included in reports than negative results. Only seven (14%) studies appeared to be free of selective outcome reporting (Florentino 1990; Rahmathullah 1990; West 1991; Dibley 1996; Benn 1997; DEVTA trial 2013; Fisker 2014). We judged 26 (55%) studies to be at unclear risk of bias and 14 (29%) studies to be at high risk of bias (Pinnock 1988; Van Agtmaal 1988; Vijayaraghavan 1990; Ross 1993 HEALTH; Ross 1993 SURVIVAL; Stansfield 1993; Ramakrishnan 1995; Pant 1996; Bahl 1999; Arya 2000; Cherian 2003; Lin 2008; Lin 2009; Lima 2014). Most of the studies did not cite a published protocol, which is why we assessed a large proportion of studies to be at unclear risk of bias."
  "Some secondary outcomes did not contain a majority of the children randomised in the review, and these results may be vulnerable to selective outcome reporting bias." Imdad et al. 2017, pp. 16, 23.

• 70

  "Nine trials reported mortality due to diarrhoea and showed a 12% overall reduction for VAS (RR 0.88, 95% CI 0.79 to 0.98; 1,098,538 participants; high-quality evidence)."
  "Six studies reported a lower risk of mortality due to measles, but the effect was not statistically significant (RR 0.88, 95% CI 0.69 to 1.11; 1,088,261 participants."
  "Lower respiratory tract infection Nine studies did not show any significant difference between the intervention and placebo group (RR 0.98, 95% CI 0.86 to 1.12; 1,098,538 participants." Imdad et al. 2017, abstract, p. 18.

• 71

  "Six studies reported a 50% decrease in measles incidence (RR 0.50, 95% CI 0.37 to 0.67; 19,566 participants…We judged this evidence to be of moderate quality." Imdad et al. 2017, p. 20.

• 72

  "Fifteen studies reported a 15% decrease in diarrhoea incidence (RR 0.85, 95% CI 0.82 to 0.87; 77,946 participants…We judged this evidence to be of low quality (see Summary of findings for the main comparison)." Imdad et al. 2017, p. 18.

• 73

  "Eleven studies reported no combined effect for VAS on LRTI incidence (RR 0.99, 95% CI 0.92 to 1.06; 27,540 participants…We judged the quality of this evidence to be low (see Summary of findings for the main comparison)." Imdad et al. 2017, p. 21.

• 74

  "In summary, the primary outcome was at low risk of bias, and the size and the significance of the effect cannot be explained by bias. While there was some evidence of small study bias for secondary outcomes, further research is unlikely to change the conclusion that VAS, delivered with high quality and coverage, prevents death among children aged 6 to 59 months in low- and middle-income countries." Imdad et al. 2017, p. 23.
  The full assessment of risks of bias for each trial is in Imdad et al. 2017, pp. 14-16.

• 75

  "Most of the trials in the review included multiple outcome measures, and positive results are more likely to be included in reports than negative results. Only seven (14%) studies appeared to be free of selective outcome reporting (Florentino 1990; Rahmathullah 1990; West 1991; Dibley 1996; Benn 1997; DEVTA trial 2013; Fisker 2014). We judged 26 (55%) studies to be at unclear risk of bias and 14 (29%) studies to be at high risk of bias (Pinnock 1988; Van Agtmaal 1988; Vijayaraghavan 1990; Ross 1993 HEALTH; Ross 1993 SURVIVAL; Stansfield 1993; Ramakrishnan 1995; Pant 1996; Bahl 1999; Arya 2000; Cherian 2003; Lin 2008; Lin 2009; Lima 2014). Most of the studies did not cite a published protocol, which is why we assessed a large proportion of studies to be at unclear risk of bias."
  "Some secondary outcomes did not contain a majority of the children randomised in the review, and these results may be vulnerable to selective outcome reporting bias." Imdad et al. 2017, pp. 16, 23.

• 76

  Dr. West told us that "smaller trials not intended, designed, powered or executed to enable the efficacy of vitamin A on child mortality to be assessed are unlikely to be informative for this outcome." Keith West, email to GiveWell, February 25th, 2024.

• 77

  "Comparing the 36 retinol-allocated versus 36 control blocks in analyses of the primary outcome, deaths per child-care centre at ages 1.0–6.0 years during the 5-year study were 3.01 retinol versus 3.15 control (absolute reduction 0.14 [SE 0.11], mortality ratio 0.96, 95% CI 0.89–1.03, p=0.22), suggesting absolute risks of death between ages 1.0 and 6.0 years of approximately 2.5% retinol versus 2.6% control." Awasthi et al. 2013a, abstract.

• 78

  "Trials assigned approximately 1,223,856 participants, with sample sizes ranging between 35 participants in Van Agtmaal 1988 to approximately 1 million participants in DEVTA trial 2013." Imdad et al. 2017, p. 12.
  Source that DEVTA was the largest randomized trial ever conducted (as of 2010): Imdad et al. 2010, p. 15.
  "DEVTA is the largest randomised controlled trial ever conducted, including approximately one million children. That is, the trial included four times the combined participants of all included studies in this review."
  Note that DEVTA was randomized at the administrative block level. This limits the statistical power of the study to some extent.
  "Participants in this cluster-randomised trial were pre-school children in the defined catchment areas of
  8338 state-staffed village child-care centres (under-5 population 1 million) in 72 administrative blocks. Groups of four
  neighbouring blocks (clusters) were cluster-randomly allocated in Oxford, UK, between 6-monthly vitamin A (retinol
  capsule of 200 000 IU retinyl acetate in oil, to be cut and dripped into the child’s mouth every 6 months), albendazole
  (400 mg tablet every 6 months), both, or neither (open control). Analyses of retinol effects are by block (36 vs 36 clusters)." Awasthi et al. 2013a, abstract.

• 79

  "Participants in this cluster-randomised trial were pre-school children in the defined catchment areas of
  8338 state-staffed village child-care centres (under-5 population 1 million) in 72 administrative blocks. Groups of four
  neighbouring blocks (clusters) were cluster-randomly allocated in Oxford, UK, between 6-monthly vitamin A (retinol
  capsule of 200 000 IU retinyl acetate in oil, to be cut and dripped into the child’s mouth every 6 months), albendazole
  (400 mg tablet every 6 months), both, or neither (open control). Analyses of retinol effects are by block (36 vs 36 clusters). The study spanned 5 calendar years, with 11 6-monthly mass-treatment days for all children then aged 6–72 months."
  "Within the participating blocks, the study population was all young children living in the defined catchment areas of 8338 village anganwadi child-care centres (AWCs; anganwadi means courtyard) that the ICDS regarded as functional (except in one block17) (figure 1). A typical rural AWC employs an ICDS-funded AWC childcare worker and serves a village with a population of about 1000 (with 10–15% aged 1.0–6.0 years); large villages can have more AWCs. AWCs register about two thirds of the children of age 1.0–5.0 years (and one-third of 5-year-olds) for possible nutritional supplementation."
  "When treatment became due (in April or October), in each treatment-allocated block a mass-treatment day was selected on which all children of apparent age 6–72 months in all study AWCs in that block would be given the trial treatment by their AWC worker. Those treated were checked off on the AWC lists, and the few children missed were often found by the AWC worker and treated soon afterwards." Awasthi et al. 2013a, abstract, p. 1470.
  "UP is an ideal setting for this study, because government programmes have already been instituted that should be able to supply food supplements and micronutrients to preschool children. The study will use this infrastructure (the anganwady system) to deliver deworming medicine and vitamin A to children in those blocks randomly allocated to one and/or other active treatment. . . . The cost-effectiveness of mass treatments . . . depends not only on the costs and effectiveness of the drugs themselves, but also on the practicability of delivering them to children reliably and sustainably at low additional cost. Both of these important questions will be addressed." Awasthi et al. 2013a Appendix, p. 2.

• 80

  The absolute risk of death from age 1-6 years in DEVTA's control group was 2.64%: Awasthi et al. 2013a, p. 1474. This implies a mortality rate of 5.3 per 1,000 child-years.

  • Calculation: The 2.64% risk of death over a five year period (ages 1-6) equates to an average annual mortality rate of 0.53% (2.64 / 5). This 0.53 deaths per 100 children per year is equivalent to 5.3 deaths per 1000 child-years (0.53 x 10).

• 81

  We selected these studies for comparison because, together with DEVTA, they account for approximately 90% of the statistical weight for the all-cause mortality analysis in the fixed-effects analysis, and around 60% of the weight in the random-effects analysis (see this section for a discussion of the difference between the analyses and how we interpret them).
  Meta-analysis weights for the fixed-effects analysis are in Imdad et al. 2017, Analysis 1.1, p. 90. The weights for each study are:

  • DEVTA: 61.7%
  • Ross et al. 1993 (SURVIVAL): 9.85%
  • West et al. 1993: 6.44%
  • Herrera et al. 1992: 5%
  • Daulaire et al. 1992: 3.81%
  • Sommer et al. 1986: 3.58%

  These weights sum to 90.38%.
  Meta-analysis weights from the random-effects analysis are available here:

  • DEVTA: 13.6%
  • Ross et al. 1993 (SURVIVAL): 11.5%
  • West et al. 1993: 10.5%
  • Herrera et al. 1992: 9.8%
  • Daulaire et al. 1992: 8.9%
  • Sommer et al. 1986: 8.7%

  These weights sum to 63%.

• 82

  Mortality risk ratios for each study are available in Imdad et al. 2017, Analysis 1.1, p. 90.

• 83

  Our understanding is that VAS was administered to children aged 6-72 months in the treatment group, but that mortality (the primary outcome) was measured for children aged 12-72 months only.
  "When treatment became due (in April or October), in each treatment-allocated block a mass-treatment day was selected on which all children of apparent age 6–72 months in all study AWCs in that block would be given the trial treatment by their AWC worker. Those treated were checked off on the AWC lists, and the few children missed were often found by the AWC worker and treated soon afterwards."
  "Table 2 shows the findings for child mortality at ages 1.0–2.9 years, 3.0–6.0 years, and 1.0–6.0 years, along with the relative risk for the age range 1.0–6.0 years. Deaths per child-care centre at ages 1.0–6.0 years during the 5-year study (the primary trial endpoint) were 3.01 retinol versus 3.15 control (absolute reduction 0.14 [SE 0.11], mortality rate ratio [RR] 0.96, 95% CI 0.89–1.03, p=0.22), suggesting absolute risks of death between ages 1.0 and 6.0 years of approximately 2.5% retinol versus 2.6% control." Awasthi et al. 2013a, pp. 1470, 1473.

• 84

  "Of 118 administrative blocks in the seven DEVTA districts in north India, 46 with many AWCs not functioning in 1998 were not included. 72 blocks remained; parts of each of them became the study areas." Awasthi et al. 2013a, p. 1470.

• 85

  The absolute risk of death from age 1-6 years in DEVTA's control group was 2.64%: Awasthi et al. 2013a, p. 1474. This implies a mortality rate of 5.3 per 1,000 child-years.

  • Calculation: The 2.64% risk of death over a five year period (ages 1-6) equates to an average annual mortality rate of 0.53% (2.64 / 5). This 0.53 deaths per 100 children per year is equivalent to 5.3 deaths per 1000 child-years (0.53 x 10).

• 86

  "The Survival Study included 21 906 children aged 6-90 months in 185 geographical clusters, who were followed for up to 26 months." Ross et al. 1993, abstract.

• 87

  "The trials took place in the guinea savannah area of Ghana in theKassena-Nankana District, on the border with Burkina Faso (figure 1)." Ross et al. 1993, p. 7.

• 88

  We note that the control group all-cause mortality rate is given as 29.9 per 1,000 child-years in the original paper. We estimate 29.5 per 1,000 child-years by dividing the overall number of control group deaths (495) by the number of years of follow-up (16,779). We’re unsure why there is a discrepancy between these two estimates, but have not investigated further because the difference is so small. "The 21906 children who entered the Survival Study were followed up for 33 287 child-years (16 508 vitamin A group, 16 779 placebo group)...There were 892 deaths among the children in the Survival Study, which gave an overall mean mortality rate for all clusters of 27.11 per 1,000 child-years of follow-up. 397 of the deaths were in vitamin A clusters (mean mortality rate 24.4 per 1000 child-years) and 495 in placebo clusters (29.9 per 1000 child-years). " Ross et al. 1993, p. 10.

• 89

  "Community trials of the efficacy of vitamin A supplementation in reducing preschool childhood mortality have produced conflicting results. To resolve the question, a randomised, double-masked, placebo-controlled community trial of 28 630 children aged 6-72 months was carried out in rural Nepal, an area representative of the Gangetic flood plain of South Asia." West et al. 1991, abstract.

• 90

  "A randomised, double-masked, placebo-controlled vitamin A supplementation trial was carried out from September, 1989, to December, 1990, in the rural, plains (Terai) district of Sarlahi. This area was selected because it has endemic vitamin A deficiency, no previous vitamin A supplementation programme, and ecological, cultural, and demographic similarities to the Gangetic floodplain communities of South Asia, with which it is continuous; the factor enhances the general applicability of the findings." West et al. 1991, p. 67.

• 91

  West et al. 1991, Table III, p. 68.

• 92

  "Previous studies of the effect of 6-monthly vitamin A supplementation on child mortality have given conflicting results. In other trials, more frequent doses of vitamin A have significantly reduced mortality among children at risk of vitamin A deficiency. We have done a double-blind, placebo-controlled trial of vitamin A supplementation in the Sudan among 28 753 children aged 9-72 months at risk of vitamin A deficiency." Herrera et al. 1992, abstract.

• 93

  "The study was conducted between June, 1988, and December,1990, among children between 9 and 72 months of age in five rural councils in northern Sudan where vitamin A deficiency was present." Herrera et al. 1992, p. 267.

• 94

  "During the 18 months of follow-up, there were 120 deaths (8.4/1000) in the vitamin A group and 112 deaths (7.9/1000) [in the control group]." Herrera et al. 1992, p. 269.
  Calculation: In the control group there were 7.9 deaths/1,000 children over a course of 18 months. To calculate the deaths per 1,000 child years, we multiply that number by 2/3. 7.9 x 2/3 = 5.3.

• 95

  "SUBJECTS--All children aged under 5 years; 3786 in eight subdistricts given single dose of vitamin A and 3411 in remaining eight subdistricts given no supplementation."
  "Risk of death for children aged 1-59 months in supplemented communities was 26% lower (relative risk 0.74, 95% confidence interval 0.55 to 0.99) than in unsupplemented communities." Daulaire et al. 1992, abstract.

• 96

  "Setting-Jumla district, Nepal." Daulaire et al. 1992, abstract.

• 97

  Daulaire et al. 1992, Table III, p. 208.

• 98

  The age profile of the children in Sommer et al. is not completely clear. The study targeted children aged 12 - 71 months, and this is the group whose outcomes are reported in the study abstract. However, mortality data is also recorded for younger children (aged 0 - 11 months) and the effect size calculation reported in Imdad et al. 2017 appears to include this group. For consistency, we use the figures reported in Imdad et al.
  "5 939 preschool children were examined at baseline and again 11 to 13 months later. Capsules containing 200 000 IU vitamin A were distributed to preschool children aged over 1 year by local volunteers 1 to 3 months after baseline enumeration and again 6 months later. Among children aged 12-71 months at baseline,mortality in control villages (75/10 231, 7.3 per 1000) was 49% greater than in those where supplements were given (53/10 919,4.9 per 1000) (p<0.05)." Sommer et al. 1986, abstract.
  "Vitamin A was not intended to have been distributed to children under the age of 12 months, but it would appear that some 0-12 month-old children received the vitamin A capsule. Outcome data were reported on a cohort of 0-12 month-old children." Imdad et al. 2017, p. 80.

• 99

  "The study was carried out in Aceh Province, which is at the northern tip of Sumatra and where xerophthalmia is prevalent." Sommer et al. 1986, p. 1169.

• 100

  This figure represents the control group mortality rate for children aged 0 - 71 months, not children aged 12 - 71 months only (the main analysis reported in Sommer et al. 1986). Sommer et al. 1986, Table VI, p. 1171.

• 101

  The primary analysis in Imdad et al. reports a risk ratio of 0.73 for Sommer et al. 1986. Risk ratios are reported in reverse in Sommer et al. 1986 (i.e., the ratio is reported as the control groups’ mortality relative to the treatment group), but our understanding is that this value corresponds to the risk ratio for children aged 0 - 11 months. Imdad et al. 2017, Analysis 1.1, p. 90; Sommer et al. 1986, Table VI, p. 1171.

• 102

  We currently estimate that 80% of the impact of VAS on child mortality is expressed through reduced diarrhea and measles mortality, and 20% of the impact is expressed through other mechanisms. See this section for further details.

• 103
  • Decline in diarrhea mortality. "Mortality from diarrhoea has declined over the past two decades from an estimated 5 million deaths among children under five to 1.5 million deaths in 2004,7 which parallels downward trends in overall under-five mortality during this period." UNICEF/WHO, Diarrhoea: Why children are still dying and what can be done, 2009, p. 5. We have not vetted this study but do believe the result that deaths from diarrhea have declined.
  • Decline in measles mortality. "In 1980, before widespread global use of measles vaccine, an estimated 2.6 million measles deaths occurred worldwide. . . . [T]he estimated number of annual measles deaths worldwide decreased from 733,000 in 2000 to 164,000 in 2008." CDC, Morbidity and Mortality Weekly Report, "Progress in Global Measles Control, 2000-2010," 2012, p. 73. We have not vetted this study but do believe the result that deaths from measles have declined.

• 104

  Awasthi et al. 2013a, p. 1472.

• 105
  • Ross et al. 1993: the mean daily prevalence of measles and diarrhea in the placebo group from were 0.1% and 15.9% respectively.
  • West et al. 1991: About 5.5% of children had measles during the four months preceding the study and 11% of children had diarrhea during the week preceding the study.
  • Herrera et al. 1992: 0.3% of children had measles in the seven days preceding the study and just over 17% had diarrhea in the seven days preceding the study.
  • Sommer et al. 1986: about 22% of children had measles at any time in the past and about 7.9% of children had diarrhea in the seven days before the study.

  Because of the differing time periods covered, these data are not directly comparable to prevalence rates among DEVTA participants, but they seem to suggest reasonably similar overall levels of illness.
  Ross et al. 1993, p. 10, Table 1; West et al. 1991, p. 68, Table I; Herrera et al. 1992, p. 269, Table I; Sommer et al. 1986, p. 1170, Table III.

• 106

  The earliest available GBD estimates are from 1990. Some of the trials were conducted before this, so we use the national-level estimates from the country where the trial was conducted as a proxy. This is a very rough method, increasing our overall uncertainty about this comparison.

• 107

  For example, if a study was conducted in a region with higher or lower levels of mortality than the national average.

• 108

  See this cell in our supplementary analysis.

• 109

  See this cell in our supplementary analysis.

• 110

  Beaton et al. 1993:

  • "The second consideration might be overall mortality rates. Figure 5.3 portrays the relative effectiveness of vitamin A supplementation in relation to control group mortality rates (a poor proxy for baseline mortality rate). No particular relationship is apparent and none could be detected in statistical analyses involving a variety of models in which individual projects were weighted (see Technical Annex)." p. 67.
  • See Figure 5.3, p. 68.

• 111

  Note: the sample of children in DEVTA who were tested for VAD was a convenience sample (i.e., children were selected by workers non-randomly, normally depending on which children were registered with the anganwadi childcare center). This leaves open the possibility that the reported data is not representative of the overall study population. Our best guess is that this data collection process is more likely to underestimate than overestimate VAD (i.e., because the children selected were probably the easiest children to reach and therefore may have had different diets, access to healthcare, poverty levels, and overall health from the general population).
  That said, our guess at the dynamics of the sampling bias could be mistaken, and we cannot entirely rule out the possibility that DEVTA’s participants had a lower incidence of VAD than participants in other studies.

• 112
  • DEVTA: Awasthi et al. 2013a, Table 1, p. 1472.
  • All studies in Imdad et al. 2017: See this section of our 2019 analysis of VAD.

• 113

  "Independently, enquiries a week after mass-treatment days about lists of named children from the mid-study census confirmed supplementation of 96% (133 602/138 966) of those registered with the AWC and 72% (80 048/111 227) of those not, largely independent of age (data not shown). Because two-thirds of under-5s and one-third of 5-year-olds were AWC-registered, overall compliance was about 86% (not 91%, as previously estimated)." Awasthi et al. 2013a, p. 1472.

• 114

  See GiveWell, Coverage estimates from RCTs of vitamin A supplementation included in Imdad et al 2017 meta-analysis for a comparison.

• 115
  • "The program was delivered very inexpensively compared with previous programs and it used existing centers and care providers. Limited investment in the implementation of the intervention (e.g. training and monitoring) may help explain the relatively poor result." GiveWell's non-verbatim summary of a conversation with Evan Mayo-Wilson, June 10, 2013.
  • "Excluding donated drug, the total cost was US $100 000 per year (US$0.10 per child), but this expense was mostly for evaluating the intervention." Awasthi et al. 2013b (deworming), p. 1484.
  • According to an account of an unpublished workshop on DEVTA, "DEVTA provided the anganwadi workers with less than half a day’s training." Sommer, West, and Martorell 2013, p. 591, citing a workshop on the DEVTA Study at Worcester College, Oxford, UK from Nov. 3-4, 2008; the workshop proceedings have not been published.
  • "But this was neither a rigorously conducted nor acceptably executed efficacy trial: children were not enumerated, consented, formally enrolled, or carefully followed up for vital events, which is the reason there is no CONSORT diagram." Sommer, West, and Martorell 2013, p. 1
  • "To achieve 96% coverage in Uttar Pradesh in children found in the anganwadi workers’ registries would have been an astonishing feat; covering 72% of children not found in the anganwadi workers’ registries seems even more improbable. In 2005–06, shortly after DEVTA ended, only 6.1% of children aged 6–59 months in Uttar Pradesh were reported to have received a vitamin A supplement in the previous 6 months according to results from the National Family Health Survey,2 a national household survey representative at national and state levels. The level of contact between anganwadi workers and children has historically been very low. Although 76% of children aged 0–71 months in 2005–06 lived in areas covered by an anganwadi worker, only 22% of children received any service from the anganwadi worker. Thus, it is hard to understand how DEVTA ramped up coverage to extremely high levels (and if it did, why so little of this effort was sustained)." Sommer, West, and Martorell 2013, p. 1.
  • Note: some of the authors of DEVTA have challenged this interpretation of the National Family Health Survey. They note that DEVTA was delivered in only 7 of 70 districts in Uttar Pradesh and only 3% of the administrative blocks in the state. This may explain the discrepancy between the coverage figures from DEVTA and the results of the National Family Health Survey.
  • "The reference implies that the DEVTA program did not achieve the compliance it had reported. However, DEVTA was completed in autumn 2004, a significant amount of time before the National Family Health Survey. Additionally, DEVTA operated in only 7 out the 70 districts in UP and provided VAS in only 3% of the blocks (sub-districts) in UP. Among those blocks, VAS was distributed only from ones that had functioning AWCs." GiveWell's non-verbatim summary of a conversation with Richard Peto and Simon Read, April 10, 2014, p. 8

• 116

  "During the third study year, monitors abstracted from local records name, sex, age, and father’s name for all (just over 1 million) pre-school children in the study areas." Awasthi et al. 2013a, pp. 1470-71.

• 117

  "Independently, enquiries a week after mass-treatment days about lists of named children from the mid-study census confirmed supplementation of 96% (133 602/138 966) of those registered with the AWC and 72% (80 048/111 227) of those not, largely independent of age (data not shown). Because two-thirds of under-5s and one-third of 5-year-olds were AWC-registered, overall compliance was about 86% (not 91%, as previously estimated18)." Awasthi et al. 2013a, p. 1472.

• 118

  "Annually, Oxford randomly selected one AWC per block for fieldwork teams to survey 30 pre-school children (six per year of age) 1–5 months after a mass-treatment day, seeking from them blood and faecal samples. In practice, full information with complete assay results was obtained for about 24 children per block. The children surveyed were not randomly chosen and were often taken from AWC lists, hence mainly registered with the AWC. Caregivers were asked whether the previous mass treatment had been received (and, if so, whether any acute illness followed) and about recent (past 4 weeks) illnesses." Awasthi et al. 2013a, p. 1471.

• 119

  "Consistent with this finding, the biomedical surveys of randomly chosen AWCs, which tended to over-sample AWC-registered children, found caregivers reporting 91% (2337/2581) had received retinol on the previous mass-treatment day." Awasthi et al. 2013a, p. 1472.

• 120

  "Unannounced visits to a quarter of the AWCs in vitamin A blocks on or just after each mass-treatment day found 99% (10 925/11 090 visits) were distributing treatment." Awasthi et al. 2013a, p. 1472.

• 121

  See the National Family Health Survey (NFHS-3) India 2005-06, table 9.19, p. 254. Note Sommer, West, and Martorell 2013 raise a similar point in their critique of DEVTA, but use an estimate of 22% for this figure. This figure is uncited and we’re not sure why there’s a discrepancy.

• 122

  The methods used are reported as follows: "Independently, enquiries a week after mass-treatment days about lists of named children from the mid-study census confirmed supplementation of 96% (133 602/138 966) of those registered with the AWC and 72% (80 048/111 227) of those not, largely independent of age (data not shown). Because two-thirds of under-5s and one-third of 5-year-olds were AWC-registered, overall compliance was about 86% (not 91%, as previously estimated 18)." Awasthi et al. 2013a, p. 1472. We’re unsure from this description how coverage was measured.

• 123

  "During the third study year, study monitors abstracted from local records name, sex, age, and father’s name for all (just over 1 million) pre-school children in the study areas." Awasthi et al. 2013b (deworming), p. 1480. Because the compliance data relied on the study to identify a sample, there is no compliance data from the first two years of DEVTA.

• 124

  It is plausible that compliance could be lower during the earlier years of the study:

  • In general, we would guess that distribution improves over time as workers and monitors become familiar with their roles.
  • In the first years of the study, workers were unable to use the Census data to identify, locate, and treat children who were absent from mass treatment days. "Those treated were checked off on the AWC lists, and the few children missed were often found by the AWC worker and treated soon afterwards." Awasthi et al. 2013a, p. 1470.

  This factor is unlikely to be very important because the mortality rate ratio during the first two years of the study (0.96) is the same as the mortality rate ratio from the last three years of the study. (0.96). Awasthi et al. 2013a, p. 1474. If DEVTA’s results were caused by lax procedures at the study’s start, we would expect to see a smaller effect in the first half of the study.

• 125

  "Annually, Oxford randomly selected one AWC per block for fieldwork teams to survey 30 pre-school children (six per year of age) 1–5 months after a mass-treatment day, seeking from them blood and faecal samples. In practice, full information with complete assay results was obtained for about 24 children per block. The children surveyed were not randomly chosen and were often taken from AWC lists, hence mainly registered with the AWC. Caregivers were asked whether the previous mass treatment had been received (and, if so, whether any acute illness followed) and about recent (past 4 weeks) illnesses." Awasthi et al. 2013a, p. 1471.

• 126

  Children in areas receiving VAS had higher levels of retinol, a lower prevalence of deficiency, and a lower prevalence of bitot’s spots (a potential precursor to blindness).
  "Among 2581 versus 2584 children surveyed during the second half of the study, mean plasma retinol was one-sixth higher (0.72 [SE 0.01] vs 0.62 [0.01] μmol/L, increase 0.10 [SE 0.01] μmol/L) and the prevalence of severe deficiency was halved (retinol <0.35 μmol/L 6% vs 13%, decrease 7% [SE 1%]), as was that of Bitot's spots (1.4% vs 3.5%, decrease 2.1% [SE 0.7%])." Awasthi et al. 2013a, abstract.

• 127

  "Annually, Oxford randomly selected one AWC per block for fieldwork teams to survey 30 pre-school children (six per year of age) 1–5 months after a mass-treatment day, seeking from them blood and faecal samples. In practice, full information with complete assay results was obtained for about 24 children per block. The children surveyed were not randomly chosen and were often taken from AWC lists, hence mainly registered with the AWC. Caregivers were asked whether the previous mass treatment had been received (and, if so, whether any acute illness followed) and about recent (past 4 weeks) illnesses…
  Consistent with this finding, the biomedical surveys of randomly chosen AWCs, which tended to over-sample AWC-registered children, found caregivers reporting 91% (2337/2581) had received retinol on the previous mass-treatment day…
  …Retinol assays were available (only during the second half of the study) for 5165 children with complete information on all assays and questionnaire replies, 2581 in retinol-allocated versus 2584 in control blocks (table 1). Among them, mean retinol was 0.72 (SE 0.01) μmol/L versus 0.62 (0.01) μmol/L, a 16% increase (p<0.00001)." Awasthi et al. 2013a, pp. 1471-1473.

• 128

  Our rough estimate assumes:

  • The overall effect size of non-DEVTA trials is 24% (the main reported effect from Imdad et al. 2010, an earlier version of the meta-analysis that excludes DEVTA).
  • Coverage in non-DEVTA trials was around 87%.

  This implies that for coverage alone to explain the difference between DEVTA and non-DEVTA trials, DEVTA coverage would need to be approximately 15% (4% / (24% / 87%)).

• 129

  "However, the GiveWell document fails to consider the problem of ascertainment bias, which may be the most important concern if substantial mortality went unreported, as is likely the case given the data collection methods. Because of the small number of research staff members, the limited training of the child-care center staff, the small proportion of reported mortality that was confirmed, and the lack of study population enumeration data and population-based mortality assessment, the methods do not seem to be sufficiently rigorous for the results to be plausible (see, for example, comments by Sommer et al, 2013)." Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, 2023 (unpublished).

• 130

  "But this was neither a rigorously conducted nor acceptably executed efficacy trial: children were not enumerated, consented, formally enrolled, or carefully followed up for vital events, which is the reason there is no CONSORT diagram." Sommer, West, and Martorell 2013, p. 1.

• 131

  Deaths were recorded by 18 study monitors going village-to-village by motorcycle. Monitors were to visit all (~500) anganwadi children’s centers over the course of six months and visit and record all households where a child under age 10 had died in the previous year. These checks were conducted again by a different monitor six months later, and 5% of all villages were visited by supervisors as an additional check.
  We would expect that this method would undercount deaths, since monitors are likely to have missed some households (particularly remote households). The number of monitors (18) relative to the study size (around one million children) makes this more likely.
  "Deaths were recorded by 18 full-time motorcycle village-to-village monitors.17 Monitors covered four neighbouring blocks at a time, one in each treatment group, and within 6 months went to all (roughly 500) study AWCs in those blocks to identify and visit all households where in the past year a liveborn infant or child younger than 10 years had died, recording age (often given only approximately, so our analyses of mortality at 1–5 completed years of age include all deaths at recorded ages 1.0–6.0 years), sex, name, parental names, and (by simple verbal autopsy) likely cause." Awasthi et al. 2013a, p. 1471.

• 132

  "A fixed-effect meta-analysis provides a result that may be viewed as a ‘typical intervention effect’ from the studies included in the analysis. In order to calculate a confidence interval for a fixed-effect meta-analysis the assumption is usually made that the true effect of intervention (in both magnitude and direction) is the same value in every study (i.e. fixed across studies). This assumption implies that the observed differences among study results are due solely to the play of chance (i.e. that there is no statistical heterogeneity)." Deeks et al., Cochrane Handbook for Systematic Reviews of Interventions, section 10.10.4, "Incorporating heterogeneity into random-effects models." Accessed May 22nd, 2023.

• 133

  The conventional choice is a normal distribution. According to Cochrane, one criticism of random-effects meta-analysis is that the shape of the distribution is not known, and there is no easy way to validate the choice of any particular distribution.
  "The random-effects meta-analysis approach incorporates an assumption that the different studies are estimating different, yet related, intervention effects (DerSimonian and Laird 1986, Borenstein et al 2010). The approach allows us to address heterogeneity that cannot readily be explained by other factors. A random-effects meta-analysis model involves an assumption that the effects being estimated in the different studies follow some distribution. The model represents our lack of knowledge about why real, or apparent, intervention effects differ, by considering the differences as if they were random. The centre of the assumed distribution describes the average of the effects, while its width describes the degree of heterogeneity. The conventional choice of distribution is a normal distribution. It is difficult to establish the validity of any particular distributional assumption, and this is a common criticism of random-effects meta-analyses. The importance of the assumed shape for this distribution has not been widely studied." Deeks et al., Cochrane Handbook for Systematic Reviews of Interventions, section 10.10.4, "Incorporating heterogeneity into random-effects models." Accessed May 22nd, 2023.

• 134

  "A random-effects model provides a result that may be viewed as an ‘average intervention effect’, where this average is explicitly defined according to an assumed distribution of effects across studies. Instead of assuming that the intervention effects are the same, we assume that they follow (usually) a normal distribution. The assumption implies that the observed differences among study results are due to a combination of the play of chance and some genuine variation in the intervention effects." Deeks et al., Cochrane Handbook for Systematic Reviews of Interventions, section 10.10.4, "Incorporating heterogeneity into random-effects models." Accessed May 22nd, 2023.

• 135

  DEVTA receives 61.7% of the weight in the fixed-effects analysis, and 13.6% of the weight in the random-effects analysis.

  • Meta-analysis weights for the fixed-effects analysis are in Imdad et al. 2017, Analysis 1.1, p. 90.
  • Meta-analysis weights from the random-effects analysis were derived from downloading the RevMan file published alongside Imdad et al. 2017, which can be found under the "Download data" link here.

• 136

  See this row in our cost-effectiveness analysis.

• 137

  "A random-effects model provides a result that may be viewed as an ‘average intervention effect’, where this average is explicitly defined according to an assumed distribution of effects across studies. Instead of assuming that the intervention effects are the same, we assume that they follow (usually) a normal distribution. The assumption implies that the observed differences among study results are due to a combination of the play of chance and some genuine variation in the intervention effects." Deeks et al., Cochrane Handbook for Systematic Reviews of Interventions, section 10.10.4, "Incorporating heterogeneity into random-effects models." Accessed May 22nd, 2023.

• 138

  For summaries of the variation between studies, see:

  • GiveWell, Vitamin A deficiency and vitamin A fortification [2019] for a comparison of locations, dates and vitamin A deficiency rates between studies.
  • GiveWell, Round 1 VAS effect size estimate, 2024 for a comparison of dosages and treatment schedules between studies.
  • Table 1 above for a comparison of baseline mortality rates in a sample of studies which receive high weight in the meta-analysis.

• 139

  Note that this may be partially offset by a reduction in our internal validity adjustment (one component of which assesses what proportion of the mortality reduction in Imdad et al. 2017 can be explained by reductions in measles and diarrhea). See this section for further details on how this adjustment works.

• 140

  See this row of our cost-effectiveness analysis for our calculation.

• 141
  • Most studies did not report how coverage was measured or how coverage data was collected (see this column of our coverage estimates analysis for details by study). We would guess that in many studies this data was based on self-reports from caregivers. This kind of data is at risk of being inflated by social desirability bias (survey respondents being more likely to overreport "good" behaviors). This implies we could be overestimating coverage (and therefore underestimating the impact of VAS on children who are treated).
  • This adjustment assumes that the findings reported in Imdad et al. 2017 are "intention-to-treat" estimates (including all children originally allocated to receive the intervention) rather than "per-protocol" estimates (including only children who actually received VAS). Imdad et al. does not report which type of analysis was used for each study. We assume that most of the effects included in Imdad et al. 2017 are intention-to-treat estimates because these are the most common estimates of interest according to the Cochrane Handbook, but we have not reviewed the underlying studies in detail to confirm this.
    • "The effect of interest in any particular analysis of a randomized trial is usually either the effect of assignment to intervention (the ‘intention-to-treat’ effect) or the effect of adhering to intervention (the ‘per-protocol’ effect). These effects are discussed in Chapter 8, Section 8.2.2. The data collected for inclusion in a systematic review, and the computations performed to produce effect estimates, will differ according to the effect of interest to the review authors. Most often in Cochrane Reviews the effect of interest will be the effect of assignment to intervention, for which an intention-to-treat analysis will be sought. Most of this chapter relates to this situation. However, specific analyses that have estimated the effect of adherence to intervention may be encountered." Higgins et al., Cochrane Handbook for Systematic Reviews of Interventions, "Chapter 6: Choosing effect measures and computing estimates of effect." Accessed August 16th, 2023.
  • This conversion does not take possible spillover effects into account (i.e., people who don’t receive VAS may nonetheless gain some protection from VAS campaigns, if VAS reduces overall transmission of infectious diseases). Imdad et al. reports that VAS reduces incidence of some infectious diseases (diarrhea and measles), so it may be that we should expect spillover effects, but we haven’t investigated this question in detail. If there were spillover effects, this could affect our estimates in two ways:
    • Spillovers to the control group, which could lead us to underestimate the effect of VAS. Our current adjustment assumes that children in the control group got no benefit. If there were spillovers to the control group, we would be underestimating the benefit of VAS. All of the top five most highly weighted trials in the meta-analysis were cluster RCTs that randomized entire communities for treatment, rather than individuals, and so we’d expect this to be unlikely (we haven’t checked for all RCTs in the meta-analysis).
    • Spillovers to children who did not receive VAS in the treatment group, which could lead us to over- or underestimate the effect of VAS. Our conversion implicitly assumes that children in the treatment group who did not receive VAS received no benefit, and all the benefits of VAS were concentrated among the children who did receive VAS. In fact, some children who did not receive VAS may have received benefits through reduced disease transmission. We would expect this not to bias our results if coverage of VAS was similar between trial contexts and contemporary VAS programs (because the conversion doesn’t affect the overall benefit of VAS, just how it is allocated between children who do and don’t receive VAS). But the conversion could yield inaccurate estimates of the impact of VAS if coverage rates differed significantly between trial contexts and contemporary VAS campaigns (due to varying transmission patterns). We’re unsure whether this would positively or negatively update our analysis.

• 142

  See this row in our cost-effectiveness analysis.

• 143

  The meta-analysis finds that VAS reduces mortality by 24%, compared to 17% for a Cochrane meta-analysis of insecticide-treated nets (one of GiveWell’s other top recommended programs as of February 2024) and 14% for Azithromycin. See this row in our supplementary analysis for further details. While this is not evidence in and of itself that the VAS estimate is too high, it did contribute to us wanting to conduct sense checks.

• 144

  See this section of the report for details on how we reach this estimate.

• 145

  We think that 59% of children in the underlying studies were VAD, the impact on mortality was 24%, 87% of children in the underlying studies received VAS, and we roughly guess that the mortality impact of VAS was only ⅓ as large for children without VAD. (24% / 87%) / (59% + ((100% - 59%) x 33%)) = ~38%. See this section of our supplementary analysis for this calculation.

• 146

  We think that 59% of children in the underlying studies were VAD, the impact on mortality was 24%, and 87% of children in the underlying studies received VAS. (24% / 87%) / 59% = ~47%. See this sheet of our supplementary analysis for rough calculations.

• 147

  See this sheet of our supplementary analysis for more details. Studies with relative risks of approximately 1:

  • Agarwal 1995: RR 1.22 [0.66, 2.25]
  • Barreto 1994: RR 1.00 [0.14, 7.08]
  • DEVTA trial 2013: RR 0.96 [0.89, 1.03]
  • Herrera 1992: RR 1.06 [0.82, 1.37]
  • Fisker 2014: RR 0.93 [0.66, 1.31]
  • Vijayaraghavan 1990: RR 1.02 [0.57, 1.82]

• 148

  Dr. Meager is an Associate Professor at the University of New South Wales.

• 149

  Dr. Więcek is the Director of WAW Statistical Consulting, and Consulting Director at the Development Innovation Lab, University of Chicago.

• 150

  This understanding is based on unpublished correspondence with Dr. Meager and Dr. Więcek. Our understanding is that this is still an interim analysis, and has not yet been published or peer reviewed. As a result, we do not report the authors’ updated estimate of the VAS effect on mortality, as we think it’s possible that this may change.

• 151

  See Imdad et al. 2017, pp. 19-20. "To test for small study bias, we repeated the analysis using a random-effects model. The overall estimate was identical to the fixed-effect estimate, though the CI widened compared to the fixed effect model, suggesting that heterogeneity is not explained by small studies reporting larger effects (RR 0.84, 95% CI 0.73 to 0.98; 15 studies). See Table 1. The funnel plot we produced was dominated by two studies accounting for 74% of the overall effect (Figure 5), and the plot was relatively flat."

• 152

  "In summary, the primary outcome was at low risk of bias, and the size and the significance of the effect cannot be explained by bias. While there was some evidence of small study bias for secondary outcomes, further research is unlikely to change the conclusion that VAS, delivered with high quality and coverage, prevents death among children aged 6 to 59 months in low- and middle-income countries." Imdad et. al. 2017, p. 23.
  The full assessment of risks of bias for each trial is in Imdad et. al. 2017, pp. 14-16.

• 153

  See this row of our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 154

  We include this adjustment because we expect that these changes to be correlated to some extent and therefore we would risk "double penalizing" our adjustment by simply multiplying each adjustment together. See this section of the report for further discussion. See this row in our cost effectiveness analysis for this adjustment.

• 155

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 156

  See this cell in our cost-effectiveness analysis, and this sheet in our supplementary analysis for our calculations.

• 157

  We use the same random-effects weights we use for our main analysis of the impact of VAS. See this column in our supplementary analysis for these weights.

• 158

  Where VAD prevalence (as measured by blood serum retinol concentration) data was not available, we used a combination of the following:

  • Rates of xerophthalmia, if measured in the trial.
  • Data on the most recent (as of 1995, close to the time of many of the trials) national-level surveys on serum retinol levels recorded in WHO, Global Prevalence of Vitamin A Deficiency, 1995.
  • Regional-level (e.g., sub-Saharan Africa) estimates of the prevalence of VAD in 1991 (close to the time of many of the trials) in Stevens et al. 2015.

  Of the 19 studies included in our analysis:

  • Five studies reported data on baseline VAD rates (DEVTA, Dibley 1996, Donnen 1998, Ross et al. 1993 Health and Ross et al. 1993 Survival).
  • Six studies reported data on baseline xerophthalmia (dry eyes, a symptom of VAD) rates, but not VAD itself (Agarwal 1995, Benn 1997, Daulaire 1992, Herrera 1992, Sommer 1986, and West 1991). Xerophthalmia is a clinical indicator of VAD and can be used to estimate deficiency (see WHO, Global prevalence of vitamin A deficiency in populations at risk 1995–2005, 2009, p. 2). We extrapolate the rates of xerophthalmia in these studies to VAD rates based on a very rough analysis of VAD rates from a 2015 study of VAD prevalence over time (Stevens et al. 2015).
  • Seven studies do not report data on either xerophthalmia or VAD (Barreto 1994, Chowdhury 2002, Fisker 2014, Pant 1996, Rahmathullah 1990, Venkatarao 1996, and Vijayaraghavan 1990). In these cases we use estimates from either Stevens et al. 2015 or a 2009 WHO estimate of VAD in populations at risk, depending on which source provides an estimate closer to the country and year in question.
  • One study (Lin 2008) receives no weight in the meta-analysis, so we exclude it from our estimate.

  See these columns in our supplementary analysis for our reasoning in each case.

• 159

  See this row of our analysis for location-specific values. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 160

  See this row in our cost-effectiveness analysis.

• 161

  See this row in our cost-effectiveness analysis. The most recent survey dates for each country were: Burkina Faso (2010), Cameroon (2009), Côte d'Ivoire (2007), DRC (1999), Kenya (2011), Mali (1997), and Mozambique (2002).

• 162

  Specifically, we assume that VAD rates fall by 1.33% per year between the date the survey was conducted and the date we conducted our analysis (2022). See this row of our cost-effectiveness analysis. This figure is based on the annualized rate of change in three measures that we think might be a proxy for VAD rates (stunting, acute malnutrition, and poverty) between national health surveys conducted in Nigeria in 2003 and 2018. We use estimates from Nigeria only and apply them to other countries because this adjustment is relatively small and have not yet prioritized it.
  See this section of our separate analysis of VAD in Nigeria for further details. Note that we have not validated how far each of these proxy measures is correlated with VAD, and we assume that the rate of improving circumstances in other countries is the same as in the Nigeria data. We therefore think of this analysis as a rough and uncertain best guess.

• 163

  Reasons for our low confidence in the Global Burden of Disease Project state-level estimates of VAD in Nigeria:

  • We investigated the state-level VAD estimates for Nigeria and found them to be implausibly low (2% to 4% in many states, see this spreadsheet).
  • These low rates are in tension with what we would expect, based on other sources of information:
    • They are considerably lower than VAD rates in other sub-Saharan African countries where Helen Keller and Nutrition International works (in the ~25% to 35% range in GBD 2017 - see this row of our cost-effectiveness analysis).
    • They are considerably lower than the level recorded in the most recent nationally representative survey in Nigeria that we have found (30%). The survey was conducted in 2001, and while we might expect some fall in VAD over time, we think this fall is too dramatic to be plausible.
  • We’re unsure what modeling changes were driving this update between GBD 2017 and GBD 2019.

• 164

  Maziya-Dixon et al. 2006 is the most recent nationally representative VAD survey we are aware of, and was conducted in 2001.

• 165

  These estimates are compiled in this section of the CEA. The 2018 Nigeria DHS survey is available here. For measures of poverty, we use the Nigerian Living Standards Survey (NLSS) estimates reported in the 2019 Poverty and Inequality in Nigeria Summary Report (available here, downloaded Feb 28th, 2022).

• 166

  See these rows in our cost-effectiveness analysis for our location-specific weights.
  Note that in Kenya, there are particularly large differences between VAD prevalence rates in the GBD estimates and the most recent national survey. The GBD 2017 estimate of VAD prevalence for 6-59 month olds is approximately 33%, compared to 9.2% in the Kenya 2011 national survey.
  We conducted an in-depth review of the 2011 survey to investigate the reliability of the data. We found several potential issues with the survey data and methodology, but nothing to justify ignoring the data altogether. We therefore decided to assign 50/50 weight to each source.
  See this document for further details on our reasoning and this column in our cost-effectiveness analysis for our calculations.

• 167

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 168

  See these rows in our cost-effectiveness analysis.

• 169

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 170

  "The evidence I identified supports a substantial effect of VAS on mortality from diarrhea and measles, but does not support a substantial effect of VAS on other prominent infectious causes of mortality in under-five children in sub-Saharan Africa. Reasoning:

  • Imdad et al. 2017 is a Cochrane meta-analysis that reports that mass VAS reduces diarrhea mortality (relative risk 0.88; 95% CI 0.75 to 0.98) and incidence (relative risk 0.85; 95% CI 0.82 to 0.87) in under-five children in RCTs. These estimates are statistically significant, have relatively tight confidence intervals, and were judged "high-quality" and "low-quality", respectively.
  • Imdad et al. 2017 reports that mass VAS (non significantly) tends to reduce measles mortality (relative risk of 0.88; 95% CI 0.69 to 1.11) and greatly reduces measles incidence (relative risk of 0.50; 95% CI 0.37 to 0.67). These estimates were judged "low-quality" and "high-quality", respectively. Other meta-analyses of mass VAS are consistent with these estimates. In the meta-analysis Sudfeld et al. 2010, VAS tended to reduce measles morbidity in children already infected with measles (relative risk of 0.63; 95% CI 0.37 to 1.08) and reduced it quite substantially in the three trials that provided higher doses of vitamin A (relative risk of 0.38; 95% CI 0.18 to 0.81). Together with the fact that the evidence underlying the Imdad et al. 2017 mortality estimate is judged "low-quality", this suggests to me that mass VAS probably reduces measles mortality, and likely to a greater degree than Imdad et al. 2017 suggests. I favor a relative risk of 0.67 to 0.69, but this relies on uncertain judgment calls."

  GiveWell, Disease-specific mortality effects of vitamin A supplementation, 2019, p. 2.

• 171
  • "VAS probably does not meaningfully impact lower respiratory tract infection (pneumonia) mortality, but I do not believe this with high certainty. Evidence from meta-analyses of VAS RCTs appears consistent that VAS does not reduce LRTI mortality or incidence. The confidence interval in Imdad et al. 2017 is relatively tight (relative risk of 0.98; 95% CI 0.86 to 1.12), suggesting little room for a meaningful effect ("low-quality" evidence). In contrast to mortality, Imdad et al. 2017 reports that VAS may substantially reduce LRTI prevalence (relative risk of 0.60; 95% CI 0.45 to 0.81), and I am uncertain how to reconcile this with the mortality estimate. However, this latter estimate is based on only two trials, while the former is based on nine trials (both are dominated by DEVTA).
  • I do not recommend including malaria mortality in our model, but I am uncertain about this due to a possible discrepancy between mortality and incidence/morbidity estimates. The most pertinent RCT did not identify an impact of VAS on malaria mortality (relative risk 1.03; 95% CI 0.74 to 1.43). However, Imdad et al. 2017 reports a significant reduction in malaria incidence (relative risk of 0.73; 95% CI 0.60 to 0.88; based on one trial), and of the three pertinent RCTs reporting malaria morbidity, one reports a significant reduction and another reports a nonsignificant reduction of a similar magnitude (the third, with very wide confidence intervals, reports a nonsignificant increase). An additional meta-analysis reports no significant impact of VAS on placental malaria risk in pregnant women (relative risk of 1.09; 95% CI 0.95 to 1.25; based on two trials). Overall, I believe the evidence weakly suggests that mass VAS probably does not substantially impact malaria mortality."

  GiveWell, Disease-specific mortality effects of vitamin A supplementation, 2019, pp. 2-3.

• 172
  • "I was unable to evaluate the impact of VAS on whooping cough incidence or mortality due to a lack of evidence.
  • The evidence I encountered on VAS and tuberculosis was insufficient to evaluate whether VAS impacts tuberculosis mortality.
  • I was unable to evaluate the impact of VAS on invasive non-typhoidal salmonella incidence or mortality due to a lack of evidence."

  GiveWell, Disease-specific mortality effects of vitamin A supplementation, 2019, p. 3.

• 173

  The review tentatively concludes: "I recommend two possible approaches for our external validity adjustment. Of these, I slightly prefer the second.

  • Assume that VAS only impacts diarrhea and measles mortality.
  • Assume that the mortality benefit of VAS is primarily expressed via diarrhea and measles mortality, but also has a small effect on overall infectious disease mortality (due to impacts on specific VAS-responsive infectious diseases that occur in sub-Saharan Africa but whose VAS-responsiveness I have not been able to identify here). I am uncertain how much to weight each cause, but 85 percent of weight on diarrhea and measles, and 15 percent of weight on everything else seems reasonable to me. This is in contrast to our current model, which weights each of these at 50 percent. I slightly favor this model because there are many possible infectious causes of death with insufficient evidence to determine how they are impacted by vitamin A status. It seems more likely than not that other infectious diseases may contribute to the mortality reduction effect of VAS to some degree."

  GiveWell, Disease-specific mortality effects of vitamin A supplementation, 2019, p. 4.
  Note that we have since updated our best guess of the relative contribution of diarrhea / measles and other infectious diseases from 85% / 15% to 80% / 20%.

• 174

  "I am slightly inclined to favor the latter approach because, with the many possible infectious causes of death it seems more likely than not that VAS reduces mortality from other less impactful diseases for which there is insufficient evidence to observe this effect. However, I am not certain which model is more accurate." GiveWell, Disease-specific mortality effects of vitamin A supplementation, 2019, p. 34.

• 175

  See this section in our cost-effectiveness analysis.

• 176

  See this sheet in our analysis of baseline mortality for our calculations. Note that:

  • GBD data is only available beginning in 1990. We use data from 1990 for trials that took place before 1990.
  • These estimates could be biased up or down if the studies took place in locations with better or worse child health than the national average.
  • Our overall estimate is a weighted average, using the study weights in Imdad et al. 2017 (random-effects analysis).

• 177

  See these rows in our cost-effectiveness analysis.

• 178

  See these rows in our cost-effectiveness analysis.

• 179

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 180

  Our current non-independence "gives back" 25% of any penalty for relative changes in causes of mortality (e.g., the -52% adjustment for changes in the causes of mortality in Guinea is effectively cut to -39%).
  We don’t know how much we should expect falls in VAD and changes in the causes of child mortality to be correlated. The two possible extremes are:

  • The two changes are totally dependent. In this case, we would take their average for our adjustment (-52% + -53%) / 2 = -52.5%.
  • The two changes are totally independent. In this case, we would take their product for our adjustment (100% - 52%) x (100% - 53%) - 100% = -78%.

  We think of our 25% adjustment as a rough best guess. See this row in our cost-effectiveness analysis.
  See these rows of our external validity analysis for further details on our reasoning and the specific calculation.

• 181

  DEVTA estimated a relative risk for the impact of VAS on mortality as 0.96, and Fisker et al. 2014 estimated a relative risk of 0.93 (more in our supplementary analysis here).

  • DEVTA: We discuss our concerns about DEVTA in the section above.
  • Fisker et al. 2014:
    • Additional VAS campaigns were conducted during the trial, and so some children (in both treatment and control groups) received VAS more than once. We do not know what proportion of children this applies to, although we note that the number of person-years of observation falls by roughly half between the analysis conducted at 6 months and to the next VAS campaign, see figure 2. We expect that this would attenuate any measurable impact of receiving VAS relative to being in the control group.
    • To account for this, the study authors conducted an analysis of mortality during only the first six months of the trial (although some children in the control group still received VAS in this period, and we’re unsure how many). The overall number of deaths in this period was low (80 non-accident deaths in total), so this analysis has low statistical power (95% CI 0.59 - 1.41).

  "The unforeseen increased frequency of VAS campaigns posed analytical challenges. Information on campaign VAS was obtained during home visits and thus available only for surviving children. Using the information obtained during visits could introduce bias, and censoring all children at the date of a campaign severely reduced the follow-up time and compromised the power. Conducting follow-up to 12 months of age would mean that most children had received VAS twice during follow-up and this would lead to dilution of any effect of the intervention. Hence, we conducted 2 types of analyses. In the main analysis, we followed children for 6 months after enrollment, irrespective of subsequent VAS; this main analysis includes some follow-up time after campaign VAS which may have diluted the effect of the experimental treatment."
  "Between August 2007 and November 2010, 7587 children were enrolled. Within 6 months of follow-up 80 nonaccident deaths occurred (VAS: 38; placebo: 42). The mortality rate ratio (MRR) comparing VAS versus placebo recipients was 0.91 (95% confidence interval 0.59-1.41) and differed significantly between boys (MRR 1.92 [0.98-3.75]) and girls (MRR 0.45 [0.24-0.87]) (P = .003 for interaction between VAS and gender)."
  Fisker et al. 2014, e742, e739.

• 182

  See this column in our analysis of the estimated effect sizes in each trial.

• 183

  We performed weighted regression using the study weights from the random-effects analysis of the Imdad et al. 2022 meta-analysis. The correlation between predicted and observed relative risk is negligible (p = 0.80, R2 = 0.004). This analysis is currently unpublished.

• 184

  Supplementation intervals for each trial can be found here.

• 185

  This analysis is currently unpublished.

• 186

  Several experts have told us that the impact of VAS on vitamin A status lasts about two to four months, depending on how it is measured (serum retinol vs. the modified relative dose response test). This is substantially shorter than the typical supplementation interval of 6 months, suggesting that VAS does not sustain vitamin A levels for the entire 6-month period.
  "It can also be helpful to wait a few months after supplementation has occurred to take measurements – for a serum retinol survey, for example, the effects of VAS fade within two months, so any survey conducted more than two months after supplementation should reflect the population’s serum retinol levels independent of the effect of vitamin A supplementation." Givewell's non-verbatim summary of a conversation with Dr. Sherry Tanumihardjo, March 26, 2018.
  We have also heard from other (unpublished) expert conversations that the impact of VAS on vitamin A status is no longer detectable after six to eight weeks using retinol binding protein or serum retinol, and three to four months using modified relative dose response (more on methods for measuring VAD here).
  This is also consistent with data from a large-scale supplementation campaign in the Philippines, that receipt of VAS is associated with a 21% relative reduction in vitamin A deficiency rates measured by serum retinol at 1-2 months after supplementation, a 14% relative reduction at 3-4 months, and no apparent reduction at 5-6 months (1.4%). This is calculated by dividing the rate of children with vitamin A deficiency in each post-supplementation category by the rate among children who did not receive VAS. Data are from table 2 of Pedro et al. 2004.
  Pedro et al. 2004, table 2, p. 323.

• 187
  • A controlled, non-randomized trial of vitamin A fortification of monosodium glutamate in Indonesia reports a mortality RR of 0.55 in the 12-60 month age group. The paper reports the odds ratio for mortality, and this is statistically significant. To calculate RR, we use the mortality rate for intervention and control groups in table 6, and divide the former by the latter. 17 / 31 = 0.55.

  The odds ratio for under-5 mortality in children 12-60 months is 1.87 [1.41, 2.48] (table 6). The denominator and numerator are reversed relative to how they are typically reported (the inverse is an OR of 0.53). We assume the finding would also be statistically significant if expressed as RR.
  "Preschool children in control villages died at 1.8 times the rate of children in program villages." Muhilal et al. 1988, abstract.

  • One of the VAS trials included in the Imdad et al. 2022 meta-analysis has a nutrition education arm intended to increase vitamin A intake (excluded from the meta-analysis because it was not a VAS intervention). The trial reports a large reduction in under-5 mortality (RR 0.64; 95% Cl 0.48-0.86). "The risk of mortality at 2 years was reduced for both the nutrition education (RR = 0.64; 95% Cl = 0.48-0.86) and capsule distribution (RR = 0.57; 95% CI = 0.42-0.77) cohorts." Pant et al. 1996, abstract.

• 188

  This analysis is currently unpublished.

• 189

  This understanding is based on conversations with Dr. Sherry Tanumihardjo, a researcher specializing in vitamin A.
  "Serum retinol concentration, measured in blood samples, is the most commonly used vitamin A status biomarker. When serum retinol concentrations are below 0.7 μmol/L, individuals are considered to be vitamin A deficient, but the World Health Organization recommends that another biomarker be used to confirm deficiency. Concentrations of retinol-binding protein (RBP) are sometimes used as a proxy for serum retinol concentrations, because tests measuring RBP are less expensive and easier to implement than serum retinol tests. But unlike serum retinol, there is not a commonly accepted deficiency cutoff for measures of RBP and it may differ depending on the assay used."
  "Serum retinol and RBP both have limitations as biomarkers for indicating vitamin A status:

  • Both of these biomarkers are responsive to inflammation, such that a low level of the biomarker could indicate either VAD or an acute phase response to infection. It may be more accurate to think of serum retinol and RBP as biomarkers of inflammation rather than of VAD.
  • In the beginning stages of VAD, levels of serum retinol and RBP may increase as the body attempts to "recycle" vitamin A.

  The results of surveys of serum retinol or RBP are sometimes adjusted to account for inflammation. If no adjustments are used, serum retinol and RBP surveys would likely overestimate the prevalence of vitamin A deficiency, because some individuals' concentrations of serum retinol or RBP would be low due to inflammation, rather than chronic low vitamin A intake." GiveWell's non-verbatim summary of conversations with Dr. Sherry Tanumihardjo, October 17 and 27, 2017, p. 2.

• 190

  "A better measure of VAD to use going forward would be the modified relative dose response (MRDR) test, because it is not as sensitive to inflammation." GiveWell's non-verbatim summary of conversations with Dr. Sherry Tanumihardjo, October 17 and 27, 2017, p. 2.

• 191

  See the highlighted cells of this column in our supplementary analysis for these studies.

• 192

  Dr. West told us that "no single indicator is ideal for all purposes. MRDR has become a gold standard for indirectly estimating liver vitamin A stores, but requires repeat visits, can be costly and less available, while a serum retinol distribution, when low as measured in labs throughout the world, reflects a vitamin A-stressed population likely to experience improved health, and perhaps survival, following vitamin A intervention." Keith West, email to GiveWell, February 19th 2024.

• 193

  For details of these methods (accurate as of GBD 2017, see GiveWell's non-verbatim summary of a conversation with the Institute for Health Metrics and Evaluation, April 5, 2019, p. 2.
  "WHO collects results from VAD surveys in its Vitamin and Mineral Nutrition Information System (VMNIS). IHME uses these results, along with additional survey results from Demographic and Health Surveys (DHS) and other sources, as inputs for its VAD prevalence estimations. For many countries, there is either no VAD survey data available or the available data is more than 10 years old. To estimate VAD prevalence for countries and years without available survey data, IHME uses the following covariates in a predictive model (all of which are inversely related to VAD prevalence):

  • VAS coverage – IHME uses UNICEF data on the proportion of targeted populations of preschool-aged children that receive vitamin A supplements.
  • Socio-demographic Index (SDI) – IHME developed SDI as a composite of income levels, educational attainment, and fertility rates.
  • Vitamin A availability in food – In collaboration with the Food and Agriculture Organization of the United Nations, IHME developed a global database (encompassing all 195 GBD countries) for the availability of 170 different nutrients. In its model of VAD prevalence, IHME treats vitamin A availability in national food supplies as a proxy for vitamin A consumption, but it does not account for consumption of vitamin A fortified foods—fortification data is often either unavailable or incomplete."

• 194

  This page includes a description of the changes in modeling assumptions between GBD 2017 and GBD 2019 for VAD. We don't have a full understanding of why these modeling changes led to such big shifts in VAD prevalence estimates.

• 195

  "In collaboration with the Food and Agriculture Organization of the United Nations, IHME developed a global database (encompassing all 195 GBD countries) for the availability of 170 different nutrients. In its model of VAD prevalence, IHME treats vitamin A availability in national food supplies as a proxy for vitamin A consumption, but it does not account for consumption of vitamin A fortified foods—fortification data is often either unavailable or incomplete." GiveWell's non-verbatim summary of a conversation with the Institute for Health Metrics and Evaluation, April 5, 2019, p. 2.

• 196

  "For many countries, there is either no VAD survey data available or the available data is more than 10 years old. To estimate VAD prevalence for countries and years without available survey data, IHME uses the following covariates in a predictive model (all of which are inversely related to VAD prevalence):

  • VAS coverage – IHME uses UNICEF data on the proportion of targeted populations of preschool-aged children that receive vitamin A supplements.
  • Socio-demographic Index (SDI) – IHME developed SDI as a composite of income levels, educational attainment, and fertility rates.
  • Vitamin A availability in food – In collaboration with the Food and Agriculture Organization of the United Nations, IHME developed a global database (encompassing all 195 GBD countries) for the availability of 170 different nutrients. In its model of VAD prevalence, IHME treats vitamin A availability in national food supplies as a proxy for vitamin A consumption, but it does not account for consumption of vitamin A fortified foods—fortification data is often either unavailable or incomplete." GiveWell's non-verbatim summary of a conversation with the Institute for Health Metrics and Evaluation, April 5, 2019, p. 2.

• 197

  We currently do include an adjustment like this for our estimates of child mortality, but not for VAD. See the discussion above for further details.

• 198

  Note: one reason to believe that this may not lead to significantly different estimates of VAD is that measures of serum retinol or retinol-binding protein, the main biomarkers used to evaluate VAD, do not appear to be very responsive to VAS. Instead, our understanding is that these biomarkers are more responsive to dietary intake of vitamin A.
  "Serum retinol distribution curves are used to evaluate program impact (111). However, the lack of change in serum retinol distribution over time in several countries that have sustained >70% coverage with vitamin A supplementation has raised the concern about the appropriate indicator (219). For this reason, the impact of supplementation programs is not measured by a change in the prevalence of low serum retinol concentrations but may be better served by evaluating coverage rates. Retinol concentrations may respond to sustained, improved dietary intakes and therefore can guide programmatic decisions about whether to maintain or change interventions (219). Thus, the use of serum retinol distributions among preschool children from cross-sectional surveys to assess the need for vitamin A interventions is still recommended." Tanumihardjo et al. 2016, pp. 16S-17S.
  "There is a growing interest in measuring the impact of VA programs in countries that have implemented national-scale programs for several years. Serum retinol concentrations do not respond to VAS, except in a transient manner (ie, for 1-2 months). While the kinetics of this transient effect have not been well characterized, it presumably reflects the rapid use of VA to support its biological functions when background dietary intake is low and/or VA losses resulting from infections. Serum retinol concentration is therefore not recommended as an impact indicator where VAS is the only strategy for addressing VAD. For this reason, the impact of VAS programs is not measured by a change in VAD prevalence in the population, and the mortality impact is instead modeled using coverage data. Serum retinol concentrations are, however, responsive to improved dietary intakes, sustained over time, and therefore can guide programmatic decisions about whether to maintain or change intervention mixes. Thus, using serum retinol distributions among preschool-aged children— in conjunction with other vitamin A status markers or demographic/ecologic risk factors—from cross-sectional surveys to assess the need for VA interventions is still recommended, even in countries that have sustained high semiannual VAS coverage over several years." Klemm et al. 2016, pp. 5-6.

• 199

  See this row of our cost-effectiveness analysis. The most recent survey dates for each country were: Burkina Faso (2010), Cameroon (2009), Côte d'Ivoire (2007), DRC (1999), Kenya (2011), Mali (1997), and Mozambique (2002).

• 200

  See this cell in our cost-effectiveness analysis and the section above for a discussion of this adjustment.

• 201

  Stevens et al. 2015, table 1, e531.

• 202

  See this spreadsheet for this list of surveys included in Stevens et al. 2015.

• 203

  The surveys produced estimates of 17% in Sierra Leone, 4% in Malawi, and 21% in Ghana. See this column in our 2019 analysis.

• 204

  See Wirth et al. 2017, pp. 6-7, for a list of countries in which biofortified crop programs have been implemented.

• 205

  See this column in our 2019 analysis for details.

• 206
  • "We conducted representative surveys in Yaoundé and Douala, Cameroon, 2 years before and 1 year after the introduction of a mandatory national program to fortify cooking oil with VA. In each survey, 10 different households were selected within each of the same 30 clusters (n = ~300). Malaria infection and plasma indicators of inflammation and VA (retinol-binding protein, pRBP) status were assessed among women aged 15–49 years and children aged 12–59 months, and casual breast milk samples were collected for VA and fat measurements. Refined oil intake was measured by a food frequency questionnaire, and VA was measured in household oil samples post-fortification." Engle-Stone et al. 2017, p. 1.
  • Adjusted prevalence of VAD among preschool-aged children (26.6%) and unadjusted prevalence of VAD among preschool-aged children (41.2%) in 2012 reported in Engle-Stone et al. 2017, p. 10, Table 4.
  • Differences between 2009 and 2012 surveys of VAD prevalence and mean RBP among preschool-aged children were not statistically significant.

• 207

  "There is likely a threshold of VAD prevalence below which VAS is unlikely to have much impact on mortality. If there is high-quality data showing low VAD in a region, Helen Keller thinks it is reasonable not to expect VAS to have a mortality impact there.
  "Organizations in the Global Alliance for Vitamin A (GAVA) currently use 10% VAD as the threshold at or above which VAS programs ought to be maintained in a region. The World Health Organization (WHO) classifies VAD rates of 20% or greater among preschool-aged children as a serious public health problem. VAD rates of less than 5% are accepted as not much of a concern.
  "Despite a lack of recent micronutrient analyses in many African countries, HKI is confident that VAD is prevalent enough in many places for VAS to remain an impactful intervention. For instance, while HKI is not aware of any recent micronutrient deficiency data in Mali, it would be surprising if VAD were not prevalent there, given Mali's child mortality and malnutrition rates." GiveWell's non-verbatim summary of a conversation with Helen Keller International, June 1, 2017, p. 2.

• 208

  "Despite the lack of data, Dr. Tanumihardjo thinks it is unlikely that oil fortification programs across sub-Saharan Africa are working well enough to render VAS programs unnecessary in most countries, given that many of the oil fortification programs are relatively new. Over the next few years, we may gain enough data on rates of VAD to make an informed decision about whether to continue or scale back VAS programs. If there were strong evidence that a country's vitamin A fortification program was effectively fortifying food and reaching target populations, it may be appropriate to scale back the programs. Dr. Tanumihardjo thinks it would be premature to start scaling back VAS programs before we have these data." GiveWell's non-verbatim summary of conversations with Sherry Tanumihardjo, October 17 and 27, 2017, p. 2.

• 209

  This estimate factored in some expected reduction in deficiency rates due to fortification, although we were very unsure about the magnitude of this because of the limited evidence available. See this column in our 2018 analysis.

• 210

  In 2019, following a conversation with the Institute for Health Metrics and Evaluation (IHME), we decided to instead use VAD prevalence estimates from its GBD project instead of our previous estimates. These were higher than our previous estimates (see this column for our 2019 estimates, and this column for our 2018 estimates). We guessed that IHME's estimates of the prevalence of VAD are more likely to be accurate than our subjective best-guesses, but a significant weakness was that IHME told us that its model does not take vitamin A food fortification into account.

• 211

  To understand how the variation in the external validity adjustment affects each location’s cost-effectiveness, we put together a undefinedrough sensitivity analysis. We found that cost-effectiveness is sensitive to the external validity adjustment:

  • All locations we analyzed were near or above our cost-effectiveness bar (10x or more as cost-effective as direct cash transfers at the time of writing) when each location’s external validity adjustment was set to the smallest adjustment we estimate across countries (-41% in Madagascar). See this row in our sensitivity analysis.
  • All locations but two fell below our cost-effectiveness bar when each location’s external validity adjustment was set to the largest adjustment we estimate across countries (-88% in Anambra state, Nigeria). See undefinedthis row of our sensitivity analysis.

  As a point of comparison, at the time of writing we estimate that roughly half the locations in our main cost-effectiveness analysis are above our 10x cost-effectiveness bar. While we’d expect the factors in our external validity adjustment (vitamin A deficiency rates and measles, diarrhea, and other infectious disease mortality) to vary substantially across locations, we’re unsure if we’re overestimating the true differences.

• 212

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 213

  Our estimates are in this sheet of our cost-effectiveness analysis for all countries other than Cameroon and Madagascar, and this spreadsheet for Cameroon and Madagascar. Our adjustments are:

  • Cameroon: We use a weighted average of mortality rates in four regions of Cameroon (Centre, Littoral, Ouest, Sud), which we think are 14% lower than the national average. We use these regions because these are the regions where Helen Keller supported VAS campaigns in the first half of 2021. Since then, Helen Keller has expanded to support all regions (see Helen Keller International, Room for More Funding Report, 2023, p. 18. We have not yet updated our analysis to account for this change).
  • Madagascar: We estimate that mortality rates are 17% higher in the six regions Helen Keller's program will support in Madagascar than in Madagascar as a whole. For this list of regions, see the highlighted cells on the Madagascar tab of our subnational mortality rates for VAS spreadsheet.

• 214

  We use microdata on deaths from 18 Demographic and Health Surveys (DHS) between 2013 and 2018 in sub-Saharan Africa and South Asia, shared by Romero-Prieto based on Romero-Prieto et al. 2021. The raw data and additional notes on our analysis are compiled in this spreadsheet.

• 215

  See this row in our cost-effectiveness analysis.

• 216

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 217

  See this section of our separate report on insecticide-treated nets for further details.

• 218

  Our approach:

  • We used VAS coverage estimates from recent Demographic and Health Survey in each country compiled in GiveWell, Analysis of baseline VAS coverage.
  • We calculated a weighted average of 49% coverage across all the most recent surveys (weighted by how outdated they are), then regressed our estimates for each country towards that average (regressing more for countries with more outdated surveys).
  • For example, in Côte d’Ivoire (where there was a recent 2021 DHS survey), we put 74% weight on that survey and 26% weight on the cross-country average. In DRC (where the most recent DHS is from 2024), we put 24% weight on the DHS and 76% weight on the cross-country average. See here in our analysis of baseline VAS coverage for this comparison.

• 219

  See this section in our cost-effectiveness analysis. This calculation uses the following formula:

  • Mortality overall = (Mortality among those not receiving VAS x % not receiving VAS) + (Mortality among those receiving VAS x % receiving VAS). This is equal to:
  • (Mortality among those not receiving VAS x % not receiving VAS) + (Mortality among those not receiving VAS x relative risk of mortality from receiving VAS x % receiving VAS). This implies:
  • Mortality among those not receiving VAS = Mortality overall / (relative risk of mortality from receiving VAS x % receiving VAS + % not receiving VAS).

• 220

  See this row in our cost-effectiveness analysis. This range is based on the following locations: Burkina Faso, Cameroon, Chad, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 221

  This understanding is based on multiple conversations with IHME researchers. Detailed modeling assumptions for the GBD estimates are available in the GBD 2019 methods appendix.

• 222

  To understand how the variation across countries’ baseline child mortality rates affect cost-effectiveness, we put together a rough sensitivity analysis. We found that cost-effectiveness is sensitive to baseline child mortality rates:

  • All locations we analyzed exceeded our cost-effectiveness bar (10x as cost-effective as cash transfers as of the time of writing), when each location’s child mortality rate is set equal to the highest mortality rate across countries (1.76% in Niger). See this row in our sensitivity analysis.
  • All locations but one fell below our cost-effectiveness bar when each location’s child mortality rate was set equal to the lowest mortality rate across countries (0.35% in Kenya). See this row in our sensitivity analysis.

  As a point of comparison, at the time of writing we estimate that roughly half the locations in our main cost-effectiveness analysis are above our 10x cost-effectiveness bar. While we’d expect baseline child mortality to vary substantially across locations, we’re unsure if we’re overestimating the magnitude of this difference.

• 223

  See this row in our cost-effectiveness analysis. Note that this estimate doesn’t account for the benefits we incorporate as rough supplemental adjustments, and so can’t be interpreted as an estimate of the total benefits of the program.

• 224
  • Our estimate of the development effects of deworming programs is based on long-run follow-ups to the experiment described in Miguel and Kremer 2004. See our report on mass deworming programs for our full assessment of the study.
  • For malaria programs, we rely on the findings of two natural experiments: Bleakley 2010 and Cutler 2010. See this section of our report on mass distribution of insecticide-treated nets (ITNs) for a detailed discussion of these studies.

• 225
  • "This study uses the malaria-eradication campaigns in the United States (circa 1920), and in Brazil, Colombia and Mexico (circa 1955) to measure how much childhood exposure to malaria depresses labor productivity. The campaigns began because of advances in health technology, which mitigates concerns about reverse causality. Malarious areas saw large drops in the disease thereafter. Relative to non-malarious areas, cohorts born after eradication had higher income as adults than the preceding generation. These cross-cohort changes coincided with childhood exposure to the campaigns rather than to pre-existing trends. Estimates suggest a substantial, though not predominant, role for malaria in explaining cross-region differences in income." Bleakley 2010, abstract.
  • "We examine the effects of exposure to malaria in early childhood on educational attainment and economic status in adulthood by exploiting geographic variation in malaria prevalence in India prior to a nationwide eradication program in the 1950s. We find that the program led to modest increases in household per capita consumption for prime age men, and the effects for men are larger than those for women in most specifications. We find no evidence of increased educational attainment for men and mixed evidence for women." Cutler 2010, abstract.

• 226

  See this section of our report on insecticide-treated nets for more detail on these estimates.

• 227

  See this row in our cost-effectiveness analysis.

• 228

  See this row in our cost-effectiveness analysis.

• 229

  "These findings suggest that SMC in children in areas with heavy malaria pressure tends to increase weight gain, but they do not offer evidence that SMC increases height gain. Except for Fernando et al. 2006, these trials were all conducted in sub-Saharan Africa in young children, so they should be roughly comparable to current VAS beneficiary contexts." GiveWell, Development effects of vitamin A supplementation, 2019, p. 23. This conclusion is based on five RCTs.

• 230

  In our 2019 review, we focused on four studies from Ramakrishnan et al. 2009 that most closely resembled the locations where GiveWell funds VAS today. These studies suggested no impact of VAS on growth.
  "Since the effect of VAS could be attenuated in higher-income countries with lower prevalence of vitamin A deficiency and lower infectious disease pressure, I further examined the results of individual RCTs included in Ramakrishnan et al. 2009, focusing on those conducted in low-income countries in sub-Saharan Africa similar to our current VAS beneficiary populations. This left four trials, which report the following:

  • Kirkwood et al. 1996 was conducted in Ghana and reported no "consistent or substantial" effect of VAS on weight, height-for-weight, or mid upper-arm circumference in two related RCTs.
  • Fawzi et al. 1997 was conducted in North Sudan and reported no effect of VAS on height, weight, or the risk of stunting or wasting.
  • Donnen et al. 1998 was conducted in Zaire [current-day DRC] and reported no effect of VAS on weight, height, or mid-upper arm circumference in the groups as a whole over the entire follow-up period.
  • Villamor et al. 2002 was conducted in Tanzania and reported no effect of VAS on height (0.0 cm; 95% CI -0.6 to 0.6), weight (0.03 kg; 95% CI -0.16 to 0.23), or the risk of stunting or wasting in the groups as a whole.

  Since possible growth effects of VAS could be mediated via reduced health impacts of infectious diseases, Kirkwood et al. 1996 is particularly informative because it is the largest VAS trial to report that VAS significantly reduces all-cause mortality. It did not report growth effects in the groups as a whole. Collectively, these findings provide little indication of an effect of VAS on growth, even in settings where VAS substantially reduces mortality risk." GiveWell, Development effects of vitamin A supplementation, 2019, pp. 21-22.

• 231

  "Below, I pool the findings of the four trials that offered VAS postnatally, focusing on overall measures of cognitive ability. Since the trials measured general intelligence in different ways, I pooled using SMD to make them commensurable. The forest plot of the results is below:
  When normalized by calculating SMD, the findings of these four trials are fairly similar, with overlapping 95 percent confidence intervals. The results suggest that VAS in infancy or childhood in populations with varying prevalence of vitamin A deficiency does not have a statistically significant effect on general intelligence when measured four months to eight years later. The nonsignificant SMD of 0.05 is roughly equivalent to a very small increase in IQ of 0.75 points, and the confidence interval implies that the finding is compatible with changes in IQ of -0.75 to 2.25. This yields a best-guess IQ effect of 0.75 points and allows us to tentatively exclude an increase in IQ of greater than 2.25 points, although this estimate comes with uncertainty." GiveWell, Development effects of vitamin A supplementation, 2019, p. 10.

• 232

  "Overall, the RCT literature presents a mixed picture of the impact of SMC on cognitive ability. Among the four trials that reported a substantial effect of SMC on a measure of malaria burden, two reported an improvement in at least one measure of cognitive ability, while the other two did not. This suggests that SMC can probably increase some measures of cognitive ability in some contexts, but the effect is inconsistent." GiveWell, Development effects of vitamin A supplementation, 2019, p. 10.

• 233

  "Malaria infections can affect the brain, sometimes with severe consequences. The World Health Organization (WHO) defines "cerebral malaria" as a coma associated with malaria infection, with no other known cause of the coma. This is a strict definition that excludes many cases of brain involvement…
  Based on a quick literature search, I very roughly estimate that 1.2 percent of under-5 clinical malaria episodes in sub-Saharan Africa result in cerebral malaria, using the strict WHO definition of cerebral malaria. Using a less strict definition that encompasses any clear sign of brain involvement, one might expect approximately twice this rate. These figures are very uncertain but suggest that cerebral malaria only affects a small minority of children per clinical malaria episode."
  "My interpretation of the malaria evidence as a whole is that clinical malaria episodes in childhood probably impair mean cognitive ability somewhat, with the burden concentrated in a small minority of children, and preventing malaria with SMC may improve some measures of cognitive ability in some contexts. However, I am not very confident about the latter conclusion." GiveWell, Development effects of vitamin A supplementation, 2019, pp. 12, 19.

• 234

  "Vitamin A deficiency (VAD) impairs body functions and may cause death. Adverse health consequences may also include xerophthalmia (dry eyes), susceptibility to infection, stunting and anaemia (Sommer 1996; Rice 2004)." Imdad et al. 2010, p. 9.
  Xerophthalmia (dry eyes), is "the leading cause of preventable childhood blindness." WHO, Global prevalence of vitamin A deficiency in populations at risk 1995–2005, 2009, p. 1.

• 235

  This is because we think they would be challenging to model effectively or their likely effect is small enough that we do not expect explicit modeling to be worth the time investment required.

• 236

  See this section of our cost-effectiveness analysis.

• 237
  • Step one: Estimate the likely size of each effect (a rough subjective best guess).
  • Step two: Evaluate each effect using three criteria, each assigned a score of up to 3:
    • Can it be objectively justified (i.e., is there direct evidence for the effect)? (Where 0 is little evidence and 3 is strong direct evidence).
    • How easy would it be to model? (Where 0 is impossible and 3 is simple).
    • Consistency - is this effect included in our cost-effectiveness analysis for other programs? (Where 1 is no, 2 is partially and 3 is yes).
  • Step three: Convert the total three criteria score out of 9 into a percentage (e.g., a total score of 5 converts to a percentage of 50%).
  • Step four: Weight our estimate of each effect by this percentage to produce our overall adjustment (e.g., an effect size guess of 15% multiplied by a three criteria score of 60% = an adjustment of 9% (15% x 60% = 9%)).

  We use the method described for all supplemental intervention-level adjustments in our cost-effectiveness analysis for VAS, except our estimate of costs saved from averted illness. We account for these savings using an explicit model, discussed in more detail in GiveWell, Cost of Illness Averted Adjustment Write-up.

• 238

  See this section of the report for a detailed breakdown by disease.

• 239

  For more details on our reasoning, see this section of the report. See this row in our cost-effectiveness analysis for this adjustment.

• 240

  "The search yielded 23 eligible studies, 21 clinical trials and 2 cohort studies, with children, teenagers, pregnant or lactating women. The meta-analysis of the clinical trials showed that VAS reduces the risk of anemia by 26% and raises hemoglobin levels, compared to non-treated group, independent of the life stage. VAS did not alter the prevalence of iron deficiency among the clinical trials conducted with children and teenagers (RR 0.82, 95% CI 0.60 to 1.12, p = 0.204), whereas a significant increase in serum ferritin levels was observed in trials conducted with pregnant and lactating women (WMD 6.61 μg/L; 95% CI 6.00 to 7.21 μg/L; p < 0.001). Therefore, vitamin A supplementation alone may reduce the risk of anemia, by improving hemoglobin and ferritin levels in individuals with low serum retinol levels." de Sá Barreto da Cunha et al. 2019, abstract.

• 241

  We have two specific concerns:

  • It includes nonrandomized trials, which have lower internal validity. However, nonrandomized trials only represent 3.3 percent of the pooled findings and they report findings that are close to the pooled mean, so they probably had little impact.
  • Some of the trials did not isolate the impact of VAS and the meta-analysis may therefore overestimate the impact of VAS. We have spot-checked the three RCTs that received the largest weights, and two isolated VAS, while the third administered a deworming drug along with VAS.
    • "All eligible anaemic pre-school children were randomly divided into three groups: group 1 received no intervention, which served as the control group, group 2 received 400 mg single-dose albendazole administration and group 3 received a 60000 μg vitamin A capsule combined with 400 mg single-dose albendazole at the beginning of the study." Chen et al. 2016, abstract. We have also read the abstract of Suharno et al. 1993 and Ahmed et al. 2001.

  See de Sá Barreto da Cunha et al. 2019, figure 3.

• 242

  See this row in our cost-effectiveness analysis.

• 243

  See this discussion of anemia in our development effects model here.

• 244

  We include an additional adjustment to account for these benefits because we would expect invested income to earn a return, meaning increased income leads to further consumption increases in the future. For example, we estimate that income invested as a result of GiveDirectly’s direct cash transfer program will earn a return of 10% (see here in our cost-effectiveness analysis). We have not investigated whether returns will be similar for income accrued through VAS development effects. See this row in our cost-effectiveness analysis.

• 245

  We currently use the following values:

  • Insecticide-treated nets: 2.5%
  • Seasonal malaria chemoprevention: 5%
  • Conditional cash transfers for vaccination (New Incentives): 5%

  See our cost-effectiveness analysis for each program for our reasoning in each case. We have not updated these values recently, and they may be out of date.

• 246

  Xerophthalmia (dry eyes), is "the leading cause of preventable childhood blindness." WHO, Global prevalence of vitamin A deficiency in populations at risk 1995–2005, 2009, p. 1.

• 247

  Imdad et al. 2017 found that VAS:

  • Reduces the incidence of Bitot’s spots (a potential precursor to blindness) by 58% (RR 0.42, 95% CI 0.33 to 0.53).
  • Reduces the incidence of night blindness by 68% (RR 0.32, 95% CI 0.21 to 0.50).

  Imdad et al. 2017 assesses both estimates as based on "moderate" quality evidence.
  See Imdad et al. 2017, Summary of Findings, p. 5.

• 248

  See GiveWell, VAS vision benefits write-up for our rough analysis.

• 249

  We use GiveDirectly's unconditional cash transfer program as a benchmark for comparing the cost-effectiveness of different programs, which we describe in multiples of "cash." Thus, if we estimate that a program is "10x cash," this means we estimate it to be ten times as cost-effective as unconditional cash transfers.

• 250

  See this section of our report on Helen Keller International, the organization GiveWell has supported most for previous VAS campaigns, for further details.

• 251

  See Helen Keller International, Distribution methods for VAS mass distribution campaigns, 2022 for details on which other interventions were co-delivered with VAS in Helen Keller-supported campaigns between 2018 and 2021.

• 252

  "Since vaccination against a specific disease commonly enhances immunity to other diseases, we believe it is plausible that some vaccines have beneficial nonspecific effects. Animal research suggests that VAS can increase the immune response to certain vaccines, although the results of human trials are inconsistent. Together, we believe this suggests that it is fairly plausible that VAS can enhance the beneficial nonspecific effects of vaccines in charity-relevant settings, but we are fairly uncertain about this conclusion. Due to a weaker evidence base, we believe the plausibility of harmful nonspecific effects of vaccines is lower but not negligible." GiveWell, The biological plausibility of interactions between vitamin A supplementation and vaccine effectiveness, 2018, p. 1.

• 253

  See this row in our cost-effectiveness analysis.

• 254

  As of February 2024, these are AMF, Malaria Consortium, Helen Keller, and New Incentives. See our "Top charities" page here.

• 255

  Our model estimates a 14% upwards adjustment for insecticide-treated nets (here), a 20% upwards adjustment for seasonal malaria chemoprevention (here) and a 21% upwards adjustment for vitamin A supplementation (here).

• 256

  "We reanalyzed the data to explore the hypothesis that VAS reduces mortality in children who had bacille Calmette-Guerin or measles vaccine as their most recent vaccine but increased mortality when diphtheria-tetanus-pertussis vaccine (DTP) was the most recent vaccine. On the basis of previous studies, we expected the effects to be strongest in girls." Benn et al. 2009, p. 629.

• 257

  "As hypothesized, the reanalysis suggests important interactions between VAS, sex, and vaccines. VAS was associated with a strong beneficial effect in children with no record of vaccination, whereas there was almost no effect for those who had been vaccinated. This differential effect was due to a difference in girls, in whom VAS was associated with a decrease in mortality in the unvaccinated but in whom VAS was associated with a nonsignificant increase in mortality in the vaccinated (Table 2). This was due to a differential effect of VAS according to vaccination type. Among girls who had already received MV at enrollment, VAS was associated with significantly higher mortality. This was only seen in girls who were missing doses of DTP at enrollment and were therefore likely to receive them during follow-up (Table 5)." Benn et al. 2009, p. 635.

• 258

  "We have hypothesized that the effect of VAS is modified by vaccines, VAS amplifying the non-specific immune-modulating effects of vaccines, thus being beneficial when provided with live vaccines but potentially harmful with inactivated vaccines." Fisker et al. 2014, p. e740.
  "As prespecified, all analyses considered interaction between VAS and gender and, in addition, previous VAS and season." Fisker et al. 2014, p. e741.

• 259

  "Between August 2007 and November 2010, 7587 children were enrolled. Within 6 months of follow-up 80 non-accident deaths occurred (VAS: 38; placebo: 42). The mortality rate ratio (MRR)comparing VAS versus placebo recipients was 0.91 (95% confidence interval 0.59–1.41) and differed significantly between boys (MRR1.92 [0.98–3.75]) and girls (MRR 0.45 [0.24–0.87]) (P= .003 for interaction between VAS and gender). At enrollment, 42% (3161/7587) received live measles vaccine, 29% (2154/7587) received inactivated diphtheria-tetanus-pertussis–containing vaccines, and 21% (1610/7587)received both live and inactivated vaccines. The effect of VAS did not differ by vaccine group." Fisker et al. 2014, p. e739.

• 260

  "Since vaccination against a specific disease commonly enhances immunity to other diseases, we believe it is plausible that some vaccines have beneficial nonspecific effects. Animal research suggests that VAS can increase the immune response to certain vaccines, although the results of human trials are inconsistent. Together, we believe this suggests that it is fairly plausible that VAS can enhance the beneficial nonspecific effects of vaccines in charity-relevant settings, but we are fairly uncertain about this conclusion. Due to a weaker evidence base, we believe the plausibility of harmful nonspecific effects of vaccines is lower but not negligible." GiveWell, The biological plausibility of interactions between vitamin A supplementation and vaccine effectiveness, 2018, p. 1.

• 261

  "In most children 6–59 months of age, a dose of 100 000–200 000 IU of vitamin A is well tolerated, although side-effects such as headache, nausea or vomiting, and diarrhoea have been reported in 3–7% of these children." WHO, Guideline: vitamin A supplementation in infants and children 6-59 months of age, 2011, p. 3.

• 262

  "In most children 6–59 months of age, a dose of 100 000–200 000 IU of vitamin A is well tolerated, although side-effects such as headache, nausea or vomiting, and diarrhoea have been reported in 3–7% of these children." WHO, Guideline: vitamin A supplementation in infants and children 6-59 months of age, 2011, p. 3.

• 263

  WHO, Adverse events following administration of vitamin A supplements:

  • "The administration of excessive amounts of vitamin A can lead to toxicity, known as hypervitaminosis A. The amount required to cause toxicity will vary among individuals." p. 1.
  • "Worldwide, the incidence of hypervitaminosis A is a very minor problem compared with the incidence and effects of vitamin A deficiency. An estimated 200 cases of hypervitaminosis A occurs annually…" p. 1.
  • "Hypervitaminosis does not result from public health intervention programs. Rather toxicity has been associated with the abuse of vitamin A supplements and with diets extremely high in preformed vitamin A (i.e., foods of animal origin). Toxic reactions provoked by large doses of vitamin A are well-known to occur following either intake of liver rich in vitamin A (e.g., polar bear, halibut or whale) or by excessive administration of vitamin A preparations (Miller & Hayes, 1982)." p. 2.
  • "Acute vitamin A toxicity (single ingestion of 25,000 IU per kg or more): Signs and symptoms may be delayed for 8 to 24 hours and include manifestations such as nausea, vomiting, diarrhea, changes in humour (irritability, drowsiness, dizziness, lethargy), increased intracranial pressure (headache, bulging of fontanelle, diplopia, papilloedema), skin changes (erythema, pruritus, desquamation). Peeling of skin around mouth may be observed from 1 to several days after ingestion and may spread to the rest of the body (Miller & Hayes, 1982; Bendich & Langseth, 1989; Hathcock et al., 1990; CPS, 1999; Parfit, 1999)." p. 2.

  Note: the WHO guidance uses the terms "toxicity" and "hypervitaminosis A" interchangeably. In 2023, we received feedback from an expert, Dr. Sherry Tanumihardjo, that the two terms should refer to different thresholds. We haven’t investigated the differences between the two in depth.
  "In the BOND report (Tanumihardjo et al., 2016), we tried to distinguish between hypervitaminosis A and vitamin A toxicity. Hypervitaminosis A occurs when there is too much vitamin A, but no clinical signs are present. Toxicity is when retinyl esters begin to circulate and the liver is compromised." Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023 (unpublished).

• 264
  • "Hypervitaminosis does not result from public health intervention programs. Rather toxicity has been associated with the abuse of vitamin A supplements and with diets extremely high in preformed vitamin A (i.e., foods of animal origin). Toxic reactions provoked by large doses of vitamin A are well-known to occur following either intake of liver rich in vitamin A (e.g., polar bear, halibut, or whale) or by excessive administration of vitamin A preparations (Miller & Hayes, 1982)." WHO, Adverse events following administration of vitamin A supplements, p. 2.
  • Helen Keller told us that receiving two doses of vitamin A supplements within a short time period would not meet toxicity thresholds:
    • "[GiveWell:] In countries where six-month contact points have been initiated, is there a risk of a child receiving a 'double dose' of VAS in a short time period (one from a facility visit when the infant is six months old, and another at the next biannual Child Health Day or door-to-door campaign)? Would receiving a double-dose potentially be dangerous? (Even if they aren't dangerous, we're also concerned about double-doses because they wouldn't be an effective use of resources.)
    • "[Helen Keller:] This is a legitimate question and one we have had to think about carefully as we started to promote and support the 6 month contact point (6MCP). First, receiving two doses in a short time frame poses some, but minimal, risks for children as the toxicity thresholds go far beyond receiving two doses (see attached document on Adverse events following administration of VAS)."

  Helen Keller responses to GiveWell's questions May 2017, unpublished.

• 265

  See this section of our discussion of the cost per supplement analysis for further details on this estimate.

• 266

  For a summary of fertility rates by country, see Our World in Data, "Fertility rate: children per woman, 2021".

• 267

  "The impact of life-saving interventions on fertility and population growth varies by context, and is rarely greater than 1:1. In places where lifetime births/woman has been converging to 2 or lower, saving one child’s life should lead parents to avert a birth they would otherwise have. The impact of mortality drops on fertility will be nearly 1:1, so
  population growth will hardly change. In the increasingly exceptional locales where couples appear not to limit fertility much, such as Niger and Mali, the impact of saving a life on total births will be smaller, and may come about mainly through the biological channel of lactational amenorrhea. Here, mortality-drop fertility-drop ratios of 1:0.5 and 1:0.33 appear more plausible." Roodman, The Impact of Life-Saving Interventions on Fertility, 2014, summary.

• 268

  We received this feedback as an entry to GiveWell’s 2022 Change our Mind contest. In 2024, the author shared an updated version with some minor edits (the version published here). Mariet Benade, Drug degradation as a potential reason for differences in the effect of Vitamin A Supplementation on all-cause mortality among children, 2024.

• 269

  "The 50 000-IU capsules that were tested were also fully potent up to and through February 2003. However, between February and July 2003, the vitamin A content of the 50 000-IU capsules declined markedly, so that the 33 capsules that were analyzed between August 2003 and January 2004 contained only 1800–23 993 IU (mean: 15 861 IU), or 32% of the expected amount." Idindili et al. 2007, p. 1315.

• 270

  "These analyses confirmed that the 200 000- and 100 000-IU capsules contained a satisfactory amount of retinyl palmitate throughout the trial period. The 25 000-IU capsules also had ≥80% potency up to July 2003, when the last dose of this type of capsule had already been given…However, between February and July 2003, the vitamin A content of the 50 000-IU capsules declined markedly." Idindili et al. 2007, p. 1314.

• 271

  If making an adjustment, we would primarily ask whether capsule degradation in contemporary VAS programs was better or worse than the RCTs in the Cochrane meta-analysis we rely on, not whether capsules degraded in contemporary VAS programs at all.
  We do not think the Tanzania RCT (Idindili et al. 2007), by itself, is very informative about degradation in contemporary programs, because:

  • It is a single, 20 year old study.
  • Three of the four types of capsule retained good vitamin A levels during the study, and only one type (50,000 IU, ½ to ¼ the dose used in Helen Keller and Nutrition International’s program) did not.
  • The sample used to assess Vitamin A degradation in the Tanzania study contained only 33 capsules. While the authors do not report confidence intervals, our best guess is that the confidence intervals for this estimate would be wide.

  However, we haven’t checked to see if more comprehensive data is available on this question.
  Our initial light-touch review of the five most highly weighted studies in the Cochrane meta-analysis (Imdad et al. 2017) found that three studies reported capsule degradation data. In all three, the capsules retained 80+% their intended levels of vitamin A when tested.

  • "Supplements were stored in sealed bottles at room temperature. Random samples of the vitamin A bottles and of the capsules were returned for testing of retinol content by Hoffman-La-Roche's laboratories in Basel; there was less than 20% loss of potency, even in vitamin A stored for up to 2 years." Ross et al. 1993, p. 8.
  • "Every 4 months two capsules of each code, sampled from the study area, were analysed for vitamin A potency by a local laboratory unaware of their code. The mean retinol contents were 53 300 (SD 3513) and 246 (27) retinol equivalents, which is equivalent to 89% and 82% potency for the vitamin A (n = 19) and control (n = 20) capsules, respectively." West et al. 1991, p. 68.
  • "After 22 months of storage, a sample of capsules of both colours was sent to the manufacturer for analysis; 15% of the vitamin A activity had been lost." Herrera et al. 1992, p. 267.

• 272

  The examples we have seen have varied by country and over time. For example:

  • DRC: Since 2021, Helen Keller has used GiveWell funding in DRC to fund campaigns at the province level, starting with three provinces in 2021 and expanding to seven provinces in 2022. We therefore interpret the impact of Helen Keller’s funding in DRC as allowing more provinces to be covered. See Helen Keller International, Room for More Funding Report, 2021, pp. 18-19.
  • Cote d’Ivoire: In 2021, Helen Keller reported that it expected VAS distributions to be canceled in some districts in the absence of additional funding. Our best guess is therefore that Helen Keller’s funding in Cote d’Ivoire as allowing more provinces to be covered. See Helen Keller International, Room for More Funding Report, 2021, p. 17.
  • Guinea: In 2016 and 2017, two planned campaign rounds in Guinea were skipped altogether because of lack of funding (see GiveWell, Helen Keller country case studies, 2017). Since 2018, campaigns have taken place every round in Guinea (Helen Keller International, Room for More Funding Report, 2021, p. 19, and Helen Keller International, Room for More Funding Report, 2022, p. 13). We therefore interpret the most likely impact of Helen Keller’s funding in Guinea as allowing campaigns to take place.

• 273

  Differences:

  • Within-organization fungibility: We used a -2% adjustment for within-organization fungibility. This is a smaller adjustment than for Helen Keller, because Chad was not a "core" location for Nutrition International and they had a relatively limited track record there. It seemed very unlikely to us that Nutrition International would allocate its limited amount of flexible funding to these funding gaps in the absence of GiveWell funding.
  • Change of priorities: We use a higher adjustment (-5% compared to 0% for Helen Keller) for changes in Nutrition International’s priorities (i.e., Nutrition International might use some GiveWell funding for things we do not believe to be useful, and that we did not envisage). This adjustment is higher because we do not have as much experience funding Nutrition International as Helen Keller.

  See this section of our cost-effectiveness analysis for further details on our reasoning.

• 274

  As of February 2024, we apply the following adjustments to our other top charities:

  • AMF (insecticide-treated nets): -4%
  • Malaria Consortium (seasonal malaria chemoprevention): -8%
  • New Incentives (conditional cash transfers for immunization): -7%

• 275

  See these rows in our cost-effectiveness analysis.

• 276

  This is driven by two factors:

  • In some of Helen Keller’s campaigns, only certain regions are surveyed. We think the regions selected for campaigns may not be selected randomly, meaning that the data may not be representative of Helen Keller’s program as a whole. See this section of our review of Helen Keller for more information.
  • In most countries Helen Keller only conducts a survey after one of two campaign rounds, and our understanding is that VAS implementers may know in advance which campaign will be surveyed. If that is the case, we would guess that workers would be more incentivized to ensure high coverage in surveyed campaigns than non-surveyed campaigns (which have less oversight).

• 277

  See this row in our cost-effectiveness analysis.

• 278

  This understanding is based on many conversations with Helen Keller over time.

• 279

  See this row in our cost-effectiveness analysis. Our leverage adjustment ranges from -1% to -5%, varying by location. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 280

  See the section above for more details on how we reach these estimates. Note that these figures don’t correspond with the proportion of costs paid by different actors in this section of our cost-effectiveness analysis. We estimate that for each $1m spent by Helen Keller, Nutrition International incurs ~$88,000 in costs and the Guinea government incurs ~$543,000.
  The reason the figures above are smaller than this is that some program costs are also borne by other philanthropic actors (in Guinea, WHO). Our best guess is that if Helen Keller didn’t fund the program, these costs would be unaffected and the Guinea government / Nutrition International costs would shrink proportionally to WHO’s costs. These are around 15% as large as Helen Keller and WHO’s costs combined. ~543,000 x 15% = ~$460,000 and ~$88,000 x 15% = ~$75,000. See this cell and this cell of our cost-effectiveness analysis for our calculations.

• 281

  We estimate that each dollar spent by a domestic government in a country receiving VAS generates 0.005 units of value, compared to 0.060 for Helen Keller’s spending on VAS in Guinea. 0.005 / 0.060 = 8.3%. For Nutrition International’s VAS capsule donations, we estimate 0.006 units of value. 0.006 / 0.060 = 10%.
  For details of how we estimate these values, see this section of the report.

• 282
  • We estimate that each dollar spent by the Guinea government generates 0.005 units of value if used for other activities and each dollar spent by Nutrition International on VAS capsule donations generates 0.006 units of value.
  • In total, we think that each $1m spent by Helen Keller causes the Guinea government to incur ~$460,000 in in-kind costs and Nutrition International to incur ~$75,000 in costs for donated capsules.
  • This implies that shifting these resources away from other activities results in 2,751 units of value being lost. (~460,000 x 0.005) + (~75,000 x 0.006) = 2,751.

  See this row in our cost-effectiveness analysis.

• 283

  See this row in our cost-effectiveness analysis.

• 284

  2,751 x 55% = 1,513. 1,513 / 60,187 = ~3%. See this row in our cost-effectiveness analysis.

• 285

  See this row in our cost-effectiveness analysis. Our funging adjustment ranges from -18% to -46% depending on the location. This range is based on the following locations: Burkina Faso, Cameroon, Côte d'Ivoire, DRC, Guinea, Mali, Madagascar, Niger, and five states in Nigeria: Adamawa, Benue, Ebonyi, Nasarawa, and Taraba (for Helen Keller) and Chad (for Nutrition International). We focus on these locations for the reasons discussed above.

• 286

  Our understanding is that UNICEF and Nutrition International are the largest philanthropic organizations supporting VAS in areas where Helen Keller works and therefore the most likely to replace its funding. See this section of our review of Helen Keller for further information.
  We revisit these probabilities for each grant we make. For Guinea, we most recently estimated these probabilities as part of a 2023 grant. See our grant page here.

• 287

  In this scenario, we do not think that Helen Keller’s funding is causing more VAS to be delivered, because other funders would have replaced its spending in any case.

• 288

  We roughly estimate that the Guinea government’s spending on other activities generates 0.005 units of value per dollar, and UNICEF / Nutrition International’s spending generates 0.012 units of value per dollar, compared to 0.060 for Helen Keller’s spending in Guinea. 0.005 / 0.060 = ~8% and 0.012 / 0.060 = 20%. See this section of the report for how we generate these estimates.

• 289
  • We estimate that each dollar spent by the Guinea government generates 0.005 units of value if used for other activities and Helen Keller generates 0.060 units of value per dollar spent on VAS in Guinea. Per $1m, the Guinea government replacing Helen Keller’s spending in Guinea would generate 60,187 units of value on VAS ($1m x 0.060) and lose 5,057 units of value ($1m x 0.005) that would have been spent on other programs. 60,187 - 5,057 = 55,130. See these cells in our cost-effectiveness analysis.
  • We estimate that each dollar spent by other philanthropic actors (UNICEF and Nutrition International) generates 0.012 units of value if used for other activities. Per $1m, replacing Helen Keller’s spending in Guinea would generate 60,187 units of value on VAS ($1m x 0.060) and lose 12,310 units of value ($1m x 0.012) that would have been spent on other programs. 60,187 - 12,310 = 47,877. See these cells in our cost-effectiveness analysis.

• 290

  See this cell in our cost-effectiveness analysis for our estimated probability of the Guinea government replacing Helen Keller's spending, and this cell for the probability for other philanthropic actors.

• 291

  Note that this breaks down into:

  • Guinea government: 55,130 x 10% = 5,513
  • Other philanthropic actors: 47,877 x 35% = 16,757

  5,513 + 16,757 = 22,270. See this cell in our cost-effectiveness analysis.

• 292

  22,270 / 60,187 = ~37%. See this cell in our cost-effectiveness analysis.

• 293

  We estimate the probability that other funders would replace GiveWell's funding in our absence by adding the probability that domestic governments would replace the funding (see this row in our cost-effectiveness analysis) with the probability that other philanthropic actors would replace the funding (see this row in our cost-effectiveness analysis). This estimate (varying by location) ranges from 20% to 50% in our analysis.

• 294

  Note that these figures don’t correspond with the proportion of costs paid by different actors in this section of our cost-effectiveness analysis. We estimate that for each $1m spent by Helen Keller, Nutrition International incurs ~$88,000 in costs and the Guinea government incurs ~$543,000.
  The reason the figures above are smaller than this is that some program costs are also borne by other philanthropic actors (in Guinea, WHO). Our best guess is that if Helen Keller didn’t fund the program, these costs would be unaffected and the Guinea government / Nutrition International costs would shrink proportionally to WHO’s costs. These are around 15% as large as Helen Keller and WHO’s costs combined. ~543,000 x 15% = ~$460,000 and ~$88,000 x 15% = ~$75,000. See this cell and this cell in our cost-effectiveness analysis for our calculations.

• 295

  Our understanding is that WHO provided costs for polio campaigns in Guinea in 2018 and 2019 and VAS was co-delivered alongside these campaigns (more discussion here).
  We exclude these costs from our leverage and funging adjustments, because we think that these polio campaigns would have gone ahead regardless of Helen Keller’s funding, and so Helen Keller does not lead to WHO contributing more or less to the campaign.

• 296

  See this section of our cost-effectiveness analysis, and these cells for the Guinea estimates.

• 297

  By way of comparison, we think that the VAS funding landscape is less crowded than the funding landscape for the main malaria interventions GiveWell funds, seasonal malaria chemoprevention and insecticide-treated nets. There are two major funders of malaria control (the Global Fund to fight AIDS, Tuberculosis and Malaria, and the US government's President's Malaria Initiative) that to a significant extent fund the most high priority needs for malaria control in low-income countries. In contrast, our understanding is that GiveWell is the largest funder of VAS campaigns in low-income countries in Africa, despite the fact that GiveWell's total funding for VAS is considerably lower than its total funding for malaria control.

• 298

  This understanding is based on many discussions over time with Helen Keller and other VAS stakeholders.

• 299

  "With GiveWell support, Helen Keller Intl covered VAS funding gaps in Guinea since 2018. Helen Keller and UNICEF are the two main actors providing VAS technical and financial support to the Ministry of Health." Helen Keller International, Room for More Funding Report, 2021, p. 19.

• 300

  "Following discussions with the Ministry of Health and partners, it appears that UNICEF’s funding will be inadequate to support VAS in more regions, although it intends to continue to support at least 3 out of the 8 regions in the country. Nutrition International’s current funding will end in 2022. Thus, it is expected that between 2023 and 2025, Helen Keller will be requested to support the remaining 5 regions for the implementation of campaigns or weeks of intensification." Helen Keller International, Room for More Funding Report, 2022, p. 13.

• 301

  See this row in our cost-effectiveness analysis.

• 302

  See this row in our cost-effectiveness analysis.

• 303

  0.005 / 0.060 = 8.3%, or 1/12.

• 304

  See this row in our cost-effectiveness analysis.

• 305

  Health

  • We use data from the Uganda National Health Expenditure Accounts in 2013-14 (available here) as a proxy for how domestic governments in low-income countries allocate their health spending. In that year, approximately 30% of the health spending was allocated to HIV/AIDS, 20% to malaria, and the rest to other programs (see this column in our counterfactual value of government funds spreadsheet).
  • We roughly guess the dollar value required to save a life for eight of the largest categories of spending (e.g., we guess $30,000 of spending on HIV/AIDS programs is required to save one life (here), compared to $10,000 for malaria, here). As a simplifying assumption, we divide each category of spending into spending intended to save adult lives (e.g., HIV/AIDS), or spending intended to save under-five lives (e.g., malaria) (see this column in our counterfactual value of government funds spreadsheet). This allows us to calculate the weighted average cost required to save one child’s life ($12,254) and one adult life ($30,000).
  • Finally, we translate these figures into standardized units of value using GiveWell’s moral weights for the value of averting a child death (116 units) and adult death (73 units). We also use our estimates of the proportion of spending allocated to programs intended to save adult lives (55%) and under-five lives (45%). This results in an overall weighted average of 0.0056 units of value per $ spent on health programs by domestic governments. See this section of our analysis of the counterfactual value of other actors' spending for our calculations.

  Education

  • As with health, we use data from Uganda as a proxy for how domestic governments in low-income countries allocate their education spending. Specifically, we rely on World Bank 2014 data that 7% of Uganda’s GDP in 2014 went towards primary education, 3% went towards secondary education and 2% went towards tertiary education (here). We then rescale these figures, resulting in estimates that 59% of education spending in Uganda goes towards primary education, 25% towards secondary education and 16% towards tertiary education (here).
  • As a simplifying assumption, we assume that the primary benefit of education spending is that it increases long-run income and consumption. We benchmark our estimates to our estimates of the impact of GiveDirectly’s unconditional cash transfer program (which we think also increases income and consumption). Specifically, we roughly guess that primary education spending is 100% as valuable as GiveDirectly’s program, secondary education is 70% as valuable and tertiary education is 50% as valuable (here).
  • We take a weighted average of these values, using the % of government spending on each category of education as weights. This results in an overall estimate of 0.0028 units of value per $ spent on education programs by domestic governments. See this section of our analysis for our calculations.

  Social security
  We use a similar approach to estimating the value of social security spending as education spending. Specifically:

  • We use data from Uganda as a proxy for how domestic governments in low-income countries allocate their social security spending. We use World Bank 2014 estimates that social security spending accounted for 0.62% of Uganda’s GDP (here). Within this, 0.29% of GDP is in-kind spending, 0.1% is social pension spending, 0.1% is conditional cash transfers, and 0.05% is other cash transfers (here).
  • We roughly guess the value generated by each of these four largest categories of social security spending, excluding the smaller categories. As with education spending, we use the simplifying assumption that the primary benefit of social security spending is that it increases income and consumption. Benchmarking to GiveDirectly’s cash transfer program, we guess that cash transfers generate 100% of the value of GiveDirectly’s program and in-kind and social pension spending each generate 70% of the value (here).
  • We take a weighted average of these values, using the % of government spending on each category of social security spending as the weights (re-scaled to ignore smaller categories). This results in an overall estimate of 0.0026 units of value per $ spent on social security programs by domestic governments. See this section of our analysis for our calculations.

• 306

  See this row in our analysis of the counterfactual value of other actors' spending.

• 307

  See this row in our cost-effectiveness analysis and this row in our analysis of the counterfactual value of other actors' spending.

• 308

  See this row in our cost-effectiveness analysis.

• 309

  See this row in our analysis of the counterfactual value of other actors' spending.

• 310

  Burkina Faso, Cameroon, Côte d'Ivoire, and DRC. See this row in our cost-effectiveness analysis.

• 311

  "UNICEF received a USD $25 million grant from Global Affairs Canada to support VAS in 15 countries (Angola, Benin, Burkina Faso, Cameroon, CAR, Chad, Cote d‘Ivoire, DRC, Guinea, Madagascar, Malawi, Mozambique, Sierra Leone, Sudan, Togo) from mid-2023 to end-2025 (5 Semesters). However, considering that some funds will be used for global studies and not all funds can be directly allocated to VAS support in the field, the actual direct VAS delivery support per semester and per country is less than USD $200,000." Helen Keller International, Room for More Funding Report, 2023, p. 2.

• 312

  Mali, Niger, Nigeria, Kenya, and Uganda. In these locations, we expect much of the funding comes from UNICEF’s uncommitted funding. We would expect the counterfactual value of these funds to be lower. See this row in our cost-effectiveness analysis.

• 313

  "The above-mentioned VAS campaign studies are not the only studies that have indicated that high-dose VAS in certain situations can be associated with harm. Our group has pointed out that a number of studies providing VAS at the time window of diphtheria-tetanus-pertussis (DTP)-containing vaccine have tended to find negative effects, particularly in females (reviewed in [4], examples in[14-17]). This observation goes hand in hand with the controversial but consistent observation that DTP vaccine is associated with negative non-specific effects and increased female mortality[18-20]. Likewise, there are indications that VAS can have
  negative effects in boys, who receive a combination of live and non-live vaccines at the same time (reviewed in [4], examples in [7, 21]), also in this case, vaccine studies have indicated that this combination of live and non-live vaccines can have negative non-specific effects in males. Hence, VAS seems to amplify the negative effect of vaccine(s) that have been shown to have negative effects on overall survival." Christine Benn, Feedback on GiveWell vitamin A supplementation (VAS) research, December 2023 (unpublished).

• 314

  "The relationship of health with vitamin A status is a U shape. Too little and there are bad effects leading to death. Too much, and there are effects on bone."
  "In the BOND report (Tanumihardjo et al., 2016), we tried to distinguish between hypervitaminosis A and vitamin A toxicity. Hypervitaminosis A occurs when there is too much vitamin A, but no clinical signs are present. Toxicity is when retinyl esters begin to circulate and the liver is compromised." Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023 (unpublished).

• 315

  "In some cases, VAS might interrupt growth if children are hypervitaminotic. In this case, we think it is because of acute toxicity causing the child to feel nausea and then not eating for several days.
  We have evidence of this in South African (van Stuijvenberg et al. 2019; Sheftel et al., 2022) and Malawian children (Williams et al., 2021). Below are two graphs that we published in South African children who regularly consume liver (attached). On the background of good dietary intake of vitamin A, the supplements are mostly stored and do accumulate causing the children to have hypervitaminosis A…
  In the Malawian school-age children we saw elevated retinyl ester concentrations. Serum retinyl esters will only be elevated if there are very high stores of vitamin A to the point of toxicity." Sherry Tanumihardjo, Review of GiveWell VAS report, December 2023 (unpublished).

• 316

  "A final issue not fully addressed in the GiveWell analysis is the potential for excessive VA status, at least in sub-sets of the population, in locations where VAS is super-imposed on VA fortification programs and other platforms that deliver additional VA (e.g, micronutrient powders and small quantity lipid-based nutrient supplements). Although I doubt that this is an important issue for public health in most of sub- Saharan Africa, it is something that needs to be monitored going forward." Kenneth Brown, Comments on GiveWell Vitamin A Supplementation cost-effectiveness analyses, November 2023 (unpublished).

• 317

  "To the contrary, some showed a statistically significant increase in respiratory infection incidence in the vitamin A compared with the control group. One of these, conducted in Indonesia, included 1407 preschool children. It was a randomised, placebo-controlled, double masked trial, published in 1996. The authors concluded: ‘High dose vitamin A supplements increased the incidence of acute respiratory illnesses (ARI) by 8%, and acute lower respiratory illnesses (ALRI) by 39%’. They also concluded: ‘These ‘detrimental effects on acute lower respiratory illnesses were most marked in children with adequate nutritional status’." Latham 2010, p. 24.

• 318

  "There was no significant effect for VAS on mortality due to measles, respiratory disease, and meningitis. VAS reduced incidence of diarrhoea (RR 0.85, 95% CI 0.82 to 0.87; 15 studies; 77,946 participants; low-quality evidence) and measles (RR 0.50, 95% CI 0.37 to 0.67; 6 studies; 19,566 participants; moderate-quality evidence). However, there was no significant effect on incidence of respiratory disease or hospitalisations due to diarrhoea or pneumonia." Imdad et al. 2017, abstract.

• 319

  Note that we have received feedback from a VAS Expert, Dr. Kenneth Brown, that we should be cautious about conclusions that VAS might increase respiratory disease risk, due to uncertainties about the methods used in field studies to detect respiratory disease. We have not deeply investigated or tried to corroborate this.
  "With regard to the statements on possible adverse effects of VAS on the incidence of respiratory infections, I believe it is important to recognize that it is notoriously difficult in community-based studies to assess the presence of lower respiratory tract infections (ALRI -- bronchitis and pneumonia), which are much more important causes of mortality than upper respiratory tract infections. In most community-based studies, the diagnosis of ALRI depends on the (reported or observed) presence of cough and age-specific elevated respiratory rate. If VAS results in a more vigorous (protective) cough response, ALRI may be diagnosed more frequently, while this (i.e., the presence of cough) actually might be a beneficial effect of VAS. Thus, caution is needed in the interpretation of these findings, and any studies that included radiographic confirmation of pneumonia should be considered more reliable." Kenneth Brown, email to GiveWell, February 8th 2024 (unpublished).

• 320

  "In settings where vitamin A deficiency is a public health problem, vitamin A supplementation is recommended in infants and children 6–59 months of age as a public health intervention to reduce child morbidity and mortality (strong recommendation)." WHO, Guideline: vitamin A supplementation in infants and children 6-59 months of age, 2011, p. 1.

• 321

  "Authors' conclusions
  Vitamin A supplementation is associated with a clinically meaningful reduction in morbidity and mortality in children. Therefore, we suggest maintaining the policy of universal supplementation for children under five years of age in populations at risk of VAD. Further placebo-controlled trials of VAS in children between six months and five years of age would not change the conclusions of this review, although studies that compare different doses and delivery mechanisms are needed. In populations with documented vitamin A deficiency, it would be unethical to conduct placebo-controlled trials." Imdad et al. 2017, abstract.

• 322

  We have seen the following commentators call for an end to mass VAS distribution campaigns:

  • Latham 2010
  • Mason et al. 2018

  Latham 2010: "Of course the administration of medicinal doses of capsules is effective in cases of clinically evident xerophthalmia, which remains a public health problem and even emergency in some locations in some lower-income countries. What is mistaken, and reprehensible, are the claims made for vitamin A capsule programmes, and the indiscriminate scale of these programmes. Evidence for the numbers claimed was never conclusive, and is increasingly embarrassingly lacking as implementation has expanded." pp. 31-32.
  Mason et al. 2018: "These programmes, intended to reduce child mortality, are no longer effective, are wasteful, and are of doubtful safety." p. 1.

• 323

  "The policy to provide vitamin A supplementation twice per year to children aged 6–59 months was based on eight trials from the late 1980s and early 1990s. However, results of two recent trials published in 2013 and 2014 showed no effect of vitamin A supplementation on overall survival, and there are other reasons to question the continuation of vitamin A supplementation." Benn et al. 2018, p. 631.

• 324

  "The first key question is: what impact do these programmes have on child mortality today? Deaths prevented by intermittent high dose vitamin A supplements in the 1980s and 90s were primarily from measles and diarrhoea; measles now has low incidence, with high vaccination coverage, and diarrhoea is better managed. Re-analysis of a Ghanaian study showed that only unvaccinated children benefited." Mason et al. 2018, p. 1.

• 325

  "As hypothesized, the reanalysis suggests important interactions between VAS, sex, and vaccines. VAS was associated with a strong beneficial effect in children with no record of vaccination, whereas there was almost no effect for those who had been vaccinated. This differential effect was due to a difference in girls, in whom VAS was associated with a decrease in mortality in the unvaccinated but in whom VAS was associated with a nonsignificant increase in mortality in the vaccinated (Table 2). This was due to a differential effect of VAS according to vaccination type. Among girls who had already received MV at enrollment, VAS was associated with significantly higher mortality. This was only seen in girls who were missing doses of DTP at enrollment and were therefore likely to receive them during follow-up (Table 5)." Benn et al. 2009, p. 635.

• 326

  "To the contrary: the evidence, supported by many years of my own experience both in the field and in international policy-making committees, which is shared by many colleagues, is that the vitamin A programmes are ineffective. They use up precious human and material resources. Most of all, they impede other approaches to the prevention of vitamin A deficiency, best initiated at national and local level, which need much more support. These include breastfeeding, and the protection and development of healthy, affordable and appropriate food systems and supplies. Such approaches also protect against other diseases, are sustainable, enhance well-being, and have social, cultural, economic and environmental benefits." Latham 2010, p. 15.

• 327
  • Weaknesses in the evidence that VAS reduces child mortality: We discuss our view of the evidence in detail in the section above. We apply a -25% internal validity adjustment to account for the risk that the main finding from the Cochrane meta-analysis is inflated because of weaknesses in the underlying trials.
  • Uncertainty about whether VAS is likely to reduce mortality in contemporary health environments: We account for this with our external validity adjustment (-41% to -79% across locations).
  • Health risks: We discuss these in the section above.
  • Diverting attention and resources away from other public health programs: We account for this through a best guess that 30% of the costs of VAS programs are in-kind government resources like staff time in our cost analysis (more). This should be seen as a rough best guess because it is based on a single ~10 year old study of a deworming program, and our analysis also assumes, based on a rough analysis, that these government resources would have been spent on considerably less cost-effective programs if they were not used for VAS (more).

  However, note that we haven’t reviewed Latham 2010 or Mason et al. 2018 in detail, and so it’s possible that there are criticisms in these commentaries we haven’t yet considered.

• 328
  • Guinea-Bissau:
    • Andersen et al. 2018.
    • Andersen et al. 2021.
  • Bangladesh: Nielsen et al. 2021.
  • Ghana: Welaga et al. 2023.

• 329

  "The starting point is four epidemiological studies from Guinea-Bissau[11, 12], Bangladesh[13] and Ghana (Welaga P et al, in press with EClinicalMedicine), aiming to study the effect of national immunization campaigns with oral polio vaccine (OPV), but also investigating the effect of other campaigns at the same time, as control exposures. They cover a period from 2002-2014 (Guinea-Bissau), 2004-2019 (Bangladesh) and 1996-2015 (Ghana), respectively.
  National immunization campaigns are essentially a natural experiment. We assumed that all eligible children would participate in the campaign and thus children could contribute risk time both as unexposed (prior to a given campaign) and exposed (after the campaign) This eliminates a good deal of confounding from e.g., socioeconomic status (SES) or education. However, it is possible for time-varying covariates to alter the observed effects, and analyses were therefore adjusted for potentially relevant covariates. The studies compared the mortality rate after campaigns with the mortality rate before campaigns in Cox regression models with age as the underlying time scale. Hence, age was inherently controlled for. Children were followed until 36 months of age, movement, death or end of study period, whichever came first.

  Study population: For the present preliminary analysis, I extracted the estimates from the published papers. The two Guinea-Bissau studies are almost overlapping: one is based on the data from within seven randomised trials, the other is based on the data from the overall HDSS from the same period. Part of the children in the randomised trials were from outside the HDSS. For the final analysis, we will include those children as well. For the purpose of this analysis, we just did it with the basis of the HDSS. The inclusion of the children outside the HDSS is unlikely to change the conclusions.

  Age groups: The estimates are based on models that estimated OPV campaign effects from age 0 months to 5 years. For the final meta-analysis, we will conduct the analysis from 6 months onwards. However, very few children received VAS prior to 6 months, and therefore this change in age group will be unlikely to change conclusions substantially.

  On the next page follows the meta-analysis (random effects analysis) of the effect of VAS given with or without deworming, in all children and in males and females where data was available." Benn, Feedback on GiveWell vitamin A supplementation research, December 2023 (unpublished)

• 330

  An alternative version of the meta-analysis, using a fixed-effects analysis, finds a statistically significant increase in mortality of 18% (relative risk 1.18, 95% CI 1.06 - 1.31). We primarily report the random-effects analysis because our understanding is that this method is more appropriate when there is significant heterogeneity between study contexts.
  However, we note that the Cochrane Handbook reports that fixed-effect methods are more appropriate for situations where outcomes are "rare". There does not appear to be an agreed-upon threshold for what makes an outcome "rare", and so in this case we’re unsure which method is more appropriate to use.
  "Methods that should be avoided with rare events are the inverse-variance methods (including the DerSimonian and Laird random-effects method) (Efthimiou 2018). These directly incorporate the study’s variance in the estimation of its contribution to the meta-analysis, but these are usually based on a large-sample variance approximation, which was not intended for use with rare events. We would suggest that incorporation of heterogeneity into an estimate of a treatment effect should be a secondary consideration when attempting to produce estimates of effects from sparse data – the primary concern is to discern whether there is any signal of an effect in the data."
  "There is no single risk at which events are classified as ‘rare’. Certainly risks of 1 in 1000 constitute rare events, and many would classify risks of 1 in 100 the same way. However, the performance of methods when risks are as high as 1 in 10 may also be affected by the issues discussed in this section. What is typical is that a high proportion of the studies in the meta-analysis observe no events in one or more study arms."
  Deeks et al., Cochrane Handbook for Systematic Reviews of Interventions, section 10.4.4 "Meta-analysis of rare events." Accessed February 16th, 2024.

• 331

  Benn, Feedback on GiveWell vitamin A supplementation research, December 2023 (unpublished).

• 332

  Dr. Benn noted that the same analysis indicated that oral polio vaccine (OPV) campaigns were associated with a significant reduction in mortality, and she did not know of biases that could explain the different results for OPV and VAS. Christine Benn, email to Givewell, February 9th, 2024 (unpublished).

• 333

  Hombali et al. 2019 is a systematic review and meta-analysis of the impact of staple food fortification on vitamin A deficiency. It found some limited evidence that fortification programs reduce deficiency, although this was based on very low-certainty evidence and the confidence intervals were extremely wide:

  • Impact of fortification on subclinical deficiency: risk ratio 0.45 (95% CI 0.19 to 1.05), two studies.
  • Impact of fortification on night blindness: risk ratio 0.11 (95% CI 0.01 to 1.98), one study.

  "Staple food fortified with vitamin A versus unfortified staple food
  We are uncertain whether fortifying staple foods with vitamin A alone makes little or no difference for serum retinol concentration (mean difference (MD) 0.03 μmol/L, 95% CI −0.06 to 0.12; 3 studies, 1829 participants; I² = 90%, very low‐certainty evidence). It is uncertain whether vitamin A alone reduces the risk of subclinical vitamin A deficiency (risk ratio (RR) 0.45, 95% CI 0.19 to 1.05; 2 studies; 993 participants; I² = 33%, very low‐certainty evidence). The certainty of the evidence was mainly affected by risk of bias, imprecision and inconsistency.
  It is uncertain whether vitamin A fortification reduces clinical vitamin A deficiency, defined as night blindness (RR 0.11, 95% CI 0.01 to 1.98; 1 study, 581 participants, very low‐certainty evidence). The certainty of the evidence was mainly affected by imprecision, inconsistency, and risk of bias." Abstract.

• 334

  Rahmathullah et al. 1990 (one of the VAS trials in the Cochrane meta-analysis) found that regular, low-dose supplementation (8,333 IU) once a week reduced mortality by 54%.
  "We conducted a randomized, controlled, masked clinical trial for one year in southern India involving 15,419 preschool-age children who received either 8.7 μmol (8333 IU) of vitamin A and 46 μmol (20 mg) of vitamin E (the treated group) or vitamin E alone (the control group). Vitamin supplements were delivered weekly by community health volunteers who also recorded mortality and morbidity. Weekly contact was made with at least 88 percent of the children in both study groups. The base-line characteristics of the children were similar and documented a high prevalence of vitamin A deficiency and undernutrition.
  RESULTS.
  One hundred twenty-five deaths occurred, of which 117 were not accidental. The risk of death in the group treated with vitamin A was less than half that in the control group (relative risk, 0.46; 95 percent confidence interval, 0.30 to 0.71). The risk was most reduced among children under 3 years of age (6 to 11 months — relative risk, 0.28; 95 percent confidence interval, 0.09 to 0.85; 12 to 35 months — relative risk, 0.46; 95 percent confidence interval, 0.26 to 0.81) and among those who were chronically undernourished, as manifested by stunting (relative risk, 0.11; 95 percent confidence interval, 0.03 to 0.36). The symptom-specific risk of mortality was significantly associated with diarrhea, convulsions, and other infection-related symptoms." Abstract.

• 335

  "Preschool children in control villages died at 1.8 times the rate of children in program villages." Muhilal et al. 1988, abstract.
  Note: the reported odds ratio for under-5 mortality in children 12-60 months is 1.87 [1.41, 2.48] (table 6). The denominator and numerator are reversed relative to how they are typically reported (taking the inverse yields an odds ratio of 0.53).

• 336

  Our best guess is that Global Affairs Canada’s grant to UNICEF for programs including VAS has been roughly $10m per year since 2016 (GiveWell first began supporting Helen Keller for VAS in 2017). This is based on:

  • Its 2016-2022 grant for Child Health Days (covering a variety of child health programs including VAS) in 15 sub-Saharan African countries being $75m over roughly 5 ½ years (approximately $10m USD, assuming the $75m is Canadian dollars).
  • Helen Keller informed us in 2023 that UNICEF had received a renewal grant of $25m USD for VAS from mid 2023 to the end of 2025 (equating to approximately $10m a year).

  "UNICEF received a USD $25 million grant from Global Affairs Canada to support VAS in 15 countries (Angola, Benin, Burkina Faso, Cameroon, CAR, Chad, Cote d‘Ivoire, DRC, Guinea, Madagascar, Malawi, Mozambique, Sierra Leone, Sudan, Togo) from mid-2023 to end-2025 (5 Semesters). However, considering that some funds will be used for global studies and not all funds can be directly allocated to VAS support in the field, the actual direct VAS delivery support per semester and per country is less than USD $200,000. As a result, UNICEF has informed Helen Keller that it can no longer provide significant support for VAS in most countries. For instance, in Cameroon, Helen Keller had to cover the entire country in 2023, which doubled the number of targeted children and the required funding for the campaigns." Helen Keller International, Room for More Funding Report, 2023, p. 2.

• 337

  This understanding is based on information from Helen Keller that UNICEF’s funding for VAS has declined in specific countries. "In 2021, Helen Keller International significantly increased its support for Vitamin A Supplementation (VAS) in multiple countries following the large reduction in funds available from one of the main VAS supporters--UNICEF. With support from GiveWell, Helen Keller was able to close part of the funding gap and distributed ~50 million capsules through semi-annual VAS campaigns in 2021. More recently, reductions in UNICEF VAS funding in 2022 were less severe overall, but remain significant in countries such as Kenya, Nigeria or Niger. Although UNICEF thought it could provide greater support to Guinea, Mali and Côte d’Ivoire in 2022, their actual level of support remains lower than in the previous years. The likelihood of increased UNICEF support in these countries in the near future is low. Filling UNICEF’s funding gaps has led Helen Keller to spend higher amounts of GiveWell funds than initially budgeted, causing estimated funding gaps in each country beginning in 2024 and creating 'room for more funding' for these countries." Helen Keller International, Room for More Funding Report, 2022, p. 2.

• 338

  "WHO recommends that consideration be given to mass drug administration of azithromycin to children 1 to 11 months of age for prevention of childhood mortality in sub-Saharan African settings in which:

  • infant mortality is > 60 per 1000 live births or under-five mortality is > 80 per 1000 live births respectively, and
  • infant and under-five mortality rates, adverse effects and antibiotic resistance (AMR)

  are continuously monitored, and
  implementation of existing child survival interventions, including seasonal malaria chemoprophylaxis where recommended, is concurrently strengthened.
  (Conditional recommendation, low quality evidence)." WHO, Guideline on mass drug administration of azithromycin to children under five years of age to promote child survival, 2020, p. ix.

• 339

  Azithromycin is hypothesized to avert mortality through its effect on respiratory infection, diarrhea and malaria in particular, and we think that the main impact of VAS is averted diarrhea mortality, although there’s significant uncertainty around both of these. See this section of our separate report on azithromycin for more details.

• 340

  Oral rehydration solution.

• 341

  We estimate that each dollar donated to GiveDirectly’s direct cash transfer program generates 0.00335 units of value. 0.005 / 0.00335 = ~1.5.

• 342

  We estimate that each dollar donated to GiveDirectly’s direct cash transfer program generates 0.00335 units of value. 0.006 / 0.00335 = ~1.8.

• 343

  We estimate that each dollar donated to GiveDirectly’s direct cash transfer program generates 0.00335 units of value. 0.012 / 0.00335 = ~3.5.

• 344

  Note that these figures don’t correspond with the proportion of costs paid by different actors in this section of our cost-effectiveness analysis. We estimate that for each $1m spent by Helen Keller in Guinea, Nutrition International incurs ~$88,000 in costs and the Guinea government incurs ~$543,000.
  The reason the figures above are smaller than this is that some program costs are also borne by other philanthropic actors (in Guinea, WHO). Our best guess is that if Helen Keller didn’t fund the program, these costs would be unaffected and the Guinea government / Nutrition International costs would shrink proportionally to WHO’s costs. These are around 15% as large as Helen Keller and WHO’s costs combined. ~543,000 x 15% = ~$460,000 and ~$88,000 x 15% = ~$75,000. See this cell and this cell in our cost-effectiveness analysis for our calculations.