Note: This page summarizes the rationale behind a GiveWell grant to Evidence Action's Iron and Folic Acid Supplementation program. Evidence Action staff reviewed this page prior to publication. The page reflects our rationale at the time the grant was recommended.
Summary
We recommended this grant because our best guess is that the program is 11.5 times as cost-effective as direct cash transfers. Although we have a high level of uncertainty about this estimate, we think there is a strong intuitive case for the cost-effectiveness of the grant.
Our main reservations are:
- We have low confidence in our estimate of how much the grant will increase IFA coverage relative to the counterfactual.
- We're uncertain about the reliability of the data that informs our estimates of anemia burden. We're also unsure how much to expect the anemia burden to decrease over time in the absence of Evidence Action's program.
- Although Evidence Action plans to collect coverage data during the grant period, we don't think this will be very informative for our understanding of the program's effect on coverage.
What we think this grant will do
Evidence Action will provide support to the government at the state and district level to improve a government-run program aimed at reducing anemia. This support includes policy advocacy, resource planning and program management, supply chain management, training, community awareness, and monitoring and evaluation (M&E). Children will be reached through schools, childhood education centers known as anganwadi centers (AWCs), and by community health workers known as Accredited Social Health Activists (ASHAs). The Evidence Action IFA program will share some staff with Evidence Action's Deworm the World Initiative, since the two programs work with a similar set of stakeholders.
The recommended grant will support four years of technical assistance (TA), four coverage surveys, advisory services from IDinsight, and evaluation of a pilot program for private school IFA supplementation.
Why we made this grant
We're recommending this grant because:
- We think this is a cost-effective use of funding. Our best guess is that the program is 11.5 times as cost-effective as direct cash transfers (or "11.5x cash").1
While we have tried to account for major considerations impacting cost-effectiveness, we still have a high level of uncertainty about this estimate relative to those of our top charities. However, we think there is a strong intuitive case for the cost-effectiveness of this grant due to the reasons listed below.
- There is a high burden of iron deficiency and anemia in states where Evidence Action plans to provide TA. In our cost-effectiveness model, we use Institute for Health Metrics and Evaluation (IHME) estimates of anemia rates and iron deficiency. IHME estimates an anemia rate of 56% among children under five across India and higher iron deficiency prevalence than average in states where Evidence Action plans to work. (More)
- Iron deficiency and anemia can have adverse physical and cognitive effects. Iron deficiency and iron deficiency anemia are associated with a wide range of negative physical, psychological, and cognitive effects. (More)
- IFA supplementation seems to be an effective intervention to reduce iron deficiency and anemia. Based on evidence from several randomized controlled trials and the highly plausible biological mechanism by which IFA supplementation operates, we consider the evidence for iron supplementation's impact on iron deficiency and anemia to be reasonably strong, though we have some pending uncertainties. (More)
- We think Evidence Action has a good chance at being effective at increasing IFA coverage. This is based on Evidence Action successfully providing TA to support deworming through Deworm the World in India, which leverages a similar platform; policymakers in India saying they believe Evidence Action’s TA will be helpful; and Evidence Action providing a plausible explanation of how its work can alleviate bottlenecks to increasing IFA coverage. Although we have previously funded Evidence Action's IFA program in India, we don't have a strong sense of Evidence Action's track record for this work, as the earlier years of the program were significantly disrupted by the COVID-19 pandemic. (More)
- There is a funding gap. Based on conversations with Evidence Action and local stakeholders, we think this project is unlikely to be supported if we decide not to fund it. (More)
Main reservations
Our key reservations and uncertainties are:
- How much this grant will increase IFA coverage over time. (More)
- How much Evidence Action will increase IFA coverage. Our estimate is based on Evidence Action's best guess for how much they will increase coverage, which we sense-checked by comparing it to other sources, including estimates from other TA programs GiveWell has supported and estimates from Indian government officials. We expect to have limited ability to learn about this during the course of the grant.
- How long any increase in coverage from the program is likely to persist after the program ends. We assume that the government's IFA program will continue to benefit from the TA Evidence Action provided for five years after Evidence Action stops providing TA. This is comparable to what we estimate for other TA programs we support, but we're highly uncertain about it. If we remove this benefit from our model (i.e., if we assume that the coverage benefits end when the program ends), the grant would fall below our cost-effectiveness bar.
- Anemia burden. We rely on IHME data to inform our estimates of anemia burden in the states where Evidence Action is providing technical assistance. However, we're unsure how reliable these estimates are or how much we should expect the anemia burden to decrease over time in the absence of Evidence Action's program. (More)
- Limited ability to learn about the impact of Evidence Action’s TA on IFA coverage. Evidence Action plans to collect coverage data in all five states at the beginning and end of the grant period. However, we expect this information to be only weakly informative of the TA’s effect on coverage because:
- It will not be possible to precisely measure Evidence Action's effect on IFA program coverage and thus to distinguish its effect from that of other programs aimed at addressing iron deficiency and anemia.
- We understand that there is potential for natural fluctuations in coverage from year to year within states, even without the intervention.
We thus expect to be left with a high degree of uncertainty about the effect of this program on IFA coverage, even after the grant. (More)
Table of Contents
Published: February 2024
The organization
This IFA supplementation technical assistance (TA) program is run by Evidence Action, an organization with which we have a long-standing relationship and to which we have provided significant support.2 In March 2018, we recommended funding to the Evidence Action Accelerator to scope a technical assistance program for IFA supplementation in India. Based on the results of this scoping work, we recommended a grant in December 2018 to Evidence Action to scale a pilot of the IFA TA program and a grant in March 2019 to evaluate the program.
The program was originally intended to reach children in schools and anganwadi centers (AWCs). However, due to the COVID-19 pandemic and subsequent school and AWC closures, the government shifted to providing IFA supplementation in community settings. Since Evidence Action did not expect this delivery method to be representative of its long-term work and impact, the midline and endline evaluation did not take place.3 That portion of funding was used instead for program costs. In July 2022, as schools and AWCs reopened, Evidence Action once again targeted school- and AWC-based delivery.4
We also considered funding an impact evaluation of this program. We decided not to make that grant because of concerns about quality of the evidence, limited generalizability outside India, and limited room for more funding for Evidence Action IFA TA in India. Evidence Action did not have concerns about this decision.5
Evidence Action–IFA activities
In 2018, the Indian government launched Anemia Mukt Bharat (AMB) with the goal of reducing anemia across India.6 The Ministry of Health & Family Welfare, under the National Health Mission, makes AMB funding and technical guidance available for states to conduct IFA supplementation activities.7 States must request AMB funding from the national government, plan, and deliver the AMB IFA program. Evidence Action supports states as they conduct these activities.
Evidence Action’s IFA program operates in five states in India: Rajasthan, Madhya Pradesh, Jharkhand, Uttarakhand, and Haryana.
In 2022-2023, these states plan to target roughly 59 million children, ages 6 months to 19 years old, for IFA supplementation.8 This includes: children 6-59 months old (reached through ASHAs and AWCs), children 5-19 years old who attend schools (reached through schools), and out-of-school girls 10-19 years old (reached through AWCs).
Evidence Action provides technical assistance for IFA supplementation at the state and district levels. Evidence Action focuses on addressing bottlenecks in the following areas:9
- Resource planning and program management10
- Supply chain management11
- Training12
- Policy and advocacy13
- Information, education and communication (IEC) and community mobilization14
Based on a 2019 survey, Evidence Action believes the key bottlenecks its assistance is addressing are lack of awareness of the IFA program and product availability.15
Evidence Action expects it can begin phasing out its TA support in states after 4-5 years, with the theory that states will have sufficiently strong systems in place to manage the program on their own. The phase-out process is expected to take about two years.16
Staff structure
Evidence Action’s IFA staff occupy roles at the global, national, and state levels. The IFA TA program will share some staff with Evidence Action's Deworm the World Initiative, including national program managers, state program managers, state program coordinators, and regional coordinators.17
Monitoring and evaluation data
Evidence Action’s monitoring and evaluation strategy includes two components: program process monitoring and program evaluation.18
Program process monitoring surveys will collect data on the outputs of Evidence Action’s TA and expected drivers of coverage at the district and block level, including the IFA supply chain, training, program awareness and community mobilization, and reporting. These surveys will be administered to district- and block-level government officials.
Evidence Action’s program evaluation will evaluate program outcomes, IFA coverage, and program processes.19 Specifically, the evaluation of program processes will include an examination of the theory of change assumptions, a review of process and coverage data, and interviews with local stakeholders.20 IDinsight will support Evidence Action with program evaluation design, stakeholder data collection, and evaluation analysis.21
Program process monitoring and program evaluation are intended to help Evidence Action improve its ability to track key outputs, refine its theory of change, and modify its approach if necessary.
Evidence Action will collect data on IFA program coverage and drivers of coverage to inform its program process monitoring and program evaluation. These data will be collected through three types of data collection:
- Coverage validation surveys,22 conducted from 2022-2025, to estimate the percentage of targeted children that receive and consume IFA supplementation.23 These surveys will also collect data on drivers of coverage – such as awareness, stock availability, training, and reporting – at the frontline worker level. Surveys will be administered to teachers, anganwadi workers (AWWs), and ASHAs for their respective beneficiary groups.
- Program process monitoring (PPM) surveys (described above).
- Qualitative interviews with key stakeholders to collect data on activities and indicators where there are gaps in the other data sources. This qualitative data is intended to complement the quantitative data and provide more detail on program processes.24
Room for more funding
We believe that Evidence Action’s IFA TA in India could absorb approximately $30 million in additional funding in the future (“room for more funding”).25 This is a rough estimate, based on Evidence Action’s estimate that the cost per child will be similar in additional states, and our assumption that 30% of India's population will be in states where there’s government buy-in for the program and the program is cost-effective.26 It also includes the cost of extending the program by three years in states covered by this grant.27
The intervention
Iron and folic acid supplementation
IFA supplementation is done by providing oral supplements, in either syrup or tablet form.28 Evidence Action supports IFA supplementation programs that aim to reach school going children ages 5-19 years and out-of-school adolescent girls ages 10-19 weekly and children ages 6-59 months twice per week.29
We expect most of the benefits of this intervention to come from iron, rather than folic acid, supplementation.30 In our intervention report on iron supplementation, we write: "There is strong evidence that iron supplementation reduces cases of anemia. There is weak to moderate quality evidence that iron supplementation, with or without folic acid, increases cognitive ability." We discuss the evidence for IFA supplementation in greater detail in that report.
We have several uncertainties about the evidence for iron fortification/supplementation programs, which we are currently working on addressing (more).
The grant
We're recommending a grant of $9,185,495 for four years, from mid-2022 to mid-2026.31 This grant is being funded by Open Philanthropy.
Budget for grant activities
Evidence Action’s overall budget for the activities covered by the grant is $10,786,671. Evidence Action currently holds $1.6 million in program reserves for the IFA TA program, which we subtracted from the budget to estimate the grant size. This includes:32
- Four years of maintenance, covering Evidence Action’s activities: $8,027,257
- Four coverage validation surveys, which will estimate the IFA coverage in Evidence Action-supported states in 2022 to 2025: $899,280
- Advisory services from IDinsight, to advise on and analyze results of coverage surveys and monitor data tracking Evidence Action’s key levers of change: $206,832
- Evaluation of a pilot for private schools and advocacy activities to promote the scale up of IFA supplementation in private schools if the pilot currently run by Evidence Action is successful33 : $52,126
The case for the grant
Cost-effectiveness
Our best guess is that the IFA program supported by Evidence Action's TA is 11.5 times as cost-effective as direct cash transfers. We estimate that, on average, over a 10-year timeline, IFA coverage with Evidence Action’s support will be 10 percentage points higher than it would be without its support.34 The 10-year timeline includes program duration and duration of benefits after the program is discontinued. Program benefits include averting morbidity from anemia (~80% of total benefits), and improving cognitive performance (and thus income) in children (~10% of total benefits) and adults (i.e., young people 15-19 years old of age, ~10% of total benefits).
Intuitive case for cost-effectiveness. Due to uncertainty around some key parameters, this cost-effectiveness estimate is more uncertain than for other programs to which we’ve recommended funding (more). Although we're uncertain about our precise cost-effectiveness estimate, there is a strong intuitive case explaining why the grant is cost-effective:
- There is a high burden of iron deficiency in India (more).
- There is reasonably strong evidence that IFA supplementation reduces iron deficiency (more).
- There is a funding gap for this work (more).
High burden of iron deficiency
We model the benefits of IFA supplementation as occurring through reduced iron deficiency, which is the most common cause of anemia.35
In our cost-effectiveness analysis, we rely on data from the Institute for Health Metrics and Evaluation (IHME). These data show anemia rates of 56% among children under 5 in 2019.36 We’ve also triangulated this against data from India’s National Family Health Survey-5, which found that 67% of children under 5 were anemic in 2019-2021 – an increase from 59% in 2015-2016.37
IHME data show the five states targeted by Evidence Action for TA have higher iron deficiency prevalence than the average rates for India.38
Evidence that IFA reduces iron deficiency
Our best guess is that receiving IFA reduces iron deficiency by about 70%.39 We view the evidence of the effects of IFA supplementation as reasonably strong because it comes from a large number of RCTs40 and because there is a plausible biological mechanism for the effect of iron supplementation on iron deficiency.41 See our intervention report on iron supplementation for school-aged children for more information.
However, we have several uncertainties about the evidence for iron fortification and supplementation programs. For example, we are uncertain about several factors that could influence iron absorption, as well as the effects of iron absorption on health outcomes.42 However, we think it’s relatively unlikely that additional work on the iron supplementation research would overturn the case for this grant.
We estimate the IFA supplementation program supported by Evidence Action decreases iron deficiency by 57%; this accounts for the fact that supplementation frequency in the program is lower than in the studies we base our estimate on.43
Evidence Action’s ability to increase IFA coverage
Three factors make us think Evidence Action is likely to be able to meaningfully increase IFA coverage through its TA activities.
- Evidence Action has a strong track record with Deworm the World in India since 2009.44 The IFA team will share some staff with Evidence Action’s Deworm the World Initiative, including state program managers, state program coordinators, and regional coordinators.45
- Policymakers in India have indicated they find Evidence Action’s TA valuable and likely to be successful at increasing IFA coverage. State government officials indicated that without Evidence Action support, IFA coverage will likely be much lower.46
- Evidence Action has provided a plausible explanation about the bottlenecks to increasing IFA coverage and how its TA will alleviate those bottlenecks. Evidence Action and IDinsight’s 2019 baseline survey (covering 5,025 schools in 11 states) identified two key bottlenecks to achieving coverage: awareness of the IFA program and product availability.47 Evidence Action plans to address awareness through training and orientations for teachers, health workers, and local government officials, while gaps in product availability will be addressed by providing supply chain support to local governments, schools, and AWCs.48
There is a funding gap
Our impression is that Evidence Action's TA for the IFA program would not be funded absent a GiveWell-recommended grant, and that there is not another organization implementing a comparable program of TA for IFA in India. This is based on conversations with Evidence Action as well as local stakeholders.49
Risks and reservations
Our key reservations and uncertainties follow.
Quantitative reservations
We're uncertain how much Evidence Action’s TA will increase IFA coverage over time
We do not have strong benchmarks to inform our estimates of how much a technical assistance program like this one is likely to increase IFA coverage. We estimate that, on average, over a 10-year timeline, IFA coverage with Evidence Action’s support will be 10 percentage points higher than it would be without its support.50 This is based on Evidence Action's best guess, which we sense-checked by comparing it to other sources, including estimates from other TA programs GiveWell has supported and estimates from Indian government officials. We think our estimate is plausible, but we're highly uncertain about it.
We're also uncertain how long any increase in coverage from the program will persist after Evidence Action stops providing technical assistance to the government. We expect the coverage benefits to persist at least for a while because we expect the government to continue to use systems and processes that were set up with Evidence Action's support; over time, it's less clear whether those systems will be maintained or if new challenges will arise that will be more difficult to address without Evidence Action's support. We assume that the benefits will persist for five years after full engagement by Evidence Action stops. This is consistent with how we have modeled other TA programs.
If we exclude the benefits from Evidence Action's TA over the five years following the end of the grant, the program would model below our cost-effectiveness bar.
We're unsure how reliable our estimates of anemia burden are and how we should expect it to change over time
We rely on IHME data to inform our estimates of anemia burden in the states where Evidence Action is providing technical assistance. However, we don't have a good understanding of how reliable these estimates are at the national level, how much stock we should put in state-level differences, and how much we should expect the anemia burden to fall over time in the absence of Evidence Action's program.51 We sense-checked the IHME estimates against National Family Health Survey data on anemia for children under five, and it appears reasonable, but we are unsure.52
Because the above reservations make us highly uncertain about the quantitative estimate of this grant's cost-effectiveness, we put a high weight on the intuitive case for the grant—the fact that there is a plausible story for how Evidence Action can have impact that is supported by some evidence.
Other reservations
We will have limited ability to learn about Evidence Action’s success at increasing levels of IFA coverage
We will have pre-post coverage estimates for states that Evidence Action works in (see above for details). However, we expect those will only be weakly informative of the program’s effect on coverage because (i) it will not be possible to distinguish Evidence Action's effect from that of other programs, especially AMB on its own, and (ii) IDinsight has noted there is potential for natural fluctuations in coverage from year to year within states, even without the intervention, which would reduce the informativeness of pre-post coverage estimates.53
Plans for follow-up
We plan to check in with Evidence Action on the grant's progress each quarter. We also plan to review baseline (2022) and endline (2024)54 surveys, and specifically:55
- Percent of all targeted school children who have consumed an IFA tablet each week in the last 4 weeks,
- Percent of all targeted out-of-school (OOS) girls ages 10-19 years who have consumed an IFA tablet each week in the last 4 weeks, and
- Percent of all targeted children ages 6-59 months who have consumed 1ml of IFA syrup twice each week in the last 4 weeks.
We plan to review IDinsight’s report on progress monitoring after the endline survey. After that, we plan to review yearly data on coverage from coverage validation surveys. We may also decide to conduct, either independently or in collaboration with IDinsight and Evidence Action, conversations with government officials and other key stakeholders on the “causal chain” from Evidence Action’s TA to higher IFA coverage.
All these check-in points will allow us to continue to update our understanding of the cost-effectiveness of this grant and determine if this grant should be renewed or expanded to cover other states in India or in other countries.
Internal forecasts
For this grant, we are recording the following forecasts:
Confidence Prediction By time 80% Evidence Action will have completed a baseline survey of IFA coverage. December 2022 80% Evidence Action will have completed an endline survey of IFA coverage. December 2024 60% The difference in IFA coverage between baseline and endline survey will be greater than 10.5% over the 18-month evaluation period (for reference, Evidence Action’s estimate of its effect on coverage is 14% after 2 years). March 2025 60% Our best guess of the cost-effectiveness of the IFA program supported by Evidence Action will be above 8x cash in all five states in India that it is currently operating in. June 2025 60% We will have recommended funding for geographic expansion of the IFA TA program in at least one location (such as additional states in India, Nigeria, or another country). June 2025 Sources
- 1
For more information on how we use cost-effectiveness estimates, see this page.
- 2
For example, see here for all grants we have recommended to Evidence Action's DeWorm the World Initiative since 2014. We have also supported the Evidence Action Accelerator, an incubator portfolio within Evidence Action focused on GiveWell-aligned, evidence-backed, and cost-effective interventions.
- 3
"The evaluation aimed to measure IFA supplementation coverage among children at schools and anganwadi centers (AWCs), designated delivery sites by the government. Baseline data was collected from August to October 2019, with the expectation that midline collection would take place in 2020 and endline collection in 2021. However, in light of COVID-19 related school and AWC closures, in April 2020, the government moved to an interim model of community-based delivery where IFA is delivered at households. While the Evidence Action team continued to provide technical support on community-based delivery of IFA supplementation, the evaluation was put on hold given our aim is to measure the impact and cost-effectiveness of our TA support for school and anganwadi-based delivery of IFA supplementation." Evidence Action, IFA Evaluation Updates for Givewell, 2022 (unpublished)
- 4
"With schools and AWCs gradually reopening, the program is preparing for a return to site-based delivery." Evidence Action, IFA Evaluation Updates for Givewell, 2022 (unpublished)
Evidence Action confirmed in its review of this grant page that this work began in July 2022.
- 5
"There are no major considerations beyond what we’ve already shared in our recent note. We do believe there is still value in conducting an evaluation for our own learning and decision-making, but we believe Option 2 in the note (a pre-/post- performance evaluation) would suffice for our needs." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 6
"With the GoI's [Government of India's] launch of the Anemia Mukt Bharat (AMB) campaign in March 2018, the government was expressing an urgency to act and reduce anemia rates." Evidence Action, IFA Phase II context and updates, 2020 (unpublished)
- 7
"As it launched this initiative, the GoI made its AMB support uniformly available to states (including available budget, technical guidance, etc.), and it was up to the initiative of state governments to leverage those resources." Evidence Action, AMB Launch Timelines, 2020 (unpublished)
- 8
Evidence Action, IFA investigation plan, 2022 (unpublished)
These state targets have been updated since we recommended the grant. This grant page reflects our understanding at the time the grant was recommended.
- 9
"Overall, our technical assistance model aims to address bottlenecks across the following areas: policy and advocacy, resource planning and program management, supply chain management, training, IEC and community mobilization, as described in this overview of our theory of change. We are in the process of finalizing updates to the theory of change and related indicators." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 10
This will include:
- "State-level IBD [(institution-based delivery)] resumption guidance prepared and provided to state-level governments
- Annual PIP support to state health depts for IFA target groups (activities, targeting, budgeting)
- Quarterly tracking on IFA activity progress/completion against the planned activities in the PIP
- Coordinate state and district inter-dept IFA program review meetings (format, cadence, participants)
- Generate annual state-level and quarterly district-level reports on IFA delivery progress for interdepartmental meetings
- Monthly follow-up by Regional Coordinators at district- and block-level on IFA HMIS reporting"
Evidence Action, IFA theory of change powerpoint slides, 2022, slide 4
- 11
This will include:
- "Support state annual/biannual quantification and forecasting
- … [Coordinate with state on the drug delivery against the ordered quantity]
- … [Support states in accessing supply status and suggest corrective actions]
- Develop drug distribution plans from the regional warehouse to school/AWC level with each new drug batch
- Monitor stock availability and pipeline at the district and block-level
- Ensure districts, blocks, and PHCs update IFA stock position data in the state government's drug logistics portal (e.g. "e-Aushadhi") on a monthly basis
- RCs troubleshoot with district- and block-level officials to ensure there are sufficient IFA supplies and timely distribution happening to lower level
- Analyze IFA supply data pulled from the state government's drug logistics portal and provide district- and block-level recommendations on a monthly basis"
Evidence Action, IFA theory of change powerpoint slides, 2022, slide 5
- 12
This will include:
- "Develop comprehensive state adaptations of the AMB training package (including technical/clinical information, supply management, reporting requirements and IFA administration guidelines) that are aligned with state requirements and the most recent national technical guidelines
- Support the development of training cascade schedules and plans (including estimating training case load) to cascade trainings to district and block officials, school nodal teachers, ANMs, AWWs and ASHAs
- Develop estimates of training caseloads and develop training monitoring databases
- Monitor training coverage, according to training cascade plans, and training quality, and recommend improvements for state IFA supplementation training content and implementation, and share them with the government"
Evidence Action, IFA theory of change powerpoint slides, 2022, slide 6
- 13
This will include:
- "Conduct district-level private school pilots in five TA states and generate evidence on the feasibility and performance of the program model
- Advocate for inclusion of private school children in state-level program guidelines and Program Implementation Plans (PIPs) [pending pilot results]"
Evidence Action, IFA theory of change powerpoint slides, 2022, slide 3
- 14
This will include:
- "Develop state IFA supplementation community mobilization strategies
- Based on the community mobilization strategies, develop state-specific IFA supplementation IEC materials using both digital and offline communication methods"
Evidence Action, IFA theory of change powerpoint slides, 2022, slide 7
- 15
Evidence Action, IFA investigation plan, 2022 (unpublished)
- 16
- "Once higher IFA coverage is achieved and government systems have been strengthened, Evidence Action would aim to transition to lighter touch support models and eventually phase out its support. Our aim is to put in place sustainable systems and tools that can outlive our support." Evidence Action, IFA investigation plan, 2022 (unpublished)
- "With DTW [(Deworm the World)], we have historically provided 3-5 years of intense engagement in a given state before moving to a less intense engagement model. The length of this support has been determined by evidence of high performance in terms of coverage and reduced worm prevalence. Some states … have required much longer runways.
With the IFA program, given it is a routine program that requires weekly/bi-weekly administration, rather than a campaign-based program, we anticipate needing to provide longer support. We propose that our engagement in a given state would last for around 6-7 years:
- ~4-5 years of more intense engagement with dedicated support at the state and sub-state levels. This would include the presence of field staff to support with addressing delivery challenges.
- ~2 years of less intense engagement, focused on state-level technical and implementation support. We expect that the budget would reduce to ~50-70% of the first phase." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 17
- State program managers, state program coordinators, and regional coordinators are key positions in the field and will be shared with Evidence Action's Deworm the World Initiative. Regional coordinators will split their time 70/30 between IFA and deworming. Evidence Action, Conversation with GiveWell, April 4, 2022 (unpublished)
- Evidence Action told us this integration is due to the deworming program relying on the same stakeholders as the IFA program. For example, the same state nodal officer may be responsible for IFA as well as deworming; Evidence Action will thus have one staff person as the point of contact for that relationship. Evidence Action, Conversation with GiveWell, April 4, 2022 (unpublished)
- Evidence Action, IFA Staff List (unpublished)
- 18
This section reflects our understanding of Evidence Action's planned monitoring, learning, and evaluation (MLE) at the time we recommended the grant. Its MLE strategy has evolved since then.
- 19
"[W]e believe Option 2 in the note (a pre-/post- performance evaluation) would suffice for our needs. As a reminder, this option attempts to calculate an average change in program coverage across all five TA states and includes a process evaluation component to analyze our theory of change and identify levers of coverage change." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 20
"The PE [(Process Evaluation)] will provide insights on the perceived 'why’s' and 'how’s' of the TA’s impact on program delivery and coverage. Consequently, it will inform EA’s decisions on whether and how to modify their TA program (...)
"To address the research questions outlined above, IDinsight will undertake the following activities:
- Examine ToC assumptions: The link between every activity and corresponding outputs and outcomes will be thoroughly studied to determine if there are other confounding factors involved.
- Conduct a ToC prioritization workshop: Owing to the scope of the technical assistance provided by EA, it is essential to identify activities and indicators which are most critical for program success. IDinsight will organize a workshop with EA to identify and prioritize the most important activities.
- Review of process data: We will conduct a thorough review of EA’s process and monitoring data. Based on the prioritized activities and indicators, we will determine if any new data needs to be collected.
- Conduct interviews with key stakeholders: We will conduct qualitative interviews with stakeholders to complement the quantitative data. This will be particularly important for activities and indicators where data gaps have been identified." IDinsight, IFA process evaluation overview (unpublished)
IDinsight is refining the PE design to be more specific and relevant for Evidence Action's evaluation.
- 21
"For pre- and post-analysis, we expect IDi will conduct the ‘baseline’ and ‘endline’ state-level analysis for the process and coverage evaluation, similar to how they planned to conduct this analysis for the impact evaluation. They will integrate the process evaluation results with detected changes in coverage to assess our performance along the TOC’s causal pathway and examine how this performance influenced any changes in coverage." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 22
Evidence Action now refers to “coverage validation surveys” as “coverage evaluation surveys.” At the time of the investigation, the term “coverage validation surveys” was used and is therefore included as such in this grant page.
- 23
- "Data collected: based on the ToC document, slide 14, we would guess you plan to collect the data listed below. Could you confirm this is right?
- % of all targeted school children (government and private, pending pilots) who have consumed an IFA tablet each week in the last 4 weeks (reported separately for pink and blue tablets, and school type), unadjusted and adjusted for average attendance
- % of all targeted OOS girls ages 10-19 years who have consumed an IFA tablet each week in the last 4 weeks
- % of all targeted children ages 6--59 months who have consumed 1ml of IFA syrup twice each week in the last 4 weeks
Yes, we are planning to collect the data required to inform all the indicators mentioned through our coverage validation (CV) surveys." Evidence Action, IFA investigation plan, 2022 (unpublished)
- "Based on an 18-month timeframe between the baseline and endline, we propose to incorporate non-layered/layered CV surveys according to the timelines below. It would be useful for us to incorporate an interim layered CV in 2023 to get a read on progress and make necessary course corrections, even if this isn’t required for GiveWell purposes.
- Nov - Dec 2022: new 'baseline' non-layered CV, assuming 3 months have passed since site-based delivery has resumed and 'steady state' has been achieved
- Aug - Sep 2023: Layered CV
- Jul - Aug 2024: 'endline' non-layered CV, timed 18 months after the baseline. It can’t be conducted in May/June due to the summer vacation period.
- Aug - Sep 2025: Layered CV" Evidence Action, Email to GiveWell, June 1, 2022 (unpublished)
- "Data collected: based on the ToC document, slide 14, we would guess you plan to collect the data listed below. Could you confirm this is right?
- 24
"To address the research questions outlined above, IDinsight will undertake the following activities:
[. . .] 4. Conduct interviews with key stakeholders: We will conduct qualitative interviews with stakeholders to complement the quantitative data. This will be particularly important for activities and indicators where data gaps have been identified." IDinsight, IFA process evaluation overview (unpublished) - 25
See here for calculations.
- 26
- "What % of the total population do you expect to be able to reach in India and Nigeria?
India
In India, there is a lot of room to support the scale up of IFA supplementation for children 6 months - 19 years of age. If funding is not a constraint, we expect that we could eventually provide TA support across most states, conditional on there being government buy-in and the intervention being cost-effective. Whether the intervention is cost-effective largely depends on the state-level anemia prevalence rates for our target populations and our ability to support the government to increase IFA coverage.
Our target population of children 1-19 years of age makes up 35% of the total population (Global Burden of Disease, 2019). Within a given state, we would take a phased approach in reaching this population, first targeting government school children, children <5 years and adolescent out of school girls and later expanding to private school children. 35% of school going children in primary through upper secondary school attend private school. We would aim to ultimately achieve 75% coverage of the total target population in partner states." Evidence Action, IFA investigation plan, 2022 (unpublished) - "Do you expect costs per child to be significantly different from current ones?
India
We expect the cost per child to be similar in expansion states as to our current states." Evidence Action, IFA investigation plan, 2022 (unpublished) - 30% is our rough guess of the proportion of India's total population that resides in states where Evidence Action is likely to get government buy-in and for the program to be cost-effective. We have not tried to model cost-effectiveness in other states.
- "What % of the total population do you expect to be able to reach in India and Nigeria?
- 27
See here for calculations.
- 28
- "Iron supplementation is the oral consumption of iron-containing compounds, typically in pill form." GiveWell, "Iron supplementation for school-age children"
- "Based on the technical guidelines, children <5 are recommended to receive 2 doses of IFA syrup per week and children 5-19 years are recommended to receive one IFA tablet weekly." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 29
"With the IFA program, given it is a routine program that requires weekly/bi-weekly administration…" Evidence Action, IFA investigation plan, 2022 (unpublished)
- 30
In the last grant we recommended to Evidence Action, we wrote, "In this report, we primarily discuss the effects of iron rather than folic acid.
We have not carefully considered the potential benefits of folic acid as distinct from iron. Our understanding is that in some of the iron supplementation studies we rely on, folic acid is also part of the treatment. However, our rough impression from reviewing the folic acid literature is that most of the potentially important public health impacts of folic acid come from providing supplements to pregnant women to reduce birth defects,10 and we expect that pregnant women would be a small portion of beneficiaries targeted by this program.11
To simplify our analysis, we have focused on the effects of iron. However, our analysis would be conservative if the public health effects of folic acid in this population are larger than we currently expect." GiveWell, "Evidence Action Beta — iron and folic acid supplementation ('Phase 2')"
We are not aware of any further work on folic acid that has been conducted since. - 31
- 32
See the breakdown of costs here.
- 33
"Our request for a private school pilot evaluation budget of $51,160 under 'Proposed Additional Budget Items' will allow for Evidence Action to independently conduct coverage validation and process monitoring surveys in the pilot districts. These survey results will help Evidence Action to build a case to state governments on the feasibility of scaling up IFA delivery to private schools in more districts in the state, with the eventual goal of state-wide scale-up." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 34
See calculations here.
- 35
“Iron deficiency is the most common cause of anemia, a condition in which hemoglobin production is diminished, and is believed to contribute to at least half of the global burden of anemia.” GiveWell, "Iron supplementation for school-age children"
- 36
Institute for Health Metrics and Evaluation, GBD results tool, "Impairment: Anemia"
- 37
Government of India, Ministry of Health & Family Welfare, National Family Health Survey (NFHS-5) 2019-21, p. 5, row 92 "Children age 6-59 months who are anaemic"
- 38
See calculations in the "prevalence of dietary iron deficiency" section of this sheet.
- 39
- We base our estimates on the results of a 2013 meta-analysis, which estimates an ~80% reduction in iron deficiency from iron supplementation. See Low et al. 2013, Table 3, section "Hematology and iron indices." We apply a downward adjustment of 90% to account for concerns about publication bias.
- Reduction in iron deficiency: 79% (Low et al. 2013) x 90% = 71.1%
- 40
- "We identified 16 501 studies; of these, we evaluated 76 full-text papers and included 32 studies including 7089 children." Low et al. 2013, p. E791
- "We included randomized controlled trials that included primary-school–aged children (5–12 yr) who were randomly assigned to daily (≥ 5 d/wk) oral iron supplementation or control. We included studies that did not specifically recruit participants from this age range if the mean or median age of participants was between 5 and 12 years, if more than 75% of participants were aged 5–12 years, or if most of the study’s recruitment age range overlapped 5–12 years. We excluded studies that included only children with a known developmental disability or a condition that substantially altered iron metabolism, including severe anemia. We included trials involving participants from all countries and socioeconomic backgrounds." Low et al. 2013, p. E792
- 41
See this section of our intervention report on iron supplementation.
- 42
Some of the factors that could influence iron absorption that we're particularly unsure about include (but are not limited to):
- Baseline levels of iron in the blood.
- Consistency of consumption of iron-fortified foods.
- For fortified foods, which foods are fortified (e.g., maize versus wheat flour).
- The extent to which certain substances inhibit the absorption of iron (e.g., tannins and phytates) or enhance the absorption of iron (e.g., ascorbic acid).
Some of our questions about the effects of iron absorption on health outcomes that we're particularly uncertain about include (but are not limited to):
- Conditional on the amount of iron absorbed, does iron have a greater effect on health outcomes for a lower initial iron status?
- How confident should we be in estimates for cognitive effects of iron supplementation?
- How serious are the effects of anemia on welfare?
- 43
- See this spreadsheet for more details.
- Reduction in iron deficiency: 79% (Low et al. 2013) x 90% (adjustment for publication bias) x 80% (adjustment for lower supplementation frequency) = 56.8%
- 44
- “Deworm the World started working in India in 2009.” GiveWell, "Evidence Action's Deworm the World Initiative: supplementary information"
- "We believe the evidence from Deworm the World's monitoring makes a relatively strong case that the programs it has supported have successfully dewormed children. Here we focus on Deworm the World's monitoring from Kenya, India, Nigeria, and Pakistan because those are its largest programs.55 Deworm the World's track record in Kenya and India is strong." GiveWell, "Evidence Action's Deworm the World Initiative – August 2022 version"
- 45
- State program managers, state program coordinators, and regional coordinators are key positions in the field and will be shared with Evidence Action's Deworm the World Initiative. Regional coordinators will split their time 70/30 between IFA and deworming. Evidence Action, Conversation with GiveWell, April 4, 2022 (unpublished)
- Evidence Action told us this integration is due to the deworming program relying on the same stakeholders as the IFA program. For example, the same state nodal officer may be responsible for IFA as well as deworming; Evidence Action will thus have one staff person as the point of contact for that relationship. Evidence Action, Conversation with GiveWell, April 4, 2022 (unpublished)
- Evidence Action, IFA Staff List (unpublished)
- 46
We discussed Evidence Action's TA in a few unpublished conversations with government officials.
- 47
- See figure, “Barriers to Regular Provision of IFA Supplementation in Schools,” Evidence Action, IFA investigation plan, 2022 (unpublished)
- Evidence Action, IFA investigation plan, 2022 (unpublished)
- 48
- "School awareness can be addressed through school teacher training and program orientation . . . We are addressing this bottleneck by supporting state governments with training and orienting district and block officials, teachers, and other frontline workers (ASHAs and anganwadi workers) on the IFA program[.]" Evidence Action, IFA investigation plan, 2022 (unpublished)
- "We’re addressing this gap we’re observing in supply availability at schools and AWCs by providing upstream and downstream supply chain support, from quantification and procurement support to last mile delivery." Evidence Action, IFA investigation plan, 2022 (unpublished)
- 49
- "What's the likelihood that the government or other philanthropic funders might support the program?
The Government of India is already supporting the program. Based on discussions with partners and donors, the appetite for funding [similar, statewide] IFA supplementation programs for children and adolescents isn't high."
Evidence Action, IFA investigation plan, 2022 (unpublished) - We asked local stakeholders about whether there are other organizations doing work that is comparable to Evidence Action. They told us that UNICEF provides some assistance but on a much smaller scale than Evidence Action and there are no obvious replacements for Evidence Action's TA. When asked about the existence of other funders for the IFA program, one stakeholder mentioned a collaboration with a medical college, while two other stakeholders indicated they could not think of any funding partners aside from the government and Evidence Action. Indian government official, conversation with GiveWell, April 14, 2022 (unpublished); Indian government official, conversation with GiveWell, April 15, 2022 (unpublished); Indian government official, conversation with GiveWell, April 18, 2022 (unpublished)
- "What's the likelihood that the government or other philanthropic funders might support the program?
- 50
See calculations here.
- 51
IHME made a significant methodological change between its global burden of disease (GBD) estimates for 2017 and its estimates for 2019 that caused iron deficiency prevalence estimates to significantly increase. We do not yet fully understand the reasons for this increase, but we are using GBD 2019 data, as 2019 is the most recent year for which data is available. Please note we are not able to easily link to past versions of GBD data to provide for comparison.
- 52
When we triangulate IHME data against NFHS-5 data for anemia among children under five (a measure collected in both), we find NFHS-5 shows higher rates of anemia and an increase in anemia over time (see our discussion here), which suggests we could be underestimating.
- 53
- Because the pre-post coverage estimates will be in the states that Evidence Action works in (TA states), and not other, non-TA states, we will not have measurement of Evidence Action's marginal impact through comparison of TA and non-TA states. This approach provides less insight on attribution of any increases in IFA coverage than methodologies that would compare the two.
- A shortcoming of using pre-post surveys to assess coverage is that coverage rates vary from year to year within and across states. Evidence Action and IDinsight, conversation with GiveWell, January 28, 2022, (unpublished)
- 54
- "Based on an 18-month timeframe between the baseline and endline, we propose to incorporate non-layered/layered CV surveys according to the timelines below. It would be useful for us to incorporate an interim layered CV in 2023 to get a read on progress and make necessary course corrections, even if this isn’t required for GiveWell purposes.
- Nov - Dec 2022: new 'baseline' non-layered CV, assuming 3 months have passed since site-based delivery has resumed and 'steady state' has been achieved
- Aug - Sep 2023: Layered CV
- Jul - Aug 2024: 'endline' non-layered CV, timed 18 months after the baseline. It can’t be conducted in May/June due to the summer vacation period.
- Aug - Sep 2025: Layered CV" Evidence Action, Email to GiveWell, June 1, 2022 (unpublished)
- At the time of the grant investigation, the 2022 coverage survey was referred to as the “baseline” survey because it was meant to capture data at the start of this grant. Since then, Evidence Action has referred to the survey as the “2022 coverage survey” so as not to be confused with the 2019 baseline coverage survey that was captured at the start of Evidence Action's TA in most states.
- "Based on an 18-month timeframe between the baseline and endline, we propose to incorporate non-layered/layered CV surveys according to the timelines below. It would be useful for us to incorporate an interim layered CV in 2023 to get a read on progress and make necessary course corrections, even if this isn’t required for GiveWell purposes.
- 55
"Data collected: based on the ToC document, slide 14, we would guess you plan to collect the data listed below. Could you confirm this is right?
- % of all targeted school children (government and private, pending pilots) who have consumed an IFA tablet each week in the last 4 weeks (reported separately for pink and blue tablets, and school type), unadjusted and adjusted for average attendance
- % of all targeted OOS girls ages 10-19 years who have consumed an IFA tablet each week in the last 4 weeks
- % of all targeted children ages 6--59 months who have consumed 1ml of IFA syrup twice each week in the last 4 weeks
Yes, we are planning to collect the data required to inform all the indicators mentioned through our coverage validation (CV) surveys." Evidence Action, IFA investigation plan, 2022 (unpublished)