What We’ve Learned from Looking Back on Our Technical Assistance Grantmaking (September 2025)

In a nutshell

Since 2019, GiveWell has recommended ~$120 million in technical assistance (TA) grants to support nine programs in 14 countries. These are grants that involve funding organizations to work with governments in low and middle-income countries (LMICs) to improve the performance of health programs. This write-up summarizes what we learned from looking back on our grants to five of these programs (totaling ~$45m).

Overall, we think these grants broadly achieved their stated objectives and we'd make them again. Coverage of targeted health programs increased after we made these grants, and government partners spoke positively of our grantees' work. We're highly uncertain about this takeaway, though, since it's hard to know what would’ve happened without our grants.

To some extent, this was not surprising. Technical assistance involves complex, multi-year efforts to influence government systems and counterfactuals are hard to establish, making it difficult to evaluate these grants after the fact. With this said, evaluation was even harder than we expected, because of disagreements between different data sources and unexpected shocks (e.g., COVID-19) that meant some grant activities had to adapt to new developments.

We also identified several ways we can better investigate and evaluate these grants. Going forward, we plan to talk to more in-country stakeholders, find ways to independently verify grantee data, and consider broader system impacts that our cost-effectiveness models don't capture.

Published: January 2026

Summary

What did we do

Our TA grants involve funding organizations to work with governments in LMICs to increase coverage of health programs that we think are highly cost-effective.

In 2025 we looked back on our TA grants to five programs:

  • Evidence Action’s Deworm the World (DtW) Initiative in India
  • Evidence Action’s support for the government’s Iron and Folic Acid Supplementation program in India
  • Evidence Action’s work to integrate HIV/syphilis dual tests into the health system in Liberia
  • PATH’s work to accelerate the rollout of the RTS,S malaria vaccine in Ghana, Kenya, and Malawi
  • Results for Development’s (R4D’s) work to increase the supply of amoxicillin-DT in government health facilities to treat pneumonia in Tanzania

    • We picked these programs because they were the first five TA programs we’d supported, and we felt enough time had elapsed since making the grant to evaluate them.

    How successful were these grants in retrospect?

    We think these grants were plausibly in range of the cost-effectiveness estimates we initially modeled and we would probably make them again.1 However, we have a low degree of confidence in this, as we’re very uncertain how much these grants led to counterfactual increases in coverage of programs like deworming, iron and folic acid supplementation (IFA), syphilis treatment, and malaria vaccination.2

    To assess these grants, we relied on coverage data, grantee activities, and qualitative evidence from government partners and other key stakeholders. We also visited three of these programs in person as part of this lookback. Each of these sources has substantial limitations:

    • Coverage estimates. In general, the data we’ve seen suggests coverage of the targeted health programs increased after we made these grants, roughly in-line with our initial predictions. For example, according to data Evidence Action sent us, the percentage of women who were tested for syphilis at antenatal clinics in Liberia increased from 11% in the year before we made the grant to 88% three years after we made the grant.3 However, we have several reservations about this data:
      • First, there were discrepancies between other grantee-reported data and government data in two of the three cases where we had both external coverage surveys and government coverage data.4 For example, Evidence Action’s coverage data suggests the percentage of primary school children consuming IFA supplements in schools increased by ~34 percentage points (pp) between the year we made the grant and two years after we made the grant. Government coverage data across the same time period suggests a smaller 17pp increase.5
      • Second, except potentially in the case of our amoxicillin-DT grant, we did not have reliable estimates of what coverage would have looked like in the absence of our grant (i.e. counterfactual coverage). For example, the percentage of children receiving albendazole6 on National Deworming Day7 looks to have increased by ~10-20pp between the year before we made the grant and three years after Evidence Action’s engagement, but we don’t have a good sense of how much coverage would have increased in the absence of their support. (more)
    • Grantee activities. Grantees generally completed the activities they set out to do, though there were some deviations from plans. For two programs (R4D’s pneumonia treatment program and Evidence Action’s syphilis testing program), we made unanticipated $1-1.2 million grants due to the programs needing more funding than we initially forecasted, and for one program (RTS,S), we agreed to a six-month no-cost extension due to delays in implementation. Some of these delays and top-ups were due to hard-to-predict disruption caused by COVID-19, but others were caused by things we should have anticipated. For example, RTS,S rollout was delayed in Kenya because of a presidential election, while we made a top-up grant for dual tests in Liberia partly to align the handover period with Global Fund replenishment cycles. (more)
    • Qualitative input from key stakeholders. For the three programs we visited (Deworm the World, iron and folic acid supplementation, and R4D’s pneumonia treatment program), government stakeholders all told us they valued the TA and gave specific examples of how our grantees work was helpful. For example, in Tanzania, an official in the Ministry of Health told us that R4D’s work in improving their demand forecasting model for amoxicillin-DT had also helped them to improve their demand forecasting for other commodities. However, we're cautious about this feedback since officials may feel pressure to report positively, especially when our grantees were present for discussions. (more)

    We did some rough estimates of retrospective cost-effectiveness, given what we found on program coverage and costs. These suggest each of these grants were around ~10x our cost-effectiveness bar, though we don’t put a lot of stock in these estimates as we haven’t revisited every assumption made in our initial cost-effectiveness model.8 (more)

    What are takeaways for our grantmaking going forward?

    This evaluation has underscored our impression that TA grants are inherently less measurable than our “direct delivery” grants, such as funding insecticide-treated net distributions or vitamin A supplementation campaigns. A few things make investigation and evaluation of these grants more challenging:

    • While our impact estimates for direct delivery programs are typically based on randomized controlled trials, these usually aren’t feasible for TA grants, as identifying a suitable control group is difficult.9
    • Our direct delivery grants generally involve more clear and predictable theories of change – e.g., we know roughly the steps that need to happen to deliver bed nets to people, and can check retrospectively whether these happened. In our experience, it’s harder to map out what needs to happen for TA grants to be successful in advance, as this is more likely affected by context and shifting government priorities.
    • We find it easier to interrogate bottlenecks to coverage for direct delivery grants. In most cases, we think the key constraint is funding – not everyone gets nets because malaria control programs are funding constrained, and so by funding net campaigns, we can ensure nets get to people who would not otherwise have received them. Funding constraints seem less relevant for the TA grants we’ve investigated, and while we think there are other plausible bottlenecks to coverage, we’ve found these harder to interrogate from afar.

    Despite this higher degree of uncertainty, we still plan to investigate TA grants, because we continue to think there are highly cost-effective opportunities in this space. Government health programs can work at enormous scale – even boosting coverage up by a few percentage points can lead to millions of people getting access to programs that we think are highly cost-effective. If we avoided investigating these opportunities in favor of only investigating what was more measurable, we think we’d probably be ignoring impactful opportunities.

    While we’re willing to accept a greater degree of uncertainty for these grants, we also think there are things we should change about our approach to maximize our chances of picking good grants in this space, and learning from them. Things we consider most important:

    • Talk to more in-country stakeholders. Before these lookbacks, we had only visited one program (Evidence Action’s DtW in 2013). For three grants (R4D’s pneumonia program, Evidence Action’s syphilis testing program, and PATH’s malaria vaccine program), we relied solely on our grantees to communicate government priorities and bottlenecks instead of speaking to ministries directly. We think we should be more proactive in investigating these, especially now that GiveWell is a larger grantmaking organization with greater research capacity. In the future, we plan to do more site visits and hold more conversations with individuals in the Ministries of Health in countries where we support these programs to improve our understanding of the implementation context. (more)
    • Look for more opportunities to verify grantee data. Given the discrepancies we found between grantee and government coverage data, we’d like to do more to build independent checks into our monitoring and evaluation (M&E) process. One thing we’ll consider is funding our own independent coverage or monitoring surveys, as opposed to outsourcing this to our grantees. We’ve already done this for one of our more recent TA grants (e.g., in-line chlorination in India). (more)
    • Plan for programs evolving differently than we modeled. Generally speaking, these grants evolved in more unpredictable directions than our direct delivery grants. For example, in Liberia, the World Food Program unexpectedly took over the management of the national medical stores and distribution of all commodities, including of the benzathine penicillin G (BPG) and dual tests, after we made the grant.10 In Tanzania, we think R4D’s work may have improved demand forecasting for other health commodities like antiretrovirals and bed nets, which weren’t targeted by our grants. We think this unpredictability should affect how we investigate these opportunities. For example, we should spend more time investigating our grantees’ ability to provide flexible support, such as looking at their previous track record in-country, and less time building detailed cost-effectiveness models, which can realistically only capture a handful of pathways to impact. (more)
    • Design simpler CEAs that better align with M&E data we’re collecting. In some cases, our main outcome measures (e.g., the percentage of children receiving iron and folic acid supplements) didn’t map to clear parameters in our cost-effectiveness model, which made it hard to assess how we ought to update our beliefs in light of our M&E. Going forward, we’ll make sure that the key performance indicators we’re collecting directly map to our cost-effectiveness model. We’ve built templates to make this easier. (more)
    • Be more specific about what activities we expect to happen and when. For example, our grant page for dual tests says we expected Evidence Action to “work with the National AIDS Control Program to train providers to use the dual rapid test,” without specifying how many providers we expected to be trained and when. Lack of specifics makes it harder to assess how far program activities deviated from our expectations. In evaluating these grants, we don’t plan to place too much weight on what percentage of milestones the grantee met, as we recognize that grants can evolve in positive directions that we did not initially anticipate. Nonetheless, we think it’s still helpful to track our expectations about milestones, and plan to do this more consistently (more)
    • Align grant activities around predictable bottlenecks. For example, RTS,S vaccine rollout was delayed in Kenya due to a presidential election, and we had to provide additional funding for our grant to support dual tests in Liberia to ensure the period when the program was handed over to the government aligned with external funding cycles. We should have anticipated both of these ahead of time and planned accordingly. (more)
    • Build more contingency into budgets and timelines. Multiple programs needed unexpected additional funding. Evidence Action's Liberia program needed an additional $1.2m due to unaccounted for inflation, COVID delays, and an extended handover period. R4D's Tanzania program needed an unplanned $1m follow-up grant due to COVID disruption. PATH's RTS,S program required a 3-6 month no-cost extension. We plan to build more contingency into initial budgets and timelines for these projects. (more)
    • Prioritize baseline data collection. For two of these programs (DtW and R4D’s pneumonia treatment program), we don’t have a reliable baseline measure of coverage before TA activities started. Though we think our decision not to collect baseline data was reasonable at the time, it makes it hard to know how coverage changed after we made the grant. We plan to prioritize baseline data collection in future (more)

    What are the limitations of these findings?

    We’ve only looked back on five TA programs and think these lessons are generally limited by this small sample. More specific limitations include:

    • We only spent one to two weeks looking back on each program, and it’s possible we’re missing out on some wins, failures, or other details as a result. We chose to prioritize breadth rather than depth to try and synthesize cross-cutting takeaways.
    • We visited three of these programs in-person and spoke with government stakeholders, and we spoke with government officials in Liberia via video call, but we did not speak with officials for the RTS,S grant in Ghana, Kenya and Malawi).
    • For four out of five of these programs (all except PATH RTS,S), grant activities are still ongoing, and it’s possible there’ll be more decision-relevant updates in the future as grant activities wrap up. We’re especially interested in what happens to coverage once TA partners exit, and may report on this in future.
    • In the retrospective cost-effectiveness models we built, we only updated cost data (if we made an unexpected top-up grant) and counterfactual coverage estimates, in light of the M&E data we’d collected. We didn’t do a full review of other parameters of these models. It’s possible that had we done this, our retrospective cost-effectiveness estimates for these programs would have changed.

    What are technical assistance grants?

    Our technical assistance (TA) grants involve funding organizations to work with governments in low and middle-income countries (LMICs) to increase coverage of health programs we think are highly cost-effective.11 Depending on the grant, the types of activities supported can range from helping the government forecast the demand for programs/commodities, monitoring stockouts in the supply chain, and training frontline service workers on how to administer the program and report basic performance indicators.12

    Since 2019, we estimate that we’ve recommended around $120m in grants towards nine TA programs, around 6% of our overall grants made between 2019 and 2024.13 We thought grants in this space could be highly cost-effective because of the scale of government programming. Intuitively, if we could nudge coverage up by a few percentage points, or accelerate the scale-up of a program by a couple of years, this could lead to millions of people getting counterfactually treated with health programs that we think are highly cost-effective, like deworming tablets or iron and folic acid supplementation.

    Table 1 summarizes the TA grants we’ve made to date. This write-up focuses on the five programs highlighted in yellow. For the others, we think it’s generally too early to assess whether these grants have been successful.14 We may look back on these at a later date.

    Table 1: GiveWell TA grants 2013-2024

    Program Partner country(s) Size of grant(s) Timeline of grant(s) Current timeline of TA engagement Aim of grants (in a nutshell)
    Evidence Action DtW India ~$10m15 2013-2023 2013-202516 Increase the coverage of school-based deworming programs
    Evidence Action IFA India ~$13m 2019-2022 2022-2025 Increase the coverage of school and community-based iron and folic acid supplementation programs
    R4D pneumonia treatment Tanzania ~$13m 2016-2022 2016-2024 Increase the supply of amoxicillin-DT in Tanzanian health facilities
    PATH RTS,S Ghana, Kenya, Malawi ~$5m 2022 2022-2024 Accelerate the rollout of the RTS,S malaria vaccine
    Evidence Action HIV/syphilis dual tests Liberia ~$5m 2020-2024 2020-2026 Accelerate the rollout of HIV/syphilis dual tests in the health system and increase syphilis treatment coverage
    Evidence Action HIV/syphilis dual tests Zambia and Cameroon ~$15m 2022 2022-2028 Accelerate the rollout of HIV/syphilis dual tests in the health system and increase syphilis treatment coverage
    PATH perennial malaria chemoprevention DRC ~$6m 2022 2023-2025 Pilot perennial malaria chemoprevention to test operational feasibility in the Democratic Republic of the Congo
    Evidence Action in-line chlorination India ~$38m 2023 2023-2028 Pilot in-line chlorination to test operational feasibility in two Indian states
    PATH RTS,S Burkina Faso, DRC, Mozambique, Nigeria, Uganda ~$15m 2024 2024-2027 Improve the quality of the rollout of the RTS,S malaria vaccine
    Total ~$120m

    Notes: grant amounts not adjusted for inflation. The latest grant is linked to in the table.

    How successful were these grants in retrospect?

    Coverage estimates

    All of these grants sought to increase coverage of a targeted health program, so program coverage was the key metric we used to assess program impact. Generally, data sent by our grantees suggests coverage has increased after we made these grants, and in some cases, faster than we predicted. For example, two years after we made a grant to PATH, 77% of targeted children had received two doses of the RTS,S malaria vaccine – 11 percentage points higher than what we had initially predicted.17

    Table 2: Summary of coverage data

    Grant Key performance indicator Baseline data? Credible counter factual? Baseline After making our grant
    Year 1 Year 2 Year 3 Year 4
    Evidence Action DtW % of 5-19 year olds dewormed on National Deworming Day N N What we predicted Not possible to tell from the model18
    Grantee M&E data19 . 70% 68% 65% 76%
    Evidence Action IFA % of 5-19 year olds consuming 4 or more tablets (full IFA consumption) in the last 4 weeks Y N What we predicted Not possible to tell from the model
    Grantee M&E data20 28% 78%21 61%
    R4D amoxicillin-DT % of government health facilities with amoxicillin-DT in stock N Maybe What we predicted 42% 62% 64% 65% 66%
    Grantee M&E data22 . 53% . . 74%
    PATH RTS,S % of eligible children vaccinated with RTS,S (3 doses) Y N What we predicted 0% 53% 66% 73% 80%
    Grantee M&E data23 0% 69% 77%
    Evidence Action dual tests % of women screened for syphilis Y N What we predicted 12% 41% 71% 71% 71%
    Grantee M&E data24 11% 53% 79% 92%

    Notes: yellow cells = data not yet collected

    However, we think there are major limitations about how much we can infer from this data about the counterfactual impact of our grants:

    Poor triangulation with alternative data sources

    For three of the five programs (DtW, IFA, and amoxicillin-DT), we have alternative coverage estimates produced by the government. In two of three cases (DtW and IFA), we found discrepancies between these sources.25 From a shallow review, we didn’t find a convincing explanation of what was driving this difference, which makes it hard to know how much weight to put on competing sources.

    Evidence Action IFA

    As part of this grant, Evidence Action commissioned independent coverage surveys. Evidence Action put these surveys out to competitive tender,26 and then hired an external data collection firm27 to collect data on what percentage of children were receiving IFA supplements in Indian schools. To do this, the data collection firm would visit a randomly selected sample of schools, look through their IFA registers to see what percentage of children had received IFA supplements in the last four weeks, and then interview the headteacher and a randomly selected sample of children to cross-check the registers and test basic knowledge of the program.28 Evidence Action would randomly spotcheck ~5% of schools themselves. This data was collected in 2022 – before Evidence Action’s TA work started – and again in 2023 and 2024, 1 and 2 years after the project began. The sampling strategy in 2023 was different to 2022, as Evidence Action oversampled certain districts to help inform program design. However, the sampling strategy in 2022 and 2024 was the same, so we think these represent the most reliable points of comparison.

    Across the same time period, the government of India has also published their own estimates of IFA coverage in schools.29 These statistics similarly rely on IFA registers in schools as ‘ground truth’ data: summary data are collected by block education officials and then handed over to block health officials,30 where they are then entered into the digital HMIS system.31 Data is aggregated at the district and then state level. As far as we know, there are no top-down adjustments that affect this aggregation process.32

    Across the five states where Evidence Action was working, these data tell different stories. If we average across states and ages (weighted by population), the data Evidence Action sent us implies that the percentage of targeted children receiving ‘full’ IFA supplementation increased from 28% before their TA began to 61% two years later. By contrast, government data implies full IFA coverage went from 57% to 74% over this same period.

    Chart description
    Source: GiveWell’s analysis of monitoring and evaluation data from our TA grants (unpublished)

    Notes: government data was pulled from the Anemia Mukt Bharat dashboard on 19th August 2025. In our experience, historic data on this platform sometimes changes, so this chart may not be replicable using data pulled at a later date. We do not have permission to publish state-specific grantee data.

    To dig into potential reasons for this discrepancy, we spoke to several people familiar with the government estimates, including people involved with IFA programming in India and several government officials. All of these people generally expressed negative views on the quality of this data. We also noticed that some of the 2022-2023 data changed when we pulled it in 2024 vs. 2025, which seems like an additional issue of data quality.33

    Evidence Action Deworm the World

    For these grants, we also have coverage estimates produced by the government and ones sent to us by Evidence Action. Evidence Action’s process for estimating deworming coverage is similar to IFA: they contract with an independent data collection company who visits a randomly selected sample of schools a few days after National Deworming Day (NDD). Once there, the enumerators look through deworming registers to see: i) how many children were enrolled/eligible (the denominator) and ii) how many of these children were marked as having received deworming tablets (the numerator). They then cross-validate these against interviews with three randomly selected students, the headteacher, and the nodal deworming teacher.34

    The government’s process is also similar: school deworming registers are the same primary input, which get collected by block education officers after NDD, handed over to the block health office, and then digitized into their HMIS system.35

    The data we’ve seen generally shows poor triangulation36 – coverage estimates produced by Evidence Action rarely line-up with what the government estimates. There doesn’t seem to be a clear directional bias – e.g. one source reporting consistently coverage than the other – but the lack of agreement generally makes us wary of putting too much stock in any one data point. Unfortunately, we don’t have permission to publish this data.

    Missing counterfactuals

    Table 2 suggests coverage of each program increased after we made the grant. However, it’s hard to infer from this that our grant caused an increase in coverage, because we don’t observe what would have happened otherwise.

    Identifying credible ways to account for the counterfactual is difficult for TA grants. First, most of our TA grants (e.g. RTS,S, dual tests, amoxicillin-DT) have sought to increase coverage nationally, which makes it hard to identify a group of people who were not affected by the intervention in some way (i.e., a control group). Second, even when TA is targeted at the subnational level (e.g. DtW, IFA), randomizing delivery usually isn’t feasible, as governments usually want assistance targeted at regions most in need.

    For the amoxicillin-DT grant, we tried to approximate the counterfactual by also collecting data on the availability of comparable medicines that were not targeted by R4D’s TA. In 2016, a supply contract with UNICEF was coming to an end in Tanzania, which meant the Ministry of Health was facing a funding cliff for amoxicillin-DT (an antibiotic used to treat lower respiratory tract infections) and other health commodities.37 GiveWell made grants to R4D to: i) initially fund procurement of amoxicillin-DT to plug an immediate commodity gap; ii) provide TA to the Ministry of Health to gradually transition the procurement of amoxicillin-DT to domestically mobilized resources.38

    To assess whether this increased coverage, we earmarked funding for R4D to collect data on the availability of amoxicillin-DT and other health commodities in Tanzanian health facilities over time. R4D have told us that the zinc and oral rehydration salts co-packs share some relevant characteristics to serve as a comparator. Like amoxicillin-DT, external funding for zinc/ORS also dried up with the expiry of the UNICEF contract. It is also a tablet-based product with a soluble accompaniment so we thought this might proxy what would have happened to amoxicillin-DT had we not made a grant.39

    R4D outsourced this data collection to an independent survey company, who visited a randomly selected sample of Tanzanian health facilities across 11 rounds to assess whether these products were in stock.40 The data suggests that amoxicillin-DT availability increased over time while zinc/ORS co-pack availability decreased over time, which might suggest our grant was causally responsible for driving the increase in coverage.

    Chart description
    Source: GiveWell’s analysis of monitoring and evaluation data from our TA grants (unpublished)

    However, even with this data, it’s hard to make strong inferences about counterfactual impact. One thing that undermines zinc-ORS co-pack’s role as a counterfactual is that it’s more easily substitutable with other products than amoxicillin-DT. The WHO only recommends amoxicillin-DT in tablet form; they recommend zinc-ORS in both tablet (i.e. co-pack) and sachet form. When the UNICEF contract expired for co-packs, the Tanzanian government may have been less incentivized to increase the supply as they would have been for amoxicillin-DT, as they could substitute towards the sachets. If so, using the supply of zinc-ORS co-pack to proxy the counterfactual supply of amoxicillin-DT would overstate the impact of our grant.

    Lack of baseline data

    For two of these programs (DtW and amoxicillin-DT), we lack a reliable measure of baseline coverage, which makes it hard to know how much coverage increased after we made our grant.

    In the case of amoxicillin-DT, we chose not to fund a baseline coverage survey for ethical reasons – we’d heard that supplies of amoxicillin-DT were critically low, and it didn’t feel right to delay plugging this commodity gap so we could establish a baseline coverage measure.41

    In the case of DtW, our decision not to fund baseline data collection was a function of GiveWell’s priorities at the time. We first supported Evidence Action’s DtW program in 2013, when GiveWell was only moving ~$10m per year and not expected to scale significantly beyond this. We now move ~$300m per year and expect this to remain relatively stable,42 which has shifted the calculus of how much it makes sense to invest in data collection to inform better grantmaking in future. Were we to make these grants today, we’d probably have earmarked more funding for baseline surveys, and we expect to do so for similar grants in future. (more)

    Lack of clear predictions

    For two of these programs (DtW and IFA), we didn’t make clear predictions in our cost-effectiveness models about how we expected coverage to change after making the grant. For DtW, we didn’t build a cost-effectiveness model of Evidence Action’s TA work in India.43 For IFA, our cost-effectiveness model didn’t contain predictions about how IFA coverage would change after we made the grant. Instead, we modeled the number of children counterfactually reached as target population and the counterfactual increase in coverage (which we can’t observe).44

    This makes it hard to know whether the coverage data we collected exceeded or missed our initial expectations. We think this was a mistake, in hindsight, and plan to always make predictions about how we expect coverage to change in our TA cost-effectiveness models in future. (more)

    Grantee activities

    Ideally, for each grant we would have a list of milestones we’d agreed upon with the grantee, which could look back on to see how many were completed and at what time. We have this in one case: for our RTS,S grant, we pre-specified project milestones, though these weren’t published on our grant page. Looking back on these, all of the agreed upon milestones were met, though there were slight (two month) delays in Kenya due to delays caused by a presidential election in August 2022.45

    Table 4: RTS,S grant milestones, predicted vs. actuals

    Milestone Predicted Actual
    All three countries make formal decisions to expand the vaccine into MVIP comparator areas. PATH provided financial and technical support for the meetings, facilitating the review of next steps and processes for expansion and the review of evidence from the pilots May 2022 April 2022
    PATH begins providing funding for the development, review, and finalization of expansion plans June 2022 Completed in Ghana and Malawi in May 2022, delayed in Kenya due to presidential election
    WHO and PATH developed a supply and shipment plan for donation doses with GSK and UNICEF June 2022 July 2022
    National and district-level health worker training completed December 2022 November 2022 (Malawi); February 2023 (Ghana and Kenya)
    Launch of vaccine implementation in the comparison areas of all three countries January 2023 November 2022 (Malawi); February 2023 (Ghana); March 2023 (Kenya)

    For other grants, we weren’t specific about what activities we expected to be completed and when, which makes it hard to quantitatively assess how much our expectations for these grants differed from reality. We plan to do this more consistently in future. (more)

    While we can’t ground this in numbers, our qualitative sense is that our grantees generally completed the activities they set out to do. For example, Evidence Action has trained representatives from all health facilities in Liberia (that were designated by the Ministry of Health as certified sites for testing pregnant women for HIV) on how to administer syphilis/HIV dual tests, as well as the appropriate treatment protocols if these tests come back positive.46 In Tanzania, R4D helped the Ministry of Health transition to a new demand forecasting model for amoxicillin-DT, which combined ‘bottom-up’ supply requests from health facilities with ‘top-down’ inputs such as population estimates and pneumonia prevalence rates.47 In India, Evidence Action developed standard operating protocols for IFA supplementation and NDD, and also trained nodal teachers on how to administer these programs, monitor adverse events, and report M&E data correctly.48

    However, compared to our direct delivery grants (e.g., funding insecticide-treated net campaigns), these grants have also moved in more surprising directions to what we initially anticipated. For example:

    • Dual tests:49 One of the core aims of this grant was to get an indicator for whether a patient had been treated for syphilis embedded into official facility registers.50 This didn’t happen as planned in 2020, which meant we had to shift our plans for monitoring how the number of women treated for syphilis (the key outcome of interest for us) changed over time. In 2023, the World Food Program (WFP) also took over the management of some key aspects of the national supply chain, including BPG (the main treatment for syphilis) in Liberia, becoming the ones responsible for managing the national medical store and arranging the logistics of transporting the drug to health clinics. We’re not sure whether this is a positive or negative development for our grant. On the one hand, Evidence Action have told us that BPG stockouts have been less frequent since WFP took over, which may mean more women got successfully treated for syphilis after testing positive. On the other hand, the government having less control over the supply chain seems like a negative update on their ability to maintain high treatment coverage in the absence of TA.
    • Amoxicillin-DT:51 The plan for this grant was to gradually taper off support for direct amoxicillin-DT procurement so that it could be sustainably financed by the government. This had to be rethought in 2020 as the COVID-19 pandemic meant the government faced severe funding constraints, so we pivoted the grant back towards direct procurement. This grant has since gone back towards tapering off direct funding support, and as of 2024, all amoxicillin-DT in Tanzania was being procured via domestically-mobilized resources.52 When we visited this program in September 2024, one surprising thing we heard from a government official was that R4D’s support in demand forecasting had also been helpful for commodities beyond amoxicillin-DT.53 In 2019, the government of Tanzania announced that all health commodities had to be forecasted via bottom-up demand quantification systems,54 including traditionally ‘top down’/vertical commodities like bed nets and antiretrovirals. The official said that R4D had helped the Ministry in making this broader transition, which is not something we had expected at the time we made the grant.
    • RTS,S: While the rollout of the RTS,S malaria vaccine in Malawi happened slightly ahead of schedule, training of community health workers was delayed by a cholera outbreak after cyclone Freddy hit in February 2023.55 This shifted the priorities of the Ministry of Health, and meant PATH had to be flexible in reorganizing the RTS,S training schedule

    Unexpected updates have pointed in both directions – both positive and negative – and we don’t have a strong sense as to which direction dominates. Generally, though, us being less able to reliably predict what will happen once our funds are dispersed feels like a negative update on our ability to pick high impact grants in this space.

    Qualitative input from key stakeholders

    We visited three of these programs (DtW, IFA, amoxicillin-DT) in September 2024 to speak with government stakeholders who had worked closely with our grantees. For DtW and IFA, we spoke with government officials at varying levels of seniority in Uttarakhand, a state where we fund both programs. We spoke with block health and education officers, district health officers, and the state official in charge of NDD and school-based IFA supplementation in Uttarakhand. Evidence Action staff were present for these meetings.56 In Tanzania, we spoke with the chief pharmacist responsible for monitoring the supply of amoxicillin-DT. R4D staff were not present for this meeting.

    Generally, each of these stakeholders spoke positively about our grantees’ work, and asked us to consider extending the engagement for the two programs where our grantees activities were winding down (DtW and amoxicillin-DT). We asked them to give examples of specific bottlenecks to coverage they thought our grantees' activities had helped resolve. Some examples they gave:

    • Monitoring procurement contracts: In India, Evidence Action back-calculates when IFA procurement contracts are about to expire and flags to the government when tenders need to be put out to ensure there’s no gap in supply. The state official we spoke to mentioned this as helpful to make sure things didn’t fall through the cracks, and that there’d likely have been more gaps in coverage without their support
    • General gap filling: The state official we spoke to in India told us she had 20 health programs in her portfolio, including IFA and deworming. She said that having a trusted partner in Evidence Action was a great help, as she could rely on them to help arrange the logistics of National Deworming Day and troubleshoot supply chain issues before they escalated to her. This gave her more time to focus on other programs in her portfolio
    • Wider systems support: The chief pharmacist we spoke to in Tanzania said R4D had been helpful in supporting the Ministry transition to bottom-up demand forecasting for all health commodities. R4D helped build the Excel-based demand forecasting models (based on the one they had built for amoxicillin-DT) and hosted workshops to train government staff on how to use them

    We view these positive stakeholder reviews as a positive update on these grants. However, we’re cautious about putting too much weight on this feedback, since officials may feel pressure to report positively, especially when our grantees were present for discussions.

    Stepping back, we’ve also found it harder to interrogate the bottlenecks to increased coverage for TA grants. For our direct delivery grants, bottlenecks to coverage seem more straightforward – some people don’t get bednets or seasonal malaria chemoprevention because National Malaria Control Programs are funding constrained, and so by funding the procurement and distribution of these commodities, our grantees can get these to people who would have not otherwise received them.

    Our TA grants, on the other hand, are usually targeted at programs where funding doesn’t seem like the key constraint. For example, in India, the national government sets aside funding for school-based deworming and IFA, but many states do not fully tap into the funds available.57 This means we need different reasons for why more children aren’t covered by these programs. From speaking with our grantees, some explanations we’ve heard include:

    • Some programs can slip through government cracks. For example, both deworming and IFA entail administering health products via schools, so awkwardly straddle both the Ministry of Health and Ministry of Education. In Liberia, responsibility for screening expectant mothers for syphilis sits between the National AIDS and STI Control Program (NACP) and the Family Health Unit. When programs don’t fit obviously into existing silos, accountability and reporting can get muddied, and it can take an external push (e.g. a TA engagement) to jumpstart programming and institutionalize protocols
    • Some programs require technical expertise. For example, our grant to PATH sought to accelerate the rollout of the RTS,S malaria vaccine. While funding for the vaccines was already guaranteed by Gavi, introducing a new vaccine into a health system is a far from trivial task – health workers need to get trained, supply contracts need to be agreed, and monitoring chains need to be established. In a context like Malawi, where capacity in the Ministry of Health is often stretched thin by unexpected shocks like cholera outbreaks, having external experts to hand can be the difference between programs getting introduced sooner rather than later

    These bottlenecks seem plausible to us, but they also seem harder to interrogate than a more straightforward supply side intervention, where the government lacks funding to administer a program. We think these grants demand a greater understanding of the local policy making context, which makes them harder to investigate and evaluate from afar. We don’t think this should stop us making grants in this space; but we do think it should affect what we prioritize when we investigate them (more).

    Backwards-looking cost-effectiveness

    We built retrospective cost-effective models of grants we made to each of these programs. These are intended to provide a rough sense of whether these grants look cost-effective, given our current assessment of the effect of the grant on coverage and program costs.

    We account for this in the following ways:

    • Effect of program on coverage: We put some weight on data reported by grantees and their independent monitors and some weight on data reported by the government.
    • Counterfactual coverage: We make rough forecasts about how coverage of these programs would have changed in the absence of TA. In each case, we assume coverage would have increased without TA. To estimate counterfactual impact, we take the “wedge” between: (i) our estimated effect of the program on coverage and (ii) how we expect coverage would have changed otherwise. You can see our assumptions about coverage and counterfactual coverage here.
    • Program costs: We incorporate actual spending on the grant, including cases where we made top-up grants.

    We haven’t revisited the ‘downstream’ assumptions in our initial cost-effectiveness models, such as the effect of childhood deworming on adult income, the anemia burden in India, or the effect of amoxicillin-DT on child mortality. We chose not to do this because it would have been time-consuming, and, e.g., revisiting the deworming treatment effect is a lower priority for us at the moment compared to other research questions. While this means the estimates below should be taken with a pinch of salt, we still think it’s informative to show how the coverage and cost data we’ve looked at updates our cost-effectiveness assessment, holding fixed the downstream assumptions.

    Cost-effectiveness we modeled at the time Retrospective cost-effectiveness estimates58
    25th percentile Best-guess 75th percentile
    Evidence Action DtW N/A59 2x 8x 16x
    Evidence Action IFA60 12x61 3x 8x 15x
    R4D amoxicillin-DT 5x62 2x 9x 13x
    Evidence Action dual tests 14x63 3x 12x 18x
    PATH RTS,S 12x64 12x 14x 16x65

    Note: Our retrospective cost-effectiveness models can be found here.

    What are takeaways for our grantmaking going forward?

    We’ve identified several things we think we should be doing more of in future. Some of these involve rectifying mistakes we think we made in investigating these grants at the time. Others involve things we’re able to do more of now. GiveWell is a very different organization now compared to when we made some of these grants. For example, when we first made a grant to R4D in 2016, GiveWell was directing $100 million a year and had 7 full-time members of the research team.66 In 2025, we expect to direct roughly $350m in funding67 and have a research team of almost 50 full-time staff. We think this should affect both what we’re able to accomplish, and what we ought to prioritize.

    Talk to more in-country stakeholders

    When we revisited our investigations of these grants, we found that we had done limited site visits and discussions with in-country stakeholders (e.g., at the Ministry of Health in countries where TA was going to be done).

    • Before these lookbacks, we had only done a site visit to one of these programs (Evidence Action DtW, which we visited back in 2013).
    • For all of these grant investigations, we didn’t speak to many stakeholders beyond our grantee, and for three (amoxicillin-DT, dual tests, and RTS,S) we didn’t speak to anyone in the government Ministry our grantee was proposing to partner with.

    While we recognized the importance of implementation context and government buy-in, we generally relied on our grantees to communicate this to us. While we had a smaller research team at the time of these grants, we still think not directly investigating this ourselves was a mistake, given how important understanding the policy making context seems in interrogating whether TA grants are likely to be successful.

    Going forward, we plan to collect more input from in-country stakeholders through more site visits and expert interviews.

    • Site visits. We’re unsure about the optimal number of site visits we should make and guess it’s not necessary for all grants. However, we think on the margin, we should do more site visits, especially in cases where we’ve made or are considering substantial grants.68 This would help us to understand coverage bottlenecks better and catch potential issues early. For example, in Liberia, one reason a syphilis treatment indicator didn’t get integrated into official registers back in 2020 was because syphilis treatment usually takes place in a different location at the facility than HIV & syphilis testing, and so the National AIDS control program was concerned that if they included the treatment indicator in their testing register, it would often be left blank. This concern could have surfaced if we’d visited Liberian health facilities beforehand.
    • Expert interviews. We also think more discussion with experts would help us to surface issues we wouldn’t have found otherwise. For example, for the amoxicillin-DT grant, we should have spoken to the government, and maybe also sought out experts that had worked on health TA programs in Tanzania or supply chain programs in other countries, to see whether the bottlenecks R4D was proposing to resolve resonated. In our grantmaking, we’ve already begun engaging more regularly with in-country stakeholders, and we plan to continue this. We also think strengthening our network of experts could help in retrospective evaluation of grants by identifying “neutral spectators” to discuss whether a grant was successful with (people that have an understanding of the activities our grantees are supporting on the ground, but don’t have clear vested interests in saying how successful they’ve been).

    Look for more opportunities to verify grantee data

    There were surprisingly large discrepancies between grantee-reported data and government reported data. For example:

    • IFA. Surveys commissioned by Evidence Action showed a large (34pp) coverage improvement while government data showed a more muted (17pp) improvement.69
    • DtW. Surveys commissioned by Evidence Action showed a large (~60pp) reduction in helminth infections whereas surveys commissioned by the government showed much more muted declines.70

    We didn’t commission our own independent surveys for any of these grants. For four of these programs, we earmarked funds for independent M&E, but left it to our grantees to contract with independent data collection agencies, design the sampling strategy, and backcheck the data.

    In future, we plan to do more to try to verify grantee data:

    • Independent surveys. In cases where we’ve made or are considering large grants, we should be more involved in the monitoring process. One way would be to contract with independent data collection firms ourselves. We’ve already done this in the case of our large in-line chlorination TA grant in India, where we’ve contracted with a firm separately to conduct tests of chlorine levels in household water supply.
    • Triangulating downstream measures. In cases where coverage estimates disagree, we might resolve some of this agreement by looking at downstream measures of impact. For example, India has a diet and biomarkers survey planned for 2026, where they plan to assess anemia rates among children. If there are no notable reductions in childhood anemia in the states where we’ve funded TA, that seems like a vote against sizable increases in IFA coverage.

    We don’t expect independent surveys to ‘solve’ the issue of conflicting data points, but we think it would mean we’re better positioned to interrogate them.

    Plan for programs evolving differently than modeled

    Our TA grants have typically evolved in more unpredictable directions than our direct delivery grants. For example:

    • Dual tests. The WFP unexpectedly took over the management of the supply chain of BPG in Liberia,71 which meant less opportunities for Evidence Action to strengthen government capacity in this area.
    • Amoxicillin-DT. R4D has been supporting commodity demand forecasting more generally, which isn’t something we anticipated at the time of making the grant.

    We think this ought to change what we spend time on when investigating these grants:

    • Less emphasis on detailed cost-effectiveness models. Our cost-effectiveness models chain together a series of assumptions that map our funding decisions (e.g. grant amount) to downstream impact (e.g. number of lives saved). More detail can be helpful when we can break down our theory of change into a series of linear and predictable steps.72 For example, to distribute bed nets to people, we have a reasonable sense of what steps need to happen along the way (e.g., people need to receive the nets; people need to use them), and can make assumptions about these based on data we collect (e.g., percentage of nets that get delivered; percentage of households that have nets hanging up when visited).73

      For TA grants, it seems harder to do this. We don’t think this means we should stop building cost-effectiveness models for these grants, but, we think it means we should put less onus on mapping out the ‘right’ pathway to impact. An example of this is our ILC grant, where we built a very detailed model focused on a handful of very specific scenarios playing out.74 In hindsight, we should have spent less time on this.

    • More emphasis on qualitative factors like grantee track record. We think we should put less weight on a handful of modeled scenarios and more weight on factors that we’d guess robustly predict impact under a wide-range of scenarios. One thing that has struck us in these lookbacks is the need for TA programs to be flexible, given the difficulties of predicting what needs to happen in advance. We’d guess a strong predictor of grantees’ ability to provide this support is the strength of their relationship with the government. Rather than detailed cost-effectiveness modeling, we think we’re better off gauging this by spending more time speaking with government stakeholders and assessing our grantee’s track record in-country, or advising on similar programs elsewhere.

    Design simple CEAs that better align with M&E data

    In two cases (DtW and IFA), the key outcome measure we were monitoring (coverage) didn’t correspond to parameters in our cost effectiveness model. For example:

    • IFA. In our cost-effectiveness model, we modeled the number of children counterfactually reached as target population and counterfactual increase in coverage (which we can’t observe). There were no predictions about how coverage would change after we made the grant.

    This matters because it makes it difficult to know how we ought to update our beliefs about cost-effectiveness in light of the M&E data we’re collecting. For example, in 2023, surveys outsourced by Evidence Action suggested 74% of 5-9 year olds received full IFA in the last 4 weeks. We have no way of clearly knowing whether this is higher or lower than what we initially expected as we don’t have a parameter that captures this in our CEA.

    We plan to fix this going forward by always making predictions in our CEAs about how we expect coverage to change if we make the grant. This is embedded in new templates we’ve built to model TA grants.

    Be more specific about what activities we expect to happen and when

    In all but one case (RTS,S), we didn’t set specific milestones about what activities we thought would be completed and when. Our description of the activities we thought the grant would support was also quite vague. For example:

    • Dual tests. In our grant page, we said we thought this grant would help Evidence Action “Support Liberia in managing and monitoring the supply chain for both dual rapid tests and penicillin”. We weren’t specific about exactly what activities this would entail (e.g., training frontline health workers on data entry or building dashboards for Ministry staff) and when we expected them to be completed.

    We plan to fix this going forward, by being more specific about milestones we expect to happen and making forecasts over them in our grant page. This is embedded in new M&E guidelines.75 A (hypothetical) example is below:

    Confidence Milestone By time
    90% A procurement contract will have been signed with a manufacturer September 2025
    75% All school nodal officers will have been trained March 2026
    80% All flyers/radio announcements/other dissemination activities will have been completed May 2026
    60% At least 100 coverage surveys will have been done August 2026

    Align grant timelines around predictable bottlenecks

    We identified cases where grant activities were delayed by bottlenecks that were predictable. For example:

    • RTS,S: the rollout of the RTS,S malaria vaccine in Kenya was delayed by a presidential election in June 2023, which we should have anticipated and worked around in advance.
    • Dual tests: originally, Evidence Action’s TA engagement was due to end one year before the Global Fund replenishment cycle. We thought this might threaten program sustainability, as getting dual tests into Liberia’s Global Fund application was a key part of how we thought the supply of dual tests would remain high after Evidence Action exited.

    In hindsight, we should have anticipated the need to align the project timeline and the Global Fund replenishment cycle in advance.

    Build more contingency into budgets and timelines

    Multiple programs needed unexpected top-up funding or no-cost extensions. For example:

    • Dual tests: Evidence Action's Liberia program needed $1.2m additional due to inflation, COVID delays, and the need for an extended handover period.
    • Amoxicillin-DT: R4D's Tanzania program needed an unplanned $1m follow-up grant due to COVID disruption, which meant some of the money earmarked for TA had to be diverted towards direct amoxicillin-DT procurement.
    • RTS,S: PATH's RTS,S program required a 3-6 month no-cost extension. PATH requested this due to delays caused by the election in Kenya, and because they thought this could ensure a smoother transition to GAVI-supported routine immunization.

    We plan to build more contingency into budgets and timelines in future.

    Prioritize baseline data collection

    For two of these programs (DtW and amoxicillin-DT), we lack a reliable measure of baseline coverage, which makes it hard to know how much coverage increased after we made our grant. For amoxicillin-DT, we decided not to collect baseline data as we didn’t want to delay programming (for ethical reasons). For DtW, we didn’t collect baseline data because we had much less staff capacity (<10 research staff) at the time of making these grants, and invested more towards investigating new grants vs. evaluating existing ones. Now that we have a larger research team, we’re prioritizing more evaluation of the programs we support.

    Sources

    Document Source
    Anemia Mukt Bharat, Homepage Source (archive)
    GiveWell, All Content on Evidence Action's Deworm the World Initiative Source
    GiveWell, Evidence Action — Iron and Folic Acid (IFA) Supplementation in India (August 2022) Source
    GiveWell, Evidence Action — Liberia Syphilis Screening and Treatment in Pregnancy Exit Grant (April 2024) Source
    GiveWell, Evidence Action — Scale-Up of In-Line Chlorination in India (September 2023) Source
    GiveWell, Evidence Action — Syphilis Screening and Treatment in Pregnancy Source
    GiveWell, Evidence Action — Syphilis Screening and Treatment in Pregnancy in Zambia and Cameroon (July 2022) Source
    GiveWell, Evidence Action IFA supplementation CEA [2022] Source
    GiveWell, Evidence Action's Deworm the World Initiative — India (January 2023) Source
    GiveWell, GiveWell's retrospective CEAs for technical assistance programs (September 2025) Source
    GiveWell, In-line Chlorination CEA (Evidence Action, India), 2023 Source
    GiveWell, Mass Distribution of Insecticide-Treated Nets (ITNs) Source
    GiveWell, PATH — Perennial Malaria Chemoprevention Pilot in the Democratic Republic of the Congo (November 2022) Source
    GiveWell, PATH — RTS,S Malaria Vaccines in Pilot Comparison Areas (January 2022) Source
    GiveWell, Results for Development — Childhood Pneumonia Treatment Program Phaseout (December 2022) Source
    GiveWell, Results for Development — Childhood Pneumonia Treatment Scale-Up Source
    GiveWell, Vitamin A Supplementation Source