Trying to Reproduce the RCTs Behind Our Top Charities

Published: May 2025

Background

Since 2014, we have recommended more than $1.2 billion2 to our four current Top Charities:

• Insecticide-treated bed nets (ITNs) by Against Malaria Foundation
• Seasonal malaria chemoprevention (SMC) by Malaria Consortium
• Vitamin A supplementation (VAS) by Helen Keller International
• Conditional cash incentives for childhood immunization by New Incentives

Our assessment of each program's effectiveness is supported by multiple RCTs, some of which have been summarized in meta-analyses. Key meta-analyses supporting our assessments of out Top Charities include:3

• Pryce et al. 2018: Insecticide-treated nets for preventing malaria
• Meremikwu et al. 2012: Intermittent preventive treatment for malaria in children living in areas with seasonal transmission
• Imdad et al. 2022: Vitamin A Supplementation for preventing morbidity and mortality in children from six months to five years of age
• Neelson et al. 2021: Financial Incentives to Increase Utilization of Reproductive, Maternal, and Child Health Services in Low- and Middle-Income Countries

Previous replication and reanalyses efforts by GiveWell and Open Philanthropy (a close partner of GiveWell’s) have influenced funding decisions. For example:

• GiveWell funded a reanalysis of a malnutrition treatment meta-analysis, which showed that we were overestimating counterfactual mortality risk.4
• Open Philanthropy funded reanalyses of one study on the long-term income effects of childhood immunization in India and another on effects of internet access in Africa. Both reanalyses found that treatment effects reported in the initial studies didn't withstand scrutiny.5

More broadly, we see value in promoting data sharing norms within academic research. In 2024, we funded a reanalysis of a cash transfer study in Kenya, which was made possible by the fact that the data and code were publicly available online. Wider adoption of open data practices would make it easier for GiveWell and others to verify important research findings.

Given this, we partnered with the Institute for Replication (I4R) to assess whether we could reproduce a portion of the evidence behind our current top programs.

Process

I4R did the following:

• Compiled a list of all 182 papers included in the four meta-analyses we asked them to consider.
• Located email addresses for authors of 170 papers (93% of total).6 The 12 papers they were not able to find email addresses for were mostly very old studies (e.g. VAS trials conducted in the 1980s).
• Sent emails to 115 authors requesting data and code sharing.7
• Followed up one month later with non-responders.

Out of 115 authors contacted, I4R received responses from 10 (9% response rate).8 These 10 authors were involved in 41 of the 182 papers.

Email response rates overall9

Contact attempt	Responded	Opened but no response	Not opened
Initial email (115 sent)	4 (3%)	33 (29%)	78 (68%)
Follow-up email (110 sent)	6 (5%)	34 (31%)	70 (64%)
Overall	10 (9%)	-	-

This included 7 authors of VAS studies, 3 authors of cash incentives for vaccination studies, and 0 responses for LLINs and SMC studies.

• For VAS, 4 people responded to say that reproducing the results would not be possible because the trials had been conducted a long time ago (20-30 years), with two of these mentioning that the data and code had been lost. An additional 2 people said that the data might still exist, but it would take work accessing it as it hadn’t been uploaded to a repository. 1 person said that the data and code should still exist and be accessible, but we would need permission from the Guinean National Ethics Committee to access it.
• For cash incentives for vaccination, 1 person said that the data was already accessible online (which we confirmed) but not the code, but said they were happy to share this with us. 1 person said the data was in the process of being uploaded to the public World Bank micro-data library, and didn’t get back to us about the code. 1 person replied that the data was not publicly available but might be accessible, but we would need permission from the Population Council to access it.

Among the limited responses received, authors described several barriers to replication:

• One VAS researcher reported that data from the 1980s was stored on physical cards at different locations and some was possibly eaten by rodents.
• Another noted their data might still exist but would require "considerable time digging it up" and was stored in "antiquated software no longer in use."
• A respondent mentioned that journals only required data retention for 20 years after publication

We were surprised by the low response rates even for more recent studies from the past 10-20 years, suggesting that data sharing challenges extend beyond just older research.

Email responses by Top Charity program

Program	Papers with email addresses	Responses received
LLINs	23/24 (96%)	0
SMC	7/7 (100%)	0
Cash incentives for vaccination	56/58 (97%)	3
VAS	84/93 (90%)	7
Total	170/182 (93%)	10 (9% of contacts)

Takeaways and next steps

We think the following reasons plausibly played a role in why we received such a limited response:

• Light-touch email outreach may not have been sufficient to get authors' attention.
• Many emails weren't opened, possibly being filtered as spam.
• We didn't offer any incentives for participation.
• Some researchers may not see value in replication efforts.
• Older studies have higher risk of data loss.
  • We think it’s unlikely we’ll be able to replicate the trials behind the VAS evidence base, given the dates that initial trials were conducted (20+ years ago for most)10 and the indication that the original data wasn’t being maintained in accessible repositories. While we didn’t get replies from anyone involved in the LLIN RCTs, we expect this is probably the case for these trials too, given the majority of studies included in the relevant meta-analysis were published in the 1990s and early 2000s.11
• Journal data retention policies may not support reproducing older trials.

Going forward, we're considering more intensive approaches to reproducing key results:

• Using warm introductions rather than cold emails.
• Offering funding for research assistant time to help locate and prepare data.
• Exploring ways to encourage routine data sharing in research.

Because we think newer evidence bases (including SMC and cash incentives for vaccination) may be more accessible, we may prioritize those.12

• 1

  This funding came from GiveWell’s operations budget and was not structured as a grant.

• 2

  See our list of all grants made to current Top Charities here.

• 3

  These four meta-analyses do not fully account for all studies underlying our assessments of our Top Charities.

• 4

  This reanalysis is published here. A discussion of how this reanalysis affected our views on malnutrition treatment can be found here.

• 5
  • The reanalysis of the long-run income effects of childhood immunization in India is published here.
    • “In sum, a causal interpretation of the SNB estimates as long-term effects of vaccination on income and spending deserves almost no weight. The results are best seen as artifacts of inflation during the survey, as well perhaps as other survey sequencing effects.” Roodman 2024 (a), p. 6.
  • The reanalysis of the employment effects of improved internet access in Africa is published here.
    • “Hjort and Poulsen (2019) frames the staggered arrival of submarine Internet cables on the shores of Africa circa 2010 as a difference-in-differences natural experiment. The paper finds positive impacts of broadband on individual- and firm-level employment and nighttime light emissions. These results largely are not robust to alternative geocoding of survey locations, to correcting for a satellite changeover at end-2009, and to revisiting a definition of the treated zone that has no clear technological basis, is narrower than the spatial resolution of nearly all the data sources, and is empirically suboptimal as a representation of the geography of broadband.” Roodman 2024 (b), Abstract.

• 6

  In most cases, this email was provided in the publication itself. Otherwise, I4R conducted a brief search to try and find the email address online. If there was no listed corresponding or lead author (only a handful of cases), I4R tried to find the email address of one randomly selected co-author.

• 7
  • Some authors were named on more than one of the 170 papers.
  • A copy of the email they sent is here.

• 8

  I4R also received some automatic replies to their emails. These were not counted as responses.

• 9

  We are not 100% sure about the not opened column, as some of the emails bounced and record keeping of this wasn’t well-kept.

• 10

  Of 47 studies included in the Imdad et al. 2022 analysis, all but 10 were published prior to 2005. See Figure 2 on pgs. 15-16.

• 11

  Pryce et al. 2018, figure 2, pgs. 14-15.

• 12
  • The studies contained in Meremikwu et al. 2012, which supports our analysis of SMC, were published between 2006 and 2011. See Figure 1 on page 10.
  • The programs reviewed in Neelson et al. 2021, which supports our analysis of cash incentives for childhood immunization, had a “median first implementation year of 2011” (pg 9). Start years for all included programs can be seen in Table 1 on pages 17-18.