Forecasting Research Institute — Soliciting Expert Forecasts on the Mortality Effect of Water Chlorination (April 2025)

Note: This page summarizes the rationale behind a GiveWell grant to the Forecasting Research Institute. Forecasting Research Institute staff reviewed this page prior to publication.

Published: May 2025

Background

Water quality interventions are one focus area of GiveWell grantmaking.1 We believe that the evidence is moderately strong that water quality interventions such as chlorination reduce all-cause mortality in children under-five, but we have high uncertainty about the size of the effect.2

We’ve identified a number of RCTs that have studied the impact of water chlorination on child health, but none were individually powered to detect effects on all-cause mortality.3 A 2022 meta-analysis by Kremer et al. found that water quality interventions were associated with a statistically significant 28%-30% reduction in under-five mortality. However, this estimate is highly sensitive to methodological choices, including which studies are included, how they are weighted, and how bundled interventions (e.g., chlorination combined with safe storage or handwashing education) are treated.

To address these concerns, GiveWell conducted our own meta-analysis focusing on five trials of chlorination-only interventions that are most relevant to the programs we evaluate. This analysis suggests a 14% reduction in all-cause under-five mortality, though the estimate is imprecise and not statistically significant. Our best guess, after adjusting for limitations in internal and external validity, is that chlorination interventions reduce all-cause mortality in children under-five by approximately 6–11%, depending on context.4

To date, we haven’t put our estimate of the mortality effect of water chlorination under as much scrutiny as other interventions we fund, like long-lasting insecticide-treated nets to prevent malaria, or seasonal malaria chemoprevention programs. This grant provides a relatively low-lift way to get detailed feedback and scrutiny on our water chlorination child mortality effect estimate.

The organization

The Forecasting Research Institute (FRI) develops forecasting methods to improve decision-making. FRI's research focuses on creating forecasting questions, developing and testing methods, and using forecasting tools to aid organizational decision-making.5

Josh Rosenberg, CEO of FRI, previously worked at GiveWell and has familiarity with our research processes and evidence standards. This is our first grant to FRI.

What we think this grant will do

This grant will fund FRI to collect forecasts on a refined version of the following question:

We plan to work with FRI to nail down the exact framing of this question – such as whether we want to ask about the intent-to-treat (ITT) effect or the treatment-on-the-treated (ToT) effect.6

Provisionally, FRI aims to recruit around 95 participants across five groups, with the following targeted time commitments:7

• ~10 subject-matter experts in water quality (5-10 hours commitment each)
• ~15 cost-effectiveness modelers from peer organizations (up to 20 hours each)
• ~10 PhD students in economics/public health (25 hours each)
• ~10 superforecasters (20 hours each)
• ~50 screened public forecasters (5 hours each)

The details of these groupings and time commitments may change as FRI further develops the design for this research project.

Approximately $70k of this grant will be earmarked towards incentive payments to encourage these experts to engage. Approximately $130k will go towards FRI staff costs and overheads.
Project activities and timeline:

1. FRI will compile information packets containing relevant GiveWell materials, academic papers, and other inputs (1-2 months)8
2. Participants will provide initial forecasts, receive feedback, participate in discussions, and update their forecasts using a modified Delphi method (3 months)9
3. FRI will analyze results and write preliminary findings for GiveWell (1-2 months)10
4. FRI will finalize a public report after incorporating feedback (1-2 months)11

The expected output is a report on the forecasters’ quantitative forecasts with accompanying qualitative rationales explaining the underlying reasoning.

The case for the grant

• Our estimate of the under-five mortality effect of water chlorination has undergone relatively less scrutiny than our estimates for other core interventions, like long-lasting insecticide-treated nets to prevent malaria, or seasonal malaria chemoprevention programs.12 We think this is a relatively light-touch way to receive feedback from several groups.
• Forecasting requires experts to quantify their beliefs—something we've historically found challenging to do. We hope that collecting forecasts through a structured process will help us better understand the extent and nature of disagreement among experts.
• FRI also plans to elicit qualitative rationale from experts to explain their forecasts. We hope this might surface specific areas of disagreement, or considerations we may have overlooked in our current analysis.
• We think there’s a chance we will fund a trial in the future that will try to measure the effect of water chlorination on all-cause under-five mortality. If we do this, comparing forecasted results vs. actual results may provide useful information about which experts were most accurate in their predictions.
• We’re broadly interested in experimenting with new ways of getting external feedback on our work. If this is a useful exercise, we may consider doing more forecasting projects on other grantmaking areas, with or without FRI’s assistance.
• Josh Rosenberg, CEO of FRI, is a previous GiveWell employee. We think his familiarity with GiveWell’s work and process will reduce the amount of time GiveWell staff will need to spend coordinating with FRI on the project.

Risks and reservations

• Some of the most relevant subject matter experts may not have the time or willingness to engage with this project, even with the incentive payments. We see forecasts from subject matter experts as the most important group from which we’d like to get forecasts.
• If the forecasters’ underlying reasoning isn’t explained clearly, we may not be able to interpret and learn from the forecasts. We have to understand where differing forecasts disagree with our current analysis and why, so that we can assess the merits of the disagreement and incorporate it thoughtfully. This could especially be a problem if subject-matter experts aren’t able to spend more than a couple of hours engaging on this project.
• If forecasters aren’t working off a shared understanding of the question, the resulting forecasts are difficult to compare and resolve differences between. We think this is a deceptively difficult problem to avoid. We believe that providing all forecasters with the same information packet will help ensure a shared understanding of the forecasting question.

Plans for follow-up

We’ll work with FRI to refine the question and the information packet at the start of the project. We may provide feedback for the forecasters after the initial round of forecasts. If the resulting report is useful we would consider making similar grants in the future.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time	Resolution
80%	FRI is able to recruit at least 5 subject matter experts to provide forecasts.	Sep 2025	-
60%	FRI is able to recruit at least 10 subject matter experts to provide forecasts.	Sep 2025	-
30%	We decide to do another forecasting project with FRI.	Dec 2026	-
40%	Pooled expert forecasts on the treatment effect are at least 20% higher or lower than our best-guess.	Mar 2026	-
20%	Pooled expert forecasts on the treatment effect are at least 50% higher or lower than our best-guess.	Mar 2026	-

Our process

We recommended this grant after conducting a shallow investigation. We reached out to FRI to see if they would be interested in a project like this because we think they’re well-situated to run the project. They agreed to share a proposal with us.
We spoke to FRI and a PhD candidate at the University of California, Berkeley who has conducted research on forecasting about the grant.

Relationship disclosures

Josh Rosenberg, CEO of FRI, previously worked at GiveWell.

Sources

Document	Source
Forecasting Research Institute (FRI), Homepage	Source (archive)
Forecasting Research Institute (FRI), Proposal to GiveWell (2025)	Source
GiveWell, All grants dashboard	Source
GiveWell, Internal forecasts	Source
GiveWell, Mass Distribution of Insecticide-Treated Nets (ITNs)	Source
GiveWell, Research Strategy: Water	Source
GiveWell, Seasonal Malaria Chemoprevention	Source
GiveWell, Water Quality Interventions	Source
Karger et al., 2021	Source
Kremer et al., 2022	Source
Wikipedia, Delphi method	Source (archive)

• 1

  See this blog post for more on our water grantmaking portfolio.

• 2

  To read more about our assessment of water quality interventions, see our intervention report here.

• 3

  See this section of our report on water quality interventions for more on the body of evidence supporting these interventions.

• 4

  See this section of our report on water quality interventions for more on how we arrived at this estimate.

• 5

  From the Forecasting Research Institute, “About us” page, “We develop forecasting methods to improve decision-making on high-stakes issues. First-generation forecasting research—spearheaded by FRI Chief Scientist Philip Tetlock and coauthors—focused on establishing a rigorous standard for prediction accuracy. The next generation of work aims to channel this approach into real-world relevance. We work with policymakers and nonprofit organizations to design practical forecasting tools and test them in large experiments.
  Our research centers on: (1) Producing high-quality forecasting questions about complex, long-run topics; (2) Novel methods for resolving unresolvable questions; (3) Testing the robustness of forecasting techniques across different domains and contexts; and (4) Using forecasting tools to help organizations make better decisions.”

• 6

  The intent-to-treat (ITT) effect refers to the average effect for everyone assigned to treatment, regardless of whether they received it. The treatment-on-the-treated (ToT) effect refers to the effect for those who actually received the treatment.

• 7

  See the “Structure” section of the Forecasting Research Institute proposal for GiveWell for more on the proposed structure of the forecasting panel.

• 8

  From the Forecasting Research Institute proposal for GiveWell, “Writing forecasting questions (including simplified spreadsheet models) and preparing ‘information packets’ for participants.

  • Duration: ~1-2 months
  • We will develop the headline forecasting questions in consultation with GiveWell…
  • We will also compile a brief overview of the relevant GiveWell materials, evidence bases, key questions, etc. that we ask all participants to read.
  • We would ask GiveWell for feedback on all materials but our team would do as much of the work as possible.”

• 9

  From the Forecasting Research Institute proposal for GiveWell, “Recruiting participants, collecting forecasts and rationales, conducting interviews with select participants, sharing responses (both from GiveWell and other respondents) with all participants, and asking all participants for final forecasts.

  • Duration: ~3 months
  • After recruiting participants as outlined above, we would provide them with the relevant information packets and ask them to provide forecasts.
    • We would incentivize forecasts on any unresolvable questions using methods like reciprocal scoring. We can provide more details on this if useful.
  • We’d then follow something like the Delphi method: providing the participants with summaries of others’ forecasts, facilitating discussion (via online forums and some facilitated video calls), and asking them if they want to update. We could potentially do multiple rounds of this.
    • As part of this stage, we could also provide them with feedback on their forecasts from GiveWell or our team. E.g. if we’re seeing that their forecasts didn’t take a key consideration into account or are very different on one dimension that we don’t fully understand, we could probe this.
  • We’d also identify forecasts and rationales that are particularly high-quality and conduct interviews via Zoom call with the participants who provided those forecasts. This would help us to develop even more detailed rationales from particularly strong participants.”

• 10

  From the Forecasting Research Institute proposal for GiveWell, “Writing up our findings, sharing with GiveWell

  • Duration: ~1-2 months”

• 11

  From the Forecasting Research Institute proposal for GiveWell, “Iterating on our report, gathering external feedback, and publishing

  • Duration: ~1-2 months”

• 12

  We’ve also directly less funding to water chlorination interventions. While we’ve directed ~$375m and ~$500m to the mass distribution of long-lasting insecticide-treated nets (LLINs) and seasonal malaria chemoprevention respectively, we’ve directed ~$120m to water chlorination grants.