Note: This page summarizes the rationale behind a GiveWell grant to the Development Innovation Lab at the University of Chicago (DIL) and Innovations for Poverty Action (IPA). DIL, IPA, and Evidence Action staff reviewed this page prior to publication.
In a nutshell
In March and April 2025, GiveWell recommended grants totalling ~$926,000 to the Development Innovation Lab and Innovations for Poverty Action. This funding is for household surveys to independently corroborate GiveWell’s estimates of the number of people reached via Evidence Action's dispensers for safe water and in-line chlorination programs in Uganda and Malawi (more).
We recommended this grant because (more):
- GiveWell sees Evidence Action’s water chlorination programs as among the most impactful global health programs that donors can support. However, both we and Evidence Action have open questions about the number of people who are successfully reached by these programs. This is mainly because of data we’ve seen from an independent study, the Kenya Study of Water Treatment and Child Survival, as well a follow-up survey in Kenya conducted by Evidence Action. Taken together, these surveys imply that Evidence Action’s internal monitoring data is currently overestimating the number of people receiving chlorinated water through its Kenya dispensers program. However, we’re unsure how far these issues apply to Evidence Action’s safe water programs in other locations.
- We expect that a high-quality, independent survey could help resolve our uncertainty and improve our analysis of the cost-effectiveness and overall promisingness of these programs. In particular, we plan to use the results to inform our decision making about two large potential grant opportunities for Evidence Action chlorination programs that we expect to consider late in 2025.
- We also expect the grant could inform potential changes to its monitoring methods that Evidence Action is actively exploring, as well as improving GiveWell’s understanding of the overall promisingness of chlorination programs in other contexts.
We have the following reservations and uncertainties about making this grant (more):
- It’s possible the survey might be uninformative because of methodological issues (e.g. because the 2 surveys taking place in each village will use different sampling approaches, and it may be challenging to isolate the same target population).
- Because of the time sensitivity of the project, GiveWell’s investigation for this grant was shorter than usual for a grant of this size and complexity. It’s possible that this increases the risk of errors or oversights.
Published: June 2025
1. What we think this grant will do
1.1 Overview and background
Evidence Action is an NGO that delivers low-cost health programs in Africa and Asia.1 Evidence Action currently operates two large-scale water chlorination programs: Dispensers for Safe Water (DSW) in Kenya, Malawi, and Uganda,2 and in-line chlorination in Malawi and India.3 Both dispensers and in-line chlorination are technologies for chlorinating water at communal water points in rural areas that aim to reduce the risk of waterborne disease.4
In 2022, GiveWell recommended a ~$65m grant to fund dispensers for safe water in Kenya at its previous size, and expand the program in Malawi and Uganda. Before making the grant, GiveWell reviewed Evidence Action’s monitoring protocol and Evidence Action incorporated suggestions based on this review.5
This grant will fund surveys to independently corroborate Evidence Action and GiveWell’s estimates of the number of people reached6 by the programs in Uganda and Malawi.7 The surveys will also inform potential changes to Evidence Action’s monitoring & evaluation methods. The surveys will be delivered by a partnership between the Development Innovation Lab at the University of Chicago (DIL) and Innovations for Poverty Action (IPA), in coordination with Evidence Action.
This grant is motivated by two new sources we’ve seen suggesting that fewer people use dispensers in Kenya than our and Evidence Action’s previous estimates. The first is an independent study part-funded by GiveWell, the Kenya Study of Water Treatment and Child Survival (KSWTCS), whose focus area partly overlaps with Evidence Action’s program in western Kenya. In March 2024, the KSWTCS research team and Evidence Action shared an analysis suggesting that, in a subset of overlapping villages between 2019 and 2021, chlorination rates were substantially lower than Evidence Action’s routine monitoring data had shown (see footnote for further details).8 However, the two datasets are difficult to compare like-for-like because KSWTCS’ study area and sample only partly overlapped with Evidence Action’s program footprint, the timing of data collection was different, COVID may have been a confounding factor, and there were methodological differences in the data collection. Therefore, it is not possible based on the KSWTCS analysis alone to validate the discrepancies or confirm their cause.
Because of the difficulty of directly comparing the two datasets, in 2024 Evidence Action decided, in consultation with GiveWell and DIL, to conduct a new survey in 69 DSW villages in Kenya to corroborate its monitoring estimates and understand the potential mechanisms behind the KSWTCS discrepancies. At the time of writing (May 2025), we have received initial results from Evidence Action’s survey, which confirmed that Evidence Action’s routine monitoring overestimated the number of people reached through its Kenya dispensers program.9 Evidence Action provisionally estimates that the survey implies 30-40% fewer people are reached by its Kenya dispensers program than its routine monitoring previously suggested,10 and is still in the process of investigating the reasons behind this overestimation.
We expect that the surveys funded by this grant will help shed further light on the discrepancies in Kenya, as well as helping us to understand how far they apply to Evidence Action’s chlorination programs in Malawi and Uganda (which use a similar monitoring approach).
1.2 Planned activities and budget
1.2.1 Objectives and activities
This grant has two main objectives:
- Primary objective: Generate an independent estimate of the number of people chlorinating household drinking water through Evidence Action’s dispensers and in-line chlorination in Uganda and Malawi.
- Secondary objective: Better understand any sources of error in Evidence Action’s routine monitoring, including whether an independent partner implementing Evidence Action’s normal monitoring processes gives substantially different results to the census-based survey. Evidence Action has asked us to explore this topic through these surveys.
To achieve these objectives, DIL and IPA will conduct two parallel research activities: (1) a census-based survey to achieve objective 1, and (2) a replication of Evidence Action’s routine monitoring processes in the same villages to achieve objective (2). See the table below for a full summary of research activities.11
| Objective/Research Question | Research activity | Notes |
|---|---|---|
| Objective 1: Independent estimate of the number of people chlorinating household drinking water through Evidence Action’s dispensers and in-line chlorination in Uganda and Malawi. | Census of water points in sampled villages to identify water points where dispensers/in-line chlorination is installed. | These activities will provide an independent estimate of the number of water points with functioning dispensers or in-line chlorination devices. |
| Survey of all water points identified in sampled villages to understand functionality of the water point and the functionality of the dispenser or in-line chlorination device. | ||
| Household census in sampled villages to identify all households in the target population. | These activities will provide an independent estimate of (i) the number of people using each water point with a functioning dispenser or in-line chlorination device and (ii) the share of those people with chlorinated water. | |
| Household survey to gather data on household size and composition, water points used, chlorination rates in tested water, and attitudes towards chlorination. | ||
| Objective 2: Better understanding of any sources of error in Evidence Action’s routine monitoring, including whether an independent partner implementing Evidence Action’s normal monitoring processes give substantially different results to the census-based survey. | Data collection conducted by IPA using an identical methodology to Evidence Action’s normal monitoring process (“adoption monitoring”).12
For dispenser monitoring, this process consists of the following key steps:
|
This would help us understand any sources of error, including if an independently delivered version of Evidence Action’s current monitoring approach matches the results from the census-based survey. |
See the footnote for a draft list of outcome measures we expect to be gathered through these surveys.13
1.2.2 Sample size and statistical power
For villages, the sample size will be 100 villages in Malawi and 60 villages in Uganda (the sample size in Malawi is larger because it includes approximately 40 villages where in-line chlorination is delivered, whereas the survey in Uganda will focus on dispenser villages only).14 For households, the sample size will be approximately ~5,100 households in total (~3,200 in Malawi and ~1,900 in Uganda).15
Our estimate of statistical precision is sensitive to a number of analytical choices where we have limited or conflicting information. However, using our preferred assumptions and based on calculations shared by DIL, we expect that the estimates based on these surveys will have the following margins of error around the mean chlorination rate:16
- Dispensers in Uganda: ~4 percentage points
- Dispensers in Malawi: ~4 percentage points
- In-line chlorination in Malawi: ~6 percentage points
1.2.3 Chlorine testing devices
All enumerators will test for the presence of chlorine in drinking household water using manual color wheel kits. These are the same devices Evidence Action currently uses for its routine monitoring. In addition, two enumerators per team (randomly selected each day) will test drinking water using digital colorimeters, an alternative (and possibly more reliable) testing tool. DIL expects that this will result in a representative sample of colorimeter-collected data to compare to the main dataset, informing Evidence Action’s decisions about which tool to use in the future (more below).
1.2.4 Budget
The total budget for this project is ~$926,000,17 of which ~$257,000 is allocated for DIL and ~$669,000 is allocated for IPA.18 DIL’s overall role in the project is as the lead for survey design, project management, data quality checks, and analysis/reporting. IPA will lead the data collection itself and manage the local ethical approval process.
2. The case for the grant
2.1 The survey could substantially improve our understanding of the reach of Evidence Action's water chlorination programs.
Because of the KSWTCS results and the findings from Evidence Action’s recent Kenya chlorination survey (described above), both GiveWell and Evidence Action currently have substantial uncertainty about the number of people who treat their drinking water through Evidence Action’s chlorination programs. We expect that a high-quality, independent survey might help resolve this uncertainty and improve GiveWell’s analysis of the cost-effectiveness and overall promisingness of these programs.
Between 2022 and the time of writing, GiveWell has made over $70m in grants to Evidence Action chlorination programs in Kenya, Uganda, and Malawi and in-line chlorination in Malawi.19 We therefore see the ~$926,000 cost of this survey as a good investment for improving our understanding about a key aspect of these programs.
We expect that the results from the survey will directly inform two grant opportunities that GiveWell is planning to consider in late 2025: (1) a possible renewal of GiveWell’s 2022 grant for dispensers in Uganda and Malawi, and (2) a possible expansion of dispensers and in-line chlorination in Nigeria and/or Sierra Leone.
2.2 The survey could help to shape future changes to Evidence Action’s monitoring processes for its chlorination programs.
Some potential learnings from the study include:
- Whether an independently implemented version of Evidence Action’s routine monitoring processes produces (see Objective 2) similar results to a “gold standard” census-based survey, which we’d expect to be more expensive (see Objective 1). This could inform decisions about whether to maintain Evidence Action’s normal monitoring methodology or switch to a different methodology, and whether it is important for the monitoring data to be independently collected. These are key questions Evidence Action has asked us to explore through these surveys.
- Whether data collected through manual chlorine color wheel kits is comparable to data collected through digital colorimeters. Evidence Action’s current monitoring process relies only on color wheel kits, which are substantially cheaper.20 While we have not researched this question in detail, our understanding is that colorimeters are considered the most reliable tool for on-the-ground survey data collection.21 Comparing the two could inform Evidence Action’s decisions about which method to use in the future.
2.3 The survey may inform GiveWell’s understanding about the promisingness of chlorination programs in other contexts.
Water chlorination is a relatively new area of grantmaking for GiveWell, and one that we are actively trying to expand in new locations.22 In December 2024, we launched a public request for information for proposals for chlorination programs, in particular dispensers and in-line chlorination. We expect that this survey will help improve our understanding of the promisingness and cost-effectiveness of this entire grantmaking portfolio.
This is a less important part of the reasoning for making the grant, because we expect that the generalizability of the survey will be somewhat limited beyond Evidence Action and the specific locations where it works in Uganda and Malawi.
3. Reservations and uncertainties
3.1 There is a risk that the surveys might be uninformative because of methodological issues.
The two types of surveys taking place in each village (census-based vs adoption monitoring, discussed above) will use different sampling approaches. The census-based survey will sample from a census of all households in each village, whereas the routine monitoring survey will sample from a list of water point users drawn up by the volunteer responsible for managing each water point. It’s possible that this difference in methodology could make comparisons between the surveys challenging. We expect this risk will be biggest for the in-line chlorination survey in Malawi. This is because some in-line chlorination devices serve multiple villages, and it would not be viable to conduct a household census in all of them. DIL, IPA, and Evidence Action are discussing how to resolve this issue as part of the survey design phase.
There is also a risk of “behavioral priming” (i.e., households surveyed twice might change their behavior as a result of the first survey, leading to bias in the second estimate). Both the census-based survey and the “adoption monitoring” survey will take place in the same villages; while this design will enable us to gather detailed information about possible mechanisms for bias, it is also possible that the surveys themselves could have an effect on chlorination behavior (e.g., if the presence of enumerators in the village acts as a reminder to households to chlorinate). At the time of writing, we are considering different options to mitigate this risk,23 but it’s possible these will be ineffective or impractical.
3.2 GiveWell’s investigation timeline for this project was relatively short.
Because of the time sensitivity of the project,24 our investigation was shorter than usual for a grant of this size and complexity (approximately 3-4 weeks). It’s possible that this increases the risk of errors or oversights.
4. Plans for follow up
During the project, we plan to discuss this grant regularly with DIL, IPA and Evidence Action (at a cadence to be determined) to confirm a number of unresolved survey design questions. We expect to receive results back from the surveys by August 2025.
5. Our process
We identified this opportunity through discussions with Evidence Action. As part of the grant investigation we conducted the following activities:
- We requested and reviewed proposals from 3 different potential survey partners.
- We engaged in multiple rounds of questions with Evidence Action and DIL.
- We had a number of calls with the survey partners.
- Evidence Action reviewed DIL’s final proposal and shared feedback.
- 2 GiveWell peer reviewers reviewed the grant and shared feedback.
6. Sources
- 1
“Evidence Action scales low-cost health interventions that improve the wellbeing of hundreds of millions of people in Africa and Asia.” Evidence Action, Home.
- 2
See the page for GiveWell’s 2022 grant for Dispensers for Safe Water here.
- 3
See the page for GiveWell’s 2022 grant for in-line chlorination in Malawi here and the 2023 grant for in-line chlorination in India here.
- 4
See Evidence Action’s overview of its dispensers program here and an overview of its in-line chlorination program in Malawi here.
- 5
See this section of GiveWell’s 2022 grant page for further details on the changes.
- 6
We use “reached” on this page as a shorthand for the number of people who drink chlorinated water as a result of Evidence Action’s programming. This factors in both the number of people using water points where dispensers or in-line chlorination are installed and the number of people whose water is effectively chlorinated as a result of using each device.
- 7
Note: these surveys will not collect data on Evidence Action’s chlorination programs in Kenya, where Evidence Action recently conducted its own survey using independent enumerators and corroborating its routine monitoring with a different method (more), or India, where its in-line chlorination program is only recently underway, and we expect that any data collected now would not be reflective of a mature program.
- 8
The KSWTCS data suggested much lower chlorination rates than expected in a study area and sample partly overlapping with Evidence Action’s program. KSWTCS researchers and Evidence Action conducted an analysis comparing their dataset with Evidence Action’s M&E from 2019 to 2021 and found that under plausible assumptions, the number of people drinking chlorinated water in the KSWTCS dataset was only 33% as high as Evidence Action's in villages where both datasets overlap. Because of the difficulty comparing the datasets like-for-like, there is substantial uncertainty around this estimate in both directions. Development Innovation Lab, “Comparison of chlorination rates from different data sources,” March 2024 (unpublished).
- 9
Evidence Action, “DSW Monitoring Review Results,” March 2025 (unpublished).
- 10
Evidence Action, conversation with GiveWell, May 6, 2025 (unpublished).
- 11
Development Innovation Lab, “Measuring chlorination in areas with active Evidence Action chlorine delivery interventions,” March 2025 (unpublished).
- 12
Evidence Action, “Dispensers for Safe Water: Chlorine Adoption and Functionality Monitoring Protocol,” 2024 (unpublished), and Evidence Action, “In-Line Chlorination: Chlorine Adoption and Functionality Monitoring Protocol,” 2024 (unpublished).
- 13
- Number of households in a village
- Percent of households that self-report using water points with chlorine dispensers
- Percent of households that self-report using water points with chlorine dispensers that test positive for chlorine
- Percent of households that self-report using water points with in-line chlorination
- Percent of households that self-report using water points with in-line chlorination that test positive for chlorine
- Functionality rate of chlorine dispensers
- Functionality rate of in-line chlorination devices
- Dosing of chlorine dispensers
- Dosing of in-line chlorination devices
- Number of households using water points with chlorine dispensers according to “promoters” (community volunteers responsible for managing water points)
- Percent of households using water points with chlorine dispensers according to promoters that test positive for chlorine
- Number of households using water points with in-line chlorination according to promoters
- Percent of households using water points with in-line chlorination according to promoters that test positive for chlorine
Development Innovation Lab, “Measuring chlorination in areas with active Evidence Action chlorine delivery interventions,” March 2025 (unpublished).
- 14
This is because GiveWell only funds the in-line chlorination program in Malawi.
- 15
These estimates assume 20 households will be surveyed in each village as part of the census-based survey (described above), and 12 households will be surveyed in each village as part of the “adoption monitoring” survey (described above
- 16
These estimates refer specifically to the chlorination rate among households using water sources with chlorine dispensers or in-line chlorination. Note that these estimates rely on the following assumptions from Development Innovation Lab/GiveWell, “Running Q&A document for chlorine surveys,” February-March 2025 (unpublished):
- The overall chlorination rate in villages with in-line chlorination is 50%, and the overall chlorination rate in villages with chlorine dispensers is 20%.
- The average share of the population in each village using water points with chlorine dispensers or in-line chlorination is 56%.
- The intra-village correlation in the chlorination rate is 0.1.
- The correlation between use of a water point with chlorine dispensers/in-line chlorination and detectable chlorine residual in household water is 0.25.
- The intra-village correlation in use of a water point with chlorine dispensers or in-line chlorination is 0.2.
- 17
Development Innovation Lab, “Measuring chlorination in areas with active Evidence Action chlorine delivery interventions,” March 2025 (unpublished).
- 18
Budget specifics:
- $669,039.12 for IPA. IPA, “Adoption of Evidence Action's DSW and ILC programs in Malawi and Uganda,” March 2025 (unpublished).
- Note the initial Development Innovation Lab budget at the time GiveWell made this grant was $233,534.56. GiveWell then subsequently signed off a top-up of $23,195.90 in April 2025 to pay for additional colorimeters to reduce the margin of error around the colorimeter data collection. $233,534.56 + $23,195.90 = $256,730.46.
- 19
This consists of:
- 20
As of the time of writing, the retail price for a Hach color wheel kit online is $90.75, compared to $690 for a Hach digital colorimeter.
- 21
See Murray et al. 2015 (abstract): “Seven commercially available colorimeters, color wheel and test tube comparator kits, pool test kits, and test strips were evaluated for use in low-resource settings by: (1) measuring in quintuplicate 11 samples from 0.0–4.0 mg/L free chlorine residual in laboratory and natural light settings to determine accuracy and precision; (2) conducting volunteer testing where participants used and evaluated each test kit; and (3) comparing costs. Laboratory accuracy ranged from 5.1–40.5% measurement error, with colorimeters the most accurate and test strip methods the least. Variation between laboratory and natural light readings occurred with one test strip method. Volunteer participants found test strip methods easiest and color wheel methods most difficult, and were most confident in the colorimeter and least confident in test strip methods. Costs range from 3.50–444 USD for 100 tests. Application of a decision matrix found colorimeters and test tube comparator kits were most appropriate for use in low-resource settings; it is recommended users apply the decision matrix themselves, as the appropriate kit might vary by context.” Note that we have not evaluated this paper in detail.
- 22
For more, see this page on our 2024 water team strategy.
- 23
For example, we could compare the adoption monitoring estimate from this survey to the data from Evidence Action’s routine monitoring. If the adoption monitoring estimate from this survey was notably higher, it could be suggestive evidence of behavioral priming.
- 24
We aim to have survey results back by August 2025 to meet our planned deadlines for the grant decisions discussed above.