Innovations for Poverty Action and the University of Michigan — Five-Year Follow-Up and Additional Survey for Raising the Village RCT (November 2024)

Note: This page summarizes the rationale behind a GiveWell grant to Innovations for Poverty Action and the University of Michigan. Staff from these organizations reviewed this page prior to publication.

Published: February 2025

Background

Raising the Village aims to increase the incomes of ultra-poor farming communities through a number of low-cost interventions, broadly encompassing agricultural productivity measures and income-generating community projects.1 Raising the Village’s current program provides all households within ultra-poor villages with six months of intensive livelihood support, including group training sessions, the provision of agricultural inputs, sanitation and health programs, and the establishment of savings groups to facilitate the creation of new businesses. The version of the program subject to this study also provides small livestock for some households.2 Following this six month period, Raising the Village conducts eighteen months of lighter-touch training and follow-up, after which villages are considered to have graduated from the program. To date, Raising the Village has reached around 1.5 million people in Uganda.3

Mahreen Mahmud and Emma Riley, at the University of Exeter and University of Michigan respectively, conducted a randomized controlled trial of Raising the Village’s program between 2020 and 2023.4 Innovations for Poverty Action provided data collection for the study.5

The results of this study are not yet public while it awaits peer review. However, we reviewed preliminary, non-peer-reviewed results that suggest Raising the Village’s program meaningfully increased consumption, which is the primary outcome that we use to evaluate ‘livelihoods’-related programs.6

However, we have three important uncertainties and reservations about the evidence currently available:

• Effect size: Since our current ‘best guess’ of the program’s cost-effectiveness is based on the results of a single, non-peer-reviewed study, further evidence could update our view of the magnitude of the program’s effect on consumption significantly.
• The duration of impact: Mahmud and Riley’s endline survey occurred around a year after the conclusion of the two-year program. However, a key driver of our estimate of the cost-effectiveness of Raising the Village’s program is the duration of the program’s effects, which a one-year follow-up is not able to capture.
• Seasonality: The study’s endline survey occurred during one of Uganda’s two annual harvest periods. A key question for us is whether the program’s effects can be sustained during the ‘lean season’ – that is, the period between harvests, when rural farmers’ income and consumption are lower. (If not, then the results of the initial endline survey would not be representative of the average impact of the program across the year.) Earlier in 2024, Mahmud and Riley conducted a follow-up survey by phone during the lean season to begin to answer this, and their initial findings suggest that program recipients ‘smooth’ their consumption across the year (i.e., save some of the money they make during harvests to support their consumption during the remainder of the year). While this is promising, additional evidence of the program’s lean season effects would increase our confidence in the overall impact of the program.

The grant

We are recommending grant funding for a five-year follow-up to Mahmud and Riley’s existing RCT, and for an additional lean season phone survey to address the seasonality uncertainty mentioned above:

• Five-year follow-up: our grants will fund an additional in-person follow-up survey in August to October 2025, timed for five years after the beginning of program implementation (consistent with the within-year timing of the study’s previous surveys). This will measure the same outcomes and use the same survey instruments as the endline survey that has already been conducted (as specified in the researchers’ pre-analysis plan).7 The original endline survey was conducted three years after the beginning of program implementation, so these grants will generate data on the effects of the program two years after the point at which the study would otherwise have concluded.
• Lean season phone survey: from November to December 2025, after the post-harvest surveys above, the researchers will conduct a phone-based survey to capture the effects of the program on consumption during the lean season between harvests, where we would expect agricultural households’ income and consumption to be lower.

The five-year follow-up is primarily designed to address our uncertainty around the duration of the effect of the program, while the lean season phone survey will help to address our uncertainty around seasonality. While neither will primarily address our uncertainty around effect size, knowing whether the program’s effects can be sustained through three years after the completion of the program would make us somewhat more confident about the plausibility of the initial short-term effects.

In addition to the five year follow-up and lean season phone survey, we are considering providing funding for a possible second randomized evaluation of Raising the Village’s program, which would assess whether or not the findings of this first trial can be replicated (thereby addressing our uncertainty around the program’s effect size directly). However, the results of the five-year follow-up and lean season phone survey will not affect our decision about whether to fund this second evaluation, since we expect to make a decision about the replication RCT before we receive the follow-up and lean season results.

We expect that the five-year follow-up and lean season survey will (a) conditional on the results being positive and us deciding to fund a second trial, increase our willingness to provide Raising the Village with some direct implementation funding and (b) give us an earlier signal about the promisingness of Raising the Village’s program than we would get if we waited for the results of a second randomized trial, which could take approximately five additional years. We think that (b) may be particularly useful in informing the decisions of other funders, who may not wish to wait for the results of a full second study prior to contributing to large-scale funding for the program.

Budget for grant activities

The overall budget for the five-year follow-up and additional lean season phone survey is $542,110. This breaks down into grants to two organizations (Innovations for Poverty Action (IPA) and the University of Michigan), as follows:

• $414,816 to IPA for data collection, of which:
  • $324,948 will support data collection for the five-year follow-up. This will include measurement of the same outcomes as the original trial, plus an additional survey on gender-based violence and psychological well-being.
  • $89,868 will go towards the lean season phone survey.8
• $127,294 to the University of Michigan, of which:
  • $68,067 will pay for Emma Riley’s time (a ‘course buyout’). We’re recommending this funding in order to speed up the time it takes to receive the results of the follow-up and phone survey to February 2026.
  • $59,227 will pay for two semesters of research assistant time.9

Timeline

We expect this grant to adhere to the following timeline:10

• In-person surveys (harvest season): August–October 2025
• Phone surveys (lean season): November–December 2025
• Data cleaning and analysis: October 2025–February 2026
• Initial internal results shared with GiveWell: 2026
• Working paper submitted for publication: 2027

The case for the grant

We are recommending this grant because:

• The results of the surveys could increase our confidence in our cost-effectiveness estimate of Raising the Village’s program, and therefore unlock cost-effective grantmaking opportunities in the future. If the results of the five-year follow-up and additional lean season survey look promising, we may conclude that Raising the Village’s program is above our cost-effectiveness bar of ten times unconditional cash transfers, and would therefore consider making direct delivery grants for Raising the Village in the future, particularly if the results are supported by a second RCT, which we plan to investigate funding. We think this argument is particularly compelling in this case, because:
  • Cost-effectiveness: We’re highly uncertain about our best guess of Raising the Village’s program’s cost-effectiveness, since this is based on the unpublished results of a single randomized evaluation, and we have large outstanding uncertainties, including about the size of the program’s effect on consumption, the duration of the effect, the impact of seasonality on the results, and implementation costs. However, our current best-guess estimate is that the program is around our cost-effectiveness bar.11 This suggests it is plausible that the additional evidence funded by this grant will place Raising the Village's program above our bar.
  • Room for more funding: We think that Raising the Village’s program could scale significantly in the future, both in Uganda (where it aims to reach 14 million people by 2030, which we speculatively estimate could cost around $408 million) and elsewhere in sub-Saharan Africa.12 Therefore, if new evidence shows the program to be above our cost-effectiveness bar, there could be very significant grantmaking opportunities in the future.

Risks and reservations

Our main reservations about this grant are:

• The case for the grant relies on the results of one RCT that has not yet been published. We think it is possible that the results of Mahmud and Riley’s trial could change substantially with peer review, or that additional evidence could lead us to think that Raising the Village's program is less promising than we currently guess. This would reduce the value of the five-year follow-up and additional lean season survey for our grantmaking. Our current cost-effectiveness analysis is particularly sensitive to the factors mentioned above: effect size, effect duration, and seasonality, as well as implementation costs.
• It’s possible this study would be funded without us. We expect the results of the five-year follow-up and lean season survey to be of interest to other funders. Therefore, it’s possible that they would have funded this study in our absence.
• The results of this study won’t come in time to inform our decision about a second RCT. We are interested in funding a second randomized trial of Raising the Village’s program in order to reduce our uncertainty around the size of its effect on consumption. The results of the surveys funded by this grant would affect our confidence in the results of a potential second RCT (and vice-versa), but the survey results won’t arrive in time to affect our decision as to whether or not to recommend funding for a second RCT.

Plans for follow up

We expect to receive the five-year consumption estimates from both the harvest and lean season surveys in February 2026, after which we will update our cost-effectiveness analysis of Raising the Village’s program. We expect the researchers to have a public working paper by June 2027.

We will schedule check-in meetings with both Raising the Village and the researchers throughout the study period, timed around key milestones, such as endline data collection during the post-harvest and lean seasons.

Internal forecasts

For this grant, we are recording the following forecasts:

Our process

• In April 2024, we were informed about this RCT by the Livelihood Impact Fund, with whom we recommended funding for an RCT of eyeglass provision in May 2024.
• From June 2024, we had several conversations about Raising the Village and this study, including with Raising the Village staff, the study’s principal investigators, and the Livelihood Impact Fund, in order to to better understand Raising the Village’s program model and plans for additional evidence generation, and the results of the lean season phone survey.
• Based on the initial results of the RCT and the lean season results, we conducted an initial cost-effectiveness analysis, which was internally peer reviewed. Thereafter, we conducted an intentionally light-touch grant investigation.

Sources

• 1

  “Raising The Village partners with last-mile farming communities in Sub-Saharan Africa over 24 months to address ultra-poverty through a multi-dimensional program model. Each dimension of our program works together to build household incomes year-after-year as communities transition from subsistence farming to income generation, breaking the chronic cycle of ultra-poverty.

  • Driving Income – by increasing agricultural productivity
  • Enabling Participation – by addressing barriers and creating opportunities
  • Ensuring Sustainability – by building local capacity and structures”

  Raising the Village, Our work.

• 2

  This was confirmed by Mahreen Mahmud and Emma Riley in their review of this grant page.

• 3
  • The 1.5 million figure was confirmed in Raising the Village, email to GiveWell, February 14, 2025 (unpublished).
  • “1,091,659 lives impacted through successful partnerships with 219,647 families and counting.” Raising the Village, Our impact Raising the Village's geographical reach is shown in Raising the Village, 2023 Impact Report, p. 16

• 4

  Mahmud and Riley, “We’re All in This Together”: Addressing Poverty in Village Economics [Presentation], April 2024 (unpublished).
  This trial was pre-registered here: AEA RCT Registry, Impact of a village-based poverty alleviation programme in Uganda.

• 5

  This was confirmed by Mahreen Mahmud and Emma Riley in their review of this grant page.

• 6

  GiveWell uses moral weights as a way to value and compare the outcomes of programs that we consider for funding. Our moral weight for increasing consumption is the only moral weight we currently use to evaluate programs that target non-health related outcomes. For more on GiveWell’s moral weights, see GiveWell, 2020 update on GiveWell's moral weights.

• 7

  See ‘Impact of a village-based poverty alleviation programme in Uganda’.

• 8

  The budget for this work is broken down as follows:

  • Household survey - $222,751
  • Village elder survey - $17,411
  • Market price survey - $9,785
  • Gender survey - $75,000
  • Phone survey - $89,868
  • Total - $414,816 (after rounding up to the nearest dollar)

  Source: Mahreen Mahmud, email to GiveWell, December 4 2024 (unpublished)

• 9

  The breakdown of costs between the RA time and the course buyout comes from an email from Mahreen Mahmud, November 7, 2024 (unpublished).

• 10

  Emails from Mahreen Mahmud, October 30, 2024 and November 20, 2024 (unpublished)

• 11

  Our BOTEC for the cost-effectiveness of Raising the Village's program can be found here.

• 12

  This room for more funding calculation is here: GiveWell, Raising the Village BOTEC, 2024.