Summary
In July 2022, GiveWell recommended a grant of $1.4 million to the Regents of the University of California, Berkeley to support a seven- to eight-year follow-up of a randomized controlled trial of GiveDirectly’s unconditional cash transfer program in Kenya.
We’re recommending this grant because:
- There is limited long-term evidence for spillover effects of unconditional cash transfers on non-recipient households and limited evidence for effects of unconditional cash transfers on child mortality. We think this follow-up study provides a unique opportunity to gain high-quality estimates for both.
- This evidence could inform how much we value not just GiveDirectly and unconditional cash transfer programs in general, but also other interventions that increase household consumption (e.g., by informing our moral weight on consumption). Because there’s a lot of room for more funding for unconditional cash transfers and other interventions that increase consumption, this has the potential to make a large difference in our funding recommendations.
- Given the large amount of attention on unconditional cash transfers, we guess this evidence could also be useful to other funders, as well as researchers and GiveDirectly itself.
Our main reservations are:
- It seems more likely than not that someone else would fund additional follow-up at some point. We view our funding as reducing the chance this doesn’t get funded, potentially improving the quality by limiting attrition and/or enabling the research team to keep its Kenya staff in place, and speeding up the availability of these results to inform decision-making by GiveWell and others.
- Because of the relatively small grant amount, we haven’t tried to model exactly how we’ll update our funding decisions based on the results, nor have we quantified the cost-effectiveness of this grant relative to other opportunities we’re considering. However, we believe that it exceeds our cost-effectiveness bar because of the high information value of this research.
Published: November 2022
Table of Contents
About the grant
GiveWell recommended a grant of $1.4 million to researchers at the University of California, Berkeley ("UC Berkeley") to support a seven- to eight-year follow-up of a randomized controlled trial of GiveDirectly’s unconditional cash transfer program in Kenya.
This research is being led by Edward Miguel and Michael Walker of UC Berkeley, and Dennis Egger of the University of Oxford. Others may be involved in collaboration, but this has not yet been determined.1
We have previously recommended other grants to support research in which Professor Miguel is involved, including:
- Identification of previously completed randomized controlled trials (RCTs) in global health and development that could be candidates for follow-up studies on long-term effects,
- the Kenya Life Panel Survey Round 4 (KLPS-4), a 20-year follow-up to Miguel and Kremer 2004,
- an investigation into the feasibility of following up on other deworming studies, and
- research support to support studies on deworming and cash transfers.
Grant activities and budget
This grant will support additional follow-up of a randomized controlled trial of GiveDirectly’s unconditional cash transfer program in Kenya. We have written previously about a paper reporting short-term follow-ups from this RCT here. That paper has been accepted for publication as Egger et al. 2022.2
In the RCT, one-time $1,000 cash transfers were provided by GiveDirectly between 2014-2017.3 Egger et al. 2022 reports results from endline surveys conducted in 2016-2017.4 A second round of endline surveys took place from 2019-2021.5
This grant will support a third endline in 2023-2024.6 This endline will include a full household census, which has not been done since the baseline.7
The study is intended to measure the following:8
- Long-term gains for recipients of cash transfers,
- Long-term spillover effects to non-recipients and the local economy, and
- Effects of cash transfers on child mortality and other outcomes that they will have greater statistical power to detect, due to a larger sample size for this round of data collection.
The budget is $1,397,102 over 3 years. It breaks down as follows:9
- Field survey data collection with consulting firm REMIT Kenya: $797,553
- Personnel costs (UC Berkeley staff): $410,694
- Travel, materials, and other costs: $49,145
- Indirect costs: $139,710
Case for the grant
- We think this study could provide an update on the benefits of cash transfers, including direct and spillover effects on household consumption, effects on child mortality, and other outcomes aside from consumption, as well as how long these effects last. We view spillovers as an area of particularly high uncertainty, which we have written about here. This study will provide additional years of data from the largest and highest-quality study of the spillover effects of unconditional cash transfers.10 Our impression is also that few studies have been able to reliably measure the effect of unconditional cash transfers on child mortality.11 One particular benefit of this follow-up study is that the household census that will be collected will permit estimating effects on child mortality.
- Updates on the impact of cash transfers could change our view on the overall cost-effectiveness of unconditional cash transfers and potentially other interventions that increase household consumption—and, in turn, how much money we allocate to these interventions in the future. There are a few avenues through which this study could change our funding decision-making, though we have not tried to quantify these:
- While we estimate that unconditional cash transfer programs are substantially below our bar for cost-effectiveness,12 there is some chance that this could change in the future if we lower our bar due to a larger amount of funds raised. If this study positively updated us on the cost-effectiveness of cash transfers, the likelihood of us directing funding to cash transfers would increase (i.e. our bar would need to lower by a smaller amount in order to consider funding cash transfers). Given the potentially large amount of room for more funding available for unconditional cash transfer programs,13 this could potentially shift our funding recommendations substantially.
- If the study finds large and persistent benefits to non-recipient households and/or benefits beyond consumption, such as reduction in child mortality, that may change how we view this and other programs that increase consumption. For example, we may decide to reevaluate our moral weight on increases in consumption to reflect benefits that are not currently incorporated into that moral weight.14 This would increase the cost-effectiveness of any program that increases consumption and could lead to changes in how much we recommend funding to programs that increase consumption and are closer to our cost-effectiveness bar. This could impact a very large amount of potential funding, given all the interventions we may consider that include an impact on consumption.15
- The study may also identify specific sub-groups that benefit more from cash transfers. These findings could lead us to prioritize investigation of unconditional cash transfers in areas where or for individuals to whom unconditional cash transfers would be more cost-effective.
- There may be additional impacts on our decisions and views of different interventions that we haven’t thought of yet.
- Results from this study could also potentially be useful to other funders considering unconditional cash transfers, as well as researchers and GiveDirectly itself. We have not explored how the findings might be used by these others, though we think these findings could be of interest to organizations beyond GiveWell, given broad interest in the impact of unconditional cash transfers, either through direct funding or use as a benchmark. The study may also provide information to researchers and to GiveDirectly.
- We think the study will be high-quality. The researchers have already successfully conducted two follow-ups with this sample,16 and the results from the first follow-up study have been accepted to a top economics journal.17 The researchers have told us they are not concerned about attrition between the previous follow-ups and this round, which could compromise the results if retention was low.18 We have not requested power calculations for effects on household consumption. For child mortality, which was not measured in previous follow-ups,19 the researchers report a minimum detectable effect (MDE) of 12% to 13%.20 The researchers have told us the study will include a pre-analysis plan that will be similar to pre-analysis plans for previous follow-ups, though the pre-analysis plan for the seven- to eight-year-follow-up has not yet been written.21
- This could help provide a fairer assessment of GiveDirectly in communicating cost-effectiveness. We use GiveDirectly as a benchmark in our cost-effectiveness analysis, but we haven’t dug particularly deeply into spillover effects, long-term impacts, or impacts on other outcomes like child mortality. Even though a shift in GiveDirectly’s cost-effectiveness (e.g., from 1x to 2x of our current estimate) would not change GiveWell’s overarching decision-making, it could be material for GiveDirectly in terms of being a fairer comparison with other giving opportunities.
- Given the relatively low cost of this grant, relative to potential benefits, this seems like a good use of funding.
Risks and reservations
- We have not tried to model the cost-effectiveness of this grant or quantify how we could change our views based on the results. We think it’s plausible this is as cost-effective or more cost-effective than other grants we would recommend funding, but we have not tried to compare it to specific opportunities, as we have done with other recent research grants.22 This includes modeling how we would incorporate updates to our best guess on spillovers. We have a number of uncertainties in how to interpret the results of Egger et al. 2022 and other studies of spillovers of unconditional cash transfers (see this page), and we have not yet addressed these. We also have not yet modeled how we would update our assessment of other interventions beyond unconditional cash transfers, based on findings from this study on spillover effects or effects on child mortality.
- This grant is off-strategy for GiveWell, compared to other research grants we’ve made recently. Many research grants we make are for programs where we think they are closer to our bar for cost-effectiveness and we think there’s a decent chance we would update toward believing the program is more or less cost-effective, based on the trial. In this case, we think it’s highly unlikely we’ll update toward believing GiveDirectly is above our bar based on the results of this study. The case for the grant is based more on the belief that there is general value to this study that could update us in ways we haven’t yet tried to quantify.
- We think it’s more likely than not that additional follow-up would be funded without us. This is based on the research team being able to secure funding for previous follow-ups. Even if the researchers were unable to secure funding within the next year, it seems unlikely that they would not receive funding to follow up sometime in the next few years. We view our funding as reducing the chance this doesn’t get funded, potentially improving the quality by limiting attrition and enabling the research team to keep Kenya staff in place,23 and speeding up when these results are available to inform decision-making by GiveWell and others.
- There is probably a smaller marginal benefit from this follow-up, compared to initial rounds. A seven- to eight-year follow-up might not add that much information, relative to the five-year follow-up (which is already underway as Egger et al. 2021). However, the five-year follow-up did not include a household census, which will be included in the seven- to eight-year follow-up and used to measure additional outcomes, such as impacts on child mortality, that require a larger sample to generate enough statistical power.24
Plans for follow-up
We expect to see child mortality data at the end of 2023, when the household census is complete.25 We expect draft results in the fall of 2024, three to six months after data collection ends in early- to mid-2024.26
Internal forecasts
We think there is a 30% chance we will positively update our cost-effectiveness estimate of GiveDirectly by 20% or more based on the results of this RCT. We think there is a 20% chance we will positively update our moral weight on increasing consumption by 20% or more based on the results of this RCT.
Process
We followed a light review process for this grant, given the small size of the grant and our belief that further investigation was unlikely to change our decision about whether or not to recommend funding.
We had one phone call with Edward Miguel and Michael Walker in January 2022, then we requested additional information through email before recommending the grant in July 2022.
Sources
- 1
“For now, the primary study authors are myself, Michael (copied here), and Dennis Egger (who will start a faculty position at the University of Oxford this summer). Other colleagues on the broader GE project would likely also to be involved (but we will need to confirm their interest in this aspect of the work), including Johannes Haushofer and possibly Paul Niehaus. We have growing collaborations with several researchers based in Kenya and it is possible some of them would be study authors. We are also working with several leading child health and development specialists on our KLPS work, including Lia Fernald, Sarah Baird, and Emily Smith, and we will also engage them to determine their interest in collaborating on this work.” Edward Miguel, UC Berkeley, email to GiveWell, June 9, 2022 (unpublished)
- 2
The Econometric Society, "New Papers Posted to Econometrica's Forthcoming Page," 2022
- 3
- “We designed and carried out a large-scale experiment in rural Kenya that provided one-time cash transfers worth roughly USD 1000 (distributed by the NGO GiveDirectly) to over 10,500 poor households in a sample of 653 villages with a population of roughly 300,000.” Egger et al. 2022 (preprint), p. 2.
- “Transfers and data collection took place from mid-2014 to early 2017, a period of steady economic growth, relative prosperity, and political stability in Kenya.” Egger et al. 2022 (preprint), p. 6.
- 4
“We conducted a total of 8,239 endline household surveys between May 2016 and June 2017.” Egger et al. 2022 (preprint), p. 9.
- 5
“The analyses outlined in this document will be primarily based on a new round of data collected between 2019 - 2021 (Endline 2), roughly 5 years after the GD cash transfers went out, as highlighted in the approximate timeline below.” Egger et al. 2021 (working paper), p. 2.
- 6
“Dr. Miguel and Dr. Walker are seeking funding for additional follow-up research, with data collection to take place in 2023-2024, about 10 years after the cash transfers were provided.” GiveWell’s non-verbatim summary of a conversation with Edward Miguel and Michael Walker, January 6, 2022
- 7
- “The researchers have already completed a household census of every household in the area [at the baseline in 2014], totaling 300,000 people. Detailed surveys were conducted with a random sample of those households. About 5,000 children have been born in households where detailed surveys were conducted. In the next household census, researchers could ask about child births and deaths since 2014. That would provide a much larger sample size, offering additional statistical power to detect effects of living standards on child mortality.” GiveWell’s non-verbatim summary of a conversation with Edward Miguel and Michael Walker, January 6, 2022
- “The one additional point to emphasize is that the sample for the child birth and mortality analysis will be far larger than the specifications used in the endline 2 analysis, since it will be based on the full census of nearly 70,000 households, rather than the representative subsample of roughly 9,000 households that have been the focus of most of the detailed econometric analysis to date.” Edward Miguel, UC Berkeley, email to GiveWell, June 13, 2022 (unpublished)
- 8“The study, which includes both household surveys and enterprise surveys, will have focus on three areas:
- long-term gains for recipients of cash transfers
- long-term spillover effects on the local economy and on those who did not receive cash transfers
- effects of cash transfers on child mortality
Because the sample size is very large, the researchers will have sufficient statistical power to examine a number of additional variables, including a subgroup analysis to determine which groups are benefiting the most and why. The household census data will be powered to identify various effects, and the detailed subset of household surveys will be able to explore the mechanisms underlying those effects.” GiveWell’s non-verbatim summary of a conversation with Edward Miguel and Michael Walker, January 6, 2022
- 9
Michael Walker, UC Berkeley, comments on a draft of this page, October 17, 2022 (unpublished).
- 10
“We pre-registered our belief that the [general equilibrium] study would play a large role in our conclusions in this post, because it is the largest and highest quality study that we have seen on spillover effects.” GiveWell, "Spillover Effects of GiveDirectly's Cash Transfers Program," 2018
- 11
- The researchers identified one randomized trial in Nigeria, (Okeke and Abubakar 2020) as well as three papers from Latin America (Ramos et al. 2021, Moncayo et al. 2019, and Barham 2010), which all study conditional cash transfers. The three Latin America papers rely on statistical analysis of existing population-based data.
- A review by the Abdul Latif Jameel Poverty Action Lab (J-PAL) in 2020 concludes that “While evidence to date points to the potential for health improvements in early childhood to lead to better educational outcomes later in life, more follow-up studies are necessary to clearly identify under which conditions and through which mechanisms health improvements translate into better education and whether those ultimately improve labor market outcomes. Nevertheless, cash transfers conditioned on uptake of healthy behaviors can improve uptake of child health interventions in the short term and improve educational outcomes in the longer term” J-PAL, "Using cash transfers to improve child health in low- and middle-income countries," 2020
- We have not reviewed these papers in depth, but our impression is that there is room for more evidence on child mortality, especially on the effect of unconditional cash transfers.
- “So my take on this is that there is still plenty of room for more research that: (1) is based in a low or lower middle income setting (like our study area in rural western Kenya, rather than far richer Latin American settings), (2) has a large sample size (which we will be able to accomplish be censusing over 65K households in this area, with a 8-9 year follow up period since cash was distributed), and (3) importantly utilizes experimental variation. An added advantage of our planned study is that for around 9K households we will have really detailed panel data over 8-9 years, allowing us to also hopefully make some progress on mechanisms and deeper understanding of heterogeneity.” Edward Miguel, UC Berkeley, email to GiveWell, February 19, 2022 (unpublished)
- 12
- We currently describe our bar for funding compared to unconditional cash transfers to GiveDirectly ("x cash"). We estimate unconditional cash transfers provide 335 units of value per $100,000 spent. See here, GiveWell, Cost-effectiveness analysis version 5, 2022. Sheet: GiveDirectly.
- Based on our current pipeline of spending opportunities and our projection of funds raised, our current bar is 10x cash, or 3,350 units of value per $100,000 spent. GiveWell blog, "An update on GiveWell's funding projections," 2022
- If, for example, we doubled our estimate of GiveDirectly's cost-effectiveness but our current pipeline and projection our funds raised stayed the same, our bar would still be 3,350 units of value per $100,000 (though we could describe this as 5x cash transfers from GiveDirectly). We would recommend funding programs producing at least 3,350 units of value per $100,000.
- It's possible that in the future our projection of funds raised increases sufficiently that we lower our bar so that it is below the cost-effectiveness of 3,350 units of value per $100,000. If, for example, we doubled our estimate of GiveDirectly's cost-effectiveness and saw an unexpectedly large increase in funds raised, our bar could drop below 670 units of value per $100,000, in which case GiveDirectly would be above our bar.
- 13"We believe that GiveDirectly is highly likely to be constrained by funding in the next three years. With additional funding, it could significantly increase the number of cash transfers it delivers in six countries and potentially expand to additional countries. Over 2021-2023, we estimate that GiveDirectly could productively use several hundred million dollars more than we expect it to receive." GiveWell, "GiveDirectly," 2020
- 14
As a related example, researchers at the Happier Lives Institute have suggested incorporating the effect of programs on life satisfaction scores or other measures of subjective well-being into our estimates of moral weights. Because this research will include effects on subjective well-being measures, it could provide an update on the moral weight assigned to increases in consumption using this type of subjective well-being approach.
- 15
This includes interventions that both improve child health and lead to development benefits.
- 16
- 17
See Egger et al. 2022 (preprint), accepted to Econometrica.
- 18
- “We are not that concerned about attrition. In the current follow-up survey round (which is just wrapping up this month), we have already achieved tracking rates in the 80-85% range for both households and firms, with balance across treatment groups. (And these rates may rise slightly as we finalize data collection.) This rate includes those who have moved out of the study area (i.e., to Nairobi); focusing solely on those living within the study area, the successful household survey rate is closer to 90%. As part of this survey round, we have collected updated and detailed contact information which will facilitate the collection of further survey rounds, and we have maintained good relationships with sample communities. We are hopeful and confident that we can achieve similarly high tracking rates in the next survey round (as we have for multiple follow up rounds in related surveys in Kenya, such as the Kenya Life Panel Survey).” Edward Miguel, UC Berkeley, email to GiveWell, June 9, 2022 (unpublished)
- “We achieved high tracking rates at endline, reaching over 90% of eligible and ineligible households in both treatment and control villages, and these rates do not systematically vary by treatment status (Table F.1).” Egger et al. 2022 (preprint), p. 9.
- 19
- "For now, our sense is that we would collect data similar to what we are collecting now in EL2, plus detailed data on all children born and their mortality, as well as their associated health issues (e.g., morbidity), access (e.g., to antenatal care), and behaviors (e.g., vaccinations, deworming, bednet use, etc.) in far more detail then we have collected to date in the census activity." Conversation notes don't explicitly say this but it's implied: "A relationship between living standards and child mortality has been hypothesized, but it has been difficult to quantify that relationship with an experimental design that has a large enough sample. The follow-up research proposed by Drs. Miguel and Walker would be able to examine this relationship. The researchers have already completed a household census of every household in the area, totaling 300,000 people. Detailed surveys were conducted with a random sample of those households. About 5,000 children have been born in households where detailed surveys were conducted. In the next household census, researchers could ask about child births and deaths since 2014. That would provide a much larger sample size, offering additional statistical power to detect effects of living standards on child mortality. The researchers could validate the data based on detailed fertility questions that they have asked a subset of households." Edward Miguel, UC Berkeley, email to GiveWell, June 10, 2022 (unpublished)
- "A relationship between living standards and child mortality has been hypothesized, but it has been difficult to quantify that relationship with an experimental design that has a large enough sample. The follow-up research proposed by Drs. Miguel and Walker would be able to examine this relationship." GiveWell’s non-verbatim summary of a conversation with Edward Miguel and Michael Walker, January 6, 2022
- 20
“After a few days of work, we have results to report on the power calculations. Here is the summary:
-- We focus on under-5 mortality as the main outcome
-- We base population and fertility numbers on the existing rounds of GE data, and the child mortality levels on KLPS data. Average under-5 mortality in KLPS over this period is 60 per 1000 births (which is pretty similar to the Kenyan national average)
-- The MDE results are based on 80% power, 95% significance (and we also mention 90% significance results), and intra-cluster correlation of 0.018 (based on fertility patterns in the GE data), in the main T vs C village comparison.
>> The MDE at 95% significance is a reduction of 7.6 per 1000 births, or 12-13% of the base level of 60 per 1000 births. At 90% significance, the MDE is a reduction of 6.7 per 1000 births, or roughly 11% on the base level.
Regarding magnitudes, this MDE is considerably smaller than the estimated under-5 mortality effects of deworming on the next generation of children (which is 18 per 1000 births, see brand new working paper attached), or of chlorine treatment (14 per 1000 births, working paper here: Haushofer et al. 2021 (working paper)). It is hard to know what child mortality effects are likely or "reasonable" in this context of cash transfers (and persistent income gains), but we see this design as reasonably well-powered to detected moderate effects.” Edward Miguel, UC Berkeley, email to GiveWell, June 16, 2022 (unpublished) - 21
“Is it possible to share either a draft pre-analysis plan for endline 3 or a list of the main outcomes you plan to look at? Rough documents are just fine.”
“We have not yet written up this document. For now, our sense is that we would collect data similar to what we are collecting now in EL2, plus detailed data on all children born and their mortality, as well as their associated health issues (e.g., morbidity), access (e.g., to antenatal care), and behaviors (e.g., vaccinations, deworming, bednet use, etc.) in far more detail then we have collected to date in the census activity. Please let us know if you would like us to flesh this out a bit more, and the degree of detail, and we will gladly draft this. The bottom line is that we are confident that we can obtain rich and reliable child birth and mortality histories in this population, providing a unique data environment -- especially when combined with a cash transfer RCT at this scale and over this time frame.” Edward Miguel, UC Berkeley, email to GiveWell, June 10, 2022 (unpublished) - 22
See, for example, GiveWell, University of Chicago — Evaluation of Mobile Conditional Cash Transfers (November 2021) or GiveWell, “Bridges to Prosperity – Trailbridge Building in Rwanda (May 2022),” 2022.
- 23
“There is no single hard deadline but knowing by June would be [very] useful for us in making staffing decisions on our team.” Edward Miguel, UC Berkeley, email to GiveWell, April 19, 2022 (unpublished)
- 24See above.
- 25
Edward Miguel, UC Berkeley, conversation with GiveWell, September 16, 2022 (unpublished)
- 26
“For sharing results, 3-6 months after the end of data collection sounds great! Can you confirm when data collection would end?
We would plan to ramp up preparations and planning during fall 2022, launch the survey in early 2023, and collect data through early-mid 2024. So 3-6 months after that would be roughly fall 2024.” Edward Miguel, UC Berkeley, email to GiveWell, June 13, 2022 (unpublished)