The International Initiative for Impact Evaluation — Scoping of Long-Term Follow-Up Study on Preschool Program (December 2024)

Note: This page summarizes the rationale behind a GiveWell grant to The International Initiative for Impact Evaluation (3ie). 3ie staff reviewed this page prior to publication.

Published: March 2025

Background

Between 2008 and 2010, Save the Children implemented a preschool program for children aged 3-5 in 30 rural communities in Gaza province, Mozambique. The program was led by community volunteers who acted as classroom instructors and delivered cognitive stimulation exercises (including games, art, and music), elementary math and reading classes, and an introduction to Portuguese (in preparation for elementary school).1 The program also organized monthly parent groups focused on breastfeeding, deworming, nutrition, literacy, and school attendance.2 The cost of the program was around $3 per student per month.3

This program was evaluated in a cluster randomized controlled trial, Martinez et al. 2017, which we think was well-conducted.4 This RCT found that, two years after the start of Save the Children’s program:

• Children who attended the preschool program saw improvements in cognitive development, communication, fine motor skills, and socio-emotional skills;5
• Children who were offered the program were 8 percentage points more likely to enroll in primary school. Among those who actually attended preschool, the increase was larger at 21 percentage points;6
• The program had generated positive spillovers on the school enrollment rates of older siblings, and adult caregivers’ participation in the workforce.7

The grant

With this grant, 3ie will scope the possibility of conducting a long-term follow-up to Martinez et al. 2017, which would measure economic, education, and health-related outcomes when the original cohort is 21-23 years old.8 As discussed below, the primary intention behind this long-term follow-up is to contribute to the evidence base on whether early childhood education can improve long-term economic outcomes. The outcomes shown in the initial study (such as improvements in cognitive development or an increase in the likelihood of enrollment in primary school) might be plausible pathways by which this could occur.

3ie’s scoping work will focus on whether 3ie can achieve a follow-up rate with the cohort of over 80%, as the researchers predict.9 The researchers’ last touchpoint with the cohort was in 2014, and consent for long-term follow-up was not planned for in the original study. If follow-up rates are lower than 80% (i.e. the scoping work finds attrition of over 20%), GiveWell would not fund the remainder of this study.

To determine the attrition rate, 3ie will:10

• Develop a tracking protocol, including for individuals that may have migrated elsewhere (either to other parts of Mozambique or internationally);
• Conduct a tracking exercise on a random subsample of the study population to estimate the likely attrition rate for the full sample;
• Submit an updated study protocol for ethical approval, and request study permissions from national, sub-national, and local authorities.

Budget for grant activities

This grant of $169,691 represents half of the total $339,381 budget for the scoping portion of this study. The remaining half will be funded by Open Philanthropy. If funded, the total budget for the overall study (including this scoping grant) would be $2,498,711, so GiveWell and Open Philanthropy would consider a follow-on grant of $2,159,330.

The budget for the scoping grant breaks down as follows:11

• $179,718 (53% of the cost of the grant): personnel
• $83,051 (24%): 3ie’s indirect costs
• $38,140 (11%): travel
• $30,000 (9%): data quality and assurance
• $7,200 (2%): equipment
• $1,173 (<1%): Institutional Review Board fees and software licensing

The case for the grant

We are recommending this grant because:

• If this scoping grant is successful, this study would help to fill an evidence gap on the long-run returns to education, which could lead to GiveWell recommending grants for education programs in the future.
  • To date, GiveWell has not recommended significant funding for education programs, primarily due to the limited evidence linking them to income gains in low-income countries.12 However, we have long held an interest in helping to fill this evidence gap should an opportunity become available, as we discussed in our 2018 intervention report. In 2022, Open Philanthropy funded the Center for Global Development (CGD) to build out the evidence base, although to date this work has focused on evaluations of primary and secondary education interventions rather than preschool studies for long-term follow-up.
  • We think this study offers a good opportunity to contribute to the evidence base on this issue. To our knowledge, it would be the first long-term follow-up study of a preschool program in a low-income setting. If it found a meaningful long-term effect on earnings, this would be notable for us, even if it wouldn’t be sufficient evidence to cause us to update our overall view of education programs (due to a lack of generalizability, as discussed below). It could also help us to identify interim outcomes (such as specific cognitive gains, or school attendance) that are predictive of long-term earnings increases.
  • We understand that Open Philanthropy is currently considering developing a consortium for education research, specifically focused on this topic, with the aim of generating multiple studies appropriate for a future meta-analysis. By contributing to this study, we are also signalling our interest in contributing to this consortium, and potentially funding other research in the future.
• We think this scoping grant will allow us to learn about the feasibility of the overall study, and could have broader benefits. Through this initial scoping work, we expect to learn whether the researchers can achieve follow-up rates of above 80%, which we think is necessary for the overall study to be informative, and which will therefore inform our decision about whether or not to recommend funding for the full study. In addition, this grant should provide a useful signal for other long-term follow-up studies, e.g. by providing a benchmark for costs and highlighting potential implementation challenges that arise when tracking a cohort over a long time horizon.
• We view this grant as ‘on strategy’ for GiveWell’s livelihoods grantmaking. This year, one of our primary goals for our livelihoods grantmaking portfolio has been to fund research that could expand GiveWell’s giving opportunities in the future. In this case, we think this grant will teach us about the feasibility and promisingness of funding follow-up studies to build out the evidence base on the long-term economic returns to early education (in addition to directly informing whether to fund a program similar to the one evaluated in this study). For example, we expect to learn whether the tracking methods in this grant are sufficient to achieve high follow-up rates for a study that requires ‘reconsenting’ (i.e., where long-term follow-up wasn’t planned for at the outset). If not, we will likely view similar opportunities to this as less promising in the future.

Risks and reservations

Our main reservations about this grant are:

• Even if the results of the full study are promising, we may not be able to recommend grants for similar programs in the future, due to a lack of comparable programs currently being implemented. We understand that the Government of Mozambique scaled an intervention modeled on the Save the Children program evaluated in Martinez et al. to 350 communities and around 50,000 children in Mozambique between 2014 and 2019, and management of the Save the Children preschools has since transferred to local communities. Though we're uncertain about whether Save the Children would implement the program in future, its relatively simple model may make it more replicable than other evaluated education programs. According to 3ie, community-based preschooling has been implemented in various settings across Africa, though we have not independently verified this claim.13
• The full study would have technical constraints, including limited statistical power, and measuring outcomes earlier in participants’ careers than we think would be ideal.
  • Power: If the researchers are able to achieve their expected attrition rate of 18%, the study would be powered to detect an effect on wages of 0.2 standard deviations, or about a 13.7% logarithmic increase.14 Based on a light review, we think that smaller effects than this could still be sufficient for this intervention to look cost-effective to us, which this study could miss. However, our impression is that this will be a challenge for most long-term follow-up studies, given that education evaluations tend to have relatively small sample sizes and effects are likely to fade out over time. Moreover, we think that the study could still provide useful information even if it cannot detect a statistically significant effect, particularly if this study can be supplemented by others in a meta-analysis.
  • Timing: Since wages are typically at their highest in middle age, we do not expect this study to detect the full earnings benefits of the intervention at this point in time, when the cohort is aged between 21 and 23 years old. However, obtaining participants’ consent for long-term follow-up now is crucial for any future longer-term evaluations, given the need to keep attrition rates as low as possible.
• This study may not be the best candidate for long-term follow-up due to the limited effect sizes it detected. Martinez et al. 2017 found moderate to large effects on cognitive development and socioemotional skills, but only modest effects on primary school enrollment, and only small positive spillovers on other household members. This makes us uncertain about whether there is a plausible mechanism through which we would expect to see large increases in participants’ earnings over time.
• We think that this study will likely have limited generalizability. Our impression is that studies of education programs typically have lower generalizability than studies of other programs GiveWell might fund, due to heterogeneity in program design (as we discuss in our intervention report on education).15 In practice, this means that the results of this study would only affect our opinion about a narrow range of similar education programs, rather than the field as a whole. However, this study could still be useful as a means of beginning to build a broader evidence base for the returns to preschool education, if it can be supplemented with other research.
• We investigated this quickly, so it’s possible that we’re missing something. Our grant investigation was brief, in proportion to the relatively small value of the grant. However, Open Philanthropy also independently decided to recommend funding, which we view as a reassuring sign about the possibility we missed something material as a result of our shortened review.

Plans for follow up

We plan to set up regular check-in calls with the 3ie researchers over the next year, to hear updates on the scoping activities. We expect to make a decision on funding the remainder of the grant by January 2026.

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence	Prediction	By time	Resolution
50%	Attrition rates are expected to be less than 20%, and GiveWell and Open Philanthropy decide to fund the follow-up study.	January 2026	-
13%	We think this program is >10x cash16 after the long-term follow-up concludes by January 2028.	January 2028	-
10%	GiveWell spends >$5m on direct delivery of education programs by January 2030.	January 2030	-
5%	GiveWell spends >$20m on direct delivery of education programs by January 2030.	January 2030	-

Our process

• GiveWell and Open Philanthropy ran a request for proposals for research to inform our grantmaking. (Once this process has concluded, we plan to publish a blog post looking at how this went.) We received 3ie’s initial expression of interest, then invited a full proposal and asked specific questions that we wanted addressed, before reviewing the final proposal we received. We also reviewed the initial study, Martinez et al. 2017.
• GiveWell and Open Philanthropy mutually decided to co-fund this grant, since both organizations were interested in this scoping work.

Sources

Document	Source
International Initiative for Impact Evaluation (3ie), Homepage	Source (archive)
Martinez et al. 2017	Source
GiveWell, Education in Developing Countries	Source
Open Philanthropy, Center for Global Development Europe — Education Research	Source (archive)
GiveWell, Internal forecasts	Source
International Initiative for Impact Evaluation (3ie), Proposal to GiveWell and Open Philanthropy	Source
International Initiative for Impact Evaluation (3ie), Budget - Long Run Returns to ECD in Africa’	Source
Dr. Sebastian Martinez, email to GiveWell, 12/16/2024 (unpublished).	Unpublished
GiveWell, GiveWell's Cost-Effectiveness Analyses	Source

• 1
  • “Given the scarcity of qualified instructors in the area, preschool teachers were not formally trained educators. Instead, instructors were recruited from within communities, provided basic training and supervision by Save the Children, and paid a nominal fee of US$10 per month.” Martinez et al. 2017, p. 9.
  • “The primary language of instruction was Changana, the local vernacular, and the curriculum gradually introduced Portuguese into learning activities to help children prepare for primary school.” Martinez et al. 2017, p. 9.

• 2

  “In addition, the project organized monthly parent group meetings to strengthen positive parenting practices at home. Meetings were facilitated by the community development agents with the aid of the pre-school teachers and a parent of the day, who was chosen by the community. Some of the topics discussed at parent meetings included breastfeeding, de-worming, nutrition, child development, pre-math and pre-literacy, playing with children, attendance and demand driven components defined by the community. Additional details of the preschool model are discussed in Martinez et al 2017.” 3ie's proposal to GiveWell and Open Philanthropy, p. 3.

• 3

  “At a cost of US$3.09 per child per month, the program provided up to three preschool classrooms per community, community mobilization activities, learning materials,instructor training, and monthly parenting meetings” Martinez et al. 2017, p. 3.

• 4
  • This is because it had successful random assignment, good baseline equivalence (see table 2, p. 38), low and non-differential attrition (5.2% in the treatment group and 5.1% in the control group), and valid outcome measures (e.g. primary school enrollment, anthropometric measurements, and the “Ages and Stages Questionnaires” (ASQ) for cognitive development).
  • “An intensive tracking effort was made to locate the target child and minimize sample attrition. If the child had moved from his or her original place of residence we attempted to interview the child (and their current household) so long as he or she maintained residence in Gaza Province (including outside the three intervention districts) or had moved to the capital city, Maputo. Overall, we successfully located 94.9% of the baseline sample, for an average attrition of approximately 2.5% per year and with no differential attrition between treatment and control (94.8% re-contact in treatment, 94.9% re-contact in control).” Martinez et al. 2017, p. 10.

• 5

  “We find that children who attend preschool demonstrate large and significant improvements in cognitive and problem-solving abilities, communication and receptive vocabulary, fine-motor skills and socio-emotional and behavioral outcomes. As such, children are better prepared for primary school and outperform their peers in the control group on these dimensions.” Martinez et al. 2017, p. 4.

• 6

  Table 6, Column: “Currently enrolled at school,” Row: “OLS: treatment community” Martinez et al. 2017, p. 44.
  “Preschools were 21 percentage points more likely to be enrolled in primary school, 14.9 pp more likely to enroll at the appropriate age, and had higher cognitive scores in first grade”. Martinez et al. 2017, Abstract

• 7

  “Children 10 to 14 years old were 2.8 percentage points more likely to have attended school and reported spending fewer hours caring for younger children. Adult caregivers were 3.7 percentage points more likely to report working in the past 30 days (effect significant at the 10% level).” Martinez et al. 2017, p. 4.

• 8

  “Children in treatment communities were more likely to enroll and flourish in primary school. By 2026, at the time of the proposed long-term re-contact, the study cohort will be 21 to 23 years old, allowing us to observe impacts into young adulthood. Long-term outcomes of interest include educational attainment, labor market participation and income, health status and risky behaviors.” 3ie's proposal to GiveWell and Open Philanthropy, p. 1.

• 9
  • “Between existing survey rounds from 2008, 2010 and 2014, we experienced about 1% attrition of the sample per year. If we extrapolate these numbers to 2026, we obtain an attrition rate of approximately 18%, or a final analytical sample for the long-term survey of approximately 1640 subjects. We are optimistic about keeping attrition to 18% or less given that we have GPS coordinates of the child’s original dwelling and contact information from relatives and neighbours.” 3ie's proposal to GiveWell and Open Philanthropy p. 5.
  • The authors currently assume an attrition rate of 18%. We think this may be optimistic, because the researchers arrive at this figure by noting they had 1% attrition per year for the first two years, and then extrapolate to 2026 and get 18%. However, it seems unlikely to us that attrition is linear: we would expect it to be substantially harder to track down the sample after 10+ years of no contact.

• 10

  3ie's proposal to GiveWell and Open Philanthropy, p. 5.

• 11

  See ‘3ie Budget - Long Run Returns to ECD in Africa’.

• 12

  “The evidence that education interventions can increase earnings is limited. Furthermore, the strongest evidence we are aware of (Duflo, Dupas, and Kremer 2017 and Bettinger et al. 2017) estimates that increases in earnings are driven by children who attend (or applied to) vocational schools, rather than academic schools, as discussed in more detail above. We expect this evidence to improve over time, particularly for children who attend academic schools, as the authors of Duflo, Dupas, and Kremer 2017 collect more rounds of data. To further strengthen this evidence, we would like to see more high quality long-term studies of the effects of a range of education interventions on labor market outcomes and earnings.” GiveWell, Education in Developing Countries

• 13

  “The community based preschool model that we evaluated was scaled to 350 communities and more than 50,000 children in Mozambique between 2014-2019, so the long-term evidence would extrapolate quite directly to this larger intervention. Presumably these newer preschools continued in operation after the program ended in 2019, so the total number of children may be closer to 100k by now. I am also aware of community based preschools in a number of other African countries including Malawi, Tanzania, Rwanda, Kenya, Cameroon, the Gambia and Senegal. I agree that quality and other factors are likely to differ from context to context, and I’m not sure how much data exists to allow for cross-country comparisons in Africa, but this is something we could look at further as part of the research project. My sense is that our study population is also quite typical of many parts of rural sub-Saharan Africa in terms of educational, health & nutrition and economic constraints. So if the long-term results were encouraging, these could support investments on both the intensive margin – investing in improving quality of existing community based preschool programs around the continent, as well as extensive margin – investing in expanding community based preschool into new areas.” Dr. Sebastian Martinez (original study author and research team member in 3ie’s proposal), email to GiveWell, 12/16/2024 (unpublished).

• 14

  "Power estimates are presented in Appendix Table A.1. Estimates suggest that modest effects of 0.2 standard deviations will be detectable for most outcomes with our existing sample, and effects of 0.25 standard deviations are detectable even under high attrition rates surpassing 60%." 3ie's proposal to GiveWell and Open Philanthropy, pp. 4-5.

• 15

  See GiveWell, Education in Developing Countries, “How generalizable is this evidence?”

• 16

  See this page for more on how we think about cost-effectiveness in our grantmaking and what we mean by “x cash”.