- Top charities
The content we created for Innovations for Poverty Action in November 2011 appears below. This content is likely to be no longer fully accurate, both with respect to what it says about Innovations for Poverty Action and with respect to what it implies about our own views and positions. With that said, we do feel that the takeaways from this examination are sufficient not to prioritize re-opening our investigation of this organization at this time.
In 2011, we declared Innovations for Poverty Action a "standout organization," which meant that we found it to be an outstanding opportunity to accomplish good but that we rated our top charities above it.
Published: November 28, 2011
IPA carries out research on aid, primarily developing-world aid, and advocates for the use of this research in decision-making.
We feel that IPA is a standout organization because:
Our full review, below, discusses our full assessment of IPA, including what we see as its strengths and weaknesses as well as issues we have yet to resolve.
IPA is focused on researching aid programs. It provides implementation services for studies (e.g., carrying out surveys) as well as assistance and training for them; it also participates in discussions with donors and the public about how to use the available research to maximize the effectiveness of aid; finally, its Proven Impact Initiative and scale-up initiatives aim to expand programs they believe to be well-supported by the research and that they have a comparative advantage supporting.1
The bulk of IPA's work relates to developing-world aid. IPA categorizes its work into seven sectors: agriculture, charitable giving, education, health, microfinance and enterprise, governance and community participation and water and sanitation.2
Note that we have not investigated the Proven Impact Fund, a sub-program of IPA "designed to allocate resources to organizations implementing the ideas selected for the Proven Impact Initiative,"3 because IPA stated to us that (a) the Proven Impact Fund is too new to be well-suited to a GiveWell evaluation; (b) IPA has committed not to use unrestricted funds for the Proven Impact Fund.4
IPA's focus is on providing and discussing research on how to make aid more effective. Thus, our key questions for it are:
Note that IPA has stated that it will be sending us more information on these topics, particularly regarding its influence with policymakers and additional evidence in support of the interventions it labels as "Proven."
This section discusses our own subjective assessment of the quality of IPA's research (as distinct from its perceived quality by others). We feel that our assessment is relevant, because
Our take on IPA's research is as follows.
IPA's studies are at the top end of quality for studies in this area (evaluating the impact of development programs). In our experience, most research in this area suffers from methodological problems and concerns that make it hard to rely on, and a particular kind of study (the randomized controlled trial) is less prone to these problems and concerns.5 Most of the studies featured on IPA's website, and all of the studies that we discuss in more detail below, are randomized controlled trials; in addition, most of the randomized controlled trials we have seen that appear highly relevant for aid are featured on IPA's website.
However, we have not examined enough individual IPA studies to make a categorical statement that all IPA studies are at the top end of quality for studies in this area. IPA’s individual projects are led by individual researchers, so presumably there will be heterogeneity in quality of output. IPA stated to us that it does understake considerable investment in staff training, research protocols and data collection strategies to maintain high quality, but we did not evaluate those internal quality control procedures.
On another page, we discuss notable research affiliated with Innovations for Poverty Action. Overall, we feel that IPA has not produced very strong evidence (as opposed to suggestive or supplementary evidence) for any particular aid-relevant conclusion to date, and that its promotion of certain programs as "Proven" is not fully warranted in some cases. Our top health interventions are generally backed by either a large number of high-quality studies or by strong macro evidence; by contrast, no particular view disseminated by IPA appears to have more than a few studies behind it.
This latter point is a function of the fact that to date, there has been relatively little research in this area of the quality which IPA studies routinely have. This point could be taken as an argument for donating to IPA with the aim of increasing the quantity of high-quality research. As stated at our discussion of IPA's notable research, there are cases in which substantial further replication is in progress.
On another page, we discuss notable research affiliated with Innovations for Poverty Action.
The case for IPA's influence is open to interpretation. Our take is that IPA has had noticeable influence in debates over school-based deworming, microlending and bednet distribution, and that there has been some preliminary funding and piloting of other interventions promoted by IPA.
Our view is that IPA's influence in the areas of deworming, microlending and bednet distribution has been positive, convincing other international actors to advocate for school-based approaches to deworming, encouraging skepticism of existing approaches and experimentation in the case of microlending and encouraging a "default to free bednets" (as opposed to selling bednets) in case of bednet distribution.
Note that the set of notable or influential studies we discuss is only a subset of all the studies IPA has carried out, i.e., not all of IPA's research ends up having noticeable influence. For a list of all IPA studies as of this writing (2-Nov-2011), see the 116 studies currently returned by a search of its publications.6
We don't feel the connection between studies and social impact is clear enough to report cost-effectiveness directly in social impact terms; instead, here we discuss how much a study typically costs.
Dean Karlan, IPA's President and Founder,7 has stated to us that a typical study costs between $50,000 and $500,000 (although some can be even more expensive, when they are studying multiple interventions at once, for example). He added that cheaper studies tend to be more operational research, for example, studies on how to get people to finish their immunization series, on optimal pricing of health products, or on takeup and usage of savings products; studies which require extensive surveying tend to be more expensive. 8 These figures seem consistent with the limited information we have on the actual costs of past IPA studies (confidential) and with the information we have on future projected studies (see below).
IPA has stated to us that it seeks more unrestricted funding for purposes such as outreach (i.e., promoting its research methods and findings), training (of evaluators), and general infrastructure (which would better position it to carry out studies in countries where it does not yet have a strong presence). It gave us specific, concrete "room for more funding" analysis up to the level of $2 million per year. The USD $2 million threshold was provided based on a uniform request from GiveWell, but IPA has stated that its research initiatives would benefit from increased funding in order to replicate interventions and evaluations in multiple contexts and to expand the scope of the research questions and that these initiatives can absorb unrestricted funds of up to $50 million for research and accompanying interventions. 9
With an additional $100,000, IPA would fund:
With an additional $500,000, IPA would fund all of the above activities, as well as:
At the $1,000,000 level, IPA would still fund all of the above activities and also:
Finally, at the $2,000,000 level, IPA would carry out the above and also be able to undertake two exciting larger projects:
The above expenses would presumably increase IPA's ability to raise funding for, and carry out, more studies. Thus, we also requested information on what areas of study it would undertake if it had its own discretion over the funds (while understanding that in many cases the studies IPA undertakes are driven by the major funders who pay for them10).
IPA provided the following possibilities. All of these are replications of projects that have previously been evaluated.11
|Project||Research questions||What does the existing evidence say?||Possible research locations||Estimated budget|
|Examining Underinvestment in Agriculture||What kinds of products can assist farmers in making more productive investments? What are the returns to fertilizer? What are the returns to other agricultural inputs, such as plow usage or labor? Are farmers risk averse, or do they lack the initial capital to invest in agricultural inputs?||Building on evidence that fertilizer (when used in specified quantities) is a profitable investment for farmers in Western Kenya, an RCT study found that farmers purchased fertilizer for the next season more often when given the chance to buy it right after harvest with free delivery at the start of the next planting season. The study did not examine directly whether crop yields or profits rose.12 The authors concluded that the program was not cost effective as implemented, but the idea of selling vouchers at harvest time is promising, if done through a cheaper delivery channel.13||Mali, Uganda, Malawi||$1 to 5 million|
|Graduating the Ultra-Poor||What is the impact of the Microfinance Graduation Pilot intervention on social and economic outcomes (income, assets, school attendance of children, health and food security)? What is the viability of “graduating” the ultra poor to food security and/or microfinance? Is mandating savings necessary for ensuring financial stability among the target group?||The first study of the program compared pre- and post-program data on participants and on those who did not qualify for the program due to being slightly better off. The study found positive results both soon after and a few years after the program ended. Preliminary results from two RCTs in India are publicly available but the authors have requested that they not be cited.14 The first program studied cost about $500 per participant.15||Mali, Liberia, Sierra Leone||$1.5 to 3 million per site|
|Incentives for Vaccines||What level of incentive works in increasing immunization rates? What is the impact of incentives when offered in public health facilities in addition to mobile clinics?||Villages were randomly assigned to receive immunization camps, immunization camps plus rewards for bringing children to the camps, or neither. Both interventions resulted in statistically significant increases in the number of immunizations and rates of children fully immunized, with incentives having a larger effect.16 The cost of the camps with incentives was $28 per child fully immunized and $56 at camps without incentives.17||Peru, Morocco (and others)||$100,000 to $1 million per site|
|Remedial Education||What is the impact of the tutoring program on children’s test scores? If the program is effective, is it due to the remedial education, or simply because of smaller class sizes? Will reducing instructional time with the regular teacher (by pulling low performing students out the regular class) reduce learning levels? If so, would remedial classes be more effective if conducted after school hours? Can teachers achieve the same results without assistants, if trained to do so?||A randomized evaluation found statistically significant improvements in tests scores among children in schools that received tutors. Improvements did not persist for schools that had received tutors in the year or two following the program. Gains for the worst-performing students were still statistically significant one year after the program, though at a considerably lower level than they were immediately following the program.18 The program cost $2.25 per student per year and $0.67 per standard deviation improvement in test scores.19||TBD||$1.2 million|
|Safe Water Project (Chlorine dispenser system)||How will the dispenser system be financed over time? How will a reliable supply of chlorine be delivered to communities? If chlorine supply is publicly and donor financed, how will funding be linked to efficient distribution? How will the dispenser hardware be installed and maintained over time?
|A randomized evaluation looked at the combination of free chlorine dispensers at water collection points and local promoters of the dispensers. The full report on the evaluation is preliminary; a summary paper reported that the intervention led to a 37 percentage point gain in chlorine use at 3 weeks and 53 percentage point gain at 3-6 months.20 Systematic reviews have found that point-of-use chlorination reduces reported diarrhea;21 we have not seen direct evidence that point-of-collection chlorination reduces diarrhea (chlorine may provide residual protection in some cases22). The dispenser study found that the program could be "extremely cost-effective, with a cost per DALY saved that could be as low as $20 at scale."23
|Ethiopia, Haiti, Kenya||Flexible (partial scale up possible)|
Innovations for Poverty Action, "Our Strategy."
Innovations for Poverty Action, "Sectors."
Innovations for Poverty Action, "Proven Impact Fund."
Dean Karlan, phone conversation with GiveWell, February 2, 2011.
Innovations for Poverty Action, "Publications."
Innovations for Poverty Action, "Board of Directors."
Conversation with Dean Karlan, 6/22/11.
Innovations for Poverty Action, email to GiveWell, July 8, 2011.
"Our approach to determining the focus of our research is generally a collaborative process between researchers, implementing partners and IPA, yet ultimately dependent on funding. As such, our priorities are also somewhat tied to funding and the levels that correspond to the projects’ varying size and need (as well as the availability of other funds for the same purpose)." Innovations for Poverty Action, [[WHERE IS THIS FROM?]]