Innovations for Poverty Action (IPA)

Print
Email
   

A note on this page's publication date

The content we created for Innovations for Poverty Action in November 2011 appears below. This content is likely to be no longer fully accurate, both with respect to what it says about Innovations for Poverty Action and with respect to what it implies about our own views and positions. With that said, we do feel that the takeaways from this examination are sufficient not to prioritize re-opening our investigation of this organization at this time.

In 2011, we declared Innovations for Poverty Action a "standout organization," which meant that we found it to be an outstanding opportunity to accomplish good but that we rated our top charities above it.

Published: November 28, 2011


Summary

IPA carries out research on aid, primarily developing-world aid, and advocates for the use of this research in decision-making.

We feel that IPA is a standout organization because:

  • We feel that IPA produces unusually high-quality research for its field and is contributing to raising standards for its field.
  • We believe that IPA has had some success in positively influencing major funders using evidence.
  • IPA is a highly transparent organization. It has publicly provided all the information that we find relevant.

Our full review, below, discusses our full assessment of IPA, including what we see as its strengths and weaknesses as well as issues we have yet to resolve.

Table of Contents

What do they do?

IPA is focused on researching aid programs. It provides implementation services for studies (e.g., carrying out surveys) as well as assistance and training for them; it also participates in discussions with donors and the public about how to use the available research to maximize the effectiveness of aid; finally, its Proven Impact Initiative and scale-up initiatives aim to expand programs they believe to be well-supported by the research and that they have a comparative advantage supporting.1

The bulk of IPA's work relates to developing-world aid. IPA categorizes its work into seven sectors: agriculture, charitable giving, education, health, microfinance and enterprise, governance and community participation and water and sanitation.2

Note that we have not investigated the Proven Impact Fund, a sub-program of IPA "designed to allocate resources to organizations implementing the ideas selected for the Proven Impact Initiative,"3 because IPA stated to us that (a) the Proven Impact Fund is too new to be well-suited to a GiveWell evaluation; (b) IPA has committed not to use unrestricted funds for the Proven Impact Fund.4

Does it work?

IPA's focus is on providing and discussing research on how to make aid more effective. Thus, our key questions for it are:

  • Are the individual studies produced and highlighted by IPA high-quality (in terms of credibility of conclusions)? We feel that IPA produces unusually high-quality research for its field and is contributing to raising standards for its field. Our view of the general quality of IPA's research is based on its emphasis of a particular type of study, the randomized controlled trial, that we find particularly credible; we have not examined enough of IPA's studies in detail to give a strong view on the quality of its research aside from this point in its favor.
  • Does IPA make well-grounded recommendations for aid? We feel that IPA's promotion of certain programs as "Proven" is not fully warranted in some cases. We believe that very strong evidence generally requires having multiple studies pointing to similar conclusions, and that IPA has not yet reached this point on most of the aid-relevant conclusions it promotes, although it does have an expressed interest in (and is making progress on) doing so.
  • Is the research produced and highlighted by IPA influential (in terms of impact on aid providers)? The case for IPA's influence is open to interpretation. Our take is that IPA has had noticeable positive influence in debates over microlending and bednet distribution, and that there has been some preliminary funding and piloting of other interventions promoted by IPA.

Note that IPA has stated that it will be sending us more information on these topics, particularly regarding its influence with policymakers and additional evidence in support of the interventions it labels as "Proven."

Is the research produced and highlighted by IPA high-quality?

This section discusses our own subjective assessment of the quality of IPA's research (as distinct from its perceived quality by others). We feel that our assessment is relevant, because

  • If IPA succeeds in influencing policy, the ultimate humanitarian value of its work depends on the extent to which its research is high-quality (not just "perceived as high-quality").
  • GiveWell's process involves reviewing and assessing a good deal of research on the impact of development programs. A substantial part of our work and value-added is gauging the quality of this research.
  • We know of no independent-of-GiveWell sources that regularly provide useful assessments of research quality.

Our take on IPA's research is as follows.

IPA's studies are at the top end of quality for studies in this area (evaluating the impact of development programs). In our experience, most research in this area suffers from methodological problems and concerns that make it hard to rely on, and a particular kind of study (the randomized controlled trial) is less prone to these problems and concerns.5 Most of the studies featured on IPA's website, and all of the studies that we discuss in more detail below, are randomized controlled trials; in addition, most of the randomized controlled trials we have seen that appear highly relevant for aid are featured on IPA's website.

However, we have not examined enough individual IPA studies to make a categorical statement that all IPA studies are at the top end of quality for studies in this area. IPA’s individual projects are led by individual researchers, so presumably there will be heterogeneity in quality of output. IPA stated to us that it does understake considerable investment in staff training, research protocols and data collection strategies to maintain high quality, but we did not evaluate those internal quality control procedures.

Does IPA make well-grounded recommendations for aid?

On another page, we discuss notable research affiliated with Innovations for Poverty Action. Overall, we feel that IPA has not produced very strong evidence (as opposed to suggestive or supplementary evidence) for any particular aid-relevant conclusion to date, and that its promotion of certain programs as "Proven" is not fully warranted in some cases. Our top health interventions are generally backed by either a large number of high-quality studies or by strong macro evidence; by contrast, no particular view disseminated by IPA appears to have more than a few studies behind it.

This latter point is a function of the fact that to date, there has been relatively little research in this area of the quality which IPA studies routinely have. This point could be taken as an argument for donating to IPA with the aim of increasing the quantity of high-quality research. As stated at our discussion of IPA's notable research, there are cases in which substantial further replication is in progress.

Is the research produced and highlighted by IPA influential?

On another page, we discuss notable research affiliated with Innovations for Poverty Action.

The case for IPA's influence is open to interpretation. Our take is that IPA has had noticeable influence in debates over school-based deworming, microlending and bednet distribution, and that there has been some preliminary funding and piloting of other interventions promoted by IPA.

Our view is that IPA's influence in the areas of deworming, microlending and bednet distribution has been positive, convincing other international actors to advocate for school-based approaches to deworming, encouraging skepticism of existing approaches and experimentation in the case of microlending and encouraging a "default to free bednets" (as opposed to selling bednets) in case of bednet distribution.

Note that the set of notable or influential studies we discuss is only a subset of all the studies IPA has carried out, i.e., not all of IPA's research ends up having noticeable influence. For a list of all IPA studies as of this writing (2-Nov-2011), see the 116 studies currently returned by a search of its publications.6

What do you get for your dollar?

We don't feel the connection between studies and social impact is clear enough to report cost-effectiveness directly in social impact terms; instead, here we discuss how much a study typically costs.

Dean Karlan, IPA's President and Founder,7 has stated to us that a typical study costs between $50,000 and $500,000 (although some can be even more expensive, when they are studying multiple interventions at once, for example). He added that cheaper studies tend to be more operational research, for example, studies on how to get people to finish their immunization series, on optimal pricing of health products, or on takeup and usage of savings products; studies which require extensive surveying tend to be more expensive. 8 These figures seem consistent with the limited information we have on the actual costs of past IPA studies (confidential) and with the information we have on future projected studies (see below).

Room for more funds?

Unrestricted funds up to 2 million dollars per year

IPA has stated to us that it seeks more unrestricted funding for purposes such as outreach (i.e., promoting its research methods and findings), training (of evaluators), and general infrastructure (which would better position it to carry out studies in countries where it does not yet have a strong presence). It gave us specific, concrete "room for more funding" analysis up to the level of $2 million per year. The USD $2 million threshold was provided based on a uniform request from GiveWell, but IPA has stated that its research initiatives would benefit from increased funding in order to replicate interventions and evaluations in multiple contexts and to expand the scope of the research questions and that these initiatives can absorb unrestricted funds of up to $50 million for research and accompanying interventions. 9

With an additional $100,000, IPA would fund:

  • Partial salary for two Deputy Country Directors to help develop new projects and expand outreach to local policymakers and organizations - $50K
  • Four to five local research dissemination events (to aid our efforts in policy dissemination, organizations partner outreach, project development, and expansion of our researcher network) - $50K

With an additional $500,000, IPA would fund all of the above activities, as well as:

  • Identification and project development for one new scale-up location for remedial education - previously estimated at $100K
  • Continued support for the development and growth of our Post-Conflict and Fragile States Research Initiative - about $100K
  • Additional financial management for global operations - $25K
  • Expanded training resources to be able to conduct more specialized trainings - $75K
  • Large-scale conference with African policy-makers on innovations in education reform and policy, to be held in Ghana - $100K

At the $1,000,000 level, IPA would still fund all of the above activities and also:

  • Explore an additional site for the expansion of our scale-up programs (could feasibly be safe water, commitment savings, deworming or remedial education, as this depends highly on our opportunities and our in-country relationships) - $100K
  • Expand the researcher outreach team to hire a coordinator who can focus efforts to diversify and grow our global network of researchers - $75K
  • Additional support for policy outreach - $75K
  • Continue to grow research initiatives and expand opportunities by adding to SME research fund and establishing a Post-Conflict research fund - $250K

Finally, at the $2,000,000 level, IPA would carry out the above and also be able to undertake two exciting larger projects:

  • Establish a replication fund to align with ongoing work to identify and communicate on studies that have been particularly promising but could benefit from further replication - $750K
  • A large-scale event similar to our October 2010 Microfinance Impact and Innovation conference. This event would, much like the first, aim to bring practitioners and researchers together to take stock of what we know in a sector and proactively engage in discussion of the evidence and how it can inform future programs and efforts - $250K

Areas of further research

The above expenses would presumably increase IPA's ability to raise funding for, and carry out, more studies. Thus, we also requested information on what areas of study it would undertake if it had its own discretion over the funds (while understanding that in many cases the studies IPA undertakes are driven by the major funders who pay for them10).

IPA provided the following possibilities. All of these are replications of projects that have previously been evaluated.11

Project Research questions What does the existing evidence say? Possible research locations Estimated budget
Examining Underinvestment in Agriculture What kinds of products can assist farmers in making more productive investments? What are the returns to fertilizer? What are the returns to other agricultural inputs, such as plow usage or labor? Are farmers risk averse, or do they lack the initial capital to invest in agricultural inputs? Building on evidence that fertilizer (when used in specified quantities) is a profitable investment for farmers in Western Kenya, an RCT study found that farmers purchased fertilizer for the next season more often when given the chance to buy it right after harvest with free delivery at the start of the next planting season. The study did not examine directly whether crop yields or profits rose.12 The authors concluded that the program was not cost effective as implemented, but the idea of selling vouchers at harvest time is promising, if done through a cheaper delivery channel.13 Mali, Uganda, Malawi $1 to 5 million
Graduating the Ultra-Poor What is the impact of the Microfinance Graduation Pilot intervention on social and economic outcomes (income, assets, school attendance of children, health and food security)? What is the viability of “graduating” the ultra poor to food security and/or microfinance? Is mandating savings necessary for ensuring financial stability among the target group? The first study of the program compared pre- and post-program data on participants and on those who did not qualify for the program due to being slightly better off. The study found positive results both soon after and a few years after the program ended. Preliminary results from two RCTs in India are publicly available but the authors have requested that they not be cited.14 The first program studied cost about $500 per participant.15 Mali, Liberia, Sierra Leone $1.5 to 3 million per site
Incentives for Vaccines What level of incentive works in increasing immunization rates? What is the impact of incentives when offered in public health facilities in addition to mobile clinics? Villages were randomly assigned to receive immunization camps, immunization camps plus rewards for bringing children to the camps, or neither. Both interventions resulted in statistically significant increases in the number of immunizations and rates of children fully immunized, with incentives having a larger effect.16 The cost of the camps with incentives was $28 per child fully immunized and $56 at camps without incentives.17 Peru, Morocco (and others) $100,000 to $1 million per site
Remedial Education What is the impact of the tutoring program on children’s test scores? If the program is effective, is it due to the remedial education, or simply because of smaller class sizes? Will reducing instructional time with the regular teacher (by pulling low performing students out the regular class) reduce learning levels? If so, would remedial classes be more effective if conducted after school hours? Can teachers achieve the same results without assistants, if trained to do so? A randomized evaluation found statistically significant improvements in tests scores among children in schools that received tutors. Improvements did not persist for schools that had received tutors in the year or two following the program. Gains for the worst-performing students were still statistically significant one year after the program, though at a considerably lower level than they were immediately following the program.18 The program cost $2.25 per student per year and $0.67 per standard deviation improvement in test scores.19 TBD $1.2 million
Safe Water Project (Chlorine dispenser system) How will the dispenser system be financed over time? How will a reliable supply of chlorine be delivered to communities? If chlorine supply is publicly and donor financed, how will funding be linked to efficient distribution? How will the dispenser hardware be installed and maintained over time?
______________________________		 A randomized evaluation looked at the combination of free chlorine dispensers at water collection points and local promoters of the dispensers. The full report on the evaluation is preliminary; a summary paper reported that the intervention led to a 37 percentage point gain in chlorine use at 3 weeks and 53 percentage point gain at 3-6 months.20 Systematic reviews have found that point-of-use chlorination reduces reported diarrhea;21 we have not seen direct evidence that point-of-collection chlorination reduces diarrhea (chlorine may provide residual protection in some cases22). The dispenser study found that the program could be "extremely cost-effective, with a cost per DALY saved that could be as low as $20 at scale."23
__________________________________		 Ethiopia, Haiti, Kenya Flexible (partial scale up possible)

Unresolved issues

  • In some cases, IPA's promotion of certain programs as "Proven" does not seem fully warranted. Our disagreement here pertains to synthesizing and interpreting the meaning of multiple studies, a topic on which reasonable people can often disagree; we agree with IPA's take in many, but not all, cases.
  • IPA's focus, research, is inherently risky and future studies could have limited influence (though they also could have substantial influence). It is difficult to predict which studies will become influential going forward, and we are unsure of the likely impact in terms of quantity of influential research (though we are clear on the use in terms of projects) of additional donations to IPA.

Sources

  • 1.

    Innovations for Poverty Action, "Our Strategy."

  • 2.

    Innovations for Poverty Action, "Sectors."

  • 3.

    Innovations for Poverty Action, "Proven Impact Fund."

  • 4.

    Dean Karlan, phone conversation with GiveWell, February 2, 2011.

  • 5.

    See our discussion of the problems of selection bias and publication bias.

  • 6.

    Innovations for Poverty Action, "Publications."

  • 7.

    Innovations for Poverty Action, "Board of Directors."

  • 8.

    Conversation with Dean Karlan, 6/22/11.

  • 9.

    Innovations for Poverty Action, email to GiveWell, July 8, 2011.

  • 10.

    "Our approach to determining the focus of our research is generally a collaborative process between researchers, implementing partners and IPA, yet ultimately dependent on funding. As such, our priorities are also somewhat tied to funding and the levels that correspond to the projects’ varying size and need (as well as the availability of other funds for the same purpose)." Innovations for Poverty Action, [[WHERE IS THIS FROM?]]

  • 11.

    Evaluations:

    • Examining Underinvestment in Agriculture: Duflo, Kremer, and Robinson 2009.
    • Graduating the Ultra-Poor: IPA notes, "IPA is conducting randomized evaluations in India, Pakistan, Honduras, Peru, Ethiopia, Yemen, and Ghana to understand the impact of this innovative model." Innovations for Poverty Action, "Ultra Poor Graduation Pilot in India."
    • Incentives for Vaccines: Banerjee et al. 2010.
    • Remedial Education: Banerjee et al. 2007.
    • Safe Water Project (Chlorine dispenser system): Ahuja, Kremer, and Zwane 2010.
  • 12. Duflo, Kremer, and Robinson 2009.
  • 13. "The SAFI program itself, with its delivery of small quantities of fertilizer to farmers by field officers, is too expensive (in terms of staff costs) to be cost effective and therefore could not be directly adopted as policy. However, preliminary results from a pilot program designed to mimic key elements of SAFI!without individual free delivery (and thus expensive visits to farms) suggest that time-limited coupons for small discounts on fertilizer could cost effectively increase take-up." Duflo, Kremer, and Robinson 2009, Pg 35.
  • 14. Papers linked from Microfinance Innovation and Impact Conference 2010, "Abstracts & Papers" under the header "Targeting the Ultra Poor."
  • 15. "This wide range of supports makes per beneficiary cost of the programme very high (about 500 USD per beneficiary)." Sulaiman and Matin 2006, Pg 3.
  • 16. Banerjee, et al. 2010.
  • 17. "The average cost to Seva Mandir of fully immunising a child was $27.94 (1102 rupees, about £16 or €19) in the reliable camp with incentives and $55.83 (2202 rupees) in the reliable camp without incentives (see appendix 2 on bmj.com for further details). The difference comes from the fact that camps had to be open from 11 am to 2 pm, regardless of the number of children present. Thus, the higher average number of children in the camps with incentives spread the daily fixed cost (mainly, the salary of the nurse and assistant) over more children." Banerjee, et al. 2010, Pg 6.
  • 18. See our review of Pratham for further details.
  • 19. "Banerjee et al. [2005] estimate the cost per standard deviation improvement of both programs under various assumptions, and compare it to other effective programs evaluated in the developing world. The Balsakhi program, at a cost of about $0.67 per standard deviation, is by far the cheapest program evaluated." Banerjee et al. 2007, Pg 20.
  • 20. Ahuja, Kremer, and Zwane 2010, Pgs 21-23.
  • 21.
    • "In all studies, outcomes were reported by the mother...Point-of-use chlorine treatment of drinking water reduced diarrhea in children in 9 of 10 studies identified, and the effect was statistically significant in 5 of the studies...The pooled effect of point-of-use chlorine treatment on child diarrhea indicates a 29% reduction in risk (relative risk, 0.71; 0.58–0.87) compared with traditional practices." Arnold and Colford 2007, Pgs 356, 361.
    • "Household chlorination was associated with a statistically significant reduction in diarrhoea." Clasen et al. 2007, Pg 5.
  • 22. "When added to water in tightly covered containers, volatilization is minimal, and chlorine disinfec- tants provide residual protection for many hours to days." Arnold and Colford 2007, Pg 354.
  • 23. Ahuja, Kremer, and Zwane 2010, Pg 23.
Clicky Web Analytics