Available impact studies for economic empowerment programs

This page provides an overview of the impact studies we received as a part of our 2009 grant process for economic empowerment programs in sub-Saharan Africa.

Table of Contents

Summary

  • Only one organization submitted a compelling impact evaluation. By "compelling," we mean a study that not only observes a positive change in clients' lives, but carefully considers, discusses, and gives strong reasons to doubt "alternate hypotheses" for such a change. (Most randomized controlled trials would qualify for this subjective criterion, but there could also be studies meeting this criterion without being randomized controlled trials.)
  • 17 of 49 organizations that applied for our grant submitted an impact study. Of these, we found 15 to have serious flaws that are noted below. One organization (Heifer International) submitted a study that remains confidential, so we cannot offer our thoughts on it. One organization, BRAC, submitted what we consider a compelling impact study.
  • We would guess that many of the 108 organizations who did not apply did not do so because they had no impact study available. As we discuss at our process page for this grant, (a) we were straightforward about our criteria for this grant and (b) of the 40 organizations that gave a reason for declining the application, 10 specifically cited a lack of technical reports.

Common methodological flaws

Below, we list common methodological flaws in the studies we reviewed.

The table below notes the flaws we felt were present in each organization's submission.

  • "Self-reported (retrospective)" refers to studies that rely on participants' reported well-being and reported improvement over time to measure a program's success. In some cases, participants were only asked about their attitude as opposed to objective measures of standard of living. We have a variety of concerns about these studies. Participants may feel pressure to tell interviewers what they want to hear; they may incorrectly recall their past conditions, changes over time, and the reasons for those changes. While we feel this sort of data could be part of a compelling study, none of the studies with this flaw listed spoke to these concerns.
  • Lack of control group. Some studies point to observed improvements in the lives of program participants, but did not include a "control group" (non-participants) or otherwise address the question of whether these improvements were due to the program or due to other factors.
  • High attrition. Some studies began with many participants but only ultimately examined the changes in outcomes for a subset of those that started in the program. In these cases, we are concerned that those individuals that completed the program and the evaluation might not be representative of all enrolled participants (and may systematically tend to have seen more favorable outcomes, thus creating a biased study result).
  • Representativeness. Some studies were not clear as to how participants were selected. In these cases, we are concerned that the group evaluated may have been systematically different than the group surveyed, biasing the study's result.
  • No baseline. Studies that reported conditions post-program but didn't offer a baseline (pre-program) assessment, and thus did not make it possible to assess what change had occurred in participants' lives.
  • Measured non-impact outcome. Studies that pointed to an outcome that does not directly indicate a change in an individual's life. For example, if a study points to knowledge of health behaviors, we didn't consider this to demonstrate impact, unless the link between knowledge of health behaviors and improved outcomes (in the case of this cause, improved standards of living) is separately established.

None of these flaws need be fatal in itself, but in the table below they are used as shorthand for major concerns raised by the studies listed. We have made the full studies available as well (with the exception of Heifer International's, which is confidential).

Organization details

The table below provides the full list of organizations that applied, whether they submitted an impact evaluation or none at all, and what led us to conclude that a particular evaluation was not compelling.

(The charity names in the table below provide links to our review pages for each organization; the "Reason not compelling" column references terms explained above.)

Organization What do they do? Submitted an impact study? Reason not compelling
Village Enterprise Fund Business grants and training Yes High attrition
Small Enterprise Foundation Microcredit No -
BRAC Microfinance, education, health, and asset transfer Yes This was the strongest study we reviewed
Microloan Foundation Microcredit No -
Initiative Development-Ghana Microcredit No -
Women for Women International Cash grants, emotional support, and training Yes Self-reported
Heifer Project International Livestock grants Yes Confidential
ACCION International Technical assistance and financing for microfinance banks No -
Grameen Foundation Technical assitance to microfinance banks No -
Opportunity International Technical assistance and financing for microfinance banks No -
Women's World Banking Technical assitance to microfinance banks No -
COMACO Trading centers and subsidized prices for farmers No -
Med-Net Microfinance No -
MicroDreams Microfinance No -
Vipani Agricultural inputs and training Yes Lack of control group
Aid To Artisans Training and marketing for artisans Yes Self-reported
Ashoka Grants for social entreprenuers Yes Lack of control group
Association for Progressive Communications Promote internet access for underserved communities Yes Retrospective, Self-reported, Lack of control group
Business Council for Peace Technical assistance for women entrepreneurs Yes Retrospective, Self-reported, Lack of control group
Endeavor Technical assistance for developing world entrepreuers Yes Lack of control group
Global Partners For Development Livestock training, vocational training, clinics, microcredit, school construction and scholarships, medical supplies, water infrastructure, food aid, etc. Yes Self-reported
International Development Enterprises (IDE) Sell simple agricultural technologies Yes Retrospective, Self-reported, Lack of control group
Mennonite Economic Development Associates (MEDA) Technical assitance to microfinance banks, financial assistance for investors investing in the developing world companies Yes Self-reported, Lack of control group
Millennium Promise Agricultural extension, education, health facilities, and infrastructure for selected villages Yes Lack of control group, Data mining
Oxfam Agriculture, disaster relief, health, education, policy advocacy, etc. Yes Lack of control group
Project Concern Healthcare, childcare, economic empowerment, human trafficking, disaster relief, sanitation education, food aid, etc. Yes No baseline, Measured non-impact outcome, Lack of control group, Representativeness
World Neighbors Agriculture, health, water, etc. Yes Lack of control group
Acumen Fund Investments in social service organizations No -
African Enterprise Business and leadership training No -
American Jewish World Service Grants to developing-world organizations No -
AMURT Schools, disaster relief, training for healthcare providers, water infrastructure, primary health care, agricultural extention, homeless shelter, food aid No -
CHF International Water infrastructure, microfinance, HIV/AIDS treatment and care, SME finance, fuel-efficient stoves, emergency relief, community development, vocational training, agriculture, etc. No -
Concern Worldwide Education, disaster relief, health, natural resource advocacy, agriculture, market access, etc. No -
EndPoverty Technical assistance and financing for microfinance banks No -
Global Fairness Initiative Labor relations, advocacy, technical assistance for women producers No -
International Center for Research on Women (ICRW) Research and advocacy No -
Missions of Hope Education, food aid, business training and loans, social work No -
Nuru Health education, agriculture loans, water infrastructure, bed net sales, teacher training, financial education No -
Outreach International Water infrastructure, food aid, school construction, housing assistance, agricultural extension and loans, childcare, infrastructure repair, literacy, etc. No -
Peer Servants Technical assistance and financing for microfinance banks No -
Prince Youth Business International Technical assistance for developing world entrepreuers No -
Shared Interest Loan guarantees for community development banks in South Africa No -
VisionSpring (AKA Scojo Foundation) Training and supplies for eyeglass sellers No -
Winrock International Environmental advocacy and research, education, agricultural extension, etc. No -
World Hope International/LEAP Africa HIV/AIDS, education, agriculture, microfinance, anti-trafficking, etc. No -