This page provides an overview of the impact studies we received as a part of our 2009 grant process for economic empowerment programs in sub-Saharan Africa.
- Only one organization submitted a compelling impact evaluation. By "compelling," we mean a study that not only observes a positive change in clients' lives, but carefully considers, discusses, and gives strong reasons to doubt "alternate hypotheses" for such a change. (Most randomized controlled trials would qualify for this subjective criterion, but there could also be studies meeting this criterion without being randomized controlled trials.)
- 17 of 49 organizations that applied for our grant submitted an impact study. Of these, we found 15 to have serious flaws that are noted below. One organization (Heifer International) submitted a study that remains confidential, so we cannot offer our thoughts on it. One organization, BRAC, submitted what we consider a compelling impact study.
- We would guess that many of the 108 organizations who did not apply did not do so because they had no impact study available. As we discuss at our process page for this grant, (a) we were straightforward about our criteria for this grant and (b) of the 40 organizations that gave a reason for declining the application, 10 specifically cited a lack of technical reports.
Common methodological flaws
Below, we list common methodological flaws in the studies we reviewed.
The table below notes the flaws we felt were present in each organization's submission.
- "Self-reported (retrospective)" refers to studies that rely on participants' reported well-being and reported improvement over time to measure a program's success. In some cases, participants were only asked about their attitude as opposed to objective measures of standard of living. We have a variety of concerns about these studies. Participants may feel pressure to tell interviewers what they want to hear; they may incorrectly recall their past conditions, changes over time, and the reasons for those changes. While we feel this sort of data could be part of a compelling study, none of the studies with this flaw listed spoke to these concerns.
- Lack of control group. Some studies point to observed improvements in the lives of program participants, but did not include a "control group" (non-participants) or otherwise address the question of whether these improvements were due to the program or due to other factors.
- High attrition. Some studies began with many participants but only ultimately examined the changes in outcomes for a subset of those that started in the program. In these cases, we are concerned that those individuals that completed the program and the evaluation might not be representative of all enrolled participants (and may systematically tend to have seen more favorable outcomes, thus creating a biased study result).
- Representativeness. Some studies were not clear as to how participants were selected. In these cases, we are concerned that the group evaluated may have been systematically different than the group surveyed, biasing the study's result.
- No baseline. Studies that reported conditions post-program but didn't offer a baseline (pre-program) assessment, and thus did not make it possible to assess what change had occurred in participants' lives.
- Measured non-impact outcome. Studies that pointed to an outcome that does not directly indicate a change in an individual's life. For example, if a study points to knowledge of health behaviors, we didn't consider this to demonstrate impact, unless the link between knowledge of health behaviors and improved outcomes (in the case of this cause, improved standards of living) is separately established.
None of these flaws need be fatal in itself, but in the table below they are used as shorthand for major concerns raised by the studies listed. We have made the full studies available as well (with the exception of Heifer International's, which is confidential).
The table below provides the full list of organizations that applied, whether they submitted an impact evaluation or none at all, and what led us to conclude that a particular evaluation was not compelling.
(The charity names in the table below provide links to our review pages for each organization; the "Reason not compelling" column references terms explained above.)
|Organization||What do they do?||Submitted an impact study?||Reason not compelling|
|Village Enterprise Fund||Business grants and training||Yes||High attrition|
|Small Enterprise Foundation||Microcredit||No||-|
|BRAC||Microfinance, education, health, and asset transfer||Yes||This was the strongest study we reviewed|
|Women for Women International||Cash grants, emotional support, and training||Yes||Self-reported|
|Heifer Project International||Livestock grants||Yes||Confidential|
|ACCION International||Technical assistance and financing for microfinance banks||No||-|
|Grameen Foundation||Technical assitance to microfinance banks||No||-|
|Opportunity International||Technical assistance and financing for microfinance banks||No||-|
|Women's World Banking||Technical assitance to microfinance banks||No||-|
|COMACO||Trading centers and subsidized prices for farmers||No||-|
|Vipani||Agricultural inputs and training||Yes||Lack of control group|
|Aid To Artisans||Training and marketing for artisans||Yes||Self-reported|
|Ashoka||Grants for social entreprenuers||Yes||Lack of control group|
|Association for Progressive Communications||Promote internet access for underserved communities||Yes||Retrospective, Self-reported, Lack of control group|
|Business Council for Peace||Technical assistance for women entrepreneurs||Yes||Retrospective, Self-reported, Lack of control group|
|Endeavor||Technical assistance for developing world entrepreuers||Yes||Lack of control group|
|Global Partners For Development||Livestock training, vocational training, clinics, microcredit, school construction and scholarships, medical supplies, water infrastructure, food aid, etc.||Yes||Self-reported|
|International Development Enterprises (IDE)||Sell simple agricultural technologies||Yes||Retrospective, Self-reported, Lack of control group|
|Mennonite Economic Development Associates (MEDA)||Technical assitance to microfinance banks, financial assistance for investors investing in the developing world companies||Yes||Self-reported, Lack of control group|
|Millennium Promise||Agricultural extension, education, health facilities, and infrastructure for selected villages||Yes||Lack of control group, Data mining|
|Oxfam||Agriculture, disaster relief, health, education, policy advocacy, etc.||Yes||Lack of control group|
|Project Concern||Healthcare, childcare, economic empowerment, human trafficking, disaster relief, sanitation education, food aid, etc.||Yes||No baseline, Measured non-impact outcome, Lack of control group, Representativeness|
|World Neighbors||Agriculture, health, water, etc.||Yes||Lack of control group|
|Acumen Fund||Investments in social service organizations||No||-|
|African Enterprise||Business and leadership training||No||-|
|American Jewish World Service||Grants to developing-world organizations||No||-|
|AMURT||Schools, disaster relief, training for healthcare providers, water infrastructure, primary health care, agricultural extention, homeless shelter, food aid||No||-|
|CHF International||Water infrastructure, microfinance, HIV/AIDS treatment and care, SME finance, fuel-efficient stoves, emergency relief, community development, vocational training, agriculture, etc.||No||-|
|Concern Worldwide||Education, disaster relief, health, natural resource advocacy, agriculture, market access, etc.||No||-|
|EndPoverty||Technical assistance and financing for microfinance banks||No||-|
|Global Fairness Initiative||Labor relations, advocacy, technical assistance for women producers||No||-|
|International Center for Research on Women (ICRW)||Research and advocacy||No||-|
|Missions of Hope||Education, food aid, business training and loans, social work||No||-|
|Nuru||Health education, agriculture loans, water infrastructure, bed net sales, teacher training, financial education||No||-|
|Outreach International||Water infrastructure, food aid, school construction, housing assistance, agricultural extension and loans, childcare, infrastructure repair, literacy, etc.||No||-|
|Peer Servants||Technical assistance and financing for microfinance banks||No||-|
|Prince Youth Business International||Technical assistance for developing world entrepreuers||No||-|
|Shared Interest||Loan guarantees for community development banks in South Africa||No||-|
|VisionSpring (AKA Scojo Foundation)||Training and supplies for eyeglass sellers||No||-|
|Winrock International||Environmental advocacy and research, education, agricultural extension, etc.||No||-|
|World Hope International/LEAP Africa||HIV/AIDS, education, agriculture, microfinance, anti-trafficking, etc.||No||-|