Freedom from Hunger

Published: May 2011

Table of Contents

Summary

Freedom from Hunger develops and tests programs that add supplementary services to microfinance and trains microfinance institutions to implement them. Freedom from Hunger's programs include providing business education, health education, and savings services. We think this is a valuable area to focus on. In our analysis of microfinance, we have noted that many microfinance institutions target financial rather than social results, and we have sought out microfinance institutions that focus on improving the lives of the people they serve. Therefore, we believe that Freedom from Hunger is trying to meet an important need.

In addition, Freedom from Hunger has a commendable focus on evaluation that is extremely rare among charities. It has been involved in highly rigorous evaluations of its programs, and it publishes numerous technical reports about its work on its websites.

Our review consisted of reviewing these rigorous evaluations and speaking with Freedom from Hunger's president, Chris Dunford. Freedom from Hunger does not currently qualify for our highest ratings because:

  • The rigorous evaluations of Freedom from Hunger's programs found some effects on participants' knowledge, but found limited or no effects on business revenues and limited effects on health behaviors (more).
  • Freedom from Hunger relies on other organizations to implement its programs. It is concerned with the question of whether partners implement its programs well, but, due to the nature of its model (in which more people are reached by not running programs itself), it has limited ability to ensure that programs are consistently implemented well (more).

What do they do?

Freedom from Hunger describes its strategy as (1) testing innovations to deal with hunger and (2) distributing tested innovations through other organizations.1 In practice, it trains a staff member of a microfinance organization to train other staff members to implement one of the following programs:2

  • Credit with Education: working with microfinance institutions to provide learning sessions at microcredit meetings.3
  • Saving for Change: setting up groups of women who collect savings from and administer small loans with interest to members.4
  • Microfinance and Health Protection: improving access to healthcare among microfinance clients.5
  • Advancing Integrated Microfinance for Youth: a new program offering savings and credit services and financial education to young people in Ecuador and Mali.6

Freedom from Hunger told us that some costs are paid for by partner organizations and that the amount each contributes is determined by whether the model has been previously tested and whether Freedom from Hunger has funding available.7

Review

In considering Freedom from Hunger's impact, we asked two main questions:

  • Do the programs that Freedom from Hunger teaches to partner organizations improve clients' lives when implemented well?
  • Are programs consistently implemented well?

As discussed below, there is some limited evidence that programs work when implemented well, but no evidence that they are consistently implemented well.

Do the programs work?

We reviewed all the relevant studies for Freedom from Hunger's programs.

In summary, the rigorous evaluations of Freedom from Hunger's programs found some effects on knowledge, but found limited or no effects on business revenues and limited effects on health behaviors.

A more detailed overview of the evidence from the most rigorous studies we found is available below. More detail on each study is available in the footnotes.

Summary of research: Credit with Education
Country Quality of study design Bottom line from abstract/summary Outcomes showing statistically significant improvement Outcomes not showing statistically significant effect Outcomes showing statistically significant deterioration
Peru8 Rigorous (randomized controlled trial) "We find little or no evidence of changes in key outcomes such as business revenue, profits, or employment. We nevertheless observed business knowledge improvements and increased client retention rates for the microfinance institution."9 8 indicators (2 of business results, 4 of business practices, 2 of institutional results) 32 indicators (6 indicators of business results, 10 indicators of business practice, 13 indicators of household outcomes, 3 indicators of institutional outcomes) None
Dominican Republic10 Rigorous (randomized controlled trial) "We find no significant effect from a standard, fundamentals-based accounting training [designed by Freedom from Hunger]. However, a simplified, rule-of-thumb training produced significant and economically meaningful improvements in business practices and outcomes."11 None 10 indicators of business practices and sales None
India12 Somewhat rigorous (randomized controlled trial with imperfect randomization) "Participation resulted mainly in improved confidence levels of the daughters (and mothers) regarding their money management. Statistically significant improvements in savings levels and effective bargaining were not detected in the quantitative studies...The only [health] topic with significant gains in the randomized controlled trial evaluation was HIV/AIDS. All other topics, such as hand-washing, diarrhea, nutrition and reproductive health, saw few significant differences...[Gains] were seen in the girls’ comfort levels in discussing the topics with their family members."13 Unmarried: 1 indicator of financial literacy, 6 indicators of health literacy; Married: 2 indicators of financial literacy Unmarried: 14 indicators of financial literacy, 44 indicators of health literacy; Married: 14 indicators of financial literacy, 48 indicators of health literacy Unmarried:1 indicator of financial literacy; Married: 2 indicators of financial literacy
Ghana14 Results not reported on intent-to-treat basis - - - -
Bolivia15 Results not reported on intent-to-treat basis -
__________________________________		 - -
____________________		 -

Summary of research: Health programs
Country Quality of study design Bottom line from abstract Outcomes showing statistically significant improvement Outcomes not showing a statistically significant effect Outcomes showing statistically significant deterioration
Ghana16 Rigorous (randomized controlled trial) "The malaria education complemented the other activities to increase knowledge and positive behaviors. Yet, even the increased knowledge and behaviors often were impeded by gaps in a family’s ability to access promoted prevention methods such as ITNs."17 1 indicator of net ownership; 4 indicators of net use by vulnerable group; Net re-treatment 3 indicators of net ownership None
Peru18 Rigorous (randomized controlled trial) "Individuals in the IMCI treatment arm demonstrated more knowledge about a variety of issues related to child health, but there were no changes in anthropometric measures or reported child health status."19 Only knowledge indicators All directly measured and reported child health indicators None
Benin20 Somewhat rigorous (randomized controlled trial with imperfect randomization) "Results revealed that that the education villages perform somewhat better than the credit-only villages in malaria knowledge indicators [and] also have somewhat better malaria behaviors...Education villages were substantially more likely than credit-only villages to perform better on HIV knowledge indicators...There were no significant differences...when assessing knowledge and behavior change as a result of the childhood illnesses module."21
__________________________________		 7 health indicators, 3 credit and finance indicators 74 health indicators; all food security, social network, and decision making indicators; 39 credit and finance indicators
____________________		 1 indicator

Saving for Change

We have a positive view of programs aiming to increase savings in the developing world. For more, see our blog posts:

Freedom from Hunger is testing its Saving for Change program in a "large-scale randomized control trial conducted by Innovations for Poverty Action (IPA), comparing 500 treatment and control villages, as well as 24 months of financial diaries (high-frequency surveys) conducted with a subset of those participating in the RCT." The baseline report on the trial indicates that project results will be completed in 2012.22

Advancing Integrated Microfinance for Youth

This is a new program, so evaluations are not yet available.23

Are programs consistently implemented well?

Freedom from Hunger's model centers on testing programs and training others to implement them. This is a strategy that allows Freedom from Hunger to reach a large number of people, with the trade-off that it has less control over how programs are implemented. Freedom from Hunger appears to be concerned with the question of whether programs are implemented well by partner organizations, but its ability to ensure high quality is limited by the nature of its model.24

As part of its efforts to monitor and improve program quality, Freedom from Hunger told us it does these things to follow up with the partner organizations it trains:

  • Freedom from Hunger publishes a status report on 38 of the approximately 120 organizations that it has trained to provide Credit with Education.25 This report includes number of clients receiving education, the size and quality of the organizations' loan portfolios, and the type of education. Data is reported by the partner organization.26 In addition, Freedom from Hunger shared with us a report on the numbers of people reached by each organization that is implementing the Microfinance and Health Protection program27 and a report on the number of members and total savings by members in Saving for Change groups.28 Freedom from Hunger told us that it is able to identify some implementation problems by observing how these numbers change over time and by asking the implementing organization about the changes.29
  • Freedom from Hunger told us that its staff stay in contact with partner organizations through phone, email, and occasional site visits.30
  • Freedom from Hunger has begun to collect "impact stories" from clients of the microfinance institutions it works with, with the goal of systematically collecting pre-program and three-year follow up narratives from a random sample of clients.31 So far, Freedom from Hunger has collected stories from randomly selected clients at nine microfinance organizations, and has not yet interviewed clients for a second time.32

It is unclear to us (a) whether these processes have enabled Freedom from Hunger to identify and respond to problems; and (b) whether past programs have been implemented well.

We asked president Chris Dunford about how Freedom from Hunger approaches the potential problems associated with relying on other organizations to implement its programs. He responded that Freedom from Hunger does as much as it can upfront to train partners and to deal with potential issues.33

To its credit, Freedom from Hunger readily acknowledges that monitoring is a key challenge and notes that, even in RCTs, ensuring proper implementation is difficult:34

In an RCT, we have to pay a lot of attention to proper implementation. You want to evaluate if it can be done right. If you aren’t paying a lot of attention, you might have a lot of implementation problems, and the RCT ends up proving that implementation was poor and that is all, which is a total waste of time and money. So the partner staff gets a LOT of help and guidance and general attention from us, much more than if they were not participating in an RCT. But the partners chosen for an RCT are not atypical partners.
...
We are concerned about the quality of implementation. It’s a high-leverage strategy, because we train people to do what we know how to do. But we have no lasting control over the MFIs and other partners. We seek to deal with this by providing high-quality training up front and follow-up technical assistance as long as funding allows.

Sources

  • 1

    "We have a dual-track strategy focused on global hunger:
    1) Evidence-based innovation on how to deal with hunger and
    2) Distribution of innovations through others to serve the hungry poor."
    Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 2

    "GiveWell: Our understanding of your model is that you develop training programs related to microfinance and train the trainers who in turn train microfinance institution (MFI) staff to implement the program. Is this correct?
    Freedom from Hunger: Our staff spends a lot of time training and in preparation for training. But the training is because we have a dual-track strategy focused on global hunger:
    1) Evidence-based innovation on how to deal with hunger and
    2) Distribution of innovations through others to serve the hungry poor.
    First we have to show that the innovation works, which means careful research, including randomized controlled trials (RCTs). Distribution means we need to be trainers. The idea is similar to a market test followed by global roll out....
    We have four basic models right now:
    1. Credit with Education. The basic package is widespread. We did the first RCT in microfinance, on Credit with Education programs in Ghana and Bolivia in the mid-1990s. It didn’t get as much attention as the more recent studies because the results were never published in peer-reviewed journals. We effectively showed the positive value of Credit with Education, and now we are at the stage in which organizations come to us and say they want to implement the program. Every new partner requires a new customization process. We are still developing education modules on new topics for dissemination through Credit with Education, and we want to roll them out to as many organizations and people as possible.
    2. Saving for Change. This program started in 2005 in Mali as a joint venture with Oxfam America and the Strømme Foundation of Norway. It is a savings-led approach to microfinance. It serves areas or people who are generally beyond the reach of MFIs. There is an RCT going on in Mali, funded by the Gates Foundation and conducted by Innovations for Poverty Action. We’ve demonstrated that it can be practical for implementation, but because the impact evaluation is incomplete, it’s one stage behind Credit with Education.
    3. Microfinance and Health Protection (MAHP). We work with MFIs to introduce programs to increase access to health services and products. We’ve done this in five different locations on four continents with five different MFIs and have completed a lot of research on this; this time we’re publishing our results through peer-review journals. Now we’re in the stage of starting to roll it out. There are still uncertainties about whether all MAHP services can be offered through Saving for Change programs; this is a new frontier for innovation.
    4. AIM Youth. This is a new innovation program to extend Saving for Change and Credit with Education to youth (13–25 years old). As a still untested innovation, we remain appropriately skeptical but optimistic about its prospects to work well in a variety of circumstances. The MasterCard Foundation has given us a major contract to fund the development and testing of AIM Youth."
    Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 3

    "At regular meetings, the women's group gathers to make repayments and deposit their savings. The women also participate in a lively and joyful learning session led by a local staff person who speaks their language and knows their culture and customs." Freedom from Hunger, "Credit with Education."

  • 4

    "Saving for Change enables groups of women to deposit savings-often starting with weekly deposits of only 20 cents-and build lump sums for predictable needs. When savings accumulate, the women in the group act as their own bankers, approving small loans to each other from their pooled savings. The interest they charge themselves for the loans goes back into the pool of savings, yielding a healthy return on the deposited savings of each member of the group." Freedom from Hunger, "Saving for Change."

  • 5

    "MAHP complements this education by enabling microfinance institutions to offer financial products and other services that improve access to actual healthcare services and medicines." Freedom from Hunger, "Microfinance and Health Protection."

  • 6

    Freedom from Hunger, "Advancing Integrated Microfinance for Youth."

  • 7

    "GiveWell: What exactly does Freedom from Hunger pay for and what do its partners pay for?
    Freedom from Hunger: It depends on the status of the innovation. If we test an unproven program, then, because of the risk, we will pay for everything. Research and development of new innovations is usually funded through grants from institutional donors. Once we’ve developed a program, but it’s still not totally proven, then there’s more cost-sharing. We’ll cover our own costs but our partners pay for their costs. A third scenario is one in which an institution will come to us with money to cover our costs to do training, or we help them to fundraise to cover our costs. "
    Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 8Peru business-training study: Freedom from Hunger's Credit with Education program was evaluated in a randomized controlled trial in Peru. Results are reported in Karlan and Valdivia 2009, Pgs 33-36, Tables 1-4. Note that statistical significance is not reported. We calculated this with a two-tailed t-test at the 95% confidence level with both the unadjusted difference-in-difference effect (column 7) and the adjusted "OLS with covariates" effect (column 9) and designated an indicator statistically significant if it was significant in either measure of effect size. We did this in order to minimize the chance of unreasonably rejecting the hypothesis that the program had an effect. For more detail, see GiveWell, "Karlan and Valdivia 2009 Analysis." Karlan and Valdivia (2009, Pg 15) note, "Since treatment was assigned randomly, the insertion of these covariates would not affect the consistency of the parameter of interest. Rather, its inclusion is used to improve estimation precision, to account for chance differences between groups in the distribution of pre-random assignment characteristics, and to account for non-random attrition in the follow-up survey."
  • 9Karlan and Valdivia 2011, abstract.
  • 10

    Dominican Republic study: A randomized controlled trial of a program implementing Freedom from Hunger financial education modules was conducted in 2006-2008. Drexler, Fischer, and Schoar (2010, Pg 9) write, " The materials and capacitator training program for the Accounting treatment were based on the financial education program designed by Freedom from Hunger, a US-based non- profit organization, and the Citigroup Foundation5 and adapted to local conditions. The ADOPEM training program is most closely related to the budgeting module of the FFH training program." Results are reported in Drexler, Fischer, and Schoar 2010, Pg 24, Table 2.

  • 11

    Drexler, Fischer, and Schoar 2010, Pg 1.

  • 12India study: A randomized controlled trial was conducted on a Credit with Education program for mothers and daughters in West Bengal India. It appears that the ultimate "treatment" and "comparison" groups were not truly randomized. Gray and Chanani (2010) write, "After the completion of the follow-up quantitative work, we discovered that actual participation rates were much lower than we had anticipated...Consequently, to detect actual changes in a small and fluctuating group of participants, we first had to find a way to identify who in the study population was intended to receive the education, who actually participated in the Financial Games, who actually participated in the Health Games, and who was considered the control group. The analysis was conducted in this manner because the basic comparison of control versus treatment was unable to detect any changes due to the fact that so few of the full 'treatment' population actually participated. Thus, we broke the analysis into the following groups:
    1. Intent-to-Treat. Those mothers and girls who were randomly assigned to participate in the Learning Games and who participated in the introductory Game [emphasis added]. They may or may not have participated in any subsequent Games.
    2. Attended Health Games. Those mothers and girls who were a subset of the Intent-to-Treat group who indicated they participated in the HIV/AIDS Game and were likely to have participated in all other Health Games. Although the HIV/AIDS session was the final session and it had the lowest participation of all Health Games, we felt this would create the most common denominator to represent all Health Games. If they participated in the HIV/AIDS Game, they were likely to have participated in the other, less-threatening Games.
    3. Attended Financial Games. Those mothers and girls who were a subset of the Intent-to-Treat group who indicated they participated in the first Savings Game and were likely to have participated in subsequent Games. Unlike the Health Games, there was consistent participation across the Financial Games.
    4. Control. The full population of mothers and girls who were randomly assigned to NOT receive the full curriculum, but who participated in the first introductory session like the intent-to-treat group."

    Results for "intent-to-treat" and "control" are reported in See Gray and Chanani 2010, Pg 50, Table 3, Pg 53-55, Table 6, Pg 60, Table 10, and Pg 63-65, Table 13.

  • 13Gray and Chananai 2010, Pgs i-ii.
  • 14Ghana and Bolivia studies: Although study communities were (partially) randomly assigned (see MkNelly and Dunford 1998, Pgs 10-11 and MkNelly and Dunford 1999, Pgs 13-14), to receive the program or not, results are given separately for each "participants," "non-participants in program communities," and control communities (see MkNelly and Dunford 1998, Pg 11 and MkNelly and Dunford 1999, Pg 15), with one exception: on MkNelly and Dunford 1999, Pg 77, children's nutritional status is compared for program communities and control communities. No difference is found for mean height-for-age or weigh-for-age, and a small difference is found for mean weight-for-height (0.07 v. -0.14 z-scores) with controls for a number of demographic variables. Thus, at the unit of analysis, the studies were not randomized controlled trials and may be biased. See our discussion of the problem of comparing those to who choose to participate in microfinance programs to those who do not.
  • 15See previous footnote.
  • 16Ghana study: Freedom from Hunger conducted a randomized controlled trial on its malaria programs in Ghana. Participating communities were randomly assigned to receive either malaria or diarrhea education. Results are reported in Grey et al. 2007, Pg 27, Table 14; Pg 20, Table 17; and Pg 31, Table 19. We considered results of the comparison between "Malaria clients" and "Diarrhea clients." To the degree to which malaria clients outperformed diarrhea clients on malaria indicators, the malaria program was considered effective. For additional information on the effectiveness of insecticide-treated nets, see our report on the effectiveness of insecticide-treated nets.
  • 17 Gray et al. 2007, Pg 2.
  • 18Peru health education study: "This study was carried out from February 2007 to February 2008 in collaboration with PRISMA, an NGO that provides microcredit loans to clients throughout Peru… After the baseline survey was completed, half of loan groups were randomized to receive an additional health education intervention, described below, in addition to the existing microcredit services… Of the 1,855 clients from the baseline survey, 1,501 (81.0%) consented to participate…PRISMA’s US-based affiliate, Freedom From Hunger, developed these sessions." Hamad, Fernald, and Karlan 2011, Pgs 3-4. Results are reported in Hamad, Fernald, and Karlan 2011, Pgs 7-8, Tables 3 and 4.
  • 19Hamad, Fernald, and Karlan 2011, abstract.
  • 20Benin study: 138 villages were randomly assigned to received education with the microfinance program or not and were also randomly assigned to either have female-only or mixed-gender groups (Gray and Ekoue-Kouvahey 2010, Pg 13). There were no statistically significant differences between the groups on indicators of food security, responsibilities and social network, or decision-making in the household. For results, see Gray and Ekoue-Kouvahey 2010, Pgs 39-52. Note that 17 of the originally included villages did not participate in the study, potentially introducing bias into the results (Gray and Ekoue-Kouvahey 2010, Pg 14). Focus groups and interviews to assess client satisfaction and reasons for leaving the program were also conducted and reported in the same paper.
  • 21Gray and Ekoue-Kouvahey 2010, Pgs 3-5.
  • 22

    "This innovative methodology combines qualitative and quantitative approaches to create a nuanced picture of the current SfC program and to document the baseline situation in an SfC expansion zone in the Segou region of Mali, where a randomized control trial (RCT) is currently underway to measure the socioeconomic impacts of the program over a three-year period (2009-2012)." Bureau of Applied Research in Anthropology and Innovations for Poverty Action, "Baseline Study of Saving for Change in Mali: Results from the Segou Expansion Zone and Existing SfC Sites," Pg 7.

  • 23

    "AIM Youth. This is a new innovation program to extend Saving for Change and Credit with Education to youth (13–25 years old). As a still untested innovation, we remain appropriately skeptical but optimistic about its prospects to work well in a variety of circumstances. The MasterCard Foundation has given us a major contract to fund the development and testing of AIM Youth." Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 24

    "GiveWell: Do you follow up with organizations that implement programs you train them on?

    Freedom from Hunger: We follow up with further training, particularly to develop or introduce new education modules to their work, and to troubleshoot with technical assistance as needed to help our implementing partners resolve the problems that naturally arise as they expand the program and integrate it within their other operations. We track the partners’ progress in our Credit with Education status report that is updated every six months with reports from the partners on financial performance and the education modules each organization is implementing. Once they are no longer dependent on Freedom from Hunger for funding or training/technical assistance, we cannot force them to report, so it is a tribute to our good relationship management that they continue to report voluntarily for years afterward.

    We are concerned about the quality of implementation. It’s a high-leverage strategy, because we train people to do what we know how to do. But we have no lasting control over the MFIs and other partners. We seek to deal with this by providing high-quality training up front and follow-up technical assistance as long as funding allows."
    Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 25

    Number of reporting organizations is from Freedom from Hunger, "Credit with Education Status Report (December 31, 2010)."

    On the total number of MFIs participating in Credit with Education: "Of 112 partners who continue to report to us (half are in Mexico and many of these just participated in training programs without follow-up technical assistance), we are highly engaged with something like 20 to 40 at this time. In some cases, we have the opportunity to go back in to see how they are doing after a long hiatus of engagement...There are some MFIs that we’ve worked with that have ceased to report. I estimate it’s about 10." Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 26

    "We track the partners’ progress in our Credit with Education status report that is updated every six months with reports from the partners on financial performance and the education modules each organization is implementing." Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 27

    Freedom from Hunger, "MAHP PM Report Data (December 2010)."

  • 28

    Freedom from Hunger, "Saving for Change Outreach (December 2010)."

  • 29

    "These data give us only weak proxy indicators of the quality of program implementation and no verification of impact. Still, when we receive data from the partner, we compare the numbers against those from the prior period and take note of any changes, positive or negative. Often this leads our relationship managers to reach out and talk to the partner; either to commend them for a growing program or to inquire about programs that are clearly struggling." Freedom from Hunger, "Response to GiveWell Draft Evaluation Report," Pg 3.

  • 30

    "Our monitoring of partners after they have been trained and guided in the customization process is done by "relationship managers"—our staff assigned to liaise with particular partners in particular countries, both during the intensive training and technical assistance phase of innovation dissemination and customization, and long afterward. As I said in the interview, the frequency with which these staff can actually make on-site visits to the assigned partners depends on dedicated funding and partner willingness to receive our visits (by far, the former is the more common constraint). However, the relationship manager stays in touch with key staff of the partner organization by phone and e-mail to stay abreast of the partner’s progress and challenges. Through these means, if not by direct observation, our staff has fairly accurate knowledge of how the partner is applying the tools and systems we have helped its staff develop and install." Freedom from Hunger, "Response to GiveWell Draft Evaluation Report," Pgs 2-3.

  • 31

    "The key steps are to randomly select clients for story collection and to collect these client stories with a systematic yet open-ended interview process that allows the client to tell as much of the full story as she or he is willing to share…Our goal is to interview people who have just recently joined a microfinance program and then find these same people about three years later (whether or not they are still participating in the program) to learn what has happened in their lives in the intervening period." Jarrel et al. 2011, Pg 9.

  • 32

    "So far, we have collected and analyzed a total of 274 client stories from random samples in eight countries with nine local microfinance-providing partners… At the time of writing this report, Freedom from Hunger had not yet begun the second round of interviews with original cohorts of incoming clients." Jarrel et al. 2011, Pgs 9 and 34.

  • 33

    "GiveWell: How would you respond to the criticism that Freedom from Hunger just comes in, provides training and support, and then leaves but doesn’t follow up to ensure that the program is going well and running effectively?
    Freedom from Hunger: We do everything we can upfront when we’re engaging with our partner to identify possible areas where the project might fail, and address those. We don’t have the resources to stay heavily engaged with our partners long-term, and in many cases, because they are independent operating entities, they might not want us to."
    Chris Dunford, phone conversation with GiveWell, February 22, 2011.

  • 34

    Chris Dunford, phone conversation with GiveWell, February 22, 2011.