BRAC’s Response to GiveWell’s Evaluation Report
Prepared by Anu M.Shetty (Program Associate, BRAC USA) and Scott MacMillan (Communications Manager, BRAC USA)
October 2, 2013
We are writing to respond to Givewell’s assessment that “despite the apparent commitment to monitoring and evaluation, BRAC’s research does not appear to make it possible to evaluate the impact of the organization as a whole. Specifically, program evaluations appear not to be available for some of BRAC's largest programs.”
Neither of the two above statements is correct. BRAC’s research website (http://research.brac.net/) is geared largely toward measuring the impact of BRAC’s individual programs, rather than the organization as a whole. However, studies are widely available online that, taken together, show the overall positive impact of BRAC, both in its native Bangladesh and beyond.
BRAC’s scale and impact
BRAC’s scale as a poverty alleviation organization is well-attested. With over 100,000 employees, it is generally recognized as the largest nongovernmental organization in the world, reaching an estimated 135 million people in 11 countries with programs in economic empowerment for women and girls, education, healthcare, agriculture, human rights, and more. Paul Collier, a scholar of development, has described BRAC as “the most astounding social enterprise in the world,” while The New York Times’s Nicholas Kristof has called it “the best aid group you’ve never heard of.” Bill Gates has said, “BRAC has done what few others have – they have achieved success on a massive scale[.]”
BRAC is also known as one of the world’s earliest and best examples of a “learning organization,” having established an independent Research and Evaluation Division as early as 1975. The organization is known for addressing its own failures by constant internal monitoring and evaluation, including through randomized control trials, while publishing its findings online.
In terms of evaluating “the organization as a whole,” a 2004 study published in The Journal for International Development examined BRAC’s poverty alleviation programs as a whole and found positive impacts on a wide range of indicators, including child survival, nutrition, expenditure patterns, family planning, and children’s education.
In terms of the availability of evaluations on “some of BRAC’s largest programs,” most of BRAC’s studies are indeed program design studies rather than impact studies, designed to improve programs by revealing flaws in their design. Moreover, several of BRAC’s largest programs in Bangladesh are so well-established, with the organization reaching three-quarters of the country’s population of 154 million, that it is difficult to find control groups needed to conduct proper randomized controlled trials (RCTs), considered the gold standard of evaluating impact.
That said, BRAC has in fact conducted extensive impact studies and RCTs, including some for its largest programs. These are available online, and several are cited below.
Overwhelming evidence shows that participation in BRAC programs, including microfinance, reduces women‘s economic dependence on their husbands and other male relatives. According to a 2001 paper published in the Journal of International Women’s Studies, women participating in BRAC programs in Bangladesh “reported that they now have an independent source of income. They no longer need to rely solely on their husbands for the purchase of personal and household items. Women have also reported an improvement in their relationships with their husbands, primarily because they provide them with capital for investment purposes.”
Empowerment of adolescent girls
In Uganda, researchers from outside institutions, including London School of Economics, University College London, and the World Bank, have recorded a significant decline in risky behaviors among participants in BRAC’s adolescent empowerment program, the membership of which exceeds 270,000 worldwide. Using a randomized control trial, the study found the program delays early marriage, improves knowledge on HIV, and leads to greater self-reported condom use and fewer incidents of pregnancy. The program also reported an 83 percent reduction in reports of unwilling sex among participants, “the clearest marker for the program changing how empowered adolescent girls are in their relations with men.” The authors call the program “a novel advance over earlier generations of standalone interventions related to HIV-education or skills provision, in that it recognizes the interlinkage between … health and economic issues.” (Download full study here.)
Targeting the ultra-poor
Impact studies, including a randomized control trial, have demonstrated the impact of BRAC’s program targeting the ultra-poor, in which more than one million households have participated. This program is designed for those trapped in ultra-poverty, for whom microfinance is not an option. BRAC’s tailored methodology has been instrumental in giving hope and a future-oriented mindset to the ultra-poor, as economists and development scholars, including Esther Duflo of MIT’s Poverty Action Lab, have pointed out.
According to an impact study published in 2010, among participants that entered the two-year program in 2002, 98.4 percent of households satisfied at least six out of ten indicators of graduating from extreme poverty (including having three to four income sources, two meals a day, a kitchen garden, a solid roof, a sanitary latrine, and school-going children), even four years after the program ended. Moreover, a large-scale randomized control trial published in 2013 tracked 7,000 eligible women over a four-year period, revealing a “dramatic change” in participants’ occupational structure, food security, life satisfaction, and earnings.
BRAC schools have educated nearly 10 million children. In Bangladesh, BRAC has allowed an entire generation to grow up with opportunities that would have been unimaginable decades previously. According to a July 2011 study, “BRAC schools, in spite of their meager facilities, performed better in terms of internal efficiency – dropout and repetition rates being lower and completion rates higher than other types of school. This is mainly because of intensive monitoring of teachers, better teacher input and the teaching process suited to the children from poor families.”
BRAC’s health program, with its 100,000 self-employed community health workers, is widely believed by global health experts to have played a major role in halving the country’s infant mortality rates. Between 1980 and 1990, BRAC’s oral rehydration program reached 14 million of Bangladesh’s 19 million households, with government surveys showing 70 percent of families in Bangladesh using oral rehydration solution, according to Stanford Social Innovation Review and others. Moreover, a 2011 study of BRAC’s urban maternal health program showed a reduction in delays associated with emergency care for life-threatening labor complications. According to the BRAC website, for the period of 2007 to 2010, in areas where the urban maternal health program operates, monitoring of progress shows a reduction in home delivery from 86 percent to 25 percent, with the maternal mortality rate in urban intervention areas decreasing to 141 per 100,000 live births, compared to the national rate of 194 – and below the Millennium Development Goal of 143.
BRAC has recorded remarkable results in changing conditions for smallholder and tenant farmers in seven countries, training more than 200,000 farmers. BRAC‘s Research and Evaluation Division for Africa, based in Kampala, studied Ugandan farmers’ retention of training after one year, finding that 68 percent had retained new knowledge on planting methods and that 75 percent used improved seeds versus 48 percent in a control sample.
In light of the evidence above and BRAC’s well-demonstrated commitment to impact evaluation, we feel the GiveWell assessment deserves revisiting.