VillageReach aims to improve the systems that distribute medical supplies to rural areas in Africa, so that life-saving supplies get to those who need them. Its programs include both technical support staff and changes in logistical setups (such as moving from a system in which health clinics collect their own supplies to a centralized delivery system).

VillageReach is a relatively small and young organization. Based on results from its pilot project, we believe its model has significant impact, well under $1000 per infant death averted.

What do they do?

VillageReach aims to improve the logistics - particularly tracking and distribution of supplies - for health systems in rural areas.1

Its specific activities include the following, with examples given from its pilot project in northern Mozambique:2

• Transportation vehicles: "Created multi-modal transport networks including land cruisers, motorcycles and bicycles. Staff inspects and repairs equipment on monthly visits."
• Cold chain: "Introduced reliable, low maintenance and cost-effective refrigerators in clinics."
• Injection safety equipment: "Installed propane burners for sterilization, incineration points and needle removers to ensure safe disposal of used syringes."
• Clinics' energy access: "Provided lighting for nighttime care, refrigerators, and sterilizers at clinics."
• Supplies-tracking: "Partnered with Iridium to utilize their global satellite system, and introduced communication system in trucks to enable near, real-time inventory tracking."
• Training: "Trained community representatives to provide basic health care."
• Creating a for-profit enterprise to improve energy supply: "Established VidaGas, a Mozambican propane distribution company to reliably supply energy to clinics, businesses and households."

VillageReach, which was founded in 2000,3 has thus far completed one project, a demonstration program of its approach, which was implemented in the Cabo Delgado province of Mozambique from April 2002 to May 2007.4 The Cabo Delgado project officially became the local government's responsibility in 2007;5 in 2006, VillageReach began to replicate the project in a second province in Mozambique, Nampula.6

Future activities

As of March 2008, VillageReach reported that it was in discussions with Malawi's Ministry of Health to implement the VillageReach program in Malawi and anticipated signing a "memorandum of understanding" by the summer of 2008.7 As of October 2008, when the Mozambique Ministry of Health requested that VillageReach implement its program nationally, VillageReach was in the process of developing an expansion plan to implement the program and raise necessary funds.8

Does it work?

One of VillageReach's primary methods of evaluating success is through tracking the progress made in administering basic immunizations. Such immunizations are a proven, cost-effective way to improve health and save lives in the developing world (more here), and so success in increasing immunization coverage - alone - likely constitutes, in our view, success in saving lives.

Below we examine evidence provided by VillageReach including (a) reporting on vaccines and equipment delivered, progress in vaccination coverage rates, and health clinic inventories; (b) an independent evaluation9 of VillageReach's impact. We conclude that VillageReach's pilot project has been effective in increasing vaccine coverage.

Delivery of vaccines and medical supplies

A key component of VillageReach's model is a shift from a "collection-based" to a "delivery-based" supply system: rather than clinics' being responsible for picking up their own supplies, VillageReach's logistics team delivers supplies and provides other logistical support.10

The tables below provide data on the goods VillageReach delivered to Cabo Delgado between 2004 and 2007.11 (Note that the project began in April 2002; we aren't sure why data has not been provided pre-2004.)

Increases in immunization coverage

The charts below show that (a) the number of children receiving DTP-3 (third dose of diphtheria-tetanus-pertussis vaccine12) immunizations increased; (b) the number of children who "dropped out" during the DTP-3 sequence - that is, they received one, but not all doses - fell; (c) reported "stock-outs" - centers with no inventory of the vaccine - fell significantly over the course of VillageReach's pilot project, which ran from April 2002 to May 2007.13. Charts are taken from VillageReach's 5-year assessment.14

Were improvements attributable to VillageReach?

VillageReach's 2008 Evaluation Report notes that during this time period, "Mozambique and most sub-Saharan African countries achieved significant improvements in their DTP-3 coverage, probably due to GAVI and its support for infrastructure development, provision of new vaccines, and safe injection equipment."15

To evaluate the question of VillageReach's role in improvements, we look at three types of information:

1. Reports from VillageReach about the problems interfering with immunizations in Cabo Delgado before their arrival.
2. An evaluation report published by VillageReach explicitly focused on addressing this question, comparing the change in immunization coverage rates in Cabo Delgado to that of another province in Mozambique, Niassa.
3. Examining the changes in immunization coverage in several countries in Africa over this period to put the observed change in context.

We also observe that the above charts showing "stock-outs" above have stock-outs falling from a very high level prior to the start of the project to a very low level shortly after the project began. Intuitively speaking, this suggests that the project was impactful in this area.

VillageReach reports of pre-arrival conditions

VillageReach provides a report of the obstacles to immunization coverage in Cabo Delgado before its arrival. We would prefer to have better documentation of these conditions, but nevertheless, we believe the report offers some support to the idea that VillageReach's services were needed in Cabo Delgado.

VillageReach reports,

Prior to the [project], health facilities were individually responsible for picking up vaccines and supplies from the DPS cold stores or district stores and taking them back to their health facility for use. This system encountered various problems:

• Intermittent closing of health facilities during business hours so health workers could pick up vaccines and supplies.
• Challenges securing transport to go to the DPS cold stores. Each district generally had one vehicle, which was for all health service trips by all health system personnel, and was also the ambulance in case of emergencies. Often, when the vehicle was needed to pick up or deliver vaccines, it was out on an emergency, in use by someone else for some other health-system function, broken down, or out of gas.
• Difficulty maintaining proper vaccine temperatures during transport.
• Uncoordinated vaccine supply requirements.
• Frequent stock-outs of vaccines in health facilities.
• Funds were often liberated late – both quarterly from the provincial level to the districts, and monthly from the district administrator to the PAV Chief who needed to purchase gas for the refrigerators, fuel up the district vehicle, and pick up and distribute vaccines.16

In 2002, before starting work, VillageReach performed an assessment of access to vaccines in Cabo Delgado province.17 This report claims that, in 2002, there were 22 health facilities in the three districts in Cabo Delgado that VillageReach assessed.18 Of these 22, 4 did not offer access to vaccination services:19 two facilities because they did not have access to a cold chain; 1 because it lacked personnel; and 1 for other reasons.20

VillageReach evaluation document

VillageReach compared improvements in Cabo Delgado to improvements on the same indicators in the nearby Niassa province, which was not served by VillageReach.21

• Baseline data collection: Because VillageReach did not have baseline data (i.e., data from a time prior to the start of the project) available for Niassa, it used data from the 1997 and 2003 Demographic and Health Surveys, which surveyed a large number of households and created estimates of the DTP-3 coverage rate.22
• Outcome data collection: Evaluators randomly selected households in the treatment area (Cabo Delgado province),23 including 474 children.24 Evaluators then also randomly selected households in the comparison area (Niassa province), interviewing households in which 1,200 children lived.25
• Results: The study found that DTP-3 coverage rates increased significantly in both provinces during this period, but rates improved significantly more in Cabo Delgado (treatment area) than Niassa (control area). The chart below shows the change in coverage rates in the two areas during this time period; the 1997 and 2003 data points are based on Demographic and Health Surveys, while the 2008 data points are from VillageReach's independent data collection.26

On one hand, we believe that this chart creates an inflated picture of VillageReach's impact. We have reason to believe that there were significant improvements in immunization coverage between 1997-2001 that were more related to Cabo Delgado's recovery from the aftermath of a civil war than to VillageReach's activities.27 However, the jump to extremely high levels of coverage as of 2008 - a change not mirrored in the nearby province - give some reason to attribute impact to VillageReach.

The evaluation report is forthright about many limitations of this comparison analysis, including limited sample size, uncertainty about the appropriateness of Niassa as a "comparison province," and issues with taking baseline and endpoint data from different sources.28 However, it concludes that "Such a comparison is consistent with international practice" and that "it appears that the Project is most likely responsible for this difference, [although] additional information about the conditions in Niassa compared to those in Cabo Delgado is needed to better understand and interpret the comparison data."29 It also notes that "It is ... unlikely that the activities of other NGO's, which are not very involved in immunization activities in Cabo Delgado, were responsible for the improvement."30 (Note, however, that it does not discuss the confounding factor that we discussed above.)31

Our comparison of Cabo Delgado to other areas in the developing world

We took a broader look at changes in African immunization coverage over the time period in question in order to further investigate the idea that Cabo Delgado's improvements may simply have reflected a wider phenomenon. Using the Demographic and Health Surveys (Measure DHS),32 we collected data on DTP-3 immunizations for countries in Sub-Saharan Africa.33

The table below summarizes this data, sorted by the country's arithmetic percentage change in immunization coverage.

This table provides information at a country level rather than province level, and variation within countries could be significant. However, it appears to rule out the idea that the observed change in Cabo Delgado purely reflected a broader (country-wide or continent-wide) change. Also note that Cabo Delgado's 2003 coverage rate was slightly below the Mozambique overall rate, while its post-project rate was above the Mozambique overall rate (and above every other country's overall rate).

Bottom line on the Cabo Delgado program between 2001-2008

We do not feel that any of the pieces of evidence above is highly compelling by itself. But we are persuaded of VillageReach's impact by the combination of the observations that VillageReach's program (a) entered an area with clearly documented logistics problems; (b) reduced stockouts - one of the clearest measures of the logistics improvement it was aiming for - to near-zero levels; (c) brought Cabo Delgado from an "average" (for the country) level of coverage to an extremely high level of coverage; (d) was reported not to have been supplemented by other nonprofits' programs.

Can improvements be maintained?

Based on the evidence above, we feel that the VillageReach program has improved capacity to deliver vaccines in Cabo Delgado. However, VillageReach does not seek to run its programs indefinitely; rather, it seeks to hand them over to the government, and there are major concerns about whether it can do so effectively. The Cabo Delgado project officially became the local government's responsibility in 2007;34 a recent report states, "The data suggests that following the discontinuation of field coordinator teams delivering supplies and performing supervision, the districts and health centers are having difficulty reliably picking up supplies, stock-outs of vaccines are beginning to occur again, there is some (not statistically significant) evidence that immunization coverage is beginning to fall, and district level budgets are not being maintained for these activities."35 (Emphasis ours.) VillageReach's representatives have stated to us that the government has been "slid[ing] back into the old collection-based system" but that they expect this slide to be temporary.36

The fact that VillageReach is monitoring the program's continuing performance, and being open about setbacks, is encouraging; but news of program deterioration is cause for concern.

It's worth noting that VillageReach could be making lasting differences in individuals' lives even if its effects on health care are only temporary, since 1-3 doses of most vaccines are sufficient to immunize children against diseases. (Details here.)

Possible negative/offseting impact

As stated here, we are generally concerned about charities' potential diversion of skilled labor and/or interference with government responsibilities. However, we believe these concerns are smaller with VillageReach than with other charities we've seen.

VillageReach's focus is on improving logistics rather than on increasing the available resources in an area. Its cost analysis argues that its program ultimately ends up saving the government money (more below), and a conversation with its representatives implies that it does not attempt to repurpose skilled labor from other areas or sectors.37 In addition, it appears to be seriously committed to handing off its programs to the government over time, as it has done in Cabo Delgado. It does not appear to grant funds directly to governments.

What do you get for your dollar?

We do not attempt to quantify the full benefits of the VillageReach program. Instead, we observe that even a relatively conservative estimate of its cost per child vaccinated would imply quite strong cost-effectiveness (in terms of cost per death averted).

The Disease Control Priorites report (DCP) estimates the cost per fully-immunized child with a basic set of vaccines at $14.21 in sub-Saharan Africa.38 According to the DCP, this implies a cost per death averted of approximately $200.39

• VillageReach's estimate: VillageReach sent us a draft of its internal review of the Cabo Delgado project's cost-effectiveness. VillageReach estimates that its program is significantly more cost-effective than the government's program, at a cost of $5.70 per child fully immunized including both VillageReach and government costs.40 This would imply a cost-per-death averted that is significantly lower than the DCP's estimate of $200, and thus easily within the range we consider highly cost-effective (discussed here).
• GiveWell's conservative estimate: We assumed that all VillageReach costs (except for those explicitly allocated to its second project in Nampula, for which vaccination coverage data is not yet available) are attributable to the Cabo Delgado project, and that there is no impact of VillageReach on immunization coverage beyond 2008 (the last year for which we have data). We also assumed that starting in 2003, the difference in immunization coverage between Cabo Delgado and Niassa can be attributed to VillageReach's program. These assumptions yield an estimate of one additional child fully immunized for every ~$35 of VillageReach's expenses.41 If the Disease Control Priorities Report is correct to estimate that $15 per fully immunized child corresponds to $200 per death averted, this would imply that Village Reach is averting a child death for every ~$460 it spends, still well within the range discussed on our overview of cost-effectiveness estimates. (This estimate ignores government costs entirely, in order to give a sense of what is accomplished for donor money.)42

Room for more funds?

VillageReach sent us a document providing its plans for the near future, stating that "Expansion of the logistics platform in Niassa and Tete and the Cabo Delgado study is expected to cost approximately $3.5 million over three years. Ministerio da Saúde (MISAU) will fund approximately $1.2 million of that amount to cover infrastructure improvement and operating costs. VillageReach is seeking private charitable funding to cover the remaining $2.3 million for its technical assistance and additional investments in the management information system."43 A VillageReach representative told us that their internal fundraising goal for 2009 is $750,000. As of June 17, 2009, VillageReach had raised $200,000 towards that goal.44

Financials/other

All data comes from VillageReach's IRS form 990s for 2002-2007, available for download through the National Center for Charitable Statistics.45

Revenue and expense growth (about this metric): VillageReach reached a large five-year, $3.3 million grant agreement with the Gates Foundation in 2004,46 which explains the large jump reported revenues in 2004.47

In 2007, both revenues and expenses fell. It's possible that this is because VillageReach had completed its work in Mozambique and was largely focused on reviewing and evaluating that project.

Assets-to-expenses ratio (about this metric): VillageReach has maintained an assets:expenses ratio of approximately 2:1, aside from the year (and year after) they received the Gates Foundation grant.

Expenses by program area (about this metric): As far as we can tell, VillageReach has only one program area, the program described above.

Expenses by IRS-reported category (about this metric): VillageReach maintains a reasonable "overhead ratio", spending approximately 70-80% of its budget on program expenses.

Unanswered questions

• Room for more funds. VillageReach is currently seeking to raise $750,000 to expand its operations in Mozambique. We have not yet seen a detailed budget projection for this expansion. We also have little sense of how likely VillageReach is to reach its goal or of how funds in excess of $750,000 would be used.
• Does VillageReach have a lasting impact? As discussed above, there is some evidence that program performance has deteriorated in Cabo Delgado since the local government assumed responsibility for continuing the program. We await more information about whether the government can and will be able to maintain VillageReach's apparent improvements.

Sources

• [30x7 sampling explanation] http://nccphp.sph.unc.edu/PHRST5/TwoStageSampling.pdf, accessed 8/17/2009.
• Audited Financial Statements. 2008. Available online here, accessed 8/17/2009.
• Cabo Delgado Logistics Assessment Report. 2002. Available online here, accessed 8/17/2009.
• Phone conversation with Director of Strategic Development John Beale, President Allen Wilcox and Finance and Program Administration Manager Becca Miller, 5/21/2009.
• Skoll Foundation Application Narrative and supporting documents. 2008. VillageReach requested that we not post this document online.
• USAID: Measure DHS Program. Online at http://www.measuredhs.com/, accessed 8/17/09.
• VillageReach Evaluation of the Project to Support PAV. 2008. Summary available online at http://villagereach.net/vrsite/wp-content/uploads/2009/08/Evaluation-Exe..., accessed 8/17/09. VillageReach has informed us that they plan to post the full text online in the future.
• VillageReach Five Year Project Report. 2008. Available online here, accessed 8/17/09.
• VillageReach Evaluation of the Project to Support PAV: Statistical Analysis. 2008. VillageReach has informed us that they plan to post this document online in the future.
• Village Reach Board of Directors and Key Personnel. Available at http://villagereach.net/about-us/our-team/ and http://villagereach.net/about-us/board-of-directors-advisors/, accessed 8/17/09.
• VillageReach Cost Estimates. Available online here, 8/17/3009.
• VillageReach. "Mozambique Expansion Summary". Available online here, accessed 8/17/2009.
• VillageReach. 2009. "Comparison of costs incurred in dedicated and diffused vaccination logistics systems: cost-effectiveness of vaccine logistics in Cabo Delgado and Niassa provinces, Mozambique." Summary available online at http://villagereach.net/vrsite/wp-content/uploads/2009/08/VillageReach-C..., accessed 8/17/09. VillageReach has asked us not to post the full text online.
• VillageReach. 2009. "Improving Vaccination Distribution Systems". Available online here and at http://villagereach.net/vrsite/wp-content/uploads/2009/08/VillageReach-M..., accessed 8/17/2009.
• VillageReach. March 2009. "Optimize Overview". Available online here, accessed 8/17/2009.
• VillageReach. 2009. Rotary Seattle Presentation. Available online here, accessed 6/30/2009.
• VillageReach. "VillageReach at a Glance". Available online here and at http://villagereach.net/vrsite/wp-content/uploads/2009/08/VillageReach-a..., accessed 8/17/2009.
• VillageReach. "Why Vaccines". Available online here, accessed 8/17/2009.
• VillageReach/GiveWell email exchange, June 2009. Available online here, accessed 8/17/2009.
• VillageReach. Organization website. Online at http://www.villagereach.org/, accessed 8/17/09.

• 1.

  "VillageReach's mission is to to save lives and improve well being in developing countries by increasing last-mile access to healthcare and investing in social businesses that address gaps in health and community infrastructure...Even as significant resources from the public and private sectors have provided new medicines, vaccines and equipment intended to benefit developing countries over the past years, significant gaps in the distribution of these health technologies to rural communities remain." VillageReach Website (http://www.villagereach.org/about.html, accessed 8/17/09).

• 2.

  http://www.villagereach.org/MOZ_project.htm, accessed 6/30/09

• 3.

  VillageReach website (http://villagereach.net/about-us/about-villagereach/, accessed 8/17/09).

• 4.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 10.

• 5.

  "We have transitioned the VillageReach model and program to the local Ministry of Health in Cabo Delgado province, home to 88 vaccination clinics. VillageReach and FDC, our local implementation partner, will continue to provide technical assistance and data reporting." See "April 2007" section of http://villagereach.org/timeline.htm, accessed 6/30/09

• 6.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 8.

• 7.

  http://villagereach.org/timeline.htm, accessed 6/30/09.

• 8.

  VillageReach. "Mozambique Expansion Summary."

• 9.

  "This report was compiled by Mark Kane, MD, MPH, a consultant, following review of these materials and extensive discussions with VillageReach staff. Many of the opinions and viewpoints in this report are those of the reviewer, and do not necessarily represent the views of the VR, its staff, or Project implementation partners." VillageReach Evaluation of the Project to Support PAV 2008, Pg 12. Mark Kane is listed as a key advisor to VillageReach, at http://villagereach.net/about-us/board-of-directors-advisors/, accessed 8/17/09.

• 10.

  "Under the previous system of distribution, clinic workers in need of vaccines and other medical supplies were required to travel many miles, often on foot, to a provincial or district warehouse to obtain supplies that were not always available. Today, in Cabo Delgado, health workers at 90 rural clinics receive monthly deliveries from one of VillageReach's three delivery trucks, specially outfitted to navigate the difficult terrain of rarely maintained roads and sustain the cold chain necessary for the safe transport of vaccines. As the VillageReach drivers leave from the provincial warehouse for two-week excursions, they bring with them the necessary vaccines, medical supplies, and energy needed by each clinic to serve their communities." http://www.villagereach.org/supply_chain.htm, accessed 6/30/09

• 11.

  VillageReach Evaluation of the Project to Support PAV 2008, Pgs 17-18

• 12.

  World Health Organization glossary at http://www.who.int/immunization_monitoring/glossary/en/index.html, accessed 6/30/09

• 13.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 10.

• 14.

  VillageReach Five Year Project Report 2008, pgs 12-14. See VillageReach/GiveWell 2009 email exchange for clarification on how data was collected.

• 15.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 24

• 16.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 14

• 17.

  Logistics Assessment Report 2002

• 18.

  Logistics Assessment Report 2002, Pg 9, Table 1.1.

• 19.

  Logistics Assessment Report 2002, Pg 10, Table 2.1.

• 20.

  Logistics Assessment Report 2002, Pg 10, Table 2.2.

• 21.

  "Comparison data was obtained from a 2007 immunization coverage cluster survey conducted by DPS in the neighboring province of Niassa, in which the Project did not operate." VillageReach Evaluation of the Project to Support PAV 2008, Pg 6.

• 22.

  VillageReach Evaluation of the Project to Support PAV: Statistical Analysis 2008, Pg 9

• 23.

  Methodological details in the Evaluation Report: Statistical Analysis 2008, Pg 10.

• 24.

  VillageReach Evaluation of the Project to Support PAV: Statistical Analysis 2008, Pg 10.

• 25.

  VillageReach Evaluation of the Project to Support PAV:Statistical Analysis 2008, Pg 16.

• 26.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 23.

• 27.

  See VillageReach/GiveWell 2009 email exchange.

• 28.

  "The Statistical Analysis for the quantitative surveys carefully describes the factors that could bias the evaluation results:

  • Baseline surveys were not done at the inception of the project
  • A “comparison” Province (or Provinces) was not designated at the initiation of the project
  • Comparing the results of surveys done with different methodologies (DHS and EPI cluster surveys) creates certain potential biases
  • Relatively small sample sizes made it difficult to detect small changes in coverage between the two age groups
  • Uncertainty about the reasons districts were chosen for the Niassa survey
  • Uncertainty about the comparability of Niassa as a “comparison” province."

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 23

• 29.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 24

• 30.

  VillageReach Evaluation of the Project to Support PAV 2008, Pg 24

• 31.

  See VillageReach/GiveWell 2009 email exchange.

• 32.

  USAID: Measure DHS Program. We accessed data through the StatCompiler tool available online at http://www.statcompiler.com/, accessed 6/30/09. We looked at all available surveys for Sub-Saharan Africa.

• 33.

  We downloaded data on DTP-3 immunizations. The percentages in the table reflect reports either from (a) the child's vaccination card or (b) a mother's report (called "either source" in the Measure DHS tables). We only include countries that had at least one survey during or before 1997 and at least one survey during or after 2003; 1997-2003 was the period over which VillageReach provided Measure DHS surveys for Cabo Delgado.