VillageReach was our top-rated organization for 2009, 2010 and much of 2011 and it has now received over $2 million due to GiveWell's recommendation. We do not believe that VillageReach has short-term funding needs and therefore prefer our two, current
highest-rated charities above it.
Published: April 26, 2011; Updated: November 29, 2011
Summary
VillageReach aims to improve the systems that distribute medical supplies to rural areas in Africa, so that life-saving supplies get to those who need them. Its programs include both technical support staff and changes in logistical setups (such as moving from a system in which health clinics collect their own supplies to a centralized delivery system). It ran its pilot program in a single province in Mozambique from 2004 to 2007 and is now working to scale up the project across the entire country. As of November 2011, it operates in 4 of Mozambique's 9 provinces.
We believe that VillageReach is a standout organization because:
- Its pilot project and current scale-up focuses on increasing immunization coverage among children. Immunizations are a proven, cost-effective way to prevent child mortality.
- It produced a credible evaluation of its pilot project that, while not a randomized controlled trial, makes a compelling case for impact.
- As an organization it has demonstrated a commitment to sharing results of its work publicly enabling it and others to learn from both its successes and failures.
- Because of the above commitment, if its approach to health logistics succeeds, we believe there is significant possibility of upside, i.e., a positive effect that significantly surpasses the direct benefits of vaccinating the children its program serves. This would occur if its project succeeds and other governments choose to implement VillageReach's approach to health logistics.
Our full review, below, discusses our full assessment of VillageReach, including what we see as its strengths and weaknesses as well issues we have yet to resolve.
We also continue to follow VillageReach's progress closely. We post regular updates on its progress to our
VillageReach updates page.
What do they do?
Broadly, VillageReach aims to improve health care access in remote, underserved places by improving the medical supply delivery systems. A key component of VillageReach's system focuses on shifting health-products distribution from a "pull" system (in which health clinics collect their own supplies) to a "push" system (in which dedicated teams deliver supples to clinics directly).
VillageReach implemented a pilot project in the province Cabo Delgado in Mozambique from 2002 to 2007 focused on delivering vaccines, and also briefly ran a similar program in the neighboring province of Nampula. It then handed off the projects. In 2010, it resumed its vaccine delivery system in Cabo Delgado, and is working to expand the system to other provinces in Mozambique and to use the system to deliver additional supplies. VillageReach has also contracted with other organizations to improve logistical systems in other countries, but generally hasn't used, and doesn't plan to use, funding from individuals for projects outside of Mozambique (with a few exceptions discussed below).
Pilot project in Cabo Delgado, Mozambique (2002-2007)
VillageReach, which was founded in 2000, conducted a demonstration of its vaccine delivery system in the province Cabo Delgado in Mozambique from April 2002 to March 2007. The Cabo Delgado project officially became the local government's responsibility in 2007, but VillageReach resumed responsibility in mid-2010 due to problems under government management.
VillageReach's pilot project in Cabo Delgado is the focus of our review because it is similar to the future activities most likely to be funded with donations. The pilot project delivered supplies to all of the facilities in the province that offered vaccinations. In addition to vaccines, VillageReach delivered propane gas, medicines, and other medical supplies to the facilities each month. The components of the delivery system included:
- Transportation vehicles: "Created multi-modal transport network including land cruisers, motorcycles and bicycles. Staff inspects and repairs equipment on monthly visits."
- Cold chain: "Introduced reliable, low maintenance and cost-effective refrigerators in clinics."
- Injection safety equipment: "Installed propane burners for sterilization, incineration points and needle removers to ensure safe disposal of used syringes."
- Supplies-tracking: "Partnered with Iridium to utilize their global satellite system, introduced communication system in trucks to enable near, real-time inventory tracking."
- Training: "Trained community representatives to provide basic health care."
- Creating a for-profit enterprise to improve energy supply: "Established VidaGas, a Mozambican propane distribution company to reliably supply energy to clinics, businesses and households."
Nampula expansion (2006-2007)
In 2006, VillageReach began to replicate the project in a second province in Mozambique, Nampula. It handed off the project to a local nonprofit in January 2007.
Project scale-up in Mozambique (2010-present)
VillageReach's current and planned future activities in Mozambique focus on expanding/replicating the model of the pilot project. These are the main activities for which VillageReach is seeking donations from individuals.
VillageReach expects to have an active presence in each of eight provinces for three years, after which point it is hoped that the government health system in that province will maintain its model without further support. VillageReach will emphasize the health supply delivery system detailed above, while also applying it to additional supplies such as rapid diagnostic tests for malaria and other diseases.
Current activities
VillageReach resumed activities in Cabo Delgado in June 2010, has been delivering vaccines to health facilities on a monthly basis, and has started to distribute diagnostic tests. As of February 2011, it had acquired two vehicles, completed a baseline survey, trained coordinators, and begun regular data collection.
In another province, Niassa, it completed a baseline survey and began distributions in early 2011 (we do not have further details on activities in Niassa).
Future plans
As of February 2011, the project scale-up plan was 6-12 months behind the schedule set out in July 2010 due to delays partly related to dialogue with the government but primarily due to delays in receiving sufficient funding (though revenue in 2010 closed VillageReach's entire "stretch funding gap" for the year, the vast majority of that revenue was not received or anticipated until December). In Maputo, VillageReach originally planned to begin full implementation of its program in the final quarter of 2010, and now intends to begin full implementation in mid-2011.
In addition to Cabo Delgado, Niassa, and Maputo, VillageReach is currently hoping to implement its model across five more of Mozambique's provinces, for a total of eight out of ten provinces in the country. For each province, implementation is scheduled to start in either 2012 or 2013. As of February 2011, VillageReach was in discussions with the Gaza province to begin work there, and planned to begin talks with the Inhambane province once it had completed a key hire.
Additional activities
VillageReach is also involved in other projects, specifically:
- A variety of contract engagements. VillageReach has contracted with specific parties to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. With one exception, all of these projects are fully funded by the party in question, and VillageReach does not seek donations for them. As such, we have not deeply examined these programs (with the exception of the one for which donations are sought).
- A number of other initiatives, detailed below.
Contract engagements
Specific parties have offered VillageReach funds to carry out projects in different parts of the world, all on the theme of improving health system logistics but sometimes different in many ways from the pilot project discussed above. Notes on these engagements follow. Projected expenses for these contract engagements have not been updated since mid-2010 and therefore will not fully reconcile with other figures given below.
| Funder | Bayview Foundation | [Currently confidential] | [Currently confidential] | [Currently confidential] | John Snow Inc. |
|---|
| Area | Malawi (Kwitanda province) | South Africa (KwaZulu Natal province) | India | Senegal | TBD |
| Description | SMS-based logistics for community health workers | General health system logistics | Vaccine-focused health system logistics | General health system logistics | Operations research |
| Expenses | $249,961 | $120,462 | $233,568 | $133,531 | $250,000 |
| Fully funded? | Yes | No | Yes | Yes | Yes |
| Needed from donations | $0 | $100,000 | $0 | $0 | $0 |
| More information | VillageReach, "President's Report;" VillageReach, "Organization Budget (2010)." | VillageReach, "President's Report;" VillageReach, "Organization Budget (2010)";' VillageReach, "South Africa Proposal." | VillageReach, "President's Report;" VillageReach, "Organization Budget (2010)." | VillageReach, "President's Report;" VillageReach, "Organization Budget (2010)." | VillageReach, "John Snow Proposal." |
As noted in the table above, all contract projects have been fully funded by the main project funder, with the exception of the South Africa project, which requires VillageReach to contribute $100,000 toward the costs of the project. Some additional notes on this project, since it is relevant to individual donors:
- VillageReach seeks to improve the general health system capacity in the Zululand district of the KwaZulu Natal province of South Africa through "business process mapping and optimization, streamlining data collection, staff training, mentoring, and supervision, expanding quality improvement methodology, and the identification of innovative uses for information and communication technology in patient care coordination at the clinic level." The project is projected to last three years.
- VillageReach's specific plans for assessing its impact are unclear to us. It states that it plans on "A baseline and endline survey to gauge the outcomes and impact of the project" as well as "track[ing] output and outcome indicators that could include stock levels of key commodities, accuracy and frequency of data collection in the clinics, frequency of supervision visits, reports of data use and understanding by clinic staff, and satisfaction of clinic staff with the program." We have not seen details of how baseline and end-of-project data will be collected, or of which indicators are included in VillageReach's long-term goal of "80% improvement in identified primary healthcare indicators." On a more short-term basis, it is also unclear to us how VillageReach plans on assessing "accuracy and frequency of data collection in the clinics."
Other plans
As of mid-2010, VillageReach's other planned activities included continuing to develop and share its management information system, pursuing new contract engagements, and ongoing support to VidaGas, a gas delivery business developed as part of its pilot program.
The following is its overall budget for the year to September 31, 2010 (projected for June-September). Note that these figures are not taken from exactly the same source, time period and date as the contract engagement figures above and have not been updated since mid-2010.
| Program | Expense |
|---|
| Mozambique Vaccination Expansion | $684,380 |
| Malawi | $262,499 |
| Other contract Engagements (aside from Malawi) | $390,012 |
| VidaGas Support Activities | $60,656 |
| IT Development | $184,814 |
| General Program | $31,866 |
| Program Development | $210,716 |
| Management/general | $185,591 |
| Fundraising/marketing | $103,868 |
Including the $100,000 matching requirement over three years detailed above (we count this as $33,333 for the year), there were a total of $810,844 in expenses or projected expenses in the year to September 31, 2010 that were neither part of the Mozambique expansion nor included in contract engagements.
Does it work?
This section focuses on VillageReach's pilot project in Cabo Delgado, then briefly discusses its hand off of its work in Cabo Delgado to the government and in Nampula to another organization. It also reports on what we know to date about its current and planned future scale-up of the project in Mozambique.
Pilot project in Cabo Delgado
One of VillageReach's primary methods of evaluating success is through tracking the progress made in administering basic immunizations. Such immunizations are a proven, cost-effective way to improve health and save lives in the developing world (more at our
report on immunization), and so success in increasing immunization coverage - alone - likely constitutes, in our view, success in saving lives.
Below we examine evidence provided by VillageReach on its pilot project in Cabo Delgado. This evidence includes: (a) reporting on vaccines and equipment delivered, progress in vaccination coverage rates, and health clinic inventories; (b) an independent evaluation of VillageReach's impact. We conclude that VillageReach's pilot project was effective in increasing vaccine coverage.
Delivery of vaccines and medical supplies
A key component of VillageReach's model is a shift from a "collection-based" to a "delivery-based" supply system: rather than clinics' being responsible for picking up their own supplies, VillageReach's logistics team delivers supplies and provides other logistical support.
The tables below provide data on the goods VillageReach delivered to Cabo Delgado between 2004 and 2007. (Note that the project began in April 2002; we aren't sure why data has not been provided pre-2004.)
Vaccine doses
| Vaccine type | August – December 2004 | 2005 | 2006 | January – April 2007 | Total |
|---|
| BCG | 39,000 | 129,200 | 131,260 | 43,700 | 343,160 |
| DTP-HepB | 37,640 | 173,310 | 171,330 | 47,020 | 429,300 |
| Polio | 87,800 | 244,480 | 303,730 | 79,020 | 715,030 |
| Measles | 19,580 | 61,210 | 77,130 | 18,390 | 176,310 |
| Tetanus | 42,630 | 182,720 | 189,230 | 61,290 | 475,870 |
Injection equipment and gas
| Equipment/Gas | August – December 2004 | 2005 | 2006 | January – April 2007 | Total |
|---|
| Syringes (0.5 ml) | 52,793 | 134,757 | 60,073 | 18,202 | 265,825 |
| Syringes (0.05 ml) | 7,769 | 25,523 | 20,570 | 7,696 | 61,558 |
| Syringes (5ml) | 991 | 15,209 | 6,684 | 2,650 | 25,534 |
| Safety boxes | 906 | 1,517 | 2,105 | 22 | 4,550 |
| Gas (in cylinders) | 485 | 2,091 | 2,286 | 692 | 5,554 |
| Gas (in Kg) | 2,668 | 11,501 | 12,573 | 3,806 | 30,548 |
Increases in immunization coverage
The charts below show that (a) the number of children receiving DTP-3 (third dose of diphtheria-tetanus-pertussis vaccine) immunizations increased; (b) the number of children who "dropped out" during the DTP-3 sequence - that is, they received one, but not all doses - fell; (c) reported "stock-outs" - centers with no inventory of the vaccine - fell significantly over the course of VillageReach's pilot project, which ran from April 2002 to March 2007.. Charts are taken from Kane 2008.
Were improvements attributable to VillageReach?
The evaluation of the project notes that during this time period, "Mozambique and most sub-Saharan African countries achieved significant improvements in their DTP-3 coverage, probably due to
GAVI [a global funder of immunization programs] and its support for infrastructure development and provision of new vaccines and safe injection equipment."
To evaluate the question of VillageReach's role in improvements, we look at three types of information:
- Reports from VillageReach about the problems interfering with immunizations in Cabo Delgado before their arrival.
- An evaluation report published by VillageReach explicitly focused on addressing this question, comparing the change in immunization coverage rates in Cabo Delgado to that of another province in Mozambique, Niassa.
- Examining the changes in immunization coverage in several countries in Africa over this period to put the observed change in context.
We also observe that the charts showing "stock-outs" above have stock-outs falling from a very high level prior to the start of the project to a very low level shortly after the project began. While it is possible that this change occurred for some reason other than VillageReach's involvement in the area, such an outcome seems unlikely.
VillageReach reports of pre-arrival conditions: VillageReach provides a report of the obstacles to immunization coverage in Cabo Delgado before its arrival. We would prefer to have better documentation of these conditions, but nevertheless, we believe the report offers some support to the idea that VillageReach's services were needed in Cabo Delgado.
VillageReach reports,
Prior to the [project], health facilities were individually responsible for picking up vaccines and supplies from the DPS cold stores or district stores and taking them back to their health facility for use. This system encountered various problems....
- Intermittent closing of health facilities during business hours so health workers could pick up vaccines and supplies.
- Challenges securing transport to go to the DPS cold stores. Each district generally had one vehicle, which was for all health service trips by all health system personnel, and was also the ambulance in case of emergencies. Often, when the vehicle was needed to pick up or deliver vaccines, it was out on an emergency, in use by someone else for some other health-system function, broken down, or out of gas.
- Difficulty maintaining proper vaccine temperatures during transport.
- Uncoordinated vaccine supply requirements.
- Frequent stock-outs of vaccines in health facilities.
- Funds were often liberated late – both quarterly from the provincial level to the districts, and monthly from the district administrator to the PAV Chief who needed to purchase gas for the refrigerators, fuel up the district vehicle, and pick up and distribute vaccines.
In 2002, before starting work, VillageReach performed an assessment of access to vaccines in Cabo Delgado province. This report claims that, in 2002, there were 22 health facilities in the three districts in Cabo Delgado that VillageReach assessed. Of these 22, 4 did not offer access to vaccination services: two facilities because they did not have access to a cold chain; 1 because it lacked personnel; and 1 for other reasons.
VillageReach evaluation document: VillageReach compared improvements in vaccine delivery in Cabo Delgado to improvements in the nearby Niassa province, which was not served by VillageReach.
- Baseline data collection: Because VillageReach did not have baseline data (i.e., data from a time prior to the start of the project) available for Niassa, it used data from the 1997 and 2003 Demographic and Health Surveys, which surveyed a large number of households and provided data on DTP-3 coverage.
- Outcome data collection: Evaluators randomly selected households in the treatment area (Cabo Delgado province), including 474 children. Evaluators then also randomly selected households in the comparison area (Niassa province), thereby including 571 children in the evaluation.
- Results: The study found that DTP-3 coverage rates increased substantially in both provinces during this period, but rates improved more in Cabo Delgado (treatment area) than Niassa (control area). The chart below shows the change in coverage rates in the two areas during this time period; the 1997 and 2003 data points are based on Demographic and Health Surveys, while the 2008 data points are from VillageReach's independent data collection.
On one hand, we believe that this chart creates an inflated picture of VillageReach's impact. We have reason to believe that there were significant improvements in immunization coverage between 1997-2001 that were more related to Cabo Delgado's recovery from the aftermath of a civil war than to VillageReach's activities. However, the jump to extremely high levels of coverage as of 2008 - a change not mirrored in the nearby province - give some reason to attribute impact to VillageReach.
The evaluation report is forthright about many limitations of this comparison analysis, including limited sample size, uncertainty about the appropriateness of Niassa as a "comparison province," and issues with taking baseline and endpoint data from different sources. However, it says that its way of comparing the results of surveys done with different methodology "is consistent with international practice," and it concludes that it appears that the Project is responsible for the immunization coverage rising more in the treatment province, Cabo Delgado, than in the comparison province Niassa; although "additional information about the conditions in Niassa compared to those in Cabo Delgado is needed to better understand and interpret the comparison data." It also notes that "it is ... unlikely that the activities of other NGO’s, which are not very involved in immunization activities in Cabo Delgado, were responsible for the improvement." (Note, however, that it does
not discuss the confounding effect of the civil war recovery that we discussed above.)
Our comparison of Cabo Delgado to other areas in the developing world: We took a broader look at changes in African immunization coverage over the time period in question in order to further investigate the idea that Cabo Delgado's improvements may simply have reflected a wider phenomenon. Using the Demographic and Health Surveys (Measure DHS), we collected data on DTP-3 immunizations for countries in Sub-Saharan Africa. The table below summarizes this data, sorted by the country's arithmetic percentage change in immunization coverage.
| Country | First year | Last year | % immunized: first year | % immunized: last year | Change |
|---|
| Mali | 1995 | 2006 | 38% | 68% | 30% |
| Ghana | 1993 | 2008 | 62% | 89% | 26% |
| Senegal | 1992 | 2005 | 59% | 78% | 20% |
| Niger | 1992 | 2006 | 20% | 39% | 19% |
| Cameroon | 1991 | 2004 | 47% | 65% | 18% |
| Burkina Faso | 1993 | 2003 | 41% | 57% | 16% |
| Namibia | 1992 | 2006 | 70% | 83% | 14% |
| Mozambique | 1997 | 2003 | 60% | 72% | 12% |
| Madagascar | 1992 | 2003 | 54% | 61% | 8% |
| Tanzania | 1992 | 2004 | 80% | 86% | 6% |
| Zambia | 1992 | 2007 | 77% | 80% | 3% |
| Nigeria | 1990 | 2008 | 33% | 35% | 3% |
| Chad | 1996 | 2004 | 20% | 20% | 1% |
| Benin | 1996 | 2006 | 67% | 67% | 0% |
| Rwanda | 1992 | 2005 | 91% | 87% | -4% |
| Malawi | 1992 | 2004 | 89% | 82% | -7% |
| Kenya | 1993 | 2003 | 87% | 72% | -15% |
| Zimbabwe | 1994 | 2005 | 85% | 62% | -23% |
This table provides information at a country level rather than province level, and variation
within countries could be significant. There does not appear to be strong evidence of a continent-wide positive trend in immunization rates, but it does not appear, on its own, to rule out the idea that the observed change in Cabo Delgado
purely reflected a broader (country-wide or continent-wide) change. We note, however, that Cabo Delgado's 2003 coverage rate was slightly below the Mozambique overall rate, while its post-project rate was above the Mozambique overall rate (and above every other country's overall rate).
Bottom line on the Cabo Delgado program between 2001-2008
We do not feel that any of the pieces of evidence above is highly compelling by itself. But we are persuaded of VillageReach's impact by the combination of the observations that VillageReach's program (a) entered an area with clearly documented logistics problems; (b) reduced stockouts - one of the clearest measures of the logistics improvement it was aiming for - to near-zero levels; (c) brought Cabo Delgado from an "average" (for the country) level of coverage to an extremely high level of coverage; (d) was reported not to have been supplemented by other nonprofits' programs.
Hand off of the Mozambique projects
Cabo Delgado
Based on the evidence above, we feel that the VillageReach program improved capacity to deliver vaccines in Cabo Delgado. However, the Cabo Delgado project officially became the local government's responsibility in 2007, and a later report stated:
The data suggests that following the discontinuation of field coordinator teams delivering supplies and performing supervision, the districts and health centers are having difficulty reliably picking up supplies, stock-outs of vaccines are beginning to occur again, there is some (not statistically significant) evidence that immunization coverage is beginning to fall, and district level budgets are not being maintained for these activities.
VillageReach's representatives stated to us, "When we turned it over to the government they let it slide back into the old collection-based system."
The fact that VillageReach monitored the program's continuing performance, and has been open about setbacks, is encouraging; but news of program deterioration is cause for concern.
As discussed
above, VillageReach has now reactivated its support role in Cabo Delgado, and conducted a baseline survey there.
Ultimately, we are skeptical about VillageReach's ambitions of handing over its model to the government. However, we note that VillageReach could be making lasting differences in individuals' lives even if its effects on health care are only temporary, since 1-3 doses of most vaccines are sufficient to immunize children against diseases. (Details in our
report on immunization.) Our recommendation of VillageReach is made under the assumption that it will
not succeed in getting its model adopted by the government, while recognizing that its ultimate cost-effectiveness would be much higher if it could.
Nampula
As described
above, VillageReach briefly worked in the Nampula province of Mozambique before handing its activities off to a local organization in January 2007.
VillageReach provided us with internally collected data from this project through August 2008, and stated to us that the data became unreliable (due to internal contradictions) after that point. We have not received clearance to post the data publicly. Overall, it showed encouraging trends that resemble the trends outlined above: increasing numbers of immunizations and substantial drops in the rates of stockouts and other logistical problems. However, because of the fact that the data terminates at an apparently arbitrary point, we have serious doubts about the impact of this project.
This project's expenses, overall, were equal to about 20% of the expenses associated with the pilot project (details in our
cost-effectiveness section). Based on multiple conversations with VillageReach representatives, it is our impression that the very limited role VillageReach took is not representative of VillageReach's typical or future activities.
Project scale-up in Mozambique
Project plans
As discussed
above, the current and planned future activities in Mozambique will be fundamentally similar to the pilot project, and evaluation will be similar as well, in that VillageReach will conduct before and after surveys of immunization coverage, and plans to track a set of key indicators such as deliveries and "stock-outs" over time, as it
did with its pilot project.
The full six-year plan does not call for surveys in a "comparison province" without VillageReach's support (which the original evaluation had). However, they do call for surveys of immunization coverage in each province, both when VillageReach enters and when it exits. VillageReach will be entering different provinces in different years, and depending on exactly when implementation and measurement end up occurring, we feel that the final figures could be quite revealing even without "comparison" provinces.
Because there has only been one demonstrated success, these activities should be considered to have a reasonable risk of failure, but they are - to us - clearly good investments because they are highly similar to activities that have worked before, and we believe VillageReach has made a credible commitment to continue documenting their success or failure.
Progress to date
VillageReach shared with us a 2010 baseline vaccination coverage survey from Cabo Delgado and a six-month report on progress on key indicators. It has told us that it has also conducted a baseline survey in a second province, Niassa, though we have not yet seen this. (VillageReach earlier told us that baseline surveys would be publicly available for 3 provinces by year-end 2010, but is behind schedule on project roll out - reasons for this are discussed
above.) From these documents, we note:
- Data collection seems to have improved, despite a major data collection issue in October 2010 that VillageReach has informed us may have been due to an ongoing election.
- Other indicators, including the particularly important "stockouts" indicator (VillageReach aims to reduce the frequency with which health centers run out of vital medical supplies), don't show noticeable improvement. We don't find this cause for major concern, as VillageReach's successful (in our view) pilot project also didn't show major improvement at the six-month mark (above).
- A survey has been conducted in Cabo Delgado to assess the existing level of vaccination coverage. Overall immunization has fallen only slightly since the 2008 end-of-project survey for VillageReach's pilot project in the province (which ended in 2007), but it has fallen significantly for children under the age of 12 months. The fact that the overall coverage rate (which includes some people immunized under VillageReach's model) has fallen slightly while the under-12-month coverage rate (only those immunized recently) has fallen a lot is consistent with the hypothesis that the switch away from VillageReach's model has led to inferior immunization coverage (though the previous, more effective system still affects the overall coverage rate).
- There appears to have been a problem with the flow of funds from a funder to the government. In the short run, this has resulted in VillageReach's lending money to the government to cover for a delay. This problem does not appear to directly involve VillageReach, and VillageReach representatives told us they are unclear on the exact nature of the problem, but they stated that they expect a relatively quick resolution (within a few months) and have included a mention of the issue in the six-month evaluation in the hopes of raising the profile of the problem and raising the odds of its quick resolution.
- Supply of pentavalent vaccine (which includes diphtheria-tetanus-pertussis, the vaccinations we focus on in our calculation of lives saved) has become disrupted due to a global problem. VillageReach provided this quote from a GAVI (The Global Alliance for Vaccines and Immunisation) representative:
Yes, there is a global shortage of pentavalent vaccines which is expected to be resolved by Q3 of 2011. Countries may be required to use their buffer stocks this year. UNICEF is trying their best to manage the situation. The shortage has occurred due to withdrawal of WHO pre-qualification status of one of the suppliers last year due to quality issues.
Unfortunately, this supplier has not been successful in regaining that status. New suppliers have been pre-qualified by WHO this year but they require lead time to produce the volume necessary to match the demand. So, in short, Mozambique is not the only country affected but all GAVI countries are affected by this.
This issue may be partly responsible for the particularly high stock out rates for this particular vaccine.
- Overall, we found the provided materials to be exceptionally clear and straightforward about VillageReach's progress, struggles and future plans.
- The six-month report does not present a chart of each of VillageReach's core indicators; for example, rather than showing the percentage of health centers with a stock-out of at least one vaccine (which is VillageReach's official definition of its "stock outs" indicator), it breaks out stockouts by individual vaccine and does not include data on non-vaccine supplies. VillageReach made the case to us over the phone that the charts shown are more informative at this stage of the project, while still making it clear that there has not been progress on key indicators to date, and we find this a reasonable argument, but we have requested that in the future, VillageReach chart all standard indicators along with whatever other charts it finds to be informative.
Contract engagements
We are less positive on VillageReach's
contract engagements, many of which we have very little information about. The South Africa project that VillageReach must raise $100,000 in "matching donations" for is particularly worrisome to us, as we find the proposal for activities and impact assessment relatively vague.
Possible negative/offsetting impact
As stated at our
discussion of impact analysis, we are generally concerned about charities' potential diversion of skilled labor and/or interference with government responsibilities. However, we believe these concerns are smaller with VillageReach than with other charities we've seen.
VillageReach's focus is on improving logistics rather than on increasing the available resources in an area. Its cost analysis argues that its program ultimately ends up saving the government money (more below), and a conversation with its representatives implies that it does not attempt to repurpose skilled labor from other areas or sectors. In addition, it appears to be seriously committed to handing off its programs to the government over time, as it did previously in Cabo Delgado. It does not appear to grant funds directly to governments.
What do you get for your dollar?
Past cost-effectiveness: pilot program
We do not attempt to quantify the full benefits of the VillageReach program. Instead, we observe that even a relatively conservative estimate of its cost per child vaccinated would imply quite strong cost-effectiveness (in terms of cost per death averted).
The
Disease Control Priorites report (Jamison et al. 2006) estimates the cost per fully-immunized child with a basic set of vaccines at $14.21 in sub-Saharan Africa. According to Jamison et al. (2006), this implies a cost per death averted of approximately $200.
- VillageReach's estimate: VillageReach sent us a draft of its internal review of the cost effectiveness of the Cabo Delgado pilot project. VillageReach estimates that its program is significantly more cost-effective than the government's program, at a cost of $5.76 per child receiving three doses of each DTP and hepatitis B vaccines (which VillageReach asserts is a proxy for a fully immunized child), including both VillageReach and government costs. This would imply a cost-per-death averted that is significantly lower than the DCP's estimate of $200, and thus easily within the range we consider highly cost-effective.
- GiveWell's conservative estimate: We assumed that all VillageReach costs are attributable to the Cabo Delgado project (including costs associated with Nampula activities, due to our uncertainty about these activities' impact), and that there is no impact of VillageReach on immunization coverage beyond 2008 (the last year for which we have data). We also assumed that starting in 2003, the difference in immunization coverage between Cabo Delgado and Niassa can be attributed to VillageReach's program. These assumptions yield an estimate of one additional child fully immunized for every ~$41 of VillageReach's expenses. If the Disease Control Priorities Report is correct to estimate that $15 per fully immunized child corresponds to $200 per death averted, this would imply that VillageReach is averting a child death for every ~$545 it spends, still well within the range discussed on our overview of cost-effectiveness estimates. (This estimate ignores government costs entirely, in order to give a sense of what is accomplished for donor money.)
DCP2"); ?>
Cost-effectiveness of future activities?
Further activities in Mozambique
VillageReach has provided estimates of the "incremental children vaccinated" projected for its six-year project. These assume that "Total children forecasted to be vaccinated with new system is 80% in the first year, 85% in the second year, and 90% in the third year. The exception is Cabo Delgado where it is 90% for all 3 years." These assumptions do not strike us as overaggressive, seeing as the pilot project achieved 95% coverage in Cabo Delgado (see above). Also note that these estimates count only incremental children vaccinated
while VillageReach is active in a given area, and thus could substantially underestimate impact if VillageReach's work has lasting effects (as intended).
We have calculated the "cost per additional child vaccinated" based both on the Mozambique-only costs and on VillageReach's overall costs as an organization, excluding contract engagements (i.e., all activities that unrestricted funds support).
The aggregate cost-per-vaccination over the duration of the project is $31.88 looking at only Mozambique costs, and $58.28 when including other costs. If, as discussed above, $15 per additional immunized child corresponds to $200 per life saved, these two estimates would imply around $400 or $800 per life saved, respectively. Note that the latter estimate assumes that VillageReach activities such as IT development will have zero impact aside from its contribution to VillageReach's main logistics program in Mozambique.
The baseline survey in Cabo Delgado found existing coverage in-line with the VillageReach's projections that we have used in our cost-effectiveness estimate. Since the project has just begun, we don't know whether the project will come in over/under budget. We'll continue to watch these figures and update the review as new information becomes available.
As discussed
above, VillageReach's current priority for unrestricted funds is its work in Mozambique. As of April 2011,
we believe that VillageReach has "room for more funding" for the Mozambique project of approximately $1.03 million in above-expectations revenue in 2011, and we would cease recommending it to donors if it reached that threshold. Note that VillageReach's expected revenues do not include donations that might result from GiveWell's research.
Details:
- VillageReach stated in February 2011 that "in order to commit to starting work in Inhambane in early FY2012 (i.e. Jan-March 2012 time frame), we would like to recognize funding by December 2011 that supports the program for the subsequent 12 months." According to our calculations, this would require that VillageReach receives about $200,000 in unrestricted 2011 revenue, and as of April 2011, we had tracked over this amount (about $360,000) in unrestricted donations since the start of the year.
- VillageReach hopes to expand to three more provinces as well, beginning in late 2012. Doing so will require an additional ~$3.2 million in funding; incorporating conservative projections of incoming revenue, we estimate an overall funding gap of ~$1.03 million. VillageReach states that additional revenue of $500,000-$1,000,000 would accelerate discussions with these additional provinces.
In discussions with VillageReach, we have learned more about the factors that go into expanding to a new province, aside from funding. In addition to funding its own operations and getting sign-off from the local health authorities, VillageReach must ensure that there is sufficient government funding earmarked to helping to carry out its program and that it has sufficient capacity of its own, which can involve a hiring process. VillageReach representatives have stated to us that these factors are relatively straightforward to address when funding on its own end is available, but these issues are worth keeping in mind as potential non-financial bottlenecks to expansion.
It appears that donors who gave to VillageReach as a result of GiveWell's recommendation have played a significant role in removing certain financial barriers. Our understanding is that VillageReach's work in Gaza and Inhambane has been made possible by such donors. VillageReach received about $2 million in unrestricted revenue in 2010, of which a little over $1 million can be attributed to GiveWell's recommendation.
Financials/other
All data comes from VillageReach's IRS form 990s for 2002-2008 and its audited financial statements for 2009 and 2010.
Revenue and expense growth (
about this metric): VillageReach reached a large five-year, $3.3 million grant agreement with the Gates Foundation in 2004, which explains the large jump reported revenues in 2004.
Both revenues and expenses have grown since a low point in 2007 (possibly due to the fact that VillageReach had completed its work in Mozambique by that time and was largely focused on reviewing and evaluating that project).
Assets-to-expenses ratio (
about this metric): VillageReach's assets to expenses ratio was low in 2009-2010. At the end of the 2010 fiscal year (which ends in September), VillageReach had only half a year's expenses on-hand. However, as discussed
above, VillageReach raised substantially more revenue than expected in the final months of 2010, primarily as a result of GiveWell's recommendation.
Expenses by program area (
about this metric): See discussion
above.
Expenses by IRS-reported category (
about this metric): VillageReach maintains a reasonable "overhead ratio," spending approximately 75-85% of its budget on program expenses.
Sources
- Barrett, Leah. VillageReach Program Manager. Email exchange with GiveWell (PDF), June 2009.
- Beale, John. VillageReach Director of Strategic Development. Email to GiveWell, December 23, 2009.
- Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), June 21, 2010.
- Beale, John. VillageReach Director of Strategic Development. Email to GiveWell (PDF), July 2, 2010.
- Beale, John, Allen Wilcox, and Becca Miller. VillageReach Director of Strategic Development, President, and Finance and Program Administration Manager. Phone conversation with GiveWell, May 21, 2009.
- Bill and Melinda Gates Foundation. VillageReach. http://www.gatesfoundation.org/Grants-2004/Pages/VillageReach-OPP30874.aspx (accessed April 26, 2010). Archived by WebCite® at http://www.webcitation.org/5pHHJonys.
- GAVI Alliance. Country information: Mozambique. http://www.gavialliance.org/performance/country_results/index.php?countID=46 (accessed April 15, 2011). Archived by WebCite® at http://www.webcitation.org/5xxyuAW9t.
- GiveWell. Expanding immunization coverage for children.
- Government of Mozambique. Phase 2 Proposal to GAVI Alliance (2006) (PDF).
- Jamison, Dean T., et al., eds. 2006. Disease control priorities in developing countries (PDF). 2nd ed. New York: Oxford University Press.
- Kane, Mark. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: An independent review for VillageReach with program and policy recommendations (PDF). Seattle: VillageReach.
- Leach-Kemon, Katie, Mariana Dionísio, and Nelia Taimo. 2008. Evaluation of the project to support PAV (expanded program on immunization) in northern Mozambique, 2001-2008: Statistical analysis (PDF). Seattle: VillageReach.
- Miller, Becca. VillageReach Finance and Program Administration Manager. Email exchange with GiveWell, December 23, 2009.
- Miller, Becca. VillageReach Finance and Program Administration Manager. Email to GiveWell (PDF), December 18, 2009.
- Measure DHS. Statcompiler. http://www.statcompiler.com (accessed June 30, 2009). Archived by WebCite® at http://www.webcitation.org/5tvz9ansa.
- UNICEF. Mozambique: Immunization plus. http://www.unicef.org/mozambique/activities_1869.html (accessed April 15, 2011). PDF "print out".
- VillageReach. About VillageReach. http://villagereach.org/about-us/about-villagereach/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pCYUtlLc.
- VillageReach. Audited financial statements:
- VillageReach. Board of Directors and Advisors. http://villagereach.net/about-us/board-of-directors-advisors/ (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDGzQgrO.
- VillageReach. Cabo Delgado: Six-month process evaluation (February 2011). VillageReach has not yet cleared this document for public consumption.
- VillageReach. Cabo Delgado: Vaccine coverage baseline survey (April 2010).
- VillageReach. Comparison of costs incurred in dedicated and diffused vaccination logistics systems: Cost-effectiveness of vaccine logistics in Cabo Delgado and Niassa provinces, Mozambique. Summary available (PDF). VillageReach has asked us not to post the full text online.
- VillageReach. Cost estimates (August 8, 2009) (XLS).
- VillageReach. Field programs. http://villagereach.org/what-we-do/field-programs/ (accessed July 27, 2010). Archived by WebCite® at http://www.webcitation.org/5rXZ0bVIB.
- VillageReach. Five year project report (PDF).
- VillageReach. Funding gap memo (PDF).
- VillageReach. Health system strengthening in Mozambique (PDF).
- VillageReach. IRS form 990:
- VillageReach. John Snow proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
- VillageReach. Key indicator descriptions (PDF).
- VillageReach. Medicines for Malaria Ventures proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
- VillageReach. Milestones (PDF).
- VillageReach. Mission report - VillageReach: Logistics support to health services - MISAU Mozambique (PDF).
- VillageReach. Monthly key indicator tabular report (PDF).
- VillageReach. Mozambique budget with GiveWell additions (February 2011, updated April 2011) (XLS).
- VillageReach. Mozambique expansion budget (JPG).
- VillageReach. Mozambique funding gap (February 2011) (XLS).
- VillageReach. Mozambique revised project plan (February 2011) (XLS).
- VillageReach. Mozambique update (February 2011) (DOC).
- VillageReach. Nampula indicators. We have not received permission to publish this document.
- VillageReach. Nigeria budget. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
- VillageReach. Northern Mozambique project. http://web.archive.org/web/20080726084956/http://www.villagereach.org/MOZ_project.htm (accessed January 11, 2010).
- VillageReach. Organization budget (2010) (PDF).
- VillageReach. President's report (September 12, 2009). Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
- VillageReach. Revenue attribution (2010) (XLS).
- VillageReach. South Africa proposal. Currently withheld due to confidentiality request due to discussion of pending contracts. Interested individuals should contact VillageReach.
- VillageReach. Supply chain. http://web.archive.org/web/20080630001351/www.villagereach.org/supply_chain.htm (accessed January 11, 2010).
- Wilcox, Allen, and Becca Miller. VillageReach President, and Finance and Program Administration Manager. Phone conversation with GiveWell, November 10, 2009.
- World Health Organization. Glossary. http://www.who.int/immunization_monitoring/glossary/en/index.html (accessed April 23, 2010). Archived by WebCite® at http://www.webcitation.org/5pDJPLQ56.
