Against Malaria Foundation (AMF) - 2014 Review

We have published a more recent review of this organization. See our most recent report on AMF.


The Against Malaria Foundation (AMF) is one of our top-rated charities, and an organization that we feel offers donors an outstanding opportunity to accomplish good with their donations.

More information:

Published: November 2014

Donate

Summary

What do they do? AMF provides funding for long-lasting insecticide-treated net distributions (for protection against malaria) in developing countries.

Does it work? There is strong evidence that distributing nets reduces child mortality and malaria cases. AMF has relatively strong reporting requirements for its distribution partners and provides a level of public disclosure and tracking of distributions that we have not seen from any other net distribution charity. AMF has begun building a track record of funding and tracking fairly large-scale ($1 million or more) distributions.

What do you get for your dollar? We estimate that the cost to purchase and distribute an AMF-funded net is $5.30 in Malawi and, very roughly, $7.50 in DRC (the two countries that AMF has worked most extensively in). The numbers of malaria cases prevented and lives saved are a function of a number of difficult to estimate factors, which we discuss in detail below.

Room for more funds? AMF currently holds approximately $6.8 million in funds that it could allocate to future distributions; this includes approximately $2 million committed to a distribution scheduled for 2017. We believe AMF could conceivably absorb as much as $25 million more in the next year. Considering AMF’s track record of finalizing distributions and the past scale of its operations, we hope to see AMF receive an additional $10 million.

AMF is recommended because of its:

  • Focus on a program with a strong track record and excellent cost-effectiveness.
  • Strong processes for ensuring that nets reach their intended recipients and monitoring whether they are used over the long-term.
  • Standout transparency and commitment to self-evaluation.
  • Room for more funding – we believe AMF will be able to use additional funds to deliver additional nets.

Major unresolved issues include:

  • Though we now have increased confidence in AMF's ability to find new partners and countries to work in, there is still significant uncertainty as to its ability to scale up.
  • AMF has completed only a small number of large distributions, the type it aims to focus on in the future, and all completed distributions have been with a single partner. Its track record of collecting the data it seeks from these distributions is good so far but limited due to the small number of distributions. Many of its upcoming distributions are in DRC and our impression is that DRC is a particularly difficult place to work.
  • The best evidence for nets was collected before they were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in nets, possibly making them less effective. Further research is needed to determine how important of a problem this may be.

Table of Contents

Our review process

We began reviewing AMF in 2009. Our review has consisted of:

  • Reviewing AMF's public records for each of its net distributions and other documents AMF has shared with us.
  • Extensive communication, including several meetings at AMF's London headquarters with AMF Founder Rob Mather, to discuss AMF's methods and funding needs.
  • A visit to AMF's distribution partner organization, Concern Universal, in Malawi in October 2011 (notes and photos from this visit).
  • Reviewing materials from AMF’s distributions in Malawi and the Democratic Republic of the Congo (DRC), including pre-distribution registration data, distribution reports, and data from post-distribution follow up surveys.
  • Conversations with Peter Sherratt AMF's Executive Chairman; Don de Savigny, a member of AMF's Malaria Advisory Group, and other individuals (who requested to remain anonymous) familiar with AMF's work and its attempts to finalize distributions.


All content on AMF, including past reviews, updates, blog posts and conversation notes, is available here.

A few key terms for this review

  • A net refers to any bednet intended to block mosquitoes, whether treated with insecticide or not.
  • An ITN is an insecticide-treated net. As discussed at our write-up on ITNs, distribution of ITNs has been rigorously studied and associated with declines in under-5 mortality and overall malaria cases.
  • A LLIN is a long-lasting insecticide-treated net, intended to serve as an effective ITN for 4-5 years (details at our write-up on ITNs). The nets distributed by AMF are LLINs,1 and (as noted in our write-up on large-scale ITN distributions) most large-scale donor-financed malaria control today utilizes LLINs.

What do they do?

AMF provides long-lasting insecticide-treated nets (for protection against malaria) in bulk to other organizations, which then distribute them in developing countries.

Prior to November 2011, AMF contributed LLINs to small-scale distributions. Since then, it has de-emphasized small-scale distributions and it expects larger-scale distributions to be the primary way in which it distributes LLINs in the future (more below).

As of November 2014, AMF had completed or was in the process of completing four large-scale distributions with two distribution partners for a total of 1.3 million LLINs (2.7 million total since its inception),2 and had signed agreements to distribute an additional 2.3 million LLINs in 2014-2017.3

AMF large-scale distributions
Location Distribution partner Number of LLINs Cost to AMF (millions) Timing
Ntcheu, Malawi Concern Universal 268,430 $1.13 Completed: December 2011-March 2012
Balaka, Malawi Concern Universal 154,230 $0.71 Completed: October-November 2013
Dedza, Malawi Concern Universal 245,000 $1.01 Completed: September-October 2014
Kasaï Occidental, DRC IMA World Health 676,000 $1.94 In progress: September-November 2014
North Idjwi Island, DRC Amani Global Works 62,000 $0.19 Planned: November-December 2014
Dowa, Malawi Concern Universal 396,900 $1.50 Planned: January-February 2015
South Idjwi Island, DRC Amani Global Works 62,000 $0.19 Planned: April-May 2015
Nord Ubangi, DRC IMA World Health 730,000 $2.34 Planned: April-June 2015
Balaka, Malawi (round 2) Concern Universal 235,000 $0.94 Planned: September-October 2015
Ntcheu, Malawi (round 2) Concern Universal 335,000 $1.34 Planned: October-November 2015
Dedza, Malawi (round 2) Concern Universal 430,000 $1.72 Planned: October-November 2017
Total completed/in progress 1,343,660 $4.79
Total planned 2,250,900 $8.22
Total 3,594,560 $13.01

Our most thorough review of AMF's processes was done for its first large-scale distribution in Ntcheu, Malawi, and much of the below discussion relates to that distribution. Since then, we have followed AMF progress and asked for updates on how its processes have changed rather than consider each distribution in depth.

AMF's role

AMF's role in LLIN distributions is to:4

  1. Identify countries with funding gaps for LLINs.
  2. Find distribution partners to carry out LLIN distributions and agree on expectations for the distribution, including who pays for costs other than the purchase price of LLINs and what information will be collected and shared with AMF.
  3. Purchase LLINs and have them shipped to the distribution partner.
  4. Work with distribution partners to collect reports on the distribution and follow up surveys. AMF posts these reports on its website.

Process for distributions

Selecting locations for distributions

  • When selecting locations for future distributions, AMF told us it consults a series of sources as it believes there is no one reliable resource with up-to-date information to determine where nets are needed. Sources it consults include the Alliance for Malaria Prevention's (AMP's) list of countries with significant net gaps, other malaria control funders, in-country technical advisors, the relevant National Malaria Control Program (NMCP), implementing organizations and the African Leaders Malaria Alliance.5

    As an example, we discuss the case of its distribution in Ntcheu, Malawi in 2012. For that distribution, AMF received estimates of country-level gaps from the Alliance for Malaria Prevention (AMP) and from the Malawi National Malaria Control Program (NMCP). Rob Mather, AMF's founder, told us that, although AMF did not have a high level of confidence in the net gap numbers presented by the NMCP, nor those it was sent by the Global Fund, it did have a high level of confidence that the net gap was significantly higher than the 250,000 LLINs AMF was considering providing.6 Note that the materials AMF has sent us from the period prior to its decision to provide funding for nets in Malawi do not include quantified estimates of Malawi's LLIN gap,7 though they do make it clear that there is a gap,8 and later reports (from after AMF had decided on Malawi) show a quantified gap for the country.9

    Note that we did not repeat this analysis for distributions that AMF funded after Ntcheu, Malawi in 2012 or for the distributions it is considering as of November 2014.

  • AMF told us that it also consults other malaria control funders, implementing organizations, and the relevant National Malaria Control Program (NMCP) to verify estimates of how many additional LLINs are required to reach full coverage in a country.10 We have seen some correspondence between AMF and the NMCP in Malawi which took place prior to AMF’s decision to fund a distribution in Malawi.11 AMF also shared with us correspondence that it had with representatives of the NMCP in Togo, with regard to potential future distributions.12
  • As AMF investigates countries with existing net gaps, it also looks into organizations working within those countries that could serve as distribution partners.13 AMF's distribution partners must have the capacity and willingness to carry out a distribution that meets AMF's requirements (more below).14 Before deciding to work with Concern Universal for the Ntcheu 2012 distribution, AMF requested a distribution proposal, including information on malaria risk in the distribution area (including data on reported malaria cases), other net distributions in the area, coordination with government, and how the distribution would be carried out.15 We have seen similar distribution proposals, which include information on malaria risk and other sources of nets in the area, for distributions in Balaka in 2013, Dedza in 2014, and North Idjwi Island in 2014. We have not yet seen distribution proposals for the in progress (as of November 2014) distribution in Kasaï Occidental or for any future distributions (full list in this spreadsheet).

Requirements for distribution partners

AMF aims to work with distribution partners who will agree to the following requirements. We detail what evidence we have seen that past distributions have complied with these requirements in this spreadsheet.

  • Pre-distribution registration: AMF requires that its distribution partners carry out a pre-distribution registration survey (PDRS) in all areas that LLINs will be distributed to establish the number of sleeping spaces and LLINs in a condition such that they have at least one year of remaining use. (In the case that such data already exist, AMF and the partner discuss whether the data is reliable and up-to-date, or whether a new PDRS should be conducted.16 ) We have seen results from both of AMF's completed large-scale distributions: full data from its Ntcheu 2012 distribution and sample results from its Balaka 2013 distribution.17
  • Distribution monitoring: An additional requirement is to provide AMF with photos and videos from the distribution and a report on the distribution, including problems encountered.18 AMF's distribution partner, Concern Universal, provided photos and/or videos for all but 2 of the 14 “sublocations” for its Ntcheu 2012 distribution.19 It did not collect photos or video for AMF's other completed large-scale distribution, in Balaka in 2013. AMF expects to receive photos and video from future distributions.20 Concern Universal has provided detailed distribution reports for the three distributions it has completed.
  • Post-distribution follow-up: AMF requires partners to conduct follow-up surveys in 5% of households at 6-month intervals for 3-4 years after a distribution.21 We have seen post-distribution survey results from the Ntcheu 2012 distribution (6, 15, and 24 months post-distribution) and the Balaka 2013 distribution (6 months post-distribution). Results below. Results are not yet expected from other distributions.

Previously, AMF expected to collect malaria case rate from the regions in which it funded LLIN distributions. AMF has asked Concern Universal to provide monthly malaria case rate data from all health centers in the three districts in which it has completed distributions: Ntcheu, Balaka and Dedza. AMF has asked for this data for 12 months preceding and 4 years following the distribution.22 AMF has shared data from Ntcheu through through December 2012. We do not believe this data is high quality. In August 2013, it told us that it planned to publish an audit of the data quality by the end of September 2013.23 Later, it told us that it expected to post data in early 2014.24 In November 2014, AMF told us that it expects to share more recent data by the end of January 2015. AMF told us it has been waiting to collect data through to September 2014 to be able to review data for a full year and compare it with the prior year due to changes in the way data is recorded in the health centers.25 AMF has not included in its Agreement with its partner in DRC, IMA World Health, a requirement to collect and share malaria case rate data. Discussions with IMA made it clear the quality of the malaria case rate data was not considered to be reliable in the majority of health centers. IMA told AMF that it believes there will be a some health centres across the distribution area that can provide historic and future malaria case rate data that is reliable. AMF and IMA have agreed to return to this issue in early 2015.26

AMF looks for partners who, in addition to agreeing to the above requirements, are clear and responsive in their communication with AMF.

Prior to the Balaka 2013 distribution, AMF asked all distribution partners to use their own funds or to find another funder for all non-net costs of the distribution.27 More recently, it has paid for some of these costs in certain distributions. AMF told us that it considers funding non-net costs in cases where (a) non-net costs are not covered by other partners, and (b) AMF feels confident that its distribution partners will manage and report on spending well.28 AMF has paid for all non-net costs for the Balaka 2013 and Dedza 2014 distributions and the 24- and 33-month post-distribution check-ups for the Ntcheu 2012 distribution, and agreed to pay for all non-net costs for the Dowa 2015, Balaka 2015, Ntcheu 2015, and Dedza 2017 distributions with Concern Universal, and shipping costs for the Nord Ubangi 2015 distribution.29 It is our understanding that AMF aims to primarily fund distributions where other partners cover non-net costs, which could limit its capacity to distribute nets, though AMF has shown some flexibility on this issue recently.30

Does it work?

On a separate page, we discuss the general evidence behind distribution of LLINs. We conclude that there is strong evidence that these distributions can be expected to reduce child mortality and malaria cases.

When evaluating the effectiveness of an LLIN distribution organization, we seek to answer the following questions:

  • Are LLINs targeted at people who do not already have them? AMF requires partners to complete pre-distribution surveys to determine the number of nets required, if reliable and recent survey data are not available. We are not fully satisfied with the pre-distribution surveys' ability to ensure that people aren't under-reporting their existing LLIN ownership in order to get more LLINs, but believe that reasonable measures are being taken to address this issue.
  • Do the LLINs reach the intended destinations? Distribution partners are asked to provide pictures and videos of LLINs being handed out (a requirement that has been complied with in one of two completed large-scale distributions), and report on how the distribution went, including problems encountered. Post-distribution surveys provide additional evidence that LLINs reached their destinations.
  • Are LLINs targeted at areas with high rates of malaria? AMF seeks out distribution partners in countries that are known to have high rates of malaria.
  • Do those who receive the LLINs install them in their homes properly? Do they utilize them consistently over the long term? AMF requires partners to conduct follow-up surveys at 6-month intervals, and for a period of up to 3-4 years. Results are available from 6, 15, and 24 months after the Ntcheu 2012 distribution and 6 months after the Balaka 2013 distribution. We have little data to compare these results to, but they seem consistent with high, proper usage of nets for an extended period after distribution.
  • Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF? The limited evidence we have suggests the former.

Details follow.

Are LLINs targeted at people who do not already have them?

AMF requires that distribution partners conduct household-level pre-distribution registration surveys (if recent, reliable household-level data are not available) to determine the number of existing LLINs with at least one year of remaining use and the number of nets required in each household.31 This process includes door-to-door surveys of every household in the targeted district by government health workers. We have seen census results from both of AMF's completed large-scale distributions: full data from its Ntcheu 2012 distribution and sample results from its Balaka 2013 distribution.32

In Malawi, Concern Universal has then conducted village meetings in which its staff read off household names and the number of LLINs that have been allocated to each household; households then indicate if any errors have been made. We observed one such village verification meeting during our visit to Malawi in October 2011.33 In DRC, IMA World Health is not conducting village verification meetings because LLINs are being distributed during the census: health workers go door-to-door checking how many LLINs each household requires and provide the LLINs at the same time.34

Concern Universal conducted pre-distribution registration surveys for the Ntcheu 2012 and Balaka 2013 distributions, and AMF shared the full results from the Ntcheu census and sample data from the Balaka census.35 Based on GiveWell's suggestion, Concern Universal also decided to send its staff to random households prior to the Ntcheu 2012 distribution to spot-check the accuracy of its data.36 After the first two weeks of spot checks, the distribution partner reported that it had not found any discrepancies between the number of nets community members said they needed during the verification meeting, and the number of nets found to be needed during the subsequent household visit.37 AMF has told us that spot checks are now carried out with all distributions.38 We have not seen results from these spot checks.

We believe it is possible that some households may over-report the number of LLINs they need. Cases of households hiding nets in order to receive extra LLINs have been observed in universal campaigns in Senegal and Nigeria.39 That said, we believe that the steps AMF and its partners are taking to minimize the ability of community members to request more nets than they need are reasonable.

In Malawi, household surveys conducted by the National Malaria Control Programme (in districts not receiving AMF-funded LLINs) have found "much larger LLIN gaps than first thought,"40 and it is not clear to us what the cause of this discrepancy is.

Do LLINs reach the intended destination?

AMF requires that partners provide photos and videos of distributions.41 Concern Universal provided photos and/or videos for all but 2 of the 14 “sublocations” in Ntcheu in which it distributed LLINs in 2012.42 It did not provide photos and/or video for the Balaka 2013 distribution, the other large-scale distribution that AMF has completed. AMF told us that it expects photos and video from future distributions.43

During its Ntcheu 2012, Balaka 2013, and Dedza 2014 distributions, Concern Universal provided weekly reports on the number of LLINs distributed, the regions which were covered, and problems encountered in reaching particular villages or individuals.44 As we report in our March 2012 update, the Ntcheu 2012 weekly reports note challenges including attempted thefts, double registrations and logistical problems.45 Concern Universal provided a similar level of detail on challenges encountered in the Balaka 201346 and Dedza 2014 distributions.47 These reports increase our confidence that Concern Universal is aware of potential problems and has a system in place to address them.48

Concern Universal has provided weekly reports of its own accord (AMF does not require such reports). AMF does ask partners to provide summary reports on how distributions have gone after they are completed, and we expect to see such reports from future distributions.49

For the Kasaï Occidental (2014) distribution, AMF's distribution partner, IMA World Health is piloting the use of smartphones to record household data, including GPS coordinates, for registration and LLIN distribution. AMF hopes that the technology can be used to increase accountability, transparency, and data accuracy.50 IMA is funding this work.51 AMF has told us that it will publish the data collected (anonymized for privacy).

Post-distribution surveys, discussed below, provide a further check on whether LLINs have reached their intended destinations. The surveys that have been completed to date have found high coverage rates (90% in the two completed 6-month surveys).

Are LLINs targeted at areas with high rates of malaria?

At the highest level, AMF appears to exclusively target countries with known malaria risk.52 In its distribution proposal for its Ntcheu 2012 and Balaka 2013 distributions, Concern Universal reported the number of malaria cases in the proposed distribution area over a twelve month period preceding the distributions.53 We have not seen similar data for Concern Universal's Dedza 2014 or Dowa 2015 distributions or for the areas in DRC where AMF has or will provide LLINs. AMF has told us that it has received malaria case rate data for Dedza and Dowa districts in Malawi (which it has not yet shared publicly) and that it does not have data from Kasaï Occidental in DRC because much of the data from that area is considered to be unreliable. AMF told us that it may in the future collect and publish data from a subset of health centers in Kasaï Occidental that have reliable data.54

Do those who receive the LLINs install them in their homes properly? Do those who receive the LLINs utilize them consistently over the long term?

AMF requires partners to conduct follow-up surveys at 6-month intervals for a period of 3-4 years to determine whether LLINs are present, are hung, and what condition they are in.55 Results are available from 6, 15, and 24 month surveys following the Dedza 2012 distribution and 6 months following the Balaka 2013 distribution. Details of the methodology used in these surveys are available in our August 2014 update on AMF (see the description of the Ntcheu 15 and 24 month surveys). We discuss the methodology of the Balaka 6 month survey in this footnote56 (details were not yet available when we published the August update). We believe the methodology used was reasonably high quality and that results are fairly, though not fully, representative of all households that received AMF-funded LLINs. We note that have not seen technical details at the level of a high quality academic study on how the surveys were carried out,57 and we therefore have not been able to fully vet the results.

Results from follow-up surveys

Data sources in footnote.58

Ntcheu: 6-month Ntcheu: 15-month Ntcheu: 24-month Balaka: 6-month
# of households surveyed 7,689 9,250 8,939 4,536
# of AMF nets received by surveyed households 15,814 18,580 18,448 7,977
% of nets surveyed of total distributed 6% 7% 7% 5%
% of nets hung 90% 85% 81% 87%
% of nets missing 1% 3% 4% 2%
% nets in very good condition 99% 89% 49% 70%
% of nets in OK condition 1% 6% 28% 25%
% of nets in poor condition 0% 2% 15% 4%
% of nets worn out 0% 3% 8% 1%
% of nets used correctly 98% 98% 90% 94%
% of people covered by nets 90% 84% 79% 90%

Definitions:

  • Nets hung: We have not seen a precise definition of this term (e.g., must the surveyor observe the net fully covering the sleeping space, hung in a position that it could conceivably cover the sleeping space, etc.). AMF told us that interviewers are asked to observe whether nets are hung by entering interviewees' houses (rather than simply asking interviewees if they are hung).59
  • Very good condition: LLIN has fewer than 2 holes of less than 2 cm in size.60
  • OK condition: LLIN has fewer than 10 small holes.61
  • Poor condition: LLIN has more than 10 small holes or has one or more large holes.62
  • Worn out: Survey reports from Concern Universal did not provide a definition of this term. AMF told us that an LLIN is considered worn out if it has multiple large holes and the LLIN is unrepairable, such that it would not provide protection against mosquitos.63
  • Used correctly: According to a survey form from 2012, interviewers are asked to "ask the householder to demonstrate how the nets are used at night" and to then select yes or no to the question "are the nets being used correctly."64 AMF told us that the procedure has not changed in more recent surveys.65

79% of individuals in households that received LLINs sleeping under a useable LLIN (at 24-month follow up in Ntcheu) is roughly in line with usage rates of the trials of bednet efficacy documented in our page on Long Lasting Insecticide Treated Nets, in which small-scale studies of bednet efficacy had net usage rates generally in the 60%-80% range.66

The "decay model" we use to anticipate the lifespan of LLINs assumes that 80% of LLINs are effective at 24 months.67 AMF found that 77% LLINs in Ntcheu were in "very good" or "OK" condition at 24 months. LLINs appear to have decayed more quickly in Balaka, where 70% of nets were in "very good" condition at 6-months (and 25% in "OK" condition), compared with 99% in "very good" condition at 6-months in Ntcheu. It is not clear to us whether these rates of decay are comparable to the assumptions in the decay model, in part because it is not clear whether the definitions of useable are similar in the model and AMF's data. We may do further analysis to more directly compare the model and AMF's data in the future.

Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF?

For the Ntcheu 2012 distribution in Malawi, AMF received data from the National Malaria Control Program on population figures and LLINs already distributed by district, as well as LLINs that had been committed to the country by other funders. AMF estimated, in September 2011, that an additional 909,586 nets were needed to achieve universal coverage in the country,68 of which AMF contributed 268,240.69 We have not done a similar analysis for later distributions.

We would guess that AMF is increasing the total number of LLINs because, across Africa, there are substantial funding gaps for LLINs (more below) and because our impression from following AMF's progress over time is that, due to AMF's more limited funding and, perhaps, greater data requirements, governments often seek funding first from larger funders (particularly the Global Fund to fight AIDS, TB, and Malaria) and then may ask AMF to fill gaps. However, we have low confidence in this guess, and note that countries are able to choose how they allocate Global Fund grants among malaria interventions (including LLINs, treatment, and diagnosis), so the availability of funding for LLINs from AMF could cause countries to allocate less Global Fund resources to LLINs.

Possible negative or offsetting impact

  • Do donated nets displace government health funding? This could be a concern if government funding that otherwise would have been spent on LLINs is spent on other, less worthwhile budget items. We have little sense of how important a concern this is in AMF's case. In the case of the 2012 distribution in Malawi, all anticipated nets were expected from international donors, and the total supply appeared to be still insufficient for country-wide universal coverage.70
  • Will insecticide-treated nets continue to be effective? As discussed at our report on insecticide-treated nets, there is strong evidence for the effectiveness of this intervention, however the best evidence for the intervention was collected before LLINs were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in LLINs, possibly making them less effective. In 2012 we reviewed the evidence on the state of insecticide resistance. We concluded, "We're very concerned about insecticide resistance. It appears that relatively little is known about the extent, causes, and control implications of resistance; that resistance could substantially affect (or even negate) the effectiveness of malaria control… That said, this issue doesn't change our bottom line that LLIN distribution is a highly cost-effective intervention… There is strong evidence that LLINs reduce malaria and save lives and only preliminary/suggestive/mixed evidence that insecticide resistance may reduce their impact."
  • Do free LLIN distributions distort incentives for recipients or distort local markets for nets? As discussed at our report on insecticide-treated nets, we feel that there is a reasonably strong case for distributing LLINs freely rather than selling them at market (or even below-market) prices.
  • Could distribution of LLINs be inequitable and unfair, causing problems in the targeted communities? We feel that AMF's process for determining needs for LLINs at the household level is well-suited to ensuring that all who want LLINs receive them.
  • Does AMF divert skilled labor from other areas? Net distribution is conducted by low-level government health staff in partnership with the staff of AMF's partner NGO.71 According to AMF's partner in Malawi, government health staff are normally involved in activities such as disseminating health-related information, reporting on levels of stunting and disease, carrying out immunization campaigns, and providing nutrition support.72 We don't know the extent to which net distribution reduces their ability to complete other duties, though we note that net distributions are generally completed within a few days in each local area.

What do you get for your dollar?

Cost per LLIN distributed

Estimate based on data from AMF

Using budgets and estimates provided by AMF, we estimate that the total cost to purchase, distribute and follow up on the distribution of an AMF-funded LLIN is about $5.30 in Malawi and very roughly $7.50 in DRC. We focus on these two countries because they are where AMF has completed recent distributions or is planning distributions.

Note that we prefer to include all costs incurred to carry out a project, not just those that the charity in question pays for itself. We believe that this gives the best view of what it costs to achieve a particular impact (such as saving a life), and also avoids the lack of clarity and complications of leverage in charity. Our estimate of the cost to distribute an AMF-funded LLIN aims to include both costs AMF pays and costs paid by others.

Full details in this spreadsheet.

The estimate for Malawi uses:

  • The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
  • Actual costs for the distribution and 6-month follow-up survey in Balaka, as reported by Concern Universal, budgeted costs for later follow-up surveys in Balaka, and budgeted costs for all of the non-net costs for the Dedza (2014) and Dowa (2015) distributions.73 AMF told us that the non-net budget includes all monetary costs incurred by Concern Universal, as well as the use of vehicles and managerial staff time;74 it may exclude non-monetary costs such as use of office space.
  • An estimate of non-monetary costs incurred by Concern Universal and local governments, based on estimates provided in 2012 by Concern Universal for its distribution in Ntcheu province.75
  • An estimate of AMF's organizational costs per net, including an estimate of the value of donated services and volunteer time. Because these costs generally do not vary with the number of LLINs distributed and remain roughly constant on a per year basis, the per LLIN estimate is highly dependent on the time period used and the number of LLINs distributed in that period. We estimate AMF's organizational costs as $0.82 per LLIN for 2013-2015. This estimate assumes that AMF will complete the distributions it currently plans for that period and does not account for any additional distributions.76

We shared our analysis with AMF before publishing and AMF wrote the following in response.

Response from AMF: Using actual costs to assess cost effectiveness

The AMF view is it more relevant to its donors to look at AMF’s actual costs when calculating its cost-effectiveness, rather than adding in costs where either no cash cost exits or support is provided pro bono.

This matters as not to do so distorts cost-effectiveness figures. AMF’s overhead costs per annum are $120,000. Using this figure the cost per delivered and monitored net would be $4.80.

GiveWell uses an overhead figure of $300,000 due to adding in a CEO salary of $100,000 (AMF’s CEO does not draw a salary) and $80,000 of other overhead costs for services provided pro bono to AMF and therefore where no cash cost exists (the organisations providing pro bono support are very happy to work for no immediate financial return since they benefit in other ways from their involvement, e.g. in improved staff morale. This means they incur no net cost from providing the pro bono service).

As a result, GiveWell calculates the cost per net for AMF at about $5.30.

GiveWell has calculated the cost per net for other organisations as $5.80.

At $5.30 per net, AMF is 9% more cost-effective than other organisations. At $4.80 per net, 17% more effective. That is a material difference.

We believe our donors are more interested in a comparison derived from AMF’s actual costs rather than penalising it (reducing its assessed effectiveness) by not considering the no-cost and pro bono cost savings they may achieve. AMF as an organisation will continue to seek pro bono support believing this is more cost effective than paying for services.

For DRC, the estimate is much rougher. It is based on costs from the Kasaï Occidental (2014) distribution, specifically:

  • The purchase price per LLIN, calculated from AMF's total LLIN budget and the number of LLINs distributed.
  • An estimate of shipping costs. We do not have an estimate of shipping costs for this distribution, so we have used a rough estimate based on the Nord Ubangi 2015 distribution (also in DRC).
  • An estimate of pre-distribution and distribution costs from incomplete data from AMF's distribution partner IMA World Health. This estimate is 2.5 times as high as the Malawi estimate. This may be explained by a considerably more difficult operating environment in DRC,77 or our data for DRC may simply be of lower quality.
  • We do not have data for the cost of follow-up surveys in DRC. We have assumed that the cost of these surveys will be about 2.5 times as expensive as in Balaka, consistent with our estimate of the increased costs to distribute LLINs in DRC.
  • We have used the same estimate of AMF organizational costs as we used for Malawi.

Global estimate

The Roll Back Malaria Partnership, which estimates funding gaps for LLINs across all African countries,78 uses an estimate of $3.30 to purchase a LLIN and $2.50 for all non-net costs of a distribution, for a total of $5.80 per LLIN.79 This assumption is used in cases where country-specific data is not available. The Co-Chair of the Roll Back Malaria Harmonization Working Group told us that non-net costs can vary significantly by country and that non-net costs in DRC may be more than $5 per LLIN.80 We have little information on how reliable this estimate is likely to be. Broadly, this estimate is consistent with the data we have seen from AMF. We have used estimates based on data from AMF in our cost per life saved calculations, because we have more information on their reliability.

Cost per life saved

Using $5.30 as the total cost per net in Malawi and $7.50 for DRC, we estimate the cost per child life saved through an AMF-funded LLIN distribution at about $3,340.81

This does not include other potential benefits of LLINs (non-fatal cases of malaria prevented, prevention of deaths in age groups other than under-5 year olds, prevention of other mosquito-borne diseases, etc.). Full details at our report on mass distribution of LLINs.

As a general note on the limitations to this kind of cost-effectiveness analysis, we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

Room for more funds?

In November 2013, we wrote, in this blog post, that we believed AMF did not have room for more funding because it had raised over $10.6 million due to GiveWell's recommendation since 2011 and had not spent the bulk of these funds. Since then AMF has completed or signed agreements for several large distributions and we now believe that AMF has room for more funding in the next year. This change is primarily due to:

  1. A year ago, AMF had completed two large-scale distributions, both of which were with a single distribution partner, Concern Universal in Malawi. It had not signed agreements with other partners. As of November 2014, AMF has signed agreements for two large distributions with IMA World Health in the DRC (one of which is in progress), helping to demonstrate its ability to finalize distributions with other partners in other locations.
  2. AMF has spent or committed a large portion of the funds it raised prior to 2014. As of November 2014, AMF held $4.9 million in uncommitted funds, of which it raised $2.65 million in 2014. Of committed funds, $1.9 million are for expenses that AMF will incur more than a year from now, and AMF might consider reallocating these funds in the near term with the expectation of raising enough to cover these commitments by the time they are due. This brings the total available funds to $6.8 million.82
  3. AMF's pipeline for potential distributions includes several distributions with established partners. Given AMF's history with these partners, we would guess that there are fewer barriers to signing agreements for additional distributions. Funding all such distributions would require AMF to raise significantly more funds. AMF may also have opportunities to fund distributions with new partners.

Though we now have increased confidence in AMF's ability to finalize large distributions, there is still significant uncertainty as to its ability to find new partners and countries to work with. To find a sufficiently large distribution often requires negotiating with the national malaria control programs of countries in sub-Saharan Africa, which we perceive to have some discretion in which funders they work with, and which we perceive to be choosing funders based on a variety of factors including size and reporting requirements. Because AMF is able to fund only a relatively small piece of a given country’s distribution, but has substantial reporting requirements, there may be fundamental reasons for governments to prefer other funders. We believe that this is the most important and fundamental explanation for AMF's difficulties in finalizing distributions in the past. We believe that other factors may have also played a role. We discuss this topic in detail in this blog post.

AMF told us that it is not in conversations with any major funders to help fill its funding gap.83

Pipeline of potential distributions

As of November 2014, AMF was considering four main types of potential distributions:

  1. Additional distributions with existing distribution partners. Due to ongoing relationships and probable need for additional LLINs, we believe that it is likely that AMF will have opportunities to fund additional distributions with these groups. AMF is considering funding up to $17.2 million worth of LLINs in 2015.
    • Malawi will be carrying out a nationwide LLIN distribution in 2015 and may need additional LLINs.84 AMF is considering funding 600,000 to 1 million LLINs85 for Concern Universal to distribute in this additional district. This would likely cost $2.4-4.0 million,86 including non-net costs (we assume AMF will fund non-net costs as it has for recent Concern Universal distributions).
    • AMF is considering two potential distributions in DRC, where it believes there is a significant funding gap for LLINs in 2015. One distribution would be for 1 to 2 million LLINs and the other would be for 3 million LLINs.87 Purchasing 4 million LLINs and paying for shipping (as AMF has agreed to do for its distribution with IMA in early 2015) would cost about $13.2 million.88
  2. Distributions of moderate size (60,000 to 100,000 LLINs) in three other countries. AMF is considering funding 60,000 LLINs in one country and 100,000 LLINs in two other countries,89 for a total cost of about $780,000.90 AMF told us that it would be able to sign agreements for these distributions quickly if it chose to do so and that the reason it has not is because it expects to use its funds for larger distributions instead. If it is unable to sign agreements for larger distributions, it may go ahead with these more moderately sized ones. It will likely agree to fund the distribution for 60,000 LLINs because the distribution would involve a pilot project to test the use of smartphone technology for tracking LLIN distributions, which AMF is interested in learning more about.91
  3. Large distributions (1-4 million LLINs) in 2-3 other countries. AMF is in fairly early stages of discussion with governments in two countries to fund 1 to 2 million LLINs ($3-6 million) and up to 4 million LLINs ($12 million), respectively, in 2015.92 AMF does not expect to pay for non-net costs in these countries.93 More speculatively, there may be an opportunity for AMF to fund up to 2 million LLINs in a third country in 2015 and up to 2 million more in 2016-2017.94 AMF has not yet begun conversations with the government of the third country.95
  4. Funding research on insecticide resistance. AMF is considering spending approximately $300,000 over three years to fund a UK university to study insecticide resistance in conjunction with AMF's distribution in Nord Ubangi, DRC in 2015. It may also fund a similar project for its Dowa, Malawi distribution in 2015, at a cost of approximately $200,000. This research would (a) establish baseline rates of mosquito susceptibility to the insecticide used on most LLINs and (b) compare the effectiveness of LLINs used in most distributions and a newer type of LLIN that is more expensive but may be more effective at killing mosquitoes.96

If AMF were able to finalize all of the distributions in its pipeline, with the exception of the speculative distribution for 4 million LLINs and the moderately-sized distributions (which it would likely drop in favor of the larger ones), it would need nearly $36 million.97 We believe it is unlikely that all discussions would be successful. AMF currently holds approximately $6.8 million in funds it could allocate to future distributions. We believe AMF could conceivably absorb as much as $25 million more in the next year. Considering AMF’s track record of finalizing distributions and the past scale of its operations, we hope to see AMF receive an additional $10 million. We note that distributions can take over a year to plan98 and funding that AMF receives in the next year may not be used or committed in that year. We expect to continue to follow AMF's progress of building a track record of finalizing and carrying out distributions over the next year.

Effect of additional funds

AMF has told us that the number of opportunities it considers at one time is limited by its funds and that the effect of having additional funds is largely to increase the size of the distributions it considers, rather than increasing the number of distributions it considered. The type of opportunities it will consider depends on how much funding it has available. If it were to raise an additional $10 million in the next year, it would likely continue to primarily consider distributions of 1 to 2 million LLINs. At this size, the failure of one or two negotiations would leave AMF with other possibilities in its pipeline, but the distribution size is still large enough to attract the interest of the type of partner AMF would like to work with. Above $10 million in additional funds, AMF would be able to increase the size of the distributions it considers. Above $20 million additional funds, AMF would primarily consider distributions of about 5 million LLINs, which would give it more leverage in its negotiations and discretion over which partners to work with. AMF told us that it has had several conversations about potential distributions that have not moved forward because AMF was unable to consider a distribution of sufficient size.99 We do not have more details on these conversations, particularly on whether other issues contributed to the discussions not going forward.

Global funding gaps for LLINs

The Roll Back Malaria Partnership's Harmonization Working Group estimates funding gaps for LLINs across all African countries.100 It most recently published a gap analysis in September 2013. At that time, it estimated that about 390 million additional LLINs would be needed in 2013-2016.101 In May and October 2014, we asked the Co-Chair of the Harmonization Working Group for updates on the size of the funding gap. On both occasions she told us that there remained significant gaps in funding for LLINs. She gave the following initial estimates for 2014-2017 (in millions of LLINs):

Country 2014 2015 2016 2017
Need 217 235 197 238
Financed 197 181 98 41
Gap 20 54 99 198

While we would guess that other funders will contribute some LLINs, the size of the gaps and the persistence of the 2014 gap late into the year, as occurred in 2013 as well,102 suggests that there is, and will continue to be, unmet need for LLINs in African countries.

AMF as an organization

We believe the Against Malaria Foundation to be an exceptionally strong and effective organization:

  • Track record: We feel AMF has built up a strong track record of raising money, finding smaller distribution partners, and getting partners to report information publicly at an unusual level, and verifiably getting bednets delivered. More recently, AMF has begun to demonstrate its ability to implement its model at significantly larger-scale. Its work on this is at an early stage.
  • Communication: AMF has always communicated extremely clearly and directly with us and given thoughtful answers to our critical questions.
  • Self-evaluation: AMF has invested heavily in self-evaluation, going above and beyond what’s usual for data collection on bednet distributions: it has demonstrated a commitment to collecting long-term net usage data.
  • Transparency: AMF appears to value transparency as much as any organization we’ve encountered. It hasn’t just shared information with us; it publishes significant amounts of useful information publicly on its own, far more than the norm. We have never seen AMF hesitate to share information publicly (unless it had what we consider a good reason).

However, we see some potential room for improvement:

  • AMF does not seem to be as aggressive and ambitious about growth and capacity as it could be. It has operated for 11 years and still has only two full-time staff members; it does not appear to us to have built as robust a fundraising operation as it could.
  • AMF has not conducted any field visits to distribution partners (as far as we are aware). We see this as a negative since we find such visits valuable.

More on how we think about evaluating organizations at our 2012 blog post.

Sources

Document Source
Alliance for Malaria Prevention Conference call minutes Source (archive)
Alliance for Malaria Prevention Conference call minutes (June 22, 2011) Source
Alliance for Malaria Prevention Conference call minutes (October 26, 2011) Source
Alliance for Malaria Prevention Toolkit (version 2.0) - Chapter 3 Source
Amani Global Works North Idjwi Island 2014 distribution proposal Source (archive)
AMF Audited financial statement (2005) Source (archive)
AMF Audited financial statement (2006) Source (archive)
AMF Audited financial statement (2007) Source (archive)
AMF Audited financial statement (2008) Source (archive)
AMF Audited financial statement (2009) Source (archive)
AMF Audited financial statement (2010) Source (archive)
AMF Audited financial statement (2011) Source (archive)
AMF Audited financial statement (2012) Source (archive)
AMF Audited financial statement (2013) Source (archive)
AMF Audited financial statement (2014) Source (archive)
AMF Countries involved Source (archive)
AMF Distributions Source (archive)
AMF Financial information (as of November 16, 2012) Source (archive)
AMF Frequently Asked Questions Source (archive)
AMF Future distributions Source (archive)
AMF How we work with distribution partners Source
AMF information we publish Source (archive)
AMF LLIN distribution proposal form Source
AMF Madagascar summary: Rejected Source
AMF Malaria Advisory Group Source (archive)
AMF Malawi universal coverage calculations (September 26, 2011) Source
AMF Net cost for GiveWell updated Source
AMF Ntcheu update (November 2012) Source (archive)
AMF page on Balaka 2013 distribution Source (archive)
AMF page on Dedza 2014 distribution Source (archive)
AMF page on Kasaï Occidental 2014 distribution Source (archive)
AMF page on non-net costs Source (archive)
AMF page on Ntcheu 2012 distribution Source (archive)
AMF Post-distribution survey example Source
AMF: "Introduction of smartphone technology to collect distribution data" Source (archive)
AMF: "Mid-distribution weekly reports for Dedza distribution, Malawi" Source (archive)
AMF: "Operational planning (12 months) and planning horizon (18 to 24 months)" Source (archive)
Balaka 6-month post-distribution check up data (April 2014) Source (archive)
Balaka 2013 and Dedza 2014 non-net cost budgets Source (archive)
Charity Commission. The Against Malaria Foundation Source (archive)
Concern Universal Balaka 2013 6-month post-distribution check-up report Source (archive)
Concern Universal Balaka 2013 distribution proposal Source (archive)
Concern Universal Balaka 2013 pre-distribution census data Source (archive)
Concern Universal Balaka 2013 week 1 report Source (archive)
Concern Universal Balaka 2013 week 5 report Source (archive)
Concern Universal Dedza 2014 distribution proposal Source (archive)
Concern Universal Dedza 2014 week 1 report Source (archive)
Concern Universal Dedza 2014 week 3 report Source (archive)
Concern Universal Ntcheu 2012 distribution proposal Source (archive)
Concern Universal Ntcheu 2012 pre-distribution census data Source
Concern Universal Ntcheu 2012 24-month post-distribution check-up report Source (archive)
Concern Universal Ntcheu 2012 mid-distribution reports Source (archive)
Dowa 2015 non-net cost budget Source (archive)
GiveWell estimate of AMF cost per net (November 2014) Source
GiveWell Notes from meeting regarding LLIN distribution in Malawi (October 21, 2011) Source
GiveWell Notes from site visit with Concern Universal in Malawi (October 2011) Source
GiveWell summary of AMF large-scale distributions Source
IMA World Health and AMF distribution agreement for Kasaï Occidental 2014 Source
IMA World Health and AMF distribution agreement for Nord Ubangi 2015 Source
IMA World Health Kasaï Occidental financial report (as of November 11, 2014) Source
Malaria Atlas Project Endemic countries Source (archive)
Melanie Renshaw, African Leaders Malaria Alliance Chief Technical Advisor, email to Rob Mather, June 26, 2011 Unpublished
Melanie Renshaw, phone conversation with GiveWell, May 23, 2014 Source
Melanie Renshaw, phone conversation with GiveWell, October 17, 2013 Source
Nonprofit Staffing New York Salary Survey Report 2011 Source
Ntcheu 15-month post-distribution check-up data Source (archive)
Ntcheu 24-month post-distribution check-up data Source (archive)
Ntcheu 6-month post-distribution check-up data Source (archive)
Rob Mather and Peter Sherratt, phone conversation with GiveWell, November 6, 2014 Unpublished
Rob Mather, AMF Founder, email to GiveWell, August 6, 2011 Unpublished
Rob Mather, AMF Founder, email to GiveWell, August 8, 2012 Unpublished
Rob Mather, AMF Founder, email to GiveWell, June 23, 2010 Source
Rob Mather, AMF Founder, email to GiveWell, June 24, 2011 Source
Rob Mather, AMF Founder, email to GiveWell, June 29, 2011 Source
Rob Mather, AMF Founder, email to GiveWell, June 30, 2014 Unpublished
Rob Mather, AMF Founder, email to GiveWell, May 22, 2014 Unpublished
Rob Mather, AMF Founder, email to GiveWell, November 2, 2011 Source
Rob Mather, AMF Founder, email to GiveWell, November 20, 2012 Unpublished
Rob Mather, AMF Founder, email to GiveWell, November 7, 2011 Source
Rob Mather, AMF Founder, email to GiveWell, November 6, 2014 Unpublished
Rob Mather, AMF Founder, email to GiveWell, November 8, 2014 Unpublished
Rob Mather, AMF Founder, email to GiveWell, November 26, 2014 Unpublished
Rob Mather, AMF Founder, phone conversation with GiveWell, February 11, 2011 Source
Rob Mather, AMF Founder, phone conversation with GiveWell, July 19, 2012 Unpublished
Rob Mather, AMF Founder, phone conversation with GiveWell, March 16, 2011 Source
Rob Mather, AMF Founder, phone conversation with GiveWell, August 15, 2013 Source
Rob Mather, AMF Founder, phone conversation with GiveWell, May 23, 2014 Source
Robin Todd, Concern Universal Malawi Director, email to GiveWell, April 27, 2012 Unpublished
Robin Todd, Concern Universal Malawi Director, email to Rob Mather, November 18, 2011 Unpublished
Robin Todd, Concern Universal Malawi Director, phone Conversation with GiveWell, March 20, 2012 Unpublished
Roll Back Malaria Partnership gap analysis (September 2013) Source (archive)
Roll Back Malaria gap analysis tool Source (archive)
Tossa Kokou, National Program Manager of the National Malaria Control Program, Togo, email to Rob Mather, May 15, 2012 Unpublished