Against Malaria Foundation - November 2017 Version

We have published a more recent review of this organization. See our most recent report on the Against Malaria Foundation.

Donate

The Against Malaria Foundation (AMF) is one of our top-rated charities and we believe that it offers donors an outstanding opportunity to accomplish good with their donations.

More information: What is our evaluation process?

Published: November 2017

Summary

What do they do? AMF (againstmalaria.com) provides funding for long-lasting insecticide-treated net (LLIN) distributions (for protection against malaria) in developing countries.

Does it work? There is strong evidence that distributing LLINs reduces child mortality and malaria cases. AMF has conducted post-distribution surveys of all completed distributions to determine whether LLINs have reached their intended destinations and how long they remain in good condition. AMF's post-distribution surveys have generally found positive results, with some exceptions, but have some methodological limitations.

What do you get for your dollar? We estimate that the cost to purchase and distribute an AMF-funded net is $4.22, or $4.02 excluding in-kind contributions from governments. The numbers of deaths averted and other benefits of distributing LLINs are a function of a number of difficult-to-estimate factors, which we discuss in detail below.

Is there room for more funding? We believe that AMF is very likely to be constrained by funding. There is high uncertainty in the maximum amount that AMF could use productively, though we expect the maximum to be much greater than what AMF is likely to receive. To fund all of the distributions that it is currently in detailed discussions about, AMF would need $50 million more than we project it will receive. The total funding gap for LLINs for 2018-2020 appears to be hundreds of millions of dollars.


AMF is recommended because of its:

  • Focus on a program with excellent evidence of effectiveness and cost-effectiveness.
  • Processes for ensuring that nets reach their intended recipients and monitoring whether they remain in homes and in good condition over the long-term.
  • Room for more funding – we believe AMF will be able to use additional funds to deliver additional nets.
  • Transparency – AMF shares significant information about its work with us and we are able to closely follow and understand its work.

Major open questions:

  • We have seen detailed data from before and during distributions. AMF also collects follow-up data after distributions. These follow-up surveys are conducted by AMF's various partner organizations and there has been variation in quality across locations. GiveWell has commissioned an ongoing project with IDinsight to better understand the survey methods used in several countries and to provide suggestions for AMF for the future.
  • The best evidence for nets was collected before they were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in nets, possibly making them less effective. It seems that insecticide resistance is a growing concern, but it remains difficult to quantify the impact of resistance. We will continue to follow several ongoing studies that may help to quantify the impact of resistance.1 We discuss this issue in more detail in our page on this topic.

Table of Contents

Our review process

We began reviewing AMF in 2009. Our review process has consisted of:

  • Reviewing documents AMF made available on its website or shared with us directly.
  • Extensive communication, including several meetings at AMF's London headquarters, with AMF Founder Rob Mather and board member Peter Sherratt.
  • A visit to AMF's distribution partner organization, United Purpose (formerly Concern Universal), in Malawi in October 2011 (notes and photos from this visit). We also spoke with United Purpose by phone in April 2016.
  • A visit to Greater Accra, Ghana in August 2016 to meet with representatives of AMF, AMF's distribution partners Episcopal Relief & Development and Anglican Diocesan Development and Relief Organization (ADDRO), Ghana's National Malaria Control Program, and other non-profit and government organizations involved in the AMF-funded LLIN distributions in Ghana in 2016. Notes and photos from our site visit are available here.
  • Conversations with Peter Sherratt, AMF's Executive Chairman; Don de Savigny, a member of AMF's Malaria Advisory Group; and other individuals (who requested to remain anonymous) familiar with AMF's work.
  • Conversations with Melanie Renshaw of the African Leaders Malaria Alliance, Marcy Erskine of the International Federation of the Red Cross, and Scott Filler of the Global Fund to Fight AIDS, Tuberculosis, and Malaria about funding needs for nets.


All content on AMF, including past reviews, updates, blog posts and conversation notes, is available here. We have also published a page with additional, detailed information on AMF to supplement some of the sections below.

What do they do?

AMF provides long-lasting insecticide-treated nets (for protection against malaria) in bulk to other non-profit organizations or government agencies, which then distribute the nets in developing countries.

As of October 2017, AMF has supported large-scale distributions in six countries (Malawi, DRC, Ghana, Uganda, Togo, and Papua New Guinea) for a total of 22 million LLINs distributed.2 In addition, AMF is funding 3 million LLINs for a planned distribution in Zambia that is expected to take place in late 2017.3

A summary of AMF's distributions can be found in this spreadsheet.

The role of AMF and its partners in LLIN distributions

AMF's role in LLIN distributions is to:4

  1. Identify countries with funding gaps for LLINs.
  2. Find distribution partners (in-country non-profit organizations or government agencies) to carry out LLIN distributions. AMF and its partners agree on expectations for the distribution, including who pays for costs other than the purchase price of LLINs (which are always covered by AMF), the process that will be used to carry out the distribution, and what information will be collected and shared with AMF.
  3. Purchase LLINs and have them shipped to the distribution partners.
  4. Work with distribution partners to collect reports on the distribution and conduct follow-up surveys.

Distribution partners implement on-the-ground activities, including registering residents in targeted areas, distributing LLINs, monitoring the registration and distribution processes, and conducting follow-up surveys.5

Details follow.

Selecting locations for distributions and finding distribution partners

When selecting locations for future distributions, AMF told us it consults a series of sources, as it believes there is no single reliable resource with up-to-date information about where there are funding gaps for LLINs. Sources it consults include the Alliance for Malaria Prevention's (AMP's) list of countries with significant net gaps, other malaria control funders, in-country technical advisors, the relevant National Malaria Control Program (NMCP), implementing organizations, and the African Leaders Malaria Alliance.6 AMF told us that it has been receiving more funding requests since it started funding larger distributions,7 and notes that its largest commitment so far—10.6 million LLINs in Uganda in 2017—was made in response to an in-bound request.8

As AMF investigates countries with existing net gaps, it also looks into organizations working within those countries that could serve as distribution partners.9 AMF looks for distribution partners that have the capacity and willingness to implement registration, distribution, and monitoring processes that meet agreed-upon requirements.10 So far, AMF has worked with United Purpose (formerly Concern Universal) in Malawi, IMA World Health in the Democratic Republic of the Congo, Episcopal Relief & Development in Ghana, National Malaria Control Programs (NMCPs) in Ghana, Togo, Uganda, and Zambia, and the Rotary Club of Port Moresby in Papua New Guinea on large-scale LLIN distributions.11

Registration and distribution

  • Registration: During the registration process, AMF's distribution partners' staff or volunteers travel door-to-door in targeted areas to collect the information used to determine the number of LLINs to allocate to each household (e.g., the number of sleeping spaces, the number of previously-owned LLINs in usable condition, and/or the number of household members), as well as the information used to identify the household for the distribution and post-distribution surveys (e.g., the name of the head of the household, and/or household location). AMF has shared full or sample registration data from each completed large-scale distribution with us.12 The specifics of the registration process and LLIN allocation strategy have differed by country (process details and registration data sources on a separate page with additional details about AMF).
  • Distribution: To distribute LLINs to recipients, AMF and its distribution partners have primarily used "point distribution" (LLIN recipients pick up their nets from a specified point in or near their community), but have also used "hang-up distribution" (staff or volunteers travel door-to-door to deliver and hang up LLINs) in one distribution in Kasaï-Occidental, DRC.13 Distribution partners manage the logistics of in-country shipping and storage of LLINs prior to the distribution. The specifics of distribution processes have varied in the different countries AMF has worked in (details on a separate page with additional details about AMF).

Monitoring

AMF's distribution partners also implement a set of monitoring activities to produce evidence on whether the registration and distribution processes operated as intended and on the long-term impact of the LLIN distribution. Monitoring activities have varied somewhat for different distributions. We describe these processes in more detail on our page with additional information on AMF. In short:

  • Process monitoring (i.e., the activities used to assess whether the registration and distribution processes operated as intended):
    • Data validation: This includes various processes, which have varied considerably in different distributions, to check the accuracy of registration and distribution data. It has generally involved looking for and following up on outliers or implausible data, and has sometimes involved re-entering a sample of data or reading registration lists out loud at community meetings and asking community members for corrections.
    • "Embedded" monitoring: (Ghana only) Staff of AMF's local NGO partner organization attended district-level planning meetings to ensure that they were operating as intended, observed the registration of households by volunteers organized by the government, and observed selected distribution points.14
    • Distribution reports: Distribution reports provide narrative summaries of activities implemented and challenges encountered by distribution partners.
    • Post-distribution validation tracing: (Ghana only) After distributions were complete, AMF's distribution partners in Ghana checked that a random sample of households (100 households per district) had actually received the number of LLINs they were allocated by calling or visiting the households.15
  • Impact monitoring (i.e., the activities used to assess the impact of the distribution):
    • Post-distribution check-ups (PDCUs): Distribution partners conduct follow-up surveys (called post-distribution check-ups, or PDCUs) in 5% of households at 6-month intervals for 2.5 years after a distribution. PDCUs collect data on whether nets are present, whether they have been hung, and what condition they are in.

      We summarize which PDCUs have been completed in this spreadsheet (see "PDCUs" sheet), and summarize the results and methods of PDCUs we have seen in this spreadsheet.

      Most scheduled PDCUs have been completed in Malawi and the Ghana PDCUs are on schedule. AMF has also shared data and reports from its first three PDCUs from Kasaï-Occidental, DRC.16 We believe the DRC PDCUs were poorly implemented and provide limited evidence on the proportion of AMF's LLINs that reached their intended destinations or the impact of AMF's distribution on LLIN usage over time in Kasaï-Occidental (see this blog post for details). The 6-month PDCU in Nord Ubangi, DRC was not completed due to security concerns. AMF has told us that the 12-month Nord Ubangi PDCU was completed in mid 2017; we have not yet seen these data.

      In June and July 2017, as part of its partnership with GiveWell, IDinsight conducted site visits to post-distribution surveys to learn about AMF's monitoring of its programs. The surveys were conducted by United Purpose (formerly Concern Universal) in Malawi and by Episcopal Relief and Development and the Anglican Diocesan Development and Relief Organization (ADDRO) in Ghana. IDinsight's notes from these two visits: Malawi site visit notes (June 2017) and Ghana site visit notes (July 2017). IDinsight is planning similar such visits to Uganda, Togo, and Zambia in 2018. IDinsight will compile a report on its findings, including the sources of potential bias in post-distribution surveys.

Other activities

AMF occasionally supports malaria control activities beyond the direct distribution of LLINs. For example:

  • AMF has begun funding research on insecticide resistance that will be carried out in conjunction with AMF distributions. AMF plans to fund research on the effectiveness of PBO LLINs in conjunction with its Uganda 2017 distribution. PBO LLINs are a newer type of net incorporating piperonyl butoxide (PBO) alongside pyrethroid insecticide used in other LLINs. These nets may be more effective than other LLINs in areas where mosquitoes have developed insecticide resistance. The work is currently projected to cost $2.5 million.17 AMF also funded the first phase of a study (which cost around $100,000) on insecticide resistance in Nord Ubangi, DRC; in November 2016, AMF told us that it no longer planned to fund the completion of this study (which would have cost around an additional $700,000) because it was funding the PBO study in Uganda.18

  • AMF is co-funding a "Malaria Control Unit" (MCU) in Malawi with United Purpose.19 The MCU will consist of up to 14 permanent staff members who will work on a variety of malaria control projects: conducting post-distribution surveys, improving malaria case rate data collection practices, monitoring the levels of malaria prevention and treatment supplies at local health centers, developing efficient methods to keep net coverage rates high in between mass distribution campaigns, and more.20 The MCU is also intended to assist with AMF and United Purpose's distributions in Malawi.21 AMF has committed approximately $636,000 over three years to this project.22

  • AMF has encouraged United Purpose staff to attend national malaria control strategy meetings in Malawi to share AMF's processes and results in other districts. As a result, AMF has told us that some of its monitoring practices were being adopted for a large national distribution (approximately 7 million nets) in 18 of Malawi's 28 districts in March to May 2016.23 The practices being implemented include (a) using sleeping spaces data (as opposed to population data) to calculate the number of nets needed, (b) "105%" registration data collection (described on another page), and (c) putting summary data (at the village or health center level) into electronic form.24 AMF told us in February 2015 that it expects to be able to see the data collected during this distribution.25 We have not yet followed up with AMF on whether this happened as expected.

Spending breakdown

The following table shows AMF's total expenditure, categorized into purchases of LLINs, spending on running the organization, and spending on other non-net costs (such as providing funding to other organizations to conduct post-distribution check-ups, which are described above). We include spending since FY 2012 (July 2011 to June 2012) because this is when AMF shifted to its current model of larger-scale distributions.

AMF expenditure by category26
Category July 2011 to June 2017 spending FY 2012 to 2017 ratio
LLIN purchases $60.0 million 82.0%
Costs of distributing nets and monitoring ("non-net costs") $11.8 million 16.1%
Insecticide resistance research $0.4 million 0.5%
Salaries and other organizational costs $0.9 million 1.3%
Total spending $73.2 million -

Prior to 2013, AMF asked all distribution partners to use their own funds or to find another funder for all non-net costs of the distribution.27 More recently, it has paid for some of these costs in certain distributions. AMF told us that it considers funding non-net costs in cases where (a) non-net costs are not covered by other partners, and (b) AMF feels confident that its distribution partners will manage and report on spending well.28

Does it work?

On a separate page, we discuss the general evidence behind distributions of LLINs. We conclude that there is strong evidence that these distributions can be expected to reduce child mortality and malaria cases.

When evaluating the effectiveness of an LLIN distribution organization, we seek to answer the following questions:

  • Are LLINs targeted at people who do not already have them? When determining how many LLINs to allocate to each household, AMF's distribution partners in Malawi and DRC checked households for previously-owned LLINs in good condition, but AMF's partners in Ghana, Uganda, Papua New Guinea, and Togo did not. In Ghana, we don't know what the LLIN coverage rate was before the distribution, though we note that the previous mass LLIN distribution was four years prior and most nets wear out in less than four years; AMF told us that a similar amount of time had passed since previous distributions in other countries.
  • Do the LLINs reach the intended destinations? The main evidence that LLINs reached their intended destinations at a high rate are (a) post-distribution surveys, where surveyors visit a sample of households that registered for nets six months after the distribution, and (b) data from registering households and, for some distributions, which households are reported to have received their nets. We have seen (a) and/or (b) from all of AMF's distributions; we discuss some limitations of the post-distribution surveys below.
  • Are LLINs targeted at areas with high rates of malaria? AMF seeks out distribution partners in countries that are known to have high rates of malaria, or where malaria rates are likely to increase significantly if LLIN distribution programs are not sustained. We note that Papua New Guinea (where AMF plans to fund LLIN distributions in 2017 and 2018) has lower (but still significant) rates of malaria than other countries where AMF has worked or plans to work.
  • Do those who receive the LLINs install them in their homes properly? Do they utilize them consistently over the long term? AMF requires partners to conduct follow-up surveys at 6-month intervals for a period of 2.5 years. We have little data to compare these results to, but in most cases the results from Malawi and Ghana seem consistent with moderate to high usage of nets for at least a year or two after distribution; we have very limited evidence past 18-months post-distribution.29 We discuss some methodological limitations of the surveys below.
  • Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF? The evidence we have seen suggests that donations to AMF increase the total number of LLINs distributed, but that a portion of the impact is offset by displacing funding from other sources.

Details follow.

Are LLINs targeted at people who do not already have them?

Malawi and the Democratic Republic of the Congo

We believe that the LLIN allocation strategy used in Malawi and DRC is reasonably likely to avoid supplying more LLINs than needed to individual households.

For AMF's past large-scale distributions in Malawi, a household's allocation of LLINs was determined by subtracting the number of LLINs in usable condition from the total number of sleeping spaces in the household.30 AMF's distribution partner in Malawi, United Purpose, told us that during registration health workers enter each household, ask about who is living in the household, and visually inspect sleeping spaces and existing nets.31 United Purpose has also conducted village meetings in which its staff read off household names and the number of LLINs that have been allocated to each household; households then indicate if any errors have been made.32 We observed one such village verification meeting during our visit to Malawi in October 2011.33 For more detail on the registration process used in Malawi, see a separate page with additional information on AMF.

We have seen registration data (which include records of numbers of sleeping spaces and usable LLINs per household) from all six of AMF's completed large-scale distributions in Malawi: full data from its Ntcheu 2012, Balaka 2015, and Ntcheu 2015 distributions, and sample results from its Balaka 2013, Dedza 2014, and Dowa 2015 distributions.34

For IMA's Kasaï-Occidental, DRC distribution, we discuss what we know about the process from reports IMA has shared here. The process appears to be largely similar to United Purpose's process for going house-to-house to determine net need and allocating LLINs by subtracting a household's usable LLINs from its number of sleeping spaces, though it did not include village verification meetings (in part because nets were distributed during the same visit in which net need was determined). We have seen full distribution data from Kasaï-Occidental, which includes data on numbers of usable LLINs and sleeping spaces per household.35 Due to logistical issues encountered with registering households and distributing nets during the same visit in the Kasaï-Occidental distribution,36 IMA and AMF implemented separate registration and distribution processes for the distribution in Nord Ubangi, DRC.37 We have seen a sample of household-level registration and distribution data from Nord Ubangi (as well as summaries of data for the entire distribution), which includes numbers of sleeping spaces per household, numbers of previously-owned LLINs, and numbers of LLINs distributed.38 The summary registration data indicates that ownership of LLINs in good condition before AMF's distribution was low, with around 18,000 LLINs in good condition owned for a population of around 1.3 million.39 We have not yet seen a distribution report from Nord Ubangi, so we do not have full details on how this data was collected.

It is possible that some households may hide previously-owned LLINs during registration; cases of households hiding nets in order to receive extra LLINs have been observed in campaigns in Senegal and Nigeria.40 We have not seen any evidence that this has occurred in AMF's distributions.

Ghana, Papua New Guinea, Uganda, and Togo

We believe that the LLIN allocation strategy used in AMF's distributions in Ghana in 2016 and in Papua New Guinea, Uganda, and Togo in 2017 may have oversupplied some households with LLINs.

Prior to the 2016 distributions in Ghana, volunteers organized by the government collected information on the number of people in each household in targeted areas.41 Registered households were allocated one net per two people (rounding up for households with an odd number of people). Previously-owned nets in usable condition were not subtracted from households' LLIN allocations.42 It seems likely to us that some households in areas targeted by these distributions already owned some LLINs in usable condition, which may have been obtained through "continuous distributions" or by purchase, so these households may have been allocated more LLINs than necessary in AMF's distribution.43 However, since the most recent mass distribution of LLINs in the Northern, Greater Accra, and Upper West Regions was in 2012 (four years before AMF's distributions), we would guess that a large proportion of targeted households were in need of new LLINs.44

We have seen complete registration data from AMF's distribution in Ghana (which includes information on the number of individuals in each household and the number of LLINs allocated to each household). AMF and the government decided not to collect information on previously-owned LLINs for this distribution because they did not expect to find many.45

AMF's distribution partners also did not subtract out previously-owned usable LLINs from households' LLIN allocations during its 2017 distributions in Papua New Guinea, Togo, and Uganda.46 AMF told us that this decision was made because, in regions where the previous mass distribution was more than three years earlier, it is likely that the number of previously-owned nets in good condition will be low.47

AMF also told us that it has not yet determined whether usable LLINs will be subtracted from households' allocations during distributions in Malawi in 2018.48

AMF notes that it interprets the data from PDCUs as indicating that LLINs should be replaced at least every three years and that subtracting usable nets (unless they have never been used) increases the period where a household has to rely on old, ineffective nets until the next LLIN campaign.49

Do LLINs reach their intended destinations?

We have seen reasonable evidence that a large proportion of AMF's LLINs have reached their intended destinations in AMF's past distributions. The strongest evidence we have seen on this question in Malawi and Ghana is from post-distribution surveys, and the strongest evidence from DRC and Papua New Guinea is from household-level data on nets received. (We have not yet seen evidence on this for recent/ongoing distributions in Togo and Uganda, but expect this later.)

The main sources of evidence that LLINs reached their intended destinations at a high rate are (a) post-distribution surveys, where surveyors visit a sample of households that registered for nets six months after the distribution, and (b) data on which households are reported to have received their nets. We have also seen narrative distribution reports for most of AMF's distributions, and a few other minor sources of evidence that LLINs reached their intended destinations.

Post-distribution surveys

AMF's distribution partners conduct post-distribution surveys, discussed in detail below, about six months after each distribution (and every six months after that up to 2.5 years after the distribution) to determine whether nets are in place, being used, and in good condition.50 The first post-distribution surveys that have been completed to date for each large-scale distribution in Malawi have generally somewhat mixed results: the first four found rates above 80% (90%, 87%, 93%, and 91%) of LLINs from the recent AMF distribution hung over sleeping spaces, while the most recent distributions have had worse results (69% and 60%; the latter was followed by a rate of 77% at 12 months). Most nets that were not hung were present in the household: the highest rate of missing nets was 5% (see this summary of results for details).51 Hang up rates at 6 months post-distribution in Ghana were relatively high: 80%, 84%, and 90% in the three regions.

We believe that the results of the first post-distribution surveys from distributions in Malawi and Ghana provide reasonable evidence that nets reached their intended locations at a high rate, though we note some methodological limitations of these surveys below.

We believe that the 8-, 12-, and 18-month surveys in Kasaï-Occidental were poorly implemented, and that the results of the surveys provide limited evidence on the proportion of AMF's LLINs that reached intended destinations (details in this blog post).52

Household-level data on nets received

In addition to recording data on the number of LLINs allocated to each household during the registration process, AMF's distribution partners also record data on the number of LLINs actually distributed to each household during the distribution process.

For Malawi, we have not seen household-level data on the number of nets received. Our understanding is that these data exist in paper form only.

For the Kasaï-Occidental distribution in DRC, households received nets at the same time they were registered, so the registration list provides some evidence that nets reached their intended destinations.53 We have seen registration and distribution data from Nord Ubangi, DRC that include numbers of LLINs actually received and hung up for each household.54 We have not yet seen a distribution report from Nord Ubangi, so we are uncertain about how data on the number of LLINs households actually received were collected.

In Ghana, data on the number of LLINs actually distributed to each household were recorded by LLIN distributors, and later entered electronically into AMF's Data Entry System, where we have viewed the data.55 We also spot-checked some paper records during our visit to AMF's distribution partner in Ghana in 2016.

Distribution reports

Distribution reports provide narrative summaries of distribution activities and discuss challenges encountered. These reports provide some evidence that distributions generally operated as intended (or that distribution partners are aware of specific challenges and have plans to address them) and that households actually received LLINs; however, we do not think this type of evidence is as useful for the question of whether LLINs reached households as the quantitative evidence discussed above.

United Purpose has provided distribution reports for four of the six large-scale distributions it has completed in Malawi.56 As we report in our March 2012 update, reports from the Ntcheu 2012 distribution note challenges including attempted thefts, double registrations, and logistical problems.57 United Purpose provided a similar level of detail on challenges encountered in the Balaka 201358 and Dedza 2014 distributions,59 and the first half of the Dowa 2015 distribution.60 These reports increase our confidence that United Purpose is aware of potential problems and has a system in place to address them.61 We have not seen reports for the Ntcheu 2015 or Balaka 2015 distributions in Malawi.

IMA World Health has provided a distribution report for the Kasaï-Occidental, DRC 2014 distribution. We have not yet seen any distribution reports from the Nord Ubangi, DRC distribution.62

We have also seen progress reports covering periods of pre-distribution, distribution, and post-distribution activities from Episcopal Relief & Development, AMF's distribution partner in Ghana, for the 2016 distributions in Greater Accra, Northern Region, and Upper West Region, and a full report on the 2016 distribution in the Northern Region.63 These reports discuss problems encountered in each of these stages.64

Other evidence

For the Kasaï-Occidental 2014 distribution, AMF's distribution partner, IMA World Health, piloted the use of smartphones to record household data, including GPS coordinates, for registration and LLIN distribution.65 AMF has sent us detailed GPS data that show the GPS coordinates for each household visited.66 The registration and distribution data from Nord Ubangi, DRC also included GPS coordinates for each household.67

In Ghana, it is our understanding that "post-distribution validation tracing" (i.e., checking, immediately after the distribution, by phone or in-person, that a randomly selected sample of households actually received the correct number of LLINs) was used for all three AMF-funded distributions in 2016, but we have not seen comprehensive results from this process (e.g., the proportion of selected households that received the appropriate number of LLINs).68 AMF told us that it expected to see data from post-distribution validation tracing from Episcopal Relief & Development in January 2017;69 we have not followed up with AMF for these data.

Are LLINs targeted at areas with high rates of malaria?

At the highest level, AMF appears to exclusively target countries with known malaria risk.70 Since 2012, AMF’s large-scale distributions have occurred in Malawi, DRC, Ghana, Uganda, Togo, and Papua New Guinea, and a distribution is planned for Zambia.71 Based on 2013 data, the World Health Organization estimated that Malawi, DRC, Ghana, Uganda, Togo, and Zambia had malaria death rates of between 50 and 99 deaths per 100,000 people, and that Papua New Guinea's malaria death rate was between 10 and 49 deaths per 100,000 people.72

Do those who receive the LLINs install them in their homes properly? Do those who receive the LLINs utilize them consistently over the long term?

AMF requires partners to conduct PDCUs at 6-month intervals (plus or minus a month) for a period of 2.5 years, or until the next community-wide net distribution in the same area, to determine whether LLINs are present, whether they have been hung, and what condition they are in. The surveys do not measure usage or reported usage.73 Surveys to date have been highly comprehensive (covering all distributions) and have some methodological limitations, discussed below. Results have been fairly positive with substantial variation across distributions.

Methods

See our summary of AMF distributions spreadsheet, "PDCUs" sheet, for details of what PDCUs have been completed. In short, AMF's PDCUs are highly comprehensive: we have seen results from all distributions (where enough time has passed since the distribution), and few of the scheduled surveys (which are intended to be conducted every six months) have been skipped.

For a description of the methodology used in AMF's PDCUs, see this spreadsheet, sheet "Methods." In short:

  • Households are now selected randomly for inclusion in the surveys. AMF generates the list of households to survey from the full list of households that were registered for the distribution.74 The list includes more households than will be surveyed, with the intention of allowing enumerators to skip some households; this may lead to bias in the results. Note that in some earlier surveys, households were not selected fully randomly.75
  • AMF has told us that it asks its partner organizations to revisit 5% of the households visited, as a means of data quality control.76 While the idea is that the results from the revisits can then be matched and cross-checked against the original results, our impression is that this matching process historically has not actually occurred. AMF believes it will be easier to match the data using the Data Entry System.77 We have not yet seen this analysis for any distributions.
  • We believe that the 8-, 12-, and 18-month surveys in Kasaï-Occidental were poorly implemented, and that the results of the surveys provide only limited evidence on the proportion of AMF-funded LLINs that are used effectively over the long term (details in this blog post).78 However, we think that the combination of the three reported rates (at 8-, 12- and 18-months) of coverage tells a plausible story of a decline in coverage over time, which increases our confidence that these surveys are to some extent representative of LLIN hang-up rates, or at least of trends in hang-up rates.
  • In 2017, GiveWell commissioned IDinsight to observe PDCUs in Malawi and Ghana. For Ghana, IDinsight noted:
    • Though enumerators were trained to observe use and condition of nets, assessment was inconsistent and did not always involve direct observation.79
    • During data entry, illegible and inconsistent data were discarded, and records were not kept on what portion of data was discarded.80

Results

Full results in this spreadsheet. Definitions of each indicator are on a separate page with additional information about AMF. In short:

  • Malawi: The data from PDCUs from the first four of six distributions in Malawi show high rates of nets hanging and in at least "viable" condition for 18-33 months post-distribution. We have more limited follow-up so far on the two more recent distributions in Malawi. Early results indicate lower rates of nets from the distribution hanging and high rates of nets in at least "viable" condition. AMF believes that the low hang-up rates for these two more recent distributions may be due to the continued use of older LLINs from the 2012 distribution in Ntcheu and use of LLINs from other sources in Dowa (or other possible explanations, see footnote).81
  • DRC: We believe the data we have seen from AMF's distributions in DRC provide some limited evidence that LLINs decayed considerably quicker than expected.
  • Ghana: The distributions in Ghana occurred in mid-to-late 2016, so the longest follow-up to date in Ghana is from 12 months post-distribution and so far shows high rates of nets hanging and nets in at least "viable" condition (i.e. not "worn out").

We note that net usage rates in the trials of bed net efficacy documented in our page on Long Lasting Insecticide Treated Nets had net usage rates generally in the 60%-80% range.82 However, these usage rates are not directly comparable to the data from AMF's PDCUs, which measure the proportion of nets hung, rather than usage.

As another point of comparison, the "decay model" we use to estimate the lifespan of LLINs assumes that 92% of LLINs are functional and in use for the first year after a distribution, 80% of nets are functional and in use for the second year, and 50% of nets are functional and in use for the third year.83 It is not fully clear to us how to compare the net quality and net hangup rates found in AMF's post-distribution check-ups to the assumptions in the decay model, in part because it is not clear whether the definition of a "functional and in use" net in the decay model is comparable to what PDCUs measure.

Do AMF's LLINs increase the total number of LLINs distributed, or would the recipients have received LLINs from other sources if not for AMF?

On a separate page, we discuss some cases where AMF was in discussions to fund a distribution, but ultimately did not. In most of these cases, the net gap AMF was in discussions to fill persisted for six or more months after AMF's discussions closed, and, in two out of the five cases we looked at, gaps persisted for long periods (18 months and ~3 years, respectively). In most cases, the gap was eventually filled by another funder. As far as we can tell, during the time between AMF withdrawing from discussions and another funder stepping in, the populations targeted for distributions did not receive nets and likely were inadequately protected from malaria.

We also discuss, on a separate page, what we have been told about what would have happened in the absence of AMF funding in two distributions that AMF did fund. In summary, in both cases our best guess is that there were no other funders who could have closed the gaps, and nets would have been at least partially targeted at higher-risk populations while others would have been left uncovered.

Across Africa, there are substantial funding gaps for LLINs (more below), and because our impression from following AMF's progress over time is that, due to AMF's more limited funding and, perhaps, greater data requirements, governments often seek funding first from larger funders (particularly the Global Fund to fight AIDS, TB, and Malaria) and then may ask AMF to fill gaps. However, we note that this dynamic may change if AMF has significant resources in the future (more below) and that countries are sometimes able to choose how they allocate Global Fund grants among malaria interventions (including LLINs, treatment, and diagnosis), so the availability of funding for LLINs from AMF could cause countries to allocate fewer Global Fund resources to LLINs.84

Are there any negative or offsetting impacts?

  • Will insecticide-treated nets continue to be effective? As discussed in our report on insecticide-treated nets, there is strong evidence for the effectiveness of this intervention; however, the best evidence for the intervention was collected before LLINs were widely used and there is some evidence that mosquitoes have since adapted to the insecticide used in LLINs, possibly making them less effective. We have reviewed the evidence on the state of insecticide resistance. We concluded, "Broadly, it seems that insecticide resistance is a larger concern now than it was when we last thoroughly evaluated the evidence in 2012, but it remains difficult to quantify the impact of resistance. Our very rough best guess (methodology described in more detail below) is that insecticide-treated nets (ITNs) are roughly one-third less effective on average across sub-Saharan Africa than they would be in the absence of insecticide resistance. ITNs remain a highly cost-effective intervention after incorporating this discount."
  • Do free LLIN distributions distort incentives for recipients or distort local markets for nets? As discussed in our report on insecticide-treated nets, we feel that there is a reasonably strong case for distributing LLINs freely rather than selling them at market (or even below-market) prices. We also think that the benefits of distributing LLINs freely to a population likely outweigh the negative consequences of distortion in local net markets, though we have not factored these potentially negative consequences into our cost-effectiveness analysis of AMF.
  • Could distribution of LLINs be inequitable and unfair, causing problems in the targeted communities? We feel that AMF's processes for determining needs for LLINs at the household level are fairly well-suited to ensuring that LLINs are distributed equitably. We have some concerns about whether AMF's process succeeds at identifying all villages or households located outside of villages.
  • Does AMF divert skilled labor from other areas? In Malawi, net distributions have been conducted by low-level government health staff in partnership with the staff of AMF's partner NGO.85 AMF's partner in Malawi told us in 2012 that government health staff are normally involved in activities such as disseminating health-related information, reporting on levels of stunting and disease, carrying out immunization campaigns, and providing nutrition support.86 We do not know the extent to which net distribution reduces their ability to complete other duties, though we note that net distributions are generally completed within a few days in each local area, and we would guess that LLIN distributions are likely among the most cost-effective work they engage in.87 Diversion of skilled labor may be more of a concern in the DRC, where 22 senior district health staff were employed as Field Supervisors for the Kasaï-Occidental distribution.88 AMF's distributions in Ghana in 2016 were planned by high-level staff from a government health agency and implemented by local government staff and volunteers; monitoring of the distribution was largely implemented by a Ghanaian non-profit organization.89 We have not investigated what other activities the government and non-profit staff and volunteers engage in, or whether the LLIN distribution interferes with their ability to perform other duties.

What do you get for your dollar?

Cost per LLIN distributed

We estimate that on average the total cost to purchase, distribute, and follow up on the distribution of an AMF-funded LLIN is $4.22. Excluding in-kind government contributions, we estimate the cost is $4.02. These estimates rely on a number of uncertain assumptions. Full details of our analysis are in this spreadsheet. For results, see sheet "Summary." More discussion follows.

Below, we also discuss how we estimate the cost per death averted in AMF distributions.

Our approach

To get the total costs of the program, we attempt to include all partners such that our cost per LLIN represents everything required to deliver the nets. In particular, in our cost per treatment analysis for AMF, we have attempted to include these categories:

  • The costs paid by AMF to purchase LLINs. This accounts for 47% of the total cost per LLIN.
  • Costs of shipping and delivering nets, monitoring the distribution, and conducting PDCUs. In most cases, the Global Fund or another partner pays for most of these costs; in almost every case, AMF has paid for the PDCUs. In total, these costs account for 43% of the total.
  • Resources contributed by governments, such as staff time, office space, etc. We roughly estimate these costs as a proportion of the total cost (excluding LLIN purchase costs) based on an analysis of a distribution in Malawi in 2012. This accounts for 5% of the total.
  • Other AMF costs: staff salaries and other organizational costs, a rough estimate of the value of the CEO's donated time and estimated value of other pro bono support, research AMF has funded on insecticide resistance, and costs of the Malaria Unit in Malawi. This accounts for 5% of the total.

We start with this total cost figure and apply adjustments in our cost-effectiveness analysis to account for cases where we believe the charity's funds have caused other actors to shift funds from a less cost-effective use to a more cost-effective use ("leverage") or from a more cost-effective use to a less cost-effective use ("funging").

We used data and estimates from completed distributions, ongoing distributions, and distributions that AMF has committed to funding in the future.

Comment from AMF: AMF would like all donors reading this to know that the costs included for the CEO and pro bono services are not actually incurred. Each of them is very happy to provide their services for free.

On a separate page, we triangulate our AMF estimates of cost per LLIN with a rough global average cost per LLIN.

Shortcomings of our analysis

There are several ways in which our analysis of AMF's cost per LLIN is uncertain:

  • The price of an LLIN has been falling and in our estimate we have used a projected price per LLIN, rather than an average of the prices AMF has paid in the past. The average price in distributions AMF funded in 2016-2018 was $2.20. Based on what AMF told us it expects in the future, we have used an estimate of $2.00 per LLIN.
  • For distribution costs not paid by AMF, we have generally used rough estimates. AMF told us that the Global Fund expects to pay $1.16 per net for these costs in Malawi in 2018 and $2.20 in DRC, and paid $1.26 in Ghana in 2016. For other countries, we use the average of these three figures.
  • It is difficult to predict where AMF will fund LLINs in the future because it pursues several discussions at once. To get an overall average cost, we have taken a weighted average of past distributions, weighted by the number of LLINs in each distribution, rather than weighted by the number of LLINs projected for the future. This means that if, for example, AMF funds a larger proportion of nets in DRC, where costs are high, in the future than in the past, our cost per LLIN would be an underestimate. (Note that this does not mean that funding LLINs in DRC is necessarily less cost-effective—our cost-effectiveness analysis also takes into account factors such as malaria mortality rates in each country.)

Cost per death averted

See our most recent cost-effectiveness model for estimates of the cost per death averted through AMF-funded LLIN distributions.

Note that our cost-effectiveness analyses are simplified models that do not take into account a number of factors. For example, our model does not include the short-term impact of non-fatal cases of malaria prevented on health or productivity, prevention of other mosquito-borne diseases, or reductions in health care costs due to LLINs reducing the number of cases of malaria. It also does not include possible offsetting impacts or other harms. We do include possible developmental impacts on children who sleep under an LLIN.90

There are limitations to this kind of cost-effectiveness analysis, and we believe that cost-effectiveness estimates such as these should not be taken literally, due to the significant uncertainty around them. We provide these estimates (a) for comparative purposes and (b) because working on them helps us ensure that we are thinking through as many of the relevant issues as possible.

The full details of our cost-effectiveness analysis are in our report on mass distribution of LLINs.

Is there room for more funding?

We believe that AMF is very likely to be constrained by funding. There is high uncertainty in the maximum amount that AMF could use productively, but we expect the maximum to be much greater than what AMF is likely to receive. To fund all of the distributions that it is currently in detailed discussions about, AMF would need $50 million more than we project it will receive. The total funding gap for LLINs for 2018-2020 appears to be hundreds of millions of dollars.

We calculate this from (more details in the sections below):

  • Total opportunities to spend funds productively: AMF has set aside $23.9 million for distributions where it expects to sign agreements soon. Beyond that, AMF told us that it is in detailed discussions with partners about the possibility of committing an additional $93 million. Based on AMF's past experience, our guess is that some of these discussions will not result in agreements for reasons other than lack of funding. AMF would likely pursue discussions beyond these if it had sufficient funds. It is currently in some early stage conversations and it appears that the funding gap for LLINs across Africa will be hundreds of millions of dollars in 2018-2020. (More)
  • Cash on hand: As of October 2017, AMF held $58.6 million in uncommitted funds of which it had set aside $23.9 million for distributions where it expects to sign agreements soon, leaving $34.8 million in funds not set aside. (More)
  • Expected additional funding: We roughly estimate that AMF will receive $8 million in 2018-2020, not including funds due to GiveWell's recommendation. This could be an underestimate—AMF expects to make fundraising a larger priority in the future. (More)

Below, we also discuss:

  • Past spending rate: AMF has a track record of identifying opportunities to spend tens of millions of dollars per year, however, we see its limited staff capacity as a risk to its ability to increase its spending in the future. (More)
  • Considerations around the size of AMF distributions: Funding a large portion of a national distribution has advantages in terms of efficiency and leverage, but also increases the risk of displacing funding from other donors. We discuss AMF's approach to this tradeoff. (More)

Available and expected funds

As of October 30, 2017, AMF held $76.5 million. Of this, $17.8 million was committed to PDCUs for previous distributions, future distributions in Uganda (2017), Malawi (2018) and Papua New Guinea (2018), central costs for two years (including hiring additional staff), the research study on PBO nets in Uganda, and the Malaria Unit in Malawi, leaving $58.6 million in uncommitted funds.91 A further $23.9 million was set aside for distributions where AMF expected to sign agreements soon, leaving $34.8 million currently available to allocate to additional future distributions.92

We expect that AMF will receive additional donations over the remainder of 2016 and in 2017 from:

  • Donors who are not influenced by GiveWell's research: Historically, the vast majority of AMF's funding has been due to GiveWell's recommendation.93 As a rough guess, we expect AMF to receive $2 million per year from non-GiveWell sources in the next year. This may be an underestimate as AMF has told us that it expects fundraising to be a key priority in 2018.94
  • Donors who give based on GiveWell's top charity list, but do not follow our recommendation for marginal funding: GiveWell maintains both a list of all top charities that meet our criteria and a recommendation for which charity or charities to give to in order to maximize the impact of additional donations, given cost-effectiveness of remaining funding gaps. We estimate that AMF will receive about $2 million from donors who use our top charity list but don't follow our recommendation for marginal donations.95 In our projections of future funding, we count only one year of top charity funding in order to retain the flexibility to change our recommendations in future years.
  • Donors who follow GiveWell's recommendation for marginal donations: Our estimate of room for more funding is used to make a recommendation to these donors.

With $58.6 million in currently available funding and $4 million expected in additional funding ($2 million from each of the first two groups), we estimate that AMF will have about $62.6 million available in 2017 and an additional $2 million per year in each 2019 and 2020, for a total of $66.6 million in 2018-2020.

Uses of additional funding

As noted above, as of October 2017, AMF has set aside $23.9 million for distributions where it expects to sign agreements soon. This amount includes distributions in 2018 and 2019 in five countries, all of which AMF has funded distributions in previously (Ghana, Malawi, Papua New Guinea, Togo, and Zambia).96

AMF told us that it is in detailed discussions about distributions that would cost an additional $93 million. All distributions in this category are expected to take place in the last quarter of 2018, 2019, and 2020 and would be in countries in which AMF has funded distributions previously (DRC, Zambia, Togo, and Uganda); our understanding is that work in DRC may involve new distribution partners, in addition to or instead of those AMF has worked with in the past. If all of the agreements that it expects to sign soon are completed, it will have $35 million (as of October 2017) available to allocate to these additional opportunities.

AMF told us that it is also in early conversations with six countries in which it has not funded distributions previously. AMF roughly estimates that all of these distributions together would cost $80 million to fund. Updating these numbers for more recent data on country-level gaps, we estimate the total at $60 million, but note that there is a large amount of uncertainty both in the data we are using and the likelihood that these discussions will result in distribution agreements.

Expected maximum

We believe that there will be a large funding gap for mass LLIN campaigns in the next three years, but we are quite uncertain about the size of the gap.

The African Leaders Malaria Alliance (ALMA) estimates funding gaps for LLINs across all African countries. In 2014, 2015, March 2016, June 2016, and October 2017 we asked Melanie Renshaw of ALMA for updates on the size of the funding gap. On all occasions she told us that there remained significant gaps in funding for LLINs.97 As of late 2016, Melanie Renshaw roughly estimated there would be a funding gap for 100 million LLINs in 2018-2020.98 As of late 2017, Dr. Renshaw noted that some of the data are preliminary, but estimated that the gap would be roughly 150-180 million LLINs in 2018-2020. Nigeria is expected to make up a large portion of this gap: 74 million LLINs.99

Dr. Renshaw believed that less funding from the Global Fund would be available for LLINs in 2018-2020 than in the previous round because of changes in the way it is structuring its funding.100

At $4.12 per net (this is in variable monetary costs only—excludes in-kind costs and AMF organizational costs)—this is a funding gap of about $620-730 million over three years. For 2018 and 2019 only, the gap is roughly $230-280 million over the two years. The 2020 gap may be larger than in 2018-2019 because some donors may make commitments at a later date and/or because countries may choose to frontload Global Fund funding—the current round of Global Fund funding covers 2018-2020—with the hope of identifying additional sources of funding later.101 As a result, our preference is for AMF to use funds received in late 2017 and 2018 to fund distributions in 2018-2019. We understand that more lead time is beneficial for countries' planning processes,102 but are concerned that committing to distributions far in advance increases the uncertainty about whether funding would have been available from another source.

The gap may be larger if countries begin using PBO or other "next generation" nets.

The 2018 gap is estimated at 19% of the total need for that year, the 2019 gap at 13% of the annual total need, and the 2020 gap at 40%. This is in line with what the end of year gaps have been in recent years. Dr. Renshaw estimated that the unfilled gaps in 2013-2015 were 22 million (13% of the total need), 38 million (15% of the need), and 39 million (18% of the need), respectively.103

Rate of funds moved

We had previously expressed concerns about AMF's ability to scale up.104 We now believe that AMF has a track record of productively spending large amounts of funding annually.

In its 2016 and 2017 fiscal years, it spent $34.3 and $28.5 million, respectively, up from less than $4 million in each of the prior years.105

Progress at signing new agreements was slow in 2017, leaving AMF with a large amount of funds on hand. We attribute this to the fact that countries spent much of 2017 applying for Global Fund funding and decisions about how much funding would be allocated to LLIN distributions for 2018-2020 and what the funding gaps would be for LLINs were being finalized in many countries as of October 2017.106 AMF noted that it did not commit to funding distributions earlier in part because GiveWell had asked AMF not to make funding commitments until the size of funding gaps were known.107

The main risk we see to AMF's ability to spend funds productively is its staff capacity. As of November 2017, AMF employs four staff: a CEO, a technology director, a program director, and a recently-hired operations manager. AMF's executive chairman also spends half his time involved in the management of AMF.108 Since 2016, AMF has had a goal of hiring more staff,109 but our impression is that progress on this has been slow. In the short term, we'd expect hiring staff to reduce AMF's staff capacity as it focuses on recruiting and training. AMF noted (in 2016) that it believed that staff capacity had not constrained its ability to sign agreements or manage distributions.110

Considerations around the size of AMF distributions

Efficiency and leverage

AMF has told us that focusing on large distributions allows it to:111

  • Be more efficient, since its staff capacity is largely limited by the number of distributions it is in discussions about and following up on.
  • Focus on the countries that it has experience working with and where it has relationships with partners.
  • Have more leverage to ask countries to carry out distributions and monitoring according to AMF's preferred processes. Funding a distribution requires negotiating with NMCPs, which we perceive to have some discretion in which funders they work with, and which we perceive to be choosing funders based on a variety of factors, including size and reporting requirements.112 In the past, AMF has been able to fund only a relatively small piece of countries' distributions (Uganda is an exception), but has maintained substantial reporting requirements. This dynamic may create fundamental reasons for governments to prefer partnerships with other funders.
  • Give countries more confidence that their distributions will be fully funded and allow for easier planning and more timely distributions.

For these reasons, AMF's preferred approach would be to offer to purchase all the nets needed for one or more countries' distributions, and, if additional funds were available, fill gaps in LLIN funding for other countries; it would not fund the non-net costs, so this would mean funding about half of the full cost of the distribution.

Fungibility

AMF's approach in the past has been to look for funding gaps—countries that do not have sufficient funding from the Global Fund and other funders for nets—and offer to fill or partially fill those gaps. Given that countries and other funders have some discretion over how funds will be used, it is likely that some portion of AMF's funding has displaced other funding into other malaria interventions and into other uses.

We would guess that this effect would be significantly greater (i.e., there would be more displacement of other funding) if AMF were to pursue the strategy of offering to purchase all of the nets needed for one or more countries, rather than filling in gaps once the Global Fund has made its allocations. It is our understanding that once the Global Fund has told countries how much funding they have been allocated for each disease, it will be difficult to change those allocations and to shift funding to countries that have funding gaps for nets. Therefore, the result of AMF providing a large amount of funding for nets to one country would be for that country to spend more Global Fund resources on other malaria interventions (including general health systems strengthening) or (less likely) to reallocate funds from malaria to AIDS and/or tuberculosis work.

Our conclusion on distribution size

We feel that the risk of displacing a large amount of funding using the approach where AMF purchases all of the nets for one or more countries outweighs the benefits. We have requested that AMF use GiveWell-influenced funding to seek out gaps that other funders are unlikely to fill.

AMF has told us that, starting in 2017, it will require information for all distributions on how the country is spending its Global Fund malaria funding in the preceding and upcoming mass LLIN distributions, and what the other sources of funding are for the preceding and upcoming distributions.113 This should give AMF and GiveWell more insight into the chances that AMF's funding is displacing funding from LLINs into other uses. We have seen partially completed reports for Uganda and Ghana. See footnote for an example from Uganda (not fully complete).114

What portion of all net distributions are funded by AMF?

In 2015, 2016, and the first three quarters of 2017, approximately 470 million nets were distributed in sub-Saharan Africa. 57% of these nets were funded by the Global Fund, 21% were funded by the President's Malaria Initiative, and 9% were funded by UNICEF.115

AMF's contributions accounted for 14.2 million nets in 2015-2017 (this is less than the number of nets that AMF purchased; we have made an adjustment to account for the fact that AMF contributes the nets while other funders pay many of the other costs).116 Therefore, AMF's contribution is about 3% of all nets distributed.117

AMF as an organization

We believe the Against Malaria Foundation to be a strong organization:

Source URL: https://www.givewell.org/charities/amf/November-2017-version