Should AMF have a gold rating?

Would I recommend AMF for a gold rating?

The new survey data could potentially switch AMF to a gold rating if it reliably showed high rates of proper use of nets over the long term and high rates of use by children/pregnant women. On the basis of the new survey data that I have examined, I wouldn't recommend that AMF switch from a silver to gold rating. The main reason for this is that there are a number of issues with the data that I have seen:

  • First, the data doesn't reveal what percent of nets originally distributed by AMF are still in use in the households surveyed. Second, there is a serious potential for selection bias in the condition of nets reports. Third, there is almost no information about what percent of nets are covering children/pregnant women or the total coverage rate of these groups. Fourth, there are potential problems with the data on whether nets are properly installed (though this concern is the least serious). Fifth, I am concerned about selection bias in which organizations report results. And finally, the number of surveys (both in locations and in duration of time post-distribution) seems still too limited to be able to reliably infer from the results to AMF locations overall.
  • I'd recommend waiting for more survey data, and also asking AMF some questions (see last section below.)

    Summary of survey data:

    AMF is now requiring its partner organizations to gather data on nets from 50 households in each of their distribution areas at set points after distribution. The long-term data reporting seems like an good addition to AMF's activities, regardless of concerns expressed below, since it clearly does provide some information on net lifespan and the degree of proper installation, coverage, etc. However, there are some concerns about the accuracy of the results. The following summarizes questions that the data is meant to address, the results from the data available so far, and possible concerns with these results.

    • What is the condition of the nets after a set period of time (6 months, 18 months, etc.)? The data shows that some are in good condition, but it's very unclear whether it gives an accurate picture of the percentage in good condition, because net condition is only reported for an average of 51% of the nets in the surveyed households.
    • What percentage of people in households are covered by nets at night? The percentage of sleeping spaces with nets (compared to total sleeping spaces) is reported: 70% on average. The survey just asks whether the nets are used at night, without asking how many sleeping spaces with nets are using the nets at night (which might lead to inaccurate reporting, if some sleeping spaces have nets, but they aren't used at night). Also, the percent of sleeping spaces covered isn't the same as the number of people covered (since each sleeping space can and often does include multiple people.) This means that we may not have a very accurate picture (from survey result summaries) of what percent of people in a household are covered.
    • What percentage of children under 5 and pregnant women are covered by nets at night? The survey doesn't explicitly ask this question. In 2/16 cases, the survey result summaries include what percent of people under nets are kids/ pregnant women / adults (even in these cases, the number of people included in the tally appears to be far fewer than the total people surveyed, so it doesn't enable us to get a sense of percentages of children / pregnant women under nets). 1 There is no information on what percentage of total children under 5 and pregnant women in households are covered by nets.
    • Do households continue to use nets over long term? Of the 16 locations with survey data available, 10 offer data from one 6-month survey, 3/16 offer data from a 18-month survey, 1/16 from a 30-month survey and 2/16 from a 42-month survey. Overall then, there is relatively little long-term data so far on AMF's nets. Another problem, though, is that we don't have a way of knowing from the surveys what percent of nets originally distributed by AMF are still in the households.
    • What percent of the nets in households are used at night? The percent of nets in the household which are reported to be in use at night is high (average of 96%). So, to some extent, we have evidence that the nets are in use (though as noted above, since this question is yes/no, we may have somewhat inaccurate results if some nets are in use at night and others aren't.) But there is a problem: since we don't know how many nets the households were given originally (compared to number in household / in use at the time of measurement), we don't know what percent of the original nets are still in use.
    • Are nets used properly? It seems so: the results are an average of 93% properly used. However, since the survey simply asks for a yes/no answer, so there may be inaccuracy in the results if some nets are installed properly and others are not.
    • Is a high enough percentage of households surveyed to yield statistically significant results? AMF asks for each organization to survey 50 households in each location. The majority surveyed 50, though some organizations surveyed fewer. In general, it's not clear how many households received nets, so it's not clear what percentage of households are being surveyed. In one case, it is reported that 4,000 households received one net each, and 100 households are surveyed. So 2.5% of households are surveyed. Is this sample size large enough for statistical significance? Are the others? (I don't know the answer to this question.)
    • Can we plausibly infer that AMF's distribution locations surveyed so far show results which are representative of AMF's distribution locations overall? Only 10% of AMF's distribution locations have survey data available (see below for exact numbers).
      • One further problem with the data so far is that there may be significant selection bias in the results. The organizations that agreed to report survey data collectively received about 80,000 nets, and those that refused to participate (2 organizations) received 143,000 nets. So we have data on only 35% of the total nets distributed to organizations from whom AMF has requested surveys so far. It's possible that the organizations (notably, Red Cross) that refused to participate have worse net conditions or use, for some reason. So it's not clear that we can infer that the information we have so far on net condition, etc, is representative of the condition of total nets distributed.

    Surveys and survey data

    Required survey data:

    • AMF is now requiring its partners to provide 1. photos and videos to show that distribution has occurred, and 2. documentation on nets after set periods, gathered by visiting and surveying 50 households in each distribution area. 2
    • So far, AMF has posted information (whether organizations have agreed to do surveys) on 18 locations (in 8 different countries) and from 10 different organizations. All but 2 organizations have agreed to conduct and share survey data 3 ; survey data is available for 16 of the locations on which AMF has provided information so far. Of the 18 locations about which AMF has posted information, 17 have provided photos and 12 have provided videos (of distribution taking place). 4
    • Percentage-wise, there is survey data now available for about 10% of the locations where AMF has nets distributed and for 12% of the organizations AMF works with. (Overall, AMF has nets distributed in 136 locations and works with 66 organizations.)

    The post-distribution survey form

    • AMF's post-distribution survey form is available online. 5 There are a couple of issues with the survey questions themselves which seem that they could potentially lead to unreliable survey data results. 6
      • 1. Net condition, as defined by number and size of holes in the nets: the size of a “small” hole is not clearly defined. It may be implied (by the fact that “very good” condition is defined as “2 holes of 2 cm or less”) that small means smaller than 2 cm, but this isn't clearly stated.
      • 2. The question “are the nets used at night?” doesn't ask how often or how many of the nets are used at night. Suppose that a certain number of sleeping spaces have nets that could be used, but often they are not (whether because it's time-consuming to hang the nets or for some other reason), but that at least one net is used every night. Then, would the right answer to the question “Are the nets used at night?” be “yes” or “no”? It's unclear. This could lead to inaccuracy in the data results.
      • 3. The question about whether nets are installed properly doesn't call for a number, but is just a yes/ no question. Is it plausible that in a number of households, some nets are installed properly and others are not? In that case, the “yes/no” answer won't give very accurate results.

    Potential problems with survey results

    Partial reporting of nets' condition:

    • First, there is a possibility of selection bias: the net condition is reported for only a fraction of the total AMF nets present in the households at the time of the survey. Out of the 16 data summaries provided, 3/16 reported condition for less than 30% of the nets, 4/16 for between 30 and 50%, and 2/16 between 50 and 60%. Overall, 9/16 of the surveys only provide condition data on 60% or less of the nets. Of the remaining, only 1 reported condition for over 80% of the nets.
      • It is possible that the nets which have a condition listed are randomly chosen, and are representative of the net condition overall. But it is also possible that the net which are listed as present but with no condition reported (which is an average of 51% across the 16 organizations) are in worse/better condition than those reported. The concern is that the unreported segment of the total is in worse condition, and that there is a systematic overevaluation of net condition.
      • One piece of information that might be useful is knowing how long a net tends to last (from other studies, if available). It would then be possible to compare AMF's percentages to results from other non-AMF studies, to get a sense of whether there is selective reporting by the groups AMF works with.

    What percent of originally distributed nets are still present in households after X months?

    • Another potential problem with the survey results is that we don't know how many nets were distributed to the surveyed households originally. This could lead to inaccurate results on two of the questions: 1. What is the condition of the nets? 2. What percent of the nets are in use at night? In both cases, the total number of nets distributed originally to the surveyed households could be higher than the total number counted in the household. One hypothetical case in which this could happen is if a number of nets were in use but become worn and were thrown away, if a number of the original nets were sold, etc.

    How is random selection of households done?

    • Another question is on how the “random selection” of households takes place. If the process involves the data collectors walking around and being invited in to certain households, there could be selection bias introduced into the process. Or it could be that households without nets or with nets in poor quality avoid having the visitors.

    Questions on survey data

    Major questions

    • Does AMF know why only 51% of the nets in households have net conditions listed?
    • How accurate are the “proper installation” results likely to be? Is there a plausible chance that even if the demonstration of how the net is used is done right, that other nets are not installed properly, or not done so regularly?
    • If there a way for AMF to find out /calculate how many nets were originally distributed to the surveyed households?
    • Can AMF change its form to ask explicitly 1. How many nets each household had originally and 2. How many children / pregnant women are in the household vs. how many sleep under nets?
    • How is the random selection of households done?

    Other questions

    • How likely is it that some nets in a household are regularly hung at night, but others are not?
    • Do the organizations pay the data collector the same regardless of what the results of the survey are?
    • Is there any chance that there's a perception among either organizations, data collectors or the households that if the nets are in bad shape too soon, the organization will not give more (creating the incentive to hide nets that aren't in good shape)?
    • Are the nets generally taken down during the day? Just pushed aside? (relevant to whether the data collector will see whether nets are properly installed during the day or whether a demonstration is generally necessary).
    • AMF chooses to place a particular symbol (good/ OK / not-OK) next to the survey data. In a couple of the survey data forms, AMF shows graphs with what it considers “very good” and “acceptable” in terms of net condition, and it appears that the “very good” rating requires over 90% in very good condition, and the “acceptable” rating requires about 80% (after 6 months). 7 Yet AMF doesn't seem to follow its charts when giving ratings, i.e., AMF gives one group a “good” though its results are 55 “Good,” 25 “OK,” and 13 “Not-OK” after 6 months. 8 What is AMF's process (i.e., cut-offs) exactly for assigning the various ratings?
    • When there are uncovered sleeping spaces in the household, is there space to hang more nets?
    • Why did Red Cross refuse to conduct surveys? Is there any reason to think that the net condition is different for them?
    • When will further survey results be released?

    Sources

    • 1 AMF, “Mayo Farms Post-Distribution Survey”
    • 2 “We ask all distribution partners to sign a legal agreement.
      This has three key elements:
      1. We ask Distribution Partners to provide us with photographic and video evidence the distribution has occurred and a written post-distribution report.
      2. We ask the Distribution Partners to perform post-distribution surveys at set intervals over the following 3 years.
      3. If the nets were not to be delivered for some reason, we get our (your) money back.” AMF, “Distribution Strategy”
      For each organization which refused to do the survey, AMF posts this note: “We are disappointed at the Distribution Partner's refusal to carry out a Post-Distribution Survey. Whilst not a requirement in the Distribution Proposal at the time we agreed to partner on this distribution (it is a requirement now), ongoing assessment of net usage is very important. If net usage levels have slipped, we all need to know that. Renewed malaria education messages within the community can ensure high levels of net use are achieved and maintained. The reality with net distributions is this community-wide intervention is sometimes required. We consider the effort required, visiting 50 households in a distribution location, to be very modest given the significant effort and funding that goes into achieving a successful bednet distribution and the importance of continued net use to protect a community. Given the majority of Distribution Partners are diligent in carrying out Post-Distribution Surveys, we would not consider working with this Distribution Partner again at this time.” AMF, “Pokause”
    • 3 For some reason, AMF has an “expected” icon for future survey dates (e.g., 18 months) even for organizations who refused to give surveys so far (e.g., at 6 months). I'm not sure if this is a website error, or if this indicates that the organizations have decided to do surveys in the future. See for example AMF, “Pojuase”
    • 4 AMF, “Reports”
    • 5 AMF, “Post-distribution Survey Form”
    • 6 AMF, “Post-distribution Survey Form”
    • 7 AMF, “Maya Forms Post-Distribution Survey”
    • 8 AMF, “Bujanko Island”