# Against Malaria Foundation - August 2015 Update

Published: August 2015

## Summary

Bottom line: We continue to recommend the Against Malaria Foundation (AMF) as an excellent giving opportunity.

• AMF has published results from its distribution in Kasaï Occidental, Democratic Republic of the Congo, including details of how the distribution was carried out, photos and video of the distribution in progress, household data, and lessons learned. This was the first large-scale distribution (i.e., a distribution on the order of hundreds of thousands of nets) that AMF completed with a partner other than Concern Universal and in a country other than Malawi. AMF received monitoring data for most of the nets that were distributed. Results from a six-month follow up survey are expected within the next few months.
• AMF has continued to follow up on past distributions. It has recently published the results of three post-distribution check ups (PDCUs) on distributions in Malawi: a 33-month PDCU for the 2011-2012 distribution in Ntcheu, a 14-month PDCU for the 2013 distribution in Balaka, and an 8-month PDCU for the 2014 distribution in Dedza.

• The distribution in Dowa, Malawi has been completed. Full documentation is not yet available and is expected in the coming months.
• The distribution in Nord Ubangi, Democratic Republic of the Congo (730,000 nets) has been delayed.

• AMF is working to finalize five new distribution agreements. AMF has told us that it is in fairly advanced stages of discussion with all five countries.
• AMF currently has $11.1 million in uncommitted funding and an additional$5.8 million that it could allocate to distributions in the next year (because the funds are committed to distributions that will not take place until late 2017 or 2018).

• AMF has continued following up long-term on completed distributions. We have seen the results from three recent surveys in Malawi: a 33-month post-distribution check up (PDCU) on the 2011-2012 Ntcheu distribution in Malawi (56% of the people surveyed were sleeping under LLINs), a 14-month PDCU on the 2013 distribution in Balaka (73% under LLINs), and an 8-month PDCU on the 2014 distribution in Dedza (82% under LLINs).
• AMF has published documentation for the Kasaï Occidental distribution (completed in November 2014), which was AMF’s largest distribution to date and its first large-scale distribution outside of Malawi. In general, monitoring of the distribution was similar to that of AMF's previous large-scale distributions, though it appears there were fewer procedures to audit the accuracy of household data. We expect data from the first post-distribution check up to be available soon.

A summary of AMF’s completed distributions, the documentation that is currently available for each of the completed distributions, and the documentation that is expected in the future, is here.

More details on the updates are below.

### Post-distribution check up surveys for distributions in Malawi

#### Ntcheu district

In December 2011 to February 2012, AMF, with Concern Universal, distributed about 270,000 nets in the district of Ntcheu, Malawi.1 Concern Universal has now completed its fourth post-distribution check up (PDCU) in the district 33 months after the distribution.2 Previously, Concern Universal had completed post-distribution check ups at 6, 15, and 24 months post-distribution.3

For this PDCU, a team of ten data collectors travelled to 37 Health Centre Areas (HCAs) and randomly selected 5% of the villages in each area to visit.4 In each village, the data collectors randomly selected 5% of the households that had received nets during the distribution to visit.5 The data collectors filled out a survey form for each household, which included information on the condition of the nets in the household and the number of nets hung over sleeping spaces.6 The survey results include data on net usage.7 The report does not specify how usage data was collected; AMF told us that the data collectors directly observed whether nets were hung and what condition they were in.8 Two supervisors randomly selected 5% of all of the households surveyed and revisited those households to check the accuracy of the data collectors’ work;9 we have not seen any details on who the supervisors were or any results from the supervisors’ random follow-up surveys. Once the data collectors submitted their data, the supervisors checked the data for errors.10 The data was then sent to four data entry clerks, who entered it into a web interface designed by AMF.11

Overall, no major issues were reported.12 Although $14,448 was budgeted for the post-distribution check up, the final cost of the survey was only$12,607,13 due in part to a change in the exchange rate between Malawi kwacha and U.S. dollars.14

Details on the results from the PDCU are below.

#### Balaka district

From October to December of 2013, AMF, with Concern Universal, distributed about 155,000 nets in the district of Balaka, Malawi.15 Concern Universal has now completed its second post-distribution check up survey in the district, 14 months after the distribution.16 Originally, Concern Universal planned to do the survey 12 months after the distribution, but the PDCU was delayed several months.17 Previously, Concern Universal completed a post-distribution check up 6 months after the distribution.18

The methodology for Balaka’s 14-month PDCU exercise was nearly identical to that of Ntcheu’s 33-month PDCU (see above for details).19 Like the 33-month Ntcheu PDCU, the 14-month Balaka PDCU came in under budget: $9,667 was originally allocated to the exercise, but only$8,337 was ultimately spent.20 This was again due to the favorable exchange rate between Malawi kwacha and U.S. dollars.21

Details on the results from the PDCU are below.

#### Dedza district

In September and October 2014, AMF, with Concern Universal, distributed about 245,000 nets in the district of Dedza, Malawi.22 Concern Universal has completed its first post-distribution check up survey in the district, 8 months after the distribution.23 Originally, Concern Universal planned to do the survey 6 months after the distribution, but the PDCU was delayed several months.24

### AMF’s current pipeline

Our last update noted that AMF was discussing potential future distributions with two countries (Country X and Country Y - AMF prefers not to share the names of countries it is negotiating with).102 Originally, AMF was under the impression that discussions with each of these countries would move forward quickly.103 However, discussions progressed more slowly than expected, so AMF decided to proceed with discussions with several more countries, in order to mitigate the risk of its discussions with Country X and Country Y not progressing.104

Currently, AMF is in discussion with five countries (including Country X and Country Y) about potential future distributions; most of these discussions are for one million to several million nets during 2015-2017.105 AMF told us that it is in fairly advanced stages of discussion with all five of the countries, meaning that:106

• AMF has sent a draft agreement to each country’s NMCP for approval and either a) has already received informal approval from the NMCP and is moving forward with other distribution arrangements or b) has had positive discussions with the NMCP but is still waiting on approval.
• AMF has a promising or committed co-funding partner for the non-net costs in each country.
• AMF has identified one or more promising candidates for its distribution partner in each of the countries, and is currently in the process of investigating the candidate(s).107

AMF does not have enough funding to cover the full net need in all five countries and has told each country that the longer it takes to come to an agreement, the less funding AMF might have (because agreements with other countries could be signed first).108 AMF is not planning to fund non-net costs for any of the countries it is in discussions with.109 AMF hopes that one or more agreements for net distributions will be signed within the next few months, but notes that there is a high degree of uncertainty around predicting when distribution agreements will be signed.110

### AMF’s current funding situation

AMF currently holds about $21 million in funding, of which$9.9 million is already committed to future distributions, leaving $11.1 million available for future distributions.111 Of the$9.9 million, $4.1 million is committed to distributions in the near future (within the next 9 months), and the remaining$5.8 million is committed to distributions in Malawi at the end of 2017 and in 2018.112

AMF has mentioned before that it might use funding committed to distributions in the far future for distributions that are occurring sooner, because AMF expects that it will be able to raise the funds for the distributions in the far future at a later date.113 This implies that AMF has $16.9 million that it could use for future distributions in the near term.114 AMF has told us that the total net need for 2015-2017 of the five countries in which it is in discussions is 32 million nets.115 At$3 per net, it would cost $96 million to fully fill the funding gap for nets in these countries.116 ## Malaria case rate data In the distribution agreements for the Ntcheu 2011-2012, Balaka 2013, and Dedza 2014 distributions, AMF requested that Concern Universal provide monthly malaria case rate data from before and after the distributions from all health centers in the three districts.117 After this data was delayed multiple times, AMF informed us in February 2015 that Concern Universal had sent the data and that AMF would publish it in a month or two.118 As of the writing of this update, the data has not been published, and when we last spoke to AMF we were told that the publication of the malaria case rate data was not a high priority.119 AMF told us that it has not made this a high priority because it does not believe that the data is high enough quality to reliably indicate trends in the malaria case rate.120 We no longer see the fact that AMF collects malaria case rate data as a consideration in AMF’s favor. ## Data collection for other distributions in Malawi We noted in our last update on AMF that the Malawi NMCP is planning to implement some of AMF’s distribution practices in districts for which AMF does not fund distributions.121 The practices will be implemented in 18 of 28 districts during a large distribution of approximately 7 million nets occurring towards the end of 2015.122 The practices that will be implemented are:123 • Organizations will collect pre-distribution registration data for all households • Organizations may require that supervisors revisit 5% of the households that were surveyed for the pre-distribution registration to verify that data collectors’ work was accurate124 AMF has also advocated for (a) counting sleeping spaces (as opposed to people) during the pre-distribution registration, and distributing nets according to sleeping spaces;125 (b) putting all data in electronic form;126 and (c) post-distribution check ups.127 These practices have not been made requirements.128 Over the last several years, AMF believes that its high net hang-up rates have begun to catch the attention of a number of groups in Malawi, including the NMCP, local organizations that distribute nets, and large international NGOs.129 AMF also told us that it encouraged Concern Universal to take a more active role in a task force on malaria control in Malawi.130 This meant that as interest in learning from AMF and Concern Universal grew, Concern Universal was available to explain AMF’s methodologies at the malaria task force meetings, and it was eventually decided to implement some of AMF’s practices for the upcoming distribution.131 ## Sources Document Source AMF all distributions page Source (archive) AMF blog post, February 18, 2015 Source (archive) AMF blog post, July 3, 2015 Source (archive) AMF dollars per net page Source (archive) AMF DRC West Kasai distribution summary numbers Source AMF financial information page Source (archive) AMF funds history and status Unpublished AMF future distributions net need Unpublished AMF how we work with distribution partners Source AMF page on Balaka 2013 distribution Source (archive) AMF page on Dedza 2014 distribution Source (archive) AMF page on Dedza 2017 distribution Source (archive) AMF page on Dowa 2015 distribution Source (archive) AMF page on Kasaï Occidental 2014 distribution Source (archive) AMF page on Nord Ubangi 2015 distribution Source (archive) AMF page on Ntcheu 2011-2012 distribution Source (archive) Balaka 2013 14-month PDCU data Source (archive) Balaka 2013 14-month PDCU report Source (archive) Balaka 2013 6-month PDCU report Source (archive) Balaka 2013 distribution proposal Source (archive) Balaka 2013 distribution report Source (archive) Dedza 2014 8-month PDCU data Source (archive) Dedza 2014 8-month PDCU report Source (archive) Dedza 2014 distribution proposal Source (archive) Dedza 2014 mid-distribution weekly reports Source (archive) Dowa 2015 mid-distribution weekly reports Source (archive) GiveWell AMF review 2014 Source GiveWell AMF update August 2014 Source (archive) GiveWell AMF update May 2015 Source (archive) GiveWell estimate of AMF cost per net (November 2014) Source Kasaï Occidental 2014 summary and sample data Source Kasaï Occidental distribution report Source (archive) Kasaï Occidental technology report Source (archive) Malawi NMCP invitation for 2018 distribution Source Ntcheu 2011-12 24-month PDCU report Source (archive) Ntcheu 2011-12 33-month PDCU data Source (archive) Ntcheu 2011-12 33-month PDCU report Source (archive) Ntcheu 2011-12 distribution report Source (archive) Rob Mather and Peter Sherratt, conversation with GiveWell, February 6, 2015 Unpublished Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015 Unpublished Rob Mather, conversation with GiveWell, February 24, 2015 Unpublished Rob Mather, email to GiveWell, February 24, 2015 Unpublished Rob Mather, email to GiveWell, June 1, 2015 Unpublished Rob Mather, email to GiveWell, June 19, 2015 Unpublished Rob Mather, email to GiveWell, September 6, 2015 Unpublished Rob Mather, email to GiveWell, June 30, 2014 Unpublished Rob Mather, email to GiveWell, May 22, 2014 Unpublished • 1. • 2. • 3. • 4. "Ntcheu district has 37 health facilities. It was recommended to collect data from 5% of households in each HCCA [note: while HCCA is never defined in the report, we believe it is the same as HCA], which meant a different number of households in each HCCA as per individual health facility populations (refer to appendix A). Between 100 and 550 households were randomly selected from each of the selected four to 22 villages, depending on the HCCA. Villages were randomly selected using the village lists generated from the pre-distribution and distribution work for the December 2011-April 2012 AMF-funded universal coverage LLIN distribution. A random number table was used to select the villages." Ntcheu 2011-12 33-month PDCU report, Pg 6. • 5. "Between 100 and 550 households were randomly selected from each of the selected four to 22 villages, depending on the HCCA. Villages were randomly selected using the village lists generated from the pre-distribution and distribution work for the December 2011-April 2012 AMF-funded universal coverage LLIN distribution. A random number table was used to select the villages." Ntcheu 2011-12 33-month PDCU report, Pg 6. • 6. "The orientation included detailed explanation of the LLIN Distribution project back ground, how the beneficiaries were identified, the process of registration, data verification and LLIN net distribution, survey objectives and the logic behind the survey form (net condition, type of nets, what sleeping spaces are, what is meant by hung nets and noting hung nets against AMF nets received) as well as having the data collectors pre-test exercise in order to fill in sample forms and ask questions to ensure their understanding of what information should be collected and how." Ntcheu 2011-12 33-month PDCU report, Pgs 5-6. • 7. • 8. "Data was collected by Health Surveillance Assistants (HSAs) via direct observation of whether the nets were hung and inspecting the nets’ condition." Rob Mather, email to GiveWell, September 6, 2015. • 9. • "The supervisors were responsible for checking the data collection exercise at the same time monitoring how the data was being collected as per requirement." Ntcheu 2011-12 33-month PDCU report, Pg 6. • "Supervisors were required to visit 5% of the households being interviewed to check the accuracy of the data collectors’ work and had to check all the completed forms submitted to them before submitting them to the Project Manager. The sampled visited households were also chosen at random so the work of all data collectors was checked." Ntcheu 2011-12 33-month PDCU report, Pg 6. • 10. "Once the data collection was complete, the data collectors submitted completed forms to their assigned supervisor who was responsible for checking the forms for obvious errors or omissions, including a lack of householder signature, before delivering the forms to the data entry team." Ntcheu 2011-12 33-month PDCU report, Pg 6. • 11. "There were four data entry clerks with knowledge in basic computing. The data entry clerks were also exposed to a questionnaire orientation where they were briefed on the forms and introduced to the online web links and how to enter the data on the electronic form, make editions and post the data. The data entry clerks were assigned specific health facilities in order to facilitate their performance monitoring. Data was entered into a database via a web interface created by AMF. An internet connection was required for this work." Ntcheu 2011-12 33-month PDCU report, Pgs 6-7. • 12. • "The operational elements that went well were: • All the selected villages were visited. • There was a positive response from the LLIN beneficiaries at community level. • The survey form was short with only one page, which was ideal for the data collectors and the respondents • Local community leaders and household heads allowed the data collectors to enter their households to see the hung nets and check the condition they were in. • Management support and commitment towards the activity by Concern Universal and District Health staff was very encouraging, hence the timely execution of the exercise. • The data collectors were committed to collecting the data. The elements that did not go so well were: • In some selected villages, on a planned data collection day, the team faced challenges due to funerals hence data collection was delayed." Ntcheu 2011-12 33-month PDCU report, Pgs 7-8. • This was the first time that Concern Universal used a small group of data collectors for the entire survey; in previous PDCUs Concern Universal used different data collectors for each of the 37 HCAs: "There are 37 Health Centres (HCs) in Ntcheu District. Each has approximately 20 staff attached to each one, the majority being salaried Health Surveillance Assistants (HSAs). However, in collaboration with the District Environmental Health Office (DEHO) and Malaria Coordinator (MC) and lessons leant from 24 month Ntcheu PDCU, it was recommended to have a focused team of 10 data collectors rather than have the HSAs as data collectors from each HCA. This was based on the following reasons. First, this would reduce the number of data collectors that would need to be monitored and trained. Second, we would be able to select reliable individuals whom we could trust to do a diligent and accurate job of collecting the data. Third, it would leave the majority of HSAs to carry on with the normal health tasks and duties. Fourth, by having the same people covering the whole exercise they will get acquainted to the task and reduce errors on data collection. This meant the data collectors would spend thirty seven days collecting data rather than the one or several days if many more data collectors were to be used. This was judged the preferable way of organising and managing the data collection phase." Ntcheu 2011-12 33-month PDCU report, Pg 5. • The new strategy seems to have worked well; Concern Universal recommended that it be used in future PDCUs: "The same data collectors should be hired to collect the data for the whole exercise in the upcoming subsequent PDCU surveys. Likewise the same data entry clerks should be involved in the next subsequent upcoming PDCUs since they are already familiar with the system." Ntcheu 2011-12 33-month PDCU report, Pg 8. • Concern Universal also recommended that permanent vehicles be used in future PDCUs to avoid transportation delays: "In order to maintain and follow the timeline and meet the deadlines permanent vehicles should be allocated to the activity." Ntcheu 2011-12 33-month PDCU report, Pg 8. • 13. "The budget was MK 5,779,367.00 (US$14,448) and the total PDCU cost was MK 5,774,064.00 (US$12,607). MK = Malawi Kwatcha." Ntcheu 2011-12 33-month PDCU report, Pg 7. • 14. Comment from AMF: "The PDCU came in exactly on budget in Malawi Kwatcha. Given the exchange rate MK:USD moved in our favour, in USD terms, the cost was approximately 15% less expensive than expected. CU has managed the budget very well." Ntcheu 2011-12 33-month PDCU report, Pg 7. • 15. • 16. • 17. "This check-up was carried out at 14 months, rather than 12 months, post-distribution due to planning delays." Balaka 2013 14-month PDCU report, Pg 3. • 18. • 19. • 20. "The budget was MK 3,866,920.00 (US$ 9,667) and the total PDCU cost was MK 3,818,565.00 (US$8,337). MK = Malawi Kwatcha." Balaka 2013 14-month PDCU report, Pg 7. • 21. Comment from AMF: "The PDCU came in on budget in Malawi Kwatcha. Given the exchange rate MK:USD moved in our favour, in USD terms, the cost was approximately 15% less expensive than expected. CU has managed the budget very well." Balaka 2013 14-month PDCU report, Pg 7. • 22. • AMF page on Dedza 2014 distribution. • Note: Since the writing of this update, AMF has updated its website to show that 290,770 nets were distributed in the Dedza 2014 distribution. This is because AMF recently completed a large mop-up distribution for the Dedza 2014 distribution; the mop-up distribution gave out approximately 45,000 nets to recipients who were missed during the initial distribution. More on the the mop-up distribution can be read here. • 23. • "This check-up was carried out at 8 months, rather than 6 months, post-distribution due to planning delays." Dedza 2014 8-month PDCU report, Pg 3. • "One of the CU vehicles transporting data collectors and the Malaria Coordinator was involved in an accident during the final week of data collection, resulting in the total loss of the vehicle as well as injuries to the staff members and the Malaria Coordinator. This accident delayed the data collection exercise as the injured staff members required time off to recover from the physical and psychological trauma. A replacement vehicle had to be arranged in order to proceed with the activities. Note: The staff injured have all recovered fully. The Malaria Coordinator suffered minor injuries that did not require medical treatment. All were able to resume work after a few days off." Dedza 2014 8-month PDCU report, Pg 7. • 24. • "The check-up was carried out at 8 months, rather than 6 months, post-distribution due to planning delays." Dedza 2014 8-month PDCU report, Pg 3. • "One of the CU vehicles transporting data collectors and the Malaria Coordinator was involved in an accident during the final week of data collection, resulting in the total loss of the vehicle as well as injuries to the staff members and the Malaria Coordinator. This accident delayed the data collection exercise as the injured staff members required time off to recover from the physical and psychological trauma. A replacement vehicle had to be arranged in order to proceed with the activities. Note: The staff injured have all recovered fully. The Malaria Coordinator suffered minor injuries that did not require medical treatment. All were able to resume work after a few days off." Dedza 2014 8-month PDCU report, Pg 7. • 25. • 26. "The PDCU budget cost was US$12,849 equal to US$1.41 per household visited (or$0.052 per net originally distributed). The actual cost are still being collated and this document will be updated to include them in the coming weeks." Dedza 2014 8-month PDCU report, Pg 3.

• 27.
• "The actual cost are still being collated and this document will be updated to include them in the coming weeks." Dedza 2014 8-month PDCU report, Pg 3.
• AMF notes that "Actual cost numbers are reported to us typically three months after the end of the activity when all cost items have been gathered by CU’s finance department. We expect to have the actual cost by the end of September. A new dedicated malaria-related accountant started at CU Malawi on 01Sep." Rob Mather, email to GiveWell, September 6, 2015.
• 28.
• 29.
• 30.

"'very good condition' (fewer than 2 holes of up to 2cms in size)." Ntcheu 2011-12 33-month PDCU report, Pg 4.

• 31.
• 32.
• 33.

"Worn out means not usable. Typically, holes will be 10 cms of cms in size and numerous and the net considered effectively unrepairable and/or structural integrity will have gone. If hung the net would not provide protection against mosquitoes given the extent of holes or tears or other loss of structural integrity." Rob Mather, email to GiveWell, June 30, 2014.

• 34.
• 35.

"Alliance for Malaria Prevention 2011 lays out a model for estimating the number of LLINs still in use after distributions: 'the number of LLINs already distributed over the last three years and considered to be available in households should be calculated and subtracted from the total need, working with a decay rate of 8 per cent at one year (0-12 months), 20 per cent at two years (13-24 months) and 50 per cent at three years (25-36 months).'…
Bottom line: We believe that the "8%-20%-50%" model is the most widely used and most reasonable approximation available at the moment for capturing the extent to which LLINs remain in use in the years following distribution, accounting for any factors that might cause LLINs to be discarded or additional LLINs to be purchased. It implies an average of 2.22 years of use for each LLIN distributed. Data and analysis on this topic appears extremely thin; we have little sense for how long LLINs last in practice." GiveWell report on the "decay model" for LLINs.

• 36.

See the table above.

• 37.

"This suggests for other distribution areas showing a similar decline in coverage over time other actions are necessary if the target is to maintain a minimum 80% sleeping space coverage throughout the typical three year period between mass universal coverage distributions. We are considering with partners various developments. One possible development is an ‘injection strategy’ that brings in a quantity of nets at 24 months, and perhaps at 12 months, to increase net coverage to a level that ensures any decline over the subsequent year leaves sleeping space coverage at or above 80%. This would involve a mechanism by which sleeping spaces in need of a net, across all households, were accurately tallied at the 24 month, and perhaps 12 month, point. This could involve a district wide ‘pre-distribution registration survey’, similar to that carried out prior to a mass distribution or could perhaps be achieved through engagement of local communities. The practicality of different approaches, associated costs and the coverage likely to be achieved is being assessed. What is clear is that sleeping space coverage is below 80% during a significant part of year three post-distribution and better malaria control is likely to be achieved through maintaining an 80% or above coverage level during this third year." Pg. 4, Ntcheu 2011-12 33-month PDCU report.

• 38.

See the table above.

• 39.
• 40.

See the table above.

• 41.
• 42.
• 43.
• 44.
• 45.
• 46.
• 47.

The distribution occurred March through May 15th. The weekly reports cover March 23 - 27, April 27 - 30, and May 4 - 8. The report from the third week notes, "To date we have distributed 184,554 nets of a total of 391,608 nets required, representing 47% of the whole district’s coverage." Dowa 2015 mid-distribution weekly reports.

• 48.
• "The major challenge we encountered during the distribution was the breaking down of our distribution vehicles, which forced us to interrupt the distribution for almost one month to have them fixed, as some spare parts could not be sourced locally." Dowa 2015 mid-distribution weekly reports, Pg 2.
• AMF does not require Concern Universal to produce weekly reports on the distribution; Concern Universal does this of its own accord. We believe these reports reflect well on Concern Universal’s commitment to transparency, and they increase our confidence in Concern Universal’s ability to discover problems with a distribution. The weekly reports from the Dowa 2015 distribution had a similar level of detail to previous weekly reports from other distributions (e.g., see the Dedza 2014 mid-distribution weekly reports).
• 49.

See our May 2015 update on AMF.

• 50.
• 51.
• 52.
• 53.
• "This distribution is the first that both parties have undertaken using smartphones to collect household-level data. We expect teething problems, as with any new introduction of technology, especially in a country such as DRC. However, we are undeterred by this, as the benefits of improving processes in this region are so significant." AMF blog post, February 18, 2015.
• Some of the benefits AMF believes come from the use of smartphones include:
"1. Acts against potential theft Data in electronic form can be shared widely and easily so net numbers can be tracked with precision and visibly. This inhibits theft, making occurrences obvious, quickly. There was no material loss of nets in this distribution.
2. Improves accountability Detailed data, at the household level, can be shared with partners demonstrating what has taken place. That was the case here.
3. Increases transparency All partners, including donors and the wider public, have access to detailed data and can see what has taken place. That was the case here.
4. Increases data accuracy Data can be processed more quickly and errors spotted with time for corrections to be made prior to the net distribution taking place. That was not the case here as the registration and distribution were contemporaneous.
5. May reduce cost Data processing is simpler compared to paper based data. We are waiting for final details of cost numbers to assess if costs were lower and if so by how much.
6. Additional data can be collected GPS data and photos can be easily collected. That was the case here.
7. Reduces operational risk As long as electronic data is backed up, the risk of data loss, compared to data in paper form, can be significantly reduced or eliminated. While the vast majority of data was accurately stored and transferred, some lessons have been learned to improve in this area." Pg. 4, Kasaï Occidental technology report.
• 54.

The distribution report includes an explanation of the methodology used for the distribution, as well as detailed reports of problems encountered during the distribution and plans to address those problems in the future. Kasaï Occidental distribution report.

• 55.
• 56.
• 57.
• 58.

AMF has told us that it has a distribution proposal from IMA but that publishing that publishing it is a lower priority task. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015.

• 59.
• "The nets were securely stored at Lofice, the transport company used with the following security: 24 hour guards, secured enclosure and secured internal warehouse. The nets remained in their containers and on the trucks for a week before onward road travel to Tshikapa District." Kasaï Occidental distribution report, Pg 5.
• "At each net storage site there was a warehouseman who wrote down the quantity of nets entering or leaving. Specifically, the number of bales was counted in/out (rather than a verbal statement of quantity by one person being accepted) and two signatures placed against the number recorded." Kasaï Occidental distribution report, Pg 19.
• "In each health area there were at least two storage sites for supplies (nets plus other materials). Daily movement of nets was strictly recorded in the daily stock ledger so that it was known how many nets had gone to each Health Area or village. Any nets remaining after the distribution were returned to the secure storage locations. Any movement of nets involved manual counting of the bales of nets with two individuals having to sign for the precise quantity leaving, or being received at, the secure storage area and being loaded on, or unloaded from, the transport vehicle. The same process was required to be done when the nets arrived at a distribution location, with the transport driver and the individual in charge of the distribution location, both counting and signing for the nets." Kasaï Occidental distribution report, Pgs 18-19.
• 60.
• 61.
• "This was a crucial stage of the net distribution that consisted of collecting relevant household data for every single household in the distribution catchment area including, but not limited to, the following ‘must have’ information:
• Number of people in the household
• Number of sleeping spaces (‘for the purpose of nets’*)
• Number of perfectly good nets with at least two years of life left in them
• Location of household (GPS coordinates)
• Household identifier"
• "The CHW asks for additional information not contained in the registration questionnaire (incidents of malaria, socio‐demographic information, what type of medication they took for malaria and information on the demographic make‐up of the household to enable the collection and integration of the data." Kasaï Occidental distribution report, Pg 17.
• 62.
• "The CHW responsible for entering household data (CHW with the smartphone) gathers the required data with the help of the head of household." Kasaï Occidental distribution report, Pg 17.
• "The CHWs visiting beneficiaries kept a hard copy list of the household information entered in the smartphone in case of data failure or a need to check data" Kasaï Occidental distribution report, Pg 19.
• We do not know if IMA will continue to collect data by paper in future distributions or if this was a one-time methodology decision as insurance against the smartphones completely failing.
• 63.
• "The CHW responsible for the hanging of the nets does so with the help of the head of household and/or other household members, using the string and nails brought along if necessary. A net is hung over each sleeping space (two people per sleeping space) and then rolled up out of the way (for example off the ground) during the day to help avoid unnecessary wear and tear or damage. It is then rolled down at night." Kasaï Occidental distribution report, Pg 17.
• It is our understanding that if a household had a high-quality net that could last an additional 2 years, IMA would hang this net up instead of providing a new one (or leave it in place if it were already hung). We believe this because a) AMF's net distributions in Malawi did not provide new nets if a household already has usable nets and b) IMA collected information on the "number of perfectly good nets with at least two years of life left in them." We think it is likely that IMA collected this information to determine how many nets to give. Kasaï Occidental distribution report, Pg 17.
• We are not entirely clear on how IMA determined the appropriate number of nets to give to each household, although we believe that IMA relied on the number of sleeping spaces in a household. The following quote is in the distribution report: "When assessing the number of sleeping spaces it is important to avoid confusion between the number of separate areas in which individuals might sleep and, given nets distributed are ‘double’ nets that will accommodate two people, the number of sleeping spaces that is equal to the number of nets that are needed. For example, in a household with mum plus dad, four children aged eight, four, six and two and a grandfather there may be six separate sleeping areas: mum plus dad, one each for the four children and one for the grandfather. However, for the purposes of nets, there would be four sleeping spaces: one for mum and dad, one for the two older children, one for the two younger children and one for the grandfather." Kasaï Occidental distribution report, Pg 16. To continue with the example quoted, we believe that this could mean that if the six separate sleeping areas in the household were spread out in such a way that 4 nets could not cover them, the household would still only receive 4 nets, and would have to move its sleeping areas closer together if it wanted to cover everyone. We are not sure how easy it is for a household to move sleeping areas, or how it was decided where to hang nets in cases like this.
• On the other hand, when we look at the sample data collected, it does not seem to support this interpretation. For example, row 9 shows 5 people living in a household, and there are 5 sleeping spaces in the household. Because the household already had 2 useable LLINs, it received 3 from IMA. This seems to imply that even if there was only one person using a sleeping space, IMA would still give a net to cover the whole sleeping space, and not require that the sleeping space be moved close to other sleeping spaces so that two people could be covered with one net. Sample Data-Anonymized sheet, Kasaï Occidental 2014 summary and sample data.
• 64.
• "In order to both a) avoid returning to a household that had already received nets and b) be able to identify the household during subsequent post‐distribution follow‐up, a CHW marked a unique household identifier using chalk on the outside of the household in an easily visible place (for example on a doorframe, door or other front area)." Kasaï Occidental distribution report, Pg 17.
• "Creating a household identifier is necessary as many households do not have clear addresses. For example, it is frequently the case in rural and semi‐rural locations that street names and household numbers do not exist. The household identifier was created as one long number, as follows: Day of distribution + Unique number of the smartphone being used to collect the data + Household number for that day’s distribution + Number of sleeping spaces + Number of nets installed. For example: 171115‐2365714‐015‐04‐04." Kasaï Occidental distribution report, Pgs 16-17.
• We are not sure if the chalk is easily wiped off or might be washed away when it rains. We are also not sure if there was another way for the CHWs to determine which households had already been covered, although we imagine that since nets were being hung up in households as the distribution went, it was somewhat easy to tell which households had already been visited.
• 65.
• "The final step for the CHWs was to pass on information to as many household members as possible regarding the correct use of the net, including: tucking the net under mats or mattresses at night; not leaving any gaps through which mosquitoes could pass; rolling up the net during the day to minimize damage; avoiding causing holes and rips in the net; the importance of repairing the net if holes develop; and lastly, the ability to wash to net 20 times in the next 3 years." Kasaï Occidental distribution report, Pg 17.
• "Once hung, the CHW took a photo using the smartphone to record an image of the successfully delivered and hung nets. If the householder did not want to be in the photo, just a photo of the net was taken." Kasaï Occidental distribution report, Pg 25.
• 66.

"At the end of each distribution day each smartphone was brought to the Health Area’s main health centre and, via a WiFi connection, the data collected on the smartphone was uploaded (transferred) to a locally located laptop." Kasaï Occidental distribution report, Pg 17.

• 67.

"At the end of each day, the CHWs returned the smartphones to the Field Supervisor and the data was imported into a computer using a WiFi router connection. The Field Supervisor checked the data to make sure the household registrations had been carried out correctly." Kasaï Occidental distribution report, Pg 25.

• 68.

"After the distribution had been completed in a Health Area and all data uploaded from the smartphones, the data team compared the electronic data with the paper data collected by each distribution site in each health area." Kasaï Occidental distribution report, Pg 18.

• 69.

"Household data went through the following steps: 1. Data collected at the household by a CHW 2. Independent supervisor uploaded the data from the smartphone to a laptop. 3. The supervisor reviewed the data and corrected obvious mistakes or inconsistencies. 4. The data across a Health Area was compiled into one data set 5. The data was transported to the Health Zone central office via a USB memory stick and compiled to give a complete data set for the Health Zone. There were between 17 and 28 Health Areas in a Health Zone. 6. The data was sent by USB memory stick and hard disc from the Health Zone level to the Assistant Data Manager at the central level for overall compilation 7. The data was analysed by the Data Manager and Assistant Data" Kasaï Occidental distribution report, Pg 18.

• 70.
• 71.
• "As the data received was after distribution of the nets, analysis of the data would not lead to cleaned and corrected data being sent back to the field to influence the number of nets distributed to a particular household. This was because household registration and distribution of nets – and indeed hang‐up of the nets ‐ was contemporaneous. IMA wished to conduct the distribution in this way to gain information on high hang‐up rates expected to be achieved through immediate hang‐up of the nets. The aim of the analysis was therefore to assess the accuracy of the distribution as judged by: 1. How accurately did the data match records for the number of nets moved to each pre‐ positioning and final distribution point? ...2. How free was the data of ‘inconsistency’ and ‘anomaly’ errors?" Kasaï Occidental technology report, Pgs 13-14.
• Note that this is different from AMF’s distributions with Concern Universal in Malawi, where pre-registration survey data is checked by a) supervisors randomly revisiting some of the households that were surveyed to ensure that the data is accurate, and b) village meetings where Concern Universal reads off household names and the number of LLINs allocated to that household, and households indicate if errors have been made. All of this happens before the distribution and thus can have an impact on the number of nets a household receives. GiveWell AMF review 2014.
• 72.

"The IMA Monitoring and Evaluation (M&E) team is currently in the process of analysing data for other indicators such as prevalence of malaria and will distribute a separate IMA report of the findings." Kasaï Occidental distribution report, Pg 18.

• 73.
• "For this distribution, Nyanga was considered a pilot area and a separate net distribution was conducted there shortly before the other HZs received nets." Kasaï Occidental distribution report, Pg 4.
• "A pilot distribution was used to test the technology at a small but sufficient scale so lessons could be learned before larger roll out." Kasaï Occidental technology report, Pg 5.
• 74.
• CHWs were estimated to be able to distribute nets to 50 households per day based on the rate of distribution in Nyanga, which impacted the number of phones purchased for the subsequent distribution.
• "The number of phones was determined by the number of CHWs needed to perform the distribution. It was estimated that CHWs could register 50 households per day. This estimation was taken from the average number of households RECOs registered in one day during the pilot distribution in Nyanga." Kasaï Occidental distribution report, Pgs 6-7.
• Note that this seems slightly at odds with another section of the distribution report: "The number of phones required and therefore bought was calculated according to the number of CHW teams needed to perform the distribution and hang‐up in two days per health zone. We estimated that each CHW team would be able to cover 40 households per day. Therefore the total number of households was divided by 80 to determine the number of phones required. The calculation did not specifically include a reserve quantity in case of failure/out of service phones." Kasaï Occidental distribution report, Pg 10.
• Additional batteries to charge phones were bought after the Field Supervisors in Nyanga found themselves travelling large distances at night to recharge phones for the next day of the distribution.
• "In the pilot distribution in Nyanga Health Zone, supervisors would swap unused phones during the data collection when a smartphone ran out of charge. Furthermore, once the net distribution team was in the field, the ability to recharge the phones (finding a generator, solar installation, logistics) was reduced drastically. Many supervisors would travel back (for some, a 2 hour roundtrip) each night to recharge the phones because power sources in the villages were non‐existent. Others were able to rent power generators locally. Thus, for this distribution, additional batteries were bought." Kasaï Occidental distribution report, Pg 11.
• Wifi routers were purchased to upload data more quickly.
• "Lessons learned during the pilot distribution stage in Nyanga influenced us to change a few things for these distributions. Specifically:

1. We purchased WiFi routers which made for quicker uploading of data." Kasaï Occidental distribution report, Pg 20.

• The registration survey on the smartphones was tweaked to make data entry easier.
• "The modification of the data entry forms made after the pilot distribution in Nyanga HZ led to cleaner data (more accurate, fewer errors). The merger of the two questionnaires (= data forms), namely for registration and for distribution reduced the time of the interview at each household and removed duplicate questions." Kasaï Occidental distribution report, Pg 27.
• IMA changed its procedures such that registration and distribution activities would occur simultaneously (during the pilot, registration happened on the first day and distribution happened the next day). Generally, IMA observed that the distribution went more quickly when registration and distribution occurred simultaneously.
• "The new method of carrying out the distribution, i.e. performing the household data entry and the handing out and hanging of the nets at the same time worked very well and saved time. In the pilot distribution in Nyanga the registration took place on the first day and the distribution of the nets required took place on the second day." Kasaï Occidental distribution report, Pgs 26-29.
• "3. Registering the household (collecting data) and distributing the nets at the same time allowed us to complete the distribution at a much faster pace." Kasaï Occidental distribution report, Pg 20.
• 75.
• 76.
• "There weren’t enough trucks available which made movement of nets more difficult than desirable. Lesson learned More attention needs to be given to this in initial logistics planning." Kasaï Occidental distribution report, Pg 27.
• "The estimates of distances between the central location, where staff would meet at the start of each day and the health areas to which they would travel, were not accurate leading to delays Lesson learned Double‐checking of estimated distances to ensure they are sufficiently accurate."Kasaï Occidental distribution report, Pgs 33, 46, and 54.
• "Vehicle breakdowns. This affected the movement of field supervisors and delayed some supervision activities. Private vehicles had to be rented. Lesson learned Consideration of the vehicles and their reliability in the planning process." Kasaï Occidental distribution report, Pgs 33, 47, 50, and 55.
This made travel more difficult and caused delays in the distribution. Lesson learned Consideration of the condition of the roads in the planning process." Kasaï Occidental distribution report, Pgs 37, 40, 43, and 50.
• "The permanent break down of some vehicles and motorbikes slowed down supervisions. Lesson learned Consideration of the vehicles and their reliability in the planning process.
• 77.
• "The level of instruction given to the CHWs was so low that additional training sessions had to be organised, covering familiarity with the equipment and its practical use to ensure the CHWs were fully familiar with the smartphone and the ODK Collect registration programme and what information had to be collected. The additional training included practical use of the smartphone. Lesson learned Greater attention needs to be given to the initial training." Kasaï Occidental distribution report, Pgs 33, and 46.
• "Some CHWs complained the pay was not enough for the work they were asked to do and this seemed to slow down the work done. Lesson learned Establish clearly the daily pay rate and ask, as CHWs are selected, they accept that daily rate." Kasaï Occidental distribution report, Pgs 33, 47, and 55.
• "Certain CHWs, even after training, were not up to the level required to carry out the job well. Lesson learned Ensure better initial screening and selection of the CHWs before training starts." Kasaï Occidental distribution report, Pg 37.
• "There was too much influence over selection by some community leaders and those selected were not good enough to do the job. Lesson learned Greater attention needs to be given to the initial selection to ensure those selected as capable for the training and the job." Kasaï Occidental distribution report, Pgs 46, 50, and 54-55.
• "Some CHGWs were found to be inebriated during working hours. This was found to be linked to a number of nurses and health staff had recruited family members and friends rather than individuals qualified to be CHWs. The individuals were dismissed, unpaid. Lesson learned Monitoring CHWs is important." Kasaï Occidental distribution report, Pg 29.
• "A Field Supervisor was found to be sitting under a tree and not working rather than supervising CHWs. He was dismissed and sent back to Tshikapa. Lesson learned Monitoring Field Supervisors’ performance is important." Kasaï Occidental distribution report, Pgs 29.
• 78.
• "IMA obtained population data for the year 2012 from the Ministry of Health and applied the MOH’s estimate for population growth of 3% per annum." Kasaï Occidental distribution report, Pg 6.
• "There were many occasions where the number of nets needed for a community was overestimated. This was discovered in the registration and hang up because of diligent work done. Lesson learned Better population estimates are needed and 1.8 people per nets should be the ratio always used." Kasaï Occidental distribution report, Pg 46.
• "In a significant majority of Health Areas more nets were pre‐positioned than were needed. The estimates of population were not accurate enough. Lesson Learned There should be a critical assessment of the population numbers used to determine net numbers." Kasaï Occidental distribution report, Pg 20.
• "The population was in fact half of that estimated. [Note: this refers to the population in a particular health zone] Lesson learned Greater attention needs to be given to the accuracy of population information and assumptions for developing estimates which are in turn used to order nets and drive logistics costs. A separate pre‐distribution phase should be considered in which detailed numbers (e.g. population, net need) are established. This may be a better solution to allow an accurate number of nets to be moved and distributed." Kasaï Occidental distribution report, Pg 27.
• Although often too many nets were transported to an area, sometimes too few nets were sent too. "Nets were deployed notably late in many instances. Only 20% of Health Areas had sufficient nets delivered in a timely fashion. Lesson learned Greater attention is needed to proper organisation and logistics management." Kasaï Occidental distribution report, Pg 33.
• "Under‐estimation and over‐estimation of the number of households in a number of areas This led to too few or too many nets pre‐positioned and led to a need to a re‐movement of nets." Kasaï Occidental distribution report, Pg 40.
• We aren’t sure if there were any leftover nets from the 86,500 nets that IMA contributed. It could be that the 45,000 figure includes leftover nets from both IMA and AMF’s contributions.
• It is our impression that AMF and IMA drastically overestimated the population. We think this because IMA and AMF were under the impression that approximately 28% of sleeping spaces would be covered with a useable LLIN (one that did not need to be replaced). However, it turned out that only 2.3% of the population was covered with a useable LLIN. Thus, AMF and IMA had to cover a much greater proportion of the population than they were expecting (an additional 25% or so), but still had a large number leftover nets. This implies that the population was significantly smaller than expected.
• "7. The suggestion from IMA pre-distribution was that 28% of sleeping spaces were already covered with nets. That 28% figure was used in estimating the number of nets needing to be purchased. The numbers here suggest that figure turned out to be 2.3%. Is our understanding correct?" AMF DRC West Kasai distribution summary numbers, cell B62.
• 79.

"i) Some householders tried to invent sleeping spaces
Some householders tried to increase unfairly the number of nets they would receive. Lesson learned The registration process finds out these situations.
i) Some CHWs tried to divert ties, nails and hammers
This would be theft for personal benefit and the loss of hammers would be a cost and cause inconvenience to the distribution. Lesson learned Counting equipment and ensuring return, and not assuming return, is important."
Kasaï Occidental distribution report, Pg 43.

• 80.
• "All distribution activities had to be stopped due to an Ebola outbreak. A period of one month after the last Ebola case was identified had to be identified before the distribution could restart and be completed. The pause was for 8 weeks. Lesson learned Relevance of thinking during the planning phase (in the province more widely) of how plans and schedules can be rearranged if a major delay is necessary in one HZ." Kasaï Occidental distribution report, Pg 43.
• "In certain cases the CHWs complained there wasn’t an appropriate place (an easily visible place on the front of the household) to mark the household identifier. Lesson learned CHWs were writing the household identifier in too large a handwriting and so were running out of space to write it on the door. They were told to use smaller handwriting." Kasaï Occidental distribution report, Pg 29.
• "Lack of consistent electricity. This caused delays. Lesson learned Allowances needed in the timetable for potential delays that are caused by elements beyond the control of the team. Develop ‘What if?’ scenarios so if there are problems or delays there is a plan of how to respond and/or decisions can be taken more quickly." Kasaï Occidental distribution report, Pg 37.
• "The coordination between the two was not as good as it should have been at times leading to gaps in information and understanding of work carried out. Lesson learned Better coordination to ensure the central location is fully aware, and in a timely fashion, of the data from the field." Kasaï Occidental distribution report, Pg 51.
• 81.
• "Initially 430 smartphones were bought but 97 of these proved to be faulty and unusable. In retrospect, the smartphones were not bought form a reliable source. Lesson learned A strong recommendation, to avoid duplicate effort, delays and unnecessary cost, is to ensure the phones are bought from a highly reliable source with each one tested out of country prior to shipment to country. Identify vendors that can deal with bulk purchases." Kasaï Occidental distribution report, Pg 20.
• "The phones were purchased from three sources. Even with the popularity and wide availability of the Samsung brand, it was difficult to source all the phone models from one source. 70 phones proved to be faulty and unusable. In retrospect, this problem occurred because one of the suppliers was an unproven vendor, and that vendor delivered a faulty batch of phones. The distribution campaign continued with 360 smartphones and it was still possible to finish the distribution in a timely fashion." Kasaï Occidental distribution report, Pg 10.
• Note that in the previous two quotes, there is disagreement about how many phones were found to be "faulty and unusable." In the first quote, it is claimed that 97 phones were found to be faulty, whereas in the second quote, it is claimed that only 70 phones were found to be faulty. We are not sure of the reason for this discrepancy.
• "i) Problems with smartphones. The two problems experienced with some phones were: i) they were no longer in good working order and data could not be entered into the phones or the ODK programme would not run; ii) they would not retain energy for beyond the first hour of two of the day’s work. This led to the number of teams working on any one day being reduced and increased the number of days over which the work had to take place. Lesson learned Greater attention needs to be paid to how the smartphones are treated and consideration could be given to the robustness of the phones. Consideration should be given in selecting appropriate phones how they perform in high ambient temperature environments." Kasaï Occidental distribution report, Pgs 33-34.
• "A mistake was made in using an over‐powerful generator to recharge the phone that led to some burning out due to the excessive voltage used. Lesson learned Attention needs to be paid to the voltage used in charging phones/batteries. This is important information that has been noted for inclusion in all future briefing sessions." Kasaï Occidental distribution report, Pg 34.
• "Not all of the phones worked all of the time causing delays and inefficiencies and some data loss. Lesson learned It is very important the smartphones are reliable and there are backup phones in case they are needed." Kasaï Occidental distribution report, Pgs 40, 43, 47, 51, and 55.
• "There were problems charging the phones and spare batteries which led to insufficient battery life during the day and some phones not working. Sometimes this was caused by generators, used for recharging, breaking down. Lesson learned Greater attention needs to be given to ensuring phones work, batteries can be charged so running out of battery power does not occur. Consideration should be given to back‐up phones that can be used when particular phones do not work so the important work to be done and data gathered is not disrupted." Kasaï Occidental distribution report, Pgs 40-41, 44, and 56
• "iv) Inappropriate equipment used in charging phones
Rather than this being unsuitable ‘phone chargers’, this problem related to generators being used to generate the electricity from which the phones were charged. Unfortunately ‘constant voltage’ transformers were not used and as a result of the fluctuating/surging charging voltage from the generators, some of the phones (and several laptops) ‘burn out’. Lesson learned Care needs to be taken when charging devices where there are power outages and generators may be need. Constant voltage transformers should be given serious consideration." Kasaï Occidental distribution report, Pg 56.
• 82.

"91% of the nets distributed had data recorded electronically (565,306 nets to 231,925 households, out of total of 624,532 distributed)." Kasaï Occidental technology report, Pg 14.

• 83.

"9% of the nets distributed did not have data recorded electronically due to:

• Failure of individual smartphones with no back up available for data collection
• Issues with data being corrupted and not retrievable from the smartphones
• Issues with the data transfer process leading to data loss"
• 84.

"We are broadly confident the 9% of nets for which we do not have data in electronic form were distributed as intended for the following reasons:

• Signed net movement forms are reported by IMA to show nets moved to the final distribution points as expected
• Explanations from IMA as to why data was lost or not collected electronically are credible
• Paper based records are reported by IMA to corroborate net distribution totals."
• 85.

AMF told us: "Our understanding is 1) some data was only collected in electronic form so its loss or corruption meant there was no back up. Other data was duplicated on paper but the task of going back to individual health centres to collect that data was onerous and IMA decided not to do so as it was considered a significant and onerous task. We discussed this with IMA and in the circumstances decided not to press this further. We did however discuss how systems would change to ensure this could not happen in North Ubnagi and have developed improved protocols and requirements." Rob Mather, email to GiveWell, September 6, 2015.

• 86.
• "96% of the household data records were without error." Kasaï Occidental technology report, Pg 15.
• "4% of the households had one error or more in the data recorded due to:
• Data inconsistencies within a household record (these can be designed out with improvements to the logic mechanism i.e. acceptable answers during data entry)
• Data anomalies within a household record suggesting data entry error (these can be reduced with improved training and/or a verification step during data analysis)
• No data analysis between data collection and nets being distributed (this can be reduced if the distribution follows a separate registration phase)
• "An inconsistency error is where two pieces of data are not consistent. For example, 4 sleeping spaces in the household but 5 nets given out; or 3 people in the household but 4 under5s recorded. Usually these are data entry errors. The data entry screens are set up with error messages so as not to allow most inconsistency errors but not all were designed out and some were possible. An anomaly error is where information looks wrong. For example, more than 20 under 5s in a household or all in a household are under 5s. The data are typically outliers against the bulk of the data. These are also likely to be data entry errors." Kasaï Occidental technology report, Pg 15.
• 87.
• 91% * 96% = 87%
• Note that this not not guarantee that the data is reliable; AMF only checked to the data for inconsistencies and anomalies. This does not rule out the possibility of falsified data.
• 88.
• 89.
• AMF told us that it was waiting to receive data from the Kasaï Occidental distribution before starting the Nord Ubangi distribution, so that AMF could feel sufficiently comfortable moving forward. AMF also wanted to apply the lessons learned from the Kasaï Occidental distribution to the Nord Ubangi distribution. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015
• "We intend to use the same data collection process in the next net distribution, adjusted to apply lessons learned." Pg 3, Kasaï Occidental technology report
• 90.
• "To reach out to every one of the 782 villages individually would have been very timely hence the villages were clustered to reduce the contact points. The clusters were formed with the assistance and identification of the HSAs. The clustering was based on the closeness of the villages and one village was identified to be a meeting point. There were 282 clusters that were formed. The cluster has a village composition ranging from one to five depending on the size and geographic proximity of the villages." Ntcheu 2011-12 distribution report, Pg 5. Also see the diagram on Pg 9.
• "At a distribution point beneficiaries from villages in the cluster gathered together with their local development and elected leaders. The distribution registers which were in duplicates were called at one village after another and the called beneficiaries lined up to receive their nets. Upon completing distribution one copy was collected for office records and the other register was left with the village leaders." Ntcheu 2011-12 distribution report, Pg 8.
• 91.

See the methodology section above.

• 92.
• "Verification of the long lasting insecticide treated nets registers was done at village level where everyone was present. The community was eagerly listening as team members were calling out the name of the household, the number of people in the household, number of LLIN in that household, number of sleeping spaces in the household in question and finally the number of nets the household was going to receive. Upon completing name calling out, those villagers who did not hear their names being called were registered so that they could be included in the data base. One HSA was assigned to cross check the call out from the printout roll with the raw data and, where the printout differed with the raw data this was noted and later sent back to the data center for data updating. The community also played an important role during the verification in that whenever a name was called out that did not exist in their village, they advised the team members to delete such a beneficiary from the list, which was done after consultation with the village head. The other strength of the roll calling during the verification was that whenever the data that the beneficiary gave on registration was different with what the communities’ perception of the household, the community could raise the issue and the beneficiary could be taken to task to justification. Whenever the community disagreed with the justification, they were advising and recommending the verification team members on the total number of nets the household should get. This improved transparency of the net distribution process." Balaka 2013 distribution report, Pg 6.
• "Based on GiveWell's suggestion, Concern Universal also decided to send its staff to random households prior to the Ntcheu 2012 distribution to spot-check the accuracy of its data. After the first two weeks of spot checks, the distribution partner reported that it had not found any discrepancies between the number of nets community members said they needed during the verification meeting, and the number of nets found to be needed during the subsequent household visit. AMF has told us that spot checks are now carried out with all distributions. We have not seen results from these spot checks." GiveWell AMF review 2014.
• 93.
• "At the end of each day, the CHWs returned the smartphones to the Field Supervisor and the data was imported into a computer using a WiFi router connection. The Field Supervisor checked the data to make sure the household registrations had been carried out correctly."Kasaï Occidental distribution report, Pg 25
• "After the distribution had been completed in a Health Area and all data uploaded from the smartphones, the data team compared the electronic data with the paper data collected by each distribution site in each health area." Kasaï Occidental distribution report, Pg 18.
• 94.

Balaka 2013 distribution report, Pgs 3-4. Note that both "useable LLIN" and "sleeping space" are fairly well defined on Pg 4.

• 95.
• We are not entirely clear on how IMA determined the appropriate number of nets to give to each household, although we believe that IMA relied on the number of sleeping spaces in a household. The following quote is in the distribution report: "When assessing the number of sleeping spaces it is important to avoid confusion between the number of separate areas in which individuals might sleep and, given nets distributed are ‘double’ nets that will accommodate two people, the number of sleeping spaces that is equal to the number of nets that are needed. For example, in a household with mum plus dad, four children aged eight, four, six and two and a grandfather there may be six separate sleeping areas: mum plus dad, one each for the four children and one for the grandfather. However, for the purposes of nets, there would be four sleeping spaces: one for mum and dad, one for the two older children, one for the two younger children and one for the grandfather." Kasaï Occidental distribution report, Pg 16. To continue with the example quoted, we believe that this could mean that if the six separate sleeping areas in the household were spread out in such a way that 4 nets could not cover them, the household would still only receive 4 nets, and would have to move its sleeping areas closer together if it wanted to cover everyone. We are not sure how easy it is for a household to move sleeping areas, or how it was decided where to hang nets in cases like this.
• On the other hand, when we look at the sample household data, it does not seem to support this interpretation. For example, row 9 shows 5 people living in a household, and there are 5 sleeping spaces in the household. Because the household already had 2 useable LLINs, it received 3 from IMA. This seems to imply that even if there was only one person using a sleeping space, IMA would still give a net to cover the whole sleeping space, and not require that the sleeping space be moved close to other sleeping spaces so that two people could be covered with one net. Sample Data-Anonymized sheet, Kasaï Occidental 2014 summary and sample data.
• 96.

See here.

• 97.
• 98.
• 99.
• See our May 2015 update of AMF.
• "2018: In addition, the NMCP through the LLINs task force is grateful for the offer from the CU/AMF partnership to continue to assist in four districts beyond the 2015 distributions, for the 2018 universal coverage distributions.
In view of the above the NMCP and LLINs task force is requesting AMF/CU to undertake LLINs mass distribution campaign in four district in 2018, which will again be in line with the NMCP’s schedule and policy." Malawi NMCP invitation for 2018 distribution
• 100.

"AMF and Concern Universal, Malawi (CU) have accepted the Malawi National Malaria Control Programme’s (NMCP) invitation to assume responsibility for the district-wide net distribution re-coverage campaigns in Ntcheu, Balaka, Dedza and Dowa districts in 2018...This timing is consistent with the three year re-coverage cycle needed to keep affected populations protected from malaria-carrying mosquitoes when they sleep at night." AMF blog post, July 3, 2015

• 101.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 102.

"AMF is attempting to finalize two new distribution agreements: one for approximately 2.5 million nets in Country X (AMF cannot publicly share the names of countries it is negotiating with), and one for approximately 5.5 million nets over the next three years in Country Y. We do not have a strong sense of how likely it is that AMF will be able to finalize these distribution agreements." GiveWell AMF update May 2015.

• 103.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 104.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 105.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 106.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 107.

AMF investigates distribution partner candidates by (1) having discussions with the candidate to see how aligned the organization is with AMF’s values and (2) discussing the candidate with other independent organizations that work in the country. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 108.
• 109.

There are a few reasons for this:

• AMF believes that having a co-funder for non-net costs is one way to mitigate its risks. AMF’s co-funding partners investigate the distributions they are agreeing to using their own methods, and can learn of (or sometimes already know of) information that impacts the distribution. For example, if AMF cannot find a co-funding partner for a specific country, this may be a warning sign about that particular country. AMF can talk to potential co-funding partners to figure out why they won’t work in a particular country.
• AMF believes that not paying for non-net costs allows its funds to go further.
• AMF is not set up in countries where distributions occur. Thus, AMF is not in a position where it can easily deliver on the reporting it requires. This makes AMF hesitant to fund an organization for all net and non-net costs; if the organization ultimately fails to deliver on the reporting that AMF requested, AMF will have a hard time resolving the issue. With a co-funding partner, especially a partner that works in-country, AMF has more assurance that the reporting it requires will be delivered.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 110.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 111.
• In early June 2015, AMF told us that it had about $22 million in funding, of which$10.2 million was already committed to future distributions ($4.4 for distributions in the near future and$5.8 for the 2018 Malawi distributions). Rob Mather, email to GiveWell, June 1, 2015 and AMF funds history and status
• In July 2015, AMF’s website showed approximately $21 million in hand (see FY 2016), and approximately$4.1 million had been used for purchases or forward commitments. We believe that AMF did not include its future commitments to the 2018 Malawi distributions in its online financials because the distributions are so far in the future. AMF financial information page
• $4.1 million +$5.8 million = $9.9 million • Note that in February, AMF told us that it had approximately$18 million in hand. So, AMF was able to raise approximately $4 million in several months’ time ($22 million - $18 million =$4 million). We know that GiveWell transferred approximately $2 million to AMF in early March (the transferred funds were donations that GiveWell had received for AMF or for our top charities). AMF told us that another approximately$700,000 came from GiveWell-influenced donors. This implies that AMF received approximately $1.3 million from other donors in March through May. Rob Mather, email to GiveWell, June 19, 2015 • 112. • 113. For example, AMF is thinking about using the funds for the 2018 distributions in Malawi sooner than 2018. AMF has told us that if it spent the funds from the 2018 Malawi distributions it would need to re-raise the funds by early 2017. Rob Mather, conversation with GiveWell, February 24, 2015 • 114.$11.1 million + $5.8 million =$16.9 million

• 115.

AMF future distributions net need

• 116.
• 117.
• We discuss case rate data from Ntcheu in our February 2013 update on AMF.
• "[AMF:] Please confirm you are able to provide monthly malaria case rate data going back at least 12 months for each health centre/clinic in the distribution area and will continue to provide monthly data for a period of four years post‐distribution. This ensures we understand pre‐distribution malaria levels and can monitor them post‐distribution.
[Concern Universal:] Malaria case rate information for each of the 16 Health Centres is available from the Balaka District Health Office using the Malawi Government’s national Health Management Information System (HMIS). This information will be sent to AMF separately." Balaka 2013 distribution proposal, Pg 2.
• Dedza 2014 distribution proposal, Pg 2, contains the same language as above
• 118.

Rob Mather, conversation with GiveWell, February 24, 2015

• 119.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 120.
• 121.

"AMF primarily distributes nets in 4 of Malawi’s 28 districts. However, AMF told us that for the second and third quarter of 2015, the National Malaria Control Program (NMCP) of Malawi is planning to implement some of AMF’s distribution practices in districts for which AMF does not fund distributions." GiveWell AMF update May 2015

• 122.

Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015

• 123.
• 124.
• 125.

If a data collector visits a house and asks how many people live there and is told "six people", but there are currently only three at the house, the data collector has no way of knowing whether or not this is true. On the other hand, it is our understanding that it is much more difficult to invent sleeping spaces, which are always in the house. This is why AMF considers it best practice to give out nets according to sleeping spaces, rather than people.
Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015.

• 126.

AMF notes that the cost of data collection (several hundred thousand) is very small compared to the cost of the overall distributions (tens of millions). AMF also notes that having the data in electronic form allows one to much more easily check that the summary headline numbers for any given distribution are accurate and not inflated.
Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015.

• 127.
• 128.
• 129.

AMF mentioned that, anecdotally, it has heard that its net hang-up rates are 20-40 percentage points higher than other organizations’ net hang-up rates. These numbers are based on small reviews that other organizations have done and the PDCUs that AMF requires for its own distributions.
Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015.

• 130.

The task force is attended by representatives from the NMCP and other large and local organizations working on net distributions in Malawi. Rob Mather and Peter Sherratt, conversation with GiveWell, June 2, 2015.

• 131.