Malaria Consortium — Monitoring and Evaluation of Net Distribution in Anambra, Nigeria (November 2021)

Note: This page summarizes the rationale behind a GiveWell grant recommendation to Malaria Consortium. Malaria Consortium staff reviewed this page prior to publication.

Summary

In November 2021, Malaria Consortium received a grant of $1,199,808 to study the effects of its campaign to distribute long-lasting insecticide-treated nets (LLINs) in Anambra state, Nigeria, in early 2022. Open Philanthropy made the grant based on GiveWell's recommendation. A previous grant is supporting the costs of the universal LLIN distribution campaign.

Universal LLIN distribution is the intervention that currently receives a plurality of the funding that GiveWell recommends1 —we expect total GiveWell-directed funding for LLIN campaigns to continue to be in the range of $50-100 million per year—so it is a high priority to learn more about the cost-effectiveness of these campaigns. This grant aims to test several assumptions we make about the cost-effectiveness of these campaigns. The information we've gotten on the impact of previous grants we've recommended for LLIN campaigns has focused on what proportion of intended recipients received and reported using LLINs, and the physical durability of LLINs over time. This grant will fund Malaria Consortium to track those metrics and, in addition, will fund Malaria Consortium to measure insecticide resistance levels in local mosquitoes when they are exposed to the chemicals in LLINs, track the durability of the insecticide over time, and generate a full cost accounting for the campaign.

We have several goals in recommending this grant:

  1. Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns
  2. Help us evaluate the cost-effectiveness of the Anambra LLIN campaign in retrospect
  3. Learn about what metrics we'd like to track for future campaigns we support and how best to measure them

Published: January 2022

Table of Contents

The organization

Malaria Consortium's seasonal malaria chemoprevention program is one of GiveWell's top charities. In March 2021, we recommended funding for the first time to Malaria Consortium for LLIN campaigns. This grant will fund LLIN campaigns in two states in Nigeria: Ondo and Anambra. The March grant followed a 2020 grant to Malaria Consortium for scoping LLIN funding gaps.

In July 2021, we recommended a grant to Malaria Consortium to collect data in Ondo, similar to, but more extensive than, the data that will be collected in Anambra.

Planned activities and budget

The grant is for $1.2 million to evaluate the impact of a $14.9 million LLIN campaign in Anambra state, Nigeria, that we recommended funding to support in March 2021.2 The campaign is scheduled to start in March or April 2022.3

Note that in the budget that Malaria Consortium provided, 42% of the costs are assigned to specific study components,4 while the remainder is primarily for cross-component costs such as staff time and overheads.5 In the budget breakdown below, we've assigned cross-component costs to study components proportionately. In reality, staff time, which is a large portion of the budget, may be split in different proportions than the component-specific budget is.

This grant will support:6

  • Tests of local mosquitoes' susceptibility to insecticide used in LLINs ($0.40 million—34% of the budget): Mosquitoes will be collected at two times, once before the LLIN campaign and once two years after the campaign, to test their degree of resistance to the chemicals used to treat LLINs. Lab tests will be conducted to measure what proportion of mosquitoes are killed by varying intensities of pyrethroid insecticides used on standard LLINs (both with and without the addition of the piperonyl butoxide [PBO] synergist, a chemical that is added to some LLINs, including the ones that will be distributed in Anambra), and what proportion have genetic markers known to confer some resistance to the insecticide.7
  • Net durability monitoring ($0.40 million—33% of the budget): A portion of nets will be labeled prior to the campaign and tracked over time. Once a year, Malaria Consortium will visit households to measure what proportion of the cohort nets are still in place and the extent of physical damage (number and size of holes). A portion of cohort nets will also be tested in a laboratory to measure the efficacy of the insecticide.8
  • Surveys to track the number of nets in each household and reported use rates ($0.34 million—28% of the budget): A random sample of households9 will be surveyed to observe the number of nets and to ask what proportion of people who slept in the house the previous night slept inside a net.10 Surveys will be conducted one month prior to the campaign and three months after the campaign. Additional information on net ownership and use will be collected during the durability monitoring surveys at one, two, and three years after the campaign.11
  • Full cost accounting ($0.03 million—3% of the budget): The full costs of the campaign, not only costs borne by Malaria Consortium, will be compiled. One or more health economists will be recruited as consultants to review existing financial records and visit Anambra to collect additional information.12
  • Meteorological tracking ($0.03 million—3% of the budget): Malaria Consortium will compile and analyze data on rainfall, temperature, and humidity from weather stations in Anambra to use in interpreting data on LLIN use rates and durability of nets.13

For Ondo, Malaria Consortium will include two additional study components (details here) which are not included for Anambra. We chose not to include these components for Anambra because:

  • Malaria incidence trend analysis: The main purpose of the Ondo study on malaria incidence is to assess whether this method yields reliable data and how much the results affect our view on the impact of the LLIN campaign. We don't expect an additional test of these methods in Anambra to yield a different conclusion. Ondo was selected over Anambra for the test because higher malaria rates in Ondo means there will be more statistical power for a given number of clinics.
  • Tracking exposure to infected mosquitoes: Given uncertainty about how we will use the results generated from Ondo, we prefer to test it there before running similar tests in other locations.

    How we plan to use the results

    We have several goals in recommending this grant:

    1. Help us evaluate the cost-effectiveness of the Anambra LLIN campaign in retrospect
    2. Inform parameters in our cost-effectiveness analyses for future campaigns we're considering supporting
    3. Learn about what metrics we might want to track for future campaigns we support and how best to measure them

    ​​

    Study component Current situation How we think this grant will help
    Tests of local mosquitoes' susceptibility to insecticide used in LLINs There appears to be a relatively large number of data points on pyrethroid resistance from Nigeria,14 with a very small number measuring the impact of PBO on susceptibility.15 We have not seen a study of insecticide susceptibility from Anambra specifically.16 We have used existing data from Nigeria to estimate an insecticide resistance adjustment in our cost-effectiveness analysis for LLIN campaigns in Nigeria. Help us evaluate the cost-effectiveness of the Anambra 2022 LLIN campaign in retrospect. The pre-campaign tests will yield a data point on both pyrethroid and pyrethroid + PBO resistance levels for Nigeria, and the first data point we're aware of for Anambra specifically. We can compare this to what we assumed in our cost-effectiveness analysis.

    Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns. We and other funders may use the second data point (from two years after the campaign) to help with selecting the most cost-effective type of LLIN for Anambra and nearby states in future campaigns.

    We also plan to use both data points in the future in the insecticide resistance adjustment in our cost-effectiveness analysis for LLIN campaigns in Nigeria.

    Net durability monitoring We have relied on a large number of data points in estimating the effective lifespan of a standard LLIN (without PBO). There appears to be significant variation found across studies and locations. We are unsure about the durability of the PBO component when it is added to nets, and our impression is that limited evidence exists on this subject.17 A similar test will be done in Ondo. Our understanding is that there are not yet any durability studies specifically of the DuraNet Plus net, the type of net that Malaria Consortium will distribute in Ondo and Anambra, which contains PBO.18 Help us evaluate the cost-effectiveness of the Anambra 2022 LLIN campaign in retrospect. We plan to use this data to update the parameter for effective years of coverage19 in our cost-effectiveness analyses.

    Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns, particularly for campaigns that will use the DuraNet Plus net and somewhat for other PBO net campaigns.

    Surveys to track the number of nets in each household and reported use rates Based on past experience with LLIN campaigns and with Malaria Consortium's work on SMC in Nigeria, we expect that rates of pre-campaign ownership of functional nets will be very low (<10%, because it has been about seven years since the last campaign),20 and that the campaign will deliver nearly all of the purchased LLINs to households (>90%),21 resulting in high coverage. We expect moderately high reported use rates.22

    We could be wrong about low pre-campaign ownership if there are other sources of LLINs, such as LLINs given out at clinic visits or through schools.

    Help us evaluate the cost-effectiveness of the Anambra 2022 LLIN campaign in retrospect. We will update the parameter in our model for "adjustment for net coverage years lost due to residual nets from previous distributions." We may also update the "net use adjustment."
    Full cost accounting Malaria Consortium has shared a detailed budget and will be able to report back on its own spending. Without further study, it would not be in a position to report on spending by other contributors, such as the state and central governments and their employees and volunteers. We have also seen information on the costs of LLIN campaigns from AMF, but AMF has limited information on costs of LLIN campaigns other than the costs of purchasing and shipping LLINs, which are the components it pays for directly.23 Help us evaluate the cost-effectiveness of the Anambra 2022 LLIN campaign in retrospect.

    Improve the accuracy of parameters in our cost-effectiveness analyses for future LLIN campaigns. This work will provide a data point for third-party costs that we will likely use to estimate such costs for future campaigns in other locations. It will also provide a high-quality estimate of distribution costs against which we can triangulate budgets for future campaigns.

    Meteorological data monitoring We haven't used this type of data before. Malaria Consortium plans to use this data "to investigate any variations between the two states [Ondo and Anambra] or among different ecological areas in terms of physical and chemical durability of nets that might be associated with climatic factors."24

    Risks and reservations

    • We haven't created a cost-effectiveness model of the value of the information from this grant. We are relying on an intuitive sense that the learning in this case is worth the cost of the research.
    • Malaria Consortium has experienced some delays in executing on the Ondo monitoring and evaluation plan. There is a risk that necessary components won't be in place to do all the baseline activities before the campaign starts. However, the work for Ondo has laid the foundations for the work for Anambra (e.g. making key hires, establishing supplier contacts) and, as of this writing, Malaria Consortium does not anticipate major challenges in completing the baseline activities.
    • For the Ondo monitoring and evaluation grant, we noted that there was additional research that Malaria Consortium suggested25 and that we ultimately decided not to include because the timeline for deciding on the scope of research was fairly short due to the timing of the campaign. For Anambra, we had more time between starting to discuss the monitoring and evaluation plans and needing to finalize a decision on the grant, so one might think we should increase the scope of research for Anambra. Furthermore, Anambra is an unusual case in that there has been a seven-year gap between campaigns,26 so baseline measures pre-campaign are closer to the counterfactual than for typical three-year intervals between campaigns (when there are more LLINs still in place from the last campaign and less time for malaria to rebound to the no-LLINs level). We decided not to pursue additional research as part of this grant because ultimately there was still not enough time to do a useful study. The additional components we had discussed with Malaria Consortium were about answering the question "how do LLINs affect the number of malaria cases and deaths?" Through working on the plans for monitoring and evaluation in Ondo and Anambra it has become clearer to us that a study that convincingly answers that question, without random assignment of communities to receive or not receive LLINs, needs to carefully track the level of LLIN coverage, since variation in LLIN coverage would be the independent variable. A convincing study would also need to be carefully thought through to consider statistical power and regression specifications. We don't believe that the plans that we were discussing with Malaria Consortium ahead of the Ondo and Anambra campaigns would have overcome these measurement challenges. We had a few more months to make plans for Anambra than for Ondo, but we felt this amount of time was still insufficient to both put in place the necessary plans and partnerships for the research and still have time to collect at least a few months of data on LLIN coverage and malaria cases before the campaign.

      Internal forecasts

      Confidence Prediction By time
      35% The full cost per LLIN delivered in the Anambra 2022 campaign is measured as more than 10% different than in the full cost in the Ondo 2021-2022 campaign (excluding the monitoring and evaluation costs). End of 2022
      25% Conditional on us completing the analysis, our retrospective cost-effectiveness analysis for the Anambra 2022 campaign will change by more than 20% relative to when we recommended the grant to fund the campaign. End of 2025
      40% At baseline, population coverage in Anambra with LLINs that have at least one year of expected remaining effective use will be more than 10%. End of 2022
      10% At baseline, population coverage in Anambra with LLINs that have at least one year of expected remaining effective use will be more than 25%. End of 2022

      Our process

      In order to develop this study design for Anambra, we relied on the work we had already done designing our study for Ondo with Malaria Consortium, and left out a few elements. The Ondo study design was developed iteratively in discussions between us and Malaria Consortium. The nearly final version was reviewed by a Senior Researcher at GiveWell who was not otherwise involved in the discussions. We have not yet sought out external feedback, beyond discussions with Malaria Consortium, on either of these grants.

      Sources

      Document Source
      Carrasco et al. 2019 Source (archive)
      GiveWell, "Against Malaria Foundation: Supplementary Information" Source
      GiveWell, "Against Malaria Foundation" Source
      GiveWell, "Malaria Consortium — Support for LLIN Distribution Campaigns in Ondo and Anambra States, Nigeria (March 2021)," August 2021 Source
      GiveWell, "Malaria Consortium—Seasonal Malaria Chemoprevention" Source
      GiveWell, Counterfactual scenarios on pre-existing nets, 2021 Source
      GiveWell, Insecticide Resistance (IR) Mapper synergist bioassay data for Nigeria, 2021 Source
      GiveWell, Ondo and Anambra LLIN budget as of March 19, 2021 (redacted) Source
      GiveWell, Ondo and Anambra LLIN distribution CEA, 2021 Source
      Gleave et al. 2021 Source (archive)
      IR Mapper Source (archive)
      Malaria Consortium, Anambra campaign M&E budget, November 2021 (redacted) Source
      Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021 Source
      Malaria Consortium, Ondo state campaign timeline, 2021 Source
      PMI VectorLink Project, "Where we work: Nigeria" Source (archive)
      WHO, "Anaemia" Source (archive)
      WHO, Fact sheet detail, "Malaria," April 1, 2021 Source (archive)
      WHO, Malaria Threats Map, Insecticide resistance status Source (archive)
      • 1This was determined using internal records
      • 2

        See here for the budget for the LLIN campaigns in Ondo and Anambra states.

      • 3

        Unpublished conversation with Malaria Consortium on January 12, 2022.

      • 4We calculated this percentage by adding up the costs designated to specific components (excluding the miscellaneous costs) and dividing by the total cost.
        Household survey: $141, 002
        Durability monitoring: $165, 672
        Entomology: $168,826
        Meteorology: $12,600
        Costing study: $14,112
        Sub-total of component costs: $502,212.07
        Total cost: $1,199,808
        Percentage of costs assigned to specific study components: 42% Malaria Consortium, Anambra campaign M&E budget, November 2021 (redacted)
      • 5

        See Malaria Consortium, Anambra campaign M&E budget, November 2021 (redacted), "Summary" tab.

      • 6See here for the list of study components and their percentage of the overall budget, and here for the cost of each study component including proportional cross-component costs.

      • 7

        "Insecticide susceptibility studies will be conducted in six sites at baseline and two years after the campaign, to understand changes in resistance levels and make recommendations on the types of ITNs for the subsequent replacement campaign. The WHO and/or CDC bottle bioassays, resistance intensity tests, synergist assays, and molecular analysis (target-site resistance markers and other mechanisms as necessary) will be carried out." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, pp. 5-6 Note: this is a description of the monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra. Discussions include:

        • Malaria Consortium, call on August 11, 2021 (unpublished)
        • Malaria Consortium, email on September 7, 2021 (unpublished)
        • Malaria Consortium, email on October 22, 2021 (unpublished)
        • Malaria Consortium, email on November 25, 2021 (unpublished)

      • 8

        "Durability of the campaign nets will be studied over three years using standard procedures. The study includes collection of data on attrition (due to loss of nets), physical durability (measured by number and size of holes), and insecticide effectiveness (measured by cone bioassay using susceptible mosquitoes and chemical analysis).

        "Durability monitoring will include identification of a cohort of campaign nets and following them up longitudinally during annual surveys, and additional bioassays of a sample of campaign nets outside the main cohort in each location. The nets for this specific campaign are labelled at the manufacturing stage for ease of identification, with additional unique labelling of the cohort nets. At the end of the third year, a sample of surviving nets from the main cohort will be used for bioassay and chemical analysis. Standard procedures will be used to study ITN durability, including attrition, calculations of the hole index, bio-efficacy, and chemical contents over time." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 5. Note: these are descriptions from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.

      • 9

        "We will use a stratified multi-stage cluster sampling of households. The first-stage sampling of wards will use PPS [probability proportional to size], followed by the second-stage sampling of households. Households within selected wards will be sampled using simple random sampling. Wards and households will be re-sampled during each survey round.

        "All households in selected wards will be visited and numbered consecutively. The numbers and names of household heads will be recorded. The list will be sent to Abuja and London for sampling to avoid bias. Simple random sampling will be done using computer-generated random numbers. Large wards will be divided into segments before selecting one segment randomly for listing of households to save costs of listing and numbering." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, pp. 3-4. Note: these are descriptions from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.

      • 10

        "Survey staff will record names, sex and age of each household member, whether they stayed in the house the previous night, and whether they slept under a mosquito net. Data on each mosquito net found in sampled households will be gathered, including: type, brand, identity of persons who slept under the net, and where and when the net was obtained." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 4. Note: this is from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.

      • 11
        • Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, Table 1, "Durability monitoring (DM) and household survey rounds," p. 3. Note: this table is from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.
        • "All durability monitoring household interviews in cohort households will be also used to collect data on ITN ownership and use." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 3. Note: this description is from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.

      • 12

        "The costing analysis will involve review of financial cost data obtained from various partners involved in the ITN campaign, and use of multiple sources to estimate economic costs incurred by each partner. Both desk-based studies and field visits will be required. Consultants (health economists) will be recruited to undertake the work, assisted by Malaria Consortium staff, national and state malaria programme staff and field assistants." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 6 Note: this description is from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.

      • 13

        "Monthly rainfall, minimum and maximum temperatures, and relative humidity data will be collected from all functional weather stations in Ondo State, for up to 10 years before the ITN campaign and during the three years period post-campaign. The data will be analyzed in conjunction with selected indicators." Malaria Consortium, Ondo comprehensive M&E concept note, June 30, 2021, p. 6 Note: these descriptions are from Malaria Consortium's monitoring and evaluation plan for Ondo; based on discussions with Malaria Consortium, we expect the same design in Anambra.
        Further information on the use of this data for Anambra is from Malaria Consortium, email to GiveWell, January 14, 2022 (unpublished).

      • 14

        The World Health Organization (WHO)'s Malaria Threats Map identifies 1,024 bioassay tests of pyrethroid resistance status conducted in Nigeria between 2010 and 2019. See more in this spreadsheet.

      • 15

        The Insecticide Resistance (IR) Mapper Anopheles map identifies two studies that conducted pyrethroid-PBO synergist bioassays in Nigeria, Awolola et al. 2005 and Djouaka et al. 2016. (See here and here in the data we retrieved from IR Mapper.) We use results from these studies, reported by IR Mapper, in our insecticide resistance adjustment for Nigeria here.

      • 16

        No measurements for Anambra state show up in any of the large insecticide resistance summary sources we're aware of, including the IR Mapper Anopheles map and the WHO Malaria Threats Map for vector insecticide resistance status. Anambra is not among the states that the President's Malaria Initiative (PMI) has selected for ongoing entomological monitoring (see here).

      • 17

        The recent Cochrane review on pyrethroid-PBO nets states: "Questions remain about the durability of PBO on nets, as the impact of pyrethroid-PBO nets on mosquito mortality was not sustained over 20 washes in experimental hut trials, and epidemiological data on pyrethroid-PBO nets for the full intended three-year life span of the nets is not available." Gleave et al. 2021, "Authors' conclusions"

      • 18

        See the "Durability of protection provided by an LLIN" section in our write-up of the grant for the Ondo and Anambra LLIN campaigns here.

      • 19

        Labeled "Equivalent coverage-years for an LLIN over a 36-month distribution" in GiveWell’s model.

      • 20

        Our cost-effectiveness model uses an estimate of 8% of LLINs distributed during a mass distribution campaign existing and effective after four years. Additional nets may be available if purchased by households or received through other programs. Our model assumes that non-campaign sources of nets are negligible. The survey described above will test that assumption.

      • 21Our cost-effectiveness analysis includes an 8% downside adjustment for nets that are purchased but do not reach households, meaning that 92% of nets are expected to reach households. See here.
      • 22

        Net use in the trials on which we base our cost-effectiveness model found imperfect usage, often in the range of 60-70%. Our model assumes that usage in current contexts is 90% of what it was in the trials. For more discussion, see the latest version of our cost-effectiveness analysis ​​here, "AMF" tab, "Net use adjustment" row.

      • 23

        See here for more detail.

      • 24

        Malaria Consortium, email to GiveWell, November 25, 2021 (unpublished)

      • 25 Malaria Consortium had suggested, in addition to the components we funded for Ondo, (a) tracking prevalence of malaria and anemia in young children pre-campaign and annually post-campaign, and (b) collecting mosquitoes monthly to analyze species, infectivity, age, and whether they had had a blood meal recently.
      • 26

        See here in our write-up of the Ondo and Anambra LLIN campaign grant: "The last LLIN campaign in Ondo specifically was at the end of 2017, and in Anambra in 2014."