You are here

Results for Development — Phase I Grant Evaluation

Published: January 2019

Note: This page summarizes our analysis of a GiveWell Incubation Grant to Results for Development in May 2016. Results for Development staff reviewed this page prior to publication.

Summary

In May 2016, Results for Development (R4D) received a GiveWell Incubation Grant of $6,400,000 to support its pneumonia treatment program in Tanzania. R4D's goal for this grant was to increase the treatment coverage rate of pediatric amoxicillin, which is the World Health Organization-recommended first-line treatment for childhood pneumonia.1 This grant was intended to support the first phase of a project that R4D estimated would cost a total of $19 million. On this page, we analyze the outcomes of Phase I of the program, in part as an input into our decision about whether to recommend a second grant to R4D to support Phase II of the program.

What did they do? R4D has implemented Phase I of its pediatric amoxicillin program to treat childhood pneumonia in Tanzania. As part of Phase I, R4D directly funded the procurement and clearance of amoxicillin dispersible tablets (amoxicillin DT), worked with the Government of Tanzania to develop a co-financing strategy and create a budget allocation for amoxicillin DT, provided technical assistance to improve guidelines, forecasting, and supply planning, and conducted monitoring and evaluation, including measuring the rate of accurate diagnosis of childhood pneumonia and the rate of prescription of pediatric amoxicillin for children with pneumonia seeking care in the public sector.

Did it work? We believe that R4D's program likely increased the availability of pediatric amoxicillin in the public sector relative to the counterfactual in 2017 and 2018 by increasing the level of procurement of amoxicillin DT and improving forecasting and supply planning. However, we are highly uncertain about the extent of its impact on these factors because we have little evidence with which to estimate the counterfactual, and therefore rely strongly on R4D's guess about what the Government of Tanzania would have done in its absence. R4D's monitoring activities identified that clinicians are correctly diagnosing pneumonia in a minority of cases in Tanzania's public health facilities.

How cost-effective was Phase I of the program? We have not conducted a formal cost-effectiveness analysis of Phase I of the program. However, we have conducted a forward-looking cost-effectiveness analysis of Phase II of the program, which we believe is likely to be in the range of cost-effectiveness of our top charities, although this is highly uncertain. We think that Phase I was slightly more cost-effective than we project Phase II will be.

What did they do?

Note: All of the content below is based on communication that GiveWell had directly with R4D.

The program

Results for Development (R4D)'s pediatric amoxicillin program to treat childhood pneumonia in Tanzania is based on the following theory of change:

  1. Increases in the volume of pediatric amoxicillin procured for distribution to public health facilities and improvements in the timeliness of procurement through improved forecasting and supply planning contribute to an increase in the availability of pediatric amoxicillin in public health facilities.2
  2. Increased availability of pediatric amoxicillin leads to increases in the proportion of children with pneumonia who receive the appropriate treatment, assuming a certain minimum level of a) public sector care-seeking, b) correct diagnosis by clinicians, and c) prescription of pediatric amoxicillin for pneumonia cases, regardless of whether pneumonia was correctly diagnosed.

Phase I of the program, which took place in 2016-2018, involved the following components:

  • R4D aimed to increase the volume of amoxicillin dispersible tablets (amoxicillin DT) procured by directly funding the purchase of amoxicillin DT and working with the Government of Tanzania to encourage it to create a budget allocation for amoxicillin DT.
  • R4D provided technical assistance to the Ministry of Health, Community Development, Gender, Elderly and Children to improve its methodology for forecasting demand for pediatric amoxicillin through the 2020 fiscal year.
  • R4D provided technical assistance to the government agency responsible for procurement of medical supplies (the Medical Stores Department (MSD)) to support its initiation of procuring amoxicillin DT and to improve the timeliness of pediatric amoxicillin procurement through better supply planning.
  • R4D conducted monitoring to determine whether the pediatric amoxicillin that was procured was consistently reaching health facilities and therefore successfully increasing availability of amoxicillin for children who need it.
  • R4D and several partners conducted a clinical study to determine how accurately clinicians are diagnosing pneumonia in children and prescribing pediatric amoxicillin in the public sector.
  • R4D identified and tested a few interventions to improve the knowledge of public health facility clinicians and private sector drug shop dispensers about childhood pneumonia treatment.
  • R4D supported the government in developing a co-financing agreement to finance amoxicillin DT through the 2020 fiscal year, with the ultimate goal of transitioning funding for amoxicillin DT procurement to the Government of Tanzania in the medium to long term.

Spending breakdown

A detailed breakdown of the budget for Phase I can be found in R4D Budget, Phase I & II, November 2018. The total cost of Phase I was $6,400,000, which was allocated as follows:

  • 41% ($2,625,000) for pediatric amoxicillin procurement
  • 26% ($1,654,136) for monitoring and evaluation (inclusive of staff costs)
  • 25% ($1,577,307) for staff costs and expenses
  • 8% ($543,557) for public and private programmatic activities

Some of the funding allocated for procurement was not spent during Phase I; this will be used to support procurement during Phase II of the program.

Did it work?

  • Did R4D increase the total volume of pediatric amoxicillin procured relative to the counterfactual? We believe so, but we are highly uncertain about the size of its impact because we have little evidence with which to estimate the counterfactual. We rely strongly on R4D's guess for what the Government of Tanzania would have done in its absence.
  • Did R4D improve the timing of pediatric amoxicillin procurement relative to the counterfactual through improved supply planning? We believe that R4D was able to improve supply planning more quickly than it would likely have been improved in the counterfactual, for example by providing technical assistance to MSD. Our assessment here is based on R4D documents and conversations with R4D staff.
  • Did R4D improve availability of amoxicillin relative to the counterfactual? We believe so, because we believe that R4D increased the amount of amoxicillin procured and made improvements to supply planning. However, we are highly uncertain about the extent of its impact because we have little evidence with which to estimate the counterfactual.
  • Did R4D assess the level of clinician accuracy in diagnosing pneumonia? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania correctly diagnosed pneumonia in about 18% of pneumonia-positive cases, as identified by a lung ultrasound examination.
  • Did R4D assess the level of clinician accuracy in prescribing pediatric amoxicillin? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania prescribed pediatric amoxicillin in 72% of cases in which they correctly diagnosed a child as having pneumonia, as identified by a lung ultrasound examination. However, they also prescribed pediatric amoxicillin in approximately half of cases in which a child with pneumonia was incorrectly given a different diagnosis.
  • Did R4D make improvements to help sustain the gains in treatment availability in the event that the Government of Tanzania takes over the financing of amoxicillin DT in the longer term? We believe that R4D has better positioned the Government of Tanzania to take over financing of amoxicillin DT in the longer term, including by agreeing to a cost-sharing plan for product costs through the 2020-21 fiscal year.

Details follow.

Did R4D increase the total volume of pediatric amoxicillin procured relative to the counterfactual?

We believe that R4D increased the total volume of pediatric amoxicillin procured relative to the counterfactual, but we are unsure about the likely size of the increase because we have little information with which to estimate the counterfactual. Our guess relies strongly on R4D's own guess about the volume of pediatric amoxicillin the Government of Tanzania would have procured in its absence. We think that guess is reasonable but highly uncertain.

Some of the considerations we took into account include:

  • Procurement under the program: R4D told us that under the program in 2017, approximately 29 million tablets, amounting to approximately 1.5 million treatments, were procured.3 R4D told us this was equivalent to slightly more than one year's worth of demand.4 We have not seen data on the number of tablets procured in 2018, and 2017 was the first year in which R4D directly administered financing for the procurement of amoxicillin DT in Tanzania.5
  • Procurement in the counterfactual: R4D told us that had it not intervened, the Government of Tanzania would have made efforts to reallocate funding from the Global Financing Facility to procure an emergency shipment of three months' worth of amoxicillin DT for 2017.6 R4D told us that it expects that in its absence, and given the available funding sources and needs at the time, the Government of Tanzania could have procured another three months' worth of amoxicillin DT in 2018 by reallocating funds in its health commodities budget away from the procurement of other child health commodities, such as zinc and oral rehydration solution (the recommended combination treatment for diarrhea7), because diarrhea and pneumonia are the two largest causes of child mortality in Tanzania and the government does not appear to prioritize treatments for one over the other.8 If these guesses are correct, the amount of pediatric amoxicillin procured under the program in 2017 was more than four times greater than it would have been in the counterfactual. R4D told us that it planned for a similar level of procurement in 2018 as in 2017, but we have not yet seen data on the actual level of procurement in 2018 and therefore do not know by how much procurement increased during Phase I overall.
  • Uncertainty over the counterfactual estimate due to a lack of data: We have very little data with which to estimate the counterfactual and are relying on R4D's guess, so we are highly uncertain about the size of the program's impact on procurement.
  • Uncertainty over the counterfactual estimate due to the possibility of alternative funders: R4D's guess for counterfactual availability assumes that no alternative funder other than the Government of Tanzania would have intervened in its absence. We believe this is a reasonable assumption.9 Amoxicillin DT had previously been funded by the Canadian government and the Reproductive, Maternal, Newborn and Child Health Trust Fund, and administered by UNICEF. However, this funding ended as of mid-2016, leaving a large funding gap around the beginning of Phase I.10

Did R4D improve the timing of pediatric amoxicillin procurement relative to the counterfactual through improved supply planning?

Improvements to supply planning can increase the availability of pediatric amoxicillin for a given volume of the drug procured by increasing the likelihood that it is stocked in the health facilities where it is needed at the right time. We believe that R4D was able to improve supply planning under the program by providing technical assistance, including to MSD. While R4D believes that the Government of Tanzania would have sustained efforts to improve supply planning on its own in the counterfactual, we think it is likely that this occurred at a faster rate due to R4D's assistance. R4D told us about three specific improvements it made, or is making, to supply planning:

  1. Measuring seasonality of pneumonia incidence in 2017 and its impact on supply planning: R4D told us that previously no analysis had been performed on the number of people with pneumonia and other respiratory illnesses seeking care in the Tanzanian public health system in a given year, nor on the distribution of cases throughout the year. MSD previously did not take seasonality into account when conducting distribution planning of pediatric pneumonia treatments and assumed constant demand for pediatric amoxicillin through the year.11 In practice, actual demand is not constant through the year, for example due to the measured seasonality of pneumonia incidence in 2017.12 R4D measured a seasonality trend in pneumonia cases in 2017 by analyzing patient register data collected as part of a three-round health facility survey. R4D shared this analysis with the Government of Tanzania and is co-developing strategies to systematically improve distribution planning at the central level and supply planning at the health facility level.13
  2. Improving the consistency of procurement: The Government of Tanzania's health commodity procurements are often reactive, with stockouts occurring before additional volumes are ordered.14 R4D told us that it has provided support to the Government of Tanzania to order pediatric amoxicillin more consistently and proactively.15 We have not asked for more details on how it has helped the Government of Tanzania to do this, but R4D told us that it made regular orders itself in 2017.16
  3. Updating health facilities' request and requisition forms to facilitate ordering of amoxicillin DT:17 R4D advocated for the Government of Tanzania to update its request and requisition forms to include amoxicillin DT.18 This reduces the effort required by health staff to order amoxicillin DT, as they would otherwise have to write out their orders manually on the form.19 We think that reducing this barrier will make it more likely that health facilities will request amoxicillin DT from the MSD stores when it is needed.

Did R4D improve availability of amoxicillin relative to the counterfactual?

We believe that R4D likely increased availability of pediatric amoxicillin relative to the counterfactual (i.e., what would have happened during the same period if R4D were not present) because it achieved a higher level of procurement and made improvements to supply planning. However, we are unsure about the size of its impact because we have little information with which to estimate counterfactual availability.

Availability under the program

Methodology:

  • R4D used a stratified two-stage sampling procedure to randomly select a sample of 624 public health facilities that we believe are likely to be fairly representative of mainland Tanzania,20 with the exception that district hospitals and possibly health facilities that serve a small number of patients are over-represented.21 The sample consisted of 53 district hospitals, 50 health centers, and 521 dispensaries, which are the three types of public health facility in Tanzania.22 More details on the sampling procedure can be found in R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018.
  • Health facility surveys were conducted in March, July, and November 2017.23
  • All surveys were conducted after the first shipment of amoxicillin DT had entered Tanzania in January 2017 to avert national stockouts, so R4D does not have data on baseline availability of pediatric amoxicillin. R4D notes that amoxicillin DT availability in facilities that are designated as part of Group A in the Government of Tanzania's Integrated Logistics System (ILS) during Round 1 or amoxicillin OS availability in Round 2 can be used to approximate baseline data (more detail in footnote).24
  • A health facility is measured as having pediatric amoxicillin available if at least one unit of either amoxicillin DT or amoxicillin OS is available in the health facility on the day of the survey.25

Results:

  • Pediatric amoxicillin availability was measured at 67% in the November 2017 survey.26
  • We focus on the results from the third survey because by this point all six shipments scheduled for 2017 had entered Tanzania.27

Availability in the counterfactual

  • R4D's guess is that counterfactual availability would have been 25% in 2017 and 2018.28 This estimate is based on R4D's best guess that the Government of Tanzania would have procured three months' worth of stock in each year (calculated as a proportion of annual demand), and the assumption that the percentage of the annual need that is procured is equivalent to the level of availability.29
  • R4D's guess may be an overestimate due to challenges in supply planning. For the reasons explained above, challenges in supply planning in the counterfactual might mean that a procurement of three months' worth of supply translates into less than 25% availability.30
  • R4D's guess may be an underestimate due to amoxicillin OS stocks left over at the start of 2017. In Round 1 of the health facility survey in March 2017, amoxicillin OS availability was measured at 25%.31 By Round 3 in November 2017, amoxicillin OS availability was measured at 16%.32 R4D told us that amoxicillin OS was not procured during 2017,33 so these stocks were left over from the previous year and would have been available in the counterfactual as well. R4D's estimate of counterfactual availability takes into account only amoxicillin DT, though the measured availability of pediatric amoxicillin includes both amoxicillin DT and amoxicillin OS. Based on amoxicillin OS alone, availability of pediatric amoxicillin would have been between 16% and 25% in 2017, so it seems plausible to us that R4D's guess underestimates availability. The extent to which a government procurement of amoxicillin DT would have increased availability beyond this level would likely depend on the degree of overlap between facilities that had amoxicillin OS stocks and facilities that would have ordered amoxicillin DT.

Did R4D assess the level of clinician accuracy in diagnosing pneumonia?

A pneumonia diagnosis and prescription study undertaken in three regions of Tanzania by R4D, the Government of Tanzania, several Tanzanian partners, and IDinsight suggests that clinicians in public health facilities in Tanzania are not accurately diagnosing pneumonia in most cases. With less accurate diagnosis, increases in the availability of pediatric amoxicillin in health facilities will translate less effectively into increases in the proportion of children with pneumonia who receive treatment.34

Methodology:

  • Three regions were purposefully selected. The diagnosis and prescription study sample is not nationally representative. The Pwani, Dodoma, and Tabora regions were sampled because they include a mix of rural and urban areas and a mix of high- and low-performing regions, they are easy to access but geographically dispersed within Tanzania, and they are areas where R4D could potentially pilot interventions to improve diagnosis.35 We are unsure whether diagnosis rates in this sample are likely to under- or overestimate diagnosis rates in Tanzania as a whole.
  • Health facilities were randomly selected within those regions. A total of 83 health facilities were randomly selected, using a stratified sampling procedure, from these three regions. See this footnote for details.36
  • In each facility, approximately one provider was randomly selected from a list of all eligible providers.37 A total of 93 healthcare providers were surveyed.38 Enumerators observed all patients seen by those providers.39
  • Misdiagnosis was measured by comparing clinicians' diagnoses to results from lung ultrasound examinations.40

Results:

  • The pneumonia diagnosis and prescription clinical study found that only 18% of children with lung ultrasound-confirmed pneumonia were correctly diagnosed, out of a sample size of 847 children.41 We have not vetted this conclusion.

Did R4D assess the level of clinician accuracy in prescribing pediatric amoxicillin?

The pneumonia diagnosis and prescription clinical study suggests that clinicians prescribed pediatric amoxicillin in the majority of cases in which they had correctly diagnosed a child as having pneumonia. However, they also prescribed pediatric amoxicillin in nearly half of cases in which they had incorrectly diagnosed a child as having something other than pneumonia. Without accurate prescription, increases in the availability of pediatric amoxicillin in health facilities will have a reduced impact on increasing the proportion of children with pneumonia who receive treatment.

Methodology:

  • R4D's prescription rate estimates were based on the same study that was used to estimate the diagnosis rate (see above).
  • Prescription rates were estimated based on provider behavior in health facilities that had pediatric amoxicillin in stock.42

Results:

  • The diagnosis and prescription study found that in health facilities that had pediatric amoxicillin in stock, clinicians prescribed pediatric amoxicillin in 72% of cases where they had correctly diagnosed a child as having pneumonia.43 We have not vetted this estimate.
  • The diagnosis and prescription study found that in health facilities that had pediatric amoxicillin in stock, clinicians also prescribed pediatric amoxicillin in 46% of cases in which they had incorrectly diagnosed a child as having something other than pneumonia.44 We have not vetted this estimate. We are unsure why clinicians were prescribing pediatric amoxicillin to children with pneumonia whom they had incorrectly diagnosed as having something other than pneumonia in such a high proportion of cases.45
  • Based on these results, R4D calculates that about 51% of children under the age of five with lung ultrasound-confirmed pneumonia who visited a public health facility were prescribed pediatric amoxicillin.46

Did R4D make improvements to help sustain the gains in treatment availability in the event that the Government of Tanzania takes over procurement and financing in the longer term?

R4D told us that it has taken several steps during Phase I to prepare for the transition of all pediatric amoxicillin procurement and financing to the Government of Tanzania in the medium to long term, increasing the chance that gains in treatment availability made during Phase I can be sustained in the future. These steps included:

  • Supporting the Government of Tanzania in improving the robustness of its methodology for forecasting demand for pediatric amoxicillin. The previous methodology for estimating the demand underestimated pediatric amoxicillin demand in two ways. First, the methodology accounted for only two of three WHO-recommended age/weight bands, and did not include the age/weight band that requires the largest number of tablets per treatment.47 Second, it assumed that children with severe pneumonia would not receive pediatric amoxicillin, which did not align with WHO recommendations and health provider practices in Tanzania.48 R4D was involved in the forecasting process in 2016 and advocated for the Government of Tanzania to update the methodology to reduce the likelihood of underestimation. R4D told us it has estimated that this led to a 27% increase in forecast demand;49 we have not vetted this estimate. The level of demand forecast by the Government of Tanzania is an important input into the amount of funding it budgets for pediatric amoxicillin.
  • Developing a co-financing plan with the Government of Tanzania to finance the product costs of amoxicillin DT. In November 2017, R4D and the Government of Tanzania agreed to a co-financing plan under which the Government of Tanzania will pay for approximately 10%, 30%, 50%, and 60% of the product costs of amoxicillin DT over the fiscal years 2017-18, 2018-19, 2019-20, and 2020-21, respectively.50 We have not seen evidence of the actual amount the Government of Tanzania has spent so far, and we are uncertain whether the Government of Tanzania will meet this commitment in the future.

How cost-effective was Phase I of the program?

We have not conducted a formal cost-effectiveness analysis of Phase I of the program. We have conducted a forward-looking cost-effectiveness analysis of Phase II of the program, which can be found in this spreadsheet. We believe that Phase II is in the range of cost-effectiveness of our top charities, though our estimates are highly uncertain, particularly due to our uncertainty about the counterfactual. Our best guess is that Phase I was slightly more cost-effective than we project Phase II will be, as we expect that the counterfactual level of availability would grow over time (and therefore be higher in Phase II compared to Phase I) as the Government of Tanzania’s role in financing and procuring pediatric amoxicillin increases.

Sources

Document Source
GiveWell cost-effectiveness analysis, R4D Phase II Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016 Source
GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016 Source
R4D Budget, Phase I & II, November 2018 Source
R4D estimate of the counterfactual level of procurement, September 2018 Unpublished
R4D, Progress toward increasing treatment coverage for childhood pneumonia in Tanzania, May 2018 Unpublished
R4D, Summary of Monitoring, Evaluation & Learning Survey Methodologies, May 2018 Source
WHO Fact sheet, diarrhoeal disease Source (archive)
WHO Fact sheet, pneumonia Source (archive)