[1] "Pneumonia is well-known to be a leading cause of death among young children worldwide." (GiveWell Incubation Grant write-up, August 2016)
[2] "R4D told us that dedicated funding sources do not exist for pneumonia as they do for malaria, tuberculosis, and HIV/AIDS. As a consequence, countries often have a lower supply of pneumonia drugs in the public sector than is needed to treat their populations. In particular, funding in Tanzania for the child-friendly dispersible tablet formulation of amoxicillin, the WHO-recommended first line of treatment for pneumonia, had not been secured past mid-2016 despite a strong existing infrastructure for treatment." (GiveWell Incubation Grant write-up, August 2016)
[3] Cammie S. Lee, Market Shaping Senior Director, Results for Development, email to GiveWell, October 30, 2019.
[4] "If we eventually recommend R4D as a top charity, which is one of the main goals of this grant, that recommendation would rely on the track record that R4D built with these funds." (GiveWell Incubation Grant write-up, August 2016)
[5] "Did it work?
- "Did R4D increase the total volume of pediatric amoxicillin procured relative to the counterfactual? We believe so, but we are highly uncertain about the size of its impact because we have little evidence with which to estimate the counterfactual. We rely strongly on R4D's guess for what the Government of Tanzania would have done in its absence.
- "Did R4D improve the timing of pediatric amoxicillin procurement relative to the counterfactual through improved supply planning? We believe that R4D was able to improve supply planning more quickly than it would likely have been improved in the counterfactual, for example by providing technical assistance to MSD. Our assessment here is based on R4D documents and conversations with R4D staff.
- "Did R4D improve availability of amoxicillin relative to the counterfactual? We believe so, because we believe that R4D increased the amount of amoxicillin procured and made improvements to supply planning. However, we are highly uncertain about the extent of its impact because we have little evidence with which to estimate the counterfactual.
- "Did R4D assess the level of clinician accuracy in diagnosing pneumonia? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania correctly diagnosed pneumonia in about 18% of pneumonia-positive cases, as identified by a lung ultrasound examination.
- "Did R4D assess the level of clinician accuracy in prescribing pediatric amoxicillin? The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania prescribed pediatric amoxicillin in 72% of cases in which they correctly diagnosed a child as having pneumonia, as identified by a lung ultrasound examination. However, they also prescribed pediatric amoxicillin in approximately half of cases in which a child with pneumonia was incorrectly given a different diagnosis.
- "Did R4D make improvements to help sustain the gains in treatment availability in the event that the Government of Tanzania takes over the financing of amoxicillin DT in the longer term? We believe that R4D has better positioned the Government of Tanzania to take over financing of amoxicillin DT in the longer term, including by agreeing to a cost-sharing plan for product costs through the 2020-21 fiscal year." (GiveWell phase I grant evaluation January 2019)
[6] See cells B89 and B93 in the "Cases ending in death w no amox" sheet of our 2016 cost-effectiveness model for R4D’s pneumonia program.
The 80% estimate was likely based on our perception that diagnostic accuracy was not a major concern among individuals we discussed this with, although the staff member who led this part of our work is no longer at GiveWell and we are unable to confirm the full list of factors that contributed to our estimate of the 80 percent guess.
[7] "Correct dispensing: R4D plans to review public clinical records and clinician reports to determine whether amoxicillin dispensing has risen and whether the treatment was prescribed correctly. R4D and GiveWell are both uncertain whether this data will be high-quality, and it is plausible that this information will not be helpful." (GiveWell Incubation Grant write-up, August 2016)
[8] "[A]cquiring the data on diagnosis and treatment has been more challenging than expected.
”Using data from medical records in public healthcare facilities
"R4D staff recently conducted an onsite visit to a public health dispensary – i.e., the lowest level of the public healthcare facility system. This dispensary is located in a rural area of the Mbeya Region, in the southern highlands of Tanzania.
"R4D expected to find some information in the facility’s outpatient records about patients’ diagnoses, treatments, and diagnostic tests performed – e.g., breaths per minute. Instead, it found that the records omitted mention of the diagnosis or treatment in many cases. In terms of diagnostic tests, only the malaria rapid diagnostic test (RDT) was recorded. While prescriptions were recorded, dosages were not – e.g., penicillin syrup was prescribed for a particular case, but the records did not specify how many bottles over what duration.
"So far, this is the only facility where R4D has seen the records… It is likely that these medical records are not representative of all healthcare facilities in the country.
"Given these findings, R4D is skeptical that medical records will provide accurate data on diagnosis and treatment trends in the country." (GiveWell's non-verbatim summary of a conversation with R4D on September 29, 2016, Pgs. 3-4).
[9] "We are uncertain whether the above activities will be able to demonstrate that children sick with pneumonia actually receive treatment. R4D considered, but decided not to use, some other monitoring and evaluation activities that would have been aimed at answering this question:
- "Video-recording clinicians: Obtaining informed consent for each video-taped patient would be too difficult for this possibility to be feasible.
- "Surveying patients outside the clinic: R4D expects the number of treatments will be too low for this to be a cost-effective source of monitoring. R4D also suspects that poor patient recall and understanding could result in low-quality data.
- "Surveying parents at home about their children's treatment: Again, R4D suspects that any data collected in this manner could be low-quality due to poor recall and understanding.
- "A randomized controlled trial focused on child mortality: We expect that this would be expensive due to the required size of the trial, though it is possible that we could return to this possibility in the future." (GiveWell Incubation Grant write-up, August 2016)
[10] "R4D is partnering with IDInsight. In addition to helping design the mentorship program, IDInsight will develop a set of options for dealing with misdiagnosis issues." (GiveWell's non-verbatim summary of a conversation with R4D on September 29, 2016, Pg. 6)
[11] "These exercises were conducted in collaboration with the Reproductive and Child Health Section (RCHS) of the Ministry of Health, Community Development, Gender, Elderly, and Children (MoHCDGEC), IDinsight, the President’s Office – Regional Administration and Local Government (PO-RALG), National Institute of Medical Research (NIMR), Muhimbili University of Health and Allied Sciences (MUHAS), and EDI Limited...
"Diagnosis and Prescription Study: clinical study conducted in three regions of Tanzania to measure the rates of pneumonia misdiagnosis and mis-prescription by comparing health care provider diagnosis to the result of a lung ultrasound examination." (R4D Summary of Monitoring Evaluation and Learning Survey Methodologies, May 2018, Pg. 1)
[12] "The R4D-facilitated pneumonia diagnosis and prescription study found that clinicians in public health facilities in Tanzania correctly diagnosed pneumonia in about 18% of pneumonia-positive cases, as identified by a lung ultrasound examination." (GiveWell phase I grant evaluation January 2019)
[13] "The diagnosis and prescription study found that in health facilities that had pediatric amoxicillin in stock, clinicians also prescribed pediatric amoxicillin in 46% of cases in which they had incorrectly diagnosed a child as having something other than pneumonia. We have not vetted this estimate. We are unsure why clinicians were prescribing pediatric amoxicillin to children with pneumonia whom they had incorrectly diagnosed as having something other than pneumonia in such a high proportion of cases." (GiveWell phase I grant evaluation January 2019)
[14] You can replicate our calculations by making a copy of our Phase II R4D cost-effectiveness analysis and changing row 38 in the "Amoxicillin procurement P2" tab. When you change the inputs in row 38 from 80% to 18%, the cost-effectiveness (cell B4 in the "Results P2" tab) changes from 6.6x to 4.8x, or falls by 27%.
[15] "Based on the assumptions described above, our best guess is that Phase II of R4D's pediatric amoxicillin program is roughly 5 times as cost-effective as cash transfers to people living in extreme poverty, which is in the range of cost-effectiveness of our top charities. However, this estimate is more uncertain than the cost-effectiveness estimates for our top charities." (GiveWell, Results for Development — Childhood Pneumonia Treatment Program (2019)
[16] GiveWell, Results for Development — Childhood Pneumonia Treatment Program (2019)