Published: August 2016
[Added December 19, 2016: GiveWell's experimental work is now known as GiveWell Incubation Grants.]
Note: This page summarizes the rationale behind a grant to Results for Development made by Good Ventures. Results for Development staff reviewed this page prior to publication.
As part of GiveWell’s general effort to support the creation of future top charities, in May of 2016 Good Ventures granted $6,400,000 to Results for Development (R4D) to support its Market Dynamics practice area’s pneumonia treatment program in Tanzania. The organization aims to use these funds to increase use of amoxicillin, the WHO-recommended first-line treatment for childhood pneumonia.
We spoke with Kanika Bahl, Jean Arkedis, Cammie Lee, Aalia Chatur, Stephanie Ahn, Nisal Liyanaarachchi, Aileen Palmer, and Nabila Hemed about updates on this grant on September 29, 2016.
Pneumonia is well-known to be a leading cause of death among young children worldwide.1 R4D told us that dedicated funding sources do not exist for pneumonia as they do for malaria, tuberculosis, and HIV/AIDS.2 As a consequence, countries often have a lower supply of pneumonia drugs in the public sector than is needed to treat their populations.3 In particular, funding in Tanzania for the child-friendly dispersible tablet formulation of amoxicillin, the WHO-recommended first line of treatment for pneumonia,4 had not been secured past mid-2016 despite a strong existing infrastructure for treatment.5 R4D intends to use this grant to fulfill as much unmet amoxicillin demand in Tanzania as is possible over the next couple of years.
Though we are somewhat uncertain on the details of R4D’s plan for our grant (beyond buying and distributing amoxicillin), our best understanding is that R4D intends to use our grant roughly as follows:
- About 40%: buy amoxicillin dispersible tablet (amox DT) for the next couple of years and donate the product to the government of Tanzania for use in public health facilities. This funding is intended to be catalytic and time-limited.
- About 25%: monitor and evaluate the intervention. (See below for more details on R4D's monitoring and evaluation plans.)
- About 10%: test programmatic interventions (e.g. mentoring) in public healthcare facilities and private drug shops to improve proper diagnosis and dispensing of treatment for childhood pneumonia.
- About 25%: hire and pay for R4D staff working on this project both in Washington, D.C. (where the organization is based) and in Tanzania.
R4D estimates that the treatment rate of pneumonia with amoxicillin was approximately 40% in 2014;6 we are uncertain about the accuracy of this estimate. The organization aims to increase the treatment rate to 55% by the end of the project in five years.7
Our impression is that the staff members at R4D working on this program have, in the past, focused their efforts on other types of work, such as analyzing the malaria bednet market to advise the Global Fund and other funders on bednet procurement.8 Starting in 2015, the R4D team launched a project to administer catalytic funding to purchase amox DT for the public sector in Ethiopia. However, our understanding is that R4D has not previously played a role similar to the one it aims to play with this grant.9 As a consequence, it has a limited track record on past projects directly relevant to this project.
Kanika Bahl, R4D's Managing Director and a member of the leadership team for this program, formerly worked at the Clinton Health Access Initiative, where she did work on direct implementation of similar projects (e.g. scaling pediatric AIDS drugs in developing countries).10
In general, we have a strong positive impression of Ms. Bahl and the R4D staff members with whom we corresponded as part of this investigation. We have communicated very well with these staff members, and they appear to place significant value on evidence-based decision-making and transparency.
If we eventually recommend R4D as a top charity, which is one of the main goals of this grant, that recommendation would rely on the track record that R4D built with these funds.
Monitoring and evaluation
In order to understand whether its activities are effectively improving the rate of pneumonia treatment in Tanzania, R4D plans to implement the following monitoring and evaluation activities:11
- National levels: R4D plans to look at national data on the procurement of amoxicillin approximately every six months to see whether additional stock is entering the public sector.12 It also plans to survey private sector importers approximately every six months to determine whether they are purchasing more stock.
- Availability data: R4D intends to survey health facilities to see whether amoxicillin is in stock and whether they have experienced stockouts in the past 30-90 days.
- Correct dispensing: R4D plans to review public clinical records and clinician reports to determine whether amoxicillin dispensing has risen and whether the treatment was prescribed correctly. R4D and GiveWell are both uncertain whether this data will be high-quality, and it is plausible that this information will not be helpful.
- Clinician understanding of appropriate pneumonia diagnosis and treatment practices: R4D plans to evaluate whether clinicians are retaining knowledge over the course of the mentoring sessions that it plans to implement. We do not expect that this data will be particularly relevant to our overall assessment of the grant’s success.
We are uncertain whether the above activities will be able to demonstrate that children sick with pneumonia actually receive treatment. R4D considered, but decided not to use, some other monitoring and evaluation activities that would have been aimed at answering this question:
- Video-recording clinicians: Obtaining informed consent for each video-taped patient would be too difficult for this possibility to be feasible.
- Surveying patients outside the clinic: R4D expects the number of treatments will be too low for this to be a cost-effective source of monitoring. R4D also suspects that poor patient recall and understanding could result in low-quality data.
- Surveying parents at home about their children's treatment: Again, R4D suspects that any data collected in this manner could be low-quality due to poor recall and understanding.
- A randomized controlled trial focused on child mortality: We expect that this would be expensive due to the required size of the trial, though it is possible that we could return to this possibility in the future.
We constructed a relatively rough cost-effectiveness model for R4D's pneumonia program, available here. Our current estimate of R4D's program puts it in the range of 50% to 200% as cost-effective as the mass distribution of long-lasting insecticide-treated nets.
There are a number of inputs to this cost-effectiveness analysis about which we are particularly uncertain:
- The current pneumonia treatment rate in Tanzania
- The level of increase in treatment rates that will be created by R4D's program
- Non-R4D costs due to the program (which are currently excluded entirely)
- The counterfactual: how would pneumonia treatment rates in Tanzania change in the short and long term in the absence of this grant?
We also have little sense of how cost-effective a future pneumonia treatment program outside of Tanzania might be. This question could become relevant if we considered recommending an organization implementing a similar pneumonia treatment program as a top charity.
Room for more funding
R4D considers this grant the first tranche of a five-year project that will cost about $19 million.
During the next 2.5 years, the organization hopes to find another funder who can replace its direct funding for amoxicillin, so that Tanzania has a sustainable source of funding for amoxicillin at the conclusion of the five-year project.13
We believe there may be substantial room for more funding for the broader intervention of scaling up evidence-backed, cost-effective health commodities, though we have not thoroughly considered this question nor carefully considered the cost-effectiveness of other well-known health commodities.
In our grant-making, we want to consider the question of fungibility: whether some other funder would have funded the project if we had not, in which case our grant would not actually lead to an increased pneumonia treatment rate.
Our impression is that this is unlikely to be the case. R4D told us that it has no other potential sources of funding.14 In addition, R4D is moving forward with a smaller grant than it originally proposed, and the organization also told us that it is not planning to try to raise additional funding because it does not have any reasonable prospects. R4D estimated a 25% or lower probability that, in the absence of this project, Tanzania would find another funding source for the amoxicillin this project plans to fund.15
Risks of the grant
We see a number of potential risks to the success of this grant:
- Monitoring and evaluation data: Due to the complexity of the intervention, it is possible that the monitoring and evaluation data for R4D's program could turn out to be of lower quality than the data for our current top charities. In particular, we are uncertain whether we will be able to see high-quality data demonstrating that children with pneumonia actually receive amoxicillin. (See above.) This could weaken the case for R4D or a similar organization becoming a future top charity.
- Current treatment rates: As stated above, we hold significant uncertainty regarding R4D's estimate for the current pneumonia treatment rate in Tanzania. It is possible that current treatment rates are higher than estimated, which could reduce the potential for this grant to raise treatment rates.
- Counterfactual: It is possible that a similar project would be funded in the absence of this grant, though our current guess is that this is relatively unlikely. (See above.)
- Transparency: We are uncertain about the level of comfort that officials in Tanzania will have with our expectations for transparency; it is possible that this could become an issue.
- Future plans: We are uncertain what R4D would do after this grant if it is successful. It is possible that the organization's subsequent plans would not lead it to be a contender for a top charity recommendation.
Plans for follow-up
As stated above, R4D considers this grant to be the first tranche of a five-year project costing around $19 million. The organization intends to seek a second tranche of funding, which could range in size from $6 million to $13 million, about twelve months from now. At that point, we expect to have some data on treatment procurements and clinic availability, which will provide us with some additional information. We are uncertain, however, whether this data will be conclusive enough for us to have a confident assessment of the grant's progress.
We expect to receive updates from R4D every few months for the next year about this grant; we plan to publish summaries of these updates when warranted. We expect to receive major updates at baseline, at eight months, at eleven months, and at fourteen months, as the organization intends to collect and share procurement and clinic availability data on this approximate timeline.16
We plan to pay particular attention to these aspects of the grant in following up:
- Operational updates: Has the program scaled as intended? Have any major challenges arisen?
- Funding updates: Have any other funders expressed interest in funding the program?
- Monitoring and evaluation updates: What does the monitoring data indicate about the program's impact? How high-quality is this data?
We are experimenting with recording explicit numerical forecasts of the probability of events related to our decision-making (especially grant-making). The idea behind this is to pull out the implicit predictions that are playing a role in our decisions, and to make it possible for us to look back on how well-calibrated and accurate those predictions were. For this grant, we are recording the following forecasts:
- Good Ventures gives R4D a second grant of approximately the same size in 12 months: 70%
- R4D is a top charity by the end of 2019: 25%
R4D contacted GiveWell in July 2015 about funding a pneumonia treatment program. Due to capacity constraints, we did not pursue the possibility. In January 2016, R4D contacted us again, and we spoke several times with the organization before deciding to make this grant. We also put significant time into constructing a cost-effectiveness model.
"The main killers of children under age five in 2015 include preterm birth complications (18 per cent), pneumonia (16 per cent), intrapartum-related complications (12 per cent), diarrhea (9 per cent), and sepsis/meningitis (9 per cent)." UN IGME - Levels & Trends in Child Mortality, Pg. 8.
From GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016: "A primary reason why R4D has chosen to focus on pneumonia is that the disease tends to receive less attention and funding than other high mortality burden diseases. Potential reasons for this include:
- Access to global, institutional funding for pneumonia treatment has been limited due to the disease's exclusion from major funding alliances such as the Global Fund to Fight AIDS, Tuberculosis, and Malaria and UNITAID. R4D has discussed this issue with funders such as the United States Agency for International Development (USAID) and the United Kingdom's Department for International Development (DFID).
- Some donors believe countries should be responsible for purchasing their own products for pneumonia treatment. This might lead countries to deprioritize pneumonia programs in favor of other programs that receive more donor attention and support. For example, in Ethiopia, 50% of the health budget comes from donors. Given the life-saving potential of a switch to Amox DT, R4D would encourage donors to consider funding Amox DT in the near-term to encourage treatment scale-up and a rapid switch from inferior pneumonia products to Amox DT." Pg 3.
- 3. In USAID - Evaluation of Country-Level Procurement Constraints, 28% of respondents representing 23 different countries said that amoxicillin stockouts were frequent, and 11% of respondents said that amoxicillin stockouts were moderate.
- Figures are from Table 5, Pg. 26.
- "This study is based on reviews of existing literature, feedback from questionnaires, telephone consultations, and face-to-face interviews with a wide range of stakeholders, including officials from Ministries of Health, Ministries of Finance, and Central Medical Stores and other health and procurement professionals…
Information in the form of survey responses or interview participation was received from 60 individual respondents from the following 23 countries: Democratic Republic of the Congo, Ethiopia, Mozambique, Tanzania, Zambia, Liberia, Nigeria, Uganda, Kenya, Madagascar, Mali, Senegal, South Sudan, Afghanistan, Bangladesh, Haiti, India, Colombia, the Dominican Republic, Guatemala, Honduras, Mexico, and Paraguay. Responses were not received from the following countries: Ghana, Rwanda, Malawi, Indonesia, Nepal, Pakistan, Yemen and the Philippines." Pg. vi.
- 4. "Previously, the World Health Organization (WHO) guidelines stated that pneumonia should be treated with cotrimoxazole, and that severe pneumonia, which has a higher mortality burden than non-severe pneumonia, should only be treated with injectable antibiotics. The guidelines now recommend Amox DT as a first line treatment for both severe and non-severe pneumonia, with severe pneumonia also requiring a referral to health centers for injectable antibiotics/supportive therapy." GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016, Pg 1.
"In mid-2016, the Tanzania program will face a funding cliff for Amox DT. The government has identified a need for $1 million to cover 1.5 million treatments over a 2-year period (2016-2017). It has made urgent efforts to adjust the guidelines and conduct trainings, so it would be unfortunate if this gap was not filled due to a lack of funding." GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016, Pg 5.
- "Assuming 60% of the 58% seeking care in the public sector receive antibiotics, and 100% of the 13% seeking care in the private sector receive antibiotics, antibiotic coverage would = 48%. We take a haircut on this to 40% because public sector fulfillment rates are likely lower than 60% currently" GiveWell's cost-effectiveness analysis for R4D's Pneumonia Program, "Deaths Averted" sheet, cell AC24.
- Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016.
"R4D started work in malaria and bednets about four or five years ago, when it determined there was an opportunity to lower the cost of nets. R4D produced two main recommendations to drive more cost-effective purchasing of bednets..." GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016, Pg 5.
- 9. "Ideally, as R4D is not a service delivery organization, other organizations would also be focused on delivering Amox DT treatment programs." GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, and Thayer Rosenberg on January 27, 2016, Pg 4.
- 10. “Ms. Bahl has experience doing similar programmatic work during her time at CHAI. For example, CHAI’s pediatric antiretroviral therapy (ARV) program provided financing in 34 countries to facilitate a switch from a suboptimal, bulky syrup that was difficult to transport and administer to a more effective single fixed-dose combination drug. Ms. Bahl was involved in launching this program in 17 countries where CHAI had not previously had a presence.” GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016, Pg 4.
All information in this section comes from the following phone conversations:
- Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016
- Kanika Bahl, Cammie Lee, Thayer Rosenberg, Jean Arkedis, and Aileen Palmer, phone conversation with GiveWell, April 21, 2016
- Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 26, 2016
- 12. The organization plans to collect data more frequently in the early stages of the grant: at baseline, at eight months, at eleven months, and at fourteen months. Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016.
- 13. Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016.
"R4D believes it is unlikely to find funding for its Tanzania program apart from GiveWell and Good Ventures." GiveWell's non-verbatim summary of a conversation with Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer on February 19, 2016, Pg 3.
- 15. Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016.
- 16. Kanika Bahl, Cammie Lee, Thayer Rosenberg, and Aileen Palmer, phone conversation with GiveWell, April 14, 2016.