In a nutshell
In January 2022, GiveWell recommended a $5 million grant to PATH to accelerate the rollout of the RTS,S malaria vaccine in specific regions of Ghana, Kenya, and Malawi. By helping with logistics, procurement, and training, we thought PATH’s technical assistance would mean the RTS,S vaccine would be rolled out to more children in January 2023 – one year before they would have otherwise been reached. (more)
This page provides a lookback on this grant. We’ve assessed this grant’s performance against initial expectations and identified key lessons learned. This shallow assessment fits into a broader cross-cutting assessment of our technical assistance grantmaking, which you can read about here.
We would probably make this grant again. Positive updates we’ve gotten include:
- After we made this grant, the RTS,S vaccine was rolled out to additional children in additional communities in Ghana, Kenya, and Malawi in early 2023 as planned (though with one to two month delays in Ghana and Kenya). Coverage appears to have scaled faster than we had anticipated. For example, we expected ~50% of eligible children to have been given all three doses of the RTS,S vaccine by the end of 2023. Administrative vaccination data from the Ministry of Health suggests that three-dose coverage was ~70%. (more)
- Implementation targets for this grant were generally met – e.g., procurement plans and healthcare worker training were completed on time, though there were slight delays in Kenya due to the general election. (more)
- We still think that, had we not made this grant, the rollout of RTS,S in these areas would have happened in January 2024 at the earliest. This is based on retrospective conversations we’ve had with both Gavi, the Vaccine Alliance and the World Health Organization. (more)
- Given the changes in RTS,S coverage, we think this grant was plausibly around our cost-effectiveness threshold (~10x our benchmark). However, these estimates have wide confidence intervals and we haven’t prioritized making other updates to assumptions in our CEA. (more)
Broad takeaways and lessons for our grantmaking: (more)
- We should talk to more in-country stakeholders. We didn’t speak to anyone in these three countries during our investigation of this grant, or during this lookback. We think we should have in hindsight, given the importance of understanding local implementation context to properly interrogate what bottlenecks the TA might resolve.
- We should arrange our grants around predictable bottlenecks. In Kenya, the vaccine rollout was delayed by the 2023 general election. This was something we should have anticipated and arranged grant timelines around.
Published: January 2026
Background
In January 2022, GiveWell recommended a $5 million grant to PATH to accelerate the rollout of the RTS,S malaria vaccine in specific regions of Ghana, Kenya, and Malawi. These regions were used as comparison areas during the operational pilots of the RTS,S vaccine, which were launched in 2019.
Based on conversations we had with Gavi, the Vaccine Alliance at the time, we estimated that Gavi-supported implementation of the RTS,S vaccine would begin in the pilot comparison areas in January 2024. We thought that countries were unlikely to roll out the RTS,S vaccine without Gavi support, since at the time, the limited supply of RTS,S doses were only available through Gavi.1
By making this grant to PATH, we thought the operational activities needed to support a vaccine rollout would happen one year earlier, accelerating the introduction of RTS,S to January 2023. This would lead to more children counterfactually receiving the RTS,S vaccine.2
Would we have made these grants again, knowing what we know now?
Yes, though with important caveats about our confidence level. We think these grants were plausibly as cost-effective as our current cost-effectiveness threshold (~10x our benchmark3 ), based on M&E data we’ve seen and conversations we’ve had with Gavi and the World Health Organization (WHO). (more)
How did implementation go?
Coverage and financing estimates
For this grant, we didn’t fund independent surveys of RTS,S vaccine coverage because we’d heard government administrative data for vaccination coverage is generally more reliable compared to other commodities. There was also no need for a baseline survey as we knew that the malaria vaccine had not been rolled out in these locations, therefore baseline coverage was zero.
Coverage data we received from the governments of each country suggest that the RTS,S vaccine was rolled out to the new areas in early 2023 as planned, and that coverage among eligible children scaled faster than we had predicted. Aggregating across all three countries (weighted by population of eligible children), the percentage of eligible children that had received all three doses of the RTS,S vaccine was 69% in 2023, and 77% in 2024.
While we’ve heard from PATH and from various other experts over the years that vaccine administrative data tends to be more reliable compared to other commodities, we’ve also heard that administrative data can be inflated.4 One reason for this is due to incentives down the supply chain – for example, district health centers may over-report the number of vaccines administered to meet performance targets.
Implementation updates
Implementation targets were generally met, though there were slight delays in Kenya caused by the 2022 general election. In hindsight, we should have anticipated this and pushed back the grant milestones.
| Milestone | Predicted | Actual |
|---|---|---|
| All three countries make formal decisions to expand the vaccine into MVIP comparator areas. PATH provided financial and technical support for the meetings, facilitating the review of next steps and processes for expansion and the review of evidence from the pilots | May 2022 | April 2022 |
| PATH begins providing funding for the development, review, and finalization of expansion plans | June 2022 | Completed in Ghana and Malawi in May 2022, delayed in Kenya due to the presidential election |
| WHO and PATH developed a supply and shipment plan for donation doses with GSK and UNICEF | June 2022 | July 2022 |
| National and district-level health worker training completed | December 2022 | November 2022 (Malawi); February 2023 (Ghana and Kenya) |
| Launch of vaccine implementation in the comparison areas of all three countries | January 2023 | November 2022 (Malawi); February 2023 (Ghana); March 2023 (Kenya) |
We agreed to a six-month no-cost extension to this grant.5 The purpose of this extension was to ensure a smoother transition from pilot programming to routine immunization in 2024. While this doesn’t affect our overall impression of how this grant went, we think it serves as a reminder to think carefully about the handover period when investigating these grants, and allow for more flexibility in grant timelines if needed.
Discussions with stakeholders
In the course of this lookback, we spoke with Scott Gordon, head of the malaria vaccine programme at Gavi, about his impression of this grant. He told us that January 2024 would be a conservative estimate of the earliest that the countries would have introduced RTS,S in these areas without this grant, which corroborates our assumption that this grant accelerated the rollout by approximately a year.6
We also spoke with Eliane Pellaux-Furrer and John Francis at the World Health Organization, who were involved in overseeing the rollout of the RTS,S vaccine in these areas. They said that without PATH’s assistance, rollout in these comparator areas likely would not have started until January 2024, though Q4 2023 might have been possible.
They generally spoke positively of the grant. When we asked what they felt were the most useful activities PATH supported, they mentioned training healthcare workers, community sensitization, redesigning and reprinting vaccine cards, and redesigning recording tools at the health facility – all of which are non-trivial activities that they believe the Ministries of Health in each country would likely have been delayed in doing had it not been for PATH’s on-the-ground assistance. They also mentioned an additional benefit we had not anticipated: supporting the rollout of RTS,S in these areas added an additional learning step in the general global rollout of the RTS,S vaccine. For example, they updated the technical guidance materials for RTS,S rollout as part of this grant, which they thought might have accelerated preparation in other countries.
Finally, we spoke with PATH staff involved in the rollout of the vaccine in these areas. They shared with us several lessons learned:7
- Simple instructional materials and job aids are necessary for health workers administering the vaccine to ensure that they are aware of key guidelines, that dose administration data is recorded properly, and that caregivers are correctly informed about when to return for subsequent doses.
- The importance of using a variety of mediums for community outreach, including community radio programs and community meetings, as well as supporting key community influencers, in order to maximize vaccine uptake.
- Standardized training for community health workers to work with health facility staff is needed to ensure that vaccine implementation is effective and consistent. Involving community health volunteers has also been instrumental in communicating key information to families about the vaccine.
- Communication about the vaccine needs to be localized to different regional contexts to ensure that target populations are receiving necessary vaccine information.
How cost-effective do we think these grants were?
Based on a retrospective cost-effectiveness model, we think these grants were plausibly around our current cost-effectiveness threshold (10x), though with wide uncertainty intervals.
| Cost-effectiveness we modeled at the time | Retrospective cost-effectiveness estimates | |||
|---|---|---|---|---|
| 25th percentile8 | Best-guess | 75th percentile | ||
| PATH RTS,S | 14x9 | 12x | 14x | 16x |
To model cost-effectiveness, we fitted our original CEA into a template we’ve built for modeling technical assistance grants. Key assumptions in this model include:
- Effect of program on coverage: We use data from the Ministry of Health in these countries about how RTS,S vaccine coverage changed after rollout.10
- How coverage would have changed: We estimate that, without PATH’s assistance, rollout of the RTS,S vaccine would have happened one year later. This is based on our conversations with Gavi and the WHO. To estimate counterfactual impact, we take the ‘wedge’ between (i) our estimated effect of the grant on RTS,S coverage and (ii) how we expect RTS,S coverage would have changed otherwise. These assumptions are illustrated below.
- Program costs: We take the 2022 grant amount (~$5 million).
We did not revisit the downstream assumptions in our RTS,S model – e.g., the malaria burden in these countries or the effectiveness of the RTS,S vaccine in preventing severe malaria events.
Did we set ourselves up to learn?
Compared to other TA grants, we were fairly explicit about what milestones we expected this grant to hit, which made evaluating the grant easier in hindsight. We plan to do this more consistently for TA grants in future.
In hindsight, we think it would have been useful to fund independent RTS,S coverage surveys to cross-check the government admin data. This would have helped us to estimate the speed at which the vaccine rollout scaled, and may have also been of interest to the government / other development partners, insofar as it could have been an independent quality check on administrative records. We should have also spoken to more stakeholders in-country during both our investigation of this grant.
How calibrated were our forecasts?
We made six forecasts about this grant. Overall, we were slightly over-optimistic about the speed of rollout – we thought there was an 80% chance implementation would begin in at least two countries by January 2023, which didn’t happen. However, rollout was only delayed by one to two months in these cases.
| Forecast | Confidence (%) | Realization date | What happened? |
|---|---|---|---|
| PATH-supported implementation begins in pilot comparison areas in all three countries | 45% | 01/01/2023 | No |
| PATH-supported implementation begins in pilot comparison areas in at least two countries | 80% | 01/01/2023 | No |
| PATH-supported implementation begins in pilot comparison areas in at least one country | 90% | 01/01/2023 | Yes |
| Gavi-supported implementation begins in pilot comparison areas in all three countries | 12% | 01/01/2024 | No |
| Gavi-supported implementation begins in pilot comparison areas in at least two countries | 42% | 01/01/2024 | No |
| Gavi-supported implementation begins in pilot comparison areas in at least one country | 80% | 01/01/2024 | No |
Sources
- 1
Source: Email exchange with Gavi (unpublished).
- 2
You can read more about our case for the grant on our 2022 grant page.
- 3
More information about our benchmark can be found on this page.
- 4
Conversations with implementers (unpublished)
- 5
Source: GiveWell’s 6-month progress report on this grant (unpublished)
- 6
“I would anticipate that January 2024 would be a conservative estimate of the earliest that the countries would have introduced without the GiveWell support”, Scott Gordon email to GiveWell
- 7
Source: GiveWell’s 6-month progress report on this grant (unpublished)
- 8In our CEA, we've modeled three scenarios with different assumptions about the impact of Evidence Action's program on coverage:
- Our best guess of the effect
- A 25th percentile guess, representing what we consider a conservative estimate of program impact
- A 75th percentile guess, representing what we consider an optimistic estimate of program impact
- 9RTS,S CEA
- 10
These data are not public.