In a nutshell
In August 2020, GiveWell recommended a $3.9 million grant to Evidence Action to provide technical assistance (TA) to the Ministry of Health in Liberia to transition from HIV tests to HIV/syphilis dual tests and to scale-up maternal syphilis treatment in routine antenatal care. Evidence Action’s technical assistance included support for the Ministry of Health in training healthcare workers on syphilis testing and treatment protocols, in its application to the Global Fund for HIV/syphilis dual tests. In 2024, we made an unexpected $1.2 million top-up grant to this program to address (i) initial underbudgeting of program activities and (ii) to extend grant activities by a year.
This page provides a lookback on those grants. We’ve assessed this grant’s performance against initial expectations and identified key lessons learned. This shallow assessment fits into a broader cross-cutting assessment of our technical assistance grantmaking, which you can read about here.
Overall, we would probably make this grant again, though we’ve had both positive and negative updates since making the grant. Positive updates include:
- Government data suggests that the number of women in Liberia tested for syphilis scaled faster than we expected: from ~50,000 in 2021 to ~190,000 in 2024. We’ve heard this data may overstate the number reached, but we don’t think this would account for the large difference we observe over this period. (more)
- The estimated share of women who tested positive and received the correct treatment in 2024 was also higher than we expected (~50% vs. ~90%), though we think there are limits to what we can learn from this. The sample sizes for the surveys run by Evidence Action were very small (~40) because syphilis cases are relatively rare, making it difficult to construct a large sample of patients. (more)
- We spoke with the Program Manager for the National AIDS & STI Control Program in Liberia, who spoke positively of Evidence Action’s work. Without Evidence Action’s TA, he said that Liberia would likely have rolled out dual tests by now, but at a much slower rate compared to what they’ve achieved with Evidence Action’s assistance. (more)
- Given the increase in syphilis testing after the start of TA, we think these grants were plausibly around our cost-effectiveness threshold (~10x our benchmark). However, these estimates have wide confidence intervals and we haven’t prioritized making other updates to assumptions in our CEA. (more)
Negative updates include:
- This grant failed to get a syphilis treatment indicator integrated into official reporting protocols in 2020, which was a major implementation target. Without this, we’ve found it hard to assess how syphilis treatment rates in Liberia have changed since we made the grant. This indicator was added in 2025, which will hopefully give us a clearer picture in future. (more)
- We made an unexpected $1.2m top-up grant in 2024 because of unexpected program delays caused by COVID-19, and mistakes in the initial budget (e.g., not accounting for inflation). We think this erodes the cost-effectiveness of our initial grant in hindsight. (more)
Broad takeaways and lessons for our grantmaking: (more)
- We should talk to more in-country stakeholders. We didn’t speak to anyone in Liberia in our investigation of this grant, and we think potential difficulties with updating facility registers could have been proactively flagged had we done so.
- We should build more buffer into our budgets and check for obvious red flags (e.g., not building inflation into cost estimates).
- We should make sure grant activities are arranged around predictable bottlenecks and activities. In this case, we should have anticipated the need to align the grant handover period with the Global Fund funding cycle.
Published: January 2026
Background
In August 2020, GiveWell recommended a $3.9m grant to Evidence Action to support the Ministry of Health in Liberia in transitioning from using HIV tests to tests that screen for both HIV and syphilis (dual tests) and to scale-up syphilis treatment in routine antenatal care from 2020 to 2025. We thought Evidence Action could assist with this by:
- Supporting the Ministry in managing and monitoring the supply chain for both dual tests and benzathine penicillin G (BPG, the main drug used to treat maternal syphilis)
- Helping the National AIDS and STI Control Program train health facility workers on how to administer dual tests and the appropriate treatment protocols in the event of a positive test
- Supporting the Ministry of Health in its application to the Global Fund for HIV/syphilis dual tests
In 2024, we made an unexpected $1.2m top-up grant due to delays in program implementation, unanticipated program costs, and oversights in the original budget (e.g. inflation not being accounted for).
Would we have made these grants again, knowing what we know now?
We would probably make these grants again, as data from the government and independent health facility surveys commissioned by Evidence Action suggest syphilis screening and treatment in Liberia has scaled faster than we had anticipated – though there are limits to what we can learn from this data (more).
We’ve also had some negative implementation updates, such as having to make a top-up grant and a key implementation milestone (integrating a syphilis treatment indicator into facility questionnaires) being initially missed in 2020.1 However, this indicator was integrated in 2025, which makes us more confident that coverage improvements will be sustained once Evidence Action exits the program (more).
How did implementation go?
Screening and treatment estimates
The two main indicators we tracked for this grant were the number of pregnant women tested for syphilis in Liberia and the number of women who tested positive that were given the correct treatment (BPG).2
Screening
Women in Liberia are meant to be tested for syphilis at their first antenatal care (ANC) visits.3 Data on syphilis testing rates is captured in HIV Testing and Counseling (HTC) registers, and then uploaded to the DHIS2 system, an electronic database that tracks various indicators of healthcare coverage.4
Based on this data, the number of women being tested for syphilis in Liberia looks to have increased faster than we initially predicted: from ~50,000 in 2021 to ~190,000 in 2024, (compared to our initial prediction of ~50,000 to ~170,000).5 While this might imply this grant exceeded expectations, we think we should be cautious about placing too much weight on this data:
- We’ve heard that government administrative data can be unreliable because of facilities forgetting to upload the data, or having incentives to over-report to meet performance targets
- It seems plausible to us that recording of syphilis testing might have improved because of this grant, in which case this improvement might partially reflect better measurement rather than ‘real’ increases in syphilis screening coverage
Treatment
Before the 2020 grant, there was no syphilis treatment indicator in any facility ANC register.6 Getting this indicator integrated into at least one register was a core objective of this grant, and how we planned to primarily track syphilis treatment rates over time.7 We also earmarked funding for facility surveys, which we planned to use to validate the administrative data, and triangulate using other estimation approaches.8
This grant failed to get a treatment indicator integrated in 2020, which meant we were reliant on survey estimates only to assess syphilis treatment rates.9 To administer these, Evidence Action contracted with an independent data collection firm, who visited randomly selected public and private government health facilities.10 If there were syphilis-positive patients recorded, enumerators would ask to see their patient chart or any other record of treatment, and record whether they had been treated appropriately (with BPG).11
This data suggests that the percentage of syphilis-positive women receiving the correct treatment was much higher than we initially anticipated. For example, while we expected 52% of syphilis-positive women to be correctly treated by 2024, the patient chart data implies the figure was closer to 90%.
However, we think there are limitations to what we can learn from this data. First, the sample sizes are very small in the first two years of data collection: only 14 patient charts were reviewed in 2022, and 40 in 2023.12 When we asked why the sample sizes were so small, Evidence Action gave the following explanation:
- There are many small volume health facilities seeing less than 15 pregnant women for their first ANC visit in a month. Since syphilis is also a relatively rare condition, it was very unlikely that there happened to be a recent syphilis patient at the time of enumerator visit. Approximately 40% of the sample in the first year consisted of these small volume sites
- Enumerators tried to look back through historic records in health facilities, but had a difficult time tracking down records that were more than 3 months old. Additionally, some facilities didn't use the HIV Testing and Counseling register or weren't filling it in completely, so there was no source record for identifying who had tested positive for syphilis
In 2024, Evidence Action introduced two changes:13
- An additional quarterly survey to obtain more data.
- A switch to probability-proportionate-to-size sampling, as opposed to stratified random sampling where facilities were stratified by county and facility caseload. This made it more likely for larger facilities (which likely had more syphilis patients) to be selected for surveying.
This increased the sample size – in 2024, Evidence Action reviewed 101 patient charts.14 While this gives us more information to estimate treatment coverage today, the sparsity of historic data points and the change in sampling strategy makes comparison difficult.
Due to the limited data provided by patient charts, Evidence Action also estimated syphilis treatment coverage via a more indirect method, using proxy variables from their health facility surveys. They estimated syphilis treatment rates using the following formula:
Syphilis treatment rate (%) = % of health facilities with BPG available * % of healthcare workers knowing correct treatment protocols when asked * % of women that had consented to treatment after being correctly diagnosed
This indirect method yielded estimated treatment rates that were generally lower than the chart reviews, but higher than GiveWell’s initial assumptions.
This also seems like a positive update, but we think there are limitations to how much we can learn from this method: it doesn’t measure syphilis treatment directly, has similarly small sample sizes (e.g., in 2024 we had BPG availability data for just 41 health facilities), and the change in sampling strategy also makes comparisons to rates prior to the start of TA difficult.
Stepping back, we think we probably underestimated the percentage of positive-testing women who would be treated for syphilis, but a 90% treatment rate feels surprisingly high to us – almost at the upper limit of what’s feasible. We've tried to triangulate this by looking at syphilis treatment rates in neighboring countries. This WHO data suggests similarly high rates in nearby Guinea and Mali (100%), but much lower rates in neighboring Sierra Leone (40%). We haven't investigated the quality of this data, and find these cross-country differences also surprising
Implementation updates
- Monitoring and evaluation (M&E) protocols: This grant had intended to get a syphilis treatment indicator integrated into official ANC registers in 2020, but that didn’t happen.15 When we asked Evidence Action about this, they thought this indicator didn’t get integrated in 2020 because of coordination gaps between the National AIDS and STI Control Program (NACP) and the Department of Family Health (DFH). The NACP believed the indicator would be integrated into the Antenatal Care register by the DFH, but DFH ultimately elected to delete it without discussion with NACP because they argued the register did not have enough space. In 2025, Evidence Action successfully appealed to NACP to add a syphilis treatment indicator to the HTC register.16 This gives us more confidence in coverage increases of treatment being sustained once Evidence Action exits, as the government be better able to track the performance of this indicator over time
- Top-up grants: We made an unexpected $1.2m top-up grant in 2024. One reason we made this grant is that the 2020 grant was under-budgeted, because inflation wasn’t accounted for in the initial budget (which we did not notice initially).17 Another reason was that Evidence Action requested an extension to the grant, partially because the initial grant period (2020-2025) didn’t align with the funding cycles of the Global Fund, a major funder of dual tests. We thought this might threaten program sustainability, as getting dual tests into Liberia’s Global Fund application was a key part of how we thought the supply of dual tests would remain high after Evidence Action exited. We extended grant activities until 2026, so that the handover period aligned with supporting proposals for the 2026-2028 Global Fund funding cycle.18 In hindsight, we think we should have anticipated the need to align this grant with this funding cycle.
Discussions with local stakeholders
We spoke with the Program Manager for the National AIDS & STI Control Program in Liberia, who spoke positively of Evidence Action’s work. Without Evidence Action’s technical assistance, he said that Liberia would likely have rolled out dual tests by now, but at a much slower rate compared to what they’ve achieved with Evidence Action’s assistance.19 He pointed to Liberia being comparatively further ahead with the rollout of dual tests compared to neighboring countries as evidence for this. We haven’t independently verified this by checking on the status of dual test programs in neighboring countries.
How cost-effective do we think these grants were?
Based on a retrospective cost-effectiveness model, we think these grants were plausibly around our current cost-effectiveness threshold (10x), though with wide uncertainty intervals.
| Cost-effectiveness we modeled at the time | Retrospective cost-effectiveness estimates | |||
|---|---|---|---|---|
|
25th percentile 20 |
Best-guess | 75th percentile | ||
| Evidence Action dual tests | 14x21 | 3x | 12x22 | 18x |
To model cost-effectiveness, we fitted our original CEA for the dual tests grant into a template we’ve built for modeling technical assistance grants. Key assumptions in this model include:
- Effect of program on coverage: We put some weight on screening data reported by government, but make slight adjustments to account for the possibility that this data might be capturing better measurement.
- How coverage would have changed: We estimate that dual tests would have been rolled out four years later in Liberia without Evidence Action’s support, based partly on conversations with stakeholders in Liberia and partly on conversations in Evidence Action. To estimate counterfactual impact, we take the ‘wedge’ between (i) our estimated effect of the program on screening coverage and (ii) how we expect syphilis screening would have changed otherwise. These assumptions are illustrated below.
- Program costs: We base cost estimates on our $3.9m 2020 grant and our $1.2m 2024 top-up grant.
We did not revisit the downstream assumptions in our dual tests model – e.g., the syphilis burden in Liberia, or the percentage of women attending antenatal care visits.
Did we set ourselves up to learn?
Our ability to assess increases in syphilis treatment coverage is limited by very small sample sizes in 2022 and 2023 and a switch to probability-proportionate-to-size sampling in 2024. In hindsight, we think we should have used this sampling strategy from the outset, as we should have anticipated that low-volume health facilities would likely have very few syphilis patients.
How calibrated were our forecasts?
We made three forecasts related to our 2020 grant. Two of them are not yet realized; the other was realized when we recommended a $15m grant to Evidence Action to support the rollout of HIV/syphilis dual tests in Cameroon and Zambia.
| Forecast | Confidence (%) | Realization date | What happened? |
|---|---|---|---|
| We will assess this work as being more than 10 times as cost-effective as our benchmark at the end of the grant | 50% | 12/31/2026 | N/A |
| Within five years, Evidence Action will have largely transitioned this work to the Liberian government and will be spending less than $150,000 per year in-country on the program (including ongoing political support, monitoring support, etc.). | 60% | 12/31/2026 | N/A |
| By the end of this grant, GiveWell will recommend at least one other grant for syphilis screening and treatment in pregnancy | 25% | 12/31/2026 | Yes |
Sources
- 1
We reported this as one of the primary objectives of the grant in our 2020 grant page here.
- 2
See this section of our 2020 grant page where we mention the primary metrics we planned to track from the grant.
- 3
This claim was relayed to us by Evidence Action at the time of our 2020 investigation. (more)
- 4
For more, see this footnote from our 2020 grant page.
- 5
Source: Data on dual testing in Liberia provided to GiveWell by Evidence Action (unpublished)
- 6
Source: Conversations with Evidence Action (unpublished)
- 7
We reported this as one of the primary objectives of the grant in our 2020 grant page here.
“Continuing to support the revision of facility reporting templates to ensure integration of syphilis screening and treatment” - 8
See this section of our 2020 grant page. Our budget included: “Monitoring and evaluation – approximately $575,000, including the cost of monitoring and evaluation staff and local enumerators for the comprehensive facility survey.”
- 9
Source: Conversations with Evidence Action (unpublished)
- 10
Source: Conversations with Evidence Action (unpublished)
- 11
Source: Conversations with Evidence Action (unpublished)
- 12
Source: Data on dual testing in Liberia provided to GiveWell by Evidence Action (unpublished)
- 13
These changes were discussed as part of our case for our 2024 grant
- 14
Source: Data on dual testing in Liberia provided to GiveWell by Evidence Action (unpublished)
- 15
- We reported this as one of the primary objectives of the grant in our 2020 grant page here.
- “Continuing to support the revision of facility reporting templates to ensure integration of syphilis screening and treatment”
- The indicator not being integrated into official registers in 2020 was reported to us by Evidence Action
- We reported this as one of the primary objectives of the grant in our 2020 grant page here.
- 16
Source: Conversations with Evidence Action (unpublished)
- 17
Source: Evidence Action, Liberia maternal syphilis program budget, 2020 (public)
- 18
Supporting procurement through the Global Fund was one of the planned activities of the grant that we mentioned in our 2020 grant page.
- 19
Source: Conversations with the Program Manager for the National AIDS & STI Control Program in Liberia (unpublished)
- 20In our CEA, we've modeled three scenarios with different assumptions about the impact of Evidence Action's program on coverage:
- Our best guess of the effect
- A 25th percentile guess, representing what we consider a conservative estimate of program impact
- A 75th percentile guess, representing what we consider an optimistic estimate of program impact
- 21See our CEA for the grant at the time of making it in 2020 here.
- 22See our retrospective CEA that we put together at the time of writing this lookback here.