Clinton Health Access Initiative (CHAI) — Program Performance Management Grant (October 2024)

Note: This page summarizes the rationale behind a GiveWell grant to CHAI. It reflects our understanding at the time we recommended the grant in October 2024. CHAI staff reviewed this page prior to publication.

Published: March 2025

Summary

What we think this grant will do

This 3-year grant will fund CHAI to:

• Create a PPM team who will conduct impact and cost-effectiveness analyses to help CHAI to improve the performance of programs across its ~$200 million per annum portfolio (a much broader range of programs than those already considered within the GiveWell-funded incubator). We believe that this $200 million per annum of CHAI expenditure influences approximately $400 million of on the ground programming given that CHAI typically partners with other implementing organizations and/or government ($3.9 million).
• Improve its systems for hiring and retaining staff ($1.2 million).

i) Program performance management (PPM) team:

GiveWell's funding will pay for establishment of a PPM team of around 5 full-time equivalent (FTE) staff for 3 years (~$650k per annum). CHAI has already hired one manager for this team, and GiveWell's funding will be used to hire a program director and analysts. They also expect to spend ~$650k per annum on consultants, data tools, and travel to support this work.

This PPM team will conduct impact and cost-effectiveness analyses, partly based on the learnings CHAI has made through the GiveWell-funded Maximum Impact Incubator. We understand that these analyses will be done on existing and prospective programs, but will focus on programs where they think it could have the highest impact (e.g., larger programs, or those where there is more room for improvement).

This work will be used by key decision makers at CHAI to guide strategic decision-making across CHAI's programming portfolio. CHAI has outlined several ways that the impact and cost-effectiveness analyses of the PPM team, in collaboration with existing CHAI teams, could impact CHAI's programming, including:

• Country selection on multi-country grants: In cases in which CHAI has flexibility to choose which countries to propose to a funder for a specific program, the PPM team, in partnership with country and program leads, can identify and suggest the most cost-effective subset of countries while also weighing other impact factors (such as feasibility) for consideration. This may enable CHAI to select countries that are more cost-effective than countries they otherwise may have selected.
• Changes to program design and targeting: The PPM team's analysis may identify specific target populations or other program design tweaks that would increase the cost-effectiveness of programs.
• Choice of which new program and funding opportunities (such as RFPs) to pursue: The analyses of the PPM team may help CHAI identify cost-effective funding opportunities, such as which RFPs' application criteria present the best opportunities to propose cost-effective programs.
• Mobilize resources across programs: The PPM team may inform the CHAI senior leadership on how to shift internal resources towards more cost-effective programs and teams.

(More)

ii) Talent and enabling infrastructure:

GiveWell funding (~$400k per annum) will allow CHAI to bring onboard a new human resources management information system (HRMIS) and a new manager on the HR team focused on "new talent partnerships and external recruitment strategies".1

We have spent very little time investigating this component of the grant. However, we think that it is likely that this enabling infrastructure will enable CHAI to improve retention of high-performing staff and improve recruitment, especially for in-country offices but also for global teams. (More)

Why we made this grant

We think this is a good use of funds because:

• The cost-effectiveness of this grant is likely above our bar, because it leverages the PPM team to influence a large dollar amount of programming across CHAI's portfolio. Our best guess is that this grant is 16x as cost-effective as cash transfers.2 This estimate assumes that the PPM team can shift 10% of the $400 million portfolio of programming CHAI is a partner on per year to be 20% more cost-effective. (More)
• We believe that this is a timely moment to institutionalize cost-effectiveness in a major global health implementer, given the level of buy-in from current senior leadership. CHAI's CEO, Dr. Neil Buddy Shah, previously worked for GiveWell. We have a strong impression of several other members of CHAI's leadership who we believe are aligned on the importance of cost-effectiveness, in part based on our work together through the incubator. (More)
• The success of the GiveWell-funded Maximum Impact Incubator so far gives us some confidence that CHAI could achieve their goals on this work. We made a $10.4 million grant to CHAI to create the incubator in 2022, and our best guess (two-thirds of the way through the grant period) is that it will achieve the goals it set out. The incubator has so far led to $23 million in grantmaking (programs to increase the uptake of zinc/Oral Rehydration Salts, and preventive TB treatment), three pilot/ground-truthing projects, and early stage reviews of approximately 20 additional programs (several of which could lead to future grants). (More)
• Improving CHAI's ability to hire and retain key staff members could have positive spillover effects for GiveWell's past and potential future investments in the Maximum Impact Incubator. Continuing to hire high quality team members in country and global program teams could improve the work that CHAI proposes to GiveWell through the incubator. (More)

A summary of our cost-effectiveness model is shown in the table below.

	Best guess
Costs
Grant size ($ million)	5.1
Benefits
Length of program (years)	3
Length of impact (years)	5
Ramp-up time to hire and onboard new team members (years)	0.5
Length of impact after ramp-up time (years)	4.5
CHAI annual programmatic budget ($ million)	200
External reach multiple	2
Annual programmatic budget (CHAI + other actors) ($ million)	400
Average cost-effectiveness of programs (xCash)	3
Proportion of programs influenced by PPM team	10%
Improvement in cost-effectiveness generated by PPM team for CHAI's programs	20%
Units of value per dollar, GiveDirectly	0.0034
Annual increase in value from improvement in programs influenced by PPM team	80508
Discount rate	4%
Probability of failure	20%
Cost-effectiveness
Total increase in value from improvement in programs influenced by PPM team	270,937
Cost-effectiveness (xCash)	16

Main reservations

Our primary reservations are:

• Key personnel risk. If key members of CHAI leadership who are strongly aligned with GiveWell's cost-effectiveness approach were to leave in the next five years, the benefits of this grant are less likely to materialize. We have tried to build this risk into our BOTEC through a 20% probability of failure, but we may be underestimating. (More)
• We could be overestimating the chance of success of the PPM team. The success of the PPM team will depend on several hard-to-predict factors such as whether other funders and governments are amenable to the program adjustments the team may suggest. Additionally, we are uncertain how difficult it might be for CHAI to hire strong recruits for this team, because this team will require a fairly specific skill set.
  • We have conducted a series of threshold analyses for the key parameters which suggest that even with fairly conservative assumptions the grant looks cost-effective. However, we have considered each parameter in isolation, so it is possible the risk of failure is greater than we've accounted for. (More)
• Even if the PPM team causes CHAI's programs to be more cost-effective, this might not improve the net cost-effectiveness across implementers in a given program context. It is plausible that if CHAI chooses to implement a more cost-effective program, this may just cause an alternative implementer to implement the less cost-effective program that CHAI would have otherwise run. (More)
• We could be funging other donors. If another funder would have funded this proposal in the absence of our grant, it would mean that our funding is effectively enabling less cost-effective organizational strengthening activities. (More)
• We are unlikely to know whether this program turned out to be a cost-effective use of GiveWell funds. We (and CHAI) do not have a strong sense of the current cost-effectiveness of CHAI's overall portfolio. As a result, we will not have a baseline measure of cost-effectiveness to compare to, which would be necessary to retrospectively evaluate the impact of this grant. We plan to gather other indicators of this grant's impact, but they will inherently be less certain without a baseline. (More)
• Work to support the PPM team could detract from the incubator team's time. We have a slight concern that the incubator team, who works closely with GiveWell to scope and scale new cost-effective programs, could get drawn into spending time supporting the PPM team, given the familiarity they have built with cost-effectiveness modelling. CHAI told us that no members of the incubator team would directly work in the PPM team, except one member who would serve on the steering committee which supports the PPM team. (More)

The organization

The Clinton Health Access Initiative (CHAI) is an international public health non-profit founded in 2002, which operates in 36 countries.3 GiveWell is the sole funder of the CHAI Incubator, a program within CHAI that exists to find, assess, and implement programs that have high cost-effectiveness and potential to scale.

Through the incubator, we have also funded CHAI to implement an oral rehydration salt and zinc distribution program in Bauchi, Nigeria and a household contact management program for tuberculosis in Bihar and Uttar Pradesh, India.

The grant

This 3-year grant will fund CHAI to:

• i) create a PPM team who will conduct impact and cost-effectiveness analyses to help CHAI to improve the impact, performance, and cost-effectiveness of a wider range of its ~$200 million per annum portfolio of programs (beyond the programs within the GiveWell-funded incubator) ($3.9 million), and
• ii) improve its systems for hiring and retaining staff ($1.2 million).

We expect this to be a one-time grant because: i) we believe that CHAI's lack of funding to improve core organizational functions might be addressed in the long-term through structural changes, or ii) CHAI might find funding to build out a business development team, which could in turn raise funds to make these functions self-sustaining.

We later (during year 3 of the grant) expect to commission an external evaluator to support a process evaluation to better understand how the grant went. We believe that the additional grant amount for this evaluation would be very small.4

What we think this grant will do

We think that this grant will enable CHAI to improve the overall impact and cost-effectiveness of its portfolio of programs through the work of the PPM team and help CHAI to hire and retain key staff members.

Main grant activities

i) Program performance management (PPM) team:

GiveWell's funding will pay for a fully dedicated PPM team, which we expect will consist of 5 full-time equivalent (FTE) staff for 3 years (~$650K per annum).5 CHAI has already hired one manager for this team, and GiveWell's funding will be used to hire the program director and analysts.6 They also expect to spend ~$420k per annum on consultants to support this work.7 They have also budgeted ~$230K per annum for data tools, travel and miscellaneous/indirect costs. Our understanding is that the PPM team will work as follows:

• They will build a system for estimating the impact of programs and setting goals, which will include conduct cost-effectiveness analyses for a portion of the portfolio using a similar approach to GiveWell, partly based on the learnings CHAI has made about our approach through the GiveWell-funded Maximum Impact Incubator. We understand that these analyses will be done on existing and prospective programs, but will focus on programs where they think it could have the highest impact (e.g., larger programs, or those where there is more room for improvement).
• They will report the outcome of those analyses directly to key decision makers at CHAI:
  • The team will report into CHAI's Chief Operating Officer (COO), who sits on their 6-person Senior Leadership team as well as a newly created and broader 17-person Leadership Council.8
  • The outputs of the PPM team, including the cost-effectiveness analyses, will be used as an input by the Leadership Council (during monthly 2-hour meetings) and other working groups when discussing strategic decisions (more below). Final sign-off on strategic and programmatic decisions is typically made by the CEO (Buddy Shah), the COO, and sometimes the relevant Vice-President, Program Director, or Country Director (by program area or geography) depending on the nature of the decision.9
• Theory of change: Whilst some of CHAI's teams think about cost-effectiveness, this is done in an ad hoc way across teams and is sometimes not given enough emphasis compared to other considerations.10 The PPM team would provide the tools for decision makers to incorporate impact modeling and cost-effectiveness into their decisions, and would enable senior leadership to set goals based on estimated impact and hold program leads to account about the impact, cost-effectiveness, and progress of their team's work.11 CHAI has outlined several specific ways that the PPM team could improve the cost-effectiveness of its portfolio:
  • Within program verticals:
    • Country selection on multi-country grants: CHAI sometimes has flexibility to choose which countries to propose to a funder, either because it is responding to an RFP which lists a broad set of countries or criteria, or because it has a relationship with a funder where it can co-design programs more proactively. The PPM team, in collaboration with country and program teams, can identify and suggest the most cost-effective subset of countries while also weighing additional impact factors (such as feasibility) for consideration.
    • Changes to program design and targeting: Within countries, CHAI has told us that they can often choose which subnational regions to work in, what implementation model to use, and which demographic groups to target.12 These also present opportunities for the PPM team to improve the impact and/or cost-effectiveness of CHAI's programs.
      • Our understanding is that even quite restrictive RFPs can have this type of flexibility.
      • In addition, CHAI has a close relationship with several large donors of whom some might be open to co-designing programs more proactively. In these cases, CHAI may have even more flexibility to propose these kinds of program design changes than with reactive business development through RFPs.
    • Choice of which RFPs to apply for: CHAI teams may have a choice of several RFPs and only have the capacity to respond to a subset. The PPM team may provide guidance to help teams respond to RFPs which present the better opportunity to design impactful and cost-effective programs.13
  • Across program verticals:
    • Mobilize internal resources across program verticals. Based on the work of the PPM team, CHAI senior leadership can prioritize attention and shift internal resources (e.g. communications and fundraising capacity) towards the programs with higher anticipated cost-effectiveness, which might make CHAI more likely to win work in those areas, make the programs more likely to be successful, or help increase the scale of the programs.
      • We understand that there are three levers CHAI can pull here, two of which ((ii) and (iii)) they are still trying to raise funding for as part of this push for organizational support: i) CHAI senior leadership has discretionary time they can spend on specific projects, ii) their business development team could try to raise additional funding for the most cost-effective programs to try to increase scale, iii) The outputs of PPM analysis would be used by CHAI teams to create sector influence to support fundraising efforts for key programs or encourage other partner organizations to change their decision making.14

  
  

  ii) Talent and enabling infrastructure:

  GiveWell funding will pay for the licence and implementation of a new human resources management information system (HRMIS) and the hiring of a new HR manager focused on new talent partnerships and external recruitment strategies.15 They have also budgeted ~$70k annually for data tools, travel and miscellaneous/indirect costs. CHAI has proposed a set of other activities that the remainder of this funding might be used for, but the final decision is still to be determined.16 However, we believe that one part of this will be paying for recruitment outreach outside of the US to help hiring in country offices.17

  CHAI believes that these activities will allow it to hire and retain high performing team members for both its global and in country offices. We have spent very little time investigating this component of the grant.18 However, our understanding of the theory of change is as follows:

  • HRMIS: CHAI told us that they do not currently have any formal human resources management software, and that this makes several HR functions harder (which can lead to team members being treated inconsistently and increase non-retention risk, as well as delaying hiring and onboarding new hires).19 CHAI believes that an HRMIS system may help address this, but we have not vetted that claim.
  • Hiring a manager on the HR team and setting up outreach programs including outside of the US to improve recruitment for country offices: CHAI staff give recruitment presentations at departments in US universities, but it is not currently doing this elsewhere. Based on the experience of the malaria and NTDs team, they believe that in some countries there are a small number of feeder universities where similar outreach could help them to identify better candidates for their country offices.20 We understand that this new manager will also focus on hiring for its global teams, not just its in country offices.21

  Budget for grant activities

  The total budget for the grant is $5.1 million. It is broken down as follows:

  Summary	Year 1	Year 2	Year 3	Total
  Personnel	$712,000	$733,360	$778,022	$2,223,382
  Software, Data Tools, & Technical Support	$679,000	$699,270	$633,325	$2,011,595
  Travel and Other Direct Costs	$68,000	$68,000	$68,000	$204,000
  Indirect	$218,850	$225,095	$221,902	$665,847
  Total	$1,677,850	$1,725,725	$1,701,249	$5,104,824

  The case for the grant

  We think this is likely above our bar for cost-effectiveness

  Our best guess is that this grant is 16x as cost-effective as cash transfers. This estimate assumes that the PPM team can shift 10% of the $400 million portfolio of programming CHAI is a partner on per year to be 20% more cost-effective. Further detail on the key assumptions underlying this cost-effectiveness estimate are as follows.

  We estimate CHAI is a partner on around $400 million worth of programming per year, which represents the total scope of programming that the PPM team could influence in theory. We came to this estimate by including programs that CHAI directly implements (on average $200 million annually22 ) as well as an estimate of the larger umbrellas of funding that CHAI influences through technical assistance, which we conservatively guess would expand the pool of funding by a factor of 2.23 According to this threshold analysis, we would only need to assume that CHAI is a partner on $250 million worth of programming per year for the grant to look 10x (which we think is very likely to be true).

  We estimate that CHAI's PPM team actually influences 10% of that $400 million portfolio each year. Our best guess is that CHAI's PPM team actually influences 10% of that $400 million portfolio each year. For the PPM team to influence a program we have to believe:

  • i) the program is not already optimized for cost-effectiveness (e.g., in geographic or other targeting),
  • ii) CHAI has the ability to influence the program (e.g., the funder and other partners are open to this),
  • iii) the PPM team has the capacity to do the work (e.g., conduct the cost-effectiveness analysis), and
  • iv) the PPM team has the ability to identify areas for improvement

  On (i) the program is not already optimized for cost-effectiveness:
  CHAI told us they think it is highly unlikely that across their $200 million portfolio there is a lack of room for improvement in terms of cost-effectiveness. The intensive work of the incubator team with program and country teams shows that there haven't been lots of ready-made programs that could meet the GiveWell bar without additional refinements for cost-effectiveness. Without standardized program-wide cost-effectiveness systems, it seems highly unlikely that CHAI programs would already be fully optimized with little room left for improvement.

  On (ii) CHAI has the ability to influence the program:

  We think this is one of the more likely failure modes for this grant. However, our understanding is that CHAI will prioritize programs that will be pitched to donors who are more likely to be aligned with the value of cost-effectiveness and developing clear stakeholder engagement plans for overcoming potential barriers with less aligned partners. We expect to learn more about this through the course of this grant.

  On (iii) the PPM team has the capacity to work on enough programs to reach 10% of CHAI's total programming:

  CHAI shared that their programs can be very "lumpy" (i.e., a small number of programs comprise an outsized portion of the dollar value of the programming). This increases our confidence that a small team of around 5 staff (FTEs) would be able to get through enough investigations to influence $20 million (i.e. 10% of CHAI's portfolio) of direct CHAI programming per year.

  On (iv): the PPM team has the ability to identify areas for improvement:

  We think this is the other most likely failure mode for this grant. The PPM team's ability to be effective will depend largely on CHAI's ability to identify and onboard people who can effectively conduct cost-effectiveness analyses and identify strategic changes. CHAI is confident they can do this because of their experience hiring staff for the Incubator team.24

  Overall we think our best guess that the PPM team could influence 10% of CHAI's portfolio is reasonable. Additionally, according to our threshold analysis, we would only need to assume that the PPM team influences 6% of CHAI's portfolio per year for the grant to look 10x.

  For the programs that it influences, the PPM team causes a 20% improvement in cost-effectiveness. This is also a rough guess. A 20% improvement seems intuitively reasonable, particularly if a central element of the PPM team's work, in partnership with country and program leads, is to help choose which RFPs to respond to and which countries to target in multi-country grants (which can drive large differences in impact and cost-effectiveness).25

  According to this threshold analysis, we would need to assume that the PPM team causes a 13% increase in cost-effectiveness for the programs it influences for the grant to look 10x, but our best guess is that the PPM team can cause a 20% improvement.

  The benefits of the PPM team's work will last for 5 years, even though the funding for the team only lasts for 3 years. We believe that even if the PPM team disbanded after 3 years, the benefits would persist at least temporarily. This is because: i) we expect the PPM team's engagement with country and global program teams will lead to a spillover of skills and methods to assess cost-effectiveness, ii) country and global program teams may internalize the value of cost-effectiveness (not least given that they will be held accountable for the cost-effectiveness of their work for the 3 years of this grant), iii) it is possible that there is some path dependence in the countries or types of program CHAI works on.26

  According to this threshold analysis, we would only need to assume that the benefits last for 2.75 years after initial ramp-up for the grant to look 10x.

  Our cost-effectiveness estimate includes the downside adjustment of a 20% chance that the PPM team fails totally. This is a very subjective guess to account for the two main ways27 we think the PPM team could fail to have any impact:

  • i) if key personnel from the Senior Leadership or Leadership Council leave the organization, and with them the buy-in to enact the recommendations of the PPM team is lost, and/or
  • ii) if there is a lack of buy-in from "mid-level" staff in country and global program teams (e.g., those just below the program leads) and they still have influence on strategic direction despite buy-in from leadership.

  Our rough cost-effectiveness analysis suggests there is significant upside potential with this grant. There is substantial scale for the PPM team to be more successful than we expect, given the large portfolio of programming CHAI could influence. In our model, we include an optimistic scenario, in which the PPM team is able to influence 15% of a $500 million portfolio over a 6 year period, and cause an average 25% improvement in the programs that it influences. In that scenario, the cost-effectiveness of the grant would jump to 45x cash.

  We believe that this is a timely moment to institutionalize cost-effectiveness in a major global health implementer, given the level of buy-in from current senior leadership

  CHAI's CEO, Dr. Neil Buddy Shah, previously worked for GiveWell. We have a strong impression of several other members of CHAI's senior leadership who we believe are also strongly aligned with the importance of cost-effectiveness. This impression is in part based on our work together through the incubator. We think this grant is a good opportunity to capitalize on this time of strong alignment to set up supportive structures to make cost-effectiveness optimization a more formal part of CHAI's processes.

  The success of the GiveWell-funded Maximum Impact Incubator so far gives us some confidence that CHAI could achieve their goals on this work

  We made a $10.4 million grant to CHAI to create the incubator in 2022, and our current best guess is that it will achieve the goals it set out.28 The incubator has so far led to $23 million in programmatic grantmaking–a $15.1 million grant to strengthen and support a community-based tuberculosis household contact management program in two states in India and a $7.8 million grant to support the distribution of oral rehydration solution and zinc in Bauchi, Nigeria (as well as an RCT of this program). The incubator has also led to three pilot projects which may lead to further cost-effective funding opportunities and early stage reviews of approximately 20 additional programs (several of which could also lead to future cost-effective grants). This success increases our confidence that CHAI is an aligned partner who understands and can implement cost-effectiveness modeling to optimize programs for impact.

  Improving CHAI's ability to hire and retain key staff members could have positive spillover effects for GiveWell's past and potential future investments in the Maximum Impact Incubator.

  The portion of this grant directed at helping CHAI improve their systems for hiring and retaining staff ($1.2 million) should allow CHAI to hire and retain high performing team members for both its global and in country offices, as explained above. Having high quality team members in country and global programs teams could improve the quality of both current GiveWell-funded programs as well as the new programs that CHAI proposes to GiveWell through the incubator.

  Risks and reservations

  If key personnel leave the organization during the grant period, the likelihood of this grant's success may diminish.

  If key members of CHAI leadership who are strongly bought into GiveWell's cost-effectiveness approach were to leave in the next five years, the benefits of this grant are less likely to materialize. We have tried to build this into our cost-effectiveness estimate through a 20% probability of failure adjustment, but we may be underestimating this risk.

  We could be overestimating the chance of success of the PPM team.

  The success of the PPM team will depend on several hard-to-predict factors such as whether other funders and governments are amenable to the program adjustments the team may suggest. Additionally, we are uncertain how difficult it might be for CHAI to hire strong recruits for this team, because this team will require a fairly specific skill set.29 These hard-to-predict factors have the potential to impede the theory of change that the case for this grant rests upon.

  We have conducted a series of threshold analyses (outlined in our section on cost-effectiveness above) which suggest that even with fairly conservative assumptions for the key parameters in our model the grant looks cost-effective. However, we have considered each parameter in isolation. If we have been overly optimistic across several of these key parameters at once, the cost-effectiveness of this grant could be below our bar. For example, if the PPM team only influenced 6% of CHAI's portfolio and only improved cost-effectiveness by 13% for those programs (the 10x threshold values for those two parameters), then cost-effectiveness would fall to 6x cash.30

  Even if the PPM team causes CHAI's programs to be more cost-effective, this may not counterfactually improve the net cost-effectiveness of programming across implementers.

  It is plausible that if CHAI chooses to implement a more cost-effective program, this may just cause an alternative implementer to implement the less cost-effective program that CHAI would have counterfactually run (and vice versa).

  This concern seems most plausible in cases where the PPM team's work informs how CHAI responds to RFPs (see footnote),31 though the same logic could apply to other ways the PPM team could improve CHAI's cost-effectiveness if we assume there is a fixed total amount of philanthropic funding (i.e., assuming that funding decisions across implementers are zero-sum). Our CEA does not capture this concern because we have modeled the effect of the PPM team on the cost-effectiveness of CHAI's portfolio only.

  While this is a reservation, this risk would be greatest in a world where there is a fixed set of program opportunities in global health, all of which are funded. In that world, CHAI prioritizing more cost-effective programs would not lead to any real impact over the counterfactual. In practice, we think it is likely that the PPM team identifies opportunities for new cost-effective programs (either in terms of geographic/demographic targeting or program design) that otherwise would not have been identified either by CHAI or others (i.e., the PPM team expands the set of program opportunities).

  It is also possible that by identifying these new programs the PPM team enables CHAI to crowd in new funding to global health (i.e., expand the pool of funding), though we are less certain about this.

  Another funder might have funded this if we didn't

  At the end of June 2024 when CHAI pitched this to us, they were also fundraising from other donors for a range of organizational strengthening activities. The two activities we are funding through this grant are among CHAI's highest funding priorities,32 and so if they are able to secure additional funding then our grant would effectively be used for the lower priority organizational strengthening activities on their list.

  We did not try to model how cost-effective those alternative activities would be and so have not made an adjustment for this in the model. CHAI has found that most donors are unwilling to fund organizational support for an organization of CHAI's size, so we think the risk of this is relatively low.33 However, this could mean our BOTEC may be slightly overestimating cost-effectiveness.

  We are unlikely to know whether this program turned out to be a cost-effective use of GiveWell funds

  CHAI does not have a good sense of the current cost-effectiveness of its overall portfolio (getting a better understanding of this is part of the purpose of the PPM team). As a result, we will not have a baseline measure of cost-effectiveness to compare to (e.g., in a pre-post evaluation design).

  However, at the end of the grant period, we will ask for a set of quantitative and qualitative indicators from CHAI to check whether steps in the theory of change materialized. We are also likely to separately commission an external evaluator to conduct a series of interviews with key CHAI team members (e.g., CEO, country directors, PPM team lead, other PPM team members).

  While these will serve as helpful indicators, they will not tell us whether the program caused an improvement in the cost-effectiveness of CHAI's work over what would have happened in a counterfactual scenario without this program.

  Work to support the PPM team could detract from the incubator team's time

  We have a slight concern that the incubator team could get drawn into spending time supporting the PPM team, given the familiarity they have built with GiveWell's cost-effectiveness modeling. CHAI told us that no members of the incubator team would directly work in the PPM team, but CHAI's Vice-president of Innovation (who leads the incubator) will be part of a steering committee which supports the PPM team.34

  Plans for follow up

  • We will have biannual check-ins with CHAI, during which time we will ask about updates on the proposed activities as well as any progress on the research questions outlined in our learning agenda.
  • During year 3 of the grant (which starts ~January 2027) we will plan to hire an external evaluator (someone with either qualitative research experience, or historian/investigative journalist skillset) to conduct an evaluation of how the grant went.
  • End of the grant (December 2027):
    • We will ask CHAI to provide responses to the research questions listed in our learning agenda.
    • We will review the analysis conducted by the external evaluator on how the grant performed.

  Predictions relevant to the grant

  For this grant, we are recording the following forecasts:

  Likelihood	that… (prediction).	By time
  60%	CHAI successfully hires a fully dedicated PPM team, consisting of roughly the equivalent of 5 FTEs, including a director and several associates/managers.	by June 30, 2025
  90%	CHAI successfully hires a new HR manager focused on new talent partnerships and external recruitment strategies	by June 30, 2025
  50%	CHAI's PPM team conducts cost-effectiveness analysis on programs whose $ value exceeds 10% of CHAI's overall program expenditure during the grant period (ending December 31, 2027)	December 31, 2027
  10%	CHAI's PPM team conducts cost-effectiveness analysis on programs whose $ value exceeds 20% of CHAI's overall program expenditure during the grant period (ending December 31, 2027)	December 31, 2027
  10%	GiveWell chooses to investigate a renewal grant for organizational development at CHAI by December 31, 2027	December 31, 2027

  Our process

  • We reviewed the following information sent by CHAI:
    • A proposal
    • A lookback at the performance of the incubator
    • A BOTEC they produced for this funding opportunity
    • A comparison of CHAI non-programmatic expenses to other large international NGOs using the US tax Form 990.
    • An overview of CHAI's program breakdown by program vertical.
    • A case study in how CHAI approaches program optimization for cost-effectiveness.
    • A budget
  • We had five follow-up meetings with CHAI leadership and staff.
  • We drafted our own lookback on the CHAI incubator and BOTEC
  • We conducted internal peer reviews.
  • We did not have any external conversations with parties other than CHAI.

  Relationship disclosures

  Dr. Neil Buddy Shah was hired in April 2022 as CHAI’s CEO. Previously, he was GiveWell’s Managing Director.

  Sources

  Document	Source
  CHAI, About Us	Source (archive)
  CHAI, Homepage	Source (archive)
  CHAI, Organizational strengthening proposal, June 2024 (unpublished)	unpublished
  GiveWell and CHAI, CHAI organisational development grant notes, 2024 (unpublished)	unpublished
  GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)	unpublished
  GiveWell, Blog - Neil Buddy Shah has been appointed CEO of the Clinton Health Access Initiative, April 2022	Source
  GiveWell, CEA of a community-based TB HCM program in India, 2024	Source
  GiveWell, CHAI organizational development grant learning spreadsheet, 2024 (unpublished)	unpublished
  GiveWell, CHAI PPM grant BOTEC, 2024	Source
  GiveWell, Clinton Health Access Initiative – CHAI Incubator (August 2022)	Source
  GiveWell, Clinton Health Access Initiative — Oral Rehydration Solution and Zinc Distribution in Bauchi, Nigeria (September 2023)	Source
  GiveWell, Clinton Health Access Initiative (CHAI) — Community-Based Tuberculosis Household Contact Management (June 2024)	Source
  GiveWell, Internal forecasts	Source
  GiveWell, Notes from Meeting with CHAI, August 6, 2024 (unpublished).	unpublished
  GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).	unpublished
  GiveWell, Notes from Meeting with CHAI, July 9, 2024 (unpublished).	unpublished

  • 1

    "The position is to hire a new manager position on the HR team focused on new talent partnerships and external recruitment strategies. While the individual will work with the recruiters, it’s an adjacent role to help us develop strategic partnerships (e.g., with universities in our program countries to enhance our talent pipeline)". GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)

  • 2

    Our estimate of the value per dollar donated to cash transfers is out of date as of 2024. We are continuing to use this outdated estimate for now to preserve our ability to compare across programs, while we reevaluate the benchmark we want to use to measure and communicate cost-effectiveness.
    For more on our updated assessment of the impact of unconditional cash transfers, see this page.

  • 3
    • "CHAI was founded in 2002 with a transformational goal: help save the lives of millions of people living with HIV/AIDS. Today, we support government priorities across many areas, including other infectious diseases, non-communicable diseases, women and children’s health, health workforce and financing, assistive technologies for disabilities, and the intersection of climate and health." CHAI, About Us
    • “We now work with governments, donors, and other partners on more than 20 programs in over 35 countries across Africa, Asia, and Latin America.” CHAI, Homepage

  • 4

    We expect that we would hire either a qualitative researcher or someone with a historian/investigative journalist skillset to investigate some of the key parameters relevant to the cost-effectiveness of this grant (ex. CHAI's ability to hire for the PPM team) as well as other factors not directly relevant to cost-effectiveness (ex. key personnel risk). We guess this will cost something in the region of $50k. GiveWell, CHAI organizational development grant learning spreadsheet, 2024 (unpublished)

  • 5

    'High level budgets', p.9. CHAI, Organizational strengthening proposal, June 2024 (unpublished)

  • 6

    CHAI staff members confirmed they have already hired a manager for the PPM and would use additional funding to hire five analysts to staff this team. GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).

  • 7

    'High level budgets', p.9. CHAI, Organizational strengthening proposal, June 2024 (unpublished)
    - The money for consultants would be used for: "interns from consulting firms/partner orgs…, technical experts for impact evaluations (either internally or externally), and/or covering partial staff time for select senior management individuals". GiveWell and CHAI, CHAI organisational development grant notes, 2024 (unpublished)

  • 8
    • CHAI's COO is Joshua Chu. The 6-person Senior Leadership team consists of: Dr. Neil Buddy Shah (CEO), Rasha Hibri (CFO), Joshua Chu (COO), Samantha Diamond (VP Health System and NCDs), Dr. Olufunke Fasawe (VP Integration), and David Ripin (EVP Infectious Diseases, Chief Scientific Officer) See CHAI, About Us.
    • CHAI recently created an 17-person Leadership Council (as part of the development of the new strategy), which includes the 6 people from Senior Leadership team plus 11 additional team members, including Neel Lakhani and Justin Cohen (who GiveWell interacts regularly with through the Incubator work). See CHAI, About Us.

  • 9

    GiveWell, Notes from Meeting with CHAI, July 9, 2024 (unpublished).
    

  • 10

    GiveWell, Notes from Meeting with CHAI, July 9, 2024 (unpublished).

  • 11
    • "CHAI proposes establishing impact measurement systems that will enable more strategic decision-making throughout all levels of the organization.", p.4. CHAI, Organizational strengthening proposal, June 2024 (unpublished)
    • "The mechanism will provide the senior leadership team with necessary information to make strategic decisions to maximize the aggregate impact of CHAI's portfolio. In practice, we believe this could include the outputs of the PPM team (including cost-effectiveness analysis) being used to track performance in quarterly and annual review meetings (which already take place across various global and country programs)." GiveWell and CHAI, CHAI organisational development grant notes, 2024 (unpublished).

  • 12

    GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).

  • 13

    GiveWell, Notes from Meeting with CHAI, August 6, 2024 (unpublished).

  • 14

    GiveWell, Notes from Meeting with CHAI, July 9, 2024 (unpublished).

  • 15
    • "The position is to hire a new manager position on the HR team focused on new talent partnerships and external recruitment strategies. While the individual will work with the recruiters, it’s an adjacent role to help us develop strategic partnerships (e.g., with universities in our program countries to enhance our talent pipeline)". GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)
    • 'High level budgets', p.9. CHAI, Organizational strengthening proposal, June 2024 (unpublished)

  • 16

    CHAI's potential activities with the remainder of funds include developing an organization-wide performance management program, implementing professional development and learning programs, enhancing internal mobility, and expanding recruitment outreach programs. CHAI, Organizational strengthening proposal, June 2024 (unpublished), p. 7

  • 17

    GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)

  • 18

    We have spent little time investigating this because: i) it is a relatively small amount of funding, ii) we are happy to defer more to their judgment on the best way to improve retention and hiring than we would with other types of expenditure (because we think they are much better placed to make those calls, given the internal context they have).

  • 19

    One example CHAI gave here is that they do not have a formal benchmarking process, meaning that different teams are benchmarking salaries in inconsistent ways. Note: we are not sure precisely how software would help them to do that process more consistently. GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished)

  • 20

    GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).

  • 21

    "The recruitment activities…will support all geographies - both in-country offices and global staff." GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)

  • 22
    • CHAI calculated this by taking the average of their revenue from 2019-2021: "The $200M figure was meant to be an approximate, rounded number for CHAI’s revenue. We have confirmed with accounting that this represents annual (not multi-year) revenue." GiveWell and CHAI, CHAI organizational development grant notes, 2024 (unpublished)
    • Not all of CHAI's revenue is spent on programmatic expenses. However, the vast majority (90+%) is used for programmatic expenses.

  • 23

    CHAI's own best guess is that including funds they influence through TA would expand the total funding pool by a factor of 4, so we have made a conservative estimate here.
    As a case study, we looked at the proportion of CHAI's technical assistance expenditure for the household contact management for TB program we recently funded as a proportion of the total expected budget for this program (including spending by government and other actors). We calculated an external reach multiple of 1.3.

    • Calculations as follows: CHAI costs = $7.6m (though this includes $240k in incentives for private sector health workers, which is unusual for CHAI), government costs = $1.5m, commodity costs = $640k. Therefore total expenditure = $9.7m, of which $7.4m could be considered "typically" CHAI (subtracting private sector incentives). 9.7/7.4 = 1.3. See column B here.

    We included a small upwards adjustment to account for other higher leverage work in CHAI's portfolio.
    CHAI's work broadly splits across a spectrum between the following categories:
    1) Implementation programs where they are typically a TA partner, or
    2) Deliberately high-leverage programs where they are advising other organizations (e.g., governments or other funders) on how to optimize their expenditure.
    The second bucket is likely much higher leverage than the first bucket. However, we suspect that the second bucket of work is less pertinent to this grant because CHAI is more likely to be already doing the type of cost-effectiveness optimization work that the PPM team will be tasked with given that the whole purpose of those projects is to optimize other organizations' expenditure.
    We have therefore benchmarked our estimate ($400 million) of the scope of programs CHAI supports, on the first category of program, with a small upwards adjustment to account for the possibility that the PPM team's work could affect some of the second category programs, as well.

  • 24

    "As for ability, we will hire and train people to run standardized CE analyses. They will work with individual program teams (who understand the program nuances) to jointly develop approaches to enhancing cost effectiveness. The success of the incubator team speaks to CHAI’s ability to source and train talent to understand and execute high fidelity, cost-effectiveness-centered investigations." CHAI, email correspondence with GiveWell, September 2024 (unpublished).

  • 25

    Differences between countries, particularly regarding disease burden and existing coverage rates, can drive very large differences in cost-effectiveness of programs. While we caution against taking these estimates too literally, it does give us reasonable confidence that 20% improvements in cost-effectiveness are attainable with the proposed activities of the PPM team. For illustration, early stage cost-effectiveness estimates for programs CHAI has pitched us through the incubator as well as cost-effectiveness estimates for our top charities can often vary by 10-20x (i.e. 1000-2000%) just based on different geographies.

  • 26

    For example, the PPM team causes CHAI to win more work in a country with a high burden of childhood disease, and beyond the end of this grant period that work leads to other business development opportunities in the same (highly cost-effective) geography.

  • 27

    Note: Other ways that the PPM team could fail to be impactful (e.g., lack of capacity of the PPM team) are built into the model assumptions described earlier.

  • 28

    The goal of the Incubator grant was to identify at least 5 years of $30 million annual room for more funding at a minimum of 12x cash by 2025, or the equivalent (e.g., higher cost-effectiveness but lower room for more funding, leading to the same total units of value). See the grant page for more details.

    At the time of making this grant, the initial Incubator grant period was two-thirds of the way through.

  • 29

    This skill set includes the ability to produce cost-effectiveness models but also not losing sight of factors that matter in the real world and are hard to model. GiveWell's own recruiting experience has shown us that it can be difficult to hire for this type of skill set.

  • 30

    See this analysis in our model here.

    • We have also done a sensitivity analysis where we reduce our best guess for each of the four key parameters by 10% (duration of impact, $ size of CHAI portfolio reach, % of programs influenced by PPM team, % improvement). Doing this, cost-effectiveness falls to 10x.

  • 31

    Suppose CHAI was going to respond to (and win) RFP A, but now responds to (and wins) RFP B instead because it thinks there is more room for a cost-effective program based on the work of the PPM team. Whilst that will increase the cost-effectiveness of CHAI's programming, it is not clear that there is any counterfactual impact for the world - since another organisation now wins the work for the not-very-cost-effective RFP A (so that work still goes ahead under the intervention), whilst in the counterfactual another organisation would have done the cost-effective program responding to RFP B.

  • 32

    GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).

  • 33

    GiveWell, Notes from Meeting with CHAI, July 9, 2024 (unpublished).

  • 34

    GiveWell, Notes from Meeting with CHAI, July 15, 2024 (unpublished).