Population Services International (PSI) - 2007 review

A note on this page's publication date

The content on this page has not been recently updated. This content is likely to be no longer fully accurate, both with respect to the research it presents and with respect to what it implies about our views and positions.



We have published a more recent review of this organization. See our most recent review of PSI.

Table of Contents

What do they do?

PSI sells and markets a variety of products intended to combat disease, as well as (in some cases) improve nutrition and prevent unwanted pregnancies. The following table gives a relatively complete picture of PSI's activities based on 2005 expenses; links go to relevant section of our problems and solutions overview. Data was provided by PSI, using materials that cannot be posted publicly but are available upon request (see below for details).

Activity Total budget Portion of PSI's budget this accounts for Primary product % of budget spent on primary product
HIV/AIDS prevention $143,386,919 50% Condoms (and other safe sexual behavior) 80%
Malaria prevention $65,432,170 23% Insecticide treated nets (ITNs) 85%
Pregnancy prevention $32,566,327 11% Contraception 100%
Diarrhea prevention $6,287,377 2% Water purification 100%
Malaria treatment $3,574,773 1% Prepacked therapy (Drugs) 100%
Diarrhea treatment $2,123,221 1% Oral Rehydration Salts 100%
Maternal/perinatal health $1,225,632 1% Multivitamins 100%
Other $33,009,180 11% Includes tuberculosis treatment and Safe Delivery Kits  
Total expenses $287,605,599 100%    

HIV/AIDS prevention - marketing condoms and promoting abstinence, monogamy, and condom use - accounts for about 50% of PSI's expenses. Combined with marketing insecticide treated nets (ITNs), we have about 73% of PSI's budget; so we focus on these two activities in later sections of this review.

It's not practical for us to examine the rest of PSI's products in depth, but generally, these activities match with our understanding of the more cost-effective ways to help people, addressing many of the biggest problems for those in Africa: maternal and perinatal death (which PSI aims to help prevent through family planning), diarrhea (which PSI fights with oral rehydration therapy, an extremely cheap and simple way to save lives), etc. We do not have a strong understanding of the Safe Water System, which promotes products that people can add to water to make it safe; we haven't seen this strategy employed in any other context, but we do not focus on it since it is a small part of PSI's budget (for the record, PSI does have materials available gauging this strategy's impact).

Why does PSI sell materials instead of giving them out?

PSI is a charity; the reason it charges for products and services is not to profit, but rather to distribute them more efficiently. As shown below, revenues only end up recovering around 15% of PSI's costs.

Through informal conversations with them and our own reasoning, we'd summarize the logic behind selling (as opposed to giving) as follows:

  • Selling may help products get into the hands of people who will use them. White Man's Burden (review with the relevant excerpt is available here) gives the anecdote of free bednets' being used as fishnets and wedding veils. Selling materials may help prevent this phenomenon and thus distribute materials more efficiently.
  • Purchasing goods and services (rather than receiving them for free) may actually make a person more likely to use and/or value them. The logic here is psychological and informal: people tend to attribute value to what they pay for, and may thus be more likely to save it and use it appropriately. (This point relates to the effect of selling on a particular individual, whereas the above point relates to the effect on distribution across individuals.)
  • Selling helps reduce costs. This is not a major factor (PSI recoups about 10-15% of its costs, as shown below), but by charging people what they can easily pay, PSI may end up spending less to serve more.

On the other hand, we can see many reasons to give rather than sell, including:

  • Selling may prevent products from reaching those most in need. Though the examples we've seen (detailed below) imply that PSI's pricing isn't too steep for most customers, it may be too steep for some of them.
  • Selling may give people one more reason to choose short-term pleasure over long-term safety. For example, condoms already have a reputation for reducing the pleasure of sex; giving them away for free may reduce hesitation to use them, and send a stronger message that the giver is trying to help, not profit.
  • Giving may have beneficial side effects, such as reducing opportunities for infection. For example, insecticide-treated bednets kill mosquitoes; distributing enough of them could significantly reduce the risk even for those who don't use nets.

The merits of selling vs. giving are often debated. We don't take a general position on which is better. Rather, we evaluate what we know about individual charities' abilities to save lives consistently and cost-effectively. We believe (as should emerge from this review as a whole) that PSI is strong on these counts.

PSI's general strategy

Across its strategies, PSI employs the following general approach (detailed more fully in Attachment B-18, Pgs 4-7):

  • Strategic pricing. One example of how PSI sets its price comes from its ITN program in Madagascar, where it conducted a survey on how much people would be willing to pay for ITNs. PSI set its price at a level where close to 100% of respondents would pay (Attachment B-8, Pg 17). Though its approach in other cases isn't necessarily identical, this represents an approach committed not to maximizing profit, but rather to reaching as many people as possible while still realizing the benefits of selling (detailed immediately above).
  • Promotion. PSI states an explicit strategy it uses to approach promotion. It starts with a goal - for example, preventing HIV/AIDS for a specific, high-risk target group of people in a certain area (Attachment B-18, pg 5). It then surveys the target group to assess which people practice risky behaviors (i.e., don't use a condom); identifies the significant differences between those who do and do not practice these behaviors (Attachment B-18, pg 10); and uses this information to plan its marketing campaign around the specific attitudes that correlate to healthy behavior. PSI uses two types of promotional strategies (examples of these strategies are described in TRaC Studies conducted by PSI, in Attachments B-6 - B-17, available on PSI's website):
    • Mass media. PSI advertises on TV, radio, in newspapers, and on billboards. Some examples of advertisements they use are available on their website, here.
    • Interpersonal communication (IPC). PSI trains individuals to talk to their friends/people they know about, for example, safe sex, condom use, HIV testing, etc. They also provide those they train with materials that support their ability to convince others. See below for examples of - and data on - this strategy. A general description of the strategy is also available, on PSI's website.
  • Continuing monitoring and evaluation. Part of PSI's model is continued surveys and examinations to see whether its plans are working as well as possible. Attachment B-30, Pg 7 lays out PSI's organization-wide approach to monitoring, with two key components:
    • TRaC (Tracking Results Continuously) reports. These are surveys on the attitudes and behaviors that PSI seeks to change, such as condom and bednet utilization.
    • MAP (Measuring Access and Performance) reports. These are reports on whether PSI's products are sold and advertised as intended (an example is given below, in discussing the Mozambique project).

    Despite PSI's stated goal of conducting TRaC and MAP reports comprehensively, its coverage is far from complete. Attachment B-31 shows its "scorecard," summarizing what monitoring it has and hasn't done; when looking for these reports on its research website, we can only find some of them (accounting for about 10% of its total activities). This raises the question of whether there may be publication bias at work. Still, while many of our applicants state a commitment to organization-wide monitoring and evaluation, PSI is one of the few that demonstrates both a comprehensive strategy for doing so and actual execution of reports for a large portion of its activities.

The following table, from PSI's 2003-2005 Unit Cost Analyses, breaks down PSI's organization-wide budget in a way that illustrates how much it spends on each of the components above. Note that the cost of goods sold (condoms, bednets, etc.) accounts for only about 25-30% of total expenses, when netted with revenues.
Total net costs Gross cost of goods sold Revenues Promotion: advertising Promotion: interpersonal communication Research & evaluation National staff Local staff Other Overhead (travel, furniture, equipment, consultants)
2003 $172,766,493 44% -13% 10% 9% 2% 7% 14% 27%
2004 $200,166,248 38% -14% 10% 7% 2% 13% 16% 27%
2005 $256,635,529 42% -14% 8% 7% 2% 13% 16% 26%

Regions

PSI mostly serves Sub-Saharan Africa, but also does significant work in Southeast Asia, Central Asia, Eastern Europe and Latin America. Activities in both Sub-Saharan Africa and Southeast Asia have grown significantly in recent years, and we expect new money to go towards programs in those regions. (All information in the tables below comes from PSI's Unit Cost Analyses - see below.)

Year Total expenses Sub-Saharan Africa South-East Asia Central Asia Central America Eastern Europe South America
1997 $52.8M 63% 12% 17% 7% 1% 0%
1998 $60.4M 64% 11% 13% 8% 2% 2%
1999 ? ? ? ? ? ? ?
2000 $85.3M 67% 13% 6% 9% 3% 1%
2001 $112.8M 68% 13% 6% 7% 4% 1%
2002 $138.7M 71% 12% 6% 6% 4% 0%
2003 $195.4M 69% 14% 7% 6% 4% 0%
2004 $227.4M 73% 13% 6% 3% 4% 1%
2005 $292.9M 72% 16% 4% 4% 3% 1%

Does it work?

Most of the items PSI markets, as discussed in our problems and solutions overview, can be expected to save lives if they are both acquired and used by people who need them. In this section, we examine the evidence that PSI is making these two things happen, focusing on its largest two programs: HIV/AIDS prevention and insecticide treated net (ITN) marketing. In both cases, we find reason to believe that (a) PSI is a leading, if not the leading, supplier in the regions it serves; (b) the people served likely use its products. We conclude with reasonable, though not complete, confidence that PSI's activities accomplish their goal of saving lives. Details follow.

PSI as a major supplier

The United Nations Population Fund estimates that 2.1 billion condoms were provided by "by bilateral donors, UNFPA and social marketing organizations" to poor countries worldwide in 2004; that same year, PSI sold 850 million condoms (Attachment B-4, Pg 4). This implies that PSI alone accounted for over 1/3 of aid-based condom distribution worldwide.

Based on this observation and the examination of more specific projects (immediately below), we believe that in its regions of focus, PSI is a leading, if not the leading, supplier of the products it markets. We give two region-specific examples, using projects that are particularly well documented: Mozambique, where PSI markets condoms (and promotes safe sexual behavior) in high-risk areas, and Madagascar, where it markets insecticide-treated nets (ITNs).

Mozambique: PSI's role in supplying condoms

As Attachment B-9, pg 1 states, PSI operates in districts of Mozambique that were recommended by the Mozambique Ministry of Health as being "high risk" in terms of HIV/AIDS; it sells condoms and uses both advertising and interpersonal communication to promote safe sexual behavior (not just condom use but also abstinence and monogamy).

The table below shows the total number of condoms sold, the estimated number of urban males aged 15-49 in Mozambique as a whole, and the implied number of "condoms per [relevant] person" sold by PSI from 1997-2006. Data on condoms sold comes from Unit Cost Analyses and Sales Reports (see below). Population and demographic information comes from Attachment B-21.

Year Condoms sold # of 15-64 yr old, urban males Condoms / targeted person
1997 10,412,328 1,017,048 10.2
1998 8,672,442 1,080,337 8.0
1999 9,858,401 1,143,625 8.6
2000 8,947,604 1,206,913 7.4
2001 12,036,632 1,223,740 9.8
2002 14,478,973 1,238,679 11.7
2003 15,264,124 1,104,229 13.8
2004 16,012,202 1,188,406 13.5
2005 18,544,874 1,225,992 15.1
2006 21,888,470 1,243,668 17.6

Note that we are dividing condoms sold by the total population of urban males aged 15-49 - including those who are outside PSI's districts of focus and those who don't use condoms at all. If we had the data to adjust for these factors, we would guess it would mean a substantially higher "condoms per person" figure. Knowing how understated these numbers are, we find them to provide some evidence that PSI is a highly significant provider of condoms in its areas of focus.

Further evidence comes from PSI's direct surveys of condom coverage:

  • PSI conducted a sample of 21 random zones in its areas of focus, and found that PSI condoms were available in at least 2 outlets in 17 of them (Attachment B-22, pg 12); by contrast, it appears (though PSI doesn't explicitly point to this) that non-PSI condoms were only available in 1 of the 21 zones surveyed (Attachment B-22, pg 12). This survey was conducted as a check on outlets' compliance with PSI's policies; it also found that all outlets that sold PSI condoms sold them for the recommended price, though they were often not as visible as desired and not accompanied by the promotional materials that PSI encouraged (Attachment B-22, pg 12).
  • In 2005, PSI measured condom coverage in the Beira district, the 2nd largest urban district in Mozambique (Attachment B-22, pg 6). It found that condoms were sold in 11 of the 19 residential neighborhoods it evaluated, while non-PSI condoms were only available in one (Attachment B-22, pg 8). As in the survey above, all outlets that sold PSI condoms sold them for the recommended price, though they were often not as visible as desired and not accompanied by the promotional materials that PSI encouraged (Attachment B-22, pg 12).

From this, we take that PSI has sold a large number of condoms - and likely has been a leading supplier of them - in its regions of focus in Mozambique.

Madagascar: PSI's role in supplying insecticide treated nets (ITNs)

As Attachment B-8 details, PSI markets insecticide-treated nets (ITNs) in Madagascar. It also promotes ITN use through television and radio ads, as well as mobile video units (Attachment B-8, Pg 7); advertising specifically targets pregnant women and caregivers to children under 5 (Attachment B-8, pg 1), the people who are generally most at risk from malaria.

The following table is taken from PSI's Unit Cost Analyses and Sales Reports (see below).

Year Gross cost # sold Gross cost per net sold
2001 $305,844 40,963 $7.47
2002 $951,646 104,848 $9.08
2003 $1,607,457 139,246 $11.54
2004 $2,434,969 335,270 $7.26
2005 $4,104,939 577,740 $7.11
2006 $5,657,091 796,194 $7.11

According to the World Health Organization, long-lasting ITNs and Olysets - 50% of the ITNs PSI sells (see Attachment B-9, Pg 28) - last between 4-5 years without needing to be retreated by insecticide. (Regular ITNs last the same amount of time but require retreatment.) In the last three years, PSI has sold 1.7 million ITNs, implying about 1.7 million direct or indirect customers - in a country with a total population of 20 million people, not all of whom are at risk from malaria or interested in owning an ITN.

We did a rough estimate of how many people in Madagascar actually gained ownership of an ITN over this time period, using PSI's survey data from Attachment A-12. This data estimates that 45% of households owned an ITN in 2006, up from 22% in 2004; combined with PSI's estimates of the total number of households and likely ITNs per household (Attachment A-11, Pg 42), we come up with about 3 million ITNs owned in 2006, vs. about 1.5 million owned in 2004. Even if all people who owned an ITN replaced it over this period, that implies that PSI sold about half of all the ITNs sold in Madagascar.

Utilization of PSI's products

It isn't enough to sell materials; relevant people must use them in order for PSI's activities to be saving lives. As discussed above, part of PSI's approach is systematic use of surveys to track changes in behavior; not all of the surveys it refers to are available, but from what we've seen, we have reason to believe that PSI's products are being used. We first look at Mozambique and Madagascar, the two countries described above, and then give a general overview of the other survey data we've seen.

Mozambique: changes in condom use

PSI's survey data from Mozambique indicate large changes in condom utilization in the areas it works in - much larger than across Africa as a whole. Given PSI's large role in these areas (discussed above), we're inclined to believe that PSI was largely responsible for increased condom use in high-risk areas, and therefore saved lives from sexually transmitted diseases. We don't know whether it also increased the practice of other safe behaviors (abstinence and monogamy). Details follow.

The table below summarizes PSI surveys on condom use in Mozambique from 1996, 2001 and 2004, according to PSI surveys. Between 340 and 830 people responded to each question. PSI did not provide the exact wording of the questions asked. (This data is taken from Attachment B-9, Pg 6 (table) and Pg 22 (chart)).

Survey item 1996 2001 2004
Condom use at last sex with a regular non-marital non-cohabiting partner 25% 30% 63%
Condom use at last sex with a non-regular partner 25% 43% 68%
Consistent condom use with a regular non-marital non-cohabiting partner - - 43%
Consistent condom use with a non-regular partner - - 52%
Condom use at last sex with a regular partner when the male has 2+ partners - 28% 59%
Condom use at last sex with a non regular partner when the male has 2+ partners - 43% 69%

We believe that it is somewhat dangerous to trust survey data about sexual activity, particularly questions about "consistent condom use" (the more specific question, "Did you use a condom in your last sexual encounter?" is less vague and easier to answer concretely). However, across the board the surveys imply large increases in condom use.

We examined the USAID Demographic and Health Surveys to get some sense of how much of this change was simply the result of the general environment (i.e., condom use increasing across Africa as a whole). While these surveys do show that condom use increased across Africa, the changes are not nearly on the scale noted above. Data from these surveys is taken from 28 countries, 16 of which offered condom use data for 2 or more years, and is available at the HIV/AIDS Survey Indicators Database.

Men reporting condom use at last sexual activity:

Country Time period Change
Benin 1996-2001 9%-16%
Burkina 1999-2004 21%-27%
Cameroon 1998-2004 16%-30%
Ethiopia 2000-2005 5%-4%
Ghana 1998-2003 15%-18%
Guinea 1999-2005 14%-17%
Kenya 1998-2003 21%-18%
Malawi 2000-2004 14%-15%
Mali 1995-2001 10%-9%
Mozambique 1997-2003 6%-12%
Nigeria 1999-2003 14%-16%
Rwanda 2000-2005 6%-5%
Tanzania 1996-2004 11%-21%
Uganda 1995-2004 9%-15%
Zambia 1996-2001 20%-19%
Zimbabwe 1994-2005 28%-24%
Average change (arithmetic) 1994-2005 3%
Largest change 1994-2005 14%

Men reporting condom use at last higher-risk sexual activity:

Country Time period Change
Benin 1996-2001 21%-31%
Burkina 1999-2004 57%-69%
Cameroon 1998-2004 29%-54%
Ethiopia 2000-2005 30%-51%
Guinea 1999-2005 33%-39%
Kenya 1998-2003 44%-47%
Malawi 2000-2004 39%-47%
Mali 1995-2001 36%-33%
Rwanda 2000-2005 51%-39%
Tanzania 1996-2004 32%-51%
Uganda 1995-2004 36%-52%
Zambia 1996-2003 40%-42%
Zimbabwe 1994-2005 59%-71%
Average change (arithmetic) 1994-2005 9%
Largest change 1994-2005 25%

PSI's regions of focus, drawn from a smaller but still significant sample, show changes closer to 40%. This difference could still reflect many factors independent of PSI, and is not conclusive. However, we believe that this comparison gives some reason to believe that PSI's large-scale activities impacted behavior, through some combination of changing attitudes and making condoms more available.

Madagascar: utilization of insecticide treated nets (ITNs)

PSI conducted surveys in both 2004 and 2006 to assess utilization of ITNs in Madagascar. The surveys selected a random set of households across all the "Fokontanys" (districts in Madagascar) where PSI worked (methodology in Attachment B-8, pg 25) and questioned pregnant women and caregivers to children under five. The sample size was 1367 in 2004 and 1553 in 2006.

Survey item 2004 2006
Owned at least one untreated mosquito net or insecticide treated net (ITN) 66% 64%
Slept under untreated mosquito net or ITN last night (among pregnant women) 47% 49%
Slept under untreated mosquito net or ITN last night (among children under five years of age) 52% 55%
Owned at least one ITN 22% 45%
Slept under ITN last night (among pregnant women and children under five years of age) 16% 38%
Slept under ITN last night (among pregnant women) 12% 28%
Slept under ITN last night (among children under five years of age) 16% 38%
Utilized % Owned: Untreated nets 75% 81%
Utilized % Owned: ITNs 67% 76%

Note that this data implies that the change in ownership of nets was dominated by a change in insecticide-treated nets (ITNs), the kind PSI sells, while ownership of untreated nets remained relatively constant. This is consistent with (though again, not implying) PSI's leading role in the change in ITN ownership.

We would guess that surveying people about last night's activities, as PSI did, likely provides more accurate data than surveying them about their habits would, since the former is less vague and easier to recall. Having slept under a bednet last night is not the same as regularly sleeping under a bednet, but as we explain in a later section, using different possible interpretations of this number has virtually no effect on our estimate of the number of lives saved.

The larger point is that the above data suggests that owners of ITNs in Madagascar largely use them, and that utilization did not fall as ownership rose - implying that some combination of PSI's selling and promotion resulted in more people's using ITNs.

More data on PSI's interpersonal communication strategy

Aside from looking in depth at the two regions above, we also examined several available evaluations of PSI's interpersonal communication strategy (IPC), which consists of small-group discussions aimed at changing behavior. We find the evidence for this strategy's effectiveness to be moderately positive but not strongly convincing.

We found seven PSI reports that directly studied the IPC strategy and surveyed people's reports of condom use over time. These reports, Attachments B-10 through B-16, are summarized below.

Location Survey 1 date Survey 2 date Survey 3 date Survey 1 sample size Survey 2 sample size Survey 3 sample size Survey 1 Used condom last time Survey 2 Used condom last time Survey 3 Used condom last time Survey 1 consistent condom use Survey 2 consistent condom use Survey 3 consistent condom use
Laos 11/01/04 06/01/06 - 288 324 - 33% 66% - - - -
India 06/24/05 06/26/05 2006 3370 3711 5888 56% 65% 75% 51% 55% 60%
China 08/01/04 02/01/06 - 668 547 - - - - 20% 29% -
Romania 12/01/05 04/01/06 3/1/2007 117 147 266 61% 64% 63% - - -
Romania 06/26/05 06/27/05 - 224 369 - 49% 64% - - - -
Russian Federation 06/21/05 06/24/05 2004 471 490 521 - - - 40% 44% 46%
Vietnam 11/01/05 08/01/06 - 417 398 - 68% 76% - 41% 69% -

In PSI's favor, 6 out of 7 of these studies show large changes with large sample sizes (and the exception is the study with the smallest sample size). On the other hand, these results don't address the question of whether PSI was mostly responsible for this change, or whether it was more due to broad changes independent of PSI.

Answering this question would be difficult, especially because PSI aims to reach as many people as possible with its promotion campaigns, and presumably cannot ensure that any "comparison group" is completely unexposed to its strategies. In the one case where PSI did attempt to use a comparison group (the 2005 Romania study), the comparison group (people without explicit exposure to PSI's strategy) saw a similar improvement to the people PSI targeted, although it's questionable how comparable these two groups really were (for example, the comparison group had twice as high a proportion of married people - see Attachment B-14, pg 5).

Weighing the evidence

Selling life-saving materials en masse is, by nature, both likely to be cost-effective if it works (more on this below) and difficult to measure with great confidence or precision. There are many concerns about using the evidence above to conclude that PSI's activities are effective:

  • Changes in survey responses don't necessarily reflect actual changes in behavior, although we generally feel that the questions are specific and concrete enough to make this a relatively minor concern.
  • All of the reports we've seen are internal reports (i.e., PSI - not an external evaluator - conducted them). This raises the question of whether the reports we've seen are representative; it's possible that reports with more negative conclusions are withheld from publication, and we have no way of knowing this for sure. This is a concern with any nonprofit organization that does not regularly employ an external evaluator (including all of our other finalists). The concern is mitigated somewhat by the fact that some reports (including the one detailed in Attachment B-6) are critical of the program and find that it had no impact. PSI has also recently informed us that it does have some externally conducted evaluation reports available, but we haven't had a chance to review them.
  • Even if PSI does not purposefully withhold less favorable reports, there may be a relationship between the countries in which it conducts good evaluations and the countries in which it performs well: certain countries may be better environments to work in and thus simpler places to carry out all these activities; or the relevant staff might be better in these countries. (This concern also applies to all our other finalists, though it is less serious for PIH and Interplast since their activities are so simple and straightforward as to seemingly require less confirmation from reports.)
  • Neither PSI's sales figures nor its survey results conclusively demonstrate an impact beyond what would have happened without PSI's intervention. It is possible that PSI simply uses its subsidized prices to outcompete more expensive sellers of similar materials, and ends up reducing people's costs but not increasing their ownership or utilization of these materials. A similar concern applies to all aid projects, including all of our finalists: people served does not equal lives affected, and it is difficult to attribute a change to the activities of a particular organization. In PSI's case, many of the observations above mitigate this concern somewhat (for example, the change in reported condom use for PSI's areas vs. other countries; the doubling of ITN ownership in Madagascar, which occurred without a correspondingly drastic economic boom).

We cannot have as much confidence in our understanding of PSI as in our understanding of PIH and Interplast, whose activities are simpler and more straightforward. However, we are far more confident in PSI than in any other organization approaching PSI's size and complexity.

PSI's strategy and approach are both consistent (i.e., it articulates a high-level strategy behind all its activities, and can present a unified view of them) and adaptive (i.e., PSI constantly collects data on its results, which it can presumably use to change what isn't working). Though what it does is by nature difficult to measure, the evidence we have suggests that it is a leading supplier of life-saving materials, and has experienced past success in promoting their use. In the end, although we don't have a complete picture of PSI, having both the high-level picture and a good understanding of example projects has made us feel that PSI is generally a strong organization: well-organized with strong execution and honest self-evaluation. This feeling makes us more inclined to give it the benefit of the doubt on issues and activities we have less understanding of.

What do you get for your dollar?

PSI gives its own estimates of how many lives its activities have significantly changed (HIV infections averted, lives saved from malaria, etc.); we look at these and give the implied cost per life changed, then compare them in two areas (condoms and ITNs) to our own estimates. PSI's estimates are within our ranges for cost-effectiveness; we therefore feel somewhat confident in using PSI's estimates as indicators of what might be expected from the activities we haven't examined.

Overall, we find that PSI's cost-effectiveness is competitive with that of our best finalists, in the ballpark of $1,000 per life saved or significantly changed.

PSI's estimates

The following summarizes PSI's estimates across several of its activities:

PSI estimates of lives significantly changed
Year HIV/AIDS: Infections averted Malaria: deaths averted Contraception: maternal deaths averted Diarrhea: deaths averted Total: lives changed Source
2003 188,014 44,848 10,332 22,211 265,405 HIV/AIDS estimates from Attachment B-29; others from Annual sales reports
2004 192,411 78,405 11,798 26,279 308,893 HIV/AIDS estimates from Attachment B-29; others from Annual sales reports
2005 209,809 92,472 12,544 36,264 351,089 HIV/AIDS estimates from Attachment B-29; others from Attachment B-4 Pg 1

PSI gross costs by program (in thousands)
HIV/AIDS Malaria Family Planning Diarrhea Source
2003 $105,220 $27,490 $35,730 $8,300 Unit Cost Analyses
2004 $118,470 $42,410 $33,090 $7,500 Unit Cost Analyses
2005 $143,390 $69,010 $32,570 $8,400 Unit Cost Analyses

Implied cost per life significantly changed
Year HIV/AIDS Malaria Family Planning Diarrhea
2003 $560 $613 $3458 $374
2004 $616 $541 $2805 $285
2005 $683 $746 $2596 $232

We do not have estimates of lives saved for PSI's other activities, but we tried to get a rough sense for how PSI views their cost-effectiveness, by comparing PSI's activities based on its implied estimates of the cost per Disability adjusted life-year (DALY) averted. We find DALY to be a problematic metric, as detailed on our DALY page and on our blog, but believe that it should correlate roughly with lives saved. Since PSI estimates it for many strategies, we find it a useful way of determining how widely it believes its own strategies to vary in terms of cost-effectiveness.

In the table below, PSI's DALY estimates come from Attachment B-5, pg 4-5, and costs come from the 2005 Unit Cost Analysis.

Product $ cost (millions) DALYs (millions) $/DALY
Condoms $116 6.8 $16
ITNs $54 2.4 $22
Family planning $30 0.6 $49
VCT for HIV/AIDS $20 0.4 $54
Water purification $6 0.4 $17
PPT for malaria $4 0.2 $14
ORT for diarrhea $2 0.1 $19

Consistent with its estimates of lives saved, PSI implicitly finds ITNs to be slightly less cost-effective than condoms, and puts three of its other strategies in the same ballpark (while family planning and VCT result in fewer lives saved per dollar, but largely have different goals - preventing unwanted pregnancies for the former, and slowing the spread for HIV/AIDS for the latter).

Our estimate of cost per HIV infection averted

We estimate HIV infections prevented by PSI's condom supplying programs, using the AVERT model (see Attachment B-23, pg 31-35), a theoretical model that predicts new HIV infections for a population based on HIV prevalence and sexual behavior patterns. Details follow on our estimate, but first we provide the spreadsheet we used as a calculator, so you can plug in your own assumptions:

GiveWell calculator for estimating HIV infections using the AVERT model.

For most factors in the model - especially those we know nothing about, such as sexual behavior patterns - we use a large range of different possible assumptions, and estimate ranges for the cost per infection averted (details below). However, we hold the following assumptions constant:

  • Transmission risk for a given sex act. We use the average of the figures from two different sources: the assumptions given in the paper on the AVERT model (Attachment B-23) and the assumptions of the Disease Control Priorities Project. These come out to:
    • 0.15% male-to-female transmission risk (i.e., a female who has sex with an HIV-infected male has an 0.15% chance of contracting HIV in that act).
    • 0.05% female-to-male transmission risk.
    • 3% male-to-male transmission risk for receptive anal sex (i.e., a male receiving anal sex from an HIV-positive male has a 3% chance of contracting HIV in that act).
    • 0.05% male-to-male transmission risk for insertive anal sex (i.e., a male giving anal sex to an HIV-positive male has a 0.05% chance of contracting HIV in that act).
    • We do not have information on the likely transmission risk for heterosexual anal sex.
  • Condom efficacy. We assume that use of a condom reduces transmission risk by 95% (see Attachment B-23).
  • Total cost per condom used. As the below table (taken from the 2003-2005 Unit Cost Analyses) shows, PSI's overall total cost per condom sold is around $0.10 (including distribution and promotion).

    Gross and net costs per condom sold
    Year Gross cost Program income Net cost Condoms sold Gross cost per condom sold Net cost per condom sold Source
    2003 $91,613 $11,993 $79,620 $782,500 $0.12 $0.10 Unit Cost Analyses
    2004 $95,204 $14,413 $80,791 $855,712 $0.11 $0.09 Unit Cost Analyses
    2005 $115,545 $17,589 $97,956 $931,681 $0.12 $0.11 Unit Cost Analyses

Holding the above assumptions constant, we estimate the cost per life saved for various populations, shown below. The populations are distinguished by their HIV prevalence, sexuality (which affects the transmission risk per sex act, as detailed above), and number of partners ("monogamous" populations have 1 partner per year; "promiscuous" populations have 10-50). For each, we generate a range for the cost per infection averted, varying the number of sex acts per year from 50-500 for both populations (50-500 sexual acts with 1 partner for "monogamous"; in total 50-500 sexual acts, but with 10-50 different partners for "promiscuous"); we vary the number of condoms used per person per year from 1-500. In most cases, these variations do not have large effects on our estimate of infections averted per condom used, and thus of the cost per infection averted.

Group HIV prevalence Cost per infection averted (low estimate) Cost per infection averted (high estimate)
Promiscuous heterosexuals 3% $2,107 $2,262
Promiscuous heterosexuals 6% $708 $770
Promiscuous heterosexuals 20% $237 $274
Monogamous heterosexuals 3% $2,245 $3,830
Monogamous heterosexuals 6% $748 $1,277
Monogamous heterosexuals 20% $249 $426
Promiscuous homosexuals 3% $140 $2,289
Promiscuous homosexuals 6% $48 $858
Promiscuous homosexuals 20% $18 $405
Monogamous homosexuals Highly variable

Note that:

  • These estimates are based only on the use of condoms. They do not take into account any effect that PSI might have on other safe sexual behavior (i.e., abstinence and monogamy).
  • These estimates look only at the cost per HIV infection averted; they do not factor in other benefits of condom use, including avoiding other sexually transmitted diseases and avoiding unwanted pregnancies (and associated maternal mortality).
  • Generally, cost-effectiveness is much higher for higher-risk populations. Since PSI explicitly targets higher-risk populations (see our description of the Mozambique program above for an example), and since we would guess that those facing bigger risks are more likely to use condoms, we expect the actual cost-effectiveness of most of PSI's activities to be more on the optimistic side of our range; we believe that condoms are more likely going to people facing high HIV prevalence and large numbers of partners, for whom it generally costs $500 or less to prevent an HIV infection.
  • Cost-effectiveness gets very complex for monogamous homosexuals or similar groups. The reason is that populations like this generally either face no risk of HIV (if their exclusive partner is not infected) or extremely high risk (if their partner is infected); in the latter case, using condoms consistently has a large protective effect, but using them inconsistently has practically no impact. By contrast, for the other populations covered here, consistent vs. inconsistent use of condoms doesn't have nearly as much impact on the reduction in risk per condom (and thus per dollar).

The average HIV prevalence in relevant countries (based on prevalence data from the WHO's Global Health Atlas), weighted by how much activity PSI conducts in each, is 3.3% (Data on PSI's condoms sold by country come from the 2005 Annual Sales Report). This by itself would imply cost-per-averted-infection numbers closer to the $2000 range, but as we state above, we would guess that PSI disproportionately targets high-prevalence populations and that those who are more at risk, and thus more likely to use condoms. Ultimately, we find it reasonable to expect costs per infection averted to be in the $200-700 range which is consistent with PSI's own estimate (~$650, above).

Our estimate of cost per life saved from malaria

In 2005, the latest year for which we have both sales and cost information, PSI sold a total of 8.2m nets at a cost of $56m (data from Unit Cost Analysis and Sales Report documents). We estimate that those activities saved between 25,000 and 90,000 lives, yielding a cost per life saved of $600 - $2400 This calculation does not credit PSI with increasing utilization rates of bednets, only with increasing ownership at a steady rate of utilization.

Details of the calculation follow. We take numbers such as "number of nets sold" at face value, since it is clear to us where PSI got this data; we also use research-based numbers when possible. When neither is available - i.e., when the only reference point we have is a PSI estimate whose source is not clear - we calculate two values, one designed to err on the side of overstating lives saved ("aggressive") and the other designed to err on the side of understating lives saved ("conservative"). In this way, we attempt to capture a reasonable range within we expect the number of lives saved (and thus the cost per life saved) to fall.

ID Step Conservative estimate Aggressive estimate Source
a # sold 8,237,277 8,237,277 2005 Sales report
b % wasted/misused 20% 5% See note 1
c # owned 6,589,822 7,825,413 a * b
d % sold to rural (at-risk) areas 70% 70% Attachment B-20
e # sold in at-risk areas 4,612,875 5,477,789 c * d
f Utilization rate 50% 80% See note 2
g # used 2,306,438 4,382,231 e * f
h # children / net 0.5 1 See note 3
i # of children protected 1,153,219 4,382,231 g * h
j malaria mortality rate (rural) 1.35% 1.35% Attachment B-20; see note 4
k # would die without net 15,568 59,160 I * J
l protective effect 50% 50% See problems and solutions overview
m annual lives saved 7,784 29,580 k * l
n years a net lasts 3 3 Attachment B-320
o total lives saved 23,352 88,740 m * n
p cost $55,281,638 $55,281,638 2005 Sales report
q $ / life saved $2,367 $623 o / p

Notes:

  1. PSI estimates a 5% rate of waste/misuse (Attachment B-20), but we don't see a source for this estimate, and so for our "conservative" estimate we put in a significantly higher 20% (this is just an estimate, with no more context than we have for the 5% number).
  2. Based on survey data in Madagascar (Attachment B-8), our "aggressive" estimate for utilization among those who own a net is 80%. Our "conservative" estimate is 50% utilization, well below any of the numbers implied by the survey results for Madagascar above. (We also ran a more complex version of the calculation to see whether it changes much when we assume that 50% of children sleep under a net every night, vs. 100% of children sleeping under a net 50% of the time - this was to deal with the fact that "slept under a net last night" is not the same as "regularly sleeping under a net." The more pessimistic and complex calculation reduces our estimate of lives saved by around 0.1%.)
  3. PSI estimates that 0.5 children sleep under each net (Attachment B-20), but we don't see a source for this estimate, and we aren't sure why nets would so rarely be used to cover children, who are the main people at risk from malaria. So our "aggressive" estimate has an average of one child under each net (keeping in mind that one net can easily be used to cover two children).
  4. PSI uses a 1.35% mortality rate for malaria specifically in high-risk (rural) areas; the general malaria mortality rate is 0.88%. Once we factor in the adjustment for how many of PSI's nets (70%) are sold in rural areas, the two different approaches come out about the same (i.e., using 1.35% as the mortality rate and assuming that only 70% of nets are sold in relevant areas is roughly the same as assuming that all nets are sold in areas with a mortality rate of 0.88%).

PSI estimate ($820 per life saved for the most recent year) is at the low end of our range, but not outside it.

Cost-effectiveness: conclusion

It is common for us to see a charity estimate its own cost-effectiveness in a way that wildly overstates it, either by leaving out important costs or by equating "people served" with "lives changed" (rather than incorporating probabilistic estimates to go from people protected to lives saved, for example). PSI's own estimates do not appear to have this problem. In both of the examples above, PSI is explicitly using the same approach that we use, though with different background assumptions; and in both cases, their estimate is consistent with ours.

With that in mind, we believe that PSI's estimates are likely as good as, and quite possibly better than, our own, and returning to the tables at the top of this section, we are inclined to have some confidence in PSI's estimates for other areas (maternal mortality and diarrhea). We are further encouraged by the fact that PSI explicitly aims to assess and maximize cost-effectiveness, though we disagree with the details of how it does so (its Annual Sales Report) states that it uses Disability adjusted life-years (DALYs), a metric we find problematic, as its chief measurement).

Having formed confidence in PSI's general approach and in the validity of its calculations, we are inclined to shade our cost-effectiveness estimates toward its own numbers. Its overall "cost per life significantly changed" (see above) is around $650; we find it reasonable to expect, from the organization as a whole, roughly a life significantly changed (usually saved from death) for every $650-1000. This does not include other benefits of its activities, such as unwanted pregnancies prevented by contraception or non-fatal malaria cases prevented by ITNs.

The organization

Size and scope. PSI is a broad, multi-national organization with programming and staff all over the developing world, including Sub-Saharan Africa, Southeast Asia, Latin America, Central Asia, and Eastern Europe. This review has largely focused on PSI's promotion of ITNs and safe sexual behavior, which together account for about 73% of its budget; we have a generally illustrative breakdown of PSI's activities that covers about 89%.

Personnel. PSI's board has background in government (specifically, international relations), public health, and the for-profit sector (Attachment C-2). PSI's senior leadership have long tenures with PSI, as well as previous experience with the State Department (Attachment C-5).

Financials. The following table is constructed from IRS Form 990s available on GuideStar (2003-2005) and audits (not posted publicly - see the note for financial attachments).

Year Revenues Expenses
2003 $204,195,573 $198,494,566
2004 $248,595,579 $236,510,197
2005 $291,911,988 $280,916,289
2006 $332,851,421 $323,527,773

Over the past few years, PSI's revenues and expenses have grown rapidly and in line with each other, consistent with (though not implying) the idea that PSI's activities can scale relatively easily with more funding. As of the most recent available audit (end of 2005), PSI held $46m in assets, equal to roughly 15% of 2006 expenses; this strikes us as relatively little cash to have on hand (if revenue stopped, it would only be 2 months' worth), but PSI may operate in this way because of relatively predictable funding, as it receives a substantial amount of its support from governments. In 2005, governments accounted for $223 million of PSI's $287 million in revenues ($122 million from the US, $101 million from other governments). Government funding as a proportion of total revenues was slightly higher in 2004.

Conclusion

We believe that PSI's overall approach - marketing life-saving materials - is (a) logical and intelligent; (b) likely to be at least as cost-effective, in terms of saving and changing lives, as any of our other applicants'. We also believe that PSI is superior to all of our other finalists in terms of its consistent, strategic, organization-wide commitment to thorough monitoring and evaluation; while our other finalists run some projects that we find as worthwhile as PSI's programs, none have given us the same kind of confidence in the organization as a whole. Between our confidence in its specific activities and in its organizational approach, we find PSI to be the best organization we know for translating donations into lives saved.

PSI's largest disadvantage, relative to our other finalists, is that its activities are relatively hard to monitor and intangible: while Interplast can directly report the number of surgeries it has performed, and PIH can directly observe its effects on patients, estimating PSI's impact requires many assumptions and projections. This makes monitoring and evaluation all the more essential for PSI, and we hope to see its commitment to these practices continue in the future, and to result in an increasing set of information about its effectiveness. Ultimately, we are willing to accept a wider range of possible outcomes, and slightly less "tangible" effects on lives, in return for our belief that in all likelihood, a donation to PSI will help people more than a donation to any other charity we know of within this cause.

We'd like to know more about:

  • Existing evidence of effectiveness. Not all of the evaluations alluded to in Attachment B-19 are publicly available; we'd like to see as many as possible.
  • How PSI chooses which projects to pursue next. PSI calls this strategic planning (Attachment A-1, pg 4), but we don't know how they choose to work in a given region over another, or target a specific group of people instead of another.

Attachments

A. Application and response:

  • Population Services International, Unit Cost Analysis:
    PSI has requested that we not post full Unit Cost Analysis reports publicly. Full reports going back to 1997 are available for interested members of the public; please contact PSI if interested. PSI provided us with excerpted reports for 2003-2005, which we are cleared to publish.
  • Population Services International, Annual Sales Reports:
    PSI has requested that we not post full Annual Sales Reports publicly. Full reports going back to 1999 are available for interested members of the public; please contact PSI if interested. PSI provided us with excerpted reports for 2005-2006, which we are cleared to publish.

D. Financial documents

We are not cleared to publicly post PSI's financial audits, but PSI has stated that they are available to interested members of the public upon request. Please contact PSI if interested.

Additional notes

We made these notes during a February 2009 review of PSI's materials. Our most recent take is available at our 2008-2009 review of PSI.

Update on PSI's transparency

In 2007-2008, PSI stood out from other charities we examined partly because of its transparency: its willingness to publicly share information about its operations, expenses and estimated impact. Since then PSI has further improved its transparency, with a new Health Impact page that provides information on its sales, expenses, and estimated health impacts broken out by product and country. Much of the information we cited from the Unit Cost Analyses and Sales Reports (which we could publicly provide only excerpts from, although we had access to the full documents) is included in the documents on the Health Impact page, although past years (prior to 2006 for some information and 2007 for other information) are not available.

Update on PSI's activities

Based on data from pages 2 and 7 of the 2006 Cost-effectiveness report, we believe that PSI's 2006 expenses were more concentrated than in previous years on (a) sub-Saharan Africa (up from 72% in 2005 to around 80% in 2006, with roughly even parts of this shift coming from other regions); (b) malaria products (up from 23% in 2005 to around 30% in 2006; HIV/AIDS and contraception products maintained roughly stable shares of expenses, with the balance coming from smaller programs). We feel that the activities we focus on in our review are still representative of PSI's activities as a whole (and if anything are more representative due to the higher share of malaria-related activities.)

Update on PSI's effectiveness and cost-effectiveness:

PSI still does not appear to publish the full details of its cost-effectiveness calculations; it also does not track impact for all its programs, though it does so more than the great majority of other charities. Our most up-to-date take on PSI's effectiveness is in our 2008-2009 review.

We urge caution in interpreting our cost-effectiveness estimates, as they involve many assumptions and are based on the few regions for which we had relevant data available. However, we do not wish to revise these estimates numerically. Other cost-effectiveness estimates for insecticide-treated nets (detailed here) are consistent with our existing estimate for PSI's program.

Other notes

We have updated material on two topics discussed in this review: