Population Services International (PSI) – February 2011 Version

    Our Report

    We have published a more recent review of this organization. See our most recent report on PSI.


    About this page

    GiveWell aims to find the best giving opportunities we can and recommend them to donors. We tend to put a lot of investigation into the organizations we find most promising, and de-prioritize others based on limited information. When we decide not to prioritize an organization, we try to create a brief writeup of our thoughts on that charity because we want to be as transparent as possible about our reasoning.

    The following write-up should be viewed in this context: it explains why we determined that we wouldn't be prioritizing the organization in question as a potential top charity. This write-up should not be taken as a "negative rating" of the charity. Rather, it is our attempt to be as clear as possible about the process by which we came to our top recommendations.

    A note on this page's publication date

    The content we created in 2011 appears below. PSI's response to this content can be found in the charity response tab (see above). The content below is likely to be no longer fully accurate, both with respect to what it says about PSI and with respect to what it implies about our own views and positions. As of early 2014, PSI has expressed interest in re-applying for a recommendation.

    Published: February 2011

    Summary

    PSI promotes and distributes materials throughout the developing world, particularly condoms, insecticide treated nets, and contraceptives, and seeks to change behavior to improve health. PSI stands out from other charities working to improve lives in the developing world for both (a) focusing on programs with proven impact and (b) monitoring whether these programs are implemented effectively.

    The evidence we have seen does not clearly show that PSI has the impact it intends.

    Previous reports on PSI
    December 2007 review
    July 2009 review

    Table of Contents

    What do they do?

    Activities

    PSI promotes and distributes materials, particularly condoms, insecticide treated nets, and contraceptives, to save lives, improve health, and help families control their fertility in the developing world. Its largest program areas focus on preventing HIV/AIDS (primarily through condom promotion/distribution), reducing the burden of malaria (primarily through promotion/distribution of insecticide-treated nets (ITNs)), and improving reproductive health, i.e., reducing unwanted pregnancies (primarily through promotion/distribution of condoms and non-condom contraceptives).1

    • Insecticide-treated nets and retreatment kits: PSI offers both traditional and long-lasting insecticide treated nets, as well as materials need to replace the insecticide on nets after it wears off.2
    • Condoms: PSI offers both male and female condoms, with male condoms accounting for 96% of its total spending on condoms.3
    • Non-condom contraceptives: PSI offers (in order of gross cost) oral contraceptives, injectable contraceptives, IUDs, emergency contraception, implants, surgical sterilization, and cycle beads.4
    • Other: In addition to the above products, PSI offers HIV testing and counseling, water purification technologies, malaria diagnosis and treatment kits, lubricant, nevirapine (the drug used to prevent mother-to-child transmission of HIV), and other health products.5

    PSI has told us that additional donations will likely be used to support the distribution of products other than condoms and ITNs (more below).

    PSI has historically focused on selling products for below-market prices, seen in our 2007-2008 discussion of this topic, but also distributes some products for free. In 2009, 16% of condoms and 89% of insecticide-treated nets were given away for free.6 PSI told us that it does not collect information at its headquarters on the size of subsidies on items that are sold. PSI encourages country programs to set prices with cost recovery in mind.7

    Monitoring and evaluation

    PSI expresses a commitment to relatively systematic monitoring and evaluation of its activities - particularly "MAP surveys," which aim to evaluate the availability (and appropriate marketing) of PSI's products in a region,8 and "TRaC surveys," which aim to evaluate relevant behavior change over time9 (for example, use of condoms, insecticide-treated nets, and modern contraceptives - see the following section).

    Results from TRaC reports are available online on PSI's website.10 Technical papers describing the methodology for data collection for specific TRaC surveys are also available on PSI's website.11

    Does it work?

    PSI distributes products that have been rigorously shown to save or improve lives and is a major supplier of these products in the countries in which it works. PSI monitors some of its projects to determine whether the types of products it distributes are used consistently. We see little compelling evidence that PSI's programs are causing significant behavior change, and mixed evidence that PSI's products are used consistently.

    Condoms and ITNs: do they save lives?

    Both condoms and ITNs (which together comprise the bulk of PSI's expenses, as noted above) are proven to improve health (reducing the risk of HIV infection and malaria, respectively) when used correctly and consistently. For more, see our full reports on the promotion and distribution of condoms and ITNs.

    Condoms and other modern contraceptives: do they prevent pregnancy?

    We have not completed a full report on the ability of modern contraceptives (which PSI defines as oral contraceptives, injectables, condoms, IUDs, sterilization, diaphragm, foams/tablets, female barrier methods, or hormonal implants)12 to prevent pregnancy, but after conducting a search for evidence reviews on this topic, reviewing the most recent review we found and reading the Disease Control Priorities report chapter on this topic, we conclude that these forms of contraceptives reduce the risk of pregnancy.13

    Monitoring and evaluation

    Setting it apart from the vast majority of other charities, PSI conducts surveys to assess whether people use the products it sells and gives away, including surveys on:

    • Condom usage14
    • Insecticide-treated net ownership and usage15
    • Modern contraceptive usage (both condoms and other contraceptives)16

    We feel that PSI, because is it primarily distributing products with strong track records of effectiveness, has a lower burden of proof for demonstrating impact. In order to feel confident in PSI's impact, we would need to see either:

    • Evidence that a high percentage of PSI's products are used. We have not seen data on usage of PSI's product brands specifically, but there is some argument that the high usage rates of such products generally implies that the products sold by PSI are likely being used. There is some encouraging evidence for condom usage rates (50-80% in countries where PSI operates large programs), but usage rates for ITNs and modern contraceptives (30-60%) are less encouraging (details below).
    • Evidence that PSI is positively affecting behavior, particularly by increasing usage of condoms, ITNs, and modern contraceptives. The evidence we have seen suggests that PSI has a mixed track record on this. The average change in product use is relatively small -- 3-4 percentage point change for condoms, 12-13 percentage points for ITNs, and 2-3 percentage points for modern contraception -- and results vary significantly across countries. For countries where product usage has increased over time, there is not enough evidence to confidently attribute this change to PSI (details below). On the other hand, country- and continent-wide trends might be legitimately attributable to PSI's activities, given PSI's large reach.

    Are PSI's products used?

    Note that PSI does not believe that the analysis in this section is a valid use of the data it collected for the purpose of tracking behavior change or a valid method of judging its impact.

    In November 2010, we complied a list of the countries in which PSI distributes the most products, as measured by total gross cost of products. We then looked at household surveys of product use conducted by PSI between 2003 and 2009 for each country that had a gross cost of products for one of PSI's focus areas (HIV, malaria, reproductive health, and child survival) of over $5 million in 2008 (the most recent year for which we had data; note that no child survival programs qualified). We excluded surveys that were directed at limited sub-groups such as truck drivers or men who have sex with men. We summarize the results of the surveys we reviewed below:17

    Condoms: We found 7 countries with total gross HIV product costs of over $5 million in 2008, and we found household-level survey data on condom usage for 5 of these. All 5 surveys found moderate to high rates of condom use (in the range of about 50%-80% of respondents reporting condom use).

    Country Year of survey High rates of condom use?
    Nigeria18 2005 Mixed evidence: relatively high reported rates of use among men (70%), but considerably lower among women (39%)
    Zimbabwe19 2006 Yes: 80.7% reported consistent use with non-regular partner
    Zambia20 2003 Moderate: 52% reported use among males, female use not reported
    Pakistan No survey found N/A
    South Africa21 2007 Yes: 91% reported use among males; 83% among females
    India No survey found N/A
    Mozambique22 2008 Moderate: 53% reported use among females; 61%-68% among males

    ITNs: We found 9 countries with total gross malaria product costs of over $5 million in 2008, and we found household-level survey data on ITN usage for 8 of these. Six surveys found moderate to high rates of ITN use, while two found fairly low rates of use.

    Country Year of survey High rates of ITN use?
    Democratic Republic of the Congo23 2009 Yes: reported use by children under 5: 66%; reported use by pregnant women: 62%
    Kenya24 2007 Moderate: reported use by children under 5: 61%; reported use by pregnant women: 36%
    Mali No survey found -
    Malawi25 2005 No: reported use by children under 5: 20%; reported use by pregnant women: 21%
    Tanzania26 2008 Moderate: reported use by children under 5: 64%; reported use by pregnant women: 37%
    Sudan27 2007 Moderate: reported use by children under 5: 46%; reported use by pregnant women: 62%
    Mozambique28 2009 No: reported use by children under 5: 35%; reported use by pregnant women: 27%
    Zambia29 2009 Moderate: reported use by pregnant women: 49%; use by children not measured
    Madagascar30 2007 Yes: reported use by children under 5: 69%; reported use by pregnant women: 58%

    Modern contraceptives: We found 4 countries with total gross reproductive health product costs of over $5 million in 2008, and we found household-level survey data on modern contraceptive usage for all of these. Two surveys found moderate to high rates of modern contraceptive and two found low usage rates.

    Survey year High rates of modern contraceptive use?
    Nigeria31 2005 No: reported use 11.2%
    Pakistan32 2007 No: reported use 33%
    Kenya33 2007 Yes: reported use 66%
    Zambia34 2007 Yes: reported use 57%

    Is PSI creating behavior change?

    In May 2010, we reviewed the results of all PSI's published TRaC studies on condom, ITN, and modern contraceptive use (we also incorporated additional data that PSI sent us directly). We note that data may not be fully representative of PSI's work because data that allows us to see change over time is not available for all of PSI's projects. The reasons why data for some projects is not available include:

    • Baseline and follow up surveys may not have be conducted in every country in which PSI works. Of the 54 countries in which PSI distributed condoms in 2008, data on condom use from at least two surveys (and for at least one group) is available for 27 countries. Of the 30 country programs that distributed ITNs in 2008, data on ITN use is available for 10 countries. Of the 56 countries in which PSI distributed modern contraceptives in 2008, data on use is available for 12.35
    • Survey data may be missing due to country programs failing to report to headquarters.36
    • Data from some surveys received by headquarters may not be reported in the online summary.37

    Overall, however, PSI appears to conduct studies on (and publish the results publicly for) a greater percentage of its activities than the vast majority of charities we have looked at.

    Survey results, in brief: (note that statistical significance was determined by PSI; there were no statistically significant decreases in use marked)38

    Condoms ITNs Modern contraceptives
    Number of surveys 60 10 12
    Number of surveys showing increase in use (statistically significant increase) 40 (17) 9 (7) 9 (5)
    Number of surveys showing decrease or no change in use 20 1 3
    PSI: average change in use (percentage points) 4 12 3
    PSI: median change in use (percentage points) 3 14 3
    Overall in Africa: average change in use (percentage points) 3 28 (limited data) 6
    Overall in Africa: median change in use (percentage points) 2 23 (limited data) 4
    • Condoms: Of the 60 TRaC studies that were conducted more than once and from which we have data, 17, or 28%, show statistically significant increases in reported condom use over time. We find the overall change in reported use of 4 percentage points across all 60 surveys to be somewhat encouraging, though we remain concerned about the representativeness of the data and the fairly low number of statistically significant increases.

      Condom use across African countries as a whole seems to follow a similar pattern of little evidence of substantial change over time.39

    • ITNs: Of the 10 TRaC studies that were conducted more than once and from which we have data, 7, or 70%, show statistically significant increases in reported use over time. The small number of surveys prevents us from drawing a conclusion about PSI's malaria programs as a whole, though we find the average increase in reported use among pregnant women and children under 5 of 12.4 percentage points across the 10 surveys encouraging.

      We found independent survey data on changes in ITN use in sub-Saharan Africa for only 5 countries, showing substantial increases in use in four out of five countries in recent years.40 In addition, we are aware of at least one country where there was significant ITN-distribution activity outside of PSI: Kenya. We discuss Kenya's ITN distribution program in our program report for distribution of insecticide-treated nets; studies of the program credit PSI with a major role, but not with sole responsibility for the encouraging results. We find it slightly concerning that PSI's discussions of trends within Kenya do not mention the context of the large-scale government-led program that was taking place at the time.

    • Modern contraceptives: Of the 12 TRaC studies that were conducted more than once and from which we have data, 5, or 33%, show statistically significant increases in reported use over time. In addition, at 3.0 percentage points, the average increase in reported use in the 12 surveys is quite low.

      Modern contraceptive use across sub-Saharan African countries as a whole seems to follow a similar pattern of little evidence of substantial change over time.41

    We note that PSI appears to be a dominant supplier of both condoms and ITNs, and to a lesser degree oral contraceptives and injectable contraceptives, within sub-Saharan Africa.42 Thus, country- and continent-wide trends might be legitimately attributable to PSI's activities.

    We have shared this analysis with PSI, and its representatives stated to us that they find it to be invalid analysis of their data. They pointed out to us that out of 163 available multi-round studies of behavior change, 82 have shown statistically significant impact on at least one behavior change indicator. We have examined these 82 studies, and while we feel they are generally high-quality, we also observe that they examine a large number of indicators, none of which seem clearly more meaningful than the indicators we examine above and many of which seem less meaningful. We thus do not feel that a conclusion about PSI's impact can be reached based on the number of studies showing improvement on at least one indicator.

    Possible negative or offsetting impact

    PSI likely sells at least some products at significantly below-market prices (or distributes them for free), and may thus be effectively replacing or out-competing private provision of these products. In addition, subsidies may not reach buyers, as they may be absorbed by sellers.

    PSI appears concerned with this issue and sent us a document explaining the problems that can arise from pricing products below the market rate, including:43

    If our price is kept too low, we effectively shut out private sector competitors, hampering market development and leading to an unsustainable situation which could collapse if/when the donor exits.

    There is also a concern that by subsidizing products, PSI is effectively transferring cash to product sellers, who may sell the products at, or close to, market prices.

    PSI told us that it was not able to provide data on the difference between market prices and PSI prices, as it does not collect market price data at the headquarters level.44 (PSI did provide data on wholesale prices as well as prices charged by other non-profit organizations and clinics.45 )

    What do you get for your dollar?

    In our 2007-2008 report, we roughly estimated that PSI's condom program prevents a case of HIV/AIDS for every $500-1,000 it spends, and that its ITN program prevents a case of malaria for every $600-2,400 it spends.46 At that time, we did not have access to the details of PSI's own calculations; we now have the details of its calculations for both condoms47 and ITNs.48 We have not discussed with PSI the details of its contraceptive cost-effectiveness calculations.

    Note that while we report on PSI’s activities by product (e.g., condoms, ITNs), PSI reports its health impact by health area. PSI's 2008 report on cost-effectiveness gives overall estimates by health area including:

    • HIV: $1,310 per HIV infection averted.49 Using the conversion factor used by the Disease Control Priorities in Developing Countries report, $1,310 per prevented infection is equivalent to $65 per disability adjusted life year (DALY).50 PSI's own estimate puts its HIV programs at $42 per DALY.51 Thise figures are well within the range of $52-$112 per DALY and $1,040-$2,240 per HIV infection averted estimated by an independent source (details in the cost-effectiveness section of our review of condom distribution programs).
    • Malaria: $533 per child's life saved.52 Using the conversion factor implied by an independent source,53 $533 per life saved is equivalent to $16 per DALY. PSI's own estimate puts its malaria programs at $18 per DALY.54 This figure is consistent with other estimates of the cost-effectiveness of ITNs (more at our analysis of ITN cost-effectiveness).
    • Reproductive health: $67 per pregnancy adverted, $13,370 per maternal death adverted, and $47 per DALY.55 We have not looked into the details of how PSI calculates these figures, but they are roughly in line with the Disease Control Priorities report's estimates for the cost-effectiveness of contraceptives.56

    Note: In September 2011, we confirmed a number of errors in the estimates for the cost-effectiveness of deworming published in the Disease Control Priorities report. Based on those findings, we are currently rethinking our use of cost-effectiveness estimates, like the DCP2's, for which the full details of the calculations are not public. For more information, see our blog post on the topic.
    PSI estimates its overall cost effectiveness at $29 per DALY for 2008.57

    We find PSI's approach to be largely similar to the approach we took, but with more detail and more informational sources than we used. We therefore defer to PSI's own cost-effectiveness estimates, with the following caveats:

    • PSI's condom estimate assumes a 10% wastage rate for purchased condoms58 and a 35% wastage rate for free condoms.59 No source is given for these estimates, and it appears that PSI's monitoring and evaluation does not include methods for determining actual wastage rates. We believe that the question of whether the condoms sold by PSI reach low-income people who use them consistently and appropriately is a major question.
    • Similarly, PSI's estimate for ITNs includes the assumption that "each LLIN [a form of ITN] sold will protect on average 1.42 people, of which 0.55 of those will be aged 0-4 years," citing Demographic and Health Surveys and Multiple Indicator Cluster Surveys from malaria endemica countries.60 Given the high variation in ITN usage rates (described above) and the relatively small amount of data on them, we remain unsure about the appropriateness of this assumption.

    It is normal for cost-effectiveness estimates to include many rough estimates and assumptions, but we have highlighted those that pertain directly to the efficacy of PSI's programs - i.e., we have highlighted questions about whether PSI's products are used as intended.

    Room for more funds?

    In July 2010, PSI informed us that it has "secured adequate funding to support its core programs in malaria and HIV prevention for the coming year; however, funding for other effective health interventions, such as family planning, tuberculosis treatment, and the prevention and treatment of common childhood illnesses has lagged."61

    To meet its goals for 2011, PSI need to raise an additional (i.e. in addition to funding secured as of July 2010):

    • $205,684,748 for reproductive health programs, primarily focusing on modern contraceptive methods.62 Specifically PSI would like to expand access to long-lasting methods of contraception, such IUDs and implants, in countries with a contraceptive prevalence rate of below 40 percent.63 PSI told us that reproductive health is its first priority for additional funding.64
    • $16,944,867 for child survival programs. PSI told us that child health is its second priority for additional funding.65

    PSI told us that it would allocate additional funding to specific projects "through an internal discretionary fund competition."66

    After filling the above funding gaps, PSI would fund expansion of tuberculosis treatment programs (which it currently runs in two countries), followed by scale-up of pilot projects in smoking cessation, obesity prevention, and diabetes management.67 PSI also mentioned that it might use additional funding to improve the variety of condom types it offers.68 We do not know how much additional funding PSI could use in the next year for each of these projects.

    PSI told us that the size of a donation is not a factor in determining how it is spent.69

    Financials/other

    Unless otherwise noted, all data comes from:

    • For 2002-2007: PSI, "IRS Form 990s (2002-2007)."
    • For 2008: PSI, "Progress Report (2010)," Pg 24.
    • For 2009: PSI, "Draft financial statement (2009)."

    Revenue and expense growth (about this metric): PSI's revenues and expenses have grown consistently over the past 6 years.

    Assets-to-expenses ratio (about this metric): PSI maintains a low assets:expenses ratio. This may be because PSI receives a large portion of its support from governments, which may provide less volatile support, so that PSI does not need as large a reserve cushion as other charities.

    Expenses by program area (about this metric): PSI focuses on distribution of ITNs and condoms.70

    Expenses by IRS-reported category (about this metric): PSI maintains a reasonable "overhead ratio," spending approximately 93% of its budget on program expenses.

    Sources

    • 1

      Data for chart from PSI, "Cost Effectiveness Report (2008)," Pg 77-80. Calculations in GiveWell, "PSI Expenses by Product Type."

    • 2

      PSI, "Cost Effectiveness Report (2008)," Pg 77-80.

    • 3

      PSI, "Cost Effectiveness Report (2008)," Pg 77-80.

    • 4

      PSI, "Cost Effectiveness Report (2008)," Pg 77-80.

    • 5

      PSI, "Cost Effectiveness Report (2008)," Pg 77-80.

    • 6

      PSI, "Answers to GiveWell Questions (2010)," Pg 1.

    • 7

      PSI, "Answers to GiveWell Questions (2010)," Pg 1. For more detail see, PSI, " DELTA Companion - Price P."

    • 8

      PSI, "Product and Service Availability and Access Studies (MAP)."

    • 9

      PSI, "Population-Based Surveys (TRaC)."

    • 10

      PSI, "Population-Based Surveys (TRaC)."

    • 11

      PSI, "Publications Catalogue." A keyword search for "TRaC study" results in a large number of such studies.

    • 12

      PSI, "PERForMance Review: Behavioral Indicators," Pg 8.

    • 13

      A recent review of available studies on the efficacy of contraceptive methods found the following failure rates in the first year of use:

      % of PSI spending on reproductive health, gross cost Perfect use failure rate (%) Typical use failure rate (%) Disease Control Priorities report says:
      Condoms 50% N/A 2.5–5.9 "Typical use results in pregnancy rates of 3 to 14 percent per year."
      Oral contraceptives 16% 0 – 1.26 0–2.18 "Has a failure rate typical use of less than 10 percent over a year."
      Injectables 13% 0 0 "Have high rates of con- traceptive efficacy (0.3 to 1.0 percent failure rate over 12 months)."
      IUD 12% N/A 0.16–1.26 "Failure rates for all copper-bearing IUDs are usually less than 1 per 100 women in the first year of use."
      Hormonal implants 2% N/A 0–0.08 "Have high rates of con- traceptive efficacy (0.3 to 1.0 percent failure rate over 12 months)."
      Female condoms 2% Not listed Not listed "The female condom is slightly less effective than the male condom, with a failure rate of 5 to 21 percent."
      Sterilization 1% N/A N/A "The most effective types of female sterilization have a 10-year cumulative pregnancy rate of 7.5 per 1,000 procedures."
      Foam/tablets Not distributed by PSI Not listed Not listed N/A

      Failure rate data from Mansour, Inki, and Gemzell-Danielsson 2010, Pg 7, Table 1. PSI spending data from PSI, "Cost Effectiveness Report," Pg 77-80. Disease Control Priorities report statements from Jamison et al. 2006, Pg 1080.

      The above failure rates can be compared against rates of pregnancy of about 89% in the first year among women who are trying to conceive and 85% among women who are trying to prevent pregnancy through the timing of intercourse but who do so imperfectly. Mansour, Inki, and Gemzell-Danielsson 2010, Pg 6.

    • 14

      Condom Use With Non-Marital, Non-Cohabiting Partner at Last Sex
      Definition: percent of men and women aged 15-24 and/or 15–49 who say they used a condom the last time they had sex with a non- marital, non-cohabiting partner of those who have had sex with such a partner in the last 12 months…
      Sex Workers: Condom Use at Last Sex With a Client
      Definition: percent of sex workers sampled who used a condom at their last sex with a client of all those who admit to having sex for money or material goods.
      Men Who Have Sex with Men (MSM): Condom Use at Last Anal Sex With a Male Partner
      Definition: percent of men who used a condom at last anal sex with a male partner of those who had anal sex with a male partner in the last 12 months."
      PSI, "PERForMance Review: Behavioral Indicators," Pgs 5-6.

    • 15

      "Usage of ITN among pregnant women
      Definition: percent of pregnant women who slept under an ITN the previous night…
      Usage of ITN among children under five
      Definition: percent of children under five who slept under an ITN the previous night."
      PSI, "PERForMance Review: Behavioral Indicators," Pgs 7-8.

    • 16

      "Usage of Modern Contraceptive Method...
      Denominator: all married women aged 15-49 surveyed in the study.
      Numerator: married women aged 15-49 who report they are currently using any form of the following contraceptive methods: pills, injectables, condoms, IUD, sterilization, diaphragm, foams/tablets, female barrier methods, or hormonal implants." PSI, "PERForMance Review: Behavioral Indicators," Pg 8.

    • 17

      More details in GiveWell, "Analysis of Largest PSI Programs (November 2010)."

    • 18

      GiveWell, "Summary of TRaC Surveys (2010)."

    • 19

      PSI, "Zimbabwe (2006): HIV Prevention TRaC Study among the General Population (15-49 years), Second Round."

    • 20

      GiveWell, "Summary of TRaC Surveys (2010)."

    • 21

      GiveWell, "Summary of TRaC Surveys (2010)."

    • 22

      PSI, "Mozambique 2008: HIV/AIDS TRaC Study Among Men and Women Aged 15- 35 in Sofala, First Round" and PSI, "Mozambique 2008: HIV/AIDS TRaC Study Among Men and Women Aged 15- 35 in Zambezia, First Round."

    • 23

      PSI, "DRC (2009): Enquete TRaC Pour le Suivi Et L’evaluation de L'utilisation de la Moustiquaire Impregnee et la Prise en Charge du Paludisme Chez les Enfants de Moins de 5 Ans."

    • 24

      PSI, "Kenya (2007): Malaria TRaC Study Evaluating Bed Net Ownership and Use among Pregnant Women and Children under 5 (years), Third Round."

    • 25

      PSI, "PSI/Malawi Project TRaC – Malaria and Diarrheal Disease."

    • 26

      PSI, "Tanzania (2008) Malaria Prevention among Children Under Age Five in Tanzania, Second Round."

    • 27PSI, "Sudan (2007): Determinants of Mosquito Net Ownership and Use in Southern Sudan."
    • 28

      PSI, "Mozambique (2009): Malaria Prevention TRaC Study Evaluating Mosquito Net Use Among Caregivers of Children under the Age of Five in Maputo, Inhambane and Zambezia Provinces, First Round."

    • 29

      PSI, "Zambia (2007): Malaria TRaC Study Evaluating the Use of Insecticide Treated Nets Among Women, Round Two."

    • 30

      PSI, "Madagascar (2008): Malaria TRaC Study Evaluating the Use of Insecticide Treated Nets among Pregnant Women and Mothers/Caregivers of Children Younger than Five Years in Endemic area, Third Round."

    • 31

      GiveWell, "Summary of TRaC Surveys (2010)."

    • 32

      PSI, "Pakistan (2007): Family Planning TRaC Study Evaluating Contraceptive Use among Married Men and Women in Rural and Urban Pakistan, First Round."

    • 33

      PSI, "Kenya (2007): Use of Modern Contraceptive Methods Among Sexually Active Women Aged 20-35 Years."

    • 34

      PSI, "Family Planning TRaC Study: Determinants of Oral Contraceptive Use Among Zambian Women, Second Round."

    • 35

      GiveWell, "Summary of Behavior Change TRaC Surveys (2010)."

    • 36

      As of May 2010, only 86% of country programs had reported TRaC data for 2008 and only 12% had reported data for 2009. PSI, "Research and Metrics Scorecard (May 2010)."

    • 37

      For example, data downloaded in June 2009 included condom use survey data from Uganda for members of the military. Data downloaded from the same source in May 2010 did not include this data. See GiveWell, "Summary of TRaC Surveys (2009)" and GiveWell, "Summary of Behavior Change TRaC Surveys (2010)." Note that for completeness we have added data missing from the 2010 download to the 2010 file.

    • 38

      GiveWell, "Summary of Behavior Change TRaC Surveys (2010)." Calculated data is on the "Summary" sheet. Data in other sheets is from PSI, "Population-Based Surveys (TRaC)" and PSI, "Performance Review (2008-2010)." Surveys that did not have data on use for more than one year were excluded. Statistical significance was determined by PSI, and is indicated at PSI, "Population-Based Surveys (TRaC)" by blue shaded boxes and in PSI, "Performance Review (2008-2010)."

    • 39

      The following data is from Measure DHS, "HIV/AIDS Survey Indicators Database." Data is for males ages 15-49 in sub-Saharan Africa for the indicator "condom use at last sex with anyone." All countries in sub-Saharan Africa with data for two or more years are included. Data can be found in GiveWell, "Demographic Health Surveys: Condom Use (May 2010)."

      Country Previous survey Most recent survey Condom use, previous survey (%) Condom use, most recent survey (%) Change in condom use (percentage points)
      Benin 2001 2006 16 16 0
      Burkina Faso 1999 2003 24 31 7
      Cameroon 1998 2004 17 32 15
      Ethiopia 2000 2005 6 5 -1
      Ghana 1998 2003 17 20 3
      Guinea 1999 2005 16 20 4
      Kenya 1998 2003 21 18 -4
      Malawi 2000 2004 15 16 1
      Mali 2001 2006 10 10 0
      Mozambique 1997 2003 7 14 8
      Namibia 2000 2006 48 57 9
      Nigeria 1999 2003 15 18 3
      Rwanda 2000 2005 7 6 -1
      Tanzania 1999 2005 18 21 3
      Uganda 2005 2006 16 18 2
      Zambia 1996 2002 22 21 -1
      Zimbabwe 1999 2006 29 25 -3
      Average change (arithmetic) - - - - 3
      Median change - - - - 2

    • 40

      The following data is from Measure DHS, "STATcompiler: Tablebuilder." Data is for sub-Saharan Africa for the indicators "percentage of children under 5 who slept under an ITN last night," and "percentage of pregnant women age 15-49 who slept under an ITN last night." All countries in sub-Saharan Africa with data for two or more years are included. Data can be found in GiveWell, "Demographic Health Surveys: ITN Ownership and Use (May 2010)."

      ITN use last night, children under 5 women
      Country Previous survey Most recent survey Previous survey (%) Most recent survey (%) Change (percentage points)
      Kenya 2003 2008-2009 6.0 46.7 40.7
      Nigeria 2003 2008 1.2 5.5 4.3
      Rwanda 2005 2007-2008 12.6 56.5 43.9
      Senegal 2005 2008-2009 7.2 29.2 22.0
      Zambia 2001-2002 2007 7.3 28.5 21.2

      ITN use last night, pregnant women
      Country Previous survey Most recent survey Previous survey (%) Most recent survey (%) Change (percentage points)
      Kenya 2003 2008-2009 5.4 49.0 43.6
      Nigeria 2003 2008 1.3 4.8 3.5
      Rwanda 2005 2007-2008 17.2 60.3 43.1
      Senegal 2005 2008-2009 8.6 29.7 21.1
      Zambia 2001-2002 2007 8.9 32.7 23.8

      ITN use last night, summary
      • Average change in use: 27.7%
      • Median change in use: 22.9%

    • 41

      The following data is from Measure DHS, "Stat Compiler: Table Builder." Data is for married women ages 15-49 in sub-Saharan Africa for the indicator "current use of any modern contraceptive method." All countries in sub-Saharan Africa with data for two or more years are included. Data can be found in GiveWell, "Demographic Health Surveys: Modern contraceptive use (September 2010)."

      Country Previous survey Most recent survey Modern contraceptive use (%), previous survey Modern contraceptive use (%), most recent survey Change in modern contraceptive use (percentage points)
      Benin 2001 2006 7.2 6.1 -1.1
      Burkina Faso 1998-1999 2003 8.8 31.7 22.9
      Cameroon 1998 2004 7.1 12.5 5.4
      Chad 1996-1997 2004 1.2 1.6 0.4
      Cote d'Ivoire 1994 1998-1999 4.3 7.3 3.0
      Eritrea 1995 2002 4.0 7.3 3.3
      Ethiopia 2000 2005 6.3 13.9 7.6
      Ghana 2003 2008 18.7 16.6 -2.1
      Guinea 1999 2005 4.2 5.7 1.5
      Kenya 2003 2008-2009 31.5 39.4 7.9
      Liberia 1986 2007 5.5 10.3 4.8
      Madagascar 2003-2004 2008-2009 18.3 29.2 10.9
      Malawi 2000 2004 26.1 28.1 2.0
      Mali 2001 2006 5.7 6.9 1.2
      Mozambique 1997 2003 5.1 20.8 15.7
      Namibia 2000 2006-2007 42.6 53.4 10.8
      Niger 1998 2006 4.6 5.0 0.4
      Nigeria 2003 2008 8.2 9.7 1.5
      Rwanda 2005 2007-2008 10.3 27.4 17.1
      Senegal 1997 2005 8.1 10.3 2.2
      Tanzania 1999 2004-2005 16.9 20.0 3.1
      Togo 1988 1998 3.1 7.0 3.9
      Uganda 2000-2001 2006 18.2 17.9 -0.3
      Zambia 2001-2002 2007 25.3 32.7 7.4
      Zimbabwe 1999 2005-2006 50.4 58.4 8.0
      Average change (arithmetic) - - - - 5.7
      Median change - - - - 3.6

    • 42

      In the 30 countries in which PSI works in sub-Saharan Africa, PSI averages a condom market share of 42.8% (data missing for 3 countries), an ITN market share of 23.8% (data missing for 2 countries), an oral contraceptive market share of 15%, and an injectables market share of 10%. Data from PSI, "Annual Health Impact Report (2008)," Pg 31. Calculations in GiveWell, "PSI Market Shares (2008)."

      PSI collects data on product sales by the public sector, other NGOs and institutions, and the commercial sector in order to determine total sales and calculate its market share. PSI, " Answers to GiveWell Questions (2010)," Pg 2.

      PSI did not provide us with technical details on the process it uses to collect raw data on sales by the public sector, other NGOs and institutions, and the commercial sector.

    • 43

      PSI, "Delta Companion - Price P," Pg 66.

    • 44

      PSI, "Answers to GiveWell Questions (2010)," Pg 1.

    • 45

      PSI, "Pricing study: Field data."

    • 46

      See our 2007-2008 cost-effectiveness analysis for PSI.

    • 47

      PSI, "Condom DALY Brief," "Condom DALY Model Description," and "Condom DALY Coefficient Worksheet."

    • 48

      PSI, "LLIN DALY Brief."

    • 49

      "The net cost of preventing an HIV infection was $1,310." PSI, "Cost Effectiveness Report (2008)," Pg 18.

    • 50

      "The estimates of cost per disability-adjusted life year (DALY) saved assume a uniform 20 DALYs lost per infected adult (Murray and Lopez 1996) and 25 DALYs lost per infected child (Marseille and others 1999) and do not account for the increasing proportion of people living with HIV/AIDS in developing countries who will have access to antiretroviral therapy over the coming years." Jamison et al. 2006, Pg 344.

    • 51

      PSI, "Cost Effectiveness Report (2008)," Pg 18.

    • 52

      PSI, "Cost Effectiveness Report (2008)," Pg 10.

    • 53

      Mathers, Ezzati, and Lopez (2007, Pg 8, Table 3) implies that each episode of malaria averted counts for 0.1 DALYs, and that 1 in approximately 320 cases (the ratio between deaths and episodes) results in a death and thus an additional 33 DALYs.

    • 54

      PSI, "Cost Effectiveness Report (2008)", Pg 10.

    • 55

      PSI, "Cost Effectiveness Report (2008)", Pg 39.

    • 56

      "The model we developed for the Disease Control Priorities Project produced costs between US$87 and US$163 per birth averted, with costs lowest in Latin America and the Caribbean and in the Middle East and North Africa... The model developed for the Disease Control Priorities Project produced regional average costs between US$5,172 and US$34,564 per maternal death averted...Average costs for regions per YLL ranged from US$37 to US$110. Costs per DALY and year of life lost (YLL) were lowest in South Asia and Sub-Saharan Africa, whereas costs per year lived with disability (YLD) were lowest in South Asia and in East Asia and the Pacific. As with costs per death averted, costs per DALY varied within regions by as much as two orders of magnitude." Jamison et al. 2006, Pg 1084.

    • 57

      "PSI had an average net cost per disability-adjusted life year (DALY) of $28.95 in 2008." PSI, "Cost Effectiveness Report (2008)," Pg 1.

    • 58

      "After adjusting for a 10% condom wastage rate, PSI estimates the number of new infection averted per condom." PSI, "Condom DALY Brief," Pg 2.

    • 59

      "The approach applies to free condom as well except that we assume an additional 25% wastage rate." PSI, "Condom DALY Brief," Pg 4.

    • 60

      PSI, "LLIN DALY Brief," Pg 1.

    • 61

      PSI, "Room for More Funds (July 2010)," Pg 1.

    • 62

      “The relatively small scale of its current family planning programs contributes to a higher cost per DALY for these interventions. Contraceptive prevalence rates have gone from 10 percent in the mid-1960s to about 60 percent today, but more funding is needed to continue the trend. PSI’s strategy is to improve access to modern contraceptive methods and increase their use in countries where rates remain low.” PSI, "Room for More Funds (July 2010)," Pg 1.

    • 63

      "PSI promotes informed access to a full range of modern contraceptive products. In 2011, this will mean significantly increasing access to long-acting contraceptive methods—specifically intra-uterine devices (IUDs) and implants—in countries with a contraceptive prevalence rate of below 40 percent." PSI, "Responses to GiveWell's Follow Up Questions."

    • 64

      "The priority would be as follows:

      • Reproductive Health
      • Child Survival
      • Tuberculosis
      • Pilots in smoking, obesity, and diabetes"

      Kim Longfield, email to GiveWell, October 23, 2010.

    • 65

      Kim Longfield, email to GiveWell, October 23, 2010.

    • 66

      PSI, "Room for More Funds (July 2010)," Pg 2.

    • 67

      PSI, "Room for More Funds (July 2010)," Pg 2. Order of priorities from Kim Longfield, email to GiveWell, October 23, 2010.

    • 68

      "PSI distributed 1 billion of the 8 billion male condoms needed in 2009 to cover all risky sexual acts (source: UNFPA). However, the majority of these are plain lubricated condoms. To increase consistent use of condoms, it’s important to offer consumers, and particularly youth, choice—including condom varieties such as those that are studded, flavored, and colored. These are rarely procured through established funding mechanisms and a funding gap exists for both their procurement and their
      promotion." PSI, "Responses to GiveWell's Follow Up Questions."

    • 69

      "The size of the donation doesn't affect how the money will be used." Kim Longfield, email to GiveWell, October 23, 2010.

    • 70

      Data from PSI, "Cost Effectiveness Report (2007)," and PSI, "Cost Effectiveness Report (2008)," Pg 77-80. Calculations in GiveWell, "PSI Expenses by Product Type."

    1. Attribution, under the “Does it work?” section

    The 2010 Givewell evaluation relied heavily on one specific data review published on PSI’s website. The review tracked standard international and donor indicators for HIV, family planning, child survival, and malaria in countries where PSI works. It contained data from 2005-2010. While it was useful for monitoring some behavior change at a national level, it didn’t capture much of PSI’s work. PSI used the review simply as a monitoring tool. We never intended it to be used to evaluate programs or attribute behavior change to PSI.

    Like other NGOs, PSI develops monitoring and evaluation (M&E) plans that are responsive to government and foundation donor requirements; most donors require a demonstration that we’ve met a set of targets for promised activities. It’s not a typical requirement or expectation among these donors that organizations demonstrate attribution in a rigorous manner: they’re mostly concerned with improvements in standard indicators over time. PSI has found that it’s important to be consistent in tracking those indicators for the government and foundation donors that fund our work.

    Given the scale of PSI’s work in 70 countries and the specificity of many donor requirements, we use cross sectional surveys for examining program exposure. In 2010, the best we could do with the methods at our disposal was look to for associations between any PSI program exposure and behavior change. The results were made available to Givewell by PSI in an additional 82 survey reports; those reports demonstrated that during the 2010 evaluation, 57% of countries demonstrated behavior change that was attributable to PSI programs.

    In 2010, PSI started working with Harvard to improve our evaluation approach for survey data, which includes defining a counter-factual and increasing the rigor by which we can detect measurable differences in behavior change between individuals who did and didn’t receive PSI’s programs. . We also work with external evaluators, including Innovations in Poverty Action (IPA), Jameel Poverty Action Lab (JPAL), and universities like UCSF, and Berkeley .to conduct external evaluations when it’s in the strategic interest of PSI and the global health community.

    2. PSI’s ability to demonstrate impact, under “Monitoring and Evaluation”

    In the 2010 evaluation, GiveWell used one specific (TRaC) survey program to evaluate how PSI estimates health impact (years of healthy life added). While there are some links, the assumptions within our health impact models do not tend to come from the (TRaC) surveys Givewell evaluated. We use data from nationally representative surveys (such as DHS or UNAIDS) so that our models are as internationally consistent and robust as possible.

    3. Use of PSI’s products, under “Are PSI’s products used?”

    GiveWell selected countries with the greatest expenditures in 2008 in three health areas – HIV, malaria, and reproductive health – and then examined one data point on product use. The assumptions and analytical purpose of this section were unclear. In most cases the data for the behavior were from years that preceded 2008. It wasn’t clear why GiveWell would expect levels of behavior to be high in countries that receive a large amount of funding to change those behaviors.

    GiveWell’s 2010 analysis doesn’t answer the question are “PSI products used?” It’s a good question and one that we grapple with. In order to know what percentage of the condoms used are PSI’s brands, you’d need to ask target audiences during surveys. PSI suffers from a “Kleenex complex.” In many countries where we work, PSI brands have become the generic term for the product category. As a result, surveyors need to show respondents photos of all of the products that are available on the market to decrease response bias and isolate the actual brand used. This is a burdensome line of questioning and one that only a few countries have opted to use. As PSI is more interested in a market level analysis, we haven’t required countries to capture data at this level.

    4. Is PSI creating behavior change?

    GiveWell began the 2010 analysis by asking whether PSI is changing behavior broadly and examined PSI’s work in all health areas. Between December 2010 and February 2011, GiveWell narrowed the focus to just the use of condoms, mosquito nets, and contraception. PSI demonstrated effectiveness on several other indicators in the health areas in which we work but this information was not included in the evaluation.

    Changing behavior is very difficult and takes time. The results posted for condom use, mosquito nets, and contraception are actually very encouraging. It’s a big achievement for an organization to be able to attribute changes in behavior to its programs. We also recognize that we need to be practical about how far and how quickly we’ll be able to change the behaviors we target.

    5. Is PSI crowding out the private sector or using subsidies inefficiently? Under, “Possible negative or offsetting impact”

    The points in this section are valid criticisms and, as GiveWell has noted, are issues of concern for PSI too.

    Efficiency, sustainability, and having the appropriate market share in countries where we work are important to us. This is being addressed through our work to evaluate markets for free, subsidized and retail products in the countries where we work.

    6. Concerns about our cost effectiveness measures, under “What do you get for your dollar”

    These are fair criticisms and GiveWell admits, “it’s normal for cost-effectiveness estimates to include many rough estimates and assumptions.” Calculating cost effectiveness is a challenge for any organization. In the past, we relied on reports of level of effort and tried to accommodate cost-effectiveness per product or service.

    In 2011, we stopped estimating cost-effectiveness at the product and service level. We now track costs and calculate cost-effectiveness at the country level, setting costs against the full set of years of healthy life added. This offers a reliable and robust measure of cost-effectiveness based on our routine data systems, i.e. it doesn’t require any modeling or arbitrary assignments of costs. This allows us to make comparisons about cost-effectiveness across countries and we can use modeling to identify the drivers of cost. To date, we have been able to demonstrate that our malaria programs are most cost-effective and that regional differences are important.

    Condom wastage: PSI no longer uses different assumptions about condom wastage for free and sold condoms. We track wastage using surveys and use data to inform our assumptions, but we do not try to accommodate isolated measures of wastage within our models. We now assume 10% wastage for free and sold condoms. There’s no published evidence that rates of wastage differ between free and sold condoms.

    Mosquito Nets: Our models are now standardized to John’s Hopkins’ Lives Saved Tool (LiST), which is supported by the Child Health Epidemiological and Research Group (CHERG), a global community of practice for measuring the impact of interventions that target maternal and child health.. In addition, we use data from DHS or Malaria Indicator Surveys to include the average household size and average number of nets per household to get to the per unit deaths averted by a net.


    Source URL: https://www.givewell.org/international/charities/PSI/February-2011-version