Aggregator

One of our two new top charities this year is Helen Keller International (HKI)’s vitamin A supplementation program. We named HKI’s vitamin A supplementation program a top charity this year because:

• There is strong evidence from many randomized controlled trials of vitamin A supplementation that the program leads to substantial reductions in child deaths.
• HKI-supported vitamin A supplementation programs are inexpensive (we estimate around $0.75 in total costs per supplement delivered) and highly cost-effective at preventing child deaths in countries where HKI plans to work using GiveWell-directed funds.
• HKI is transparent—it has shared significant, detailed information about its programs with us, including the results and methodology of monitoring surveys HKI conducted to determine whether its vitamin A supplementation programs reach a large proportion of targeted children.
• HKI has a funding gap—we believe it is highly likely that its vitamin A supplementation programs will be constrained by funding next year.

HKI’s vitamin A supplementation program is an exceptional giving opportunity, but as with the case for donating to any of our other top charities, not a “sure thing.”

I’m the Research Analyst who has led our work on HKI this year. In this post, I discuss some key questions about the impact of Helen Keller International’s vitamin A supplementation program and what we’ve learned so far. I also discuss GiveWell’s plans for learning more about these issues in the future.

In short:

• Is vitamin A deficiency still a major concern? Our best guess is that vitamin A deficiency is considerably less common today where HKI works than it was among children who participated in past trials of vitamin A supplementation, but not so rare that vitamin A supplementation would not be cost-effective. We are quite uncertain about our estimate of the prevalence of vitamin A deficiency where HKI works because little high-quality, up-to-date data on vitamin A deficiency is available. We plan to consider funding new surveys of vitamin A deficiency to improve our understanding of the effectiveness of HKI’s programs.
• Have improvements in health conditions over time reduced the need for vitamin A supplementation? Child mortality rates remain quite high in areas where HKI plans to use GiveWell-directed funding for vitamin A supplementation programs. We think it’s unlikely that health conditions in these countries have improved enough for vitamin A supplementation to no longer be effective.
• How strong is HKI’s track record of supporting fixed-point vitamin A supplement distributions? HKI expects to primarily support fixed-point vitamin A supplement distributions (rather than door-to-door campaigns) going forward. Results from monitoring surveys have found that, on average, HKI’s fixed-point programs have not reached as high a proportion of targeted populations as its door-to-door programs, but these monitoring surveys may not have been fully representative of HKI’s programs overall. Our best guess is that future fixed-point programs will achieve moderate to high coverage.

Is vitamin A deficiency still a major concern?

Vitamin A deficiency, a condition resulting from chronic low vitamin A intake, can cause loss of vision and increased severity of infections. If vitamin A deficiency is less common today than it was among participants in trials of vitamin A supplementation, today’s programs may prevent fewer deaths than the evidence from the trials suggests.

We estimate that the prevalence of vitamin A deficiency was high (around 60%) in the populations studied in trials included in the Cochrane Collaboration review of vitamin A supplementation programs for preschool-aged children, Imdad et al. 2017.1See the “Imdad 2017 – VAD prevalence estimates” sheet here for details. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

The map below, from Our World in Data, presents the World Health Organization (WHO)’s most recent estimates of the prevalence of vitamin A deficiency among preschool-aged children by country, covering the period from 1995 to 2005. WHO categorizes prevalences of vitamin A deficiency among preschool-aged children of 20% or above as a severe public health problem.2WHO Global prevalence of vitamin A deficiency in populations at risk 2009, Pg 8, Table 5. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Since WHO’s most recent estimates are now considerably out-of-date, we decided to investigate a variety of additional sources in order to create best-guess estimates of rates of vitamin A deficiency today in countries in sub-Saharan Africa where HKI works.

We learned that there is very little useful, up-to-date data on vitamin A deficiency in countries in sub-Saharan Africa. In many countries, the most recent surveys of vitamin A deficiency were completed ten or more years ago. Many governments have also recently mandated the fortification of vegetable oil or other foods with vitamin A, but little information is available on whether foods are actually adequately fortified in practice.3See this spreadsheet for the information we collected on the most recent vitamin A deficiency surveys and on vitamin A fortification programs in countries where HKI has supported vitamin A supplementation programs. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Taking the limited available data into account, our best guess is that prevalence of vitamin A deficiency in countries where HKI works today is likely to be considerably lower than the prevalence of vitamin A deficiency among children who participated in vitamin A supplementation trials—closer to 20% prevalence than 60% prevalence.

We find that HKI’s vitamin A supplementation programs still appear highly cost-effective, even when taking our estimate of the change in the prevalence of vitamin A deficiency over time into account (see our most recent cost-effectiveness analysis for full details). But we remain quite uncertain about our estimate of the prevalence of vitamin A deficiency in countries where HKI works—new information could cause us to update our views on HKI’s cost-effectiveness considerably.

Next year, we’ll continue to follow research relevant to estimating vitamin A deficiency rates where HKI works. We also plan to consider funding new vitamin A deficiency surveys ourselves through a GiveWell Incubation Grant.

Have improvements in health conditions over time reduced the need for vitamin A supplementation?

In a blog post last year, we wrote that vitamin A supplementation has a mixed evidence base. There is strong evidence from many randomized controlled trials conducted in the 1980s and 1990s that the program reduces child mortality, but a more recent trial in northern India with more participants than all the other trials combined (the Deworming and Enhanced Vitamin A trial, or DEVTA) did not find a statistically significant effect.

There have been broad declines in child mortality rates over the past few decades. Participants in the control group in the DEVTA trial had a mortality rate of 5.3 deaths per 1,000 child-years, lower than the mortality rates in the control groups in earlier trials that found statistically significant results (ranging from 10.6 to 126 deaths per 1,000 child-years). One potential explanation for the difference between the results of the DEVTA trial and earlier trials is that the some types of deaths prevented by vitamin A supplementation in previously studied populations had already been prevented through other means (e.g., increased access to immunizations and medical care) in the DEVTA population.

We looked into child mortality rates in countries in sub-Saharan Africa where HKI plans to use GiveWell-directed funding in the near future—Guinea, Burkina Faso, and Mali—as well as other countries where HKI has recently worked. Mortality rates among preschool-aged children in Guinea, Burkina Faso and Mali remain quite high—around 13 deaths per 1,000 child-years, within the range of mortality rates among control groups in vitamin A trials that found statistically significant results.4The control group mortality rate in the DEVTA trial was 5.3 per 1,000 child-years. See this spreadsheet for child mortality rates in Burkina Faso, Guinea, and Mali (13 deaths per 1,000 child-years is the simple average of “Average of GBD and UN IGME data” child mortality rates for the three countries), and see here for more information on control group mortality rates in other vitamin A supplementation trials. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); Based on these high child mortality rates, we don’t believe it’s very likely that overall health conditions have improved enough in these countries for vitamin A supplementation to no longer be effective at preventing deaths.

It is also possible that changes in causes of child deaths between the 1980s and 1990s and today could mean that vitamin A supplementation is now less effective than it was in the past. Different vitamin A experts have different views on whether vitamin A primarily prevents deaths due to a few specific causes (we’ve seen diarrhea and measles most frequently pointed to) or whether it reduces deaths due to a wider range of conditions by, perhaps, strengthening the immune system against infection. In our view, the research on this is inconclusive. According to the data we’ve seen, infectious disease overall and diarrhea in particular cause a similar proportion of total deaths among young children today as they did in the 1980s and 1990s; measles causes a substantially lower proportion of total deaths today than it did in the past.5See the final bullet point in this section of our review of HKI for more on this topic. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); We’ve added an adjustment to our cost-effectiveness analysis to account for changes in the composition of causes of child mortality since the vitamin A trials were implemented—HKI’s work still appears highly cost-effective following this adjustment.

We may conduct additional research next year to learn about child mortality rates in places where HKI works at a more granular (e.g., regional or sub-regional) level. We may also conduct additional research on the impact of changes in cause-specific mortality rates on the effectiveness of vitamin A supplementation.

How strong is HKI’s track record of supporting fixed-point vitamin A supplement distributions?

In many past HKI-supported campaigns, healthcare workers have traveled door-to-door to administer vitamin A supplements to preschool-aged children. Funding was already available from other sources for sending teams of healthcare workers door-to-door to administer polio vaccinations, and adding vitamin A supplementation to these campaigns was relatively simple and cheap.

In fixed-point distributions, caregivers are expected to bring their children to a central location to receive vitamin A supplements. Due to recent progress in polio elimination, many door-to-door programs have recently been scaled-down or eliminated, so HKI expects to primarily be supporting fixed-point distributions going forward.

It may be more challenging to reach a large proportion of a targeted population with fixed-point distributions. HKI’s recent monitoring surveys have found that, on average, its door-to-door distributions have achieved higher coverage rates (around 90%) than its fixed-point distributions (around 60%). The average of around 60% for fixed-point programs reflects surveys finding high coverage in a few campaigns in the Democratic Republic of the Congo and Mozambique, and relatively low coverage in campaigns in Nigeria, Tanzania, and Kenya.

A complication for assessing HKI’s track record is that HKI often chose to conduct coverage surveys in areas where it expected coverage to be particularly low, so we would guess that these results are not fully representative of HKI’s work on fixed-point distributions.

Based on the available information, our best guess is that HKI-supported fixed-point vitamin A supplementation distributions next year will achieve moderate to high coverage.6To be more precise about what I mean: in Guinea (the program I am most familiar with, following our site visit in October), I’m 70% confident that coverage surveys representative of the distribution as a whole will indicate that the first vitamin A supplement distribution in 2018 reached at least 55% of targeted children across the country. jQuery("#footnote_plugin_tooltip_6").tooltip({ tip: "#footnote_plugin_tooltip_text_6", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); HKI has told us that it will conduct representative monitoring surveys (not only in areas where it expects coverage to be low) following its vitamin A supplement distributions supported with GiveWell-directed funding next year—we expect that these surveys will provide data useful for assessing how successful the programs were overall.

Notes   [ + ]

1. ↑ See the “Imdad 2017 – VAD prevalence estimates” sheet here for details. 2. ↑ WHO Global prevalence of vitamin A deficiency in populations at risk 2009, Pg 8, Table 5. 3. ↑ See this spreadsheet for the information we collected on the most recent vitamin A deficiency surveys and on vitamin A fortification programs in countries where HKI has supported vitamin A supplementation programs. 4. ↑ The control group mortality rate in the DEVTA trial was 5.3 per 1,000 child-years. See this spreadsheet for child mortality rates in Burkina Faso, Guinea, and Mali (13 deaths per 1,000 child-years is the simple average of “Average of GBD and UN IGME data” child mortality rates for the three countries), and see here for more information on control group mortality rates in other vitamin A supplementation trials. 5. ↑ See the final bullet point in this section of our review of HKI for more on this topic. 6. ↑ To be more precise about what I mean: in Guinea (the program I am most familiar with, following our site visit in October), I’m 70% confident that coverage surveys representative of the distribution as a whole will indicate that the first vitamin A supplement distribution in 2018 reached at least 55% of targeted children across the country. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Key questions about Helen Keller International’s vitamin A supplementation program appeared first on The GiveWell Blog.

When our cost-effectiveness analysis finds robust and meaningful differences between charities, it plays a large role in our recommendations (more on the role it plays in this post).

But while our cost-effectiveness analysis represent our best guess, it’s also subject to substantial uncertainty; some of its results are a function of highly debatable, difficult-to-estimate inputs.

Sometimes these inputs are largely subjective, such as the moral weight we assign to charities achieving different good outcomes (e.g. improving health vs increasing income). But even objective inputs are uncertain; a key input for anti-malaria interventions is malaria mortality, but the Institute for Health Metrics and Evaluation estimates 1.6 times more people died in Africa from malaria in 2016 (641,000) than the World Health Organization does (407,000; pg. 41).1Differences in their methodology have been discussed, with older figures, in a 2012 blog post by the Center for Global Development. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Before we finalized the recommendations we released in November, we determined how sensitive our results were to some of our most uncertain parameters.

In brief:

• Comparisons between charities achieving different types of good outcome are most sensitive to the relative value we assign to those outcomes (more on how and why we and other policymakers assign these weights in this post).
• Our deworming models are very uncertain, due to the complexity of the evidence base. They are also sensitive to the choice of discount rate: how we value good done today vs. good done in the future.
• Our malaria models (seasonal malaria chemoprevention and long-lasting insecticide-treated nets) are less uncertain than our deworming models, but are particularly sensitive to our estimate of the long term effects of malaria on income.

In this post, we discuss:

• The sensitivity of our analysis to moral weights (more) and other parameters (more).
• How this uncertainty influences our recommendations (more).
• Why this sensitivity analysis doesn’t capture the full scope of our uncertainty and ways in which we could improve our assessment and presentation of uncertainty (more).

The tornado charts at the bottom of this post show the results of our full sensitivity analysis. For a brief explanation of how we conducted our sensitivity analysis see this footnote.2Each contributor to our cost-effectiveness analysis inputs their own values for particularly uncertain parameters in our cost-effectiveness analysis. We use the median of contributors’ final cost-effectiveness results for our headline cost-effectiveness figures. To simplify the sensitivity analysis, we used the median of contributors’ parameter inputs to form a central cost-effectiveness estimate for each charity. The results below therefore differ slightly from our headline cost-effectiveness figures. To determine how sensitive the model is to each parameter, we flexed each parameter between the highest and lowest contributors’ inputs, while holding all other parameters constant. For more details, see our sensitivity analysis spreadsheet. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Sensitivity to moral weights

Some of the inputs in our model rely on judgement calls, which reasonable, informed people might disagree on. For example, we assign quantitative weights to our relative valuations of different good outcomes. These inputs capture important considerations in our decision-making, but are often difficult to justify precisely.

We ask contributors to our cost-effectiveness analysis (mostly staff) to input how many people’s income would have to double for 1 year to be equally valuable to averting the death of a child under 5. Contributors’ values vary widely, between 8 and 100 (see Figure 1).3You can see each of our contributors’ inputs for moral weights, and other uncertain parameters, on the Moral weights and Parameters tabs of our cost-effectiveness analysis. This year, contributors were also asked to provide a brief justification for their inputs in the cell notes. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Differences in cost-effectiveness between charities which primarily prevent child deaths (Helen Keller International, Malaria Consortium, Against Malaria Foundation) and charities which primarily increase income (Deworm the World Initiative, Schistosomiasis Control Initiative, Sightsavers, No Lean Season, End Fund) are highly sensitive to different plausible moral weights (See Figure 2).

The orange points represent the median estimated cost-effectiveness of our charities (in terms of how many times more cost-effective than GiveDirectly we model them to be). The blue bars represents the range of cost-effectiveness for different valuations of preventing the death of an under-5 child between 8x and 100x as good as doubling consumption for one person for one year (holding all other parameters in the model constant). Deworming sensitivities

Our deworming models are very uncertain, due to the complexity of the evidence base, and the long time horizons over which we expect the potential benefits to be realized. Aside from our moral weights, our deworming charities are highly sensitive to three inputs:

• Replicability adjustment. We make a “replicability adjustment” for deworming to account for the fact that the consumption increase in a major study we rely upon may not hold up if it were replicated. If you’re skeptical that such a large income increase would occur, given the limited evidence for short-term health benefits and generally unexpected nature of the findings, you may think that the effect the study measured wasn’t real, wasn’t driven by deworming, or relied on an atypical characteristic shared by the study population but not likely to found among recipients of the intervention today. This adjustment is not well-grounded in data. (For more discussion see our deworming intervention report and blog posts here, here, here and here).4You can read more about how contributors settled on the values they used for this parameter in the cell notes in row 16 of the Parameters sheet of our November 2017 cost-effectiveness model. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• Adjustment for years of treatment in Baird et al. vs. years of treatment in charities’ programs. Our charities aim to deworm children for up to 10 years, which is longer than the intervention studies in Baird et al. 2015 (where children in the treatment group received 4 years of deworming). There may be diminishing returns as the total years of treatment increase, although this is difficult to estimate.
• Discount rate. The discount rate adjusts for benefits that occur at different points in time. For a number of reasons, individuals may believe it is preferable for income to rise now than at some point in the future.

Figure 3 shows how the cost-effectiveness of Deworm the World Initiative5The sensitivity of other deworming charities is largely dependent on the same parameters. Charts are presented in the Appendix jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); varies depending on different contributor inputs for different parameters (more on how to interpret these parameters here).

The orange line represents the median estimated cost-effectiveness of our charities (in terms of how many times more cost-effective than GiveDirectly we model them to be). The blue bars represents the range of cost-effectiveness for different inputs from our contributors for that parameter (holding all other parameters in the model constant). The figures in square brackets represent the range of contributor inputs for those parameters. Malaria sensitivities

Our malaria models are less uncertain than our deworming models, but are still sensitive to our estimate of the long term effects of malaria on income (see Figures 4 and 5).

Interpreting the evidence base for the effect of malaria prevention on long run income is complex, and contributors differ widely in their interpretation. We’re planning to do more research on this topic further but summarize our current understanding here.

What does this mean for our recommendations?

When we model large differences in cost-effectiveness, we generally follow those recommendations. When charities are closer on cost-effectiveness, we pay more attention to qualitative considerations, such as the quality of their monitoring and evaluation, and potential upside benefits which are difficult to quantify (e.g. scaling a novel program).

What counts as a meaningful difference in modelled cost-effectiveness depends on a number of factors, including:

• Do the programs target the same outcomes? We place less weight on modelled differences between charities which have different good outcomes because our cost-effectiveness analysis is sensitive to different reasonable moral weights.
• How similar are the programs? We’re more confident in our comparison between our deworming charities than we are between deworming charities and other charities targeting income such as GiveDirectly. This is because we expect the most likely errors in our deworming estimates (e.g. based on our interpretation of the evidence) for different charities to be correlated.
• Are there important qualitative reasons to differentiate between the charities? We place less relative weight on cost-effectiveness analysis when there are important qualitative reasons to differentiate between charities.

For a more detailed explanation of how we made our recommendations this year, see our recent announcement of our top charities for giving season 2017.

What are the limitations of this sensitivity analysis?

This sensitivity analysis shouldn’t be taken as a full representation of our all things considered uncertainty:

• The charts above show the sensitivity of the cost-effectiveness analysis to changing one input at a time (holding all other constant). The ranges don’t necessarily imply any particular credible interval, and are more useful for identifying which inputs are most uncertain than for reflecting our all things considered uncertainty around the cost-effectiveness of a particular charity.
• We don’t ask multiple contributors to input their own values for all uncertain inputs (e.g. because we think the benefits of using the inputs of the contributors with most context outweigh the benefit of getting inputs from many contributors). These inputs have not been included in the sensitivity analysis.
• Model uncertainty. Explicitly modelling all the considerations relevant to our charity would be infeasible. Even if all our inputs were fully accurate, we’d still retain some uncertainty about the true cost-effectiveness of our charities.

We’re considering a number of different options to improve our sensitivity analysis and communication of uncertainty in the future, such as expressing inputs as probability distributions or creating a Monte Carlo simulation. But we’re uncertain whether these would create sufficient decision-relevant information for our readers to justify the substantial time investment and additional complexity.

If you’d find such an analysis helpful, let us know in the comments.

Appendix

In this section, we present tornado charts for each of our top charities. You can see more detailed descriptions of how to interpret these parameters here, or in the cell notes of our cost-effectiveness analysis.

Notes   [ + ]

1. ↑ Differences in their methodology have been discussed, with older figures, in a 2012 blog post by the Center for Global Development. 2. ↑ Each contributor to our cost-effectiveness analysis inputs their own values for particularly uncertain parameters in our cost-effectiveness analysis. We use the median of contributors’ final cost-effectiveness results for our headline cost-effectiveness figures. To simplify the sensitivity analysis, we used the median of contributors’ parameter inputs to form a central cost-effectiveness estimate for each charity. The results below therefore differ slightly from our headline cost-effectiveness figures. To determine how sensitive the model is to each parameter, we flexed each parameter between the highest and lowest contributors’ inputs, while holding all other parameters constant. For more details, see our sensitivity analysis spreadsheet. 3. ↑ You can see each of our contributors’ inputs for moral weights, and other uncertain parameters, on the Moral weights and Parameters tabs of our cost-effectiveness analysis. This year, contributors were also asked to provide a brief justification for their inputs in the cell notes. 4. ↑ You can read more about how contributors settled on the values they used for this parameter in the cell notes in row 16 of the Parameters sheet of our November 2017 cost-effectiveness model. 5. ↑ The sensitivity of other deworming charities is largely dependent on the same parameters. Charts are presented in the Appendix function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post How uncertain is our cost-effectiveness analysis? appeared first on The GiveWell Blog.

When our cost-effectiveness analysis finds robust and meaningful differences between charities, it plays a large role in our recommendations (more on the role it plays in this post).

But while our cost-effectiveness analysis represent our best guess, it’s also subject to substantial uncertainty; some of its results are a function of highly debatable, difficult-to-estimate inputs.

Sometimes these inputs are largely subjective, such as the moral weight we assign to charities achieving different good outcomes (e.g. improving health vs increasing income). But even objective inputs are uncertain; a key input for anti-malaria interventions is malaria mortality, but the Institute for Health Metrics and Evaluation estimates 1.6 times more people died in Africa from malaria in 2016 (641,000) than the World Health Organization does (407,000; pg. 41).1Differences in their methodology have been discussed, with older figures, in a 2012 blog post by the Center for Global Development. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Before we finalized the recommendations we released in November, we determined how sensitive our results were to some of our most uncertain parameters.

In brief:

• Comparisons between charities achieving different types of good outcome are most sensitive to the relative value we assign to those outcomes (more on how and why we and other policymakers assign these weights in this post).
• Our deworming models are very uncertain, due to the complexity of the evidence base. They are also sensitive to the choice of discount rate: how we value good done today vs. good done in the future.
• Our malaria models (seasonal malaria chemoprevention and long-lasting insecticide-treated nets) are less uncertain than our deworming models, but are particularly sensitive to our estimate of the long term effects of malaria on income.

In this post, we discuss:

• The sensitivity of our analysis to moral weights (more) and other parameters (more).
• How this uncertainty influences our recommendations (more).
• Why this sensitivity analysis doesn’t capture the full scope of our uncertainty and ways in which we could improve our assessment and presentation of uncertainty (more).

The tornado charts at the bottom of this post show the results of our full sensitivity analysis. For a brief explanation of how we conducted our sensitivity analysis see this footnote.2Each contributor to our cost-effectiveness analysis inputs their own values for particularly uncertain parameters in our cost-effectiveness analysis. We use the median of contributors’ final cost-effectiveness results for our headline cost-effectiveness figures. To simplify the sensitivity analysis, we used the median of contributors’ parameter inputs to form a central cost-effectiveness estimate for each charity. The results below therefore differ slightly from our headline cost-effectiveness figures. To determine how sensitive the model is to each parameter, we flexed each parameter between the highest and lowest contributors’ inputs, while holding all other parameters constant. For more details, see our sensitivity analysis spreadsheet. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Sensitivity to moral weights

Some of the inputs in our model rely on judgement calls, which reasonable, informed people might disagree on. For example, we assign quantitative weights to our relative valuations of different good outcomes. These inputs capture important considerations in our decision-making, but are often difficult to justify precisely.

We ask contributors to our cost-effectiveness analysis (mostly staff) to input how many people’s income would have to double for 1 year to be equally valuable to averting the death of a child under 5. Contributors’ values vary widely, between 8 and 100 (see Figure 1).3You can see each of our contributors’ inputs for moral weights, and other uncertain parameters, on the Moral weights and Parameters tabs of our cost-effectiveness analysis. This year, contributors were also asked to provide a brief justification for their inputs in the cell notes. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

Differences in cost-effectiveness between charities which primarily prevent child deaths (Helen Keller International, Malaria Consortium, Against Malaria Foundation) and charities which primarily increase income (Deworm the World Initiative, Schistosomiasis Control Initiative, Sightsavers, No Lean Season, End Fund) are highly sensitive to different plausible moral weights (See Figure 2).

The orange points represent the median estimated cost-effectiveness of our charities (in terms of how many times more cost-effective than GiveDirectly we model them to be). The blue bars represents the range of cost-effectiveness for different valuations of preventing the death of an under-5 child between 8x and 100x as good as doubling consumption for one person for one year (holding all other parameters in the model constant). Deworming sensitivities

Our deworming models are very uncertain, due to the complexity of the evidence base, and the long time horizons over which we expect the potential benefits to be realized. Aside from our moral weights, our deworming charities are highly sensitive to three inputs:

• Replicability adjustment. We make a “replicability adjustment” for deworming to account for the fact that the consumption increase in a major study we rely upon may not hold up if it were replicated. If you’re skeptical that such a large income increase would occur, given the limited evidence for short-term health benefits and generally unexpected nature of the findings, you may think that the effect the study measured wasn’t real, wasn’t driven by deworming, or relied on an atypical characteristic shared by the study population but not likely to found among recipients of the intervention today. This adjustment is not well-grounded in data. (For more discussion see our deworming intervention report and blog posts here, here, here and here).4You can read more about how contributors settled on the values they used for this parameter in the cell notes in row 16 of the Parameters sheet of our November 2017 cost-effectiveness model. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
• Adjustment for years of treatment in Baird et al. vs. years of treatment in charities’ programs. Our charities aim to deworm children for up to 10 years, which is longer than the intervention studies in Baird et al. 2015 (where children in the treatment group received 4 years of deworming). There may be diminishing returns as the total years of treatment increase, although this is difficult to estimate.
• Discount rate. The discount rate adjusts for benefits that occur at different points in time. For a number of reasons, individuals may believe it is preferable for income to rise now than at some point in the future.

Figure 3 shows how the cost-effectiveness of Deworm the World Initiative5The sensitivity of other deworming charities is largely dependent on the same parameters. Charts are presented in the Appendix jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); varies depending on different contributor inputs for different parameters (more on how to interpret these parameters here).

The orange line represents the median estimated cost-effectiveness of our charities (in terms of how many times more cost-effective than GiveDirectly we model them to be). The blue bars represents the range of cost-effectiveness for different inputs from our contributors for that parameter (holding all other parameters in the model constant). The figures in square brackets represent the range of contributor inputs for those parameters. Malaria sensitivities

Our malaria models are less uncertain than our deworming models, but are still sensitive to our estimate of the long term effects of malaria on income (see Figures 4 and 5).

Interpreting the evidence base for the effect of malaria prevention on long run income is complex, and contributors differ widely in their interpretation. We’re planning to do more research on this topic further but summarize our current understanding here.

What does this mean for our recommendations?

When we model large differences in cost-effectiveness, we generally follow those recommendations. When charities are closer on cost-effectiveness, we pay more attention to qualitative considerations, such as the quality of their monitoring and evaluation, and potential upside benefits which are difficult to quantify (e.g. scaling a novel program).

What counts as a meaningful difference in modelled cost-effectiveness depends on a number of factors, including:

• Do the programs target the same outcomes? We place less weight on modelled differences between charities which have different good outcomes because our cost-effectiveness analysis is sensitive to different reasonable moral weights.
• How similar are the programs? We’re more confident in our comparison between our deworming charities than we are between deworming charities and other charities targeting income such as GiveDirectly. This is because we expect the most likely errors in our deworming estimates (e.g. based on our interpretation of the evidence) for different charities to be correlated.
• Are there important qualitative reasons to differentiate between the charities? We place less relative weight on cost-effectiveness analysis when there are important qualitative reasons to differentiate between charities.

For a more detailed explanation of how we made our recommendations this year, see our recent announcement of our top charities for giving season 2017.

What are the limitations of this sensitivity analysis?

This sensitivity analysis shouldn’t be taken as a full representation of our all things considered uncertainty:

• The charts above show the sensitivity of the cost-effectiveness analysis to changing one input at a time (holding all other constant). The ranges don’t necessarily imply any particular credible interval, and are more useful for identifying which inputs are most uncertain than for reflecting our all things considered uncertainty around the cost-effectiveness of a particular charity.
• We don’t ask multiple contributors to input their own values for all uncertain inputs (e.g. because we think the benefits of using the inputs of the contributors with most context outweigh the benefit of getting inputs from many contributors). These inputs have not been included in the sensitivity analysis.
• Model uncertainty. Explicitly modelling all the considerations relevant to our charity would be infeasible. Even if all our inputs were fully accurate, we’d still retain some uncertainty about the true cost-effectiveness of our charities.

We’re considering a number of different options to improve our sensitivity analysis and communication of uncertainty in the future, such as expressing inputs as probability distributions or creating a Monte Carlo simulation. But we’re uncertain whether these would create sufficient decision-relevant information for our readers to justify the substantial time investment and additional complexity.

If you’d find such an analysis helpful, let us know in the comments.

Appendix

In this section, we present tornado charts for each of our top charities. You can see more detailed descriptions of how to interpret these parameters here, or in the cell notes of our cost-effectiveness analysis.

Notes   [ + ]

1. ↑ Differences in their methodology have been discussed, with older figures, in a 2012 blog post by the Center for Global Development. 2. ↑ Each contributor to our cost-effectiveness analysis inputs their own values for particularly uncertain parameters in our cost-effectiveness analysis. We use the median of contributors’ final cost-effectiveness results for our headline cost-effectiveness figures. To simplify the sensitivity analysis, we used the median of contributors’ parameter inputs to form a central cost-effectiveness estimate for each charity. The results below therefore differ slightly from our headline cost-effectiveness figures. To determine how sensitive the model is to each parameter, we flexed each parameter between the highest and lowest contributors’ inputs, while holding all other parameters constant. For more details, see our sensitivity analysis spreadsheet. 3. ↑ You can see each of our contributors’ inputs for moral weights, and other uncertain parameters, on the Moral weights and Parameters tabs of our cost-effectiveness analysis. This year, contributors were also asked to provide a brief justification for their inputs in the cell notes. 4. ↑ You can read more about how contributors settled on the values they used for this parameter in the cell notes in row 16 of the Parameters sheet of our November 2017 cost-effectiveness model. 5. ↑ The sensitivity of other deworming charities is largely dependent on the same parameters. Charts are presented in the Appendix function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post How uncertain is our cost-effectiveness analysis? appeared first on The GiveWell Blog.

GiveWell’s impact is a function of the quality of our research and the amount of money we direct to our recommended charities (our “money moved”). Historically, we’ve focused mostly on research because we felt that the quality of our recommendations was a greater constraint to our impact than our money moved.

This has changed. Outreach is now a major organizational priority. The goal of this work is to increase the amount of money we direct to our top-recommended charities.

In April 2014 I wrote about our work on outreach to explain why we hadn’t prioritized it: in brief, our growth had largely been driven by inbound interest in GiveWell, and proactive outreach efforts (beyond building relationships with existing donors) hadn’t yielded results that were worth the cost.

What changed?

• We believe that the amount of money we move is now a greater constraint to our impact than additional improvements in the quality of our research. Over the last two years, we’ve added five new top charities (three of which implement programs that weren’t previously represented on our top charities list), and we expect that our top charities, collectively, will have more than $200 million in unfilled funding gaps once they’ve received the funding that we expect to direct to them. (This calculation excludes GiveDirectly, which we believe could absorb and distribute hundreds of millions of dollars.) At the same time, the quality of our research and our capacity for research is higher than it’s ever been, so the returns to adding staff there (in terms of the pace at which we identify significantly better giving opportunities) are now lower.
• Increased capacity for outreach. In our 2014 post, we wrote that one of our key constraints was that senior staff (which at the time meant primarily GiveWell Co-Founder Holden Karnofsky and me) were necessary for most outreach-related work. This has changed. We now have capacity to take on outreach work as other staff have been hired and trained on this type of work.
• Better information on the impact of GiveWell’s outreach. We now have better information about the returns to outreach because:
  1. We’ve collected better data (via an improved donations processing system and outreach efforts) about where donors find out about us. Because of our ability to track donors, we know that a single appearance on NPR or major podcasts tends to drive $50,000+ in annual donations.
  2. More time passing has demonstrated that the lifetime value of the donations of a first time donor is higher than we expected. In several cases, we’ve seen major donors (i.e., those giving $10,000-$100,000) increase their annual giving by a factor of 10 or more.

We’re in the early stages of figuring out how we can proactively invest time and money in outreach to significantly increase our money moved. For now, we’ve taken some opportunities that we think will have positive returns; these are the three that we’ve invested the most time and money in to date:

• Podcast advertising. We’ve been advertising on podcasts that we believe our target audience listens to, based on interviews with current donors and GiveWell staff. In February and March, we ran a small experiment with a few ads on FiveThirtyEight’s Politics podcast and Vox’s The Weeds.1We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  In total, we spent approximately $20,000 on ads for this initial experiment. We ask donors who give via our website to tell us where they learned about GiveWell when they donate. GiveWell received approximately $8,000 in donations between February 1 and November 20 from donors who reported that they had learned about us via these podcasts.

  The donations we received were from first-time donors; to assess the impact of our advertising, we need to estimate the lifetime value of acquiring a new donor. In work we’ve done to assess our retention rate, we’ve seen that (a) approximately 20-25% of the donors who make a first-time donation of less than $1,000 give again in the subsequent year but (b) because many first-time donors increase the size of their donation over time, collectively, the donors who recur give more than 100% of the value of what they give in their first year.

  At higher donation levels ($1,000-$100,000), we measure 40-45% retention among donors, which leads to retention of approximately two-thirds of dollars given.2I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  We therefore estimated the net present value of expected future donations (over the next five years) from these podcasts ads as somewhere between approximately $20,000 (assuming two-thirds dollar retention for the first two years and 100% dollar retention subsequently) and $45,000 (assuming 100% dollar retention).3We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  A few additional facts are worth keeping in mind about the above figures:

  • We ran this experiment in February and March; most donors give at the end of the calendar year. We consistently see donors who find out about GiveWell during the course of the year, but donate in December. Other things equal, we expect that our advertising would have had greater measured returns in December than earlier in the year.
  • We are only able to track donors who (a) fill out our donation form telling us where they learned about us and (b) give directly through our website rather than to our top charities. Less than 50% of donors who give via credit card (and a smaller percentage of donors who give via check) tell us where they learned about GiveWell. Also, roughly speaking, approximately 50% of the donors and dollars we influence come through GiveWell rather than going to our top charities.4I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
  • It’s certainly possible that donors who learn about us via podcast would be more likely to give through our website than an average donor, more likely to report on how they found us (since their source is clear), or less likely to be retained. My best guess is that donors who learn about us via podcast ads behave similarly to our other donors, but I won’t be surprised if they don’t.

  With all that in mind, I believe that the impact of our podcast advertising is higher than what we directly measured.

  The results we saw from February to November this year were promising enough that we decided to increase the size of our experiment by spending approximately $100,000 on podcast ads. We’re currently running ads on FiveThirtyEight’s Politics podcast and Ezra Klein’s podcast and The Weeds at Vox.

• Earned media outreach. Mentions of GiveWell in the media have historically been a strong driver of growth. We aimed to increase mentions of GiveWell in high-quality, high-profile media where we’ve had the most past success as measured by dollars donated (i.e., media like The New York Times, NPR, The Wall Street Journal, and Financial Times). We retained a PR firm that came strongly recommended; we also increased 1-to-1 outreach by GiveWell staff to members of the media who have covered GiveWell in the past. It’s very hard to attribute the impact of the additional effort we’ve invested—overall, our effort has been fairly limited, and it’s hard to easily draw the causal lines between our work and the stories that appear—but my guess is that our increased efforts have led to more coverage of GiveWell and our top charities this giving season than in the recent past.
• Website improvements. Companies that sell products online invest significant effort into optimizing their websites and checkout pages to maximize their revenues. We retained a marketing consultant, Will Wong of Mission Street, and we’ve been A/B testing different donation pages and plan to test other pages on our website such as our homepage or top charities page to see whether we can increase our conversion rate (i.e., the percentage of visitors to our website who give to one of our top charities). For context, our current conversion rate is 1%. Our understanding is that a standard conversion rate for e-commerce companies is 2%, and that international nonprofits have a similar conversion rate.5See Pg 51 of the study downloadable here. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); An increase in our conversion rate to the industry average would lead to a significant increase in the amount of money we direct to our top charities.

Notes   [ + ]

1. ↑ We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. 2. ↑ I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. 3. ↑ We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. 4. ↑ I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. 5. ↑ See Pg 51 of the study downloadable here. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Update on our work on outreach appeared first on The GiveWell Blog.

GiveWell’s impact is a function of the quality of our research and the amount of money we direct to our recommended charities (our “money moved”). Historically, we’ve focused mostly on research because we felt that the quality of our recommendations was a greater constraint to our impact than our money moved.

This has changed. Outreach is now a major organizational priority. The goal of this work is to increase the amount of money we direct to our top-recommended charities.

In April 2014 I wrote about our work on outreach to explain why we hadn’t prioritized it: in brief, our growth had largely been driven by inbound interest in GiveWell, and proactive outreach efforts (beyond building relationships with existing donors) hadn’t yielded results that were worth the cost.

What changed?

• We believe that the amount of money we move is now a greater constraint to our impact than additional improvements in the quality of our research. Over the last two years, we’ve added five new top charities (three of which implement programs that weren’t previously represented on our top charities list), and we expect that our top charities, collectively, will have more than $200 million in unfilled funding gaps once they’ve received the funding that we expect to direct to them. (This calculation excludes GiveDirectly, which we believe could absorb and distribute hundreds of millions of dollars.) At the same time, the quality of our research and our capacity for research is higher than it’s ever been, so the returns to adding staff there (in terms of the pace at which we identify significantly better giving opportunities) are now lower.
• Increased capacity for outreach. In our 2014 post, we wrote that one of our key constraints was that senior staff (which at the time meant primarily GiveWell Co-Founder Holden Karnofsky and me) were necessary for most outreach-related work. This has changed. We now have capacity to take on outreach work as other staff have been hired and trained on this type of work.
• Better information on the impact of GiveWell’s outreach. We now have better information about the returns to outreach because:
  1. We’ve collected better data (via an improved donations processing system and outreach efforts) about where donors find out about us. Because of our ability to track donors, we know that a single appearance on NPR or major podcasts tends to drive $50,000+ in annual donations.
  2. More time passing has demonstrated that the lifetime value of the donations of a first time donor is higher than we expected. In several cases, we’ve seen major donors (i.e., those giving $10,000-$100,000) increase their annual giving by a factor of 10 or more.

We’re in the early stages of figuring out how we can proactively invest time and money in outreach to significantly increase our money moved. For now, we’ve taken some opportunities that we think will have positive returns; these are the three that we’ve invested the most time and money in to date:

• Podcast advertising. We’ve been advertising on podcasts that we believe our target audience listens to, based on interviews with current donors and GiveWell staff. In February and March, we ran a small experiment with a few ads on FiveThirtyEight’s Politics podcast and Vox’s The Weeds.1We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  In total, we spent approximately $20,000 on ads for this initial experiment. We ask donors who give via our website to tell us where they learned about GiveWell when they donate. GiveWell received approximately $8,000 in donations between February 1 and November 20 from donors who reported that they had learned about us via these podcasts.

  The donations we received were from first-time donors; to assess the impact of our advertising, we need to estimate the lifetime value of acquiring a new donor. In work we’ve done to assess our retention rate, we’ve seen that (a) approximately 20-25% of the donors who make a first-time donation of less than $1,000 give again in the subsequent year but (b) because many first-time donors increase the size of their donation over time, collectively, the donors who recur give more than 100% of the value of what they give in their first year.

  At higher donation levels ($1,000-$100,000), we measure 40-45% retention among donors, which leads to retention of approximately two-thirds of dollars given.2I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  We therefore estimated the net present value of expected future donations (over the next five years) from these podcasts ads as somewhere between approximately $20,000 (assuming two-thirds dollar retention for the first two years and 100% dollar retention subsequently) and $45,000 (assuming 100% dollar retention).3We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  A few additional facts are worth keeping in mind about the above figures:

  • We ran this experiment in February and March; most donors give at the end of the calendar year. We consistently see donors who find out about GiveWell during the course of the year, but donate in December. Other things equal, we expect that our advertising would have had greater measured returns in December than earlier in the year.
  • We are only able to track donors who (a) fill out our donation form telling us where they learned about us and (b) give directly through our website rather than to our top charities. Less than 50% of donors who give via credit card (and a smaller percentage of donors who give via check) tell us where they learned about GiveWell. Also, roughly speaking, approximately 50% of the donors and dollars we influence come through GiveWell rather than going to our top charities.4I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
  • It’s certainly possible that donors who learn about us via podcast would be more likely to give through our website than an average donor, more likely to report on how they found us (since their source is clear), or less likely to be retained. My best guess is that donors who learn about us via podcast ads behave similarly to our other donors, but I won’t be surprised if they don’t.

  With all that in mind, I believe that the impact of our podcast advertising is higher than what we directly measured.

  The results we saw from February to November this year were promising enough that we decided to increase the size of our experiment by spending approximately $100,000 on podcast ads. We’re currently running ads on FiveThirtyEight’s Politics podcast and Ezra Klein’s podcast and The Weeds at Vox.

• Earned media outreach. Mentions of GiveWell in the media have historically been a strong driver of growth. We aimed to increase mentions of GiveWell in high-quality, high-profile media where we’ve had the most past success as measured by dollars donated (i.e., media like The New York Times, NPR, The Wall Street Journal, and Financial Times). We retained a PR firm that came strongly recommended; we also increased 1-to-1 outreach by GiveWell staff to members of the media who have covered GiveWell in the past. It’s very hard to attribute the impact of the additional effort we’ve invested—overall, our effort has been fairly limited, and it’s hard to easily draw the causal lines between our work and the stories that appear—but my guess is that our increased efforts have led to more coverage of GiveWell and our top charities this giving season than in the recent past.
• Website improvements. Companies that sell products online invest significant effort into optimizing their websites and checkout pages to maximize their revenues. We retained a marketing consultant, Will Wong of Mission Street, and we’ve been A/B testing different donation pages and plan to test other pages on our website such as our homepage or top charities page to see whether we can increase our conversion rate (i.e., the percentage of visitors to our website who give to one of our top charities). For context, our current conversion rate is 1%. Our understanding is that a standard conversion rate for e-commerce companies is 2%, and that international nonprofits have a similar conversion rate.5See Pg 51 of the study downloadable here. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); An increase in our conversion rate to the industry average would lead to a significant increase in the amount of money we direct to our top charities.

Notes   [ + ]

1. ↑ We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. 2. ↑ I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. 3. ↑ We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. 4. ↑ I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. 5. ↑ See Pg 51 of the study downloadable here. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Update on our work on outreach appeared first on The GiveWell Blog.

GiveWell’s impact is a function of the quality of our research and the amount of money we direct to our recommended charities (our “money moved”). Historically, we’ve focused mostly on research because we felt that the quality of our recommendations was a greater constraint to our impact than our money moved.

This has changed. Outreach is now a major organizational priority. The goal of this work is to increase the amount of money we direct to our top-recommended charities.

In April 2014 I wrote about our work on outreach to explain why we hadn’t prioritized it: in brief, our growth had largely been driven by inbound interest in GiveWell, and proactive outreach efforts (beyond building relationships with existing donors) hadn’t yielded results that were worth the cost.

What changed?

• We believe that the amount of money we move is now a greater constraint to our impact than additional improvements in the quality of our research. Over the last two years, we’ve added five new top charities (three of which implement programs that weren’t previously represented on our top charities list), and we expect that our top charities, collectively, will have more than $200 million in unfilled funding gaps once they’ve received the funding that we expect to direct to them. (This calculation excludes GiveDirectly, which we believe could absorb and distribute hundreds of millions of dollars.) At the same time, the quality of our research and our capacity for research is higher than it’s ever been, so the returns to adding staff there (in terms of the pace at which we identify significantly better giving opportunities) are now lower.
• Increased capacity for outreach. In our 2014 post, we wrote that one of our key constraints was that senior staff (which at the time meant primarily GiveWell Co-Founder Holden Karnofsky and me) were necessary for most outreach-related work. This has changed. We now have capacity to take on outreach work as other staff have been hired and trained on this type of work.
• Better information on the impact of GiveWell’s outreach. We now have better information about the returns to outreach because:
  1. We’ve collected better data (via an improved donations processing system and outreach efforts) about where donors find out about us. Because of our ability to track donors, we know that a single appearance on NPR or major podcasts tends to drive $50,000+ in annual donations.
  2. More time passing has demonstrated that the lifetime value of the donations of a first time donor is higher than we expected. In several cases, we’ve seen major donors (i.e., those giving $10,000-$100,000) increase their annual giving by a factor of 10 or more.

We’re in the early stages of figuring out how we can proactively invest time and money in outreach to significantly increase our money moved. For now, we’ve taken some opportunities that we think will have positive returns; these are the three that we’ve invested the most time and money in to date:

• Podcast advertising. We’ve been advertising on podcasts that we believe our target audience listens to, based on interviews with current donors and GiveWell staff. In February and March, we ran a small experiment with a few ads on FiveThirtyEight’s Politics podcast and Vox’s The Weeds.1We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  In total, we spent approximately $20,000 on ads for this initial experiment. We ask donors who give via our website to tell us where they learned about GiveWell when they donate. GiveWell received approximately $8,000 in donations between February 1 and November 20 from donors who reported that they had learned about us via these podcasts.

  The donations we received were from first-time donors; to assess the impact of our advertising, we need to estimate the lifetime value of acquiring a new donor. In work we’ve done to assess our retention rate, we’ve seen that (a) approximately 20-25% of the donors who make a first-time donation of less than $1,000 give again in the subsequent year but (b) because many first-time donors increase the size of their donation over time, collectively, the donors who recur give more than 100% of the value of what they give in their first year.

  At higher donation levels ($1,000-$100,000), we measure 40-45% retention among donors, which leads to retention of approximately two-thirds of dollars given.2I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  We therefore estimated the net present value of expected future donations (over the next five years) from these podcasts ads as somewhere between approximately $20,000 (assuming two-thirds dollar retention for the first two years and 100% dollar retention subsequently) and $45,000 (assuming 100% dollar retention).3We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

  A few additional facts are worth keeping in mind about the above figures:

  • We ran this experiment in February and March; most donors give at the end of the calendar year. We consistently see donors who find out about GiveWell during the course of the year, but donate in December. Other things equal, we expect that our advertising would have had greater measured returns in December than earlier in the year.
  • We are only able to track donors who (a) fill out our donation form telling us where they learned about us and (b) give directly through our website rather than to our top charities. Less than 50% of donors who give via credit card (and a smaller percentage of donors who give via check) tell us where they learned about GiveWell. Also, roughly speaking, approximately 50% of the donors and dollars we influence come through GiveWell rather than going to our top charities.4I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });
  • It’s certainly possible that donors who learn about us via podcast would be more likely to give through our website than an average donor, more likely to report on how they found us (since their source is clear), or less likely to be retained. My best guess is that donors who learn about us via podcast ads behave similarly to our other donors, but I won’t be surprised if they don’t.

  With all that in mind, I believe that the impact of our podcast advertising is higher than what we directly measured.

  The results we saw from February to November this year were promising enough that we decided to increase the size of our experiment by spending approximately $100,000 on podcast ads. We’re currently running ads on FiveThirtyEight’s Politics podcast and Ezra Klein’s podcast and The Weeds at Vox.

• Earned media outreach. Mentions of GiveWell in the media have historically been a strong driver of growth. We aimed to increase mentions of GiveWell in high-quality, high-profile media where we’ve had the most past success as measured by dollars donated (i.e., media like The New York Times, NPR, The Wall Street Journal, and Financial Times). We retained a PR firm that came strongly recommended; we also increased 1-to-1 outreach by GiveWell staff to members of the media who have covered GiveWell in the past. It’s very hard to attribute the impact of the additional effort we’ve invested—overall, our effort has been fairly limited, and it’s hard to easily draw the causal lines between our work and the stories that appear—but my guess is that our increased efforts have led to more coverage of GiveWell and our top charities this giving season than in the recent past.
• Website improvements. Companies that sell products online invest significant effort into optimizing their websites and checkout pages to maximize their revenues. We retained a marketing consultant, Will Wong of Mission Street, and we’ve been A/B testing different donation pages and plan to test other pages on our website such as our homepage or top charities page to see whether we can increase our conversion rate (i.e., the percentage of visitors to our website who give to one of our top charities). For context, our current conversion rate is 1%. Our understanding is that a standard conversion rate for e-commerce companies is 2%, and that international nonprofits have a similar conversion rate.5See Pg 51 of the study downloadable here. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); An increase in our conversion rate to the industry average would lead to a significant increase in the amount of money we direct to our top charities.

Notes   [ + ]

1. ↑ We’ve also been running ads on Julia Galef’s Rationally Speaking podcast since then. Because it’s much smaller and more targeted, we’ve excluded it from this analysis. Measured returns to advertising on Rationally Speaking have been significantly better than the more mainstream podcasts discussed in this post. 2. ↑ I say “measure” retention because we’ve learned that many donors give subsequent donations directly to our top charities and don’t report those donations to us. We’ve tried to follow up with lapsed donors and with charities to track these donors down. 3. ↑ We only projected donations over five years. This is fairly arbitrary because we don’t have long-term enough data to know whether or not this is a reasonable assumption. We capped it to prevent our assessment being driven by speculation about how much money would be donated many years in the future. 4. ↑ I took this rough estimate from footnote 26, on page 15, of GiveWell’s 2015 metrics report. 5. ↑ See Pg 51 of the study downloadable here. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Update on our work on outreach appeared first on The GiveWell Blog.

We recently discussed how you can give to reduce the administrative burden on charities. This post will focus on how you can save money on fees and give tax-efficiently so that more of your charitable budget can go directly to the organizations you want to support. This is an updated version of a post we originally ran in 2012; some content is the same, other content has been added or updated.

1. Don’t wait until the last minute. Many donors wait until the very end of the calendar year to give. If you’re hoping to make a donation by that deadline, we strongly advise against this. Here’s why:
   • Some methods of donating require some planning and preparation, such as giving appreciated stock.
   • Checks are tax-deductible according to the postmarked date on the envelope—you can’t write a check in 2018, backdate it to 2017, and claim a deduction. So, please head to the post office before the new year if you’re looking to get a tax deduction this year.
   • Leaving little time between making your donation and the deadline means you’ll have limited time to react if something unexpected happens, such as your credit card charge being declined.

   We recommend building in a cushion of a week or two if you’re aiming to donate by a particular deadline. The earlier you can give, the less likely you’ll have any issues. For end-of-year giving, we recommend a target date of December 24 or earlier.

2. Try to get a tax benefit. Details vary by country and personal situation, but a tax deduction can allow you to give much more to charity at the same cost to yourself. (That said, as discussed later in the post, we believe it is more important to give to the most effective possible charity than to get the maximum tax benefit.) Below, we discuss our understanding of donation methods for tax-advantaged giving, although please note that none of this information should be construed as legal or tax advice.

   Donors in the United States may make tax-deductible gifts to any of our nine top charities by giving to GiveWell. There are also a large number of tax-deductible options for giving to our top charities in other countries; please see the table here for more information.

   Donors in certain U.S. states and income brackets who are interested in maximizing their tax deduction may also consider “donation bunching,” or making two donations in one year rather than one donation in each of two years to take advantage of the standard deduction in one year and maximize the size of their itemized charitable deduction in a subsequent year. Considerations related to donation bunching are discussed in this post by former GiveWell intern Ben Kuhn.

3. Avoid the large transaction fees and delays associated with large online donations. When donating via credit card, you will almost always be charged standard credit card processing fees. Making a large donation via credit card may also trigger your card’s fraud protection (though a call to the credit card company can generally resolve the situation quickly).

   We discussed some of the tradeoffs between the ease of donating via certain platforms and the fees for donors and the administrative costs to charities for processing them in a previous post. In short, we do not advise making donations via credit card if you’re planning to give $1,000 or more.

4. Give appreciated stock and cryptocurrency. In the U.S., if you give stock or cryptocurrency (such as Bitcoin) to a charity, neither you nor the charity will have to pay taxes on capital gains (as you would if you sold the stock yourself). If you have stock or cryptocurrency that you acquired for $1,000 (and has a cost basis of $1,000) but is now worth $2,000, you can give the stock to charity, take a deduction for $2,000, and not have to pay capital gains tax on the $1,000 of appreciation. This can result in significant savings.

   Due to the administrative cost associated with processing donations of stock, we ask that donors donate stock directly to GiveWell only if the value of the stock at the time of transfer is estimated at approximately $1,000 or more. More information on giving appreciated stock to GiveWell, either through E*Trade or GiveWell’s Vanguard donor-advised fund, is available here. You can also use Betterment to donate appreciated stock to GiveWell. If you’re interested in making a Bitcoin donation to GiveWell, please email us at donations@givewell.org to receive instructions on how to give.

5. Look into donor-advised funds to make the process smoother and more consistent year-to-year. Donor-advised funds allow donors to make a charitable donation (and get a tax deduction) now, while deciding which charity they’d like to support later. The donation goes into a fund that is “advised” by the donor, and the donor may later recommend a grant from the fund to the charity of his/her choice.

   We see a couple of advantages to this setup. One advantage is that you can separate your “decision date” (the date on which you decide which charity you’d like to support) from your “transaction date.” That means that if you aren’t ready to decide which charity to support yet, you can still get started on the process of transferring funds and getting a tax deduction for the appropriate year. Another advantage is that if you change the charity you support from year to year, you’re still working with the same partner when it comes to transactions, so the process for e.g. donating stock will not change from year to year. Donor-advised funds are often set up to easily accept donated stock or non-traditional assets, whereas charities may or may not be.

   Many large investment companies—Vanguard, Fidelity, Schwab—offer donor-advised funds. They generally charge relatively modest management fees. We also maintain our own donor-advised fund for donors interested in supporting our recommended charities; the minimum size for a donation is $5,000. The GiveWell donor-advised fund is likely most helpful for donors interested in giving certain types of securities, such as Vanguard mutual funds, that are not accepted by E*Trade.

6. Find out if your company offers donation matching. Many companies offer to match employees’ gifts up to a certain amount. We recommend checking with your employer if you’re unsure whether they offer this option. Some employers have a limited list of charities to which they will match donations; consider asking your employer whether they would add the charity of your choosing if it isn’t already on the list.
7. Consider the political environment. If you believe that your likelihood of taking charitable deductions is higher in 2017 than it will be in 2018, consider increasing your giving this year.
8. Choose your charity wisely. Saving money on taxes and transaction fees can be significant, in some cases approaching or exceeding a 50 percent increase in the amount you’re able to give. However, we believe that your choice of charity is a much larger factor in how much good your giving accomplishes.

   Our charity recommendations make it possible to support outstanding, thoroughly vetted organizations—which we’ve investigated by reviewing academic evidence, interviewing staff, analyzing internal documents, conducting site visits, assessing funding needs, and more—without needing to do your own research. We publicly publish the full details of our process and analysis, so you can spot-check whatever part of our work and reasoning you’d like to.

Final notes

If you support our recommended charities (on the basis of our recommendation) but you don’t give through our website, please fill out this form to let us know about your gift; doing so helps GiveWell track our impact.

We believe that even when dealing with a relatively complicated gift (for example, a gift of stock), it’s possible to give quite quickly and with only minor hassle. The much more difficult challenge is choosing a charity, and we’ve tried to make that easy as well. We hope you’ll give this season, even if you’re just starting to think about it now.

If you’d like more advice about how to donate, please don’t hesitate to contact us. We’re happy to talk.

The post Maximizing the impact of your donation: saving on fees means more money for great charities appeared first on The GiveWell Blog.

We recently discussed how you can give to reduce the administrative burden on charities. This post will focus on how you can save money on fees and give tax-efficiently so that more of your charitable budget can go directly to the organizations you want to support. This is an updated version of a post we originally ran in 2012; some content is the same, other content has been added or updated.

1. Don’t wait until the last minute. Many donors wait until the very end of the calendar year to give. If you’re hoping to make a donation by that deadline, we strongly advise against this. Here’s why:
   • Some methods of donating require some planning and preparation, such as giving appreciated stock.
   • Checks are tax-deductible according to the postmarked date on the envelope—you can’t write a check in 2018, backdate it to 2017, and claim a deduction. So, please head to the post office before the new year if you’re looking to get a tax deduction this year.
   • Leaving little time between making your donation and the deadline means you’ll have limited time to react if something unexpected happens, such as your credit card charge being declined.

   We recommend building in a cushion of a week or two if you’re aiming to donate by a particular deadline. The earlier you can give, the less likely you’ll have any issues. For end-of-year giving, we recommend a target date of December 24 or earlier.

2. Try to get a tax benefit. Details vary by country and personal situation, but a tax deduction can allow you to give much more to charity at the same cost to yourself. (That said, as discussed later in the post, we believe it is more important to give to the most effective possible charity than to get the maximum tax benefit.) Below, we discuss our understanding of donation methods for tax-advantaged giving, although please note that none of this information should be construed as legal or tax advice.

   Donors in the United States may make tax-deductible gifts to any of our nine top charities by giving to GiveWell. There are also a large number of tax-deductible options for giving to our top charities in other countries; please see the table here for more information.

   Donors in certain U.S. states and income brackets who are interested in maximizing their tax deduction may also consider “donation bunching,” or making two donations in one year rather than one donation in each of two years to take advantage of the standard deduction in one year and maximize the size of their itemized charitable deduction in a subsequent year. Considerations related to donation bunching are discussed in this post by former GiveWell intern Ben Kuhn.

3. Avoid the large transaction fees and delays associated with large online donations. When donating via credit card, you will almost always be charged standard credit card processing fees. Making a large donation via credit card may also trigger your card’s fraud protection (though a call to the credit card company can generally resolve the situation quickly).

   We discussed some of the tradeoffs between the ease of donating via certain platforms and the fees for donors and the administrative costs to charities for processing them in a previous post. In short, we do not advise making donations via credit card if you’re planning to give $1,000 or more.

4. Give appreciated stock and cryptocurrency. In the U.S., if you give stock or cryptocurrency (such as Bitcoin) to a charity, neither you nor the charity will have to pay taxes on capital gains (as you would if you sold the stock yourself). If you have stock or cryptocurrency that you acquired for $1,000 (and has a cost basis of $1,000) but is now worth $2,000, you can give the stock to charity, take a deduction for $2,000, and not have to pay capital gains tax on the $1,000 of appreciation. This can result in significant savings.

   Due to the administrative cost associated with processing donations of stock, we ask that donors donate stock directly to GiveWell only if the value of the stock at the time of transfer is estimated at approximately $1,000 or more. More information on giving appreciated stock to GiveWell, either through E*Trade or GiveWell’s Vanguard donor-advised fund, is available here. You can also use Betterment to donate appreciated stock to GiveWell. If you’re interested in making a Bitcoin donation to GiveWell, please email us at donations@givewell.org to receive instructions on how to give.

5. Look into donor-advised funds to make the process smoother and more consistent year-to-year. Donor-advised funds allow donors to make a charitable donation (and get a tax deduction) now, while deciding which charity they’d like to support later. The donation goes into a fund that is “advised” by the donor, and the donor may later recommend a grant from the fund to the charity of his/her choice.

   We see a couple of advantages to this setup. One advantage is that you can separate your “decision date” (the date on which you decide which charity you’d like to support) from your “transaction date.” That means that if you aren’t ready to decide which charity to support yet, you can still get started on the process of transferring funds and getting a tax deduction for the appropriate year. Another advantage is that if you change the charity you support from year to year, you’re still working with the same partner when it comes to transactions, so the process for e.g. donating stock will not change from year to year. Donor-advised funds are often set up to easily accept donated stock or non-traditional assets, whereas charities may or may not be.

   Many large investment companies—Vanguard, Fidelity, Schwab—offer donor-advised funds. They generally charge relatively modest management fees. We also maintain our own donor-advised fund for donors interested in supporting our recommended charities; the minimum size for a donation is $5,000. The GiveWell donor-advised fund is likely most helpful for donors interested in giving certain types of securities, such as Vanguard mutual funds, that are not accepted by E*Trade.

6. Find out if your company offers donation matching. Many companies offer to match employees’ gifts up to a certain amount. We recommend checking with your employer if you’re unsure whether they offer this option. Some employers have a limited list of charities to which they will match donations; consider asking your employer whether they would add the charity of your choosing if it isn’t already on the list.
7. Consider the political environment. If you believe that your likelihood of taking charitable deductions is higher in 2017 than it will be in 2018, consider increasing your giving this year.
8. Choose your charity wisely. Saving money on taxes and transaction fees can be significant, in some cases approaching or exceeding a 50 percent increase in the amount you’re able to give. However, we believe that your choice of charity is a much larger factor in how much good your giving accomplishes.

   Our charity recommendations make it possible to support outstanding, thoroughly vetted organizations—which we’ve investigated by reviewing academic evidence, interviewing staff, analyzing internal documents, conducting site visits, assessing funding needs, and more—without needing to do your own research. We publicly publish the full details of our process and analysis, so you can spot-check whatever part of our work and reasoning you’d like to.

Final notes

If you support our recommended charities (on the basis of our recommendation) but you don’t give through our website, please fill out this form to let us know about your gift; doing so helps GiveWell track our impact.

We believe that even when dealing with a relatively complicated gift (for example, a gift of stock), it’s possible to give quite quickly and with only minor hassle. The much more difficult challenge is choosing a charity, and we’ve tried to make that easy as well. We hope you’ll give this season, even if you’re just starting to think about it now.

If you’d like more advice about how to donate, please don’t hesitate to contact us. We’re happy to talk.

The post Maximizing the impact of your donation: saving on fees means more money for great charities appeared first on The GiveWell Blog.

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

You can view our September 2017 open thread here.

The post December 2017 open thread appeared first on The GiveWell Blog.

Our goal with hosting quarterly open threads is to give blog readers an opportunity to publicly raise comments or questions about GiveWell or related topics (in the comments section below). As always, you’re also welcome to email us at info@givewell.org or to request a call with GiveWell staff if you have feedback or questions you’d prefer to discuss privately. We’ll try to respond promptly to questions or comments.

You can view our September 2017 open thread here.

The post December 2017 open thread appeared first on The GiveWell Blog.

GiveWell’s research aims to help donors by recommending charities we believe can put donations to use efficiently to save or improve lives. Our research focuses on maximizing the good donors can accomplish with their gifts by identifying where to donate effectively.

This is the first of two posts discussing another important aspect of giving effectively: how you donate. The second post will discuss how to maximize your gift (via tax deductions, employer matches, and other strategies) and to ensure the greatest percentage of your donation reaches the charity, rather than being taken up by fees. This post will discuss how to reduce the administrative burdens on charities by choosing your donation method wisely.

How you choose to donate—whether you write a check, fill out a PayPal form, or enter your credit card online—can make a huge difference for the charities that receive your gift. You can save charities time and administrative headaches by being strategic about your method of donation. Our advice follows.

1. Give predictably. We’re often asked whether it’s better to give a recurring monthly donation or to give once every year. We don’t think one is inherently better than the other. We do think it’s important that you communicate with charities about your plans; this enables them to budget for your donation.
2. Consider the fees and processing costs of your donation method. Some donation methods require more administrative work for charities to process than others. For example, check donations require multiple manual steps (such as mail retrieval, scanning, data entry, and composing a receipt), whereas online donations can be recorded and thanked automatically. The table below summarizes administrative burden and fees for donations made to GiveWell via the following methods:
   Processing burden Fees GiveWell recommends for what size gift? Online credit card form Low 2.15% (3.20% for AMEX) + $0.28 on each transaction. GiveWell pays the fee. Under $1,000 Check High None for domestic checks. Currency conversion and vetting fees may apply for international checks. GiveWell pays these fees. Over $1,000 Wire transfer Medium Variable. Donor pays the fee. Over $1,000 Appreciated securities High Variable. For E*Trade and Vanguard, GiveWell pays the fee. Over $1,000

   The fees associated with each method also matter. Online credit card donations generally have the highest fees, but the lowest processing time for GiveWell. Checks, on the other hand, have no fees but the highest processing time.

   As a result, we recommend that donors giving under $1,000 make a donation via our online credit card form. For donors giving over $1,000, the calculus changes – we recommend giving via a check. Note that due to the high processing time required for each check we receive, we would advise donors giving via check to make an annual gift, rather than dividing their gift into monthly check payments.

3. Tell charities where you heard about them and why you chose to support their work. Knowing how donors who use our research or support our operations found GiveWell and why they chose to make a donation is extremely helpful for improving our outreach and research product. We recommend giving charities this basic information. Supporters of GiveWell and our recommended charities can do this by filling out this survey.
4. Keep in touch! Charities love hearing from their supporters. Ask them questions about their work, let them know if your giving plans change, and offer them feedback on how to improve.

A separate post will discuss how you can save money on fees and give tax-efficiently so that more of your charitable budget can go directly to the organizations you want to support.

The post Give efficiently and reduce the work for charities appeared first on The GiveWell Blog.

GiveWell’s research aims to help donors by recommending charities we believe can put donations to use efficiently to save or improve lives. Our research focuses on maximizing the good donors can accomplish with their gifts by identifying where to donate effectively.

This is the first of two posts discussing another important aspect of giving effectively: how you donate. The second post will discuss how to maximize your gift (via tax deductions, employer matches, and other strategies) and to ensure the greatest percentage of your donation reaches the charity, rather than being taken up by fees. This post will discuss how to reduce the administrative burdens on charities by choosing your donation method wisely.

How you choose to donate—whether you write a check, fill out a PayPal form, or enter your credit card online—can make a huge difference for the charities that receive your gift. You can save charities time and administrative headaches by being strategic about your method of donation. Our advice follows.

1. Give predictably. We’re often asked whether it’s better to give a recurring monthly donation or to give once every year. We don’t think one is inherently better than the other. We do think it’s important that you communicate with charities about your plans; this enables them to budget for your donation.
2. Consider the fees and processing costs of your donation method. Some donation methods require more administrative work for charities to process than others. For example, check donations require multiple manual steps (such as mail retrieval, scanning, data entry, and composing a receipt), whereas online donations can be recorded and thanked automatically. The table below summarizes administrative burden and fees for donations made to GiveWell via the following methods:
   Processing burden Fees GiveWell recommends for what size gift? Online credit card form Low 2.15% (3.20% for AMEX) + $0.28 on each transaction. GiveWell pays the fee. Under $1,000 Check High None for domestic checks. Currency conversion and vetting fees may apply for international checks. GiveWell pays these fees. Over $1,000 Wire transfer Medium Variable. Donor pays the fee. Over $1,000 Appreciated securities High Variable. For E*Trade and Vanguard, GiveWell pays the fee. Over $1,000

   The fees associated with each method also matter. Online credit card donations generally have the highest fees, but the lowest processing time for GiveWell. Checks, on the other hand, have no fees but the highest processing time.

   As a result, we recommend that donors giving under $1,000 make a donation via our online credit card form. For donors giving over $1,000, the calculus changes – we recommend giving via a check. Note that due to the high processing time required for each check we receive, we would advise donors giving via check to make an annual gift, rather than dividing their gift into monthly check payments.

3. Tell charities where you heard about them and why you chose to support their work. Knowing how donors who use our research or support our operations found GiveWell and why they chose to make a donation is extremely helpful for improving our outreach and research product. We recommend giving charities this basic information. Supporters of GiveWell and our recommended charities can do this by filling out this survey.
4. Keep in touch! Charities love hearing from their supporters. Ask them questions about their work, let them know if your giving plans change, and offer them feedback on how to improve.

A separate post will discuss how you can save money on fees and give tax-efficiently so that more of your charitable budget can go directly to the organizations you want to support.

The post Give efficiently and reduce the work for charities appeared first on The GiveWell Blog.

For this post, GiveWell staff members and contributors wrote up the thinking behind their personal donations for the year. We made similar posts in previous years (2013, 2014, 2015, 2016). Staff and contributors are listed in order of their start dates at GiveWell.

You can click the below links to jump to an entry:

• Elie Hassenfeld
• Natalie Crispin
• Josh Rosenberg
• Sophie Monahan
• Catherine Hollander
• Andrew Martin
• Chelsea Tabart
• Christian Smith
• Isabel Arjmand
• James Snowden

Elie Hassenfeld

This year, I’m planning to donate to GiveWell for granting to top charities at its discretion.

GiveWell is currently producing the highest-quality research it ever has, which has led to more thoroughly researched, higher-quality recommendations that have been compared to more potential alternatives than ever before. Personally, I only spent about a third of my time on top-charities-related research in 2017, so I’m thrilled by the quality of the research the GiveWell research team produced this year.

In making this decision I also talked to Lewis Bollard (about animal welfare) and Nick Beckstead (about effective altruism movement building and long-term future giving opportunities) but ultimately felt that the funding gaps that GiveWell’s top charities face were more pressing given the Open Philanthropy Project’s support of their respective portfolios.

Natalie Crispin

I continue to believe that GiveWell top charities are the best option for impact-focused giving for individuals and I plan to give my annual gift this year to GiveWell for granting at its discretion to top charities. I am grateful for all the work, thoughtfulness, and hours of debate that my colleagues put into the recommendations, and I believe that the recommendations are as strong as they’ve ever been. I am excited to support the most effective charities I know of.

Josh Rosenberg

This year I’m planning to give:

• 80% to GiveWell to grant to top charities at its discretion. I believe that GiveWell’s top charities are among the most effective ways to help people. I know how intensely our team has scrutinized these giving opportunities and am excited to give based on our research.
• 10% to the long-term future EA Fund. I would like to see future generations thrive. This podcast provides a good summary of some arguments for the moral importance of helping future generations. Based on my experience following the Open Philanthropy Project, I believe that donations to this fund will be put to good use.
• 10% to charities focused on farm animal welfare. I believe that the welfare of farm animals is a particularly important and neglected problem. I expect to choose a farm animal welfare charity to give to based on Animal Charity Evaluators’ recommendations. I look forward to using its research and would be excited to see similar charity evaluation organizations exist in other domains (e.g., policy-related giving).

I chose to diversify a portion of my giving because I want to signal my interest in a variety of important cause areas that I believe people should be considering and I want to continue to engage with the strongest giving opportunities in other domains to help myself reflect on which donations seem to be most effective. I focused most of my giving on global health and development since GiveWell’s top charities have the most pressing funding gaps I am aware of. If I knew of a strong case for a particular giving opportunity in another cause area, I would be open to changing my allocation in the future.

I also considered giving to the global development EA Fund. I think that this fund would be a good option for donors who (a) are open to higher-risk, higher-reward giving opportunities in global development, and (b) have a high degree of trust in GiveWell (Elie is the manager of the fund). The description of the fund notes that GiveWell’s Incubation Grants (GIG) program has not been hampered by insufficient funding to date. However, I think it may be useful to know if there is a large pool of donors who would like to see more GIG-type giving on the margin; giving to this fund would be a good way to show support for GIG. I ultimately chose not to donate to this fund because a substantial portion of my job is to work on GIG, and I would rather leave it to external observers to assess whether they think it deserves further support. (I see GiveWell’s top charities differently because we’ve thoroughly publicly justified the case for giving to these charities.)

Sophie Monahan

I believe that all of GiveWell’s recommended charities are excellent giving opportunities. I believe that donating to GiveWell for granting to recommended charities is an excellent option (allowing valuable flexibility) for donors whose values align well with GiveWell’s. (On values: more, more, more.)

This year, I am giving to No Lean Season for the following reasons:

• I place greater value on reducing near-term poverty for adults and children of all ages relative to preventing deaths of very young children, compared to GiveWell as a whole. I also value certainty in the near-term impact of programs relatively more. According to my values, No Lean Season and GiveDirectly are undervalued by GiveWell. Therefore I believe that their funding gaps are a higher priority for people with values like mine.
• I am giving to No Lean Season despite the fact that this year, GiveWell recommended sufficient funding to No Lean Season to fill its highest priority funding gap (to implement its program in Bangladesh in the next three years), citing reasons other than pure considerations of cost-effectiveness,1“While No Lean Season’s cost-effectiveness is at the lower end of our top charities (~5x cash transfers), we see additional reasons to prioritize this gap. We believe No Lean Season is the top charity where there is the strongest case to be made for “upside”; our cost-effectiveness analysis may not capture the potential impact of scaling a new program that could lead to greater visibility and funding for a novel type of program.” GiveWell blog: Our top charities for giving season 2017 jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); which may not repeat in future years. Therefore, I believe that marginal donations to No Lean Season are likely to increase No Lean Season’s multiyear funding.
• I am also moved by reasons of sentiment—I led GiveWell’s evaluation of No Lean Season and was very impressed—and to promote the visibility of this new top charity.

Catherine Hollander

I plan to give 90% of my donation to the Against Malaria Foundation (AMF) and 10% to No Lean Season. I trust GiveWell’s review process and its recommendation of AMF as having one of the highest-value funding gaps to fill.

Like Sophie, I plan to support No Lean Season to increase the visibility of a new charity on the list, and because it was the charity review I engaged with most deeply and personally in my time at GiveWell—I traveled with Sophie and Christian to Bangladesh to visit No Lean Season and review its work in September. I plan to limit my donation to No Lean Season to 10% of my total gift because I do not believe its need for funding is as pressing as that of AMF.

Andrew Martin

This year, I’m planning to donate to GiveWell for granting to top charities at its discretion.

I think the case remains as strong as ever that donating to GiveWell’s top charities is an exceptional opportunity for donors who want to maximize their impact. Even after accounting for Good Ventures’ $75 million in grants based on GiveWell’s recommendations this year, we believe that our top charities still have a large amount of room for more funding and that additional donations will accomplish a lot of good.

I’ve decided to donate to GiveWell for granting to top charities at its discretion—rather than donating directly to individual top charities—because I believe it’s valuable for GiveWell to have the flexibility to provide funding to whichever top charities have the most pressing funding needs.

Chelsea Tabart

I plan to donate to GiveWell for granting to top charities at its discretion this year. Another year of exposure to the thoughtful, rigorous work of my colleagues has increased my belief in GiveWell’s research process, and I’m excited that my giving can be a small part of the exceptional work our top charities do.

Christian Smith

I’m planning to direct all of my year-end donations to GiveWell for granting to top charities at its discretion. Several of our charities have large, high-priority funding gaps, and I’m excited to be supporting work that I expect to have a large positive impact.

I think there are reasonable worldviews and ethical positions that would make thoughtful giving in other cause areas (e.g. basic research, animal welfare, or improving the far future) appear much more cost-effective than thoughtful giving to organizations involved in global health and development. I considered directing some of my donations towards these cause areas, but ultimately had a preference for supporting causes in global health and development. I feel fine about this decision, but I may have approached my giving differently if I was not working for GiveWell.

Isabel Arjmand

The allocation of my charitable giving this year will be quite similar to what I did last year, though with a slightly higher proportion going to GiveWell’s top charities. As a general note, I’m inclined to diversify my giving between (1) organizations that are promising from a utilitarian point of view (like GiveWell’s top charities) and (2) those that appeal to different moral considerations.

I’m excited to give 75% of my donation to GiveWell for granting to recommended charities at its discretion. My understanding of GiveWell’s research is much deeper than it was at this point last year (when I was a fairly new staff member), and I remain very enthusiastic about the quality of GiveWell’s recommendations. Especially as someone whose moral values are very close to the median values in our cost-effectiveness analyses, I think giving for granting at GiveWell’s discretion is my best option for impact-focused giving. I considered giving directly to Malaria Consortium’s seasonal malaria chemoprevention program, as I think that it may have the highest-impact remaining funding gap of our top charities, but ultimately the flexibility of giving to GiveWell’s discretionary fund and my trust in GiveWell’s judgment lead me to prefer that option.

Additionally, I plan to give 5% of my donation to GiveDirectly, a GiveWell-recommended organization that I also supported last year. I’m supporting GiveDirectly because I think it’s an exceptionally strong, innovative organization with high potential for ‘upside,’ including the potential to serve as a model for other organizations. I’m thinking of the rationale for this portion of my giving as somewhere in between what I describe in the previous paragraph and what I describe below.

Like last year, I’m thinking of the remainder of my charitable contribution (in this case, 20%) as serving a different, less impact-focused purpose. My goals with this portion of my giving are to promote more justice-focused causes, further my own civic engagement, and signal support for work I’d like to see more of. I’d be surprised if any of the organizations below were as cost-effective as GiveWell’s top charities; I also haven’t vetted them with an intensity that comes anywhere close to the rigor of GiveWell evaluations. Of the four organizations among which I plan to divide this donation, the first two are organizations I supported last year, and the remaining two are new to my list.

• Causa Justa :: Just Cause: As I wrote last year, I see supporting Causa Justa :: Just Cause—a Bay Area-based grassroots organization supporting housing rights, immigrant rights, and racial justice—as a means of supporting the community in which I live.
• Planned Parenthood: Reproductive justice and access to healthcare continue to be important to me. Particularly given the absence of a GiveWell-recommended organization providing these services abroad (which I’d guess may be more cost-effective), I’m happy to donate to a U.S.-based organization that I’m personally familiar with and have confidence in.
• ProPublica: I’m donating to ProPublica in support of its high-quality independent journalism, which I think is critical to a well-functioning civic society.
• Earthjustice: I decided this year to support an organization working on climate change and environmental justice, and after researching the space briefly, I found Earthjustice—which focuses on environmental protection via legal advocacy—most compelling.

James Snowden (Research Consultant)

This year, I donated monthly through the Effective Altruism Global Health and Development Fund, so I’ve already made the majority of my personal annual donations this year. At the end of year, I set my priorities for the next year, and adjust my automatic monthly donations accordingly.

In practice, donating to the Global Health and Development Fund is similar to donating to GiveWell for granting to top charities at its discretion, as Elie is the fund manager. My primary reason for donating to Effective Altruism Funds rather than directly to GiveWell is that I want to signal support for a project I think is valuable.

I plan to continue giving 80% of my donations to the Global Health and Development fund, but now donate 10% each to the Animal Welfare fund and Long-term Future fund. I think animal welfare and improving the long-term future are extremely important (more so than I did last year). I don’t feel I have enough context to independently evaluate organizations in this area so want to outsource my decisions to Lewis Bollard (for animal welfare), and Nick Beckstead (for global catastrophic risks). I’m uncertain whether, given Good Ventures’ support for animal welfare and the relatively small number of funding opportunities, there’s substantial room for more funding in that area. But I think the ‘worst case scenario’ is that I funge with Good Ventures, which I’d still think was a reasonably good outcome.

I also considered:

• Donating to the Centre for Pesticide Suicide Prevention. I was the primary researcher working on this GiveWell Incubation Grant and believe it’s a potentially very cost-effective (though risky) giving opportunity. I decided not to donate because the Incubation Grant is intended to fully cover their costs for two years.
• Donating to Malaria Consortium’s Seasonal Malaria Chemoprevention (SMC) program. Having spent more time looking into SMC this year, I believe it’s a more cost-effective giving opportunity than Against Malaria Foundation (our recommendation to donors this year is 70% Against Malaria Foundation and 30% Schistosomiasis Control Initiative). I decided not to donate to the SMC program this year because (i) we recommended Good Ventures grant $27.9m to Malaria Consortium this year (vs $2.5m for AMF), and I don’t have a strong view that SMC would be more cost-effective on the margin after this grant (ii) I think allowing GiveWell to regrant at its discretion allows for more flexibility (iii) I place more weight in GiveWell’s aggregate view than the inside view of any individual researcher (including myself!)—although I think there’s value in thinking about this independently to identify if GiveWell is making decisions I disagree with.
• Continuing to donate only to the Global Health and Development fund. I think there’s a strong argument for just donating to the opportunity you think is best in expectation, rather than diversifying. I decided to diversify a fairly small amount because (i) it more accurately signals that I care about those areas (ii) it motivates me to learn more about those areas than I otherwise would have (although I don’t expect this to be a major priority for me).
• Donating a larger proportion to the Animal Welfare and Far Future funds. Given my relatively greater knowledge in global health and development, I don’t yet feel comfortable giving a greater proportion to areas I know less about.

Aside from my personal giving, I advise a small foundation on their grantmaking. We haven’t yet decided where to give this year, and this will partly depend on the priorities of others involved in the decision.

Notes   [ + ]

1. ↑ “While No Lean Season’s cost-effectiveness is at the lower end of our top charities (~5x cash transfers), we see additional reasons to prioritize this gap. We believe No Lean Season is the top charity where there is the strongest case to be made for “upside”; our cost-effectiveness analysis may not capture the potential impact of scaling a new program that could lead to greater visibility and funding for a novel type of program.” GiveWell blog: Our top charities for giving season 2017 function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Staff members’ personal donations for giving season 2017 appeared first on The GiveWell Blog.

For this post, GiveWell staff members and contributors wrote up the thinking behind their personal donations for the year. We made similar posts in previous years (2013, 2014, 2015, 2016). Staff and contributors are listed in order of their start dates at GiveWell.

You can click the below links to jump to an entry:

• Elie Hassenfeld
• Natalie Crispin
• Josh Rosenberg
• Sophie Monahan
• Catherine Hollander
• Andrew Martin
• Chelsea Tabart
• Christian Smith
• Isabel Arjmand
• James Snowden

Elie Hassenfeld

This year, I’m planning to donate to GiveWell for granting to top charities at its discretion.

GiveWell is currently producing the highest-quality research it ever has, which has led to more thoroughly researched, higher-quality recommendations that have been compared to more potential alternatives than ever before. Personally, I only spent about a third of my time on top-charities-related research in 2017, so I’m thrilled by the quality of the research the GiveWell research team produced this year.

In making this decision I also talked to Lewis Bollard (about animal welfare) and Nick Beckstead (about effective altruism movement building and long-term future giving opportunities) but ultimately felt that the funding gaps that GiveWell’s top charities face were more pressing given the Open Philanthropy Project’s support of their respective portfolios.

Natalie Crispin

I continue to believe that GiveWell top charities are the best option for impact-focused giving for individuals and I plan to give my annual gift this year to GiveWell for granting at its discretion to top charities. I am grateful for all the work, thoughtfulness, and hours of debate that my colleagues put into the recommendations, and I believe that the recommendations are as strong as they’ve ever been. I am excited to support the most effective charities I know of.

Josh Rosenberg

This year I’m planning to give:

• 80% to GiveWell to grant to top charities at its discretion. I believe that GiveWell’s top charities are among the most effective ways to help people. I know how intensely our team has scrutinized these giving opportunities and am excited to give based on our research.
• 10% to the long-term future EA Fund. I would like to see future generations thrive. This podcast provides a good summary of some arguments for the moral importance of helping future generations. Based on my experience following the Open Philanthropy Project, I believe that donations to this fund will be put to good use.
• 10% to charities focused on farm animal welfare. I believe that the welfare of farm animals is a particularly important and neglected problem. I expect to choose a farm animal welfare charity to give to based on Animal Charity Evaluators’ recommendations. I look forward to using its research and would be excited to see similar charity evaluation organizations exist in other domains (e.g., policy-related giving).

I chose to diversify a portion of my giving because I want to signal my interest in a variety of important cause areas that I believe people should be considering and I want to continue to engage with the strongest giving opportunities in other domains to help myself reflect on which donations seem to be most effective. I focused most of my giving on global health and development since GiveWell’s top charities have the most pressing funding gaps I am aware of. If I knew of a strong case for a particular giving opportunity in another cause area, I would be open to changing my allocation in the future.

I also considered giving to the global development EA Fund. I think that this fund would be a good option for donors who (a) are open to higher-risk, higher-reward giving opportunities in global development, and (b) have a high degree of trust in GiveWell (Elie is the manager of the fund). The description of the fund notes that GiveWell’s Incubation Grants (GIG) program has not been hampered by insufficient funding to date. However, I think it may be useful to know if there is a large pool of donors who would like to see more GIG-type giving on the margin; giving to this fund would be a good way to show support for GIG. I ultimately chose not to donate to this fund because a substantial portion of my job is to work on GIG, and I would rather leave it to external observers to assess whether they think it deserves further support. (I see GiveWell’s top charities differently because we’ve thoroughly publicly justified the case for giving to these charities.)

Sophie Monahan

I believe that all of GiveWell’s recommended charities are excellent giving opportunities. I believe that donating to GiveWell for granting to recommended charities is an excellent option (allowing valuable flexibility) for donors whose values align well with GiveWell’s. (On values: more, more, more.)

This year, I am giving to No Lean Season for the following reasons:

• I place greater value on reducing near-term poverty for adults and children of all ages relative to preventing deaths of very young children, compared to GiveWell as a whole. I also value certainty in the near-term impact of programs relatively more. According to my values, No Lean Season and GiveDirectly are undervalued by GiveWell. Therefore I believe that their funding gaps are a higher priority for people with values like mine.
• I am giving to No Lean Season despite the fact that this year, GiveWell recommended sufficient funding to No Lean Season to fill its highest priority funding gap (to implement its program in Bangladesh in the next three years), citing reasons other than pure considerations of cost-effectiveness,1“While No Lean Season’s cost-effectiveness is at the lower end of our top charities (~5x cash transfers), we see additional reasons to prioritize this gap. We believe No Lean Season is the top charity where there is the strongest case to be made for “upside”; our cost-effectiveness analysis may not capture the potential impact of scaling a new program that could lead to greater visibility and funding for a novel type of program.” GiveWell blog: Our top charities for giving season 2017 jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); which may not repeat in future years. Therefore, I believe that marginal donations to No Lean Season are likely to increase No Lean Season’s multiyear funding.
• I am also moved by reasons of sentiment—I led GiveWell’s evaluation of No Lean Season and was very impressed—and to promote the visibility of this new top charity.

Catherine Hollander

I plan to give 90% of my donation to the Against Malaria Foundation (AMF) and 10% to No Lean Season. I trust GiveWell’s review process and its recommendation of AMF as having one of the highest-value funding gaps to fill.

Like Sophie, I plan to support No Lean Season to increase the visibility of a new charity on the list, and because it was the charity review I engaged with most deeply and personally in my time at GiveWell—I traveled with Sophie and Christian to Bangladesh to visit No Lean Season and review its work in September. I plan to limit my donation to No Lean Season to 10% of my total gift because I do not believe its need for funding is as pressing as that of AMF.

Andrew Martin

This year, I’m planning to donate to GiveWell for granting to top charities at its discretion.

I think the case remains as strong as ever that donating to GiveWell’s top charities is an exceptional opportunity for donors who want to maximize their impact. Even after accounting for Good Ventures’ $75 million in grants based on GiveWell’s recommendations this year, we believe that our top charities still have a large amount of room for more funding and that additional donations will accomplish a lot of good.

I’ve decided to donate to GiveWell for granting to top charities at its discretion—rather than donating directly to individual top charities—because I believe it’s valuable for GiveWell to have the flexibility to provide funding to whichever top charities have the most pressing funding needs.

Chelsea Tabart

I plan to donate to GiveWell for granting to top charities at its discretion this year. Another year of exposure to the thoughtful, rigorous work of my colleagues has increased my belief in GiveWell’s research process, and I’m excited that my giving can be a small part of the exceptional work our top charities do.

Christian Smith

I’m planning to direct all of my year-end donations to GiveWell for granting to top charities at its discretion. Several of our charities have large, high-priority funding gaps, and I’m excited to be supporting work that I expect to have a large positive impact.

I think there are reasonable worldviews and ethical positions that would make thoughtful giving in other cause areas (e.g. basic research, animal welfare, or improving the far future) appear much more cost-effective than thoughtful giving to organizations involved in global health and development. I considered directing some of my donations towards these cause areas, but ultimately had a preference for supporting causes in global health and development. I feel fine about this decision, but I may have approached my giving differently if I was not working for GiveWell.

Isabel Arjmand

The allocation of my charitable giving this year will be quite similar to what I did last year, though with a slightly higher proportion going to GiveWell’s top charities. As a general note, I’m inclined to diversify my giving between (1) organizations that are promising from a utilitarian point of view (like GiveWell’s top charities) and (2) those that appeal to different moral considerations.

I’m excited to give 75% of my donation to GiveWell for granting to recommended charities at its discretion. My understanding of GiveWell’s research is much deeper than it was at this point last year (when I was a fairly new staff member), and I remain very enthusiastic about the quality of GiveWell’s recommendations. Especially as someone whose moral values are very close to the median values in our cost-effectiveness analyses, I think giving for granting at GiveWell’s discretion is my best option for impact-focused giving. I considered giving directly to Malaria Consortium’s seasonal malaria chemoprevention program, as I think that it may have the highest-impact remaining funding gap of our top charities, but ultimately the flexibility of giving to GiveWell’s discretionary fund and my trust in GiveWell’s judgment lead me to prefer that option.

Additionally, I plan to give 5% of my donation to GiveDirectly, a GiveWell-recommended organization that I also supported last year. I’m supporting GiveDirectly because I think it’s an exceptionally strong, innovative organization with high potential for ‘upside,’ including the potential to serve as a model for other organizations. I’m thinking of the rationale for this portion of my giving as somewhere in between what I describe in the previous paragraph and what I describe below.

Like last year, I’m thinking of the remainder of my charitable contribution (in this case, 20%) as serving a different, less impact-focused purpose. My goals with this portion of my giving are to promote more justice-focused causes, further my own civic engagement, and signal support for work I’d like to see more of. I’d be surprised if any of the organizations below were as cost-effective as GiveWell’s top charities; I also haven’t vetted them with an intensity that comes anywhere close to the rigor of GiveWell evaluations. Of the four organizations among which I plan to divide this donation, the first two are organizations I supported last year, and the remaining two are new to my list.

• Causa Justa :: Just Cause: As I wrote last year, I see supporting Causa Justa :: Just Cause—a Bay Area-based grassroots organization supporting housing rights, immigrant rights, and racial justice—as a means of supporting the community in which I live.
• Planned Parenthood: Reproductive justice and access to healthcare continue to be important to me. Particularly given the absence of a GiveWell-recommended organization providing these services abroad (which I’d guess may be more cost-effective), I’m happy to donate to a U.S.-based organization that I’m personally familiar with and have confidence in.
• ProPublica: I’m donating to ProPublica in support of its high-quality independent journalism, which I think is critical to a well-functioning civic society.
• Earthjustice: I decided this year to support an organization working on climate change and environmental justice, and after researching the space briefly, I found Earthjustice—which focuses on environmental protection via legal advocacy—most compelling.

James Snowden (Research Consultant)

This year, I donated monthly through the Effective Altruism Global Health and Development Fund, so I’ve already made the majority of my personal annual donations this year. At the end of year, I set my priorities for the next year, and adjust my automatic monthly donations accordingly.

In practice, donating to the Global Health and Development Fund is similar to donating to GiveWell for granting to top charities at its discretion, as Elie is the fund manager. My primary reason for donating to Effective Altruism Funds rather than directly to GiveWell is that I want to signal support for a project I think is valuable.

I plan to continue giving 80% of my donations to the Global Health and Development fund, but now donate 10% each to the Animal Welfare fund and Long-term Future fund. I think animal welfare and improving the long-term future are extremely important (more so than I did last year). I don’t feel I have enough context to independently evaluate organizations in this area so want to outsource my decisions to Lewis Bollard (for animal welfare), and Nick Beckstead (for global catastrophic risks). I’m uncertain whether, given Good Ventures’ support for animal welfare and the relatively small number of funding opportunities, there’s substantial room for more funding in that area. But I think the ‘worst case scenario’ is that I funge with Good Ventures, which I’d still think was a reasonably good outcome.

I also considered:

• Donating to the Centre for Pesticide Suicide Prevention. I was the primary researcher working on this GiveWell Incubation Grant and believe it’s a potentially very cost-effective (though risky) giving opportunity. I decided not to donate because the Incubation Grant is intended to fully cover their costs for two years.
• Donating to Malaria Consortium’s Seasonal Malaria Chemoprevention (SMC) program. Having spent more time looking into SMC this year, I believe it’s a more cost-effective giving opportunity than Against Malaria Foundation (our recommendation to donors this year is 70% Against Malaria Foundation and 30% Schistosomiasis Control Initiative). I decided not to donate to the SMC program this year because (i) we recommended Good Ventures grant $27.9m to Malaria Consortium this year (vs $2.5m for AMF), and I don’t have a strong view that SMC would be more cost-effective on the margin after this grant (ii) I think allowing GiveWell to regrant at its discretion allows for more flexibility (iii) I place more weight in GiveWell’s aggregate view than the inside view of any individual researcher (including myself!)—although I think there’s value in thinking about this independently to identify if GiveWell is making decisions I disagree with.
• Continuing to donate only to the Global Health and Development fund. I think there’s a strong argument for just donating to the opportunity you think is best in expectation, rather than diversifying. I decided to diversify a fairly small amount because (i) it more accurately signals that I care about those areas (ii) it motivates me to learn more about those areas than I otherwise would have (although I don’t expect this to be a major priority for me).
• Donating a larger proportion to the Animal Welfare and Far Future funds. Given my relatively greater knowledge in global health and development, I don’t yet feel comfortable giving a greater proportion to areas I know less about.

Aside from my personal giving, I advise a small foundation on their grantmaking. We haven’t yet decided where to give this year, and this will partly depend on the priorities of others involved in the decision.

Notes   [ + ]

1. ↑ “While No Lean Season’s cost-effectiveness is at the lower end of our top charities (~5x cash transfers), we see additional reasons to prioritize this gap. We believe No Lean Season is the top charity where there is the strongest case to be made for “upside”; our cost-effectiveness analysis may not capture the potential impact of scaling a new program that could lead to greater visibility and funding for a novel type of program.” GiveWell blog: Our top charities for giving season 2017 function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Staff members’ personal donations for giving season 2017 appeared first on The GiveWell Blog.

Summary

• Four of GiveWell's top charities support deworming---the mass distribution of medicines to children in poor countries to rid their bodies of schistosomiasis, hookworm, and parasites.
• GiveWell's recommendation relies primarily on research from western Kenya finding that deworming in childhood boosted income in adulthood. GiveWell has also placed weight on a study by Hoyt Bleakley of the hookworm eradication effort in the American South 100 years ago.
• I reviewed the Bleakley study and reach a different conclusion than he did: the deworming campaign in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation.
• GiveWell research staff took the conclusions of this post into account when updating their recommendations for the 2017 giving season. GiveWell continues to recommend deworming charities.
• I also reviewed a separate Bleakley study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. My reading there is more supporting. I'm finalizing the write-ups now and will share them soon.

Read More

The post Questioning the evidence on hookworm eradication in the American South appeared first on The GiveWell Blog.

Summary

• Four of GiveWell's top charities support deworming---the mass distribution of medicines to children in poor countries to rid their bodies of schistosomiasis, hookworm, and parasites.
• GiveWell's recommendation relies primarily on research from western Kenya finding that deworming in childhood boosted income in adulthood. GiveWell has also placed weight on a study by Hoyt Bleakley of the hookworm eradication effort in the American South 100 years ago.
• I reviewed the Bleakley study and reach a different conclusion than he did: the deworming campaign in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation.
• GiveWell research staff took the conclusions of this post into account when updating their recommendations for the 2017 giving season. GiveWell continues to recommend deworming charities.
• I also reviewed a separate Bleakley study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. My reading there is more supporting. I'm finalizing the write-ups now and will share them soon.

Read More

The post Questioning the evidence on hookworm eradication in the American South appeared first on The GiveWell Blog.

Summary

• Four of GiveWell’s top charities support deworming—the mass distribution of medicines to children in poor countries to rid their bodies of schistosomiasis, hookworm, and parasites.
• GiveWell’s recommendation relies primarily on research from western Kenya finding that deworming in childhood boosted income in adulthood. GiveWell has also placed weight on a study by Hoyt Bleakley of the hookworm eradication effort in the American South 100 years ago.
• I reviewed the Bleakley study and reach a different conclusion than he did: the deworming campaign in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation.
• GiveWell research staff took the conclusions of this post into account when updating their recommendations for the 2017 giving season. GiveWell continues to recommend deworming charities.
• I also reviewed a separate Bleakley study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. My reading there is more supporting. I’m finalizing the write-ups now and will share them soon.

Introduction

After the latest refresh, GiveWell’s list of top charities includes four that support deworming—the mass distribution of medicines to children to rid their guts of certain parasites. Several dozen randomized studies measure the short-term effects of deworming programs (within a year or so) on everything from body weight to being in school.1The 2016 Campbell review finds 52 short-term studies with follow-up duration under five years. Most last one to two years. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); If intestinal worms were often fatal, then short-term gains against them might be measured in lives saved, which could on its own make a decisive case for deworming. But the symptoms are normally subtler. On the other hand, some research finds that the aftereffects last into adulthood. This is why the long-term effects of deworming dominate GiveWell’s estimates of the cost-effectiveness of charities that support it.

Unfortunately, only a handful of experimental studies assess deworming’s impacts over the long haul, and most of those are based on a single experiment in Kenya. For summaries, see this 2016 post, in the section entitled “The research on the long-term impacts of deworming.” This paucity of experimental evidence has led GiveWell to place weight on a non-experimental, historical study of deworming. Hoyt Bleakley‘s 2007 paper tracks the impacts of the campaign to eradicate hookworm from the American South a century ago.

As part of an ongoing effort to scrutinize the evidence on the long-term impacts of deworming (this, this), GiveWell worked over the past year to revisit the Bleakley study. With huge assists from Christian Smith, Zachary Tausanovitch, and Claire Wang, I have formed a fresh and critical assessment of the evidence. The hookworm eradication effort in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation. For example, after the eradication campaign, outcomes such as school attendance indeed rose faster for children in historically worm-endemic areas, which could be taken as a sign of success. But that trend began decades before eradication. The full write-up is in this new working paper.

As John D. Rockefeller, arguably the richest human in history, entered philanthropy just over a century ago, he was persuaded to back large-scale, scientifically informed public health campaigns—not unlike Bill Gates in our era. In 1910, he gave $1 million to create the Rockefeller Sanitary Commission for the Eradication of Hookworm Disease. Across eleven southern states from North Carolina to Texas, the RSC soon launched what today would be called the War on Worms. Drugs were dispensed to treat infected children. Doctors, teachers, and the public were educated about the importance of sanitation, especially the use of proper privies.

From a researcher’s point of view, the suddenness and success of the campaign, and its broad geographic sweep, offer hope for credible impact assessment. If, for example, school attendance rates jumped just as infection rates plunged, that could be a compelling sign of the knock-on effects of mass deworming of children. The Bleakley (2007) study recognizes and exploits this opportunity for impact assessment. Paralleling the modern research out of Kenya, the study finds that after the RSC campaign, children in formerly worm-afflicted areas went to school more (a short-term development) and earned more as adults (a long-term effect).

In this post, I’ll explain how the GiveWell reanalysis of the Bleakley (2007) hookworm research differs from Bleakley’s original. Then I will show you some graphs that tell most of the analytical story.

I have also reviewed the related Bleakley (2010) study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. There, my conclusion is more positive. I hope to release and blog that review in the next few weeks.

What we did

The reanalysis of the Bleakley (2007) hookworm study included the following steps:

• Returning to primary sources to reconstruct the data set. The data and computer code for the study are not publicly available. In correspondence starting a year ago, Hoyt Bleakley stated that they are effectively inaccessible now. Re-gathering the data was a major undertaking because Bleakley culled nearly 50 variables from obscure, century-old books and articles. Some, such as the student-teacher ratio in each county of the eleven southern states, were found in state government reports that varied in completeness and reporting conventions. Christian Smith, Claire Wang, and, especially, Zachary Tausanovitch, poured many hours into this effort.
• Expanding the census data sets. Bleakley (2007) tracks outcomes such as school attendance, literacy, and income using U.S. census data. These come to us not from old books, but from the IPUMS online database. Until recently, all the IPUMS data sets were samples from a given year’s census records, taking, for example, one household from every fifth page of the enumeration. (Here’s a sample page from 1920 with my great-grandparents and family in rows 3–6.) When carrying out this research in 2003–05, Bleakley appears to have used the biggest sets then available, such as the 1-in-250 sample from the 1910 census and the 1-in-100 sample from 1920. No data were then to be had from 1930. The GiveWell reanalysis takes advantage of the newer, bigger samples, including preliminary 100% samples for 1910–40. In aggregate, the new data set is about 100 times larger than that in Bleakley (2007).
• Copying choices from one Bleakley (2007) table or figure to another. For example, one table in the paper estimates impacts on school enrollment, school attendance, and literacy. A corresponding figure, discussed soon, only depicts impacts on attendance. In the new paper, I rerun the figure for all three outcomes.
• Imposing an arguably tougher standard for proof of impact. I concur with Bleakley that after the eradication campaign swept through the South in 1911–14, prospects improved disproportionately for children born in areas historically prone to hookworm. This catch-up, or convergence, surfaces in the data whether comparing counties within the South (low-lying counties tended to have more hookworm than mountainous ones), or comparing southern states to other states. But that observation alone leaves me unconvinced that ridding children’s bellies of hookworm was the cause. What if the trend began well before eradication or continued well after? I therefore focus on this question: Did convergence temporarily accelerate in tandem with eradication? The Bleakley (2007) tables and figures do not approach this question so aggressively.

We shared drafts of the paper and this post with Hoyt Bleakley. This did not yield any additional insight into why our analysis differs from the original.

The short-term impact on schooling

The figure below, adapted from one edition of the Bleakley study, illustrates the finding that I just mentioned, that after eradication, school attendance surged among kids living where hookworm had been common.2Versions of Bleakley (2007) appeared in the Quarterly Journal of Economics, a World Bank report, and the site of the National Center for Biotechnology Information. They are nearly the same. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); I will convey the gist of the figure first, then explain it more precisely. You can see that the central red line stays essentially flat from 1870 to 1910. Then it jumps to about zero between 1910 and 1920, census years bracketing the Rockefeller campaign. Thereafter, the red line mostly again holds steady. The one-time jump looks like a fingerprint of eradication.

What does the red line mean exactly? For each census round with available data between 1870 and 1950, Bleakley (2007) computes the association within Southern counties between the school attendance rate of 8–16-year-olds and the hookworm infection rate as measured at the start of eradication, circa 1910.3The regressions for each census year control for the interactions of sex and race on the one hand and age on the other. They do not include the other Bleakley (2007) controls. Samples are restricted to eleven Southern states. The unit of observation is the State Economic Area, which is an aggregation of several counties. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); That the red line starts around –0.1 in 1870 means that on average, if a county’s child hookworm infection rate was 10 percentage points higher when measured around 1910, its school attendance rate in the 1870 census was 1 percentage point lower. More plainly, counties with more worms in kids had fewer kids in school. But between the 1910 and 1920 censuses, that bad-news association abruptly faded. As of 1920, a child in a historically high-hookworm county was no less likely to be in school. The black, dashed lines show confidence intervals for these census-by-census estimates—probably 95% confidence, but I cannot tell for sure.

Here is the best replication of that graph using the reconstructed data and code. I have drawn it differently to emphasize that we only have data from certain decennial censuses, and to depict the gradations of confidence within the 95% confidence intervals.4The 1890 census records were destroyed in a fire. 1930 records had not been digitized at the time Bleakley did this work. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

I discern a resemblance between the original graph and the reconstruction. In both, school enrollment rises especially quickly between 1910 and 1920 and then declines slightly. But there is a difference too, and it is more than cosmetic. Now it appears that children in hookworm-infested areas gained substantially on school attendance not just between 1910 and 1920 but between 1880 and 1900 as well—and maybe throughout 1880–1910. For lack of access to Bleakley’s data and code, I cannot explain the discrepancy between this reconstruction and the original. There could be an error in the new or the old, or some subtle difference in data or method.

The new graph’s ambiguous mix of confirmation and contradiction forces a question that is at once conceptual and practical. How do we systematically judge whether the signal of hookworm eradication is present amidst the noise of other influences? To what degree does the new graph confirm or contradict the old?

I think there is no one best way to answer that question. One approach that I took is depicted with the red lines in the reconstructed graph above, and in the p values in the bottom-left corner. I drew the red lines to connect the dots that surround the eradication campaign. I wanted to quantify how much the red contour bends upward in 1910 and downward in 1920—as in Bleakley’s graph—and with what statistical significance. That is: Suppose the education gains took place at a constant pace between 1900 and 1940 with no acceleration around the campaign in the early 1910s. (I would have substituted 1930 for 1940 were 1930 data available in this graph.) What is the chance that we would see as much bending in the red line as we do? The computer says that for the upward kink at 1910, the answer is 0.37, which is not very low. On the other hand, the deceleration around 1920 is quite hard to ascribe to pure chance, at p = 0.03. Still, the new graph casts doubt on the proposition that the campaign brought a big break with the past.

Having settled on an analytical approach, the next step was to add all the census data that has been digitized since Bleakley did his work. This brings an obvious change (see below; now that 1930 data are included, I extend the third red line only that far). Now it looks far more as though the high-hookworm parts of the South began closing the schooling gap with low-hookworm parts around 1880, some 30 years before the hookworm campaign:

In a final test, I recomputed the graph while incorporating all the Bleakley (2007) control variables. Hookworm eradication was hardly a clean experiment, in the sense that the geographic reach of the disease was not random going in. The South had it more than the rest of the country; within the South, the coastal plains had it most. If the beneficiaries of eradication differed systematically from the rest before eradication, they could continue to differ after for reasons having little to do with hookworm prevention, creating a false appearance of impact. Striving to statistically remove such initial differences, Bleakley (2007) introduces into some of the regressions an aggressive set of controls. They relate to education, health, agriculture, and race. The paper includes these controls in some of the schooling regressions reported in a table, but does not bring them to the schooling graph shown above. It turns out that doing so (in our expanded-data graph) removes most signs of any long-term gains:

The lack of upward trend here does not mean that the historically hookworm-burdened parts of the South did not after all close a schooling gap between 1880 and 1920. It does suggest that the closure was correlated with, and therefore potentially caused by, the non-hookworm factors that Bleakley sometimes controls for.5Consistent with this graph, while the Bleakley (2007) full-controls regressions continue to put a statistically significant coefficient on the treatment proxy, the reconstructions do not. This is one of the few cases where the original results are not recognizable in the reconstruction. See Table 6, panel B, of the new paper. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

In sum, I do not see robust evidence that schooling and literacy improved at an historically anomalous rate circa 1910, in a way naturally attributable to hookworm eradication.

The long-term impact on earnings

What the first half of the Bleakley (2007) study does for short-term impacts on schooling, the second does for long-term impacts on earnings. Here too, the conclusion is encouraging. “Long-term follow-up,” writes Bleakley, “indicates a substantial income gain as a result of the reduction in hookworm infection.” This finding resonates strongly with the GiveWell cost-effectiveness analysis, which makes a key assumption about how much deworming children boosts future income. The number we use for that impact comes from modern, experimental research in Kenya; yet Bleakley’s inference from American history had boosted our confidence in the Kenya number. (That said, GiveWell has discounted the Kenya number by 80–90% out of fear that it won’t replicate to other settings.6See the “Replicability adjustment for deworming” row of the “Parameters” tab of the cost-effectiveness analysis spreadsheet. jQuery("#footnote_plugin_tooltip_6").tooltip({ tip: "#footnote_plugin_tooltip_text_6", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });)

The Bleakley (2007) graph I will focus on draws together data from censuses as ancient as 1870 and modern as 1990. One problem with measuring impacts on income over this span is that not until 1940 did Census takers begin asking people how much money they made. For this reason, the IPUMS census database provides proxies for income that reach back farther. One is the occupational income score (OIS), which is, approximately speaking, the average income in 1950 associated with a person’s census-reported profession. Thus, if lawyers averaged $10,000 in income in 1950, then any self-described lawyer between 1870 and 1990 is taken to earn that much. The OIS is expressed in hundreds of dollars of 1950, and is an example of an index of “occupational standing.”

Before scrutinizing the evidence of long-term impacts on occupational standing, I need to describe a twist that Bleakley (2007) introduces in moving from short- to long-term. As one tries to follow people over longer periods of time, the analytical tack that Bleakley took for schooling starts to break down. For it looks at how the people in given places fared over time. The problem is that sometimes people move—across the state or across the country. And in this analytical set-up, the researcher does not follow them. If deworming gave children in coastal Georgia more agency in life—better health, more education—perhaps they exercised that agency by moving to Atlanta. If we only looked at the incomes of the people who stayed behind, we would miss the full story.

To minimize this attrition from migration, in studying long-term impacts, Bleakley (2007) groups census records not by place of residence at the time of census, but by place of birth. Then, if a person was born in Georgia in 1915, showed up in the census in 1940 as a bricklayer in Atlanta, in 1950 as a general contractor in Lexington, and in 1960 as the manager of a construction company in Phoenix, all three census records would be associated with Georgia in 1915. After organizing the data this way, Bleakley (2007) could study whether children born in certain areas after eradication went on to earn more than those born in the same places before eradication.

Reorganizing the data this way generates two ripple effects. First, while census takers record place of residence with extreme precision, they only record place of birth by state. We cannot differentiate people by whether they were born in hookworm-prone areas within, say, Mississippi. We can only differentiate by whether they were native to a historically high-hookworm state such as Mississippi or a low-hookworm one such as Michigan. Thus, while the short-term analysis compares counties within 11 southern states, the long-term analysis compares states across the continental U.S.

The second ripple effect is that the data come to us at higher temporal resolution: by birth year, not census decade. In response, Bleakley (2007) hypothesizes that how much hookworm depressed adult earnings depended on the percentage of one’s childhood spent where it was endemic. If we take eradication to have occurred in 1910 and assume with Bleakley (2007) that childhood lasts 19 years, then babies born in or before 1891 would have reached adulthood before eradication, too soon to benefit. Babies born in endemic areas in 1892 would have been helped for one year (between their 18th and 19th birthdays); in 1893 for two; and so on. Those born in 1910 or later stood to reap the full 19 years of benefit. Bleakley (2007) therefore hypothesizes that the impact of eradication follows a sort of diagonal step shape with respect to birth year. The step starts rising in 1891 and stops in 1910. Bleakley depicted that contour with dashed lines in this figure:

As you can see, Bleakley (2007) fit this contour to data, to see how well it could explain historical patterns. These dots are derived much as in the earlier Bleakley (2007) figure. For example, the leftmost dot is for the year 1825, and has a vertical coordinate of about –2. That means that among people born in 1825, being native to a state whose hookworm infection rate circa 1910 was 10 percentage points (0.1) higher corresponded to having an Occupational Income Score 0.20 lower. That means $20/year less income throughout adulthood, in the dollars of 1950. The graph shows that this association was generally negative in the mid-19th century and generally positive after 1910: formerly, coming from a hookworm zone depressed lifetime earnings. And the graph suggests that the transition followed the step pattern expected if the cause was hookworm eradication.

Below is my best reproduction of that graph. As before, I have plotted both the dots and their 95% confidence intervals. I have avoided superimposing the step-like contour the way Bleakley (2007) does because I worry that it tricks the eye into believing that the contour fits the data better than it really does. But I have marked the years when the contour kinks, 1891 and 1910:

Here is the same graph when using the 100-times-bigger census data sets now available7In addition to adding data, this version mimics the rest of the Bleakley (2007) analysis in adding blacks and in fitting directly to census microdata rather than aggregates, in order to include controls for race, sex, and their interaction. jQuery("#footnote_plugin_tooltip_7").tooltip({ tip: "#footnote_plugin_tooltip_text_7", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });:

And here is the graph when I copy Bleakley (2010) in incorporating all the controls for cross-state differences in health and health policy, education policy, and other traits8As I noted, when looking at short-term impacts on education, Bleakley (2007) does not plot a graph while incorporating all controls. But now, when looking at long-term impacts on occupational standing, Bleakley (2007) does also include such a graph. See the bottom right of this figure. jQuery("#footnote_plugin_tooltip_8").tooltip({ tip: "#footnote_plugin_tooltip_text_8", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });:

Does it look to you like the upward trends in these last two graphs accelerated around 1891 and decelerated around 1910, as predicted by the Bleakley (2007) theory? To me, I have to say, not much. The climbs look steady and longer-term.

Since “not much” is muddy, I moved once again to formalize my interpretation. In analogy with my earlier graphs for schooling, I fit lines to the data points in the 19 years between 1891 and 1910, as well as to the 19 years on either side. Then I checked whether any bending in 1891 and 1910 is statistically significant. The final two graphs fit lines to the dots in the previous two. The dots in these next graphs are the same as in the previous two. It doesn’t look that way because I erased the grey confidence bars in order to expand the vertical scales.

In the both graphs the trend looks quite straight over the three generations surrounding the eradication campaign. The p values, shown in the bottom-right of each plot, are high.

Conclusion

Reanalyzing the Bleakley (2007) study left me unconvinced that the children who benefited from hookworm eradication went to school more or earned more as adults. Conceivably, if I had access to the original data and code, the confrontation with the reconstructed versions would expose errors in the GiveWell version that would alter my view. But this seems unlikely. The new census data sets are much bigger than the old, which improves precision. And most of the differences probably do not arise from clear-cut errors on either side, but from minor differences in implementation, such as taking education spending from a different edition of an annual government report. If the conclusions swing on such modest and debatable discrepancies, then they are not robust and reliable.

Finally, even if the two versions of the data matched exactly, I might still disagree on interpretation, since I use tests, illustrated above, that focus more exclusively on whether the time trends contain the temporal fingerprint of hookworm eradication. For me, that fingerprint is characterized not merely by once-high-hookworm areas catching up with low-hookworm ones, but catch-up that accelerates and decelerates at times that fit the timing of the eradication campaign.

The data and code for this study are here (1.6 GB). The full write-up is here.

 

Notes   [ + ]

1. ↑ The 2016 Campbell review finds 52 short-term studies with follow-up duration under five years. Most last one to two years. 2. ↑ Versions of Bleakley (2007) appeared in the Quarterly Journal of Economics, a World Bank report, and the site of the National Center for Biotechnology Information. They are nearly the same. 3. ↑ The regressions for each census year control for the interactions of sex and race on the one hand and age on the other. They do not include the other Bleakley (2007) controls. Samples are restricted to eleven Southern states. The unit of observation is the State Economic Area, which is an aggregation of several counties. 4. ↑ The 1890 census records were destroyed in a fire. 1930 records had not been digitized at the time Bleakley did this work. 5. ↑ Consistent with this graph, while the Bleakley (2007) full-controls regressions continue to put a statistically significant coefficient on the treatment proxy, the reconstructions do not. This is one of the few cases where the original results are not recognizable in the reconstruction. See Table 6, panel B, of the new paper. 6. ↑ See the “Replicability adjustment for deworming” row of the “Parameters” tab of the cost-effectiveness analysis spreadsheet. 7. ↑ In addition to adding data, this version mimics the rest of the Bleakley (2007) analysis in adding blacks and in fitting directly to census microdata rather than aggregates, in order to include controls for race, sex, and their interaction. 8. ↑ As I noted, when looking at short-term impacts on education, Bleakley (2007) does not plot a graph while incorporating all controls. But now, when looking at long-term impacts on occupational standing, Bleakley (2007) does also include such a graph. See the bottom right of this figure. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Questioning the evidence on hookworm eradication in the American South appeared first on The GiveWell Blog.

Summary

• Four of GiveWell’s top charities support deworming—the mass distribution of medicines to children in poor countries to rid their bodies of schistosomiasis, hookworm, and parasites.
• GiveWell’s recommendation relies primarily on research from western Kenya finding that deworming in childhood boosted income in adulthood. GiveWell has also placed weight on a study by Hoyt Bleakley of the hookworm eradication effort in the American South 100 years ago.
• I reviewed the Bleakley study and reach a different conclusion than he did: the deworming campaign in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation.
• GiveWell research staff took the conclusions of this post into account when updating their recommendations for the 2017 giving season. GiveWell continues to recommend deworming charities.
• I also reviewed a separate Bleakley study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. My reading there is more supporting. I’m finalizing the write-ups now and will share them soon.

Introduction

After the latest refresh, GiveWell’s list of top charities includes four that support deworming—the mass distribution of medicines to children to rid their guts of certain parasites. Several dozen randomized studies measure the short-term effects of deworming programs (within a year or so) on everything from body weight to being in school.1The 2016 Campbell review finds 52 short-term studies with follow-up duration under five years. Most last one to two years. jQuery("#footnote_plugin_tooltip_1").tooltip({ tip: "#footnote_plugin_tooltip_text_1", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); If intestinal worms were often fatal, then short-term gains against them might be measured in lives saved, which could on its own make a decisive case for deworming. But the symptoms are normally subtler. On the other hand, some research finds that the aftereffects last into adulthood. This is why the long-term effects of deworming dominate GiveWell’s estimates of the cost-effectiveness of charities that support it.

Unfortunately, only a handful of experimental studies assess deworming’s impacts over the long haul, and most of those are based on a single experiment in Kenya. For summaries, see this 2016 post, in the section entitled “The research on the long-term impacts of deworming.” This paucity of experimental evidence has led GiveWell to place weight on a non-experimental, historical study of deworming. Hoyt Bleakley‘s 2007 paper tracks the impacts of the campaign to eradicate hookworm from the American South a century ago.

As part of an ongoing effort to scrutinize the evidence on the long-term impacts of deworming (this, this), GiveWell worked over the past year to revisit the Bleakley study. With huge assists from Christian Smith, Zachary Tausanovitch, and Claire Wang, I have formed a fresh and critical assessment of the evidence. The hookworm eradication effort in the American South did not coincide with breaks in long-term trends that would invite eradication as the explanation. For example, after the eradication campaign, outcomes such as school attendance indeed rose faster for children in historically worm-endemic areas, which could be taken as a sign of success. But that trend began decades before eradication. The full write-up is in this new working paper.

As John D. Rockefeller, arguably the richest human in history, entered philanthropy just over a century ago, he was persuaded to back large-scale, scientifically informed public health campaigns—not unlike Bill Gates in our era. In 1910, he gave $1 million to create the Rockefeller Sanitary Commission for the Eradication of Hookworm Disease. Across eleven southern states from North Carolina to Texas, the RSC soon launched what today would be called the War on Worms. Drugs were dispensed to treat infected children. Doctors, teachers, and the public were educated about the importance of sanitation, especially the use of proper privies.

From a researcher’s point of view, the suddenness and success of the campaign, and its broad geographic sweep, offer hope for credible impact assessment. If, for example, school attendance rates jumped just as infection rates plunged, that could be a compelling sign of the knock-on effects of mass deworming of children. The Bleakley (2007) study recognizes and exploits this opportunity for impact assessment. Paralleling the modern research out of Kenya, the study finds that after the RSC campaign, children in formerly worm-afflicted areas went to school more (a short-term development) and earned more as adults (a long-term effect).

In this post, I’ll explain how the GiveWell reanalysis of the Bleakley (2007) hookworm research differs from Bleakley’s original. Then I will show you some graphs that tell most of the analytical story.

I have also reviewed the related Bleakley (2010) study of the impacts of malaria eradication in the United States, Brazil, Colombia, and Mexico. There, my conclusion is more positive. I hope to release and blog that review in the next few weeks.

What we did

The reanalysis of the Bleakley (2007) hookworm study included the following steps:

• Returning to primary sources to reconstruct the data set. The data and computer code for the study are not publicly available. In correspondence starting a year ago, Hoyt Bleakley stated that they are effectively inaccessible now. Re-gathering the data was a major undertaking because Bleakley culled nearly 50 variables from obscure, century-old books and articles. Some, such as the student-teacher ratio in each county of the eleven southern states, were found in state government reports that varied in completeness and reporting conventions. Christian Smith, Claire Wang, and, especially, Zachary Tausanovitch, poured many hours into this effort.
• Expanding the census data sets. Bleakley (2007) tracks outcomes such as school attendance, literacy, and income using U.S. census data. These come to us not from old books, but from the IPUMS online database. Until recently, all the IPUMS data sets were samples from a given year’s census records, taking, for example, one household from every fifth page of the enumeration. (Here’s a sample page from 1920 with my great-grandparents and family in rows 3–6.) When carrying out this research in 2003–05, Bleakley appears to have used the biggest sets then available, such as the 1-in-250 sample from the 1910 census and the 1-in-100 sample from 1920. No data were then to be had from 1930. The GiveWell reanalysis takes advantage of the newer, bigger samples, including preliminary 100% samples for 1910–40. In aggregate, the new data set is about 100 times larger than that in Bleakley (2007).
• Copying choices from one Bleakley (2007) table or figure to another. For example, one table in the paper estimates impacts on school enrollment, school attendance, and literacy. A corresponding figure, discussed soon, only depicts impacts on attendance. In the new paper, I rerun the figure for all three outcomes.
• Imposing an arguably tougher standard for proof of impact. I concur with Bleakley that after the eradication campaign swept through the South in 1911–14, prospects improved disproportionately for children born in areas historically prone to hookworm. This catch-up, or convergence, surfaces in the data whether comparing counties within the South (low-lying counties tended to have more hookworm than mountainous ones), or comparing southern states to other states. But that observation alone leaves me unconvinced that ridding children’s bellies of hookworm was the cause. What if the trend began well before eradication or continued well after? I therefore focus on this question: Did convergence temporarily accelerate in tandem with eradication? The Bleakley (2007) tables and figures do not approach this question so aggressively.

We shared drafts of the paper and this post with Hoyt Bleakley. This did not yield any additional insight into why our analysis differs from the original.

The short-term impact on schooling

The figure below, adapted from one edition of the Bleakley study, illustrates the finding that I just mentioned, that after eradication, school attendance surged among kids living where hookworm had been common.2Versions of Bleakley (2007) appeared in the Quarterly Journal of Economics, a World Bank report, and the site of the National Center for Biotechnology Information. They are nearly the same. jQuery("#footnote_plugin_tooltip_2").tooltip({ tip: "#footnote_plugin_tooltip_text_2", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); I will convey the gist of the figure first, then explain it more precisely. You can see that the central red line stays essentially flat from 1870 to 1910. Then it jumps to about zero between 1910 and 1920, census years bracketing the Rockefeller campaign. Thereafter, the red line mostly again holds steady. The one-time jump looks like a fingerprint of eradication.

What does the red line mean exactly? For each census round with available data between 1870 and 1950, Bleakley (2007) computes the association within Southern counties between the school attendance rate of 8–16-year-olds and the hookworm infection rate as measured at the start of eradication, circa 1910.3The regressions for each census year control for the interactions of sex and race on the one hand and age on the other. They do not include the other Bleakley (2007) controls. Samples are restricted to eleven Southern states. The unit of observation is the State Economic Area, which is an aggregation of several counties. jQuery("#footnote_plugin_tooltip_3").tooltip({ tip: "#footnote_plugin_tooltip_text_3", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] }); That the red line starts around –0.1 in 1870 means that on average, if a county’s child hookworm infection rate was 10 percentage points higher when measured around 1910, its school attendance rate in the 1870 census was 1 percentage point lower. More plainly, counties with more worms in kids had fewer kids in school. But between the 1910 and 1920 censuses, that bad-news association abruptly faded. As of 1920, a child in a historically high-hookworm county was no less likely to be in school. The black, dashed lines show confidence intervals for these census-by-census estimates—probably 95% confidence, but I cannot tell for sure.

Here is the best replication of that graph using the reconstructed data and code. I have drawn it differently to emphasize that we only have data from certain decennial censuses, and to depict the gradations of confidence within the 95% confidence intervals.4The 1890 census records were destroyed in a fire. 1930 records had not been digitized at the time Bleakley did this work. jQuery("#footnote_plugin_tooltip_4").tooltip({ tip: "#footnote_plugin_tooltip_text_4", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

I discern a resemblance between the original graph and the reconstruction. In both, school enrollment rises especially quickly between 1910 and 1920 and then declines slightly. But there is a difference too, and it is more than cosmetic. Now it appears that children in hookworm-infested areas gained substantially on school attendance not just between 1910 and 1920 but between 1880 and 1900 as well—and maybe throughout 1880–1910. For lack of access to Bleakley’s data and code, I cannot explain the discrepancy between this reconstruction and the original. There could be an error in the new or the old, or some subtle difference in data or method.

The new graph’s ambiguous mix of confirmation and contradiction forces a question that is at once conceptual and practical. How do we systematically judge whether the signal of hookworm eradication is present amidst the noise of other influences? To what degree does the new graph confirm or contradict the old?

I think there is no one best way to answer that question. One approach that I took is depicted with the red lines in the reconstructed graph above, and in the p values in the bottom-left corner. I drew the red lines to connect the dots that surround the eradication campaign. I wanted to quantify how much the red contour bends upward in 1910 and downward in 1920—as in Bleakley’s graph—and with what statistical significance. That is: Suppose the education gains took place at a constant pace between 1900 and 1940 with no acceleration around the campaign in the early 1910s. (I would have substituted 1930 for 1940 were 1930 data available in this graph.) What is the chance that we would see as much bending in the red line as we do? The computer says that for the upward kink at 1910, the answer is 0.37, which is not very low. On the other hand, the deceleration around 1920 is quite hard to ascribe to pure chance, at p = 0.03. Still, the new graph casts doubt on the proposition that the campaign brought a big break with the past.

Having settled on an analytical approach, the next step was to add all the census data that has been digitized since Bleakley did his work. This brings an obvious change (see below; now that 1930 data are included, I extend the third red line only that far). Now it looks far more as though the high-hookworm parts of the South began closing the schooling gap with low-hookworm parts around 1880, some 30 years before the hookworm campaign:

In a final test, I recomputed the graph while incorporating all the Bleakley (2007) control variables. Hookworm eradication was hardly a clean experiment, in the sense that the geographic reach of the disease was not random going in. The South had it more than the rest of the country; within the South, the coastal plains had it most. If the beneficiaries of eradication differed systematically from the rest before eradication, they could continue to differ after for reasons having little to do with hookworm prevention, creating a false appearance of impact. Striving to statistically remove such initial differences, Bleakley (2007) introduces into some of the regressions an aggressive set of controls. They relate to education, health, agriculture, and race. The paper includes these controls in some of the schooling regressions reported in a table, but does not bring them to the schooling graph shown above. It turns out that doing so (in our expanded-data graph) removes most signs of any long-term gains:

The lack of upward trend here does not mean that the historically hookworm-burdened parts of the South did not after all close a schooling gap between 1880 and 1920. It does suggest that the closure was correlated with, and therefore potentially caused by, the non-hookworm factors that Bleakley sometimes controls for.5Consistent with this graph, while the Bleakley (2007) full-controls regressions continue to put a statistically significant coefficient on the treatment proxy, the reconstructions do not. This is one of the few cases where the original results are not recognizable in the reconstruction. See Table 6, panel B, of the new paper. jQuery("#footnote_plugin_tooltip_5").tooltip({ tip: "#footnote_plugin_tooltip_text_5", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });

In sum, I do not see robust evidence that schooling and literacy improved at an historically anomalous rate circa 1910, in a way naturally attributable to hookworm eradication.

The long-term impact on earnings

What the first half of the Bleakley (2007) study does for short-term impacts on schooling, the second does for long-term impacts on earnings. Here too, the conclusion is encouraging. “Long-term follow-up,” writes Bleakley, “indicates a substantial income gain as a result of the reduction in hookworm infection.” This finding resonates strongly with the GiveWell cost-effectiveness analysis, which makes a key assumption about how much deworming children boosts future income. The number we use for that impact comes from modern, experimental research in Kenya; yet Bleakley’s inference from American history had boosted our confidence in the Kenya number. (That said, GiveWell has discounted the Kenya number by 80–90% out of fear that it won’t replicate to other settings.6See the “Replicability adjustment for deworming” row of the “Parameters” tab of the cost-effectiveness analysis spreadsheet. jQuery("#footnote_plugin_tooltip_6").tooltip({ tip: "#footnote_plugin_tooltip_text_6", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });)

The Bleakley (2007) graph I will focus on draws together data from censuses as ancient as 1870 and modern as 1990. One problem with measuring impacts on income over this span is that not until 1940 did Census takers begin asking people how much money they made. For this reason, the IPUMS census database provides proxies for income that reach back farther. One is the occupational income score (OIS), which is, approximately speaking, the average income in 1950 associated with a person’s census-reported profession. Thus, if lawyers averaged $10,000 in income in 1950, then any self-described lawyer between 1870 and 1990 is taken to earn that much. The OIS is expressed in hundreds of dollars of 1950, and is an example of an index of “occupational standing.”

Before scrutinizing the evidence of long-term impacts on occupational standing, I need to describe a twist that Bleakley (2007) introduces in moving from short- to long-term. As one tries to follow people over longer periods of time, the analytical tack that Bleakley took for schooling starts to break down. For it looks at how the people in given places fared over time. The problem is that sometimes people move—across the state or across the country. And in this analytical set-up, the researcher does not follow them. If deworming gave children in coastal Georgia more agency in life—better health, more education—perhaps they exercised that agency by moving to Atlanta. If we only looked at the incomes of the people who stayed behind, we would miss the full story.

To minimize this attrition from migration, in studying long-term impacts, Bleakley (2007) groups census records not by place of residence at the time of census, but by place of birth. Then, if a person was born in Georgia in 1915, showed up in the census in 1940 as a bricklayer in Atlanta, in 1950 as a general contractor in Lexington, and in 1960 as the manager of a construction company in Phoenix, all three census records would be associated with Georgia in 1915. After organizing the data this way, Bleakley (2007) could study whether children born in certain areas after eradication went on to earn more than those born in the same places before eradication.

Reorganizing the data this way generates two ripple effects. First, while census takers record place of residence with extreme precision, they only record place of birth by state. We cannot differentiate people by whether they were born in hookworm-prone areas within, say, Mississippi. We can only differentiate by whether they were native to a historically high-hookworm state such as Mississippi or a low-hookworm one such as Michigan. Thus, while the short-term analysis compares counties within 11 southern states, the long-term analysis compares states across the continental U.S.

The second ripple effect is that the data come to us at higher temporal resolution: by birth year, not census decade. In response, Bleakley (2007) hypothesizes that how much hookworm depressed adult earnings depended on the percentage of one’s childhood spent where it was endemic. If we take eradication to have occurred in 1910 and assume with Bleakley (2007) that childhood lasts 19 years, then babies born in or before 1891 would have reached adulthood before eradication, too soon to benefit. Babies born in endemic areas in 1892 would have been helped for one year (between their 18th and 19th birthdays); in 1893 for two; and so on. Those born in 1910 or later stood to reap the full 19 years of benefit. Bleakley (2007) therefore hypothesizes that the impact of eradication follows a sort of diagonal step shape with respect to birth year. The step starts rising in 1891 and stops in 1910. Bleakley depicted that contour with dashed lines in this figure:

As you can see, Bleakley (2007) fit this contour to data, to see how well it could explain historical patterns. These dots are derived much as in the earlier Bleakley (2007) figure. For example, the leftmost dot is for the year 1825, and has a vertical coordinate of about –2. That means that among people born in 1825, being native to a state whose hookworm infection rate circa 1910 was 10 percentage points (0.1) higher corresponded to having an Occupational Income Score 0.20 lower. That means $20/year less income throughout adulthood, in the dollars of 1950. The graph shows that this association was generally negative in the mid-19th century and generally positive after 1910: formerly, coming from a hookworm zone depressed lifetime earnings. And the graph suggests that the transition followed the step pattern expected if the cause was hookworm eradication.

Below is my best reproduction of that graph. As before, I have plotted both the dots and their 95% confidence intervals. I have avoided superimposing the step-like contour the way Bleakley (2007) does because I worry that it tricks the eye into believing that the contour fits the data better than it really does. But I have marked the years when the contour kinks, 1891 and 1910:

Here is the same graph when using the 100-times-bigger census data sets now available7In addition to adding data, this version mimics the rest of the Bleakley (2007) analysis in adding blacks and in fitting directly to census microdata rather than aggregates, in order to include controls for race, sex, and their interaction. jQuery("#footnote_plugin_tooltip_7").tooltip({ tip: "#footnote_plugin_tooltip_text_7", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });:

And here is the graph when I copy Bleakley (2010) in incorporating all the controls for cross-state differences in health and health policy, education policy, and other traits8As I noted, when looking at short-term impacts on education, Bleakley (2007) does not plot a graph while incorporating all controls. But now, when looking at long-term impacts on occupational standing, Bleakley (2007) does also include such a graph. See the bottom right of this figure. jQuery("#footnote_plugin_tooltip_8").tooltip({ tip: "#footnote_plugin_tooltip_text_8", tipClass: "footnote_tooltip", effect: "fade", fadeOutSpeed: 100, predelay: 400, position: "top right", relative: true, offset: [10, 10] });:

Does it look to you like the upward trends in these last two graphs accelerated around 1891 and decelerated around 1910, as predicted by the Bleakley (2007) theory? To me, I have to say, not much. The climbs look steady and longer-term.

Since “not much” is muddy, I moved once again to formalize my interpretation. In analogy with my earlier graphs for schooling, I fit lines to the data points in the 19 years between 1891 and 1910, as well as to the 19 years on either side. Then I checked whether any bending in 1891 and 1910 is statistically significant. The final two graphs fit lines to the dots in the previous two. The dots in these next graphs are the same as in the previous two. It doesn’t look that way because I erased the grey confidence bars in order to expand the vertical scales.

In the both graphs the trend looks quite straight over the three generations surrounding the eradication campaign. The p values, shown in the bottom-right of each plot, are high.

Conclusion

Reanalyzing the Bleakley (2007) study left me unconvinced that the children who benefited from hookworm eradication went to school more or earned more as adults. Conceivably, if I had access to the original data and code, the confrontation with the reconstructed versions would expose errors in the GiveWell version that would alter my view. But this seems unlikely. The new census data sets are much bigger than the old, which improves precision. And most of the differences probably do not arise from clear-cut errors on either side, but from minor differences in implementation, such as taking education spending from a different edition of an annual government report. If the conclusions swing on such modest and debatable discrepancies, then they are not robust and reliable.

Finally, even if the two versions of the data matched exactly, I might still disagree on interpretation, since I use tests, illustrated above, that focus more exclusively on whether the time trends contain the temporal fingerprint of hookworm eradication. For me, that fingerprint is characterized not merely by once-high-hookworm areas catching up with low-hookworm ones, but catch-up that accelerates and decelerates at times that fit the timing of the eradication campaign.

The data and code for this study are here (1.6 GB). The full write-up is here.

 

Notes   [ + ]

1. ↑ The 2016 Campbell review finds 52 short-term studies with follow-up duration under five years. Most last one to two years. 2. ↑ Versions of Bleakley (2007) appeared in the Quarterly Journal of Economics, a World Bank report, and the site of the National Center for Biotechnology Information. They are nearly the same. 3. ↑ The regressions for each census year control for the interactions of sex and race on the one hand and age on the other. They do not include the other Bleakley (2007) controls. Samples are restricted to eleven Southern states. The unit of observation is the State Economic Area, which is an aggregation of several counties. 4. ↑ The 1890 census records were destroyed in a fire. 1930 records had not been digitized at the time Bleakley did this work. 5. ↑ Consistent with this graph, while the Bleakley (2007) full-controls regressions continue to put a statistically significant coefficient on the treatment proxy, the reconstructions do not. This is one of the few cases where the original results are not recognizable in the reconstruction. See Table 6, panel B, of the new paper. 6. ↑ See the “Replicability adjustment for deworming” row of the “Parameters” tab of the cost-effectiveness analysis spreadsheet. 7. ↑ In addition to adding data, this version mimics the rest of the Bleakley (2007) analysis in adding blacks and in fitting directly to census microdata rather than aggregates, in order to include controls for race, sex, and their interaction. 8. ↑ As I noted, when looking at short-term impacts on education, Bleakley (2007) does not plot a graph while incorporating all controls. But now, when looking at long-term impacts on occupational standing, Bleakley (2007) does also include such a graph. See the bottom right of this figure. function footnote_expand_reference_container() { jQuery("#footnote_references_container").show(); jQuery("#footnote_reference_container_collapse_button").text("-"); } function footnote_collapse_reference_container() { jQuery("#footnote_references_container").hide(); jQuery("#footnote_reference_container_collapse_button").text("+"); } function footnote_expand_collapse_reference_container() { if (jQuery("#footnote_references_container").is(":hidden")) { footnote_expand_reference_container(); } else { footnote_collapse_reference_container(); } } function footnote_moveToAnchor(p_str_TargetID) { footnote_expand_reference_container(); var l_obj_Target = jQuery("#" + p_str_TargetID); if(l_obj_Target.length) { jQuery('html, body').animate({ scrollTop: l_obj_Target.offset().top - window.innerHeight/2 }, 1000); } }

The post Questioning the evidence on hookworm eradication in the American South appeared first on The GiveWell Blog.

If you’re thinking about where to give to charity this year and it would be helpful to speak with a member of GiveWell’s staff about your decision, please let us know. We’re happy to answer questions sent to info@givewell.org or to schedule a call via the form here.

We know we publish a lot of information, and we’re glad to provide a brief overview of our headline recommendations. We can also answer questions about our process for finding top charities, the strengths and weaknesses of our top charities, or how your personal values might point you toward one organization we recommend over another. Conversations like these also help us understand how people use our research and what questions they have.

Due to limited staff capacity, it’s possible we won’t be able to speak with everyone who requests a call, although based on past experience we hope to be able to connect with anyone who gets in touch.

We look forward to hearing from you!

The post Want to talk to someone at GiveWell about your giving decision? appeared first on The GiveWell Blog.

If you’re thinking about where to give to charity this year and it would be helpful to speak with a member of GiveWell’s staff about your decision, please let us know. We’re happy to answer questions sent to info@givewell.org or to schedule a call via the form here.

We know we publish a lot of information, and we’re glad to provide a brief overview of our headline recommendations. We can also answer questions about our process for finding top charities, the strengths and weaknesses of our top charities, or how your personal values might point you toward one organization we recommend over another. Conversations like these also help us understand how people use our research and what questions they have.

Due to limited staff capacity, it’s possible we won’t be able to speak with everyone who requests a call, although based on past experience we hope to be able to connect with anyone who gets in touch.

We look forward to hearing from you!

The post Want to talk to someone at GiveWell about your giving decision? appeared first on The GiveWell Blog.