Evaluation of Givewell Blogs

This post will provide a brief overview of GiveWell’s progress in a number of areas so far this year. In summary,

1. Research: We are making progress on reaching charities that might be a good fit for a GiveWell recommendation and asking them to apply. We are also moving forward with GiveWell’s intervention prioritization goals.
2. Operations: The separation of GiveWell and the Open Philanthropy Project was a major organizational priority in the first half of the year and was finalized on June 1. We’ve also increased the specialization on the operations team and outsourced some of GiveWell’s operations work.
3. Outreach: Outreach is now a major organizational priority. We hope to develop a strategy for significantly increasing money moved to our recommended charities by September.

We hope you will let us know if you have any questions about our work this year in the comments.

Research

Our work on research now falls into two primary categories: Traditional top charities work, consisting of research into promising programs and evaluations of charities implementing them as potential GiveWell top charities, and GiveWell Incubation Grants, our work to grow the pipeline of potential top charities and improve our understanding of our current recommended charities.

Top charities

• Encouraging charities to apply for a GiveWell recommendation.

  In recent years, we were surprised by how few charities reached out to GiveWell to apply for a recommendation. We guessed that some part of this may be driven by (a) a lack of understanding of GiveWell’s research priorities and which organizations we might be interested in recommending, and (b) a lack of understanding or misconceptions about GiveWell’s charity review process or the value added of a GiveWell recommendation in increasing a charity’s funding.

  We took two steps to address this problem in 2017. First, GiveWell Research Analyst Chelsea Tabart is now serving as GiveWell’s “charity liaison.” In this role, Chelsea connects with groups that may be a good fit for a GiveWell recommendation to learn more about their work and to encourage them to apply if a fit seems promising, and to explain GiveWell’s review process and value added. Second, we published a blog post on why we think more charities should consider applying for a GiveWell recommendation. We are now considering a number of charities as potential top-charity contenders and attribute this in part to taking the steps described above.

• Intervention prioritization.

  A major goal this year is to assess a large number of interventions as potential GiveWell priority programs. This “intervention prioritization” work involves surveying the literature for a variety of interventions to identify the most cost-effective and evidence-backed programs.

  We have made good progress on intervention research in 2017. (We plan to write about our progress in more detail in a future post.) We completed 50 quick evidence assessments in the first half of the year and published interim intervention reports on:

  • Sayana Press
  • Distribution of eyeglasses in developing countries
  • Antiretroviral therapy to treat HIV/AIDS
  • Interventions to promote handwashing
  • SMS reminders for vaccination.

  We also published an intervention report on surgery to repair obstetric fistula.

GiveWell Incubation Grants

We continue to expand our work on GiveWell’s Incubation Grants program:

• Partnership with IDinsight. We continued our partnership with IDinsight. IDinsight is working on a number of projects with GiveWell, including cost-effectiveness and monitoring assessments of surgery charities (cataract and fistula), running a randomized controlled trial of New Incentives’ pilot program to provide incentives for immunizations, and helping one of our top charities, the Against Malaria Foundation, improve its monitoring.
• Considering new top charities via GiveWell Incubation Grants. We’re considering some GiveWell Incubation Grant recipient charities as potential 2017 top charities. We’ve published blog posts on Zusha! and No Lean Season as contenders for a recommendation at the end of the year.

Recruiting

We hired Caitlin McGugan as a Senior Fellow and James Snowden started working with us as a research consultant. We hope they will increase GiveWell’s output of intervention reports. We also have one summer research analyst, Scott Weathers, working with us.

Operations

The separation of GiveWell and the Open Philanthropy Project was a major priority for GiveWell’s operations team in the first half of 2017. The separation was finalized on June 1.

The operations team continues to increase in specialization (historically, GiveWell operations work has been done by generalists on staff). We hired Maryana Pinchuk to serve as a Donations Manager and Erin Wolff as a Donations Relations Assistant; a search for a Controller to manage our finance and accounting is underway. We have also started to work with new vendors to outsource some operations work, which we hope will increase our available staff capacity and improve the quality of our operations.

Outreach

Outreach is now a major organizational priority for GiveWell. In the past, we focused very little on efforts to reach new potential donors with GiveWell’s work. Now, we think outreach is more of a limiting factor than research—the high-value funding gaps we’ve identified exceed the amount of donations we expect to direct to those gaps.

We have developed a list of ideas for how to significantly increase the money GiveWell directs to our recommended charities, and are planning to work on the most promising ideas over the next few months. For example, we think there may be relatively low-intensity steps to take in areas like podcast advertising; in February, we advertised on a small number of podcasts and plan to do so again based on the cost of running advertisements and the additional donations to top charities we tracked as a direct result of those ads.

We are also trying to hire another Research Analyst, Outreach Focus to expand our capacity to communicate with donors and other individuals who rely on GiveWell’s research.

The post Mid-year update on GiveWell’s progress appeared first on The GiveWell Blog.

This is the third of four posts that form our annual review and plan for the following year. This post reviews and evaluates GiveWell’s progress last year as an organization and sketches out some high level goals for the current year. The first two posts covered GiveWell’s progress and plans on research. The last post in the series will look at metrics on our influence on donations in 2016.

First, a point of clarification. GiveWell as a legal entity currently employs both (a) staff whose work is described on givewell.org (finding outstanding evidence-backed, cost-effective programs) and (b) staff who work on the Open Philanthropy Project. We expect Open Philanthropy to become a separate organization this year (more below), pending board approval. The scope of this post is limited to (a) – the parts of the organization that will not become part of Open Philanthropy. Open Philanthropy has written about its progress and plans in this post.

Below, we first note three high-level points about where GiveWell is as an organization today. We then reflect on four questions that are important for thinking about our performance as an organization:

• Do we have sufficient staff capacity?
• Does our impact justify our operating expenses?
• Does GiveWell have a positive and accurate public image?
• Are we in a stable financial position?

Major organizational developments

Separation of the Open Philanthropy Project

We had aimed to complete the transition of Open Philanthropy staff to a new entity by the end of 2016 and did not accomplish this goal, though we are now effectively operating as two separate teams. We now expect, pending board approval, to complete the legal split by mid-2017. After the split, there will continue to be some shared staff between the organizations (GiveWell staff will track the time they spend on work for Open Philanthropy and GiveWell will bill Open Philanthropy for the time). We will continue to share office space.

GiveWell as an entity currently employs 35 staff members. After the split, we anticipate that GiveWell will continue to employ 15-20 of the current employees and that Elie Hassenfeld will remain as Executive Director of GiveWell. Holden Karnofsky, Co-Founder of GiveWell, currently spends very little time on GiveWell and will work full time for Open Philanthropy.

Outreach is now more of a limiting factor than research

We’ve gone from feeling that we had more funding available than we had good giving opportunities to a situation where we believe that strong giving opportunities have surpassed available funding. We estimate that we left over $100 million worth of very strong opportunities (top charity execution level 1 or 2 gaps, excluding GiveDirectly) unfilled last year.

This is due to increased research output (we added three new top charities and two new standouts) in 2016, an expectation of increased research output in the future (from our standard process and Incubation Grants), and decreased expectations of funding from Good Ventures. In a change from the previous year, Open Philanthropy’s tentative guess is currently that the “last dollar” it will give (from the pool of currently available capital) has higher expected value than gifts to GiveWell’s top charities today, leading it to recommend that Good Ventures cap its giving to GiveWell’s top charities at $50 million in 2016.

We expect to put more emphasis on expanding our outreach to potential donors interested in following our recommendations in 2017 than we have in past years. We are at early stages of thinking through what that might involve.

Organizational maturity

GiveWell will be 10 years old this year and we feel that we’ve reached a relatively stable place in our development. We are now making a major effort to strengthen our organizational infrastructure through filling specialized roles, particularly in operations (finance, donations management, technology, etc.); formalizing policies and procedures; and creating contingency plans for replacing senior staff.

Four key questions

Below we pose and respond to four questions about how we are doing as an organization.

Do we have sufficient staff capacity?

Operations: To date we have not had sufficient capacity for operations and have been slower to make improvements to our systems than we would have liked. In the last year, we have begun to make major changes to GiveWell’s operations team to try to correct for this. Sarah Ward was named Director of Operations, a new role, and we are pursuing a strategy of (a) hiring specialized firms to handle more of the HR and IT work that generalist staff have done in the past; (b) replacing our external accountants and auditors with firms that specialize in non-profits; and (c) moving current staff into and hiring for specialized roles, such as a donations manager, donor relations assistant, controller, and office manager. Our number of generalist operations staff has decreased; we expect to continue to have a need for a small number of generalist staff to manage relationships with external firms and fill gaps between specialist domains.

Our current operations team includes a Director of Operations, two operations generalists (who work on the website, accounting, recruiting, personnel management, donation processing, and IT), an Office Manager, an Administrative Assistant, a Donations Manager, a Donations Assistant, and a Donor Relations Assistant. We are hiring for an Operations and Legal Program Manager and expect to hire for additional roles in the coming months. After the expected spinoff of Open Philanthropy into a separate organization, the office manager, administrative assistant and one of the operations generalists will divide their time between the two organizations and Sarah will manage operations for both organizations temporarily; Open Philanthropy will begin building a separate operations team this year.

Research: Seven staff work on GiveWell’s research full time or close to full time. Elie Hassenfeld, GiveWell’s Executive Director, spends about half his time on GiveWell research. Elie spends the other half of his time on a combination of the Open Philanthropy project (about 20% of his time currently) and overseeing outreach, recruiting, and operations for GiveWell.

Josh Rosenberg and I have taken over much of the research work that Elie and Holden, co-founders of GiveWell, used to do, including all updates on current top charities, reviewing top charity contenders, managing research staff, and some intervention assessments. Holden now spends almost no time on GiveWell research.

We feel that we have sufficient capacity to follow up with our current top charities, consider promising contenders for top charity recommendations, and make decisions about Incubation Grants. We do not yet have sufficient capacity for reviewing the evidence for and modeling cost-effectiveness of interventions. We aim to make at least one hire for this work in the next few months. More on this in our post about our research plans for the year.

Outreach: As noted above, we feel we’ve reached the point where we are identifying outstanding giving opportunities more quickly than we can expand our reach to donors to fill the opportunities. Throughout most of our history, we felt that the opposite was true, that the amount of funding we could influence surpassed the opportunities we had identified, so this represents a significant shift for us. We don’t yet have concrete plans for future outreach work, but expect to give outreach significantly more attention than we have in the past.

We currently have one staff member, Catherine Hollander, who works on outreach full-time. Our outreach priorities in 2016 were to speak or meet with all major donors who were interested in talking to us, take any opportunities that came up to discuss our work with the media, and continue posting regularly to our blog. We feel that we accomplished our goals for connecting with major donors and keeping up with media requests, and fell short on blogging.

Catherine is leading the search for a Research Analyst, Outreach Focus to do more of the types of outreach we’ve focused on in the past, namely connecting with more media and major donors, and increasing the frequency of blog posts.

Does our impact justify our operating expenses?

GiveWell’s impact on donations (or “money moved”) to our recommended charities likely decreased somewhat in 2016. We are in the process of gathering and analyzing data on our influence on donations, but expect it to be in the range of $80-90 million to recommended charities and $9.2 million for Incubation Grants. Money moved to top charities in 2015 was $110 million.

Good Ventures’ giving to top charities fell from about $70 million to $50 million, due to changes in the way it is allocating funding across priorities and to a large one-off grant to GiveDirectly in 2015. Based on GiveWell’s recommendations, Good Ventures also funded $9.2 million in Incubation Grants, up from about $400,000 to $500,000 in each of 2014 and 2015.

Over the same period, we spent approximately $2 million on our operations. In total, GiveWell as an entity spent about $5.5 million on operational expenses, of which $3.5 million was spent on the Open Philanthropy Project.

We previously wrote that we believe that expenses that are 15% of money moved are well within the range of normal, so we feel comfortable with the relative size of our operating expenses at this point.

Does GiveWell have a positive and accurate public image?

We believe that GiveWell’s public image is largely positive and reasonably accurate. This is true for all or nearly all of the major media coverage we have received. See, for example, coverage on NPR and in The Atlantic, Esquire and Vox.

There are two aspects of our public image that we would like to change. First, media has sometimes portrayed our top charities as having guaranteed impact and as being the “best” charities—for example, a 2015 article in The Atlantic said, “If what you want is to save lives with certainty, several people said, you have to go to GiveWell.” We believe that our top charities offer the highest expected value among evidence-backed opportunities that we have found to date, but are not risk-free and may not be the best giving opportunities for donors with different values or unique expertise, connections, or resources. Second, charities may have an inaccurate view of the costs and benefits of engaging with us—more in this post.

Our biggest public image project in the last year was launching a redesigned website. This project took much longer than expected. The original launch date was April 2015, but due to unexpected problems and lack of staff capacity, it didn’t go live until September 2016. Our previous website had an outdated look and confusing architecture. We think the new one is a large improvement, though we aim to make some further improvements in the future.

Are we in a stable financial position?

The short answer is yes.

In 2016, we raised about $3 million in revenue available for funding our operations that was not specifically for funding Open Philanthropy Project expenses (Open Philanthropy has, recently, been fully funded by Good Ventures). We have roughly projected GiveWell’s expenses (excluding pre-split Open Philanthropy expenses) at $2.7 million in 2017 and $3.2 million in 2018. Given our money moved to top charities and our experiences with fundraising in the past, it seems reasonable to expect that we will be able to raise this funding, though we expect to do a more detailed analysis of our financial situation once the details of the split with Open Philanthropy have been fully worked out.

We do not expect revenue available for operations to decrease as a result of splitting with Open Philanthropy because most major donors have told us that they support GiveWell due to our work identifying top charities. We think it is likely that Good Ventures will continue to support 20% of GiveWell’s operational budget, as it has for the last several years.

The post GiveWell as an organization: progress in 2016 and plans for 2017 appeared first on The GiveWell Blog.

This is the second of four posts that form our annual review and plan for the following year. The first post reviewed our progress in 2016. The following two posts will cover GiveWell’s progress and plans as an organization and metrics on our influence on donations in 2016.

Our primary research goals for 2017 are to:

• Speed up our output of new intervention assessments, by hiring a Senior Fellow and by improving our process for reviewing interventions at a shallow level.
• Increase the number of promising charities that apply for our recommendation. Alternatively, we may learn why we have relatively few strong applicants and decide whether to change our process as a result. Research Analyst Chelsea Tabart will spend most of her time on this project.
• Through GiveWell Incubation Grants, fund projects that may lead to more top charity contenders in the future and consider grantees No Lean Season and Zusha! as potential 2017 top charities.
• Further improve the robustness and usability of our cost-effectiveness model.
• Improve our process for following the progress of current top charities to reduce staff time, while maintaining quality. We also have some specific goals (discussed below) with respect to answering open questions about current top charities.

We discuss each of these goals in more depth below.

Intervention assessments

Intervention assessments are key to our research process. We generally only consider recommending funding for programs that are implementing one of our priority programs (an exception is if an organization has done rigorous evaluation of its own program, though in practice we have found this to be very rare). In recent years, we have completed few intervention reports, which has limited our ability to consider new potential top charities. We plan to increase the rate at which we form views on interventions this year by:

• Hiring a Senior Fellow (or possibly more than one). We expect a Senior Fellow to have a Ph.D. in economics, public health, or statistics or equivalent experience and to focus on in-depth evidence reviews and cost-effectiveness assessments of interventions that appear promising after a shallower investigation. In addition, Open Philanthropy Project Senior Advisor David Roodman may spend some more time on intervention related work.
• Doing low-intensity research on a large number of promising interventions. We generally start with a two to four hour “quick intervention assessment,” and then prioritize interventions for a 20-30 hour “interim intervention report” (example). We don’t yet have a good sense of how many of these of these we will complete this year, because we’re unsure both about how much capacity we will have for this work and about how many promising interventions there will be at each step in the process.
• Continuing to improve our systems for ensuring that we become aware of promising interventions and new relevant research as it becomes available. We expect to learn about additional interventions by tracking new research, particularly randomized controlled trials, in global health and development and by talking to select organizations about programs they run that they think we should look into.

Charity applications

In the past few years, we have been surprised by how little interest there has been from charities in applying for a GiveWell recommendation. Our impression is that for global health and development charities there are relatively few funders of our size: in 2015, we tracked $110 million given due to our research; we are in the process of compiling the data for 2016, but expect it to be in the range of $80-90 million to recommended charities. We would like to better understand whether we have failed to get the word out about the potential value we offer or communicate well about our process and charities’ likelihood of success, or, alternatively, whether charities are making well-informed decisions about their fit with our criteria. (More on why we think more charities should consider applying for a GiveWell recommendation in this post.)

This year, we have designated GiveWell Research Analyst Chelsea Tabart as charity liaison. Her role is to increase and improve our pipeline of top charity contenders by answering charities’ questions about our process and which program(s) they should apply with, encouraging promising organizations to apply, and, through these conversations, understanding what the barriers are to more charities applying.

We aim by the end of the year to have a stronger pipeline of charities applying, have confidence that we are not missing strong contenders, or understand how we should adjust our process in the future.

Incubation Grants

We made significant progress on Incubation Grants in 2016 and plan in 2017 to largely continue with ongoing engagements, while being open to new grantmaking opportunities that are brought to our attention.

Among early-to-mid stage grants, we plan to spend the most time on working with IDinsight and New Incentives (where our feedback is needed to move the projects forward), and a smaller amount of time on Results for Development and Charity Science: Health (where we are only following along with ongoing projects).

Another major priority will be following up on two later-stage grantees, No Lean Season and Zusha!, groups that are contenders for a top charity recommendation in 2017. For No Lean Season, a program run by Evidence Action, our main outstanding questions are whether the program will have room for more funding in 2018 and whether monitoring will be high quality as the program scales. We have similar questions about Zusha! and in addition are awaiting randomized controlled trial results that are expected later this year.

Cost-effectiveness model

We plan to continue making improvements to our cost-effectiveness model and the data it draws on (separate from adding new interventions to the model, which is part of the intervention report work discussed above). Projects we are currently prioritizing include:

• Making it more straightforward to see how personal values are incorporated into the model and what the implications of those values are.
• Revisiting the prevalence and intensity adjustment that we use to compare the average per-person impact of deworming in places that our top charities work to the locations where the studies that found long-term impact of deworming were conducted. More in this post.
• Improving the insecticide-treated nets model by revisiting how it incorporates effects on adult mortality and adjustments for regions with different malaria burdens and changes in malaria burden over time.

Current top charities

Our goal this year is to maintain the quality of top charity updates while decreasing the amount of staff time we spend and we ask top charities to spend on this work. Below, we detail our plans for following up with each charity.

Deworm the World Initiative, GiveDirectly, and Schistosomiasis Control Initiative (SCI)

We have now followed these groups for several years and do not have major outstanding questions about them. We plan to ask for updates on financial information, monitoring results, and room for more funding and have regular phone calls with them to learn about operational changes that might lead us to ask additional questions.

Against Malaria Foundation (AMF)

We have two major outstanding questions about AMF that we hope to make progress on this year:

1. Will AMF’s monitoring processes be high quality? We wrote about our concerns about AMF’s past monitoring last year and expect new information to be available this year.
2. Going forward, AMF aims to fund larger distributions and commit funding further ahead of when a distribution is scheduled to occur than it has, for the most part, done in the past. Will this increase the extent to which AMF funds displace funds from other sources, or will there continue to be evidence that AMF’s funds are largely adding to the total number of nets distributed? More on this question in our review of AMF.

To help us make progress on these questions, we and AMF have agreed to have monthly calls to discuss questions we have about the monitoring AMF is producing and what AMF is learning about distributions it is considering funding. We will likely also seek out calls with AMF’s partner organizations to discuss these questions.

In order to estimate AMF’s room for more funding, we will seek out information on the location and size of funding gaps for mass net distribution campaigns from AMF, the African Leaders Malaria Alliance, and possibly other funders of nets. As we have in the past, we will use this information in conjunction with conversations with AMF about non-funding bottlenecks to its ability to fill various gaps.

The END Fund – deworming program

Compared with the charities we have recommended for several years, we have more open questions about the END Fund. The main questions we plan to seek more information on this year are:

1. We have not yet seen monitoring on par with that from our other top charities from the END Fund. We expect results from coverage surveys from END Fund programs this year. Will these surveys be high quality and demonstrate that the END Fund is funding successful programs?
2. We have not yet tried to compare the cost-effectiveness of the END Fund to our other top charities in our cost-effectiveness model. We will be seeking additional information from the END Fund about cost per treatment and baseline infection rates.
3. Questions around room for more funding: the extent to which funding due to GiveWell’s recommendation increases the amount that the END Fund spends on deworming versus other programs, actual and projected revenue from other sources, and what deworming grantmaking opportunities the END Fund expects to have.

We visited the END Fund’s programs in Rwanda and Idjwi island, DRC in January 2017 and will publish notes and photos from our visit shortly.

Malaria Consortium – seasonal malaria chemoprevention program

As with the END Fund, we have more open questions about Malaria Consortium than we do for the charities we have recommended for several years. Our main priorities are:

1. Further research on the evidence of effectiveness, cost-effectiveness, and potential downsides of seasonal malaria chemoprevention (SMC) (due to time constraints we have not yet completed a full intervention report, though we felt sufficiently confident in the intervention to recommend Malaria Consortium).
2. Getting a better understanding of the methodology Malaria Consortium uses for estimating coverage rates.
3. Completing a more in-depth room for more funding analysis for the program for 2018 than we did for 2017.

Malaria Consortium expects to have several new studies of its SMC programs to share in April 2017 (details).

We may visit a Malaria Consortium seasonal malaria chemoprevention program in summer 2017.

Sightsavers – deworming program

As with the END Fund and Malaria Consortium, we have more open questions about Sightsavers than we do for the charities we have recommended for several years. We expect to make limited progress this year because the first deworming mass drug administration funded with GiveWell-influenced funds is not expected to take place until September at the earliest and monitoring results aren’t expected until early 2018. Because Sightsavers has done fairly little deworming in the past year, we don’t expect to be able to learn much from its ongoing programs. Our main priorities for the year are:

1. Getting more information from Sightsavers about baseline prevalence and intensity of worm infections in the areas it is working, to inform our cost-effectiveness analysis.
2. Using Sightsavers’ budget for the projects and planned treatment numbers to improve our estimate of the cost per treatment – another input into our cost-effectiveness analysis. Our current cost per treatment estimate is very rough.
3. Completing a room for more funding analysis for 2018.

Standout charities

Standout charities are groups that we have a large amount of information about and that meet some but not all of our criteria. Because we have not followed them closely over time, it is possible that they may now be a stronger fit (or that they no longer focus on the program we reviewed). We plan to have at least one phone call with each of these groups to discuss whether anything has changed that might lead us to reopen consideration of the organization as a potential top charity. Due to our focus on organizations that are most likely to become top charities, we don’t expect to make this work a priority beyond that.

The post GiveWell’s research plans for 2017 appeared first on The GiveWell Blog.

This is the first of four posts that form our annual review and plan for the following year. This post reviews and evaluates last year’s progress on our work of finding and recommending evidence-based, thoroughly vetted charities that serve the global poor. The following three posts will cover our plans for GiveWell’s research in 2017, GiveWell’s progress and plans as an organization and metrics on our influence on donations in 2016.

Summary

We feel that 2016 was a highly successful year for GiveWell’s research. We accomplished or made significant progress on all of our top priorities and accomplished some of the goals we didn’t know if we would have time for. Our research output was greater than in any past year. We added three new top charities and two new standout charities. We made important progress on building the pipeline of future GiveWell top charities through our work on GiveWell Incubation Grants. The research team’s staff capacity has continued to increase and we expect output to continue to grow.

More subjectively, we feel that the quality of our research has continued to improve. Of particular note are our improved understanding of room for more funding for insecticide-treated nets and of the evidence for deworming.

We compare our annual output for 2012-2016 in this spreadsheet.

Our progress in 2016 relative to our plans

In early 2016, we laid out our goals for research in 2016. Below we discuss each goal in two of the categories we set out last year, “top priorities” and “other research we will undertake if we have the time to do so,” and what progress we made on each.

“Top priorities”

Supporting the development of potential future GiveWell top charities: making grants to organizations that could become top charity contenders in the future or supporting research that could lead to more organizations that are a strong fit with our criteria. […]

We investigated and recommended that Good Ventures make grants to five early stage projects: No Lean Season (migration assistance), New Incentives (conditional cash transfers), Zusha! (road safety), Charity Science: Health (immunizations), and Results for Development (childhood pneumonia treatment). We have published grant descriptions for most of these grants; three are forthcoming. We have also begun working closely with IDinsight to partner with charities that work on priority programs to strengthen their monitoring and evaluation to increase the chances that the charities meet our criteria in the future.

Considering additional funding for insecticide-treated nets [beyond the Against Malaria Foundation] […]

We spent relatively little time on this priority because (a) the Against Malaria Foundation (AMF) succeeded in signing agreements for several major distributions early in the year, increasing our estimate of its ability to absorb additional funds; and (b) our initial conversations with large funders of nets seemed unlikely to result in a top charity recommendation. At the end of the year, we found that the Against Malaria Foundation was able to absorb considerably more funding than we directed to it. This was in part due to Good Ventures’s shift away from continued growth in funding for GiveWell top charities. Separately, we also gained a stronger understanding of the size and nature of funding gaps for nets globally through country case studies and conversations to understand the global funding landscape.

Intervention prioritization: quick investigations on a large number of interventions with the goal of finding more priority programs. […]

This was a major priority for us and we made some progress, but not as much as we wanted to. The main things we did were (a) quick reviews (3-10 hours spent reviewing the evidence base) for ~30 programs, which ultimately led us to prioritize seasonal malaria chemoprevention (SMC) and recommend Malaria Consortium’s work on SMC; (b) publishing three “interim intervention reports” (on SMC, integrated community case management, and severe acute malnutrition) which have provided a template for mid-level assessments and enabled more staff to produce such reports in 2017 (one of which, on Sayana Press, has been published); and (c) writing a full intervention assessment of voluntary male medical circumcision, which is now one of our priority programs.

Current top charities: continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. […]

We answered our most important questions about the top charities we recommended in 2015. In particular, we had stronger answers at the end of the year on AMF’s progress at signing agreements, the quality of AMF’s monitoring, Schistosomiasis Control Initiative’s past spending and financial position, and had an overall much stronger understanding of Deworm the World Initiative, particularly its work in Kenya.

However, we feel that we spent too much time on this work in 2016. Three staff spent the majority of their time on top charity updates and, while following the progress and plans of our top charities is a crucial piece of GiveWell’s work, the value we got from this work felt out of proportion with the time spent. In 2017, we plan to have a single staff member do most of this work and expect it to take a half to two-thirds of a full-time job. Three other staff will spend a small portion of their time, totaling approximately the equivalent of one full-time job, on this work.

New evidence on deworming and bednets. The next round of follow up on a key deworming study is expected to be available later this year and could make a big difference to our view of deworming. We’re also looking more into the degree to which insecticide resistance may be reducing the impact of bednets. […]

We completed an evidence review on the impact of insecticide resistance on malaria control. David Roodman, Senior Advisor at the Open Philanthropy Project spent several months revisiting the evidence for deworming and summarizing his findings in two blog posts.

We wrote last year that we expected to see a new round of follow-up data on a key deworming study that could significantly affect our view of deworming. We have seen preliminary results and hope to get more complete results later this year and write about them at that time. We also decided to recommend a ~$1 million grant to support a more intensive 20-year follow-up to the Worms at Work study (writeup forthcoming).

“Other research we will undertake if we have the time to do so”

Micronutrient fortification charities. […]

We completed interim reviews of Food Fortification Initiative and Project Healthy Children and added them to our list of standout charities.

Neglected tropical disease (NTD) charities [and…] other organizations – if organizations apply for a recommendation and seem sufficiently promising, we will aim to review them.

Perhaps the most important development in 2016 came out of two secondary goals for 2016: continuing investigations of deworming programs (which led to recommending Sightsavers and the END Fund for their work on deworming) and continuing to be open to applications from charities (which led to recommending Malaria Consortium for its work on seasonal malaria chemoprevention).

At the same time, in the past few years, we have been surprised by how little interest there has been from charities in applying for a GiveWell recommendation. We have come to believe that charities may have misconceptions about our process or lack the information about whether they would be a fit for our criteria, and that this could be improved by us reaching out to more promising organizations and taking the time to understand the reasons why they have not applied in the past. We discuss what we are doing this year on charity outreach in the next post in this series.

Surgery charities. We have had several conversations with organizations that work on cataract surgery and we may reach out to organizations that work on obstetric fistula surgery. […]

In addition to the conversations with organizations that work on on cataract, we also spoke to several groups that work on obstetric fistula. We have not identified a group we would like to invite to apply. This work is now moving ahead primarily through our engagement with IDinsight to work with charities to strengthen their monitoring and evaluation.

Publishing research we largely completed in 2015: updates on standout charities (GAIN, IGN, and Living Goods), interim reviews of charities we began investigating in 2015 (Sightsavers, END Fund, and Project Healthy Children), and intervention reports (folic acid fortification, surgery for cataracts, trachoma and fistula, measles immunization campaigns, mass drug administration for lymphatic filariasis, and “Targeting the Ultra Poor”).

We made some progress on this goal. We published updates on standout charities (GAIN, Iodine Global Network, and Living Goods), reviews of charities we began investigating in 2015 (Sightsavers, the END Fund, and Project Healthy Children), and one of the intervention reports we hoped to publish in 2016 (cataract surgery). We did not publish the other intervention reports we hoped to (folic acid fortification, surgery for trachoma and fistula, measles immunization campaigns, mass drug administration for lymphatic filariasis, and “Targeting the Ultra Poor”).

Other work

Compared to previous years, we made a lot of progress on improving the usability of and getting staff engagement with our cost-effectiveness model. We designated a staff member, currently Chris Smith, to work on this close to full-time. Chris re-formatted the file to make it easier for staff and others to input uncertain and subjective values, integrated the seasonal malaria chemoprevention analysis into the model, and restructured the bed nets model to take into account country-level variation in malaria rates.

The post GiveWell’s progress on research in 2016 appeared first on The GiveWell Blog.

In addition to evaluations of other charities, GiveWell publishes substantial evaluation of ourselves, from progress against our goals to our impact on donations. We generally publish quarterly updates regarding two key metrics: (a) donations to top charities and (b) web traffic (though going forward, we may provide less frequent updates).

The tables and chart below present basic information about our growth in money moved and web traffic in the first quarter of 2016 compared to the previous two years (note 1).

Money moved and donors: first quarter

Money moved by donors who have never given more than $5,000 in a year increased about 50% to $1.1 million. The total number of donors in the first quarter increased about 30% to about 4,500 (note 2).

Most of our money moved is donated near the end of the year (we tracked 70% or more of our total money moved in the fourth quarter each of the last three years) and is driven by a relatively small number of large donors. Because of this, we do not think we can reliably predict our growth and think that our year-to-date total money moved provides relatively limited information about what our year-end money moved is likely to be (note 3). We therefore look at the data above as an indication of growth in our audience.

Web traffic through April 2016

Growth in web traffic excluding Google AdWords increased 10% in the first quarter. GiveWell’s website receives elevated web traffic during “giving season” around December of each year. To adjust for this and emphasize the trend, the chart below shows the rolling sum of unique visitors over the previous twelve months, starting in December 2009 (the first period for which we have 12 months of reliable data due to an issue tracking visits in 2008).

We use web analytics data from two sources: Clicky and Google Analytics (except for those months for which we only have reliable data from one source). The raw data we used to generate the chart and table above (as well as notes on the issues we’ve had and adjustments we’ve made) is in this spreadsheet. (Note on how we count unique visitors.)

Note 1: Since our 2012 annual metrics report we have shifted to a reporting year that starts on February 1, rather than January 1, in order to better capture year-on-year growth in the peak giving months of December and January. Therefore, metrics for the “first quarter” reported here are for February through April.

Note 2: Our measure of the total number of donors may overestimate the true number. We identify individual donors based on the reported name and email. Donors may donate directly to our recommended charities and not opt to share their contact information with us, or donors may use different information for subsequent donations (for example, a different email), in which case, we may mistakenly count a donation from a past donor as if it was made by a new donor. We are unsure but would guess that the impact of this issue is relatively small and that the data shown are generally reflective of our growth from year to year.

Note 3: In total, GiveWell donors directed $2.6 million to our top charities in the first quarter of 2016, compared to $2.0 million that we had tracked in the first quarter of 2015. For the reason described above, we don’t find this number to be particularly meaningful at this time of year.

Note 4: We count unique visitors over a period as the sum of monthly unique visitors. In other words, if the same person visits the site multiple times in a calendar month, they are counted once. If they visit in multiple months, they are counted once per month.

The post Update on GiveWell’s web traffic / money moved: Q1 2016 appeared first on The GiveWell Blog.

GiveWell is dedicated to finding outstanding giving opportunities and publishing the full details of our analysis. In addition to evaluations of other charities, GiveWell publishes substantial evaluation of our own work. This post lays out highlights from our 2015 metrics report, which reviews what we know about how our research impacted donors. Please note:

• We report on “metrics years” that run from February through January; for example, our 2015 data cover February 1, 2015 through January 31, 2016.
• We differentiate between our traditional charity recommendations, our work on the Open Philanthropy Project, and other charitable giving.
• More context on the relationship between Good Ventures and GiveWell can be found here.

Summary of influence: In 2015, GiveWell influenced charitable giving in several ways. The following table summarizes our understanding of this influence.

Total money moved: In 2015, GiveWell tracked $110.1 million in money moved to our recommended charities. Our money moved only includes donations that we are confident were influenced by our recommendations.

Open Philanthropy Project: As part of our work on the Open Philanthropy Project, we advised Good Ventures to make grants totaling $6.4 million. This was in addition to Good Ventures’ support for our recommended charities.

Money moved by charity: Our four top charities received the majority of our money moved. Our four standout charities received a total of $2.2 million.

Money moved by size of donor: In 2015, the number of donors and amount donated increased across each donor size category. In 2015, 95% of our money moved (excluding Good Ventures) came from about 15% of our donors, each of whom gave $1,000 or more.

Donor retention: The total number of donors who gave to our recommended charities or to GiveWell unrestricted increased about 60% year-over-year to 15,274 in 2015. This included 10,669 donors who gave to our recommended charities for the first time. Among all donors who gave in the previous year, about 40% gave again in 2015, up from about 33% who gave again in 2014.

Our retention was stronger among donors who gave larger amounts or who first gave to our recommendations prior to 2013. Of larger donors (those who gave $10,000 or more in either of the last two years), about 80% who gave in 2014 gave again in 2015.

GiveWell’s expenses: GiveWell’s total operating expenses in 2015 were $3.4 million. Our expenses increased from about $1.8 million in 2014 as the size of our staff grew, largely to support the Open Philanthropy Project. We estimate that about one-third of our total expenses ($1.1 million) supported our traditional top charity work and about two-thirds supported the Open Philanthropy Project. In 2014, we estimated that expenses for our traditional charity work were about $900,000.

Donations supporting GiveWell’s operations: Prior to 2013, GiveWell relied on a small number of donors to provide unrestricted support for our operations. Since 2013, we have asked more donors to support our operational costs and asked donors to support us at a higher level than we had in previous years. In 2015, we raised $4.9 million, up from $3.0 million in 2014. Several institutions and the six largest individual donors contributed about two-thirds of GiveWell’s funding in 2015.

Donor demographics: We continued to collect information on our donors. We found the picture of our 2015 donors to be broadly consistent with previous information. Based on reports from donors who gave $2,000 or more, we found:

• The most common ways that donors found us were via Peter Singer and personal referrals.
• About 70% of our donors are under 40, and about 60% work in technology or finance.

Web traffic: Unique visitors to our website increased by 12% in 2015 compared to 2014 (when excluding visitors driven by AdWords, Google’s online advertising product).

For more detail, see our full metrics report (PDF).

The post GiveWell’s money moved and web traffic in 2015 appeared first on The GiveWell Blog.

Over the past couple of years, we’ve put a lot of effort into hiring and training staff and we now have significantly more capacity to do research than we have in past years. Some of our increased capacity will support the Open Philanthropy Project, which we hope will be a separate organization by the end of 2016; its plans for the year will be discussed on the new Open Philanthropy Project blog. We also expect to have more capacity for GiveWell’s work of finding outstanding evidence-based charities.

At the same time, we have come to believe that the kind of work we’ve recently been doing to find top charities – deeply investigating the most promising-seeming charities we know of, based largely on which interventions they carry out – has limited promise. In past years – and at the beginning of this year – we hoped that these investigations would lead relatively quickly to new top charities. Now, we believe that we’ve already (previously) identified most of the strongest charities by our criteria, and there aren’t many strong candidates left (though there are a few that we continue to investigate, and we remain willing and eager to investigate further promising groups if we come across them). With that in mind, we have begun seeing more potential in other research priorities, such as supporting the development of new organizations and new evidence bases.

A future post will elaborate on why we’ve formed this view. This post focuses on laying out our plans for GiveWell’s research work in 2016, taking this view into account.

In brief, in 2016:

• We plan to focus much of our capacity on a small number of initiatives that are unlikely to result in new top charities in 2016, but which we hope will lead to new top charities that are competitive with our current top charities in 2017 or 2018.
• We plan to intensify our work following our current top charities and are tentatively planning to make site visits to distributions funded by the Against Malaria Foundation and work supported by Evidence Action’s Deworm the World Initiative.
• We are also planning a substantial project focused on the question of whether or not we should recommend that Good Ventures give significantly more than it has in the past to support insecticide-treated nets, arguably the most promising area we know of for substantial additional funding.
• We also hope to take on additional work (described in detail below) but plan to prioritize this work below the items listed above.
• We plan to put more staff time into donor outreach than we have in the past and discuss our priorities for that work below.

This plan represents a significant shift from previous years, when our primary goal was improving the list of top charities we published at the end of each year. We plan to write more about the reasoning behind this shift in a future post.

What we’ve done so far this year

In January and February 2016 we:

• Put significantly more effort into getting input on our plans from non-senior staff than we have in past years. To start, we asked staff and a small number of GiveWell followers to make probabilistic predictions about which charities would become top charities if we investigated them this year. The results of this exercise led to this initial list of possible priorities (listed in no particular order). This list represents the organizations that we would guess are most likely to become top charities at the end of 2016. When we later revised our plan, we held a series of staff meetings to discuss details of the plan and what the new plan might be missing.
• Had exploratory conversations with several charities on that list and others in the field of global health and development. The goal of conversations with charities was to explain our application process and ask them basic questions about their programs, monitoring and evaluation, and need for additional funding. The goal of conversations with others working in global health and development was to generate a list of additional organizations to contact.

Ultimately, this work made us more pessimistic that prioritizing work on all the organizations listed above would lead to new top charities by the end of 2016, as discussed in the introduction of this post, and we refined our plans for the year as a result. A future post will elaborate on this development.

Top priorities for research

Our top priorities are:

• Supporting the development of potential future GiveWell top charities: making grants to organizations that could become top charity contenders in the future or supporting research that could lead to more organizations that are a strong fit with our criteria. This work is unlikely to result in new top charities in 2016, but we hope it will lead to new top charities that are competitive with our current top charities in 2017 or 2018. This work might include:
  • Providing early stage funding to organizations that aim to scale up programs with strong evidence of effectiveness and cost-effectiveness. (For example, New Incentives or Evidence Action’s No Lean Season program.)
  • Funding research on programs that are candidates to become priority programs. (For example, this grant to support research on an incentives for immunization program.)
  • Funding organizations that run priority programs to increase or improve their monitoring, or funding a third party to do this monitoring.
• Considering additional funding for insecticide-treated nets: A significant funding gap exists for insecticide-treated nets, and this gap appears to be as cost-effective an opportunity as any other we have found. This project involves determining whether there are high quality opportunities to provide significantly more funding for insecticide-treated nets than we have in the past. It will involve conversations with the major bednet funders (e.g., Global Fund to fight AIDS, Tuberculosis, and Malaria and the President’s Malaria Initiative) and others familiar with how to identify funding gaps for bednets and what the options are for monitoring distributions. We have also been discussing with the Against Malaria Foundation (AMF) what it would take to quickly scale up AMF’s work. The goal of this work is to identify additional funding opportunities for funding insecticide-treated nets in 2016.
• Intervention prioritization: quick investigations on a large number of interventions with the goal of finding more priority programs. We have looked at many interventions historically, but regularly learn of programs that we do not know very much about.
• Current top charities: continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. More on this below.
• New evidence on deworming and bednets. The next round of follow up on a key deworming study is expected to be available later this year and could make a big difference to our view of deworming. We’re also looking more into the degree to which insecticide resistance may be reducing the impact of bednets.

Other research we will undertake if we have the time to do so

• Micronutrient fortification charities. Last year, we tried but were unable to find compelling evidence that the Iodine Global Network (IGN) or the Global Alliance for Improved Nutrition (GAIN) had successfully contributed to the impact of salt iodization programs (write-ups forthcoming). We also began investigating Project Healthy Children. We may continue some of these investigations this year and have also reached out to the Micronutrient Initiative and the Food Fortification Initiative.
• Neglected tropical disease (NTD) charities. We began investigating Sightsavers and END Fund’s work on deworming last year and may continue with those organizations this year and expand the investigations to cover multiple NTDs. We have reached out to the Carter Center and Helen Keller International (HKI) about their NTD programs. HKI declined to participate at this time.
• Surgery charities. We have had several conversations with organizations that work on cataract surgery and we may reach out to organizations that work on obstetric fistula surgery. Our initial impression from these conversations is that it will be very challenging to understand the impact that these charities’ programs have. We may also consider other surgical interventions (such as trachoma).
• Other organizations. If organizations apply for a recommendation and seem sufficiently promising, we will aim to review them.
• Publishing research we largely completed in 2015: updates on standout charities (GAIN, IGN, and Living Goods), interim reviews of charities we began investigating in 2015 (Sightsavers, END Fund, and Project Healthy Children), and intervention reports (folic acid fortification, surgery for cataracts, trachoma and fistula, measles immunization campaigns, mass drug administration for lymphatic filariasis, and “Targeting the Ultra Poor”).

Research we considered but do not expect to undertake

The following investigations are ones that we considered doing this year but don’t currently expect to get to. This could change if some of the higher priority work turns out to be less promising than expected.

• Mega-charities. We could try to work with one or more large organizations with very diverse programs to figure out how to scale-up work on one of our priority programs.
• Charities that work on programs that are probably more cost-effective than cash transfers but not by a large enough margin that it seems worth highly prioritizing work on them.
  • Voluntary medical male circumcision. We are interested in talking to PSI, the only major organization we know of working on this program, but do not plan to prioritize this program beyond that.
  • “Targeting the ultra poor” or “graduation” programs.
  • Lymphatic filariasis.
  • Incentives for immunization. We previously funded research on this program and have been working with IDinsight on a cost-effectiveness analysis.
• Immunization programs. We have put in a fair amount of work into looking for room for more funding for scaling up immunization programs and have largely failed to find opportunities (2012 write-up; recent example).

More detail on potential further research on current recommended charities

One of our top priorities for 2016 is continuing to follow our current top charities and trying to answer our highest priority unanswered questions about these groups. We moved over $100 million to these groups in 2015 and whether we recommend a similar (or greater) level of support in 2016, and how we recommend allocating funds among them, depends on answering: (1) what is our best estimate of the organization’s impact and cost-effectiveness? and (2) how much room for more funding do they have?

Top charities

In past years, we’ve updated our top charity reviews once a year, in November. This year, we plan to refresh these reviews twice, in June and November. As we have at the end of the year, we expect to reconsider what recommendation we make to donors about how to allocate donations amongst our top charities in June.

Summary of our research plans for each of our top charities (note that the strategy documents were written in February):

• Against Malaria Foundation (AMF). We plan to follow AMF’s progress closely in 2016. Key questions include (a) how quickly is AMF committing funding to new distributions, and (b) can we get a more detailed understanding of how data is collected in pre- and post-distribution surveys. More details here.
• Schistosomiasis Control Initiative (SCI). The amount of time we spend on SCI this year depends on whether we see a significant improvement in the quality of SCI’s financial information (how it has spent funds, how much funding it holds, and projected expenses). If it does not improve, we will likely deprioritize much additional work on SCI. If it does, we would be interested in exploring the research questions detailed here.
• Deworm the World Initiative. Of our top charities, we feel that there is the largest gap between what we could know and what we do know for Deworm the World. In particular, we’ve focused on Deworm the World’s work in India, because in the past most unrestricted funds were used in India. Going forward, unrestricted funds will largely be used in new programs. We aim to follow Deworm the World’s progress in new countries closely and to better understand its past work by learning more about its program in Kenya. Details here.
• GiveDirectly. Our main goals from following GiveDirectly are to see if the quality of monitoring remains high, it is able to enroll new recipients quickly, and we can learn more about the impact of its work with partners to make cash a baseline against which other development programs are judged. Details here.

Standout charities

• Development Media International. We’re not planning to consider DMI as a possible top charity in 2016. The results from a randomized controlled trial (RCT) of its program that DMI shared last year were not in line with what we would have wanted to see for DMI to become a top charity. More recently, DMI shared some additional results from the RCT (which are not yet public). We believe that taken together these results provide conflicting evidence for DMI’s impact. DMI stands out for its commitment to transparency and rigorous evaluation and we will consider working with DMI to continue to build the evidence base around behavior change through mass media. We see this as a long-term project that is unlikely to result in DMI’s being a top charity in 2016.
• Iodine Global Network. We are planning to follow up with IGN about a few case studies that IGN thought might provide additional evidence of its impact.
• The Global Alliance for Improved Nutrition (GAIN) – Universal Salt Iodization (USI) program. We’re not planning to consider GAIN’s USI program as a possible top charity in 2016. We have not been able to establish clear evidence of GAIN successfully contributing to the impact of iodization programs, and think it is unlikely that more work on this will be useful.
• Living Goods. It’s fairly unlikely that we will consider Living Goods as a possible top charity in 2016. We would revisit this if we were to see significant improvements in the rigor of Living Goods’ monitoring or if we significantly changed our cost-effectiveness estimate for its work.

Plans for donor outreach

We have not historically prioritized outreach at GiveWell, instead choosing to devote staff capacity primarily to our research work. Now, with the addition of new research staff as well as the continued growth of GiveWell’s donor base, we feel it is appropriate to dedicate more capacity to outreach for GiveWell in service of our mission to make our research available to help individuals decide where to give.

In 2016, we plan to have 1.5 staff members devoted to outreach related to GiveWell and the Open Philanthropy Project. Due to this being early on in our outreach work, we’re tentatively planning to reassess our priorities every month for the first half of the year, and then every quarter. As of the publication of this blog post, we expect the following to be top priorities for GiveWell outreach in 2016:

• Donor calls and meetings. We expect that connecting with individuals who have donated to GiveWell will be an important part of our outreach going forward, although as we’re relatively new to prioritizing this, we plan to survey donors about whether this is something that they find useful. We’re hoping to learn more about the donors who use our work and any questions or feedback they have, as well as to offer an opportunity for donors to stay up to date on GiveWell’s work. More here.
• Launching a redesigned website. The redesign will largely improve the look and feel of the site with some minor improvements in navigation and content organization.
• Improving GiveWell’s written communications. This includes:
  • Revisiting and refreshing content on our website (e.g., a recent update to our criteria page) to ensure it’s up-to-date and clearly presented, particularly for individuals who aren’t familiar with our research.
  • Publishing content to our blog, in the hopes of highlighting research and providing additional insight into our values, process, and findings. We will need to put more effort into writing blog posts in order to maintain our previous pace of about one blog post per week, since many types of blog posts that previously appeared here will now be appearing on the Open Philanthropy Blog.

The post GiveWell research plans for 2016 appeared first on The GiveWell Blog.

This post reviews and evaluates last year’s progress on our traditional work of finding and recommending evidence-based, thoroughly vetted charities that serve the global poor. It has two parts. First, we look back at the plans we laid out in early 2015 and compare our progress against them, providing details on some of the most significant accomplishments and shortcomings of the year. Then, we reflect on the overall impact of our traditional work and critically evaluate some of our major strategic decisions. In our next post, we will cover our plans for GiveWell’s work in 2016.

SummaryIn brief, when evaluating ourselves against the goals we laid out in early 2015, we feel that we broadly achieved our primary goals for the year while we generally fell short on several of our secondary goals.

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015, as we tracked more than $100 million that was donated to our recommended charities as a direct result of our research.

This self-evaluation post focuses primarily on how we have grown as an organization and the recent strategic decisions we have made since these are most relevant to thinking about the impact of recent GiveWell-focused work.

Our progress in 2015 relative to our plansIn our “2015 plan” blog post, we wrote:

This year, our primary goals are to:

• Build management and research capacity for GiveWell’s traditional work while further reducing senior staff time (note 1) spent on this work, primarily by reallocating Elie Hassenfeld’s management responsibilities related to GiveWell’s traditional work.
• Maintain our core research product by completing updates on all eight 2014 recommended charities and determining which of them should be recommended as top charities for the 2015 giving season.

Our secondary goals for 2015 are to:

• Continue to seek outstanding giving opportunities by reviewing 2-4 new charities and publishing 2-4 new intervention reports.
• Improve the cost-effectiveness analyses and room for more funding analyses in charity reviews.
• Finish and launch a redesigned GiveWell website.
• Make further progress on experimental work to “seed” potential recommended charities.We expect our total output on “top charities” work to be roughly comparable to last year’s, despite a growing staff, because (a) a major focus of the coming year is training, and we expect to trade some short-term efficiency for long-run output; (b) we may be reallocating some capacity from our “top charities” work to the Open Philanthropy Project this year.

We feel that we broadly achieved our primary goals for the year, while we fell short on several of our secondary goals.

Goals that we feel we accomplished include:

• Building management and research capacity for GiveWell’s traditional work while further reducing the time that Elie spent on this work. Specifically:
  • All GiveWell staff track how they spend time at work. In 2014, Elie spent 36% of his time on GiveWell’s traditional work, and in 2015, Elie spent 26% of his time on this work.
  • In 2014, Elie was primarily responsible for managing all research staff. In 2015, Natalie Crispin and Josh Rosenberg took on research management responsibilities, and at the end of the year, managed 10 research staff between them. Timothy Telleen-Lawton and Eliza Scheffler also took on management responsibilities, primarily for operations-focused staff.
  • Most of the other staff working on GiveWell’s traditional work in 2015 were relatively new. Because they were new, most of the time they spent was focused on training. We discuss staff time allocations in greater detail below.
• Maintaining our core research product. We completed and published updates on all four top charities from 2014. We also conducted deeper investigations of 3 out of 4 of our standout charities (Global Alliance for Improved Nutrition (GAIN), Iodine Global Network (IGN), and Development Media International (DMI)), as planned, in an effort to determine whether they should be recommended as top charities for the 2015 giving season. We ultimately did not recommend any of these charities as top charities. We did not publish our updates on GAIN and IGN, and those are still forthcoming.
• Improving our room for more funding analyses in charity reviews, which we wrote about extensively in our November 2015 blog post announcing our updated recommendations.
• Continuing to make limited but steady progress on experimental work to “seed” potential recommended charities via grants to Evidence Action’s No Lean Season Program, New Incentives, and two randomized controlled trials focused on incentives for immunization.

Areas in which we fell short include:

• We did not publish updates (or even reach tentative conclusions internally) about any new potential recommended charities, and we published only 2 new intervention reports. We engaged with Project Healthy Children, UNICEF, The END Fund, Episcopal Relief & Development’s NetsforLife® Program, and Sightsavers. (We published a page on Children Without Worms, but it was extremely short.) We published 2 new intervention reports — on vitamin A supplementation and tetanus immunization campaigns — but both of these largely relied on work we had completed in 2014.
• As discussed in our November 2015 blog post announcing our updated recommendations, we felt more confident in our cost-effectiveness analyses (CEAs) at the end of 2015 than we had in previous years, and we made some of the improvements we planned. Nevertheless, we don’t believe that we made as much progress as we had hoped for when we set the goal to improve our CEAs in early 2015. In particular, key inputs and judgment calls in our CEA were decided on by different staff members at different points in time, which made it more difficult than ideal for individual staff members to understand all details of the CEA. We hope to resolve this issue by making at least one staff member responsible for fully understanding and explaining all aspects of our core CEAs in the future.
• We did not finish and launch a redesigned GiveWell website. The GiveWell website is large and it took longer than the firm we worked with expected to complete a version that was working properly. The firm we contracted with also built the Open Philanthropy Project website, and in September 2015, we explicitly prioritized completing the Open Philanthropy Project (“Open Phil”) site before the GiveWell site.

Overall, we succeeded in passing significant responsibilities from Elie to others, but we saw somewhat less overall research output than we had hoped for. Part of this was because, in the process of transferring responsibilities, we made some mistaken decisions – i.e., in some cases staff put substantial work into assignments that we ultimately determined were not a fit for them.

Other self-evaluation questionsWhat was GiveWell’s overall impact in 2015?

The overall impact of GiveWell’s recommendations continued to increase substantially in 2015. We tracked more than $100 million that was donated to our recommended charities as a direct result of our research. Excluding Good Ventures’ giving, we moved more than $30 million to our recommended charities. (More details on our 2015 money moved will be in our forthcoming 2015 metrics blog post.) This was a major increase relative to 2014, when we tracked about $27.8 million in money moved to our recommended charities, with about $13 million coming from non-Good Ventures donors.

Does our impact justify our staff size, in absolute terms?

As mentioned above, in 2015 we moved more than $100 million to our recommended charities. Over the same period, we spent approximately $3.8 million on our operations, of which $1.3 million was spent on GiveWell’s traditional work and $2.5 million on the Open Philanthropy Project.

We previously wrote that we believe that expenses that are 15% of money moved are well within the range of normal, so we feel comfortable with the relative size of our operating expenses at this point.

Is recent growth in staff justified? Has growth in output matched growth in staff?

Staff size has grown significantly over the past few years: from 11 people at the end of 2013 to 32 at the end of 2015. We think it is reasonable to question whether this growth has been justified and productive, in light of the fact that:

• We haven’t significantly increased the number of new major reports (charities or interventions reviewed in depth). We estimate that we published 4 major new reports in 2013, 5 in 2014, and 6 in 2015 (details on our rough estimate in this Google sheet including year-to-date information for 2016).
• Most of our 2015 impact (in terms of money moved) likely could have been achieved with reduced research work in 2015. For example, if we had only stayed up-to-date on our past top charities, we would have been in a position to have the same top charities list as what we ultimately published at the end of 2015. The bulk of additional research work that we did in 2015 did not seem to have significant direct impact, largely because the new giving opportunities that we investigated did not lead to new top charities, though it would have been difficult to confidently predict that this would happen in advance (we reflect on whether we chose the best research priorities in 2015 below).

On the other hand:

• Since 2013, we have greatly expanded the Open Philanthropy Project, which is currently formally housed at GiveWell though we hope to separate the organizations in 2016. Much of our increase in staff size has been for the Open Philanthropy Project.
• Many of our most senior staff have switched over to working primarily on the Open Philanthropy Project, and senior staff are particularly challenging to replace. We don’t think we could have maintained the same output if we had simply hired one new person for each senior staff member who switched over.
• Capacity building – hiring, training and evaluating new staff – is itself a time-consuming and long-term project. In addition, increased staff size creates the need for more work in and of itself. For example: (1) in 2015, we outgrew our office and had to find new office space. Finding new office space was only necessary because of our staff size. (2) Because our staff is larger and we have a larger office, we now need an office manager. (Early in our history, we didn’t have an office, or we worked in a shared workspace where someone else played this role.)
• As our money moved grows, it becomes more worthwhile to be more thorough (as we believe we have been), and it becomes more worthwhile to try to find new top charities even if doing so is a long-term and uncertain proposition. While the work we did in 2015 did not lead to any immediate new top charities, we believe that some of it may have significant returns in the future (e.g., the “GiveWell experimental” work discussed above and the preliminary work that we did on new charities and interventions that we have not finished evaluating).

We previously discussed our reasoning about why we believe it is worthwhile to continue to expand GiveWell’s research capacity despite limited recent returns in last year’s self-evaluation post. This year, we decided to take a more in-depth look at specifically where our increased capacity has gone, while adjusting for staff seniority. Specifically, we roughly estimated the number of full-time, co-founder-equivalent people working on each area in each year. (Co-founder time was given a score of 1 but time spent by newer staff was scaled down significantly; for example, we multiplied time spent by entry-level staff in their first year at GiveWell by 0.1. These figures are far from precise, and are intended only to give a rough guess at our changes in total capacity over time.)

We’ve divided our work into several categories:

• GiveWell’s traditional research: time spent on evaluating new charities or interventions and updating our work on existing top charities or interventions.
• Open Phil: time spent on work related to the Open Philanthropy Project.
• Other output: work that directly furthers GiveWell’s and the Open Philanthropy Project’s missions but isn’t research related. Among other things, this includes work related to donation processing (e.g., opening the mail, entering donations into our database, sending thank you receipts), donor communication (answering donor questions, staying in touch with larger donors), and general outreach (giving talks, speaking to the media, and improving our web content). All of these categories of work have increased significantly as our public profile has grown.
• Overhead: work necessary due to the increased size of our staff, such as changing offices and having an office manager (as mentioned above).
• Capacity building: time spent by relatively senior staff to increase our future capacity. This includes time spent (a) recruiting (e.g., reviewing resumes, interviewing and evaluating candidates) and (b) training and evaluating new staff with the goal of increasing future capacity, not generating additional short-term research output. For example, in 2015, Elie estimates that he spent approximately 60% of his time building capacity.
• Training: time spent by newer staff being trained. These staff members are primarily working on projects where the goal is training and evaluation, not short-term research output.

The table shows that our overall capacity has increased significantly since 2012 (from approximately 3 co-founder equivalents then to 9 today). Most of that increase has gone to Open Phil (3.4 co-founder equivalents), capacity building (.8), and overhead (.9). The number of co-founder equivalents focused on GiveWell’s traditional work has stayed low (and relatively constant) since 2012.

Year GiveWell trad-itional research Other output Capacity building Train-ing Open Phil Over-head Total co-founder equiv. Total staff at year-end 2012 0.9 0.4 0.7 0 0.7 0.2 2.9 5.5 2013 0.8 0.6 0.5 0.3 1.4 0.3 3.9 11 2014 0.9 0.4 0.8 0.6 2.6 0.4 5.7 18 2015 0.9 0.7 1.5 1.1 4.1 1.1 9 32

Overall, we find this picture reasonable. While our work on GiveWell’s traditional research has stayed relatively constant in the short term, we think of it as a major benefit of increased staff that we’ve been able to do this while building the Open Philanthropy Project. And we also believe we’ve been laying the groundwork for future capacity increases, since much of our increased staff time has gone into capacity building itself.

Over time, as new staff train and are able to produce more, we hope that this picture begins to shift and that we can devote more capacity to GiveWell’s traditional research, other output, and Open Phil. Our guess is that we will start to see some of this shift in 2016.

Did we make mistakes in allocating staff capacity in 2015?

We see mistakes in our allocation of staff capacity in 2015. Specifically, we didn’t move as quickly as we should have to move staff from one type of work to another. We had several cases where we continued to try to train staff to work on research projects even though we should have already recognized that they were unlikely to be a good long-term fit for thIs type of research, and would be a better fit for another role. It is generally difficult to predict fit, so we classify this as a relatively easy-to-make mistake, though a costly one, since capacity building is a large time investment for relatively senior staff.

Did we make the right choices about how to spend the time that we had for research (i.e., did we choose the best research priorities)?

In addition to staying up to date on our top charities, we prioritized researching the below opportunities with the goal of maximizing our chances of identifying new top charities.

• Deworming organizations. At the end of 2014, we felt that deworming was one of the most promising interventions we were aware of, but the giving opportunities we had found to scale up deworming had limited room for more funding. In 2015, we prioritized finding new deworming organizations and began evaluations of the END Fund, Sightsavers, and Children Without Worms.
• Development Media International (DMI). At the end of 2014, we believed there was a significant probability that we would name DMI a top charity at the end of 2015. We prioritized work on DMI with this in mind, but the results from DMI’s randomized controlled trial were not in line with what they would have needed to be in order for DMI to become a top charity.
• Micronutrient fortification organizations. We prioritized this area because micronutrient fortification programs reach many people at relatively low cost and they are often backed by strong evidence of effectiveness. We prioritized evaluating salt iodization programs run by IGN and GAIN. We also began an evaluation of Project Healthy Children’s programs and invited the Micronutrient Initiative (MI) to apply for a recommendation. MI declined to participate.
• Bednet organizations. We invited Nothing but Nets to apply for a recommendation (it declined) and we began an evaluation of Episcopal Relief & Development’s NetsforLife® Program.

We believe these priorities were reasonable, given the amount of time we had available and the information we had at the time. However, as these investigations – and further investigations in 2016 – have had fairly disappointing results (in terms of not resulting in new top charities), we are rethinking our approach significantly going forward. More on this in a future post.

Have we failed to publish materials that we should?

We have struggled to publish information about many research projects that we have essentially completed. For example, we have formed views on several interventions for which we have not yet published intervention reports. One major reason for this is that we often try to resolve almost all of our major questions about an intervention before publishing a report on that intervention. A possible solution would be to publish more materials that explicitly note that we are sharing preliminary views that could be substantially affected by additional research. We plan to experiment with publishing more research about preliminary views we’ve reached in the future.

However, a continuing challenge in publishing our research work is that we have limited management capacity to review and sign off on public write-ups. We hope that building management and research capacity will steadily reduce this bottleneck over time.

The post GiveWell’s progress in 2015 appeared first on The GiveWell Blog.

In addition to evaluations of other charities, GiveWell publishes substantial evaluation of itself, from the quality of its research to its impact on donations. We publish quarterly updates regarding two key metrics: (a) donations to recommended charities and (b) web traffic. This post is being published late due to staff focusing on updating GiveWell’s charity recommendations in the fourth quarter; it also includes a preliminary view of our money moved since the end of our third quarter.

Preliminary estimate of 2015 money moved (since February 1, 2015)

As of early January 2016, we have tracked about $98 million in money moved to our recommended charities. Excluding Good Ventures, we have tracked about $28 million (of which, roughly half has come from donors giving $1 million or more).

These data are preliminary. We expect that in some cases we are currently overstating our impact (e.g. due to double counting or incorrect attribution of our influence) and in other cases we are understating our impact (since there are several weeks left in our metrics year and there are delays entering data); overall, we would guess that we are currently underestimating our annual money moved. We plan to publish our annual metrics (covering February 1, 2015 – January 31, 2016) in March, at which point we will have more confidence in our data and be able to share more details.

GiveWell’s web traffic / money moved through Q3 2015

The tables and chart below present basic information about our growth in money moved and web traffic in the first three quarters of 2015 compared to the previous two years (note 1).

Money moved and donors: first three quarters

Money moved by donors who have never given more than $5,000 in a year increased about 80% to $2.44 million. The total number of donors in the first three quarters increased about 80% to about 8,300 (note 2). These growth rates are reasonably consistent with the growth we previously reported in our first and second quarter metrics.

Web traffic through October 2015

Growth in web traffic excluding Google AdWords increased about 25% in the first three quarters. Last year, we saw a drop in total web traffic because we removed ads on searches that we determined were not driving high quality traffic to our site (i.e. searches with very high bounce rates and very low pages per visit).

GiveWell’s website receives elevated web traffic during “giving season” around December of each year. To adjust for this and emphasize the trend, the chart below shows the rolling sum of unique visitors over the previous twelve months, starting in December 2009 (the first period for which we have 12 months of reliable data due to an issue tracking visits in 2008).

We use web analytics data from two sources: Clicky and Google Analytics (except for those months for which we only have reliable data from one source). The raw data we used to generate the chart and table above (as well as notes on the issues we’ve had and adjustments we’ve made) is in this spreadsheet (note 3, on how we count unique visitors).

Note 1: Since our 2012 annual metrics report we have shifted to a reporting year that starts on February 1, rather than January 1, in order to better capture year-on-year growth in the peak giving months of December and January. Therefore, metrics for the “first three quarters” reported here are for February through October.

Note 2: Our measure of the total number of donors may overestimate the true number. We identify individual donors based on the reported name and email. Donors may not share all of this information or may update it (for example, using a different email), in which case, we may mistakenly treat a donation as if it was made by a new donor. We plan to investigate how large of an overstatement there may be and possibly adjust the total for our next annual metrics report.

Note 3: We count unique visitors over a period as the sum of monthly unique visitors. In other words, if the same person visits the site multiple times in a calendar month, they are counted once. If they visit in multiple months, they are counted once per month.

The post Update on GiveWell’s web traffic / money moved appeared first on The GiveWell Blog.