Nick Otis — Forecasting Research

Published: June 2018

Note: This page summarizes the rationale behind a GiveWell Incubation Grant to Nick Otis. Nick reviewed this page prior to publication.

Summary

In May of 2018, Nick Otis received a GiveWell Incubation Grant of $10,000 to support forecasting work relevant to GiveWell's top charities. This is a relatively small grant to support research that we expect to be potentially useful for our decisionmaking.

Table of Contents

About the grantee

Nick Otis ("Nick" throughout this page) is a first-year PhD student in Health Economics at UC Berkeley who has done some forecasting research previously. We first considered funding this project after a conversation between Nick and GiveWell staff at a conference (“EA Global 2017”). Nick has also received about $20,000 in funding for this project from the Harvard Weiss Fund.

Nick asked us to share that he can be contacted at notis@berkeley.edu.

About the grant

This grant will support:

Nick also received $20,000 from the Weiss Fund to collect forecasts from beneficiaries in Kenya about other studies; he hasn't received any other funding for surveying academics or Mechanical Turk respondents. We expect it to be valuable to collect forecasts from multiple groups and see which group's forecasts are most accurate.

The forecasting is set to happen within the next several months. The planned output of this project is a paper by Nick summarizing his work and results.

Because of its small size, we vetted this grant somewhat less thoroughly than usual.

Goals for the grant

We expect our grantmaking decisions would be improved by having a method for collecting external forecasts on 1) the outcomes of planned RCTs and other studies, and 2) other key questions relevant to our cost-effectiveness analyses. Our cost-effectiveness analyses are an important input to our decisionmaking and frequently involve difficult judgment calls; in the long-term, we think it could be valuable to have a system for quickly getting external input on key questions, particularly from people working in development economics and global health whom we’d expect to have useful perspectives. We also expect this project to be useful in the near-term by piloting potential forecast collection methods on studies that are relevant to our top charities.

Plans for follow-up

We plan to follow up when Nick's paper is complete (roughly a year from now). Key questions for follow-up include:

  • What were respondents' forecasts on the outcomes of the key studies?
  • How accurate were those forecasts?
  • Did these forecasts update our views on the expected cost-effectiveness of any programs?
  • Has Nick developed methods that GiveWell could use for collecting forecasts about other key parameters of our cost-effectiveness analyses and grantmaking decisions?

Internal forecasts

For this grant, we are recording the following forecasts:

Confidence Prediction By Time
90% Nick produces a paper summarizing his work on this project. End of 2019
60% Nick collects forecasts from at least 10 academics on at least four studies. End of 2019
65% The academics' pooled forecast of the probability that New Incentives' intervention increases vaccine coverage by 15 percentage points differs from GiveWell's internal forecast by at least 10 percentage points (for instance, the academics give a 45% chance while we give a 60% chance). End of 2019

Source URL: https://www.givewell.org/research/incubation-grants/may-2018-forecasting-research-grant