# GiveDirectly: Supplementary Information - January 2017 Version

We have published a more recent supplementary information page. See our most recent supplementary information page for GiveDirectly.

This page contains further discussion and information for our review of GiveDirectly. This page is intended to provide supplementary information on topics covered in the main review and is not intended to be read independently of the main review. The information on this page is less frequently updated than our main review; there is a note at the top of each section indicating when it was last updated.

## Grant structure

Section last updated: November 2016

GiveDirectly's standard model involves grants of approximately $1,000 (USD) over about four months, after which recipients become ineligible for further grants.1 GiveDirectly has told us that it adjusts its transfer sizes for purchasing power; as of late 2015, in Kenya, GiveDirectly transferred approximately$1,040 to each enrolled household, while in Uganda, it transferred approximately $875.2 We are not sure what the size of transfers in Rwanda will be, but expect it to also be about$1,000 per household.

This is a different approach from the approach we've seen in government cash transfer programs. One way of putting the difference (which has been reflected in GiveDirectly's communications with us) is that government programs aim for "income transfers" (small supplements to income over many years), whereas GiveDirectly's standard program aims for "wealth transfers" (large, one-off transfers that hopefully give people more flexibility to make large purchases and investments). GiveDirectly's basic income guarantee program (expected to launch in late 2016) will be structured as "income transfers."

GiveDirectly's standard transfer schedule involves a small initial transfer (or "token" payment) of about $90 (USD), followed by two larger transfers of about$475 (USD).3 GiveDirectly aims to send these transfers over a period of approximately 4 months.4 GiveDirectly has an ongoing study of behavioral interventions that will allow some recipients the ability to choose when they receive their transfers (see this spreadsheet).

Note that when we reviewed household data from Kenya several years ago, we found that household size varies substantially: while the mean household size was ~4.7 and the median size was 4, 16% of households had 1 or 2 people, ~20% had 6 or more, and the maximum household size was 16.5 We estimate that the average family receives $288 per capita from GiveDirectly, which is 121% of baseline annual consumption per capita for recipients in Kenya.6 ## GiveDirectly's process Section last updated: November 2016 GiveDirectly currently operates in Kenya, Uganda and Rwanda (more details about how GiveDirectly chose those countries in the footnote).7 GiveDirectly is not prioritizing expansion to other countries, as there remain many poor households to serve in the countries in which it operates.8 It does, however, consider opportunities in other countries when compelling reasons are presented, such as the opportunity to impact policy in a way that could only be done in that country or the opportunity to serve a significant number of poor households but only by operating in a donor’s preferred country.9 GiveDirectly's typical process is as follows: 1. Selection of a local region: Once GiveDirectly has selected a country, it narrows down the geographic region in which it would like to work based on a variety of factors, heavily weighting poverty statistics. For example: • GiveDirectly told us that it initially chose to work in western Kenya and eastern Uganda based on poverty statistics.10 • GiveDirectly considers poverty data, population density, logistical and security factors, and the presence of other poverty-focused NGOs when it selects a district or county to work in.11 • In early 2015, when selecting sub-counties and sub-locations in Kenya, GiveDirectly considered poverty data, the number of potentially eligible households, how easily it could transfer staff capacity to the new locations, and how rural each area was.12 Note that we have reviewed the data GiveDirectly used in some of the examples above (see footnotes). 2. Selection of villages: GiveDirectly selects villages primarily based on poverty level and location.13 For details on how GiveDirectly has targeted villages historically, see this footnote.14 For recent campaigns in Kenya and Uganda, GiveDirectly has estimated poverty levels through census data.15 3. Obtaining permission from local officials: Before beginning to work in a given area, GiveDirectly obtains permission from local officials. This process can involve officials from the national to the village level and generally requires a series of conversations to get all the relevant stakeholders on board.16 GiveDirectly signs written agreements with or obtains approval letters from local officials to formalize permissions.17 4. Village meeting: A village meeting is held to "introduce GiveDirectly and its programming to the village residents, to answer questions anyone may have about the program, and to clarify that [GiveDirectly is not] affiliated with a political party, government agency, etc."18 5. Enrollment process: • Census: GiveDirectly has field staff from its census team visit the village to create a census of all households.19 The field staff collect data about each household and note if the household is eligible for transfers (the criteria for eligibility in a campaign depend on where the campaign is located – see our review of GiveDirectly for more).20 • Registration: GiveDirectly has a separate set of field staff from its registration team visit households marked as eligible in the census and register them.21 Registration involves a) helping recipients set up a payment system to receive transfers (if they don't already have such a system in place), and b) collecting an additional round of data from the household that can be checked against the initial data from the census (more detail in footnote).22 A registered household is formally enrolled only after all phases of enrollment (census, registration, back check, and audit) have been completed and the household has obtained a mobile money account (if necessary).23 • Back check: GiveDirectly sends a separate team of field staff from its back check team to revisit every registered household and collect data about that household that can be compared to data collected during census and registration.24 GiveDirectly field staff also ask households if they were asked to pay a bribe to register.25 GiveDirectly is currently testing a more streamlined version of its program that does not include the back check step; GiveDirectly hopes to know by the end of 2016 if it can maintain the quality of its program without this step.26 • Audits: GiveDirectly sends field staff to revisit a portion of the registered households for audits.27 GiveDirectly determines which households to audit based on the extent of the discrepancies between data collected at different phases in enrollment.28 GiveDirectly field staff resolve discrepancies during audits to determine whether households are eligible or ineligible. Households found to be eligible through this process are then considered formally enrolled, in addition to the households considered eligible after back check and not selected for audit.29 GiveDirectly aims to enroll all eligible households.30 If eligible members of the household are not home during a phase of enrollment, GiveDirectly staff revisit the household several times until they can be found.31 We have reviewed (and made public) data collected during each step of the enrollment process for most of GiveDirectly's campaigns, with deletions to preserve anonymity.32 6. Sending transfers to recipients: GiveDirectly sends transfers to recipients via mobile money providers (and, in one campaign, via a bank) (more in our review of GiveDirectly).33 GiveDirectly is currently testing a more streamlined version of its program that does not include the token payment; GiveDirectly hopes to know by the end of 2016 if it can maintain the quality of its program without this element.34 7. Conducting follow up calls: GiveDirectly field staff make multiple phone calls and, for vulnerable recipients, in-person visits, to all recipients as transfers are being sent.35 The schedule of follow up calls has varied somewhat by campaign.36 In 2016, GiveDirectly changed its policies such that recipients cannot receive their next transfer installment until they have been reached for follow-up.37 In addition to the follow-up calls, GiveDirectly maintains a phone "hotline" for recipients to call if they have any questions about the transfers or issues in obtaining funds.38 Recipients can also report issues to GiveDirectly field staff when they are in the village; GiveDirectly created a formal mechanism for recording these reports.39 ## Signed partnership project agreements Section last updated: November 2016 GiveDirectly has signed agreements or memoranda of understanding (MOUs) for the following partnership projects: • Rwanda benchmarking project: In 2015, GiveDirectly finalized an agreement for a partnership project in Rwanda. GiveDirectly will be implementing cash transfers in two randomized controlled trials; the studies will cost$4 million and are co-funded by an institutional funder and Google.org.40 The studies will test cash transfers as a benchmark against other aid programs funded by the institutional funder.41 GiveDirectly started enrollment for one of the studies in 2016, for which it expects results in late 2017, and as of August 2016 was finalizing the structure of the second study.42 GiveDirectly has provided us with some additional details about the studies, which are not yet public.
• MOU with large institutional funder: In 2016, GiveDirectly signed an agreement with an institutional funder (whom it is partnering with for the Rwanda benchmarking project) which provides a mechanism through which multiple benchmarking projects (projects comparing cash transfers to other types of aid programs) can be launched.43 The funder and GiveDirectly have each offered to contribute up to $15 million (for a total of$30 million) to support four different benchmarking projects with GiveDirectly acting as the implementer for the cash arm.44 We describe what we know about the process of setting up the benchmarking projects in this footnote.45 We do not yet have details of which aid programs will be evaluated or how the evaluations will be carried out. By the end of 2016, GiveDirectly hopes to identify two countries in which it would like to do a benchmarking project.46

## Staff structure

Section last updated: November 2016

GiveDirectly delivered its first cash transfers in 2011.47 Starting in January 2011 it had one full-time staff member.48 In early 2013 it hired a second full-time staff member to serve as COO (Domestic).49 GiveDirectly has since expanded its staff significantly. As of February 2016, its organizational structure in East Africa included:50

• Chief Operating Officer International (COO-I): The COO-I provides oversight and quality control of cash transfer programming and international operations. The COO-I oversees the Country Directors.
• Country Directors (CDs) and Field Directors (FDs): Both CDs and FDs are primarily in charge of overseeing field operations. The Country Directors oversee operations in a given country; the Field Director position is a slightly more junior role. Combined, GiveDirectly had four CDs and FDs in early 2016.51
• Field Managers and Associate Field Managers: The Field Managers supervise Associate Field Managers, focusing on quality control, management, and training of Field Officers.52 Associate Field Managers manage the logistics of transfer rounds and oversee Field Officers, as well as conduct high-level analysis of field operations and work on technology integration.53 GiveDirectly had 10 Field Managers and Associate Field Managers in early 2016.54
• Field Officers (FOs): FOs implement the steps required on the ground to enroll and follow up with households. They have the most face-to-face interaction with recipients and are all hired within the country of the transfers. There is a separate group of FOs for each of the first three pre-transfer stages: census, registration, and back checks. FOs are also hired to conduct audits and follow-up surveys with recipients post-transfers; some of the FOs hired for these roles may have previously worked on the census, registration, or back check phases.55 GiveDirectly had 71 Field Officers in early 2016.56

It is our impression that GiveDirectly has grown substantially in 2016 and the numbers above may no longer accurately represent its current size.

## Segovia

Section last updated: November 2016

In mid-2014, three members of GiveDirectly's board of directors began the for-profit technology company Segovia, which develops software that NGOs and developing-country governments can use to help implement their cash transfer programs.57 Paul Niehaus and Michael Faye, co-founders of GiveDirectly and Segovia, split their time between the two organizations.58 They previously told us that they track their time allocation to projects and would be able to share details of how much time they each spend on GiveDirectly and Segovia; however, when we requested this information, GiveDirectly told us that it had previously understood that we no longer wanted this information so it had stopped explicitly time tracking. This was a miscommunication, but we decided not to ask GiveDirectly to track this going forward. GiveDirectly estimates that the time Faye and Niehaus collectively spend on both organizations amounts to each organization having the equivalent of a full-time CEO.59 One other staff member who was previously working full-time at GiveDirectly now works part-time for each entity.60 We discuss potential risks from the overlap in staff in this blog post.

When Segovia was created, it expected to provide its services to GiveDirectly for free, in order to avoid conflicts of interest between the two organizations.61 However, in 2016, after realizing that providing free services to GiveDirectly was too costly for Segovia (customizing the product for GiveDirectly required much more Segovia staff time than initially expected), the two organizations negotiated a new contract under which GiveDirectly will compensate Segovia for its services.62 GiveDirectly wrote about this decision on its blog. GiveDirectly told us that it recused all people with ties to both organizations from this decision and evaluated alternatives to Segovia.63 We have seen information on what portion of Segovia's revenues GiveDirectly accounts for, but we do not have permission to share that figure publicly.64

Although we believe that there are possibilities for bias in this decision and in future decisions concerning Segovia, and we have not deeply vetted GiveDirectly's connection with Segovia, overall we think GiveDirectly's choices were reasonable. However, we believe that reasonable people might disagree with this opinion, which is in part based on our personal experience working closely with GiveDirectly's staff for several years.

## Targeting strategies

Section last updated: November 2016

### The assets and vulnerability status approach

In 2015, GiveDirectly started to work in Homa Bay County in Kenya, where families are less likely to have thatch-roofed houses due to a scarcity of grass.65 Consequently, GiveDirectly changed its eligibility criteria for Homa Bay County to better capture the poorest households.66 The algorithm GiveDirectly uses to determine eligibility in Homa Bay takes into account a range of factors including household assets and the vulnerability status of potential recipients; we are unable to elaborate because GiveDirectly would prefer to keep the criteria confidential so as to prevent households from gaming the system (more detail in footnote).67 More detail on how the algorithm was developed is in this footnote.68 Note that GiveDirectly may adjust its eligibility criteria for other campaigns based on its experience in Homa Bay.69

GiveDirectly tried to choose criteria that (a) included recipients who would benefit the most from the transfer, (b) were difficult to fake, (c) were low cost to implement, and (d) were perceived as fair both by community members and by GiveDirectly staff.70 GiveDirectly believes that, compared to the housing materials approach, the assets and vulnerability status criteria are more difficult to fake, somewhat more expensive to administer, and more difficult to explain (which might lead to people believing the criteria are not fair).71 Note that recipients are not made aware of the full criteria (as a measure to prevent cheating), which may also contribute to decreased perceptions of fairness.72 However, because the criteria explicitly put weight on vulnerability, they could also increase perceptions of fairness, or at least offset other fairness concerns.73

GiveDirectly's assets and vulnerability criteria may help GiveDirectly expand to new areas more easily and could provide valuable guidance for other cash transfer programs (although we are unsure if GiveDirectly will be able to share learnings from this project since it hopes to keep its algorithm confidential). However, our evidence for GiveDirectly's impact and for low rates of conflict within villages is based on previous campaigns in which GiveDirectly used different eligibility criteria, and it is possible—although we think unlikely74—that these criteria will substantially change these outcomes.

### Housing materials approach

As part of the baseline survey for the RCT of its program, researchers collected in-depth information on poverty levels of recipients. GiveDirectly has shared the full survey form used to interview participants, as well as its own summary of the data collected as of March 2012:75

Well over half of adults skip meals, less than half of household members eat until they are content, people commonly go to sleep hungry and a paltry 18% report having enough food for tomorrow in their household. Those living in eligible homes are even worse-off than the average household, consuming less and holding fewer assets. Overall, mean and median daily per capita consumption among eligible households are $0.65 and$0.55 at nominal rates, and 74% are below the Kenyan poverty line, indicating a very poor population.

GiveDirectly reports that recipients in Uganda have a slightly higher average daily consumption of $0.83.76 GiveDirectly also provided charts that show a clear difference in the consumption, expenditures, and assets of households in mud and thatch homes compared to those in cement homes, but fairly small differences between those living in mud and thatch homes and those living in mud and iron roof homes.77 End-line data on food consumption among control group recipients from GiveDirectly's RCT also suggests that the thatched-roof eligible households are extremely poor.78 This data shows that "20% [sic] of the control group reports that not all household members usually eat until they are content, 23% of respondents report sleeping hungry in the last week, and only 36% report having enough food in the house for the next day."79 Other results related to food consumption are measured as well, which are, in our view, consistent with the notion that recipients are extremely poor. ## Concerns about GiveDirectly's targeting strategy Section last updated: November 2016 How much poorer are those selected by GiveDirectly's criteria? It is not clear to us that people in thatched-roof homes (eligible for transfers) are substantially and consistently poorer than people in iron-sheet-roofed homes with mud walls and floors (not eligible for transfers in a standard campaign). In community-based targeting pilots, GiveDirectly recipients identified households that did not meet GiveDirectly's standard targeting criteria but seemed comparably poor.80 GiveDirectly has also received feedback from field staff and recipients that using housing materials as the targeting criteria systematically misses some households that are viewed within communities as comparably poor to those in thatched-roof houses.81 GiveDirectly still feels that housing materials are an effective means of targeting the poorest of the poor, on average, in areas where it has worked to date.82 Note that the concern that GiveDirectly's criteria do not select the poorest households could also apply to the assets and vulnerability status criteria. However, from a sample of 423 people, GiveDirectly found that its algorithm selected recipients with an average consumption of$0.50 per day, compared to a community average consumption of $0.86 per day.83 We don't believe these numbers are highly reliable, but they lend some support to the claim that GiveDirectly on average targets poorer households.84 What do housing materials or assets indicate about financial management? To the extent that there are differences in income or wealth between residents of eligible homes and those who live in non-eligible homes, it seems possible that these differences come down to fortune/luck (e.g., people in iron-sheet homes have been more fortunate and thus able to afford iron sheets), but we also think it may come down to differences in choices regarding financial management (e.g., people in iron-sheet homes may have demonstrated better financial management and planning, thus allowing them to acquire iron sheets). If the latter is the case, there is a potential risk that GiveDirectly is systematically targeting the people who are less likely to use additional money well. GiveDirectly comments: "The most informative data available on this point are the differential impacts we’re seeing within the set of eligible households – specifically, poorer families seeing bigger impacts on nutrition while richer households see bigger impacts on tangible investment."85 Are the benefits of targeting the poorest worth the costs? We also wonder if attempting to target only the poorest members of a community (with any eligibility criteria) is worth the costs, given that we expect almost everyone in the communities that GiveDirectly works in to be quite poor. In addition to the cost of staff time needed to select eligible households and verify their eligibility, giving cash transfers to some members of a community and not others has the potential for increased conflict. GiveDirectly's follow up surveys demonstrate that cash transfers can lead to tension between recipients and non-recipients.86 Though follow up surveys report low levels of tension and conflict, we would expect these to be underreported by recipients to GiveDirectly staff, a dynamic that GiveDirectly has seen play out in past cases.87 GiveDirectly conducted a small-scale study in Kenya to see whether more inclusive targeting criteria could reduce tension and conflict within villages. We find the results inconclusive. However, we note that GiveDirectly will be testing universal enrollment again as part of its basic income guarantee study.88 When we spoke with three field staff in Uganda, two of them suggested that it would be better for GiveDirectly's transfers to reach more people in a village, even if it meant reducing the size of a standard transfer. According to the Assistant Field Manager, the current targeting model causes bragging and unrest in the communities, potentially motivating those who don't benefit to steal from those who do. He said it would be better for GiveDirectly to provide transfers to everyone in a village, even if some transfers were small.89 ## Refusals in Homa Bay, Kenya Section last updated: November 2016 While we believe GiveDirectly currently has robust systems in place to identify recipients who are attempting to game its enrollment process, GiveDirectly has recently struggled to persuade some potential recipients to start the enrollment process, even if they are eligible.90 GiveDirectly started enrolling recipients from Homa Bay county, Kenya in mid-2015.91 There, it encountered unexpectedly high rates of refusals from potential recipients; while refusal rates in Uganda and Siaya, Kenya have historically been low (around 5%), refusal rates in Homa Bay have been about 45%.92 GiveDirectly believes the refusals are due to widespread skepticism towards GiveDirectly's program and rumors that GiveDirectly is associated with the devil.93 Although GiveDirectly has created an outreach team to address the issue, in part to prevent the problem of refusals from spreading, as of September 2016 it was still facing high rates of refusals.94 GiveDirectly told us that while the refusals have reduced GiveDirectly's efficiency somewhat, GiveDirectly was still ahead of its enrollment targets as of late August 2016.95 ## Data from follow-up surveys on how recipients spend their cash Section last updated: November 2016 For several of GiveDirectly's past campaigns, GiveDirectly staff surveyed recipients on how they used their cash transfers during the follow-up calls.96 The surveys were conducted at different points in the transfer cycle of each campaign.97 We summarize the data from the more recent campaigns in Kenya and Uganda below. The spending data from Kenya covers portions of the Kenya 2M, Kenya 1.2M, and Kenya rolling campaigns, and covers dates from February 2014 to September 2015. The spending data from Uganda covers some of the Uganda pilot campaign from October 2013 to April 2014.98 Note that we do not put much weight on this data, as it is all self-reported and we have no control group to compare it to. Amount of reported funds spent, by category Kenya Uganda Category Amount of funds reported to be spent in category (KES) % of total funds reported to be spent in category Amount of funds reported to be spent in category (UGX) % of total funds reported to be spent in category Food 8,996,160 5.0% 20,667,800 4.4% Clothing 1,448,061 0.8% - - Household items 8,590,151 4.8% 56,122,24099 11.9% Building 100,863,660 55.9% 194,449,559 41.2% Land 5,499,000 3.0% 19,603,000 4.1% Livestock 13,621,595 7.6% 66,344,250 14.0% Farm business 1,896,405 1.1% 10,536,000 2.2% Non-farm business 8,007,323 4.4% 8,414,000 1.8% School 9,664,617 5.4% 49,246,000 10.4% Medical 1,421,347 0.8% 13,434,010 2.8% Water 25,800 0.0% 0 0.0% Debt 837,951 0.5% 6,444,000 1.4% Savings 8,551,415 4.7% 19,258,500 4.1% Life event 5,571,655 3.1% 750,000 0.2% Family 1,429,030 0.8% 866,000 0.2% Church 105,450 0.1% 141,000 0.0% Transport 1,448,285 0.8% - - Alcohol - - 5,000 0.0% Other 2,331,600 1.3% 6,190,000 1.3% Total 180,309,505 100.0% 472,471,359 100.0% Note that GiveDirectly has also presented some limited data on spending in a single village on its website.100 This data indicates that the vast majority of recipients (over 75%) in the village used their transfer to buy an iron roof.101 The next three largest categories of spending were on other home improvements, livestock, and furniture.102 ## Data from follow-up surveys on problems experienced by recipients Section last updated: November 2016 We summarize survey data from several campaigns in Kenya and the pilot campaign in Uganda for some of the questions included in these surveys. This table includes follow-up survey data primarily from the Kenya 2M, Kenya 1.2M, Kenya rolling enrollment, and Kenya behavioral optimization campaigns (survey results are from 2014 and 2015) and from the Uganda pilot campaign, the Uganda 2M campaign, and the Uganda model variations campaign (survey results are from 2013, 2014, and 2015). Note that recipients may have been surveyed more than once and would therefore be included more than once in the data presented.103 Percentages reported in this table represent the number of recipients who are marked as having responded "yes" (that they had the issue) out of those for whom a response is recorded in the data.104 Kenya Uganda Issue # of reports/# of respondents % reports of total respondents # of reports/# of respondents % reports of total respondents Trouble collecting 141 / 17,289 0.8% 39 / 1,950 2% Complaints 2,314 / 39,554 5.9% 159 / 5,467 2.9% Theft105 490 / 18,802 2.6% 18 / 5,511 0.3% Bribes106 67 / 39,547 0.2% 33 / 5,552 0.6% Shouting 558 / 39,547 1.4% 69 / 5,521 1.2% Crime 311 / 39,544 0.8% 24 / 5,530 0.4% Domestic violence 428 / 17,905 2.4% 1 / 3,555 0.0% Household argument 182 / 39,546 0.5% 34 / 5,547 0.6% ## Spending breakdown Section last updated: November 2016 Below we break down GiveDirectly's total spending through July 2016 by activity; however, note that the table does not include data for the period July 2015 - February 2016, because GiveDirectly's breakdown of the data for that time period did not match the categories used at other times.107 Costs not included in GiveDirectly's total spending were at least some of the research costs of the independently-run studies of GiveDirectly's program (these costs are not funded by GiveDirectly)108 and the reserves that GiveDirectly had set aside to cover staff salaries in the event that GiveDirectly has a funding shortfall.109 Breakdown of GiveDirectly's total spending by activity - through July 2016, excluding July 2015 - February 2016110 Cost category Spending % of total costs Direct grants to recipients$30,430,766 83.2%
Enrollment costs $723,748 2.0% Transfer costs$458,022 1.3%
Follow-up costs $185,773 0.5% Core operations111$2,574,585 7.0%
Other (excluding fundraising) $54,379 0.1% Fundraising$1,727,153 4.7%
Value of President's time pre-FY 2014 $400,000 1.1% Total$36,554,425 100.0%

For the period that is missing from the table above, we know that GiveDirectly spent $13,576,343 and that$10,832,798 of that went to direct transfers (79.8%).112

## Differences in cost-effectiveness of GiveDirectly's programs

Section last updated: November 2016

### Basic income guarantee cost-effectiveness

We have limited information on the cost-effectiveness of GiveDirectly's basic income guarantee program. While we have modeled the basic income guarantee program in our cost-effectiveness analysis, we do not have a high degree of confidence in our results. Our model indicates that GiveDirectly's standard cash transfer campaigns are roughly 1.5 to 2 times more cost-effective than the basic income guarantee program.113 The difference in cost-effectiveness is partially driven by the fact that GiveDirectly is raising funding for the basic income guarantee study upfront and investing it in a low-risk portfolio. On average, because one arm of the study will receive cash transfers for 12 years, this funding won't be spent for multiple years, so the benefits from the funding are discounted.

It is possible that the basic income model will be less cost-effective than GiveDirectly's standard model because long-term, smaller transfers may reduce incentives or ability to invest the funds, or because it is possible that it will be more expensive per dollar transferred for GiveDirectly to deliver funds.114 However, it is also possible that the program will be significantly more cost-effective, perhaps by allowing participants to make longer-term plans or through influencing other funders and governments to implement basic income guarantees. We have not incorporated estimates of this type of impact into our model. GiveDirectly has told us that while it expects the cost-effectiveness of direct transfers through this project to be lower than its standard program, it believes the potential for beneficial policy impact, which is hard to quantify, outweighs any difference.115

### Benchmarking partnership projects cost-effectiveness

As we have mentioned above, it is our impression that GiveDirectly has increased its focus on experimentation and partnerships. We expect the structure of benchmarking cash transfer programs could look substantially different from GiveDirectly's standard model in some cases. For example, GiveDirectly has told us that if it is benchmarking cash against a program that distributes food stamps, GiveDirectly might disburse smaller and more frequent payments (which recipients are more likely to spend on food) to make the programs more comparable.116 As with the basic income guarantee program, and for similar reasons, benchmarking projects could be more or less cost-effective than GiveDirectly's standard program.

## Are the size and structure of the cash transfers well-thought-through and appropriate?

Section last updated: November 2016

GiveDirectly has considered experimenting with transfer size but does not view this as a high priority, in part because it feels that although further research on this question may improve GiveDirectly's program, it would be unlikely to influence other cash transfer programs.125 GiveDirectly is not concerned that people will run out of good uses of funds from $1,000 transfers.126 The Rarieda RCT included both a$300 transfer treatment group and a $1,000 transfer treatment group, but did not provide strong evidence on what the best transfer size would be, because of small sample sizes.127 ## All sources for GiveDirectly review Section last updated: November 2016 Document Source Carolina Toth, conversation with GiveWell, November 12, 2015 Unpublished Carolina Toth, email to GiveWell, October 20, 2015 Unpublished Carolina Toth, email to GiveWell, September 14, 2015 Unpublished Carolina Toth, email to GiveWell, September 25, 2015 Unpublished Carolina Toth, email to GiveWell, November 10, 2015 Unpublished Carolina Toth, email to GiveWell, May 3, 2016 Unpublished Conversation with Carolina Toth, Field Director, GiveDirectly, October 24, 2013 Unpublished Conversation with Carolina Toth, GiveDirectly, November 20, 2014 Unpublished Conversation with GiveDirectly field staff, October 20-21, 2014 Source Conversation with GiveDirectly, April 8, 2014 Source Conversation with GiveDirectly, December 7, 2013 Source Conversation with GiveDirectly, July 7, 2014 Source Conversation with GiveDirectly, October 6, 2014 Source Conversation with GiveDirectly, September 5, 2014 Source Conversation with Paul Niehaus, November 14, 2014 Unpublished Conversation with Paul Niehaus, President, and Joy Sun, COO, Domestic, GiveDirectly, August 27, 2013 Source Conversation with Paul Niehaus, President, and Joy Sun, COO, Domestic, GiveDirectly, July 18, 2013 Source Conversation with Paul Niehaus, President, and Joy Sun, COO, Domestic, GiveDirectly, July 18, 2013 (unpublished) Unpublished Conversation with Paul Niehaus, President, and Joy Sun, COO, Domestic, GiveDirectly, October 16, 2013 Unpublished Conversation with Paul Niehaus, President, and Michael Faye, Director, GiveDirectly, October 6, 2012 Source Conversation with Paul Niehaus, President, and Rohit Wanchoo, Director, GiveDirectly, March 18, 2013 Unpublished Conversation with Piali Mukhopadhyay, COO, International, GiveDirectly, October 22, 2013 Unpublished Conversation with Piali Mukhopadhyay, GiveDirectly, October 20-21, 2014 Source Conversation with Piali Mukhopadhyay, GiveDirectly, October 20-21, 2014 (unpublished) Unpublished Conversation with Stuart Skeates, GiveDirectly, October 20-21, 2014 Source Conversation with Stuart Skeates, GiveDirectly, October 20-21, 2014 (unpublished) Unpublished Dylan Matthews, Vox article, April 15, 2016 Source (archive) Carolina Toth, GiveDirectly, email to GiveWell, November 14, 2014 Unpublished Carolina Toth, GiveDirectly, email to GiveWell, September 12, 2014 Unpublished Email from Paul Niehaus, President, GiveDirectly, and Joy Sun, COO, Domestic, GiveDirectly, November 18, 2013 Unpublished Email conversation with anonymous funder, May 2016 Unpublished Faizan Diwan, Innovations for Poverty Project Associate, conversation with GiveWell, November 8, 2012 Source GiveDirectly - Evidence - Research at GiveDirectly Source (archive) GiveDirectly blog, Fighting fraud in Uganda Source (archive) GiveDirectly FY 2011 Form 990 Source (archive) GiveDirectly, 100,000 households plan Unpublished GiveDirectly, Aspirations study proposal Unpublished GiveDirectly, Blog post, August 21, 2015 Source (archive) GiveDirectly, Blog post, August 25, 2015 Source (archive) GiveDirectly, Blog post, January 21, 2016 Source (archive) GiveDirectly, Blog post, September 5, 2016 Source (archive) GiveDirectly, Blog post, September 22, 2016 Source (archive) GiveDirectly, Budget summary, July 2013 Unpublished GiveDirectly, Check in with GiveWell, September 2014 Source GiveDirectly, clarifications on GiveWell's draft review of GiveDirectly Source GiveDirectly, Consumption data for targeting work Unpublished GiveDirectly, Contextualizing transfer size Source GiveDirectly, Coffee study design Source GiveDirectly, Distributed cash out follow up with vulnerable recipients Source GiveDirectly, Eligibility check Source GiveDirectly, Eligibility criteria presentation Unpublished GiveDirectly, Enrollment speed of distributions - Siaya and Rarieda Source GiveDirectly, Estimate of personnel 2015 Source GiveDirectly, FAQs 2015 Source (archive) GiveDirectly, Final report Nike girls study Source GiveDirectly, Follow-up tracker, July 2013 Source GiveDirectly, Follow-up tracker, October 2014 Source GiveDirectly, GE research and measurement plan Unpublished GiveDirectly, GE study design Source (archive) GiveDirectly, Give now Source GiveDirectly, Google enrollment database Source GiveDirectly, Google follow-up data - disaggregated (LS - long) Source GiveDirectly, Google transfer schedule, July 2013 Source GiveDirectly, Google verification, September 2013 Source GiveDirectly, GW scratch sheet Source GiveDirectly, How it works 2013 Source (archive) GiveDirectly, How it works 2014 Source (archive) GiveDirectly, Inflation analysis - Kenya Source GiveDirectly, Kenya 1.2M enrollment database Source GiveDirectly, Kenya 2M census results, July 2013 Source GiveDirectly, Kenya 2M enrollment database, September 2013 Source GiveDirectly, Kenya follow up data, November 2014 Source GiveDirectly, Kenya hotline log, July 2013 Unpublished GiveDirectly, Kenya randomized sample of adverse events, 2014-2015 Source GiveDirectly, Kenya rolling campaign enrollment database - Homa Bay Unpublished GiveDirectly, Kenya rolling campaign enrollment database - Siaya Unpublished GiveDirectly, Kenya top 10 adverse events 2015 Source GiveDirectly, Kenya verification template, August 2013 Source GiveDirectly, Kenya, Uganda, and Rwanda enrollment database, 2016 Source GiveDirectly, Monthly operations report, August 2015 Source GiveDirectly, Monthly operations report, February 2016 Source GiveDirectly, Monthly operations report, October 2014 Source GiveDirectly, Nike enrollment database Source GiveDirectly, Nike follow-up data - disaggregated Source GiveDirectly, Nike instrument Source GiveDirectly, Nike verification (combined), May 2013 Source GiveDirectly, Nike verification (final), September 2013 Source GiveDirectly, Nike verification (short version), June 2013 Source GiveDirectly, Offering Memorandum (January 2012) Unpublished GiveDirectly, Operational process overview Source GiveDirectly, Performance - Quality of Service, September 2016 Source (archive) GiveDirectly, Rachuonyo S. Villages Unpublished GiveDirectly, Rarieda Top-up Verification (short) Source GiveDirectly, Rarieda transfer schedule, August 2013 Source GiveDirectly, Rarieda verification (top ups), May 26, 2013 Source GiveDirectly, Rarieda verification stats Source GiveDirectly, RCT Enrollment Database Source GiveDirectly, Revenue by referral source 2015 Unpublished GiveDirectly, Rockefeller index insurance update, July 2015 Unpublished GiveDirectly, Room for funding update for GiveWell, October 2016 Source GiveDirectly, Rwanda technical application Unpublished GiveDirectly, Saturation analysis Source GiveDirectly, Siaya enrollment database Source GiveDirectly, Siaya follow-up data - disaggregated Source GiveDirectly, Siaya poverty data by location Source GiveDirectly, Siaya verification stats Source GiveDirectly, Siaya verification, June 15, 2013 Source GiveDirectly, Siaya village index Source GiveDirectly, Survey for randomized controlled trial Source GiveDirectly, Targeting criteria analysis summary Unpublished GiveDirectly, Targeting focus group results Unpublished GiveDirectly, Targeting process overview Source GiveDirectly, Team Source (archive) GiveDirectly, UBI cost-effectiveness estimate Unpublished GiveDirectly, Uganda 2M campaign enrollment database Unpublished GiveDirectly, Uganda model variations quality audits - census Unpublished GiveDirectly, Uganda model variations quality audits - registration Unpublished GiveDirectly, Uganda pilot enrollment database - Akumure Source GiveDirectly, Uganda pilot enrollment database - Kanyamutamu Source GiveDirectly, Uganda pilot enrollment database - Kawo Source GiveDirectly, Uganda pilot enrollment database - Kosile Source GiveDirectly, Uganda pilot follow up data, April 2014 Source GiveDirectly, Uganda randomized sample of adverse events, 2014-2015 Source GiveDirectly, Uganda targeting data, July 22, 2013 Source GiveDirectly, Uganda top 10 adverse events 2015 Source GiveDirectly, Update for GiveWell on experimentation, September 2016 Source GiveDirectly, Update for GiveWell, April 2014 Source GiveDirectly, Update for GiveWell, February 2015 Source GiveDirectly, Update for GiveWell, February 2016 Source GiveDirectly, Update for GiveWell, July 2013 Source GiveDirectly, Update for GiveWell, July 2014 Source GiveDirectly, Update for GiveWell, May 2015 Source GiveDirectly, Update for GiveWell, October 2014 Source GiveDirectly, Update for GiveWell, September 2015 Source GiveDirectly, Update on process changes, August 28, 2013 Source GiveDirectly, Updated data (March 31, 2012) Source GiveDirectly, Verification data (November 17, 2011) Source GiveDirectly, Verification template (November 7, 2011) Source GiveDirectly, Verification template (October 1, 2012) Source GiveDirectly, Village selection process Kenya Source GiveDirectly, Village targeting regression Source GiveDirectly, What We Do - Operating Model Source (archive) GiveDirectly, What We Do - Operating Model, October 2016 Source (archive) GiveDirectly, What We Do - Who We Serve, September 2016 Source (archive) GiveDirectly staff, conversation with GiveWell, October 6, 2016 Unpublished GiveDirectly staff, responses to monitoring questions, October 11, 2016 Source GiveWell Household size analysis Source GiveWell Site visit notes Source GiveWell site visit to GiveDirectly, October 2014 Source GiveWell visit to M-PESA agent, November 8, 2012 Source GiveWell, GiveDirectly financials 2015 Source GiveWell, GiveDirectly financials - May 2016 Source GiveWell, GiveDirectly financials - 2016 Source GiveWell, GiveDirectly follow up surveys summary - Kenya, September 2015 Unpublished GiveWell, GiveDirectly follow up surveys summary - Uganda, September 2015 Source GiveWell, spot checks of Segovia registration sample 2016 Source GiveWell, spot checks of Segovia follow-up data sample, 2016 Source GiveWell's non-verbatim summary of a conversation with Carolina Toth, GiveDirectly, October 1, 2014 Source GiveWell's non-verbatim summary of a conversation with Paul Niehaus and Carolina Toth, September 7, 2015 Source GiveWell's non-verbatim summary of a conversation with Paul Niehaus, Carolina Toth, and Ian Bassin, February 23, 2016 Source GiveWell's non-verbatim summary of a conversation with Paul Niehaus, Carolina Toth, and Ian Bassin, August 12, 2016 Source GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016 Source Haushofer and Shapiro 2013 Source (archive) Haushofer and Shapiro 2013 Appendix Source (archive) Haushofer and Shapiro 2013 Policy Brief Source (archive) Ian Bassin and Carolina Toth, email to GiveWell, June 14, 2016 Unpublished Ian Bassin and Piali Mukhopadhyay, conversation with GiveWell, August 23, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 17, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 25, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 8, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 11, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 14, 2016 Unpublished Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 17, 2016 Unpublished Ian Bassin, edits to GiveWell's review, November 10, 2016 Unpublished Jean Junior, The Perspectives of Young Women in Siaya County, Kenya: Their Lives and Their Thoughts on Cash Transfer Programs Source Johannes Haushofer and Jeremy Shapiro, Welfare Effects of Unconditional Cash Transfers: Pre-Analysis Plan, June 27, 2013 Source (archive) Johannes Haushofer and Paul Niehaus, DIL Demonstration Proposal Source Lydia Tala, GiveDirectly Field Assistant, conversation with GiveWell, November 7, 2012 Unpublished Michael Faye and Paul Niehaus, Slate article, April 14, 2016 Source (archive) Paul Niehaus and Carolina Toth, conversation with GiveWell, May 28, 2015 Unpublished Paul Niehaus and Carolina Toth, conversation with GiveWell, September 7, 2015 Unpublished Paul Niehaus and Ian Bassin, conversation with GiveWell, September 15, 2016 Unpublished Paul Niehaus and Johannes Haushofer, Optimizing Impact for the Mobile Era - Final Report Source Paul Niehaus, AMA on Reddit, May 31, 2016 Source (archive) Paul Niehaus, Carolina Toth, and Ian Bassin, conversation with GiveWell, February 23, 2016 Source Paul Niehaus, Carolina Toth, and Ian bassin, conversation with GiveWell, August 12, 2016 Unpublished Paul Niehaus, email to GiveWell, October 11, 2016 Unpublished Paul Niehaus, GiveDirectly Founder, conversation with GiveWell, October 22 2012 Unpublished Paul Niehaus, GiveDirectly Founder, email to GiveWell, November 20, 2012 Unpublished Paul Niehaus, Michael Faye, and Piali Mukhopadhyay, conversation with GiveDirectly supporters, August 11, 2015 Unpublished Piali Mukhopadhyay, COO, International, GiveDirectly, conversation with GiveWell, November 7, 2012 Unpublished Piali Mukhopadhyay, COO, International, GiveDirectly, conversation with GiveWell, November 8, 2012 Unpublished Piali Mukhopadhyay, COO, International, GiveDirectly, email to GiveWell, November 23, 2012 Unpublished Piali Mukhopadhyay, COO, International, GiveDirectly, email to GiveWell, August 25, 2016 Unpublished Richard Sedlmayr, conversation with GiveWell, February 19, 2016 Unpublished UCSD, Policy Design and Evaluation Lab, "Tracking the Impact of GiveDirectly Transfers with Mobile Surveys in Kenya" Source (archive) XE currency converter, Kenya shillings to US dollars, September 25, 2015 Source (archive) XE currency converter, Uganda shillings to US dollars, September 25, 2015 Source (archive) • 1. • 2. • 3. • 4. • 5. GiveWell Household size analysis. Note that this data is based on a small sample from one of GiveDirectly's first campaigns (the Siaya campaign). • 6. • 7. • Kenya: GiveDirectly told us that it chose to work in Kenya due to the robustness of M-Pesa as a mobile banking platform and the large population of people meeting its criteria who have access to mobile technology: • Uganda: In choosing a second country in which to work, GiveDirectly said that it considered whether there was a mobile money provider accessible to the very poor, how costly it would be to operate in the country, how politically stable the country is, and how common corruption is in government affairs. The ease of moving staff between Kenya and Uganda was also a factor, as GiveDirectly's current COO (International) oversees the work in both places. Conversation with Paul Niehaus, President, and Rohit Wanchoo, Director, GiveDirectly, March 18, 2013 • Rwanda: GiveDirectly chose to work in Rwanda to pursue a partnership with a bilateral aid donor after staff at the Rwanda office of that donor approached GiveDirectly about a possible cash transfer program there. The employee heard about GiveDirectly through an NPR news piece and became excited about the prospect of GiveDirectly working in Rwanda. GiveDirectly and the donor were then able to secure interest from Google in co-funding such a project through mutual connections. Paul Niehaus and Carolina Toth, conversation with GiveWell, May 28, 2015 Rwanda also has a growing mobile money system and a large population of extremely poor potential recipients. • "While the results of this project will be applicable in many places, Rwanda is an ideal setting in which to conduct it. Rwanda features a) a burgeoning mobile money landscape, with 17% mobile money penetration in mid-2013 as per National Bank of Rwanda and cell phone penetration at 72% in September 2014, b) a sizeable population of poor households (45% below the poverty line), and c) a track record of prioritizing transparent and innovative aid programming." GiveDirectly, Rwanda technical application Pg 5. • GiveDirectly tries to ensure that if its operations in a country are severely hampered or shut down, it has other locations that it can move to and scale up quickly in. GiveDirectly currently works in East Africa and is interested in expanding to different areas of the world, so that it can be more resilient to potential developments in the East Africa region that could negatively affect GiveDirectly's operations. Paul Niehaus, Carolina Toth, and Ian Bassin, conversation with GiveWell, February 23, 2016 • 8. Ian Bassin, edits to GiveWell's review, November 10, 2016 • 9. Ian Bassin, edits to GiveWell's review, November 10, 2016 • 10. • 11. • Kenya: To select counties in Kenya, GiveDirectly told us that its executive staff uses data on poverty, population density, security, and presence of poverty-focused NGOs (with the goal of avoiding overlapping with these organizations). • Uganda: For Uganda, GiveDirectly told us that it chose a county to target initially based on poverty statistics, logistical factors and security considerations. • "Factors that informed decision to locate initial campaign in […] County: poverty rate, […] logistical ease for set up activities and cross-country management, […] minimum security for staff." GiveDirectly, Uganda targeting data, July 22, 2013 We have seen poverty data for Uganda. GiveDirectly sent us the poverty data, which we internally reviewed. • Rwanda: GiveDirectly has not yet selected districts in Rwanda to work in. • 12. • "We prepared a '100,000 households plan' in early 2015 as a roadmap to select sub-counties for the next 100,000 households for GiveDirectly to enroll after the households in Ugunja and Ugenya that are a part of the GE study. We wanted to start with households that could be enrolled with operations based out of our Kisumu office and with our current Luo-speaking staff members. In the far right of the '20160216 100,000 households plan', you can see that we calculated a ranking that drew from World Bank and Kenya National Bureau of Statistics poverty data. It weighted a given sub-county’s rural poverty, distance from Kisumu, and if the region speaks Luo. The manual adjustments put regions we were already operating in at the top...We selected Rachuonyo North in Homa Bay from this shortlist as it had one of the highest estimated poverty rate from the World Bank (see summary tab in that file) and the largest number of potential eligible households...After selecting the sub-county, we select sub-locations. For the current sub-county, Rachuonyo South, we selected sub-locations that had < 10% urban populations and > 40% 'poor' according to World Bank estimates in the attached '20151005 Rachuonyo S. Villages.'" GiveDirectly, Village selection process Kenya • We have spot-checked the poverty data that GiveDirectly used to select sub-counties in Homa Bay; we believe GiveDirectly's processes were reasonable. GiveDirectly, 100,000 households plan • We have previously reviewed poverty data that GiveDirectly sent us for divisions within Siaya district. GiveDirectly, Siaya poverty data by location • We have not seen examples of or asked GiveDirectly to describe in detail how it selected sub-county locations in Uganda or Rwanda. • 13. "Recently we have been relying on sublocation-level poverty data and looking at village names and locations to omit villages that would be too urban for our current model. We have also been working with satellite data to approximate village-level poverty." Conversation with Carolina Toth, GiveDirectly, November 20, 2014 • 14. • Rarieda, Kenya: In Rarieda (the site of the RCT and the first transfers GiveDirectly provided), GiveDirectly sought poorer districts (based on 2005 census data) that were in places with sufficient accessibility, M-PESA usage, population density to make it more convenient, proximity to Innovations for Poverty Action (the RCT implementer) offices and where there would be a sufficient number of potential recipients in this district (# of thatched roof houses). GiveDirectly chose to work in Rarieda District because it was slightly poorer than the nearby Siaya district. In Rarieda, there are slightly more than 300 villages, and GiveDirectly conducted a census in each village to determine the number of eligible (thatched-roof) and ineligible households in each. It then selected the 100 with the highest proportion of thatched-roof to non-thatched-roof households, and randomly selected 60 of those to serve as the treatment and control groups in its trial. Faizan Diwan, Innovations for Poverty Project Associate, conversation with GiveWell, November 8, 2012 • Siaya, Kenya: GiveDirectly shared the full details of its village selection process for Siaya, including data for each village and the method for weighting the different factors used to select villages in that campaign. GiveDirectly chose to work in Siaya District, the location of the three other sets of transfers GiveDirectly has initiated, because it shared local administration with the Rarieda District, making expansion easier; it chose not to remain in Rarieda because it did not want to overlap in areas in which the RCT was being conducted. Using administrative data, it chose 3 locations within Siaya that it believed had the highest poverty levels. It then ranked the 100 villages in these locations. See GiveDirectly, Siaya village index for the calculations that GiveDirectly did to create "poverty scores" for different villages in Siaya. The weights placed on each indicator (in constructing the index) were determined using the process described in GiveDirectly, Village targeting regression: the more detailed "poverty scores" from GiveDirectly's Rarieda study were regressed on indicators such as "village population," "number of boreholes," etc. GiveDirectly was not able to contact all Village Elders to obtain data (staff estimate they reached 85 out of 100) and it excluded villages whose Village Elders it was not able to reach. In June 2012, it selected the 7 villages which its model ranked as highest poverty to receive the Siaya transfers. For the project funded by the Nike Foundation, GiveDirectly selected the next 36 villages in its ranked list of 100 villages in Siaya District, as this number of villages provided a sample size sufficient to meet their target size. In the Google transfers, GiveDirectly is continuing to work down the list of 100 villages to target those not already targeted in the Siaya or Nike Foundation transfers. Piali Mukhopadhyay, COO, International, GiveDirectly, conversation with GiveWell, November 8, 2012 • Homa Bay, Kenya: We have reviewed the data GiveDirectly used to select villages within Homa Bay, and we believe GiveDirectly's process was reasonable. Once GiveDirectly had selected locations within Homa Bay County (see process above), it selected sublocations based on the poverty rate and the percentage of the population living in an urban area. For each sublocation that met GiveDirectly's criteria, every village within that sublocation was included as one that GiveDirectly would visit. See GiveDirectly, Rachuonyo S. Villages • Pilot campaign in Uganda: For the pilot campaign in Uganda, GiveDirectly relied on publicly available poverty data, which we have reviewed, as well as data that it received from local officials, which we have not reviewed. Conversation with Piali Mukhopadhyay, COO, International, GiveDirectly, October 22, 2013 GiveDirectly also sent field staff to test cell phone reception and measure proximity to market centers, and used these as inputs into its village selection (optimizing for good reception and longer distances from market centers). Conversation with Piali Mukhopadhyay, COO, International, GiveDirectly, October 22, 2013 GiveDirectly also considered population size in the villages, so that it could enroll all eligible households in each village and not exceed the budget for the Uganda pilot campaign, which was funded by part of its Google Global Impact Award. • 15. • Previously, in the Kenya 1.2M campaign, GiveDirectly selected villages by manually estimating the proportion of thatch- to iron-roof homes with satellite imagery. In the Kenya rolling enrollment campaign, GiveDirectly used a machine learning algorithm to estimate thatch-iron proportions at the village level based on satellite imagery. In the Uganda 2M campaign, GiveDirectly relied on parish-level census data with poverty measures, as well as mobile money coverage. GiveDirectly, Update for GiveWell, April 2014, Pg 3. • In the Kenya rolling campaign, as GiveDirectly moved into Homa Bay, it selected villages using World Bank census data. See GiveDirectly, Rachuonyo S. Villages • 16. • Seeking government approvals for GiveDirectly cash transfer campaigns By now, GiveDirectly understands well the process for seeking government approvals in Kenya and does not see acquiring approvals as a major risk. GiveDirectly said that in Kenya it is important to maintain relationships with government officials at the county and district levels; district commissioners introduce GiveDirectly to chiefs, and chiefs introduce GiveDirectly to Village Elders. In Uganda, there are no counties, so GiveDirectly coordinates with a few people at the district level to acquire approvals, and from there connect with officials at the local level […] As part of its networking in Kenya, GiveDirectly staff have met with the Permanent Secretary for Devolution and Planning. This is someone who could help GiveDirectly acquire permission to work in new counties in Kenya. Conversation with Piali Mukhopadhyay, GiveDirectly, October 20-21, 2014 • "When entering a new area, the COO meets with a series of officials to explain the project, obtain permission, and establish a relationship in case any problems arise: • District Commissioner • Chief • Assistant chiefs • Village elders" • In Uganda, GiveDirectly had to spend more time meeting with officials early on in the process, because there is a greater bureaucratic structure than in Kenya. This engagement tapered off after the early stage, though GiveDirectly remains in touch with officials by phone to let them know when GiveDirectly has started conducting field activities. Conversation with Piali Mukhopadhyay, COO, International, GiveDirectly, October 22, 2013 • 17. • "Govn’t relations: Signed MOUs with local officials to maximize buy-in and formalize relationship" GiveDirectly, Update on process changes, August 28, 2013 • Typical approval process Kenya: • "Seek buy-in from County and District Commissioner and sign written agreement w/district • Ensure Governor’s office and relevant Country admin officials informed of expansion activities" Uganda: • "Attain approval letter from Resident District Commissioner for natl renewal • Attain approval letters from RDC, District Security Officer, District Intelligence Officer, and District Development Officer for local renewal" • 18. • 19. Data collected during census can be found in enrollment databases, for example: GiveDirectly, Kenya 1.2M enrollment database • 20. • "Enumerators enter villages, engage a local to serve as guide for the day, and enumerate all households living in the village, noting which homes are eligible." GiveDirectly, Operational process overview, Pg 2. • Data collected during census can be found in enrollment databases, for example: GiveDirectly, Kenya 1.2M enrollment database • "Household level – we enroll households living in mud and thatch homes." GiveDirectly, Operational process overview, Pg 2. • "Dropped mud walls as eligibility requirement." GiveDirectly, Update for GiveWell, October 2014 • "We've now completed our targeting pilots in Kenya, and have selected a new targeting criteria for Homa Bay. We looked at a range of alternatives from simple proxy means tests to more complicated rules to subjective scoring to community-based methods, and evaluated them along dimensions of accuracy, perceived fairness, gameability, and cost. The results were useful in the near term for Homa Bay, and also provide a framework and starting point for criteria evaluation in new geographies going forward." Carolina Toth, email to GiveWell, October 20, 2015 • GiveDirectly also gives each household a "token" during the census: "During census, we provide each household with a card with a number on it (a “token”). When we then seek to enroll eligible households, we ask to see the token. This helps ensure that the household being enrolled is the same as the one that was censused and we deemed eligible based on that information." GiveDirectly staff, responses to monitoring questions, October 11, 2016, pg. 1. • 21. "Enrollment. A second, distinct enumerator returns to enroll households identified as eligible, give them a SIM card and instructions on how to register if needed." GiveDirectly, Operational process overview Pg 1. • 22. • Kenya: In Kenya, registration involves giving the household member a SIM card (if they do not already have an M-PESA account), which is used to transfer funds through the M-PESA system, and collecting other data that can be checked against the initial data from the census. Recipients are also given the option of purchasing a cell phone from GiveDirectly at the time of registration, the cost of which is removed from the recipient's transfer (Conversation with Paul Niehaus, President, and Michael Faye, Director, GiveDirectly, October 6, 2012). • Uganda: In Uganda, GiveDirectly uses a similar registration process. Additionally, GiveDirectly helps recipients in Uganda obtain national ID cards and arranges for mobile money agents to visit villages to register recipients in the mobile money system: • "Do recipients need to have a mobile phone to participate? No. Households need at least a SIM card to participate, and we give SIM cards to households that do not already have one. We also give recipients the option of purchasing a phone from us at bulk rates in order to make it easier to communicate with them. When recipients choose this option we deduct the value of the phone from their transfer. Historically the large majority of recipients in both Kenya and Uganda have chosen to buy a phone." GiveDirectly, FAQs 2015 • "How do you prevent corruption? The two main corruption risks that typically arise in transfer programs involve (a) manipulation of the list of eligible recipients and (b) diversion of transfers sent to eligible recipients. We address the first through a comprehensive audit process, using multiple independent checks to ensure that recipients are eligible and have not been charged bribes to get on the list. These checks include in-person visits by different staff members, in-person audits by senior management, remote audits of image and satellite image data, and phone calls with each recipient, all prioritized using modern analytics. We address the second through identity-matching between our records and those of our payment providers, through comprehensive follow-up calls to ensure money is reaching the intended recipients, and in some cases through direct staff monitoring of cash-out points." GiveDirectly, FAQs 2015 • GiveDirectly told us that in Uganda, it is possible to purchase ID cards that can be formally approved by the signature of one's Local Councilperson. GiveDirectly helped recipients obtain ID cards by purchasing the cards, sending field staff to villages to take photographs of recipients, printing the photographs for the ID cards at a local printer, working with Local Councilpeople to approve the cards, and arranging for recipients to collect their cards. GiveDirectly told us that the mobile money agents did not charge a fee to visit villages to register recipients, and that GiveDirectly field staff were present to supervise the process. Conversation with Piali Mukhopadhyay, COO, International, GiveDirectly, October 22, 2013 • The logistics are significantly harder in Uganda than in Kenya. For example, when GiveDirectly enters a new village in Uganda, over 90% of recipients need SIM cards because they did not previously have cell phones, and about 70-80% of recipients need national IDs. GiveDirectly coordinates registration drives for people to get national IDs - they buy national ID booklets, print a photo of each recipient to put in the booklets, and have the Local Councilperson stamp the booklets to approve them. GiveDirectly was able to reach 85-90% of people through these registration drives, returning IDs within about 1 week of visiting eligible households. In the Uganda 2M campaign, there are 9 villages, and GiveDirectly was able to put them all through the national ID registration process within 1 month, so that 90% of eligible households were ready to receive transfers when payments started. (The remaining households will receive their transfers on a delayed schedule, once they complete registration.) GiveDirectly also facilitates recipients signing up for a mobile money account with MTN by having an agent visit the villages. Once recipients have signed up for an account, MTN generally activates their line within 2-3 weeks. By the time the backcheck team visits villages, most recipients’ lines are active. Conversation with GiveDirectly field staff, October 20-21, 2014 • "During census, we provide each household with a card with a number on it (a “token”). When we then seek to enroll eligible households, we ask to see the token. This helps ensure that the household being enrolled is the same as the one that was censused and we deemed eligible based on that information." GiveDirectly staff, responses to monitoring questions, October 11, 2016, pg. 1. • "[GiveWell]: How are the questions about recipients being an imposter asked? (Columns DM and DN) Are these questions asked of neighbors, or judgment calls made by the enrollment team, or something else? [GiveDirectly staff]: These are usually asked of neighbors, the resident guiding us through the village or sometime a village elder. An example of the question asked is: 'Does so and so live here? How long have they lived here?'" GiveDirectly staff, responses to monitoring questions, October 11, 2016, pg. 2. • "Field officers are asked to record when they suspect a recipient of being dishonest. For example, a polygamous husband may try to claim that one of his daughters is actually an additional wife in order for her to qualify for an additional payment (since we enroll each wife in a polygamous household). A field officer might discover this when reviewing documents provided by the daughter or others in the household (e.g, if a document says the family has 3 daughters but they present as having only 2). Usually, the field officers also ask the neighbours and their guides, whose information they use to corroborate what the recipient told them during the interview." GiveDirectly staff, responses to monitoring questions, October 11, 2016, pgs. 2-3. • 23. • 24. Data collected during back checks can be found in enrollment databases, for example: GiveDirectly, Kenya 1.2M enrollment database • 25. "Back-checking. Another enumerator, distinct from the census and enrollment workers, revisits each enrolled household to check that they eligible, didn’t have to pay a bribe to enroll, etc." GiveDirectly, Operational process overview, Pg 3. • 26. • "In early 2016, GiveDirectly allocated a fairly large sum of money to test a new implementation model in Kenya. A separate field director and team were hired approximately six months ago, and the experiment is taking place in an area close to GiveDirectly's other field operations. The primary aim is to double the program's size in 2017 without significantly increasing the management structure. The new model seeks to increase throughput per manager by eliminating the token payment and back check steps. GiveDirectly will assess gains in throughput as well as costs, which might occur in the areas of comprehension and fraud. Results and data should be available after a few months of disbursements. By the end of 2016, GiveDirectly hopes to have a blueprint for implementing a similar model in other countries." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pgs 4-5. • "Approach to checking quality for campaigns with steps removed: • Recipient Comprehension: With the removal of back check, we are tracking any deviation in recipient comprehension below 90% (our preferred rate). We are tracking this through additional survey questions in both audit and our follow up surveys where we ask recipients their understanding of our program. Additionally, Associate Field Managers will be conducting quality check surveys at audit stage for another data point on recipient comprehension. • Transfer Integrity & Adverse Events Detection. Given additional risk of two payment structure, we will be assessing (on a bi-weekly basis) the percent of transfers reversed and percentage of adverse events happening after first payment and after second payment. This will then be compared to our standard quality bar to assess significant deviations. Given a greater need for over the phone assistance, we will also be tracking through our follow up program the percentage of recipients who received no customer service and tried. • Fraud Detection. With the removal of back check and a two payment structure that makes each payment riskier, we will be checking the percentage of recipients flagged for additional audit stage and then the percentage of those deemed ineligible after audit is conducted. We also have an internal audit team in place that will be conducting surveys post-payment to see if fraud occurred." • 27. • "How do you prevent corruption? The two main corruption risks that typically arise in transfer programs involve (a) manipulation of the list of eligible recipients and (b) diversion of transfers sent to eligible recipients. We address the first through a comprehensive audit process, using multiple independent checks to ensure that recipients are eligible and have not been charged bribes to get on the list. These checks include in-person visits by different staff members, in-person audits by senior management, remote audits of image and satellite image data, and phone calls with each recipient, all prioritized using modern analytics." GiveDirectly, FAQs 2015 • In Kenya, the field staff who do audits are not involved in earlier enrollment activities: "[In Kenya] the follow up team sends some of its members to do audits and staff are not pulled from prior enrollment teams." Conversation with Carolina Toth, GiveDirectly, November 20, 2014. • In Uganda, the field staff who have done audits in past campaigns were from earlier enrollment teams. Conversation with GiveDirectly field staff, October 20-21, 2014 (Note that this point was not included in the notes from this conversation). • However, GiveDirectly has told us that it ensures that staff members never revisit households that they helped through previous steps of the enrollment process: "In GiveDirectly's standard operating model, each field officer is responsible for completing one type of step in the enrollment process; for example, a field officer might be assigned to the 'backcheck team.' In two of its experimental campaigns (the experiment to increase throughput in Kenya and its work in a coffee farming community), GiveDirectly is experimenting with a "wave" approach. In this scenario, the same officers perform multiple steps; however, for steps intended to provide a check on previous steps, officers do not perform multiple steps in the same village. This ensures that the same field officer does not visit a household twice during the enrollment process." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 5. • 28. • In August 2016, we asked GiveDirectly for an update on which discrepancies cause it to audit a household: • Previously, GiveDirectly's procedure for identifying households to audit: • GiveDirectly collects information about recipients during the first three stages of a campaign: census, registration, and backcheck. Some information, such as recipient name, GPS location, housing materials, and identifying photograph, is collected at more than one stage and then checked for mismatches. (These checks are currently conducted using Excel but will eventually be automated through Segovia technology. One exception is comparing identifying photographs of recipients, which is done using a crowdsourced work platform called Mechanical Turk.) • Each mismatch in recipient information is assigned a certain weight depending on how likely it is to be an indication of gaming. (GiveDirectly said that it determined the likelihoods of various mismatches indicating gaming by conducting an analysis of the mismatches present in past cases of gaming. We did not review this analysis.) GiveDirectly said that the mismatches with the highest weights are mismatches in identifying photographs and housing materials. • Each recipient is assigned a total mismatch "score" (the composite of all their weighted mismatches). Recipients with scores above a certain level are audited. • 29. • 30. • 31. “People have to be at home at registration and back check so that they can be given phones, have safety information explained etc. If they are not at home during the first attempt to visit, we re-visit them several times until they can be found." Conversation with Carolina Toth, GiveDirectly, November 20, 2014 • 32. • 33. "We use electronic payment systems; typically, recipients receive an SMS alert and then collect cash from a mobile money agent in their village or nearest town." GiveDirectly, What We Do - Operating Model, Overview tab • 34. • "In early 2016, GiveDirectly allocated a fairly large sum of money to test a new implementation model in Kenya. A separate field director and team were hired approximately six months ago, and the experiment is taking place in an area close to GiveDirectly's other field operations. The primary aim is to double the program's size in 2017 without significantly increasing the management structure. The new model seeks to increase throughput per manager by eliminating the token payment and back check steps. GiveDirectly will assess gains in throughput as well as costs, which might occur in the areas of comprehension and fraud. Results and data should be available after a few months of disbursements. By the end of 2016, GiveDirectly hopes to have a blueprint for implementing a similar model in other countries." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pgs 4-5. • "Approach to checking quality for campaigns with steps removed: • Recipient Comprehension: With the removal of back check, we are tracking any deviation in recipient comprehension below 90% (our preferred rate). We are tracking this through additional survey questions in both audit and our follow up surveys where we ask recipients their understanding of our program. Additionally, Associate Field Managers will be conducting quality check surveys at audit stage for another data point on recipient comprehension. • Transfer Integrity & Adverse Events Detection. Given additional risk of two payment structure, we will be assessing (on a bi-weekly basis) the percent of transfers reversed and percentage of adverse events happening after first payment and after second payment. This will then be compared to our standard quality bar to assess significant deviations. Given a greater need for over the phone assistance, we will also be tracking through our follow up program the percentage of recipients who received no customer service and tried. • Fraud Detection. With the removal of back check and a two payment structure that makes each payment riskier, we will be checking the percentage of recipients flagged for additional audit stage and then the percentage of those deemed ineligible after audit is conducted. We also have an internal audit team in place that will be conducting surveys post-payment to see if fraud occurred." • 35. • 36. • 37. "For its standard Kenya and Uganda programs, GiveDirectly has implemented a rule in Segovia to block payments until the previous one has been confirmed: for example, first and second lump sum payments are blocked until token and first lump sum payments, respectively, have been confirmed via a follow-up call or visit. In an effort to ensure implementation quality, GiveDirectly seeks to keep the percentage of households with blocked payments under 5%. The delay between the first and second lump sum payments tends to be the longest. After the final payment, GiveDirectly aims to follow up with a geographically representative sample of roughly 50-60% of households. Staff in Kenya and Uganda have been focused on following up on the first two payments because of the new rule described above; they are in the process of catching up on the schedule for final follow-ups." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 4. • 38. • 39. • GiveDirectly told us that for the Kenya 2M campaign, its enrollment field staff conducted a short survey with anyone who approached the field staff to complain that they had been unfairly or mistakenly skipped during the census. Though [in campaigns prior to and including Kenya 2M] GiveDirectly [did] not add recipients after the census [had] been conducted, it intends to continue carrying out the surveys for future campaigns, as a way of tracking complaints, recognizing potential issues with the census, and assessing changes intended to improve GiveDirectly's census process. Conversation with Carolina Toth, Field Director, GiveDirectly, October 24, 2013. We have not reviewed the results of this survey. As of November 2015, GiveDirectly informed us that it still conducts these surveys, and that people who are skipped in error have an opportunity to be enrolled. Carolina Toth, email to GiveWell, November 10, 2015 • Starting with the Kenya rolling enrollment campaign in early 2014, GiveDirectly now adds into the enrollment process households that complain about having been skipped at census. Conversation with Carolina Toth, GiveDirectly, November 20, 2014 • 40. • 41. • 42. • 43. • 44. • 45. • Location: The benchmarking projects could occur in multiple different countries. GiveDirectly may decline to work in countries that it does not wish to enter. GiveDirectly intends to choose which countries it will work in according to: • How logistically simple it is to work in the country • How much working in the country would increase GiveDirectly's geographic diversity. GiveDirectly would like to test its program outside of East Africa. • How quickly GiveDirectly will be able to set up operations in the country. The easiest countries for GiveDirectly to set up in will likely be those in East Africa. • Whether or not GiveDirectly can find an on-the-ground staff member who is willing to push forward the benchmarking project. GiveDirectly's experience in Rwanda has taught it that having such a "champion" can be invaluable. • Process • According to its agreement with the institutional funder, GiveDirectly is not allowed to reach out to country-level staff to gauge their interest in a benchmarking program. Rather, first the staff must express interest in a partnership with GiveDirectly. • Once interest is expressed, GiveDirectly has a high-level conversation with the country staff. • If that conversation goes well, GiveDirectly will be invited to submit a proposal for a benchmarking project to the country staff. • GiveDirectly and the country staff then negotiate the details of the proposal and sign a contract. In Rwanda, GiveDirectly signed the contract while it still left many details to be worked out later, and this caused difficulties down the road. GiveDirectly now intends to bring up many details about the program during the negotiation stage. • Once the contract is signed, GiveDirectly and the country staff can enter the implementation phase. • 46. • As of August 2016, conversations had been initiated with approximately 8 countries. GiveDirectly thinks it is unrealistic to expect any contracts to be signed by the end of the year. Ideally, GiveDirectly would like to roll out the benchmarking projects in a sequential way to avoid stretching its management capacity too thin. Paul Niehaus, Carolina Toth, and Ian bassin, conversation with GiveWell, August 12, 2016 • "One of GiveDirectly's strategic goals for 2016 is to finalize these negotiations, which it believes would lead to two new countries it will work in. It aims to have tentative agreements in place by the end of the year. GiveDirectly aims to balance its priorities between moving forward quickly with new projects and rolling them out sequentially, which is easier from a management standpoint. Starting and scaling up projects in 4 new countries at once would be a significant challenge." GiveWell's non-verbatim summary of a conversation with Paul Niehaus, Carolina Toth, and Ian Bassin, August 12, 2016, Pgs 4-5. • 47. • 48. "GD has had one person working full-time since January of 2011. Jeremy Shapiro was full time during 2011 and Piali Mukyopadhyay is full-time now." GiveDirectly, clarifications on GiveWell's draft review of GiveDirectly • 49. Conversation with Paul Niehaus, President, and Rohit Wanchoo, Director, GiveDirectly, March 18, 2013 • 50. • 51. GiveDirectly, Monthly operations report, February 2016 • 52. • 53. • 54. GiveDirectly, Monthly operations report, February 2016 • 55. • 56. GiveDirectly, Monthly operations report, February 2016 • 57. • "Three members of GiveDirectly‘s board of directors (Paul Niehaus, Michael Faye, and Chris Hughes) are planning to start a for-profit technology company, Segovia, aimed at improving the efficiency of cash transfer distributions in the developing world. Segovia plans to sell software to developing-country governments for use in implementing their cash transfer programs." (From GiveWell's update on GiveDirectly, June 20, 2014) • What GiveDirectly uses Segovia for: "Segovia, the product GiveDirectly uses for end-to-end data management, enables GiveDirectly to track recipients' progress through each step of its enrollment, payment, and follow-up processes and manage other tasks such as data merging, matching checks between different steps, audit flags, and roster generation. Currently, Segovia automatically generates lists of pending payments, but GiveDirectly must log in to a mobile payment portal, such as M-Pesa or MTN, to upload payments; this additional step takes time and could pose a security risk. Segovia is working to bring the entire payment process into Segovia by directly integrating GiveDirectly's platform with the payment portals. This new feature should be available within the next quarter. Segovia's monitoring and reporting tools, including its dashboards, provide significant value to GiveDirectly, particularly as its campaign numbers increase." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 1. • 58. "Mr. Faye and Dr. Niehaus split their time between Segovia and GiveDirectly, and make efforts to ensure that each organization always has the equivalent of one full-time CEO. For the sake of efficiency, staff member Melissa Harpool, who manages Mr. Faye's and Dr. Niehaus' schedules across Segovia and GiveDirectly, is employed by both organizations. All of the other staff members are employed by only one of the two organizations. There have been no issues related to staff leaving one organization for the other; the organizations' respective staff profiles are quite different." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 3. • 59. Ian Bassin, edits to GiveWell's review, November 10, 2016 • 60. "For the sake of efficiency, staff member Melissa Harpool, who manages Mr. Faye's and Dr. Niehaus' schedules across Segovia and GiveDirectly, is employed by both organizations." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 3. • 61. We mention this in this blog post. If GiveDirectly pays for Segovia's services, then in some sense GiveDirectly is contributing additional income to its co-founders, who are also the co-founders of Segovia. • 62. "Segovia intended and did provide free services to GiveDirectly initially, but the scale of GiveDirectly's needs increased considerably. And the unique nature of different GiveDirectly campaigns required considerable ongoing technical assistance from Segovia that was not part of the original assumptions about what would be required in terms of ongoing, active support. Given that, it was both necessary for Segovia and fair for GiveDirectly to provide some compensation for those services." Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 25, 2016 • 63. • "Michael Faye and Paul Niehaus are co-founders and board members of both GiveDirectly and Segovia. In order to mitigate the potential legal and ethical risks of this situation, they are recused from GiveDirectly's decisions about Segovia. Ms. Mukhopadhyay and Gavin Walsh, Director of Finance and Information Systems, manage the decision to do business with Segovia and the details of that contract. During recent contract negotiations, they assessed the quality and cost implications of building a similar product to Segovia in-house, because no similar product is offered by any other vendors. They concluded that using Segovia's product was the best option and negotiated a pricing model with its product lead and head engineer. GiveDirectly received a discounted quote in line with the planned volume of transfers. Ms. Mukhopadhyay and Mr. Walsh's proposal was accepted by the other independent board members. In making this decision, board members reviewed information about the extent to which Segovia's growth relies on GiveDirectly's business to ensure that GiveDirectly was not too large a percentage of Segovia revenue; this was out of a concern that such a scenario could cause some to question the relationship even if the decision to use Segovia is independently sound." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pgs 2-3. • "Our independent board members did ask what % of Segovia's revenue our contract would represent in order to make sure it was not so high as to raise appearance concerns that could be detrimental to the GD brand and mission. They only considered that information in that regard, not otherwise. Because that information is not ours, however, and is Segovia's that was shared with our independent board members solely for that purpose, we're not at liberty to share it." Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 25, 2016 • "The current agreement term ends on February 28, 2017. The parties intend to discuss renewal beyond that point." Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 25, 2016 • 64. Paul Niehaus, email to GiveWell, October 11, 2016 • 65. • 66. • [GiveDirectly] is also considering modifying its targeting criteria to include certain types of people who may be especially vulnerable whether or not they live under an iron roof. Conversation with GiveDirectly field staff, October 20-21, 2014 • GiveDirectly is considering expanding its eligibility criteria to include: • Widows living in iron-roofed houses • Houses with iron roofs that are severely corroded • Households with partially cemented floors Conversation with GiveDirectly, October 6, 2014 • "Methods being piloted • Community based targeting (multiple variants) • Subjective rankings (by staff, external parties) • Proxy means tests • PPI (Progress out of Poverty Index) • MPI (Multidimensional Poverty Index) • Other additional proxies (e.g., widows)" • "We've now completed our targeting pilots in Kenya, and have selected a new targeting criteria for Homa Bay." Carolina Toth, email to GiveWell, October 20, 2015 • 67. • 68. • To test possible proxies for poverty to use as its new criteria, GiveDirectly attempted to determine the validity and replicability for each metric, and also solicited community feedback (more detail on GiveDirectly's process in the footnote). • Paul Niehaus and Carolina Toth, conversation with GiveWell, September 7, 2015 • "Additional detail on process and excel file context: 1. Conducted desk research on most promising targeting methods 2. Piloted MPI, PPI, numerous types of subjective assessment, numerous types of CBT, different proxies, and blends of these different approaches across 50 villages in Homa Bay 3. Collected recipient and non-recipient feedback after token transfers were sent (See “Targeting Focus Group results”). The focus groups were done on three of the most successful and widely used criteria: [redacted] 4. Collected consumption data as part of our household census for ~500 households in Homa Bay (see “Consumption data for targeting work” which includes the most complete survey versions)—the monthly per-capita consumption figure gathered from this was the measure of each household’s poverty. 5. Analyzed consumption data to see if there were strong predictors of poverty among household’s observable characteristics. Requested analysis and advice of several data scientist volunteers. Unfortunately, there was no single, strong predictor of poverty like thatch. 6. Developed multiple possible criteria (i.e. models) (See “Targeting criteria analysis summary”) based on the piloting and analysis experience, using factors that were the strongest predictors of poverty, fair, and difficult to game. 7. Selected one of the models based upon our priorities of accuracy, perceived fairness, lack of gameability, and cost." GiveDirectly, Targeting process overview • "To inform and structure this decision, we tested out several different targeting methods and eligibility criteria to evaluate their pros and cons, and the circumstances in which we thought they would be most and least useful. For example, we tested: a variety of proxies, such as per-capita housing space and housing materials; community-based targeting, where members of a village nominate, through various means, which they think are the poorest households; points-based systems such as the Progress out of Poverty Index (PPI) and the Multidimensional Poverty Index (MPI); subjective assessments like our field officers rating on a 1-5 or 1-10 scale the poverty of a household or the quality of their house; and various blends of these different approaches." GiveDirectly, Blog post, January 21, 2016 • GiveDirectly put together focus groups to solicit feedback about the criteria it was testing, asking questions about how well GiveDirectly did at identifying the poorest people in the community and whether or not GiveDirectly missed particularly poor households or included not-poor households. GiveDirectly, Targeting focus group results and GiveDirectly, Targeting criteria analysis summary • For example, GiveDirectly tested the same criterion on the same group of people at different times to see if respondents gave consistent answers that led to the same group of eligible recipients each time. • GiveDirectly gave an example of one inconsistent criteria model that it tested: some versions of community-based-targeting were inconsistent. GiveDirectly found that in some community-based targeting models, unique community groups would rate the same family's level of poverty quite differently; while one community group would claim the family was extremely poor, another would tell GiveDirectly that the family was comparatively wealthy. These discrepancies led to GiveDirectly discarding several community targeting models as eligibility models. Paul Niehaus and Carolina Toth, conversation with GiveWell, September 7, 2015 • 69. • 70. • GiveDirectly, Eligibility criteria presentation • Carolina Toth, email to GiveWell, October 20, 2015 • "We wanted to evaluate how well different techniques worked along several dimensions. We looked for accuracy (did the method actually identify poor households), perceived fairness (did community members think that the method was fair), gameability (how easy was it to cheat the system), and cost (how expensive was the method to administer). We will write more about the targeting project on our blog over the coming months. Besides testing and evaluating different techniques, the project allowed us to settle on usable eligibility criteria for our new home in Homa Bay. We found that there was no single, objective replacement for thatched roofs as a criterion (e.g. mud floors or the presence of a latrine) in our new location that also met the bar for accuracy, perceived fairness, gameability, and cost. We also found that some methods that might work well in other contexts didn’t work well in Homa Bay. For example, community-based targeting was perceived as fair and was cost-effective, but it was not particularly accurate, in part due to the high prevalence of clanism in Homa Bay – villagers were sometimes nominating those in their own clan over the poorest households in the community." GiveDirectly, Blog post, January 21, 2016 • 71. • "There are major differences between this method and the thatch and homeless criteria we used in Siaya. Because the new criteria take into account many factors, it’s harder to game the system. Also, the new criteria include a broader range of vulnerable statuses than the criteria in Siaya by adding widows and child-headed households into the algorithm. This is likely to increase the perceived fairness of our eligibility criteria. The new criteria also have some weaknesses: it’s slightly more expensive from an operational standpoint, because the criteria involve more questions, and it is more difficult to explain to community members why a given household was or was not eligible. This may counteract the perceived fairness of enrolling more vulnerable-status recipients. Taking all of these dimensions into account, our new eligibility criteria will allow us to identify and serve the poorest of the poor in Homa Bay. Even though the new criteria make this system slightly more expensive, that cost will likely be mitigated by a decrease in gameability and an increase in accuracy. And although the new algorithm may be perceived as harder to understand, we hope that by accounting for vulnerable-status groups the criteria will be perceived as more fair in the communities where we work." GiveDirectly, Blog post, January 21, 2016 • "Major differences between this criteria in Kendu vs thatch criteria in Siaya Pro: harder for all parties to game, as algorithm is not known Pro: includes vulnerable groups important to fairness perceptions Con: more expensive to administer as there are more questions to answer. Cost savings from reduced gaming may offset this. Con: difficult to explain to communities why someone was or was not included, decreasing fairness perception Taking all of these: we think cost and fairness differences are probably going to be about the same, and so the major gain is the fact that it is difficult to game" Carolina Toth, email to GiveWell, October 20, 2015. • 72. • "Pro: harder for all parties to game, as algorithm is not known" Carolina Toth, email to GiveWell, October 20, 2015. • "The new criteria also have some weaknesses: it’s slightly more expensive from an operational standpoint, because the criteria involve more questions, and it is more difficult to explain to community members why a given household was or was not eligible. This may counteract the perceived fairness of enrolling more vulnerable-status recipients." GiveDirectly, Blog post, January 21, 2016 • 73. "And although the new algorithm may be perceived as harder to understand, we hope that by accounting for vulnerable-status groups the criteria will be perceived as more fair in the communities where we work." GiveDirectly, Blog post, January 21, 2016 • 74. We asked GiveDirectly if it has heard of any increased conflict in locations where they new criteria are being used, and GiveDirectly noted: "Our field directors have not seen or heard any evidence of this." Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 8, 2016 • 75. • 76. GiveDirectly, What We Do - Operating Model, see the Uganda tab. • 77. • 78. Haushofer and Shapiro 2013 Policy Brief • 79. • 80. • "Majority of households are widows, almost half have one family member that is disabled or very sick.” • ”Households seem to be as or more needy than typical GD recipients" • 81. • "Some (but not all) mabati [iron-roofed] or even permanent HH are as deserving as thatched HH […] 6/6 groups mentioned deserving special cases." GiveDirectly, Saturation analysis, Pg 4. • Mr. Ekeu thinks that roofs are too rough a way to target poverty because some people may live under an iron roof but actually be very poor (e.g., someone who inherited an iron-roofed house from his grandfather, or a widow whose late husband built her an iron-roofed house long ago). Ms. Mukhopadhyay said that GiveDirectly hears this kind of feedback from a lot of its field staff, but believes that building materials are still a good criteria on average. Conversation with GiveDirectly field staff, October 20-21, 2014 • 82. Mr. Ekeu [the Senior Field Officer in Uganda] thinks that roofs are too rough a way to target poverty because some people may live under an iron roof but actually be very poor (e.g., someone who inherited an iron-roofed house from his grandfather, or a widow whose late husband built her an iron-roofed house long ago). Ms. Mukhopadhyay said that GiveDirectly hears this kind of feedback from a lot of its field staff, but believes that building materials are still a good criteria on average. It is also considering modifying its targeting criteria to include certain types of people who may be especially vulnerable whether or not they live under an iron roof. Conversation with GiveDirectly field staff, October 20-21, 2014, Pg 4. • 83. GiveDirectly, Consumption data for targeting work • 84. Our impression is that the average consumptions were calculated based on surveys where recipients were asked to report what they had spent in the last year in a number of categories. These amounts were summed, then divided by 12 and the number of people per household to obtain the monthly consumption per person. We believe that self-reported responses, especially about what has been spent in the last year (which is difficult to recall), are unlikely to be highly accurate. GiveDirectly, Consumption data for targeting work • 85. • 86. • 87. • One adverse event reported in the Kenya follow-up tracker involved a case of domestic abuse resulting in the death of the mother and child, where the particular instance of conflict may have been related to the use of transfer funds. When GiveDirectly investigated the event, the parents of the deceased expressed that there was not anything GiveDirectly could have done to prevent this adverse event. People in the community, including the parents of the deceased, also said that they did not want to report the incident because they were afraid that it would cause GiveDirectly to cancel the cash transfer program, which they did not want to happen. Conversation with GiveDirectly, July 7, 2014, Pg 5. • "During this process there were some reports of problems during paydays - recipients were hesitant to come forward initially."Carolina Toth, GiveDirectly, email to GiveWell, September 12, 2014 • 88. Ian Bassin, edits to GiveWell's review, November 10, 2016 • 89. We asked the field officers what they think about the current transfer size ($1000), and whether they’d choose to keep it at that level, increase it, or decrease it, given the effects that an adjustment would have on how many people GiveDirectly would be able to serve.

• Mr. Okello: Typically there are multiple households on one compound, each inhabited by relatives of the same family, and any household that meets the targeting criteria can receive transfers. Mr. Okello said that it may make more sense for GiveDirectly to group some households on a compound together so that transfers are shared across them, rather than each eligible household receiving the full $1000. Mr. Okello also said that if GiveDirectly increased the size of the transfers, that could create a high level of dependency. One of the messages that field officers send is that people should use the$1000 transfers to develop themselves as much as possible, but if someone knew they were getting $2000, they may stop farming, for example. With$1000 people can get some things but not everything; it is the right amount.

• Mr. Ekeu: Mr. Ekeu prefers reducing the amount of money in each transfer and expanding the recipient base to reach everyone in the village. He said that the current targeting model causes bragging and unrest in the communities. The people who don’t benefit may be brought to use force to get some of the money, such as by breaking into recipients’ homes. Mr. Ekeu suggested that it would be better for GiveDirectly to provide all households in a village with some amount of money, even if it was less for households that are currently deemed ineligible (e.g., $100). This way, each of the households would be busy figuring out how they would spend their own money rather than how to get money from another. • Mr. Olinga: Mr. Olinga said that to reduce extreme poverty the bigger transfer is better, but he didn't have a strong opinion on$1000 transfers to some people versus 500 transfers to twice as many. [This is how we posed the question to Mr. Olinga.] • 90. "Recently, however, we’ve seen lower than usual participation rates in parts of Kenya. In July 2015 we entered Homa Bay, a new county and our first venture outside of Siaya. In Homa Bay and the neighboring areas, roughly 45% of the households we speak with decline to be enrolled into the program." GiveDirectly, Blog post, September 5, 2016 • 91. "In July 2015 we entered Homa Bay, a new county and our first venture outside of Siaya." GiveDirectly, Blog post, September 5, 2016 • 92. "In general, an overwhelming majority of eligible recipients opt to receive cash transfers from GiveDirectly. In Siaya, where GiveDirectly Kenya has operated from 2011 to the beginning of 2016, over 95% of recipients who are given the opportunity to be a part of the program accept it. In Uganda and Rwanda more than 96% of eligible recipients have opted in, respectively. These figures are high relative to participation rates in typical development programming, which is not surprising given the unconditional nature of our transfers. For example, Manuela Angelucci and Orazio Attanasio found that Oportunidades, a conditional cash transfer program in Mexico, had take up rates of roughly 50%. Oriana Bandiera et al. report that a BRAC training program targeted at adolescent girls in Uganda expected participation rates of roughly 20%. Recently, however, we’ve seen lower than usual participation rates in parts of Kenya. In July 2015 we entered Homa Bay, a new county and our first venture outside of Siaya. In Homa Bay and the neighboring areas, roughly 45% of the households we speak with decline to be enrolled into the program. As it turns out these challenges have been common for NGOs working in the area. Other development programs focused on HIV, water and sanitation, agricultural development, education, and female empowerment have also faced community resistance." GiveDirectly, Blog post, September 5, 2016 • 93. • "We’ve done a lot to try to understand what has led to this increase in refusals. We’ve found that people typically refuse out of skepticism. Potential recipients find it hard to believe that a new organization like GiveDirectly would give roughly a year’s salary in cash, unconditionally. As a result, many people have created their own narratives to explain the cash, including rumors that the money is associated with cults or devil worship." GiveDirectly, Blog post, September 5, 2016 • "While GiveDirectly is still determining the root causes of this problem, it has developed a few hypotheses. These include the presence of vocal religious or political leaders who might believe they have reason to mobilize the community against the payments (such as politicians who worry the organization or payments may be aligned with their opponents), or a history in the area of hostility to NGOs born either of bad prior experiences or otherwise. Recently, GiveDirectly engaged in productive conversations with some of these stakeholders. The problem might also be exacerbated by the upcoming national elections." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 5. • "GiveDirectly has seen an uptick in the rate of refusal to participate in its cash transfer program in Homa Bay. The root of this development is not clear, and GiveDirectly has not yet identified a solution. In some cases, community members are led by local religious leaders or local government to mistrust the program. In Siaya County, this issue did not arise, possibly because GiveDirectly covered such a large portion of the county that in any new area it entered, people were already aware of the program and knew that it was trustworthy. GiveDirectly has attempted to allay these suspicions by meeting with local government and religious leaders and speaking on local radio shows to explain the program’s purpose. In villages where the refusal rate was high, GiveDirectly put some recipients on an accelerated schedule so they could receive their transfers more quickly and serve as an example to others. After returning to these villages, however, GiveDirectly found no change in public opinion. For this reason, it has not attempted to re-enroll those who refused, but might do so in the future." GiveWell's non-verbatim summary of a conversation with Paul Niehaus, Carolina Toth, and Ian Bassin, February 23, 2016, Pg 5. • 94. • "We’ve debated how much effort to expend trying to address the refusal issue. We considered leaving Homa Bay since working in an area with lower refusals would be better for efficiency targets and staff morale. However, ultimately we decided to stay because we are in the process of implementing a large scale research project in Homa Bay, and leaving the area would result in meaningful delays and cost overruns for our research partners. We also believe it’s important to try to address the issue here or else rumors could spread to other areas we’ve not yet been. So far we’ve tested several different approaches, with varying degrees of success. We’ve significantly expanded our outreach effort by creating a team dedicated to grassroots mobilization. We’ve also gone on the radio several times, enlisted the help of local opinion leaders, leveraged testimonials of past recipients, and much more. In some areas the results were promising. In parts of Nyando subcounty we saw refusal rates dip from 80% to 20% within a two month period. Other areas proved more difficult; in another subcounty (Rachuonyo East) refusal rates have stubbornly hovered around 40% for weeks." GiveDirectly, Blog post, September 5, 2016 • "GiveDirectly created an outreach team to focus on this issue, which has been operating for three months. The team is employing a variety of different tactics: for example, it is communicating with relevant community stakeholders and using media to disseminate recipient testimonial videos and other materials. It will use A/B testing to assess the effectiveness of different strategies. During a period of a couple of months, the refusal rate dropped from an average of 70% to 20-30%, but it has rebounded significantly in the last couple of weeks." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 5. • 95. "High refusal rates primarily impact the productivity of census teams; for example, if 50% of households refuse registration, the team must replace them with new households. In order to continue to reach enrollment targets in Kenya, GiveDirectly has provided additional support by increasing census team sizes and adjusting team members' locations. As lists generated by census teams only include eligible and interested recipients, refusal rates are much lower during subsequent steps. In Kenya, GiveDirectly's enrollment numbers are still roughly 10% above target; this margin is even greater in its Uganda program." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 6. • 96. "Following its Google-funded campaign, GiveDirectly surveyed recipients in detail on how they spent their transfers. Given the limitations of this kind of self-reported data, GiveDirectly has not continued this practice. It prefers to rely on more accurate data gleaned through randomized controlled trials (RCTs), and expects to collect more [of] this type of information in future studies or campaigns, such as its ongoing RCT in coffee farming communities." GiveWell's non-verbatim summary of a conversation with Ian Bassin and Piali Mukhopadhyay, GiveDirectly, August 23, 2016, Pg 4. • 97. Note that in some cases, we have cleaned this data (where it was obvious what the error was in the data entry). These cases have been marked with cell comments. In other cases, when it was not clear what the error was (for example, when an amount of funds was mistakenly placed in a spending category column, and it was not clear whether the amount was intended to be placed in the former or latter column), we deleted the data and commented on the cell to indicate that data had been deleted. Deletions were made no more than 5 times in any given spreadsheet. • 98. • The tables include follow up survey data from the Kenya 2M, Kenya 1.2M, Kenya rolling enrollment, and Kenya behavioral optimization campaigns and from the Uganda pilot campaign. Note that recipients may have been surveyed more than once and would therefore be included more than once in the data presented. • We have seen some spending data for the Ug-201404 campaign (the Uganda 2M campaign), but we have not analyzed it. We have not seen spending data from the Ug-201503 campaign (the Uganda model variations campaign) yet; it is our understanding that GiveDirectly did not collect spending data for that campaign: "Please note that for the Uganda 201503, we no longer collect spending data. I believe Carolina has mentioned this before - we felt the data from our RCT was more reliable on this and we had enough of it to feel confident we understood spending trends without needing to continue to collect it in a form less optimal than in the RCT." Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, August 17, 2016 • 99. In follow up surveys administered in Uganda, recipients were asked about spending on large household items and small household items. The figure reported here is the combination of those two categories. • 100. • 101. GiveDirectly, What We Do - Who We Serve, September 2016 See the chart in the upper left section of the web page. • 102. GiveDirectly, What We Do - Who We Serve, September 2016 See the chart in the upper left section of the web page. • 103. • 104. Note that not every follow-up survey asks about precisely the same issues (e.g. some do not ask about domestic violence or trouble collecting), which is part of what accounts for the different sizes of each group of respondents. • 105. In the Uganda follow up data, this issue is denoted "stole_item." • 106. In the Uganda follow up data, we identified 4 issues that we believe all asked about bribes ("bribe," "pay to collect," "others_bribes," "agent_bribe"). We only include the "bribe" issue in this table for Uganda (which has the highest rate of complaints out of the four). • 107. GiveDirectly sent us its July 2015 - February 2016 financial documents with a different breakdown than we have worked with before, so have not combined the data into one comprehensive summary. • 108. For example, the National Institutes of Health funded Innovations for Poverty Action to conduct the RCT of GiveDirectly's program: "GiveDirectly is conducting a longer-term evaluation to provide more detailed, context-specific evidence on how its recipients use cash transfers. The study is coordinated by an external research organization, Innovations for Poverty Action, led by Dr. Johannes Haushofer of Harvard University, and funded by the National Institutes of Health." GiveDirectly, Offering Memorandum (January 2012), Pg 26. • 109. • 110. GiveWell, GiveDirectly financials - 2016 Sheet: "2016-efficiency ratio." • 111. Includes Core Operations and Core Operations-general. Excludes fundraising. • 112. GiveWell, GiveDirectly financials - 2016 Sheet: "2016-efficiency ratio." • 113. • 114. Note that GiveDirectly estimates that the efficiency of the basic income program will be similar to that of GiveDirectly's standard program: • "We expect efficiency for the UBI program to be modestly worse than our typical program, averaging across the 3 arms to 84% (v. ~90% typically). • Long-term UBI: • Transfers per recipient are larger for the long-term arm (~3K). We expect nominal enrollment costs per adult recipient to be modestly more than for our typical program (but to be a smaller %).
• FX costs maintain the same percentage, and mobile money costs are moderately worse (because smaller withdrawals are charged higher %s).
• We expect to incur roughly the same follow-up costs as our lump sum program for each year of the long-term UBI. As a result, follow-up costs make up a larger proportion of the overall costs of the project than they do typically.
• The short-term and lump sum arms are less efficient than the long-term arm, because they incur equal nominal enrollment costs, and comparable % follow-up, management, and transfer fees, relative to a total transfer amount that is lower.
• We will refine these estimates as we pilot the project in one village this month and finalize our workplan for the full study enrollment."

Ian Bassin, COO, Domestic, GiveDirectly, email to GiveWell, October 8, 2016

• 115.

Ian Bassin, edits to GiveWell's review, November 10, 2016

• 116.

This is also because GiveDirectly wants to structure payments such that the experiment is as informative as possible for its partners. GiveDirectly believes this strategy is most likely to lead to policy impact. Paul Niehaus, Carolina Toth, and Ian bassin, conversation with GiveWell, August 12, 2016

• 117.
• 118.
• 119.

GiveDirectly, Contextualizing transfer size

• 120.
• $0.65 in pre-cash-transfer consumption per person per day implies (365*$0.65) = $237.25 per person per year. If each person receives$288 in a year from GiveDirectly, that's (288/237.25) = 121%.
• Note that recipients in Uganda live on $0.83, according to GiveDirectly's website (we believe this is referring to consumption because GiveDirectly's comparable figure for Kenya is$0.65). GiveDirectly, What We Do - Operating Model, see the Uganda tab. So the calculation in Uganda would be (365*$0.83)=$302.95. ($288/$302.95)=95%.
• Note that GiveDirectly has told us that recipients in Homa Bay have a slightly lower average daily consumption of $0.50. GiveDirectly, Consumption data for targeting work So, the calculation in Homa Bay would be (365*$0.50)=$182.50. ($288/$182.50)=158%. • 121. Paul Niehaus, GiveDirectly Founder, email to GiveWell, November 20, 2012. We have not reviewed the data GiveDirectly used to reach this conclusion. • 122. In our conversations with recipients and field staff, we phrased this question two different ways: • Do you think it would be better for GiveDirectly to provide$1000 transfers to households in one village or $500 transfers to households in two villages? • Do you think that GiveDirectly should keep the transfer size the same or reduce the transfer size but provide transfers to a greater number of people? • 123. GiveWell site visit to GiveDirectly, October 2014 • 124. We asked the field officers what they think about the current transfer size ($1000), and whether they’d choose to keep it at that level, increase it, or decrease it, given the effects that an adjustment would have on how many people GiveDirectly would be able to serve.

• Mr. Okello: Typically there are multiple households on one compound, each inhabited by relatives of the same family, and any household that meets the targeting criteria can receive transfers. Mr. Okello said that it may make more sense for GiveDirectly to group some households on a compound together so that transfers are shared across them, rather than each eligible household receiving the full $1000. Mr. Okello also said that if GiveDirectly increased the size of the transfers, that could create a high level of dependency. One of the messages that field officers send is that people should use the$1000 transfers to develop themselves as much as possible, but if someone knew they were getting $2000, they may stop farming, for example. With$1000 people can get some things but not everything; it is the right amount.

• Mr. Ekeu: Mr. Ekeu prefers reducing the amount of money in each transfer and expanding the recipient base to reach everyone in the village. He said that the current targeting model causes bragging and unrest in the communities. The people who don’t benefit may be brought to use force to get some of the money, such as by breaking into recipients’ homes. Mr. Ekeu suggested that it would be better for GiveDirectly to provide all households in a village with some amount of money, even if it was less for households that are currently deemed ineligible (e.g., $100). This way, each of the households would be busy figuring out how they would spend their own money rather than how to get money from another. • Mr. Olinga: Mr. Olinga said that to reduce extreme poverty the bigger transfer is better, but he didn't have a strong opinion on$1000 transfers to some people versus \$500 transfers to twice as many. [This is how we posed the question to Mr. Olinga.]
• 125.

Conversation with GiveDirectly, September 5, 2014

• 126.

Conversation with GiveDirectly, September 5, 2014

• 127.

Conversation with GiveDirectly, September 5, 2014