J-PAL and IRD — Incentives for Immunization Studies

[Added December 19, 2016: GiveWell's experimental work is now known as GiveWell Incubation Grants.]

Note: Abdul Latif Jameel Poverty Action Lab (J-PAL) staff reviewed this page prior to publication.

Published: October 2015

We published an update on the cost-effectiveness of this intervention in August 2016. We spoke with Dr. Subhash Chandir of IRD about updates on this grant on July 12, 2018.

Background

Initial incentives for immunization study

From 2004 through 2007, the Abdul Latif Jameel Poverty Action Lab (J-PAL) ran a randomized controlled trial (RCT) in Udaipur, India to test whether providing reliable immunization camps, in which a mobile team administers vaccinations on a fixed date each month, and small non-cash incentives for parents who vaccinate their children would increase the rate of children who received a full course of immunization.1 The study found that providing reliable immunization camps alone (i.e., without incentives) increased the rate of children who were fully vaccinated from 6% (control group) to 17%. Offering small incentives in addition to the reliable supply of vaccinations increased the rate of fully vaccinated children to 38%.2

Due to these promising results, J-PAL and Interactive Research and Development (IRD), a Pakistani non-governmental organization, are now planning to conduct two replication studies of the immunization incentive concept in Haryana, India and Pakistan.

Haryana replication study background

In Haryana, the replication study will be led by J-PAL South Asia at the Institute for Financial Management and Research. The proposed Haryana study is an RCT designed to study the effect of text message (SMS) reminders and non-cash incentives on immunization rates.3

The primary outcome of interest for the study will be the increase in fully immunized children (i.e., the difference in the change in fully immunized children in the treatment vs. control group).4 The study will also measure the total number of immunizations that children in the treatment group receive relative to children in the control group (see, e.g., Figure 1 in J-PAL Policy Briefcase November 2011), and it will provide a detailed breakdown of which vaccine doses each group received (e.g., it will collect data on how many pentavalent vaccine doses children received in the treatment vs. control group).5 The study will also serve as a pilot to test the scalability of immunization incentives programs in Haryana and other Indian states.6

For more details on this study, see J-PAL proposal for the Haryana replication RCT.

Pakistan replication study background

In Pakistan, J-PAL-affiliated researchers are working with researchers and staff at IRD to evaluate a variant of the incentives program. In this variant, each time that a parent brings his or her child to receive a vaccination at the appropriate time, he or she will be entered into a lottery to win vouchers that are redeemable at a local grocery store. The vouchers will be delivered to winners via mobile phone.7

This RCT will test the efficacy of small incentives on immunization, and the relative effectiveness and cost-effectiveness of different types of incentives structures, as well as the efficacy of SMS reminders alone.8 The main outcome of interest for the study will be the increase in fully immunized children (i.e., the difference in the change in fully immunized children in the treatment vs. control group).9 The study will also measure the total number of immunizations that children in the treatment group receive relative to children in the control group (see, e.g., Figure 1 in J-PAL Policy Briefcase November 2011), and it will provide a detailed breakdown of which vaccine doses each group received (e.g., it will collect data on how many pentavalent vaccine doses children received in the treatment vs. control group).10

For more details on this study, see J-PAL/IRD Pakistan study design brief (first two sections).

Budgets

Haryana study budget

USAID has granted $1,260,000 to support the Haryana replication study.11 J-PAL projects a remaining funding gap of $506,200 for the study, which includes the following:12

• $223,000: Incentives implementation monitoring
• $101,400: Census for endline survey
• $94,300: Text messaging intervention
• $87,500: Endline Survey

Good Ventures’ gift of $100,000 to this study is in the form of general support; we believe it will support some portion of these activities.13

Pakistan study budget

IRD has estimated a total budget of $986,595 for the Pakistan study.14

Good Ventures’ gift of $100,000 to the study is in the form of general support; we believe it will support the implementation and evaluation of the intervention.

IRD is planning to seek funding from the Global Innovation Fund and the International Initiative for Impact Evaluation (3ie) to cover the remaining costs of the study.15

Transparency

J-PAL and IRD are pre-registering both RCTs, and plan to share the final study data and statistical analysis code publicly after the studies are published.16

Rationale for the gifts

Good Ventures is making two gifts of $100,000 each to support the incentives for immunization replication studies in Haryana and Pakistan.

Our primary reasons for recommending these gifts are as follows:

• These gifts seem to be promising opportunities to advance our experimental work to “seed” potential new top charities. As high-quality replications of a potentially cost-effective intervention, these studies will help us to determine whether incentives for immunization should be one of our priority programs. If incentives for immunization becomes a priority program, we would begin searching for charities that implement this program to determine whether they should become top charities.
• We consider building the evidence base of promising global health and development interventions to be an important project. We want to improve incentives for evaluators and implementers to replicate and scale up evidence-based programs. Providing funding for these replication studies is a way to support these goals.
• J-PAL is a well-regarded research group, and its interest in immunization incentives carries some weight in our decision-making.

Major remaining questions

Cost-effectiveness

We published an update on the cost-effectiveness of this intervention in August 2016.

At our request, J-PAL and IRD created preliminary cost-effectiveness analyses (CEAs) for the interventions in both Haryana and Pakistan. For Haryana, J-PAL estimated the cost-effectiveness of the program would be between $200 per death averted and $2,500 per death averted.17 For Pakistan, IRD's estimates ranged from about $1,400 to about $5,200 per death averted.18

However, we are uncertain about the cost-effectiveness of these interventions because the CEAs seem to be highly sensitive to a few core assumptions:

• Sources for mortality rates. There are several possible sources for the cause-specific mortality rates used in the CEAs.19 Under the scenario of the CEA that we analysed most closely, by varying only the source used for the mortality rates, the cost per death averted in Haryana can range from about $700 to about $34,000.20 We do not know which data source is most accurate, and it would likely take a substantial amount of research capacity to learn more about the strengths and weaknesses of the available sources.
• Effects of marginal vaccinations. The cost-effectiveness of the incentives for immunization programs seems to depend on which part of the immunization course the intervention causes children to receive. Providing additional immunizations may have larger effects on completely unimmunized children than on partially immunized children.21 The mortality reduction rates in the J-PAL CEA assume that the children who become fully immunized as a result of the program would have been completely unimmunized in the absence of the program.22 We have not done enough research to understand the most accurate way to model this factor.23
• The timing of deaths vs. the timing of the immunization schedule. Children may not receive immunizations in time for the immunity to prevent death. For example, tetanus deaths can occur in the first month of life, but the first dose of DPT (diphtheria, pertussis, and tetanus) vaccine typically isn’t given until children are 6 weeks old in Pakistan.24 Therefore, it may not be straightforward to combine current mortality rates from DPT with the measured protective effects of DPT vaccines to estimate the deaths averted by DPT vaccines as J-PAL told us is commonly done in the literature.25 We have not thought carefully about how to adjust the CEA to account for this factor.

If we were to continue investigating the cost-effectiveness of these interventions, we would attempt to learn more about the above factors and adjust our CEA accordingly. Due to constraints on our capacity, we have deprioritized further investigation into these questions.

Path to a top charity

The end goal of our experimental work is to develop promising candidates for our top charity recommendations. We have not fully considered the path from immunization incentives being a cost-effective, evidenced-backed intervention to being implemented by a potential top charity. Relevant considerations include:

• Scalability: Is there an organization that could use additional funds to effectively implement an immunization incentives program at a large scale?
• External validity: Would an incentives-for-immunization program that is effective in India and Pakistan be effective in other contexts where charities may try to implement it (e.g., African countries)? What would we need to know to evaluate the program's likely effectiveness in other contexts?
• Changing behavior: Will program recipients respond differently to the incentives after the immunization incentives program exists in an area for a long period of time?

In addition to the above, there are likely to be many additional considerations that could affect whether it would be reasonable to recommend a charity implementing this program that are currently unknown to us.

Plan for follow-up

The Pakistan study is projected to run in total over a period of roughly 36 months. The Haryana study interventions are planned to begin by the end of the year and run for 12 months.26 We do not know when we should expect results from these studies to be available.

We plan to periodically (1-2 times a year) check in with J-PAL staff about the status of the studies and produce public conversation notes if warranted.

GiveWell may do further analysis of the studies when the results become available.

Our process

In late 2014, we had conversations with J-PAL about the possibility of supporting the Haryana and Pakistan studies. In the ensuing months, we corresponded further with J-PAL and IRD about the details of the studies and began investigating the cost-effectiveness of the interventions. After some work on the cost-effectiveness estimates, we recognized that answering our outstanding questions would require a substantial time commitment without necessarily moving us to a clear decision point. This being the case, we decided to suspend further investigation and partially fund the studies.

Sources

Document	Source
Banerjee et al 2010	Source (archive)
Childhood Immunization in Pakistan Policy Brief	Source (archive)
Conversation with John Floretta, Deputy Director of J-PAL South Asia, on May 14, 2015	Unpublished
Disease Control Priorities Project incidence rates	Source (archive)
Email from J-PAL and IRD to GiveWell, August 6, 2015	Unpublished
Email from John Floretta, Deputy Director of J-PAL South Asia, to GiveWell, May 5, 2015	Unpublished
Email from Rachel Glennerster, Executive Director of J-PAL, to GiveWell, March 31, 2015	Unpublished
GiveWell back-of-the-envelope cost-effectiveness estimate	Source
GiveWell's non-verbatim summary of a conversation with John Floretta, Deputy Director of J-PAL South Asia, on December 22, 2014	Source
Global Burden of Disease comparative tool	Source (archive)
IRD deaths averted calculation (Pakistan) (first three worksheets)	Source
J-PAL/IRD Pakistan study design brief (first two sections)	Source
J-PAL Policy Briefcase November 2011	Source (archive)
J-PAL proposal for the Haryana replication RCT	Source
J-PAL Haryana study USAID application	Unpublished
J-PAL responses to GiveWell questions about the Haryana project (first response)	Source
J-PAL summary of the immunization incentives RCT in Udaipur	Source (archive)
Million Death Study	Source (archive)
Tetanus deaths in the GBD tool	Source (archive)
UNICEF, “Immunisation and Pentavalent Vaccine”	Source (archive)
World Health Organization incidence rates	Source (archive)

• 1
  • This RCT tested two interventions. Intervention A improved immunization infrastructure by setting up reliable immunization camps and advertising their availability to villages:

    "Intervention A: Seva Mandir (a local NGO) hired a mobile immunization team including an ANM [Auxiliary Nurse Midwife] and assistant to conduct monthly immunization camps in villages. The camps were held from 11am - 2pm on a fixed date of the month and the presence of the ANM was verified by timed and dated photographs of them in the villages, as well as regular monitoring. Records indicate that 95 percent of planned camps took place, and were not disrupted by provider absence. A Seva Mandir social worker who lived in each village informed mothers of immunization camp availability and educated them on the benefits of immunization." @J-PAL summary of the Immunization Incentives RCT in Udaipur@

  • Intervention B included the immunization infrastructure of intervention A. In addition, parents were offered small incentives (1 kg lentils) for each immunization administered to their children, and a set of metal plates when the course of immunizations was completed:

    "Intervention B: Using the same immunization camp infrastructure as intervention A, intervention B also offered parents 1 kg of lentils per immunization administered, and a set of thalis (metal meal plates) upon completion of a child’s full immunization course. The value of the lentils was about Rs. 40 (less than one dollar), equivalent to three quarters of one day’s wage. The incentives were provided as an agent to help offset the opportunity cost of taking a child to be vaccinated." @J-PAL summary of the Immunization Incentives RCT in Udaipur@

  • The full course of immunization was 1 dose of BCG vaccine (prevents tuberculosis), 1 dose of measles vaccine, 3 doses of the oral polio vaccine, and 3 doses of the DPT vaccine (prevents diphtheria, pertussis, and tetanus):

    "In this study children received the WHO/Unicef extended package of immunisation, provided by the Indian government. This includes one dose of BCG vaccine, three doses of DPT (diphtheria-pertussis, tetanus) vaccine, three doses of oral polio vaccine, and one dose of measles vaccine." Banerjee et al 2010, Pg. 2
    

• 2

  “Among children aged 1-3 years, 38.3 percent were fully immunized in intervention B villages, compared to 16.6 percent in intervention A villages, and 6.2 percent in control villages.” @J-PAL summary of the Immunization Incentives RCT in Udaipur@

• 3
  • "The study will use a randomized controlled trial (RCT) design to evaluate the effectiveness of incentives and SMS reminders in increasing full routine immunization coverage in the Indian state of Haryana. The evaluation will test the scalability of this approach through the government." J-PAL proposal for the Haryana replication RCT, Pg. 1
  • "The study will cover a total of 140 Primary Health Centres (PHCs) across six or seven low performing districts of the state. These 140 PHCs will be randomly divided into two groups. One group of 70 PHCs, the treatment group, will receive non-cash incentives to distribute with the regular supply of vaccines at the immunization camps … Impact will be measured by comparing full immunization rates to a second group of 70 PHCs which will not run the incentives program." J-PAL proposal for the Haryana replication RCT, Pg. 1
  • "From the villages covered by each of the 140 PHCs, 7 villages per PHC would be randomly sampled, calculating to a total sample of 980 villages. Within each of the 980 villages, 15 eligible households i.e. households with children between 0-36 months would be sampled, calculating to a total sample size of 14,700 households." J-PAL proposal for the Haryana replication RCT, Pg. 1
  • "A cross-randomized experiment will test the effectiveness at scale of text messages to mothers reminding them to immunize their children … This experiment will be carried out in a sub-sample of 300 villages across the six study districts." J-PAL proposal for the Haryana replication RCT, Pg. 2

• 4
  • "This proposal, a joint project of the government of Haryana and J-PAL South Asia, aims to increase full immunization rates in these seven districts by providing parents with small incentives at each of the five vaccination sessions required to fully immunize their children." J-PAL Haryana study USAID application, Pg. 2. (note: J-PAL has requested that this source be kept private, except for specific quotes)
  • "Full immunization" in this study refers to receiving 1 dose of BCG vaccine (prevents tuberculosis), 3 doses of pentavalent vaccine (“pentavalent vaccine is a combination of five vaccines in one: diphtheria, tetanus, whooping cough, hepatitis B and Haemophilus influenza type b (the bacteria that causes meningitis, pneumonia and otitis,” UNICEF, “Immunisation and Pentavalent Vaccine”), and 1 dose of measles vaccine. Email from J-PAL and IRD to GiveWell, August 6, 2015
  • Explanation of the primary outcome of the study also comes from Email from J-PAL and IRD to GiveWell, August 6, 2015

• 5

  Email from J-PAL and IRD to GiveWell, August 6, 2015

• 6
  • "The largest anticipated impact, however, will be the project’s role in developing hard evidence to building a case for a scale up of a demand-focused immunization intervention across Haryana and India. This project is viewed by the Government of Haryana as a pilot to inform the potential scale-up of the scheme throughout the state." J-PAL Haryana study USAID application, Pg. 3 (note: J-PAL has requested that this source be kept private, except for specific quotes)
  • The Haryana replication RCT will have a substantially larger sample (14,700 households with children aged 0-3) than the initial RCT in Udaipur (1,640 children aged 1-3 at endpoint):
    • "From the villages covered by each of the 140 PHCs, 7 villages per PHC would be randomly sampled, calculating to a total sample of 980 villages. Within each of the 980 villages, 15 eligible households i.e. households with children between 0-36 months would be sampled, calculating to a total sample size of 14,700 households." J-PAL proposal for the Haryana replication RCT, Pg. 1
    • "Participants 1640 children aged 1-3 at end point" Banerjee et al 2010, Pg.1
  • J-PAL is cooperating with the Haryana state government for this RCT. Promising results could result in a statewide scale-up of the program:
    "In Haryana, we will be running the incentives program in approximately one-sixth of the Primary Health Centres (PHCs) in the state. In addition to strong support for the program from the program from the State Immunization Officer (who will join the study as a co-Principal Investigator) and Director of the state’s National Health Mission department, we also now briefed the Additional Chief Secretary (Health), the state’s top health official, on the project. In our meeting he stated that if the project is successful in increasing full immunization rates, he will push for scale up to the remainder of the state. This would reach approximately 400,000 children each year. We believe scale-up throughout Haryana is likely if the program is effective." J-PAL responses to GiveWell questions about the Haryana project (first response), Pg. 2-3 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)

• 7

  "Interactive Research and Development (IRD) in Pakistan has piloted an innovative variant to this project, delivering the incentive via a mobile lottery. Each time an infant makes a successful (timely) vaccination visit, the mother is presented with a lottery that entitles her to a shopping coupon at a local grocery store if she wins." J-PAL/IRD Pakistan study design brief (first two sections), Pg. 2

• 8
  • "The experiment will be a randomized controlled trial with a cross-cutting design."J-PAL/IRD Pakistan study design brief (first two sections), Pg. 4 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)
  • "It should be noted that in addition to the levels and schedule of payouts in experiment (1), we will also include an arm that provides only SMS reminders to parents to bring their child in for a timely vaccination." J-PAL/IRD Pakistan study design brief (first two sections), Pg. 4 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)
  • Email from J-PAL and IRD to GiveWell, August 6, 2015

• 9
  • "We will include a quality control component that allows us to independently check the core outcome of the project (child immunization)." J-PAL/IRD Pakistan study design brief (first two sections), Pg. 9. (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)
  • “Child immunization” in this study refers to receiving one dose of BCG vaccine (prevents tuberculosis), three doses each of oral or injectable polio vaccine (OPV/IPV), three doses of pentavalent vaccine (“pentavalent vaccine is a combination of five vaccines in one: diphtheria, tetanus, whooping cough, hepatitis B and Haemophilus influenza type b (the bacteria that causes meningitis, pneumonia and otitis.” UNICEF, “Immunisation and Pentavalent Vaccine”), and two doses of measles vaccine. Email from J-PAL and IRD to GiveWell, August 6, 2015
  • Explanation of the primary outcome of the study also comes from Email from J-PAL and IRD to GiveWell, August 6, 2015
  • "The main data on immunization coverage will be administrative data that is directly gathered by IRD’s monitoring system. This will include immunization data gathered from cards with RFID chips associated with children being immunized, scanned fingerprints of guardians that bring children in for vaccination and data on transfers/lottery outcomes from mobile server data. The reliance on finger prints to verify the identity of guardians means there will be high integrity of administrative data." J-PAL/IRD Pakistan study design brief (first two sections), Pg. 8 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)

• 10Email from J-PAL and IRD to GiveWell, August 6, 2015
• 11

  "USAID has approved $1.26 million for J-PAL’s immunization incentives study in Haryana (a state in India)." GiveWell's non-verbatim summary of a conversation with John Floretta, Deputy Director of J-PAL South Asia, on December 22, 2014, Pg. 4

• 12

  J-PAL responses to GiveWell questions about the Haryana project (first response), Pg. 3 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)
  

• 13

  J-PAL has told us that Good Ventures' gift will support "enhanced monitoring structures," Email from John Floretta, Deputy Director of J-PAL South Asia, to GiveWell, May 5, 2015

• 14

  Email from J-PAL and IRD to GiveWell, August 6, 2015

• 15

  Email from Rachel Glennerster, Executive Director of J-PAL, to GiveWell, March 31, 2015

• 16

  Conversation with John Floretta, Deputy Director of J-PAL South Asia, on May 14, 2015

• 17

  For J-PAL's Haryana CEA estimates, see J-PAL responses to GiveWell questions about the Haryana project (first response), Pg. 2

• 18

  For IRD's Pakistan CEA estimates, see the "Deaths Averted Cost Scenarios" worksheet in IRD deaths averted calculation (Pakistan) (first three worksheets). The range of $1,400 to $5,200 per death averted is for the "High Incentive" variant of the program.

• 19

  Potential sources include: the Institute of Health Metrics and Evaluation's Global Burden of Disease comparative tool, the World Health Organization incidence rates, the Disease Control Priorities Project incidence rates, and mortality estimates from the Million Death Study.

• 20
  • GiveWell back-of-the-envelope cost-effectiveness estimate compares a replication of J-PAL's Scenario 1 ($729 per death averted, see J-PAL responses to GiveWell questions about the Haryana project (first response), Pg. 2) to an estimate that uses a similar structure, but with inputs from the Global Burden of Disease comparative tool.
  • J-PAL told us that its cost-effectiveness analyses followed the norms used in medical literature to handle the variance in cause-specific mortality rates. We have not tried to independently verify that J-PAL's approach followed norms in medical literature. Email from J-PAL and IRD to GiveWell, August 6, 2015

• 21

  For example, children may experience larger benefits from receiving their first dose of DPT vaccine (diphtheria, pertussis, and tetanus) than their third.

• 22

  J-PAL told us (in Email from J-PAL and IRD to GiveWell, August 6, 2015) that since there is limited data on case fatality rates under partial coverage, the mortality reduction rates in the J-PAL CEA assume that the children who receive, e.g., a third dose of DPT (diphtheria, pertussis, and tetanus) as a result of the program did not receive the first two doses of DPT. However, it seems like it may be reasonable to try to model the alternative assumption that marginal doses of the same immunization have somewhat diminishing returns.

• 23

  There is some information about which immunizations children received as a result of the initial incentives for immunization program in Table 2 and Figure 3 in Banerjee et al 2010, on Pg. 6. We have not analyzed this information closely.

• 24
  • See Childhood Immunization in Pakistan Policy Brief, table on Pg. 1 for the DPT vaccine schedule. (Note that DPT is part of the “pentavalent” vaccine as well.)
  • According to the Global Burden of Disease comparative tool, most tetanus deaths occur within the first 28 days of life (see Tetanus deaths in the GBD tool). Vaccines given at 6 weeks would not prevent these deaths.
  • We have not closely analysed how the timing of mortality aligns with the immunization schedules for all the diseases being treated by these interventions.

• 25

  Email from J-PAL and IRD to GiveWell, August 6, 2015

• 26
  • "The project time frame is currently projected to be 36 months. This will include an initial 3 month period for developing the software platform which will manage vaccination records and payouts. After this, we will need a 6 month period to recruit new areas (i.e. service delivery points and parents who use those) of Karachi into the program. The remaining time will be used to roll out the incentivized immunization program." J-PAL/IRD Pakistan study design brief (first two sections), Pg. 9 (note: J-PAL has requested that the portion of the document cited here be kept private, except for specific quotes)
  • "Before and after the 12 month period of program implementation, surveys of approximately 15,000 households, randomly chosen to be representative of the catchment areas under each PHC, will be conducted in all program areas to assess full immunization rates." J-PAL Haryana study USAID application, Pg. 3 (note: J-PAL has requested that this source be kept private, except for specific quotes)
  • Email from J-PAL and IRD to GiveWell, August 6, 2015