Development Media International - November 2015 Version

We have published a more recent review of this organization. See our most recent report on Development Media International.

Donate

Development Media International (DMI) does not meet all of our criteria to be a GiveWell top charity but is a standout charity. Although we don't recommend these organizations as strongly as we do our top charities, they stand out from the vast majority of organizations we have considered.

More information: What is our evaluation process?

A note on this page's content

In November 2015, Development Media International (DMI) privately shared preliminary endline results from a randomized controlled trial (RCT) of its program with us. The large majority of this page was written before DMI sent us the preliminary endline results, and thus reflects our view of DMI prior to incorporating the endline results. We have placed notes throughout the review as a reminder that the current content on this page does not necessarily reflect our up-to-date view.

The preliminary endline results did not find any effect of DMI's program on child mortality (it was powered to detect a reduction of 15% or more), and it found substantially less effect on behavior change than was found at midline. We cannot publicly discuss the details of the endline results we have seen, because they are not yet finalised and because the finalised results will be embargoed prior to publication, but we have informally incorporated the results into our view of DMI's program effectiveness.

DMI believes that there were serious problems with endline data collection (note that we have not yet tried to independently assess this claim).

DMI has subsequently shared results with us that showed, based on health facility data (collected separately from the endline survey), an increase in facility visits in the treatment group during the intervention.

DMI is a standout giving opportunity because of its strong transparency, rigorous self-evaluation, and its work on a potentially cost-effective program.

Published: November 2015

Since publishing this review, we have published notes from conversations with DMI in October 2020, July 2020, April 2020, March 2020 (here and here), January 2020, May 2019, June 2018, October 2017, October 2016, and March 2016. In addition, DMI shared the following documents with us in 2016:

Summary

What do they do? Development Media International (DMI, developmentmedia.net) produces radio and television programming in developing countries that encourage people to adopt improved health practices, such as exclusive breastfeeding of infants and seeking treatment for symptoms associated with fatal diseases. The program aims to reduce mortality among children less than five years old. (More)

Does it work? From 2011 to 2015, DMI conducted a randomized controlled trial (RCT) of its program in Burkina Faso to measure impact on behavior change and child mortality. At midline, DMI found some positive effects on key self-reported health behaviors that suggested DMI’s program may cause a substantial reduction in child mortality.

In June 2015, DMI told us that it no longer expected to find a statistically significant effect on child mortality at endline because overall child mortality declined extremely rapidly in Burkina Faso over the course of the RCT, making it more difficult to distinguish the effect of DMI’s program.

In November 2015, DMI privately shared preliminary RCT endline results with us. The preliminary endline results did not find any effect of DMI's program on child mortality, and it found substantially less effect on behavior change than was found at midline. DMI believes that there were serious problems with endline data collection (note that we have not yet tried to independently assess this claim), and with the support of the trial’s Independent Scientific Advisory Committee, is planning to conduct another endline survey in late 2016 (with results available in 2017). We cannot publicly discuss the details of the endline results we have seen, because they are not yet finalised and because the finalised results will be embargoed prior to publication, but we have informally incorporated the results into our view of DMI's program effectiveness. We expect some further information about the RCT endline results to be available in 2016. We discuss the effectiveness of DMI's program in greater detail below. (More)

What do you get for your dollar? Taking into account the preliminary endline results we have seen, we do not have strong evidence to suggest that DMI's program cost-effectiveness is within the range of our other priority programs. (More)

Is there room for more funding? In September 2015, DMI estimated that over the next two years, it could productively use approximately $12 million in unrestricted funds to scale up and launch more campaigns to reduce under-5 mortality. In November 2015, DMI provided an update that it plans to use some of its existing funding to conduct a second endline survey for its RCT because it believes that serious data collection issues occurred during the first endline survey. DMI estimates that this survey will cost in the region of $200,000 to $300,000. (More)

DMI is one of our standout charities because of its:

  • Unusually strong self-analysis – particularly in supporting a randomized-controlled trial (RCT) on its program.
  • Standout transparency – it has shared significant, detailed information about its program with us.
  • Work on a potentially cost-effective program – conceptually, mass media interventions have the potential to be highly cost-effective, though we have not seen strong evidence that they are cost-effective in practice.

Major unresolved issues include:

  • The main evidence underlying DMI’s program is from a single RCT’s results that found some positive effects on key self-reported health behaviors at midline but that preliminarily find no effect of DMI's program on child mortality and substantially less effect on behavior change at endline. We cannot publicly discuss the details of the endline results we have seen, because they are not yet finalised and because the finalised results will be embargoed prior to publication, but we have informally incorporated the results into our view of DMI's program effectiveness. Other programs we recommend are generally supported by multiple strong sources of evidence.
  • Even if we assume DMI’s program is saving a large number of lives, we are unsure how the efficacy and costs of the program will compare in new countries where the cultural barriers to behavior change may be very different, and where DMI will not be spending as much time and money on each radio station from which it broadcasts as it did in its RCT.

Table of Contents

Our review process

To date, our review process has consisted of:

  • Conversations with DMI CEO Roy Head, Director of Development Will Snell, Research Manager Jo Murray, and Public Engagement and Innovation Manager Cathryn Wood.1
  • A site visit (photos) to DMI’s operations in Burkina Faso, including:
    • Extensive conversations with Country Director Matthew Lavoie, Radio Executive Producer and Trainer Pieter Remes, and other staff.2
    • A visit to one of DMI’s seven partner radio stations to watch preparation and live broadcast of DMI programming and speak with station staff.3
    • Meetings with national government officials that have supported DMI’s campaign in the Ministry of Health and the Ministry of Communication.4
    • Meetings with representatives from Micronutrient Initiative and the Initiatives Conseil International.5
    • Interviews with professionals in the rural health system.6
  • Two conversations with the lead researcher conducting the trial to independently evaluate DMI’s program, Professor Simon Cousens, Epidemiology and Medical Statistics, London School of Hygiene & Tropical Medicine (LSHTM).7
  • Conversations with independent actors with perspective on DMI or possible challenges to launching campaigns in new countries.8
  • Reviewing documents DMI sent in response to our queries, including details of the midline results from its ongoing randomized controlled trial.

Previous versions of this page available here.

What do they do?

DMI produces and broadcasts radio and television programming that encourages improved health practices in order to reduce maternal and child mortality.9

DMI currently works in Burkina Faso, the Democratic Republic of the Congo (DRC), and Mozambique. Our review focuses on DMI's “child survival” campaigns (campaigns primarily aimed at reducing child mortality) because (a) we believe that any GiveWell-influenced donations to DMI are most likely to lead to scaling up child survival campaigns (more in the “Is there room for more funding?” section below), and (b) these are the campaigns for which we have seen the most evidence of effectiveness.

Table 1a summarizes the campaigns we focus on in this review. Table 1b summarizes most of DMI’s other recent and planned activities, which we do not focus on in this review; these activities have been supported by restricted funds in the past, and we expect them to only be supported by restricted funds in the future, so they are substantially less relevant to GiveWell-influenced donors (more below).10

Table 1a: DMI campaigns focused on in this review
Country Campaign Timeframe Stage Approximate
cost/year11
Burkina Faso Child survival radio campaign and RCT 2011 to 2015 Completed $2,328,67412
Burkina Faso Child survival radio campaign13 2015 to 201714 Ongoing $1,928,00015
DRC Child survival radio campaign 2015 Ongoing $1,472,27916
DRC Child survival radio campaign (second phase) 201617 Planned $850,00018
Mozambique Child survival, family planning, and maternal health radio and TV campaign19 2016 to 201920 Planned $2,250,00021

Table 1b: Other DMI activities, not focused on in this review
Country Campaign Timeframe Stage Approximate
cost/year22
Burkina Faso Viral videos mobile health pilot23 2014-2015 Completed $49,00024
DRC Family planning radio and TV campaign in Kinshasa25 2015-2016 Ongoing $468,51626
Burkina Faso Family planning RCT27 2015-2019 Planned $1,225,00028
Tanzania Nutrition and stunting campaign29 2015-2020 Planned $991,40030
Tanzania Early childhood development campaign31 2016-2018 Planned $899,00032

DMI estimates its total expenditure for its 2015-2016 fiscal year to be at least $3,932,704.33 This figure does not include funds allocated to the upcoming family planning RCT in Burkina Faso or the campaigns in Tanzania. In the 2014 calendar year, DMI's total expenditure was approximately $2,328,674.34

Where do they work?

Burkina Faso

From March 2012 to January 2015, DMI broadcast messages from seven radio stations in Burkina Faso with the aim of reducing the child mortality rate in the broadcast area. The project was evaluated by an RCT (more below).35

Starting in May 2015, DMI scaled up its child survival campaign to 29 stations in Burkina Faso. DMI plans to continue broadcasting through at least 2017.36

Concurrent with the scale-up of its child survival campaign, DMI is conducting a nutrition campaign in the Sahel region of Burkina Faso. Additionally, DMI plans to begin a family planning campaign in 2016, which will be evaluated by an RCT.37 We have not evaluated either of these programs.

Democratic Republic of the Congo

In 2015, DMI conducted a child survival radio campaign in the DRC, broadcasting on 35 stations in eight provinces. DMI plans to expand this campaign in 2016 if it is able to raise funding for the project.38

DMI also conducted a radio and television family planning campaign in Kinshasa in 2015.39 We have not evaluated this campaign.

Mozambique

DMI plans to launch a child survival, family planning, and maternal health radio and television campaign in Mozambique in 2016.40 DMI's Mozambique country representative is currently doing preparatory work for this campaign.41

What health practices do they encourage?

DMI says that it chooses which health topics to focus on based on its predictions of which behaviors are likely to save the most lives per broadcast in a given country.42

DMI's messaging during its child survival campaign in Burkina Faso has focused on:43

  • Getting routine care during pregnancy and delivering in a health facility.
  • Seeking health care for young children who manifest symptoms including fever, cough, difficulty breathing, and diarrhea with blood.
  • Treating diarrhea with oral rehydration salts (ORS) and increasing intake of liquids and foods.
  • Savings during pregnancy.
  • Initiating breastfeeding within an hour after birth.
  • Exclusive breastfeeding up to 6 months after birth.
  • Delaying the first bath and promoting skin-to-skin contact for low birthweight babies.
  • Complementary feeding during the transition from exclusive breastfeeding to family foods.
  • Appropriate handwashing with soap.
  • Using bednets for children and pregnant women.
  • Using latrines and safely disposing of children's stools.
  • Participating in national vitamin A campaigns.

In the DRC, DMI's messaging has focused on child survival topics similar to those of the Burkina Faso campaign. The DRC campaign also contains some messaging on family planning topics, as well as some messaging on health topics like registering births and drinking water purification.44

The upcoming child survival campaign in Mozambique is focusing on messages similar to those broadcast during the child survival RCT. The Mozambique campaign will also include messaging on family planning and maternal health.45

What is the format of DMI’s broadcasts?

In Burkina Faso and the DRC, DMI uses 60-second recorded spots for its radio broadcasts. In Mozambique, DMI plans to produce radio spots as well as content for television.46

The 60-second radio spots are acted scenarios advertising a specific health behavior message such as, "Start breastfeeding newborns within an hour after birth." Stations broadcast the same spot at least ten times per day for a week before DMI switches to a new spot.47

Here is DMI’s English translation of a spot encouraging mothers to feed their babies colostrum (first breastmilk):48

Grandmother: My grandson is really handsome!

Baby: Who are you? Why are you surrounding me?

Illnesses: We’re a group of illnesses, and we’re going to make you thin and weak and stop you growing.

Baby: No, mummy and grandma won’t let you.

Illnesses: Hah! They can’t see us. And they can’t understand baby language! Your grandma is going to throw away your mummy’s colostrum because she doesn’t know that colostrum is our most powerful enemy. It’s the best protection against us, but she’s going to throw it away!

Grandmother: Daughter, your baby is hungry. Give him your breast so he can drink your colostrum. It contains everything he needs to grow up healthy and strong.

Baby: You see! Grandma’s wise and knows what’s best for me.

Illnesses: Noooo! The baby’s drinking colostrum! We’re defeated!

Narrator: Give your baby colostrum in the first hour after birth so that they’ll grow up fit and strong.

For its national child survival campaign in Burkina Faso, DMI has phased out the 10- to 15-minute-long live drama modules that it produced during the child survival RCT. DMI believes the live drama modules to be less cost-effective than the spots, and phased them out of child survival campaigns to reduce campaign costs.49

In Burkina Faso, DMI is also piloting a viral video project. Short videos, produced in the local language and containing a health message relevant to child health, are distributed and shared via mobile phones.50

How is the broadcast material produced?

Once a health topic is chosen, there are several stages of qualitative research and production involved in creating spots. Here, we outline the standard production process for DMI's radio campaigns. DMI followed this process for its RCT campaign in Burkina Faso and plans to follow a similar production process for ongoing and future campaigns.51

Discerning specific obstacles to behavior change

DMI has a qualitative research team with three full-time employees based in Burkina Faso, and employs one qualitative researcher in the DRC.52 Before DMI broadcasts on a specific health topic, the research team seeks to understand the biggest limiting factors that prevent people in the broadcast areas from adopting the behaviors DMI plans to encourage. They do so by visiting rural villages within range of broadcasting radio stations and interviewing caregivers of children under five, chiefs, religious leaders, midwives, community health workers, and others. They combine what they learn in interviews with World Health Organization (WHO) recommendations to create 2-3 recommended messages for each health topic. The team writes a one-page message brief for each recommended message to give context for it (see footnote for examples).53

Here is one of the two behavior messages in DMI’s message brief on pneumonia, and the three barriers to that behavior it hopes to overcome:54

Behaviour to promote
If your child has a cough and rapid breathing or difficult breathing, they may be suffering from pneumonia, a severe lung disease. Bring the child to a health centre for treatment immediately as the disease can be fatal.
...
Barriers to behaviour change
  • Many children die of pneumonia at home because their parents do not understand the seriousness of this disease and do not know that it requires immediate professional medical help.
  • Many people do not recognize the signs of pneumonia. They often confuse the symptoms of pneumonia with various forms of cough, and also with those of other diseases such as malaria and meningitis. It is necessary to emphasize the importance of recognizing the key signs of pneumonia: fast or difficult breathing.
  • Many people will first self-medicate. They use traditional herbal “tisanes” or various concoctions. Only after the failure of various attempts at home or when the child develops more serious symptoms, such as fever, loss of appetite or sleep, will they make use of the skilled care available at the health centre.

Writing and producing spots

Burkina Faso

In Burkina Faso, scriptwriters at DMI’s office in Ouagadougou each write two scripts for the particular message that will be the focus of an upcoming week’s broadcast (see above for a sample script).55 The scriptwriting team then narrows down the scripts to their approximately 10 favorite scripts, which are sent to London for DMI's creative director to review. The creative director makes sure that the scripts are clear and adhere to the message brief. The top four or five scripts of the batch are selected by the London office and returned to the country office. The scriptwriting team then produces these spots for pretesting.56

Democratic Republic of the Congo

For its 2015 child survival campaign in the DRC, DMI made alterations to its message creation process – in addition to producing original scripts for the campaign, scripts from Burkina Faso were repurposed and reviewed by a researcher familiar with the DRC. To assist with the spot production process, two experienced DMI scriptwriters from Burkina Faso joined the DRC team.57

Spots were selected for broadcast using a procedure similar to that used in Burkina Faso.58

For its 2016 campaign in the DRC, DMI plans to reuse the spots it used in 2015.59

Pretesting sample spots in the field

The qualitative research team brings recordings of the sample spots to a sample of villages (e.g., in the case of Burkina Faso, the sample spots were brought to two villages within one of the seven broadcasting zones). Each of the three researchers (two female, one male) interviews one focus group in each village composed of 12-13 adults of the same gender as the researcher. The researchers play all the spots and lead discussions to determine whether people understand and appreciate them, and to learn if the content matches local reality. Using this information, DMI selects four spots to air in future weeks (for examples of the “pretesting synthesis” reports that result from the spot-testing research, see this footnote).60

Sending spots to each station in the local language

The country office finishes producing the selected spots in the languages spoken in the broadcast zones. DMI then distributes the recordings to each partner radio station for broadcast.61

Soliciting feedback from listeners after broadcasts

During the RCT in Burkina Faso, the qualitative research team visited villages in one of the broadcast zones about once a month to see how broadcasts on recent messages had affected behavior in those areas. As with the visits for pretesting sample spots, the researchers each met with one focus group of 12-13 people of their own gender in each village. To solicit feedback, the researchers asked questions to understand whether people had heard the spots, understood the messages, and changed their behaviors. They also asked questions to understand what other influences were affecting their health behaviors. In some cases, this feedback affected the messaging of future spots, such as when DMI realized that some people were interpreting DMI’s message on exclusive breastfeeding to prescribe excluding drinks and liquids but not traditional herbal concoctions, which may cause diarrhea (for examples of reports produced by this feedback research, see this footnote).62

We are not sure if these feedback research visits are being conducted during the national campaign in Burkina Faso, though DMI did tell us that its qualitative research team is continuing to operate during the national campaign.63

For discussion of the qualitative research from the DRC campaign we have seen, see below.

How does DMI choose and partner with radio stations?

Burkina Faso child survival campaign

For its national child survival campaign in Burkina Faso, DMI is working with 29 radio stations.64

Some stations broadcast spots for DMI's nutrition campaign, and some stations will broadcast family planning messaging in addition to child survival messages during the family planning RCT (which will run from 2016 to 2019). We have not confirmed whether these stations are the same stations as those that are broadcasting DMI's child survival messaging.65

When selecting which radio stations or networks to partner with, DMI considers the station’s management, staff, listenership, and cost.66

DMI believes that it has developed a good reputation among Burkina Faso radio stations, which has made it easy to find stations to partner with for the national campaign.67 DMI is currently making small airtime payments to its national campaign partner stations. If more funding became available, DMI would increase these payments. DMI is also considering purchasing solar panels for partner stations as an alternative to airtime payments, but does not yet have the funding to do this.68

Democratic Republic of the Congo child survival campaign

In the DRC, DMI is working with a partner organization that handles distribution of broadcast spots and relationships with community radio stations. DMI has chosen this arrangement in part due to timing constraints (the 2015 DRC campaign was funded for one year – 4 months of setup and 8 months of broadcasting), in part due to the difficulty of traveling in the DRC, and in part because setting up a ‘new’ network of community radio stations in the DRC would unnecessarily duplicate work already done by other organizations.69

DMI told us that it had difficulty working with its DRC distribution partner this year, and plans to work with a different distribution partner for its 2016 scale-up of the DRC child survival campaign.70

Mozambique child survival, family planning, and maternal health campaign

In Mozambique, DMI is in broadcast negotiations with the national radio network and television network. DMI is optimistic about developing a positive relationship with the Mozambican national networks. DMI also plans to broadcast on community radio stations in areas where the national networks have weak coverage and/or audience share, in order to maximize the geographic coverage and audience reach of the campaign.71

Does it work?

Note: The following section was written before DMI sent us preliminary endline results for its RCT in Burkina Faso; see the details on the preliminary results at the top of this page.

As such, the below discussion does not fully reflect our current view of DMI's program effectiveness and much of the discussion is outdated.

Additionally, since we wrote the below section, the midline results from DMI’s RCT were published (see Sarrassat et al. 2015). The published midline results are not substantially different from the midline results we discuss below.

The main evidence for the effectiveness of DMI's program comes from the midline results of a randomized controlled trial (RCT) on its child survival campaign in Burkina Faso. We discuss the results of the trial in detail below.

We focus on four questions that most affect our understanding of DMI's impact on health and mortality:

  • Does DMI’s program change listeners’ behavior? (More)
  • Do these behavior changes result in saved lives? (More)
  • How does DMI monitor and evaluate its ongoing programs? (More)
  • What monitoring and evaluation does DMI plan to implement for future campaigns? (More)

Does DMI’s program change listeners’ behavior?

The strongest evidence of DMI’s impact on listener behavior comes from the midline survey of a randomized controlled trial of DMI’s intervention.72 The midline survey measures uptake of various health practices and compares uptake between the treatment and control groups. Below, we discuss the RCT’s design, the midline results, and issues that influence our interpretation of these results. We have not used other studies to supplement the argument that mass media can successfully improve health behaviors.73

DMI’s study measured moderate increases in self-reported behavior on several important health outcomes. However, we are unsure of how to interpret these results generally and believe that the issues noted below require that any interpretation be made with caution.

Note that the results released to date are interim results. DMI plans to collect mortality data as part of its final survey, which may help answer some of our outstanding concerns about the reliability of the results.74 Update: see note above about preliminary endline results.

RCT design

From the midline results report:75

Background

[...] Fourteen local radio stations across the country, with high listenership, were selected for the evaluation, of which seven were randomly selected to broadcast short spots of 1 minute duration 10 times per day and long format programs for two hours per day, five days each week. Topics covered include antenatal consultations, health facility delivery, breastfeeding, nutrition, bed net use, sanitation and appropriate health care seeking for malaria, pneumonia and diarrhoea.

[...] This report presents the results of the midline survey which was conducted in November 2013, 20 months after the launch of the campaign. The endline survey will be undertaken within the last months of the intervention, after 30 plus months of campaigning. The objective of the midline survey was to provide mid-term estimates of behaviours changes in order to adjust messages addressed by the campaign.

Methods: sampling

Within each cluster, a random sample of 9 villages was drawn with probability proportional to size from villages surveyed during the baseline survey. In each of these villages, a sample of women was selected by simple random sampling using the census data collected during the baseline survey. These women were randomly listed and fieldworkers were asked to interview the first forty available women who met the inclusion criteria in the survey: being aged from 15 to 49 years old and mother of a child less than five years old.

Results

DMI’s draft midline results report increases in some behaviors, particularly curative behaviors such as seeking treatment for a child who has malaria, diarrhea or pneumonia, but limited evidence of an effect on other behaviors, particularly ongoing preventative behaviors such as washing hands with soap after cleaning a child who has defecated.76

Two papers and an editorial discussing the midline results have been published in Global Health: Science and Practice, an open-access journal.77

DMI reported 54 metrics in its midline report, many of which relate to the same behavior (such as, “things you may do in response to finding that your child has diarrhea”), and a few of which measure behaviors not addressed by DMI’s campaign at all. Lacking a clear way to choose the most relevant metrics, we excluded behaviors that were strict subsets of other measured behaviors,78 and grouped the remaining 32 into the eight categories below, then calculated the average difference in difference for each category (category definitions in this footnote).79

Note that Tables 2a and 2b report summary data from a draft version of the midline results paper. A more recent version of these results is available in Sarrassat et al. 2015.

Table 2a: Curative behavior changes
Curative behavior categories Average % improvement in control group Average % improvement in intervention group Average % intervention improvement minus % control improvement
Sought medical assistance 9.6 18.3 8.8
Received treatment 12 12.9 0.9
At-home diarrhea responses 4.8 16.3 11.4
Used full treatment of antibiotics/antimalarials -10 -7.7 2.3

Table 2b: Preventative behavior changes
Preventative behavior categories Average % improvement in control group Average % improvement in intervention group Average % intervention improvement minus % control improvement
Used a health facility during pregnancy and birth 6.3 5.4 -0.9
Sanitation and hygiene 5.8 5.5 -0.3
Food and nutrition 6.2 7.6 1.3
Other health behavior 12.6 12.2 -0.4

Our cost-effectiveness analysis of DMI's program is driven by changes to three curative behaviors: treatment of pneumonia, malaria, and diarrhea. The midline results measured treatment seeking and treatment received for each of these behaviors. Of these six measures, only diarrhea treatment seeking (adjusted), diarrhea treatment received (unadjusted), and pneumonia treatment received (adjusted) were statistically significant at the p < .05 level.80

Issues related to the DMI RCT

The following factors lead us to interpret DMI’s results with caution. Due to the below, there is room for a plausible story in which the midline results significantly overstate the impact that DMI’s program has had. However, it is possible that these factors have little effect on the results.81

The control group was noticeably better off than the treatment group at baseline

At baseline, there were noticeable differences between the treatment and control groups in terms of child mortality rates, distance and access to health facilities, remoteness from the capital city, and proportion of women giving birth in a health facility. These differences could conceivably make health behavior change easier for one of the two groups. For example, it may be easier to improve conditions from a worse baseline, which could cause the RCT to overstate impact. Alternatively, factors that have made the control group better off could also mean the secular trend in the control zones is steeper than that in the intervention zones, which could cause the RCT to understate the impact.82

  • The control group appears to have had better access to health facilities than the treatment group. The study reports that "40% of women lived less than 2 km away from a health facility in the control arm versus 18% in the intervention arm. Among women sampled for the midline survey, these proportions were 47% versus 16% respectively."83
  • The treatment clusters had higher rates of child mortality in the baseline survey, perhaps due to the fact that they tended to be further from the capital of Burkina Faso and receive less investment than areas that are closer.84
  • There were differences in ethnic and cultural composition between the treatment and control arms.85

DMI told us that, having reviewed these differences between the control and treatment groups, the trial’s Independent Scientific Advisory Committee recommended a cluster-level adjustment for baseline levels in the analysis, including use of a composite confounder score to adjust for the key baseline imbalances, which has been followed by the LSHTM team in the midline results.86

Better partner station opportunities in the treatment group than in control group

The seven radio stations selected for the intervention group by the randomization process represented more promising “opportunities to establish effective working partnerships” than those selected for the control, a difference which may have made it easier to create a successful campaign.87

The midline results are measures of self-reported behavior change

Health behaviors were assessed through self-report, which may be biased.88 For example, respondents in the treatment group who listened to the radio program may inflate the effect it had on their behavior if they believe the surveyor wants or approves of a certain answer.

Declining overall child mortality

The mortality rate in Burkina Faso as a whole has recently declined faster than that of most comparable countries.89 There are two potential issues here: (a) if the programs that caused child mortality to fall affected treatment and control clusters differently, this could bias the results of the RCT, and (b) declines in mortality could lead to the study lacking sufficient statistical power to detect a program impact on mortality.

Other interventions focused on child mortality in Burkina Faso

  • Population Media Center (PMC) conducted a nationwide mass media campaign to attempt to increase family planning behaviors in Burkina using two 156-episode serial dramas while DMI’s trial was taking place. PMC programs were broadcast on the national radio stations across the country and in 22 community stations that did not include DMI’s control stations (at DMI’s request) but did include six of DMI’s seven intervention stations. PMC believes it was likely to have had some impact on childhood mortality because:
    • PMC believes its program was successful at affecting family planning and other health behaviors, based on its endline survey (which did not have a randomized control).90
    • PMC expects that successful family planning, which reduces the rate of child bearing in families, increases maternal and child health as well as family resources during childhood, thus decreasing childhood mortality.91
    • PMC’s dramas touched on behaviors that affect childhood mortality along with other health behaviors in addition to family planning.92

    Our impression is that DMI’s campaign was significantly more intensive than the PMC campaign. From limited information, we would guess that PMC’s broadcasts did not have a significant effect on the key behavior changes measured in DMI’s midline results (seeking and receiving treatment for children with diarrhea or symptoms of malaria or pneumonia) and so have not affected our current estimates of DMI’s impact.

    However, if PMC’s broadcasts in community radio stations did have an effect on the mortality rate in some of DMI’s intervention zones, it would inflate DMI’s endline mortality results. Prof. Cousens has noted that even in that scenario, the results would still represent evidence of reduced childhood mortality from mass media campaigns.93

  • The Bill & Melinda Gates Foundation and other funders supported a large training program for Burkinabé health workers during the trial. The program is active in two clusters, one in the treatment group and one in the control group, and may have affected one cluster more than the other.94
  • There was a large national bednet distribution program in Burkina Faso during the trial as well as national immunization days that included vitamin A supplementation. The RCT researchers do not believe that the programs affected control and treatment clusters differently.95
  • Other health programs may have reduced child mortality during the trial period.96

Decline in statistical power

DMI's RCT was designed to have an 80% chance of rejecting the hypothesis of no difference between the mortality rates in the treatment and control groups (at the 95% confidence level), assuming that the campaign reduced mortality by at least 19.9%. Assuming that the campaign reduced mortality by at least 15%, the RCT was designed to have a 50% chance of rejecting the hypothesis of no difference between groups. This study design was based on an assumed child mortality rate of 168 deaths per 1,000.97

However, DMI has told us that child mortality in Burkina Faso has declined to 96 deaths per 1,000 in the five years after the study was designed. Due to this decline, DMI no longer expects its RCT to demonstrate a statistically significant effect on child mortality at endline.98

Lack of blinding

Surveyors were aware of whether respondents were in the treatment or control groups, which could lead to bias in the results.99 Since some of the questions were about whether the participants had heard DMI’s spots, blinding the surveyors would have been difficult.

Contamination in the control group

Two-thirds of a control group cluster with 375 women was excluded from analysis because about half of the women surveyed in their villages reported listening to the radio campaign in the week before the midline survey.100 Including these women in the analysis might underestimate the impact of the campaign, while excluding them may introduce bias, though it would likely be small.

Unusually intensive intervention

DMI has told us its first programmatic priority during the trial was to make sure there would be a measurable decrease in lives saved in the RCT through mass media activities.101 While the primary mechanism for achieving the reduction was broadcasting the 60-second spots, DMI went above and beyond what it would typically include in a mass media campaign. This increased effort makes it difficult to estimate the impact of future, less-intensive campaigns.

Note that DMI plans to conduct less-intensive campaigns in the future because it believes that many of the elements of its current campaign to be less cost-effective than the campaign as a whole; if this is correct, then removing each of these elements will improve the cost-effectiveness of future campaigns despite a possible decrease in impact. Still, the changes add uncertainty to the expected impacts. Some examples of DMI activities not to be included in future campaigns are listed below.

  • Negotiating with PMC to keep PMC broadcasts out of community radio stations in DMI control zones (PMC did broadcast programming in DMI intervention zones and on national radio).
  • Producing 10 long-format module scripts per week that dramatize health messages. Producing this content was originally planned as part of the RCT as a way to help radio stations build quality content, thus developing a better listenership and improving DMI’s relationship with the radio stations. Early in the RCT, DMI decided to produce more long-format modules than originally intended and to have them focus on the same health messages as the spots instead of merely providing entertainment content.102
  • Renting and buying generators, solar panels, and broadcast equipment for radio stations. DMI plans to continue to help radio stations avoid power outages, such as possibly loaning or giving solar panels to each station in lieu of cash payments, but does not plan to respond as aggressively to future outages and equipment failures.103
  • Intervening heavily to improve management of struggling radio stations. DMI has spent significant staff time and energy assisting station management when necessary, including employing two individuals to work full-time at problematic partner stations. Because DMI will not be running an RCT during future campaigns, it will have more flexibility to stop working with stations that collapse or do not broadcast DMI content regularly. This flexibility may significantly reduce program costs, although doing less to assist radio stations may have negative effects on program impact, such as lower quality programming and less engaged listeners.104

The investments made during the trial may have affected the midline results in several ways:

  • Enabling the stations to broadcast more of DMI’s spots than they otherwise would have by remaining on the air.
  • Reinforcing the health messages in the minute-long spots with the 10-15 minute modules which also included similar messages in longer narratives.
  • Improving listenership of partner radio stations through increased airtime and higher quality programming.
  • Incentivizing partner stations to prioritize airing DMI’s spots or using DMI’s suggestions for prime-time programming due to the perceived benefits of the relationship with DMI.

Unreported results

One complicating factor in interpreting the midline results is that we are not sure which results speak most directly to the success of DMI’s program so far. The midline survey, on which the midline results are based, contains some potentially relevant questions for which we have not yet seen the results, including:105

  • For your most recent birth, is your child still alive, or at what age did they die?
  • When your child has diarrhea, what should you do?
  • If your child received treatment for diarrhea/malaria/pneumonia, may I see the marked health card/prescription/medicine packaging?
  • How many days were there between your child’s first symptoms and first receiving treatment?
  • For how many days was your child sick?

Do changed behaviors result in saved lives?

Are health supplies and services available when sought?

Some of the health activities that DMI promotes, including the three that DMI believes saved the most lives during the RCT (quickly treating diarrhea, malaria, and pneumonia in children), require that families are able to access health facilities or providers and that appropriate medicines are available.106

We are not confident in our understanding of how accessible these treatments are to those that seek them. DMI’s midline results measured self-reported treatments received in addition to treatment sought; we use the former in our estimate of DMI’s effectiveness.107

To determine whether targeted populations have access to key health supplies in countries in which DMI is considering launching a program, DMI uses Demographic and Health Survey (DHS) data to determine the portion of those who say they sought treatment for a child’s symptoms (such as a fever) that also report receiving treatment (such as antimalarials).108

DMI notes that the DHS program has the best data on health supply availability and health center access, and that DHS data suggests that availability and health center access are similar in Burkina Faso and the DRC.109 We have not verified this claim.

Do the individual health practices save lives?

We have not completed a review of each of the practices DMI encourages through its broadcasts, but we believe that many of them – e.g., pneumonia treatment, malaria treatment, and oral rehydration therapy – represent well-established ways to prevent child mortality.110

How does DMI monitor and evaluate its ongoing programs?

DMI has told us that it uses various mechanisms to monitor its ongoing campaigns to determine whether (a) programs have aired at the scheduled times, (b) people have listened to DMI's programming, and (c) listeners have correctly understood the intended messages.111 We have seen some data from DMI’s monitoring of its Burkina Faso RCT campaign, its national campaign in Burkina Faso, and its campaign in DRC. When possible, we compare data from the RCT to the two post-RCT campaigns to try to understand whether the quality of DMI’s RCT campaign is being maintained as it scales up.

Below we discuss the evidence we have seen. Broadly, in DMI’s post-RCT campaigns (which have less intensive monitoring and data collection than the RCT), we have some evidence that some broadcasts are occuring and are heard by some portion of DMI’s audience, but we are very uncertain about how frequently broadcasts are occurring, what portion of DMI’s target audience is hearing DMI’s spots, and what portion of people who hear the spots understand the intended messages. Overall, we have limited evidence on the quality of DMI’s national campaign in Burkina Faso and its campaign in the DRC.

How reliably do the broadcasts happen?

Burkina Faso RCT

We would guess that broadcast frequency (i.e., how often programs were aired) was fairly high in the RCT largely because midline results from the RCT suggest that a large portion of women in the broadcast zones heard one of DMI’s spots and may have changed their health behavior.112 DMI also says it had frequent direct contact with each of its partner stations,113 and we have seen a large number of anecdotes about DMI’s efforts to keep its partners broadcasting frequently.114

Burkina Faso national campaign

We have limited information about broadcast frequency in DMI’s national campaign in Burkina Faso:

  • As far as we know, DMI has not yet conducted qualitative research on whether people are listening to its programs or changing their health behaviors as a result of its national campaign.
  • We do not know if DMI will be able to maintain the intensity of its contact with partners that it had during its RCT now that it is working with significantly more partners and generally scaling up its operations,115 but we would guess that its contact with partners will lessen somewhat.
  • Taken at face value, the broadcast monitoring data (more details below) suggest that DMI may be falling short of its broadcasting target in this campaign, but we do not put much weight on this evidence.

DRC

We have limited information about broadcast frequency in DMI’s campaign in the DRC:

  • As far as we know, DMI has not had much direct contact with its partners in the DRC. Instead, it has contracted its distribution partner to monitor local stations,116 but we have not seen monitoring data from its partner.
  • We have seen limited data from only 3 of DMI’s 35 DRC broadcast zones on whether people are hearing DMI’s spots in DRC and whether they’re changing their behaviors. We discuss this data in the next section.
  • Taken at face value, the broadcast monitoring data suggest that DMI may be falling short of its broadcasting target in this campaign, but we do not put much weight on this evidence.
  • DMI may expect its broadcasting frequency to be lower than usual in this campaign; it has encountered some issues with this campaign.117

Details on how we reached the above conclusions follow.

Monitoring processes

Though details vary by location, DMI generally uses a variety of methods to determine whether its spots are being broadcast and how frequently they are aired:

  • Hiring “broadcast monitors.” For its campaigns in Burkina Faso and the DRC, DMI employs individuals in its broadcast zones to listen to the radio and track the frequency of DMI broadcasts.118 DMI tries to hire two to three broadcast monitors for each broadcast zone. The monitors are instructed to record each time they hear a DMI spot (which are tagged with a "laughing baby" identifier), and to report when the station is off the air or has technical difficulties.119 The monitors are independent, do not know that the other is employed by DMI, and do not know how frequently DMI spots are supposed to be played. Partner stations do not know the identity of the monitors, and the monitors are instructed to not disclose their identity to the radio station.120 Discussion of results from this monitoring are below.
  • Direct contact with stations. DMI uses a combination of communication with radio stations (e.g., visits, phone calls), requesting and reviewing broadcast reports from stations, and, in some cases, employing staff at radio stations to determine whether the stations are broadcasting.121 With the exception of anecdotes about issues that stations encountered during DMI’s RCT,122 we have not seen documentation from these activities.
  • Surveying people in target areas about whether they have heard DMI spots and whether they have changed health behaviors. DMI employs qualitative researchers to survey people in target areas about whether they have heard DMI’s spots and whether they have changed their health behaviors. These surveys may provide some evidence about whether and how often DMI’s spots are being broadcast. We discuss this evidence in the sections below.

Broadcast monitoring results

Overall, we feel that the broadcast monitoring data does not provide a strong indication of how frequently DMI spots are being broadcast; we could easily imagine these figures overestimating or underestimating the true broadcast frequency. The usefulness of this data seems limited by the fact that:

  • We are not aware of an auditing process for this data, though we have not asked DMI about this explicitly.
  • Monitors of the same station often report sizable discrepancies in the station's broadcast frequency. We are unsure how to explain these discrepancies. There also seems to be a significant amount of missing data that we are unsure how to interpret.

However, we believe this data provides weak evidence that broadcasts are occurring, partly because monitors sometimes seem to simultaneously report when a station is off the air, which provides some support to the idea that it is meaningful when monitors report that spots are being played.123

Table 3 summarizes our analysis of a sample of broadcast monitoring data we have seen from 2015.124

Table 3: Summary of broadcast monitor data
Campaign Period analyzed Average broadcasts logged per day Average discrepancy between monitors % of station-weeks when both monitors did not record DMI spots125
Burkina Faso RCT campaign 1/9/2015 to 1/29/2015 10.47 spots 3.16 spots 4.8%126
Burkina Faso national campaign 6/26/2015 to 10/4/2015 4.86 spots127 2.58 spots128 13.4%129
DRC child survival campaign 5/11/2015 to 10/4/2015 4.75 spots130 1.93 spots131 38.2%132

We believe that DMI generally intended to broadcast 6-10 spots per day to achieve “saturation,” which it believes is best for the effect of its program.133 We do not know how important it is for DMI to meet its threshold for saturation. Our best guess is that lower broadcast intensity implies less behavior change, though we are uncertain about this.

Methodological details by campaign:

  • Burkina Faso RCT campaign: In the above table, we analyze one of the four months of data that DMI shared with us (January 2015); we analyzed only one month due to time constraints (we chose to analyze January 2015 because it was the most recent of the four) and the four months of data that DMI provided were randomly chosen by us.134 The January 2015 monitoring data contains data for each of the seven intervention zones in the RCT.135

    We also looked for instances of stations being off the air in the funder reports that DMI shared with us; details in this footnote.136

  • Burkina Faso national campaign: DMI shared broadcast monitoring data from the Burkina Faso national child survival campaign that started in May 2015; it shared data from late June to the beginning of October for what appears to be all 29 stations it is working with, from which we analyzed a random sample of 8 stations due to capacity constraints.137 Many more details about this data are in this footnote.138
  • Democratic Republic of the Congo: DMI has told us that establishing good campaign monitoring is more difficult in the DRC than in Burkina Faso; it told us that it must recruit and manage monitors by phone.139 As of July 2015, DMI had recruited 61 broadcast monitors in the DRC.140 Its campaign in DRC began in May 2015.141 DMI shared broadcast monitor logs from mid-May to the end of September for what appears to be at least all 35 stations that it partnered with in its campaign; we randomly selected a sample of 11 stations for more thorough analysis.142 Many more details about this data are in this footnote.143

Do people hear DMI’s broadcasts?

Burkina Faso

There is fairly strong evidence that a large proportion of people in DMI’s RCT broadcast zones in Burkina Faso heard DMI’s spots. However, we have not seen qualitative research or any other evidence on whether people are listening to DMI’s broadcasts for DMI's 2015 post-RCT, national campaign in Burkina Faso.

DMI’s midline survey found that 75% of women in DMI’s RCT broadcast zones in Burkina Faso reported recognizing at least one of two recorded spots (that had been broadcast soon before the women were surveyed) when it was played for them during the survey, compared with 20% of women in the control zone.144 This result was statistically significant.145

Democratic Republic of the Congo

There is some evidence from DMI’s qualitative surveys that some people are hearing DMI’s spots in 3 of the 35 broadcast zones where DMI is working in the DRC. We are uncertain what to conclude about what portion of people are hearing DMI’s spots in these zones, and are highly uncertain about how representative we should expect these results to be of DMI’s work in the DRC generally.

In the DRC, DMI conducted qualitative research in August 2015 in villages that were in the broadcast zones of three of DMI’s 35 partner stations: Nsemo, Tomisa, and Kimvuka na Lutondo.146 Because the research was conducted in just three of the campaign's 35 broadcast zones (in areas close to Kinshasa, which were easier to reach and may be better developed than more remote provinces), we are unsure of how representative this evidence is. The research consisted of interviews with individuals and focus groups in eight villages in these provinces. Villages were randomly selected but the provinces were not. Focus group participants were not randomly selected; community leaders assisted in assembling a group by making an announcement. The first twelve people that responded and who matched the participant profile formed the group.147 Because of potential bias introduced by this survey methodology, we are uncertain what to conclude about the representativeness of survey responses within these zones.

In total, 179 mothers and fathers of children under 5 were interviewed in single-gender focus groups of about 12 participants each. Additionally, in Idiofa 15 boys and girls under 8 were interviewed in a focus group. 11 community leaders were interviewed individually.148

Table 4 displays the number of interviewees who said they listened to DMI's partner station.149

Table 4: Percent of interviewees who listened to DMI's partner station
Radio station Village # of focus group participants # of participants who reported listening to the station % who reported listening to the station Average broadcasts/day logged by monitors150
Nsemo Idiofa Cendre 51 35 68.6% 5.03
Nsemo Itshuam 24 6 25% "
Tomisa Kikwit - - - 3.02
Tomisa Kongila 24 13 54.2% "
Tomisa Langa 24 -151 - "
Kimvuka na Lutondo Kenge Centre 24 17 70.8% 4.52
Kimvuka na Lutondo Makiala 24 9152 75%153 "
Kimvuka na Lutondo Kongomitela 23 6 26.1% "

Overall, approximately 54% of focus group participants reported listening to the DMI partner station that was broadcasting in their area.154 Of the people who reported listening to the station that broadcasts DMI’s messages, all of them reported hearing spots tagged with a laughing baby (the identifier DMI attaches to its spots).155 The research report states that in all three broadcast zones, almost all participants who reported listening to the partner station also reported listening to subject matter associated with child and maternal health, such as messages on bednet use and the treatment of fever, diarrhea, and cough.156 We do not know the details of how people were asked about whether they had heard DMI’s messages, though we have seen the question guide used by the qualitative researcher.157 We have not seen granular data on interviewee’s responses.

Do people understand the broadcasts?

DMI’s qualitative research adds some weak, anecdotal evidence that people in DMI’s broadcast zones understand DMI’s health messages. We have a fairly large amount of qualitative research from DMI’s Burkina Faso RCT campaign, have not yet seen any qualitative research from DMI’s 2015 national child survival campaign in Burkina Faso, and have only seen a small sample of such research from DMI’s work in the DRC.

Burkina Faso

During the child survival RCT in Burkina Faso, DMI’s qualitative research team asked participants to explain the messages they heard on the radio during feedback focus groups. We have seen summary reports from these feedback groups and a synthesis report summarizing this feedback research across all seven intervention zones.158 Because the qualitative research team did not conduct similar interviews in the control zones, it is difficult to tell how much of participants’ message recall is due to DMI’s program alone.

DMI reported that most individuals are able to recall at least one of DMI’s broadcast messages and are able to explain the recalled health messages accurately. However, some individuals make errors in recalling the messages.159 The messages that were voluntarily recalled most frequently were taking children to a health center immediately when they become sick, and messages relating to malaria, diarrhea, and prenatal care.160 We have also reviewed several regional research syntheses from 2014 on breastfeeding topics.161 These syntheses seem to provide some anecdotal evidence that listeners have understood DMI's breastfeeding messaging (see following footnote for details).162

We have not seen qualitative research on child survival topics such as the treatment of malaria, diarrhea, and pneumonia for DMI's 2015 national campaign in Burkina Faso, so we do not know if listener understanding of these topics has changed during the transition from the RCT campaign to the national campaign.

Going forward, we expect future qualitative research and time series data from DMI's planned evaluation system (see below) to provide information on listener understanding of DMI campaign topics in Burkina Faso.

Democratic Republic of the Congo

DMI's qualitative research in Bandundu province (see above for further discussion of methodology and results) provides some anecdotal evidence that listeners have understood the content of DMI's messaging. Specifically, the research reports that interviewees were able to repeat the content of DMI messages on bednets, prevention of diarrhea, treatment of malaria in children, exclusive breastfeeding, and vaccination.163

The research report also noted that traditional healthcare practices remain prevalent and sometimes take precendence over Western treatments.164

We have not reviewed granular data related to this research. Because interviews were conducted in just three of the campaign's 35 broadcast zones (in areas close to Kinshasa, which may be more developed than more remote provinces), we are unsure of how representative this evidence is.