MiracleFeet — Monitoring and Evaluation (May 2023)

Note: This page summarizes the rationale behind a GiveWell grant to MiracleFeet. MiracleFeet staff reviewed this page prior to publication.

Summary

In May 2023, GiveWell recommended a $596,500 grant to MiracleFeet to conduct monitoring exercises of its program as it scales up in the Philippines and begins working in Chad and Côte d’Ivoire. This grant will fund MiracleFeet to work with external evaluators to estimate current clubfoot treatment rates across these three geographies, as well as clubfoot treatment rates after MiracleFeet has been operating for some time. Surveys of clubfoot treatment providers will be conducted before MiracleFeet expands in an area or enters into a new area, and then again following a few years of activities in those locations. We hope this will allow us to estimate the level of clubfoot treatment in the absence of MiracleFeet’s support, as well as the degree to which MiracleFeet’s support can increase the number of children being provided appropriate and effective treatment. We plan to use this information to update our estimates of the cost per child treated with MiracleFeet support, which is a key driver of cost-effectiveness. This information could help inform future grant decisions.

We are recommending this grant because:

  • We think it will address substantial uncertainties we have about an implementation grant we previously made, and could affect whether we continue to fund MiracleFeet or other clubfoot-focused organizations in the future.
    • We think this grant is cost-effective because the monitoring is relatively inexpensive and could inform large future funding decisions in the long term.
    • We know the program budget for our current implementation grant to MiracleFeet, but this monitoring grant will give us information on the number of children treated during the grant period and whether they would be likely to have been treated without the grant. These figures have a large impact on the estimated cost-effectiveness of the grant overall.
    • MiracleFeet’s program is relatively close to our cost-effectiveness threshold for funding, so we think changes to our estimate of the program’s cost-effectiveness would be more likely to change our funding decision than if we thought the program was significantly above or below our threshold for funding.
  • We think that it is unlikely that MiracleFeet would have the capacity to fund a similar evaluation without us. (More here.)

Our key reservations are:

  • We will still have uncertainties about several other parameters in our model following the evaluation, and don’t expect to resolve these. This means that we will still have moderate uncertainty about our clubfoot treatment model in the future.
  • The evaluation is a pre-post design, which means that it will measure clubfoot treatment before and after MiracleFeet begins programmatic activities. Therefore, if other factors cause clubfoot treatment rates to change over the course of the evaluation, we may still have uncertainty about the share of any change attributable to MiracleFeet.
  • MiracleFeet has less experience with impact evaluation than GiveWell's typical grantees. As a result, there’s a risk the quality of the evaluation won't be as high as we would like.
  • Even if we learn about MiracleFeet’s marginal impact as they expand in the Philippines and begin work in Chad and Côte d'Ivoire, it's possible we still wouldn’t feel confident about their marginal impact in other areas (or the impact of other clubfoot treatment implementers). This could limit the future room for more funding affected by this monitoring grant. (More here.)

Published: August 2023

Table of Contents

The intervention

Clubfoot is a congenital condition where one or both feet twist inward and upward, causing pain, mobility loss and possibly reducing income if left untreated.1 MiracleFeet supports the treatment of clubfoot by the Ponseti method, which we understand to be the gold standard of care. The treatment consists of a series of casts to correct the alignment issue, and in the majority of cases, an Achilles tenotomy (surgical cutting of the tendon) to improve the foot’s flexibility, followed by the use of braces while sleeping for four to five years to maintain the correction.2

MiracleFeet partners with local NGOs to support treatment of clubfoot through the routine healthcare system, including training providers to identify clubfoot and carry out the treatment, providing supplies for casting and braces, and community sensitization efforts.3 MiracleFeet also supports the use of a monitoring and evaluation app called CAST to collect data on treatment and ideally maintain a high-quality program.4

At the moment, we have significant uncertainty regarding levels of clubfoot treatment in the absence of NGO support as well as the degree to which NGO support can increase treatment coverage. We have provided MiracleFeet with an implementation grant to scale up its existing clubfoot treatment program in the Philippines and begin working in Chad and Côte d’Ivoire over five years.

See our intervention report for more details on clubfoot, the Ponseti method, and our assessment of its effectiveness. See our grant page for more information about our previous implementation grant to MiracleFeet.

The grant

GiveWell is recommending a grant of approximately $600,000 to MiracleFeet to fund a pre-post evaluation of MiracleFeet’s program as it expands in the Philippines and begins working in Chad and Côte d'Ivoire. We aim to address our uncertainty about the level of clubfoot treatment in the absence of NGO support, as well as the degree to which MiracleFeet’s support can increase the number of children treated.

The grant will cover the following activities:

  • MiracleFeet will work with external evaluators (who we expect will be based in the locations where the program is being implemented) to develop evaluation plans.
  • The evaluators will conduct baseline surveys of health facilities to estimate the number of children currently treated for clubfoot in the relevant contexts.
  • The evaluators will conduct a similar “endline” survey two to three years after the program expands/begins.
  • At the time of the endline survey, a data audit will also be conducted to validate the accuracy of data collected through CAST (the app used by MiracleFeet-supported clinicians).

Budget for grant activities

The budget for the grant is $596,500, broken down as follows:

  • $248,000 for baseline data collection activities, of which
    • $163,000 covers evaluator costs in the Philippines, Chad, and Côte d’Ivoire.
    • $85,000 covers MiracleFeet staff time and travel costs.
  • $348,500 for endline data collection and data audit activities, of which
    • $255,000 covers evaluator costs in the Philippines, Chad, and Côte d’Ivoire.
    • $93,500 covers MiracleFeet staff time and travel costs.

See MiracleFeet’s initial budget here (p. 12) and revised budget incorporating full expected staff costs here.

The case for the grant

We are recommending this grant because:

  • We believe that this grant would allow us to reduce our uncertainties about the implementation grant we provided to MiracleFeet. The cost per child treated is a key driver of the cost-effectiveness of the program, but we are very uncertain about it. While we have made our best estimates, we don't know how many children are currently being treated for clubfoot in the areas MiracleFeet will work or how many children the program will cause to be treated. The evaluation funded by the grant is intended to provide an estimate of the number of children that are treated for clubfoot without MiracleFeet’s support and how many more would be treated with MiracleFeet's support in the Philippines, Chad, and Côte d’Ivoire. This will allow us to better understand counterfactual treatment rates and the marginal impact of MiracleFeet’s program.

    Following evaluation activities, we may update:

    1. Our level of confidence in MiracleFeet’s program and internal data collection.
    2. Our level of confidence in the marginal impact of MiracleFeet’s program.

    This will allow us to decide whether to continue directing funding to MiracleFeet, to direct more funding to them or other clubfoot treatment programs in the future, or to redirect funding to alternative opportunities.

    This grant is relatively low cost, and we think we are moderately likely to learn something that will update us. MiracleFeet’s program is estimated to be relatively close to our funding bar in these contexts, so we think changes to our estimate of the program’s cost-effectiveness would be more likely to change our funding decision than if we thought the program were significantly above or below our threshold for funding.5 We roughly estimate the value of information from this grant to be around 14 times as cost-effective as unconditional cash transfers.6 When considering whether a value-of-information grant meets our cost-effectiveness bar, we incorporate factors like the size of the grant, how much funding the results could influence in the future, and the likelihood that the results will update us on the cost-effectiveness of the program.

  • We think it is unlikely that a similar evaluation would occur without direct funding from GiveWell. We are only aware of one previous evaluation conducted by MiracleFeet. This evaluation has been conducted over the past five years, and has surveyed caregivers in a number of countries about their child’s health outcomes and their personal experience with care providers.7 We are unaware of any other donor interest in funding an evaluation to estimate the quantitative impact of MiracleFeet’s program.

Risks and reservations

We have the following primary reservations about this grant:

  • We will still have a number of key uncertainties about MiracleFeet’s program following this evaluation. We will continue to feel uncertain about several key parameters affecting our assessment of clubfoot treatment.8 It is possible that we will feel that these uncertainties are very substantial, and a more rigorous evaluation might have allowed us to consider factors like treatment adherence, healthcare worker knowledge, and outcomes for treated kids that would inform our adjustments for the rate of relapse and partial treatment.
  • The evaluation is a pre-post design, which could result in us being skeptical of the results. The evaluation will involve conducting surveys of clubfoot treatment providers before and after MiracleFeet begins or expands its program in the relevant geographies. We do not think that a randomized controlled trial would be practical in this context (particularly because patients may travel for treatment), so despite its inherent limitations, we think that a pre-post evaluation is most appropriate. If other factors cause clubfoot treatment rates to shift substantially between the two surveys, we may incorrectly attribute this change to MiracleFeet’s program. We hope to mitigate this risk by requesting that external evaluators report on factors that may affect the supply and demand of clubfoot treatment.9 However, this could limit how much this grant would allow us to resolve our uncertainty on the effect size of MiracleFeet’s program.
  • We believe that MiracleFeet as an organization has less experience with impact evaluation than our typical grantees (although we note that it has engaged with us willingly as we investigated this monitoring and evaluation grant). We see this as both a potential risk and an opportunity. It’s possible that the partnership will be more challenging than we expect, and that because MiracleFeet will be directly managing the external evaluators, we may have some doubts about the quality of the evidence generated from this grant. We have met frequently with MiracleFeet as they have refined the proposal, and we expect to continue collaborating closely with them throughout the evaluation process. We also expect that there are a number of organizations with similar levels of impact evaluation experience that may be doing cost-effective work, and we see this grant as an opportunity to learn about the feasibility of working on an evaluation with such groups.
  • It is possible that we will still be uncertain about the marginal impact of MiracleFeet’s program in other locations following this evaluation. This evaluation will give us information about the impact of the program in the Philippines, Chad, and Côte d'Ivoire, but we are unsure how generalizable those findings will be to other contexts. We may be particularly uncertain if we learn counterfactual treatment levels are significantly different from what we previously believed, which could cause us to be unsure about the reliability of stakeholder information. If this were the case, it could limit the future room for more funding affected by this monitoring grant.

    Plans for follow up

    We plan to follow up with MiracleFeet via calls and informal reports every six months. We will consider this program for renewal in July 2027. The current implementation grant includes one year of exit funding in the event that we decide not to renew in 2027.10

    We will receive updates from MiracleFeet as they select the external evaluators, and as they develop the plan for the baseline and endline evaluation. We expect to have regular calls with MiracleFeet throughout the baseline and endline planning and surveying periods, and to support the planning process where helpful to ensure evaluation quality. We will receive reports following completion of the baseline and endline evaluations.

    Internal forecasts

    For this grant, we are recording the following forecasts:

    Confidence Prediction By time​​
    50% Baseline analysis of the Philippines will cause us to believe the treatment coverage rate for clubfoot in geographic areas without MiracleFeet’s support is ≤ 15%. January 2024 (or whenever baseline analysis is available)
    80% Baseline analysis in Chad and Côte d'Ivoire will cause us to believe the treatment coverage rate for clubfoot in geographic areas without MiracleFeet’s support is ≤ 5%. April 2024 (or whenever baseline analysis is available)
    70% We will feel confident using data collected by CAST as an accurate estimate of MiracleFeet’s treatment numbers. June 2027 (or whenever endline analysis is available)
    • In one country – 60%
    • In two countries – 40%
    • In all three countries – 20%
    		Endline analysis will cause us to believe that treatment coverage will exceed 50% in MiracleFeet’s FY2027 (from July 2026 to June 2027) June 2027 (or whenever endline analysis is available)

    Our process

    We did the following as part of this grant investigation:

    • Met with MiracleFeet to align on the goals of the evaluation. The proposed evaluator for the Philippines (the country where we expect survey activities will first be conducted) joined two calls.
    • Requested, received, and reviewed MiracleFeet’s proposal.
    • Asked additional questions and received answers and additional information from MiracleFeet, including an updated budget.

    Sources

    Document Source
    GiveWell, "MiracleFeet – Clubfoot treatment (January 2023)" Source
    GiveWell, "Ponseti casting for clubfoot," 2022 Source
    GiveWell, MiracleFeet cost-effectiveness analysis, 2023 Source
    GiveWell, MiracleFeet monitoring grant value of information CEA, 2023 Source
    Grimes et al. 2016 Source (archive)
    Harmer and Rhatigan 2014 Source
    MiracleFeet, "External evaluations showcase MiracleFeet’s remarkable impact," 2023 Source (archive)
    MiracleFeet, Evaluation grant: Proposal for GiveWell, 2023 Source
    MiracleFeet, Proposal for GiveWell, 2022 Source
    MiracleFeet, Revised budget, 2023 Source
    Owen, Capper, and Lavy 2018 Source
    Staheli 2009 Source
    World Health Organization, "Congenital anomalies" Source (archive)
    World Health Organization, World report on disability, 2011 Source (archive)
    • 1
      • “Clubfoot is an inborn deformity of the foot, where either or both feet are twisted inward, causing the child to walk on his ankles. Left untreated, the condition causes severe lifelong disability.” World Health Organization, "Congenital anomalies"
      • “If left untreated, clubfoot can result in physical deformity, pain in the feet, and impaired mobility, all of which can limit community participation, including access to education.” World Health Organization, World report on disability, 2011, p. 99.
      • “Untreated CTEV can lead to severe deformity, but with timely and correct management it is curable. CTEV is defined as an inward rotation of the foot, with four components: cavus, forefoot adductus, hind foot varus and equinus. Two of three patients occur in boys and in every other case, the condition affects both feet. The deformity causes patients to walk on the side or back of their feet leading to callus formation, and potential infections in the skin and bone. With severely hampered mobility, these patients are unable to work; thus, this deformity contributes not only to ill health but also to poverty.” Grimes et al. 2016, p. 1.

    • 2
      • “The Ponseti casting method has become common practice in high-income countries during the last decade and a half. It has changed the treatment of clubfoot so that complex posterior medial release of multiple tendons and joint capsules, which was once the standard, is now rarely needed. This new treatment involves sequentially stretching the deformed foot and holding the stretches in a series of casts to restore the correct alignment of the foot gradually (Fig. 2). Following the stretching sequence, a minor surgical procedure, percutaneous tenotomy, is nearly always required but can often be done in an outpatient clinic with local anesthetic.” Harmer and Rhatigan 2014, p. 839.
      • "The Ponseti method is now considered to be the gold standard of treatment in the USA, and its use has spread widely throughout high-income countries (HICs), largely replacing previously used surgical and conservative techniques. It is also more cost-effective, less invasive and has lower risks of complications than surgical treatments. As such, it is an ideal solution for low-resource settings." Owen, Capper, and Lavy 2018, p. 1.
      • "Most clubfoot can be corrected by brief manipulation and then casting in maximum correction. After approximately five casting periods, the cavus, adductus and varus are corrected. A percutaneous heel-cord tenotomy is performed in nearly all feet to complete the correction of the equinus, and the foot is placed in the last cast for 3 weeks. This correction is maintained by night splinting using a foot abduction brace, which is continued until approximately 2 to 4 years of age.” Staheli 2009, pp. 6-7.

    • 3 “MiracleFeet’s approach to addressing the global problem of untreated clubfoot is to strengthen local treatment capacity in LMICs and reduce the barriers to treatment that families typically face. By partnering with local NGOs working in disability and rehabilitation, we build upon each country’s existing health infrastructure to create and support a nationwide network of clinics—located in existing district or provincial hospitals—capable of delivering high-quality care. We train providers in the Ponseti method and ensure clinics are equipped with all necessary treatment supplies, including braces. We establish early detection and referral pathways by raising awareness of clubfoot in communities and training frontline health workers (such as nurses and midwives) in clubfoot identification, with the goal of ensuring families access treatment before a child’s first birthday (when treatment is easier on the child and optimal results are most likely). Once children are enrolled, our clinic teams educate and follow-up with parents to minimize patient dropout and optimize long-term outcomes. MiracleFeet complements this grassroots approach with “top-down” advocacy efforts with Ministries of Health to mainstream clubfoot care and ensure that high-quality Ponseti will be routinely available to children in the long-term.” MiracleFeet, Proposal for GiveWell, 2022, p. 1.
    • 4"In addition to increasing enrollments and geographic reach each year, MiracleFeet is highly focused on the quality of treatment delivered. We invest heavily in the ongoing mentorship of providers, and all partners are trained in the use of MiracleFeet’s mobile phone-based patient management and M&E system, known as CAST. Built on Dimagi’s Commcare platform, CAST is currently used by providers in 30 LMICs to track enrollments, record treatment data, and manage appointments. Providers enter patient data during in-clinic appointments, and data then flows into a Salesforce database where individual patient records are aggregated into key performance indicators on treatment quality. These are measured against the following globally accepted benchmarks based on standards for best clinical practice:
      • At least 75% of children enrolled in treatment are less than one year old
      • Each child receives 4-6 casts (the average number of casts needed to achieve a full correction)
      • At least 80% of children receive a tenotomy
      • Less than 10% of children drop out of the casting stage of treatment
      • Less than 20% of children drop out of the bracing stage of treatment

      Real-time reports and dashboards are visible by local partners, the MiracleFeet team, and regional program managers, democratizing access to key information and empowering data-driven decision-making. MiracleFeet’s regional program managers work with our Technical Team to review performance against targets at the clinic and country levels on a monthly and quarterly basis to identify areas of concern and devise action plans to address issues through additional organizational support, clinical training, community awareness, and/or parent education." MiracleFeet, Proposal for GiveWell, 2022, p. 7.

    • 5
      • Note that (a) our cost-effectiveness analyses are simplified models that are highly uncertain, and (b) our cost-effectiveness threshold for directing funding to particular programs changes periodically. As of May 2023, our bar for directing funding is about 10 times as cost-effective as GiveDirectly's unconditional cash transfers. See GiveWell’s Cost-Effectiveness Analyses webpage for more information about how we use cost-effectiveness estimates in our grantmaking.
      • Our current cost-effectiveness analysis estimates that MiracleFeet’s program may be about 8 times as cost-effective as unconditional cash transfers over the course of the grant period and about 9 times as cost-effective as unconditional cash transfers at scale.

    • 6See GiveWell, MiracleFeet monitoring grant value of information CEA, 2023, “Cost-effectiveness in multiples of cash, M&E only” row. This involves several key assumptions, including that:
      • We would fund MiracleFeet’s program or other clubfoot treatment programs for 10 years if we found it met our bar for cost-effectiveness.
      • There is an equal likelihood that the evaluation will update us positively (whereby more additional children are treated as a result of MiracleFeet's program than previously expected), negatively (whereby fewer additional children are treated as a result of MiracleFeet's program than previously expected), or not at all.
      • Our future bar for funding will be unchanged from our current threshold of 10 times as cost-effective as unconditional cash transfers.

    • 7“Driven by a desire to understand the long-term effects of our programs on the children and families we serve, MiracleFeet contracted impact measurement firm 60_decibels to evaluate outcomes and experiences of caregivers whose children have received gold-standard treatment for clubfoot through MiracleFeet programs in nine countries over the past five years (2019-2023).” MiracleFeet, "External evaluations showcase MiracleFeet’s remarkable impact," 2023
    • 8Uncertainties we have include the appropriate disability weight of clubfoot, the amount of disability that’s averted by treatment and the level of adjustments we should make for relapse, the impact of clubfoot treatment on consumption outcomes, and whether other surgeries are crowded out as a result of more tenotomies. See our intervention report for more details.
    • 9“Other factors affecting demand, supply, or access to quality treatment of clubfoot in the national healthcare system will be sought and examined.” MiracleFeet, Evaluation grant: Proposal for GiveWell, 2023, p. 11.
    • 10“This $5,210,581 grant, funded by Open Philanthropy and individual donors, will support: …A fifth year of funding (2028) to serve as exit funding in the event that we decide not to renew the grant.” GiveWell, "MiracleFeet – Clubfoot treatment (January 2023)"