GiveWell Reader to GiveWell, 5/4/2011
Holden-
Thank you for starting GiveWell, I enjoy reading the exchanges that you foster and appreciate that, as an organization, you value the application of empirical knowledge and systematically apply it to your impact analysis.
I often hear funders say "we fund high impact educational organization." In order to define if an educational organization is in fact "high impact", it is critical to have an operating definition of a high-quality education. Do we want all students to graduate high school with the skills necessary to excel in college? Do we want all students to be lifelong learners? Do we want all students to be ready to be active participants in our democracy? Do we want all students to have the skills necessary to advocate for themselves?
On your website, I had trouble finding your definition of the type of education you want to catalyze and therefore, without a clear definition, it is difficult, if not impossible, for you to determine if what you fund achieves your desired educational impact. The closest thing I did find on your website was the following statement with respect to the achievement gap:
There are widespread differences of opinion on the root causes behind the achievement gap. For our purposes, we find it unnecessary to take a position on what factor, or combination of factors, plays the largest part in the disparities discussed above; but we believe it is important to emphasize the largely undisputed point that the achievement gap cannot be entirely attributed to differences in people's environment and resources after the age of five; important differences in academic achievement are already present by the time people enter kindergarten
Clearly the goal of your funding is to eliminate the widely documented achievement gap. But, I couldn't find a clear articulation of how you propose to do this. I'm curious, what is your theory of change? What do you think are the critical skills for success?
You clearly state that you do not find it necessary to take a position on "what factor, or combination of factors, plays the largest part in the disparities." How can you determine the impact of an educational organization without first understanding the root causes of the achievement gap? By not addressing factors such as social capital, you are likely to find organizations that I like to call "false positives", organizations whose short term data suggests they have "impact", perhaps based on a positive trend in standardized tests scores, but in the long term do not achieve the desired impact.
On your website, you rank KIPP as your top Education organization. You say, "The Knowledge is Power Program (KIPP) is a charter school network aiming to improve academic achievement for low-income students. KIPP stands out from the rest of the charter school organizations we considered, and we believe it improves educational
outcomes for the students it serves."
The White House website promotes a "curriculum that fosters critical thinking, problem solving, and the innovative use of knowledge to prepare students for college and career." I have to admit, I am skeptical that a school that implements strategies such as those Doug Lemov describes in
Teach Like a Champion creates a culture of critical thinking. Further, I wonder the extent to which a quasi-militaristic culture prepares students to excel in a college or university culture.
You note that KIPP improves educational outcomes for the students it serves. The most recent study released by KIPP suggests that about half of their graduates (which are themselves a selective group based on the type of family that chooses KIPP and they type of family who stays at KIPP) are either in college or have completed college. While many are celebrating this; I wonder, how are the other 48% (discounting the students still in school) of the students are doing? How does it feel to have college be set as the expectation and then not persist? How many students, both those that graduated and those that did not graduate, exited with a mountain of debt and/or their confidence completely destroyed? How does KIPP prepare students who don't want to go to college? Deborah Meier writes: "It's exhausting work at best. Still we dare not rest until we can look about us and say that there is not a
single school to which we would not willingly--I don't say gladly, just willingly--send our own children." Would you willingly send your child to a KIPP school?
The achievement gap is an incredibly injustice. However, eliminating the achievement gap in test scores does not necessarily mean that students will be prepared for life. I challenge you to broader your goal to not only eliminate the achievement gap but also to consider what type of education you want your future children to receive and, as Deborah Meier suggests, to focus your educational funding on organizations that consistently produce this type of impact for low-income students.
Thank you for your time.
GiveWell to GiveWell Reader, 5/5/2011
Hello, I really appreciate the thoughtful comments. To clarify our position a bit:
- Because we serve individual donors, we tend to look for programs with proven and demonstrable track records.
- If we knew of education interventions with strong longer-term impact, we would prefer these to interventions where the main evidence is short-term impact on test scores. But we don't know of interventions in the former category; the latter is the closest we've seen to "proven impact on education." Since we don't want to point individual donors to experimental programs (without track records) and we don't want to tell them not to give, we recommend these organizations.
- I maintain that we don't need a theory of the root causes of the achievement gap in order to assess charities' track records in closing it.
Best,
Holden Karnofsky
www.givewell.org
GiveWell Reader to GiveWell, 5/7/2011
Hi Holden-
Thank you for your response. I appreciate the clarity you have with respect to your mission and your focus on providing individual donors with clear information to make data-driven decisions.
While, perhaps, it is not imperative for an evaluator to have a theory of the root causes of the achievement gap in order to assess charities' track records in closing it, it is imperative that you have a clear definition of what you are trying to achieve. You note, "Because we serve individual donors, we tend to look for programs with proven and demonstrable track records." While this is entirely logical, there is an inherent assumption that we, as a society, all agree on what an education should be. And, clearly, this is not the case. I find it notable that, given all the hype around KIPP-like schools, upper-middle class parents are not entering their students in the lotteries. Further, if we thought TFA was in the best interest of all children, why don't we also develop a system where upper-middle class schools save spots for first year teachers who will likely absorb the schools resources for a few years and then leave the teaching profession?
As a philanthropic advisor, it is imperative that you are transparent with respect to what indicators you value and why. Do you have a matrix of metrics that allows you to compare and contrast similar organizations on the same metrics? If so, what are these metrics and how do they align to your definition of the of the type of education you want to catalyze? How do you track the extent to which the organizations you fund ensure that students gain the skills with which you (and your donors) value?
You might say, evaluators don't run programs and therefore they should not get involved with day-to-day programming. And, I could not agree more with the premise. However, it is naïve to think that we are staying "neutral." When I hear that schools are doubling the amount of time spent on reading and math, I wonder what they are cutting. When I hear that teachers and students are working 10-hour days, I wonder if this is healthy and or sustainable for the students or the teachers. When I see test scores in math and reading skyrocket, I wonder the extent to which we are creating a system of incentives that forces teachers to focus on these subjects and drastically reduce the time spent on other valuable subjects, like science and social studies. I think, by staying "neutral" we are in effect perpetuating a system of educational institutions with which we would not feel comfortable sending our own children to but are "good enough" for other peoples children.
I believe that we, as a society, need to challenge ourselves beyond "closing" the achievement gap and focus on "opening" the doors for students that, for decades, have been allocated a disproportionately low number of resources and been provided an inadequate education. I encourage you to consider adopting (and sharing) a system of metrics that are aligned with your organizations beliefs with respect to aspects of a quality education with which you apply to all the education-related organizations you evaluate.
Thank you for your time.
GiveWell to GiveWell Reader, 5/11/2011
Hello,
I concede that we should have - and haven't - published an account of which metrics we find more meaningful than others, and why. Because we've de-prioritized further work on this cause, I can't commit to doing so anytime soon.
I do feel that there aren't a lot of difficult judgment calls in our choices to accept some metrics and not others. As I mentioned previously, we haven't found any organizations with evidence based on longer-term, more clearly meaningful measures; evidence based on test scores and/or graduation rates is the best we've seen.
Your criticism is legitimate and I'd like to make it more public with your permission. Would it be OK by you if we published this email exchange to the web and alerted people to it via our public email list (groups.yahoo.com/group/givewell)? If there are any edits you'd like me to make first, I'm happy to.
A couple of other notes:
- We don't see ourselves as "neutral." Our goal is transparency, not neutrality or objectivity.
- I feel that there's a good case that struggling/disadvantaged children need a fundamentally different kind of education from more privileged children. The achievement gap is present from the day they enter school (http://www.givewell.org/united-states/achievement-gap) and the best "equalizing intervention" we know of, scholarships, seems to have a disappointing track record (http://www.givewell.org/united-states/charities/Childrens-Scholarship-Fund#Doesitwork). I don't think the right question is "Would I send my children to a KIPP school rather than the school they currently attend?" but "Would people in underprivileged communities send their children to a KIPP school rather than the alternatives?"
Best,
Holden
GiveWell Reader to GiveWell, 5/27/2011
Hi Holden-
I apologize for not getting back to you sooner --- for some reason it did not appear in my inbox until recently. You are welcome to publish it as is. I very much admire your willingness to be self-reflective and look forward to your updates in the future.
Thanks.
GiveWell Reader to GiveWell, 5/28/2011
Hi Holden-
Another thought. I would pose the question slightly differently. The question "Would people in underprivileged communities send their children to a KIPP school rather than the alternatives?" is not particularly useful because we know that too many of the "alternatives" are incredibly under resourced and have been subject to decades of haphazard reform initiatives. Many parents in these communities would choose anything over the traditional neighborhood school --- especially if the school is safe and if it comes with the (by and large unproven) promise of preparing their child for college. I would tend to agree that some KIPP schools are better than the traditional neighborhood schools. However, if our metric is creating something better than dysfunctional, we will be lucky if we end up with a system of mediocre schools. If we are "successful" and create a system of mediocre schools, I still believe we have failed our obligation to provide all students with an education.
Instead, we should be asking the question "What type of education does a community want for their children and how can we foster a system of schools that meets these goals for all children?" In many ways, as I referenced in an earlier e-mail, we as a society know the general characteristics of a quality education. The question is, does KIPP succeed in providing this type of education to all the students that walk in their door? If the answer to this question is yes, then I agree with your funding recommendation. However, given an unusually high transfer-out rate, the emphasis on test prep and their less than stellar college graduation rates, I have reasons to doubt that they are meeting our countries educational vision.
Thank you for doing what you do.
GiveWell to GiveWell Reader, 6/1/2011
Thanks again for your thoughts and your permission to post these. GiveWell focuses on finding the best available option for donors, not on holding charities to an absolute standard. We feel KIPP makes a better case that it is improving the situation - relative to what was there before - than any other organization we've seen in this area. If you feel there is another more worthy organization, we'd be happy to take a look.
