About this pageGiveWell aims to find the best giving opportunities we can and recommend them to donors. We tend to put a lot of investigation into the organizations we find most promising, and de-prioritize others based on limited information. When we decide not to prioritize an organization, we try to create a brief writeup of our thoughts on that charity because we want to be as transparent as possible about our reasoning. The following write-up should be viewed in this context: it explains why we determined that (for the time being), we won't be prioritizing the organization in question as potential top charity. This write-up should not be taken as a "negative rating" of the charities. Rather, it is our attempt to be as clear as possible about the process by which we came to our top recommendations.
A note on this page's publication dateThe last time we examined the charities working primarily in the U.S. was in 2010. As of 2011, we have de-prioritized further work on this cause. The content we created in 2010 appears below. This content is likely to be no longer fully accurate, both with respect to what it says about the organization and with respect to what it implies about our own views and positions.
Table of Contents
What do they do?According to its website, the National Academy Foundation's (NAF's) mission is "to sustain a national network of career academies."1 A career academy provides academic, career, and technical education around a career theme (in the case of NAF: finance, IT, engineering, or hospitality & tourism) to prepare students for college and careers; and it directly connects students to employers in the field.2 NAF works with schools to either (a) assist existing career academies to become NAF career academies that meet NAF standards, or (b) create new NAF career academies in schools that don't have a career academy.3 Of the NAF career academies created in the 2008-2009 school year, 86% were new and 14% were existing career academy that became NAF career academies. In the 2009-2010 school year, 69% were new and 31% were existing.4 NAF provides ongoing support to its career academies by developing curricula for career academies to follow,5 training teachers in the school so that they implement the NAF model,6, and providing staff who monitor activities in the school and consult with the teachers about issues they face in implementing the NAF model.7 NAF does provide financial grants to academies, though this appears to be a relatively small portion (5-10%) of its overall activities.8 There are over 500 NAF career academies nationwide, serving more than 50,000 students.9
Does it work?
Evidence of impactThere is strong evidence for the career academies model, which has been the subject of a randomized controlled trial evaluation that found strong, positive results.10 Two of the academies that participated in that evaluation were National Academy Foundation academies.11 We summarize the results from this trial in our evidence review of career academies.
Ongoing monitoringAccording to NAF, regional staff perform informal monitoring and evaluation of NAF-supported career academies to determine whether the NAF program is being implemented appropriately.12 NAF recently began a new initiative to more formally monitor and evaluate its partner career academies. NAF asks each academy to assess itself on a number of factors that is has identified as important to its model, such as curriculum (e.g., are NAF courses taught according to NAF standards?), internships (e.g., does a sizeable proportion of the academy class participate in internships?), and advisory board (e.g., does the board have an adequate number of members? Is the board involved with curricular issues?).13 NAF does not continuously monitor longer-term outcomes for students such as alumni earnings or employment, or college graduation rates. NAF did commission one study,14 published in 2004, which surveyed 177 alumni15 and which assessed outcomes such as educational attainment,16 employment status,17 and former students' employment in sectors related to their academies' focuses.18 In summary, we believe there is strong evidence for the career academies model, but we are not confident in the impact of the National Academy Foundation.
What do you get for your dollar?We have extremely limited information about the total cost (including costs to both schools and NAF) and cost-effectiveness of career academies. We have the following information:
- NAF spent $14.5 million in 2009 to assist in the creation of 32 NAF academies and to support its network of approximately 500 academies. In 2008, NAF spent $12.8 million to assist in the creation of 40 academies and to serve its network.19
- The Coalition for Evidence-Based Policy states that schools running career academies face an additional cost of $675 per student per year.20
Room for more fundingWe don't have a precise picture of how NAF would use additional funding. Representatives told us several initiatives NAF would implement were they to receive additional funds:21
- Roll out a technical assistance and service delivery model, which includes changing from one director per region to a 4-5 person team per region.
- Improve NAF's evaluation system.
- Improve support for internships and advisory board members.
- Add at least 60 new career academies in 2011.23 Last year NAF added about 47.24
- Expand the network of NAF career academies by focusing on areas that are already advocates for the career academy model - $1.5 million.25
- Improve academy quality and performance by, for example, launching eCollege (an online tool for professional development) - $3 million.26
- Create a nationally-recognized NAF Certificate - $1 million.27
- Impact public policy and school reform - $200,000.28
- Must all new academies maintain fidelity to the model that was evaluated in the randomized controlled trial (Kemple and Snipes 2000)?
- NAF works with schools to either (a) assist existing career academies in becoming NAF career academies that meet NAF standards, or (b) create new NAF career academies in schools that don't have career academies.
- Is there evidence that NAF career academies have a greater impact than the existing career academies they replace?
- To what extent is NAF creating new career academies in schools where career academies would not have existed without NAF assistance?
- Brand, Betsy. 2009. High school career academies: A 40-year proven model for improving college and career readiness (PDF).
- Coalition for Evidence-Based Policy. Career academies. http://evidencebasedprograms.org/wordpress/?page_id=90 (accessed October 13, 2010). Archived by WebCite® at http://www.webcitation.org/5tRSYN6tP.
- Devery, Colleen. National Academy Foundation Assistant Vice President, Strategic Initiatives and Administration. E-mail to GiveWell, November 11, 2010.
- Devery, Colleen. National Academy Foundation Assistant Vice President, Strategic Initiatives and Administration. Phone conversation with GiveWell, August 26, 2010.
- Devery, Colleen. National Academy Foundation Assistant Vice President, Strategic Initiatives and Administration. Phone conversation with GiveWell, October 26, 2010.
- Kemple, James J., and Cynthia J. Willner. 2008. Career academies: Long-term impacts on labor market outcomes, educational attainment, and transitions to adulthood (PDF). New York: MDRC.
- Kemple, James J., and Jason C. Snipes. 2000. Career academies: Impacts on students' engagement and performance in high school (PDF). New York: MDRC.
- National Academy Foundation. Annual report (2008) (PDF).
- National Academy Foundation. Career academies new planning tool that will accompany self-assessment tool (draft) (DOC).
- National Academy Foundation. Career academies new self-assessment tool (draft) (PDF).
- National Academy Foundation. Data point wish list (XLS).
- National Academy Foundation. Financial statements and schedule December 31, 2008 and 2007 (with independent auditors' report thereon) (PDF).
- National Academy Foundation. Financial statements and schedule December 31, 2009 and 2008 (with independent auditors' report thereon) (PDF).
- National Academy Foundation. GiveWell follow-up questions (DOC).
- National Academy Foundation. Inside our schools. http://naf.org/inside-our-schools (accessed June 11, 2010). Archived by WebCite® at http://www.webcitation.org/5qPW5Uwfn.
- National Academy Foundation. IRS form 990: 2008 (PDF).
- National Academy Foundation Leadership summit: Timeless learning: Policy & practice (2010) (PPT).
- National Academy Foundation. New vs existing career academies (XLS).
- National Academy Foundation. Our mission and approach. http://naf.org/our-mission-history (accessed June 11, 2010). Archived by WebCite® at http://www.webcitation.org/5qPV2KQCw.
- National Academy Foundation. Proposal to the U.S. Department of Education Investing In Innovation: Strengthening career academies for student success (DOC).
- National Academy Foundation. Recommendations from NAF site visits that were used in the revision of the self-assessment tool (PDF).
- National Academy Foundation. Self-assessment 1-20 (XLS).
- National Academy Foundation. Self-assessment example (XLS).
- National Academy Foundation Self-assessment tool: Standard version (AOF, AOIT, AOHT): Fall '09 pilot version (XLS).
- National Academy Foundation. Self-assessment validation (DOC).
- National Academy Foundation. Short-term data (DOC).
- National Academy Foundation. Statistics and research. http://naf.org/statistics-and-research (accessed June 11, 2010). Archived by WebCite® at http://www.webcitation.org/5qPWCr7zG.
- National Academy Foundation. Strategic imperatives (2011) (DOC).
- National Academy Foundation. The NAF learning handbook (October 2009) (PDF).
- National Academy Foundation. Theory of change (DOC).
- National Academy Foundation. Year of planinng. http://naf.org/yop (accessed June 11, 2010). Archived by WebCite® at http://www.webcitation.org/5qPXJA7vD.
- Orr, Margaret Terry, et al. 2004. The National Academy Foundation's career academies: Shaping postsecondary transitions (DOC). IEE Working Paper 17.
- Stepman, Jessica, and Rebecca Batterman. Alumni and Corporate Relations Manager, and Development Assistant. Phone conversation with GiveWell, August 10, 2010.
- Stern, David, Charles Dayton, and Marilyn Raby. 2010. Career academies: A proven strategy to prepare high school students for college and careers (DOC).
- Tech Prep California. The Kern/South Tulare Consortium. http://www2.bc.cc.ca.us/techprep/partnershipplus.html (accessed October 13, 2010). Archived by WebCite at http://www.webcitation.org/5tRUi8qNZ.
- Webber-N'Dour, Karen. 2010. Written testimony of Karen Webber-N'Dour Principal, National Academy Foundation high school before the U.S. Senate HELP Committee: Reauthorization of ESEA: Improving America's secondary schools (PDF).
- 1. "The mission of the National Academy Foundation (NAF) is to sustain a national network of career academies to support the development of America's youth toward personal and professional success in high school, in higher education, and throughout their careers." National Academy Foundation, "Our Mission and Approach."
The Coalition for Evidence-Based Policy describes the key characteristics of career academies:
- "They are organized as small learning communities (150 to 200 students) to create a more supportive, personalized learning environment;
- They combine academic and career and technical curricula around a career theme; and
- They establish partnerships with local employers to provide career awareness and work-based learning opportunities for students."
- "Schools have to apply to open a NAF career academy. They might already have a career academy but not a NAF career academy. For example, they may have a finance academy in their high school, but might not follow all our rules for being a NAF academy, such as having an advisory board with community members that work in finance, along with other characteristics." Jessica Stepman and Rebecca L. Batterman, phone conversation with GiveWell, August 10, 2010.
- "GiveWell: Is NAF mainly helping existing career academies meeting NAF's requirements so that they become NAF career academies, or do you mainly help create new NAF career academies where there was no previous career academy? NAF: It's a mix, I can send you information on that." Colleen Devery, phone conversation with GiveWell, August 26, 2010.
- 4. National Academy Foundation, "New vs Existing Career Academies."
- 5. "We develop the curricula and we support field staff. Funding does trickle down to career academies. There's a specific model that we design that they follow." Jessica Stepman and Rebecca L. Batterman, phone conversation with GiveWell, August 10, 2010.
- 6. "Once accepted, academies go through a “year of planning” where they prepare to become NAF academies. Every July, we have an institute for staff development. We are normally working with public schools and the staff and teachers that are already there. It can be that we match a math teacher to teach finance, or it can be a guidance counselor that learns to be a NAF director. We really try to work with and make sure that what we're asking them to do falls under what they're doing already." Jessica Stepman and Rebecca L. Batterman, phone conversation with GiveWell, August 10, 2010.
- 7. "We have 20 staff in the office at our headquarters in New York and about 30 around the country. The regional staff members assess what the academies need help with, and they're very involved in the regions, with the directors, and in what is going on at those schools. For example, they check that academies use our curriculum and use it the right way." Jessica Stepman and Rebecca L. Batterman, phone conversation with GiveWell, August 10, 2010. "We felt like missing from the “What they do” section is a sense that the curriculum development, teacher professional development, and technical assistance are ongoing supports for our academies." Colleen Devery, e-mail to GiveWell, November 11, 2010.
- 8. In 2008, the National Academy Foundation spent $771,828 on grants out of total program expenses of $9,735,108. National Academy Foundation, "IRS Form 990 (2008)," Pgs 1-2.
- 9. "During the 2009-2010 school year, NAF Academies operate in 376 public high schools nationwide and serve 219 school districts in 41 states,the District of Columbia and the U.S. Virgin Islands. NAF supports a national network of: * 243 Academies of Finance * 95 Academies of Hospitality & Tourism * 123 Academies of Information Technology * 29 Academies of Engineering." National Academy Foundation, "Statistics and Research." "Each year, NAF serves more than 50,000 students in over 500 Academies in 40 states and the District of Columbia through the Academy of Finance, the Academy of Engineering, the Academy of Hospitality & Tourism, and the Academy of Information Technology." National Academy Foundation, "Annual Report (2008)," Pg 24.
- 10. Publications from the evaluation include Kemple and Willner (2008), and Kemple and Snipes (2000).
- 11. "The participating programs were drawn from most of the major established networks of Career Academies across the country, with four from the California Partnership Academy network, two from the National Academy Foundation network, one from the Florida network of Academies for Career Development and Applied Technology, and one from the network of Academy programs created by the District of Columbia Public Schools. Two of the participating Academies were developed independently through local high school or district initiatives." Kemple and Snipes 2000, Pg 16.
- 12. "We have regional directors responsible for maintaining contact with career academies in their respective region, and for directing the academies to people who can help. And we're piloting a deeper set of supports: where we now have one person we will expand to a team of 5 to 6 experts. We will still have the director but also more dedicated support on the ground... We have been consistently monitoring programmatic data, and our regional directors know in detail what is going on in each academy." Colleen Devery, phone conversation with GiveWell, August 26, 2010.
- 13. "I can provide you with information on the NAF career academies self-assessment tool, sample data from one (anonymous) academy, and the aggregate data. We piloted it in 2009 on half of the NAF career academies." Colleen Devery, phone conversation with GiveWell, August 26, 2010. The assessment tool is National Academy Foundation, "Self-assessment Standard Version." Summary reported data from the tool is in National Academy Foundation, "Self-Assessment 1-20."
- 14. Orr et al. 2004.
- 15. "In all, there were 177 completed interviews (78 percent of those with viable contact information)." Orr et al. 2004, Pgs 16-17.
- 16. Orr et al. 2004, Pg 50, Table 11.
- 17. Orr et al. 2004, Pg 53, Table 12.
- 18. "Data from the alumni also suggest that the academies strengthened student interest in the academy-related industries, because the alumni pursued further employment and careers in these fields. Indeed, most alumni (90 percent) continued to work in the academy-related industries after high school and after college. For some, this employment was with an academy-affiliated employer specifically. ... Of the alumni who worked the fall after high school graduation, 38 percent had positions with their summer internship employer or another affiliated employer (Table A-16). Five and ten years after graduation, 28 percent of working alumni were employed in jobs that were directly related to their academy-affiliated industries of finance and travel and tourism, while 54 percent found employment in the affiliated field of business. Moreover, 5 percent of working alumni were still working for their original internship employer or an academy-affiliated employer (not shown)." Orr et al. 2004, Pg 55.
- Expenses from National Academy Foundation, "Financial Statements and Schedule December 31, 2009 and 2009 (With Independent Auditors' Report Thereon)," Pg 3.
- Number of academies created from National Academy Foundation, "New vs Existing Career Academies."
- Number of academies in the network: "Each year, NAF serves more than 50,000 students in over 500 Academies in 40 states and the District of Columbia" National Academy Foundation, "Annual Report (2008)," Pg 24.
- Regarding NAF's ongoing support of NAF career academies: "GiveWell: What type of support does NAF provide to career academies over time? NAF: We do professional development for the teachers and develop courses and curriculum to help the students develop the skills they need to pursue the career of their choice. We don't hire the teachers or have a NAF person come in and teach the class, instead we give the teacher support to teach in accordance with NAF's methodology. What we do is mostly professional development, developing curricula and courses, research, starting internship programs, and providing the academies support on how to reach out to the community to get internships. There are more details than this, and I can send you more information about all of this." Jessica Stepman and Rebecca L. Batterman, phone conversation with GiveWell, August 10, 2010.
- 20. "The per-student cost of Career Academies varies widely depending on the specific features of the school. One estimate is that, in California, a high school operating a three-year Career Academy (grades 10-12) incurs an additional cost of approximately $675 per Career Academy student, per year in 2009 dollars." Coalition for Evidence-Based Policy, "Career Academies." Note that we have not been able to determine how that estimate was reached. We would guess that the source for this claim is: "Each state-funded Academy receives a yearly grant. Only those students that meet attendance and credit standards established in law receive funding. The maximum grant is $81,000 per year, which averages about $600 per student. Most Academy support comes from in-kind local sources, especially people's time: teacher time to coordinate the many program elements, and employee time to be speakers, mentors, and job supervisors." Tech Prep California, "The Kern/South Tulare Consortium." However, this source is relatively old, is only for the state of California, and does not discuss whether schools incur additional costs above and beyond those that the grant pays for. (We would guess that the Coalition for Evidence-Based Policy adjusted this $600 average for inflation yielding $675 in 2009 dollars).
"GiveWell: What is your room for more funding? I.e. what specifically would you do if you got additional funds? We read your i3 application but we are unclear about what specific programs you intended to implement.
NAF: I'll describe the work we hoped to do, have undertaken to some degree, and would do at an increased pace with more funding.
The first is to test out the technical assistance and service delivery model. For the last seven or eight years, we've had the country divided into regions, each with a director who act as the sole technical assistance provider. Although, if the issue is outside the director's particular realm of experience, the director helps the academy get the appropriate assistance.
In a pilot starting this summer we moved to a four or five person team in two regions of the country. A team includes
- the Director, which is a similar role to old regional director and they are responsible for deploying the team;
- a Partnership Manager who works with academies on involving the community, their advisory boards, connecting to businesses for volunteers and internships;
- a Academy Development Manager who is an expert in the structure of NAF academes, for example they could work with the school on scheduling;
- a Curriculum Specialist who can work with the teachers on implementing the NAF curriculum, creating integrated projects across subject areas, etc.;
- and a part-time Network Liaison who handles the administrative needs of the team
- 22. "GiveWell: Do you have a priority among field services rollout, internship support, and developing the health services theme? NAF: Different funders support each of these activities, so the priority depends on the funders agreeing to support the activity. In most cases the funders we are approaching for each of these initiatives are specifically interested in that work. GiveWell: What would you do with unrestricted funding? NAF: For the most part, we're raising restricted funding. Of course, we are trying to increase unrestricted funding but it's more difficult to raise. Most of our technical assistance support (the field teams) is paid for using unrestricted funds." Colleen Devery, phone conversation with GiveWell, October 26, 2010.
- 23. "We average about 30-50 academies each year, and are planning to add at least 60 new academies in 2011 which will significantly add to the number of students we reach. The new academies added in Health Sciences would be on top of that." Colleen Devery, e-mail to GiveWell, November 11, 2010.
- 24. "GiveWell: Do you think about allocating resources to creating more NAF career academies? NAF: That's not as much of an issue for us because the bottleneck to creating more NAF career academies is not our capacity but rather, in our view, the academies readiness to implement the model. There were about 57 academies that applied last year and we chose about 47 for implementing the NAF career academy model (these are approximate figures off the top of my head). And we have the ability to serve 40-60 academies per year, so we're not nearing the limit of our capacity. We work with the academies that we didn't accept to assist them in putting the right team in place and reapplying next year if that's what's right for them." Colleen Devery, phone conversation with GiveWell, October 26, 2010.
- 25. "Strategically grow the NAF network ($1.5M): NAF will focus on expanding its network through a “saturation” strategy, focusing on cities and districts that are already advocates for the career academy model. NAF will develop “wall-to-wall academies” in which one school building will house two or more NAF academy themes, and help academies recruit more students, particularly student populations that are traditionally underserved. NAF will also add new academies to the network in key geographic areas, with particular emphasis on the Academy of Engineering to reach out goal of 110 academies in this theme by 2012. Another strategy for growth is the Academy of Health Sciences, which like the engineering theme, will address the critical achievement gap in STEM and contribute to the pipeline of students prepared to pursue STEM-related degrees and professions." National Academy Foundation, "Strategic Imperatives (2011)."
- 26. "Improve the quality of academies and performance across the network ($3.0M): NAF will continue developing and disseminating improved tools and services designed to help academies implement the model fully in their schools and achieve student outcomes. NAF will work with each academy to evaluate the strengths and areas of improvement, and develop a professional development plan for each school including coaching and technical assistance by NAF staff, regional convenings for teachers, and the Annual Institute for Staff Development where teachers, administrators, and advisory board members will learn and share best practices with field experts and each other. Professional development includes the launch of eCollege, a new online campus where Academy teachers and network members can participate in professional development courses and workshops that help them to refine their instructional practice and fully implement the NAF model. NAF will also continue collecting and analyzing data to demonstrate the efficacy of the NAF model, deploy resources strategically, share learnings across the network, and identify areas for program growth." National Academy Foundation, "Strategic Imperatives (2011)."
- 27. "Create a nationally-recognized NAF Certificate ($1.0M): NAF is currently working on the development of a NAF Certificate that will add value to its efforts to prepare students for college and will become a portable credential of a student's rigorous academy experience. The Certificate will demonstrate proficiency in 21st Century Skills through a cumulative portfolio of student projects, end-of-term assessments, and an internship evaluation. The certificate will be one of the few multiple assessment systems for career education in the country emphasizing both content knowledge and general workplace skills. NAF will strengthen the Certificate through articulation agreements and partnerships with colleges and universities, who will agree to recognize the NAF Certificate, recruit NAF students to their campuses, and lend expertise in content areas as needed. " National Academy Foundation, "Strategic Imperatives (2011)."
- 28. "Impact public policy and school reform ($200k): NAF's work to position the organization as a major participant in broader high school reform efforts and the career academy as a key strategy has taken the form of advocacy on the state and federal level. JD Hoye recently met with several staffers in Washington, DC as well as Senator Kay Hagan of North Carolina. Additionally, NAF is a member of America's Promise and the National Coalition of Career Academies, and participates in school reform convenings such as the Pathways to Prosperity Conference." National Academy Foundation, "Strategic Imperatives (2011)."