NC RttT Overall Impact and Implementation Findings: Final Report

Executive Summary

Background

North Carolina’s Race to the Top Plan

The ultimate goal of North Carolina’s Race to the Top (RttT) plan was to build statewide capacity that could support sustained, long-term improvements in public education; namely, increases in student achievement, reductions in achievement gaps, and increases in graduation rates. To address these outcomes, North Carolina’s proposal focused on strengthening the education workforce by providing great teachers for every student and a great principal for every school. Initiatives were designed to provide a coherent approach with mutually-reinforcing components across multiple reform pillars.

The strategies funded by RttT were required to address the federal grant program’s four targeted pillars of reform:

  1. Ensure the state standards and accountability system reflects internationally benchmarked standards;
  2. Establish advanced data systems that measure student success and inform educator practice;
  3. Increase teacher and principal effectiveness; and
  4. Turn around the State’s lowest-achieving schools, so that all students get the support they need to be successful.

North Carolina’s Race to the Top Evaluation

North Carolina’s RttT proposal included a commitment to an independent external evaluation of individual initiatives and of the overall impact of the reform efforts across the state. For this evaluation, the North Carolina Department of Public Instruction (NCDPI) contracted with the Consortium for Education Research and Evaluation–North Carolina (CERE-NC) to provide formative and summative feedback on implementation and outcome findings.[1] As of July 2015, the Evaluation Team has completed over 50 evaluation reports across 13 state and local initiatives.[2]

Evaluation Questions

The purpose of this report is to assess the overall, statewide impact of North Carolina’s RttT efforts over the course of the four-year grant period. Achieving this goal in full is somewhat limited by the nature of the implementation of the initiatives, which in most cases prevented our ability to identify valid comparison groups— groups who are like the group impacted by RttT initiatives in almost every way except that they were not exposed to RttT initiatives. Despite this key limitation, the Evaluation Team was able to use a three-pronged mixed-methods approach to chart the performance of North Carolina’s education system during the years of RttT implementation (2010-11 through 2013-14). First, intra-state changes were assessed for key outcomes before and after RttT implementation by using North Carolina’s rich data to determine whether North Carolina exceeded anticipated performance based on prior data. Second, inter-state comparisons were made between North Carolina and selected non-RttT states on common national education measures. Finally, implementation findings and recommendations were synthesized from previous reports with input from members of the Evaluation Team in order to provide qualitative information about implementation and potential sustainability of individual-level initiatives.

Taken together, these strategies were used to address the following five quantitative and qualitative evaluation questions:

Quantitative

  • Evaluation Question 1: Exceeding Expected Performance: How are North Carolina publicschools performing after the implementation of RttT in relation to their expectedperformance based on pre-RttT data?
  • Evaluation Question 2: Exceeding Growth in Other States: How does North Carolinaperform on common measures of educational outcomes compared to selected groups ofstates.

Qualitative

  • Evaluation Question 3: Fidelity of Implementation: To what extent was each RttT initiativeimplemented as intended with regard to timeline, scale, and objectives?
  • Evaluation Question 4: Enhanced Capacity: To what extent has RttT enhanced NorthCarolina’s capacity to support the four RttT pillars of reform and strengthen the education workforce?
  • Evaluation Question 5: Sustainable Capacity: What initiatives demonstrate the capability tosustain implementation and promise toward ultimately impacting student and teacherperceptions, behaviors, and achievement?

Data and Methods

Quantitative Analysis – To determine whether and the extent to which North Carolina’s performance exceeded expected performance, the Evaluation Team used prior data on schools, teachers, and students to establish the state’s trajectories on multiple important outcomes (teacher effectiveness, student achievement, and the achievement gap). By comparing these pre-RttT trajectories to trajectories of the same measures during the RttT years, we generated one estimate of the effects of RttT.

To determine North Carolina’s performance relative to other states, growth on ten national education measures was compared for North Carolina and several other groups of states, including other RttT grant recipients, all other non-RttT awardees, seven neighboring states in the southeast,[3] and the ten non-RttT recipients with scores closest to North Carolina on each outcome prior to RttT.[4]

Qualitative Analysis – RttT evaluation leads were surveyed to gauge (based on their initiative-level evaluations) their perceptions of the extent to which each initiative met identified targets for the three qualitative evaluation areas of interest: Implementation, Sustainability, and Outcomes. Leads were asked to rate objectively elements in each of the three categories as well as to provide narrative justification and/or clarifications of those ratings.

In addition, findings from the initiative-level evaluation reports were used to assess the extent to which each initiative showed promise for achieving immediate, intermediate, and longer-term outcomes and for enhancing the state’s capacity to support the four RttT reform pillars.

Findings

Evaluation Question 1: Exceeding Expected Performance

Exceeded expected performance: Overall, North Carolina exceeded its expected performance in two of six education outcomes: 1) student graduation rates continued an upward trend that began before RttT implementation for an overall increase of 9.5 percentage points over the last three years of the grant; and 2) graduation attainment gaps narrowed. The graduation attainment gap between white students and minority students was cut in half, from 13.9 percentage points for the graduating class of 2009 to 7.0 points for the class of 2014, and the graduation attainment gap between economically disadvantaged students and their less-disadvantaged peers narrowed from 14.8 to 10.6 percentage points over the same time period.

Met expected performance: There were no significant differences found in teacher effectiveness as measured by principal evaluation ratings, teacher value-added, or in the narrowing of the racial and economic achievement gaps.

Declined in performance: Only one outcome—student achievement—declined over the RttT period, but the decline may reasonably be attributed to North Carolina’s adoption of new achievement exams aligned with the Common Core State Standards, along with adoption of higher proficiency standards.

Outcomes with mixed results: Teachers’ perceptions of their work environments and practices were mixed. Responses on the Omnibus Teacher and Leader survey administered annually by the Evaluation Team indicate that teachers felt that they improved in areas targeted by the state (knowledge-sharing and formative assessment), but that they did not perceive positive changes in the actions or services provided by the state, such as professional development opportunities or teacher evaluations.

Evaluation Question 2: Exceeding Growth in Other States

Overall, findings from inter-state comparisons across ten national scholastic outcomes suggest that RttT states performed similarly to non-RttT recipients during the period of the federal grant on most outcomes but may have improved faster on National Assessment of Educational Progress (NAEP) reading scores. North Carolina posted greater gains than other states on NAEP reading and 8th grade mathematics scores, Scholastic Aptitude Test (SAT) mathematics scores, and cohort graduation rates but posted smaller gains on Advanced Placement exam-taking and saw SAT reading and writing scores decline.

Evaluation Question 3: Fidelity of Implementation

Out of the eleven state-level initiatives evaluated, nine started on time or within a year of planned implementation, eight were implemented to scale, and eight met their implementation objectives.

Evaluation Question 4: Enhanced Capacity

Over the course of the grant, North Carolina implemented initiatives that enhanced capacity in each of the four RttT pillars, including: statewide professional development to enhance local capacity to transition successfully to new Common Core State Standards; an online Instructional Improvement System; a revised teacher and principal evaluation process that includes new measures of student growth; and an expanded school turnaround effort to improve the proficiency of the state’s lowest-performing schools.

Evaluation Question 5: Sustainability

Nine initiatives met at least three of four sustainability characteristics (structure, personnel, infrastructure, and ongoing demand). In addition, initiative-by-initiative progress toward attaining immediate, intermediate, and longer-term outcomes was encouraging, with five initiatives achieving all of their intended outcomes and four at least partially meeting all outcomes.

Limitations

There are several key limitations to an evaluation of this scope that should be considered when assessing the estimations in this report of the impact of this multi-year state-level intervention comprised of multiple initiatives: 1) the absence of valid comparison groups precluded the ability to establish causality; 2) the presence of competing or conflating initiatives may have influenced outcomes of interest; 3) other external factors such as changes in leadership and the economic climate also may have influenced outcomes; and 4) the length of time between initial implementation and measurement of some outcomes may not be sufficient to capture the eventual effects of some initiatives.

Conclusions

This overall impact evaluation of North Carolina’s RttT plan revealed notable successes in program implementation, the presence of key sustainability factors, and improvements in the critical education outcome of increased student graduation rates. The comparison between North Carolina and other states also demonstrated that North Carolina posted faster gains than other states on some important academic measures.

While analyses of other key education outcomes (such as teacher effectiveness) did not result in significant positive findings, and some measures of student achievement declined in the wake of adoption of new standards, taken as a whole, the evaluation data support the conclusion that North Carolina’s education infrastructure and important markers of educational attainment are better off now than they were before the start of RttT.

The state’s commitment to the more promising RttT initiatives, especially school turnaround and transformation, should not end once grant funding has been exhausted. In order to maintain these successes and improve opportunities for all students to receive an adequate education, the state should continue to commit time and resources to sustaining the successful components of RttT for the foreseeable future.

Emerging Lessons Learned

Not everything worth learning about North Carolina’s RttT efforts can be gleaned from analyses of data alone. For this final overall evaluation, the Evaluation Team also reviewed the findings and recommendations in all of the individual initiative evaluation reports that preceded this final report and identified several emerging lessons learned—potentially of use not only to implementers but also to grantors—that cut across multiple initiatives:

  • Prioritize and actively promote collaboration throughout the reform process. Communication, a coordinated vision for implementation, and collaboration among leaders can reduce conflicting agendas and implementation challenges in a complex reform process.
  • Value simplicity. Usually, the most successful initiatives are those with the fewest moving parts.
  • Engage the broader community. A comprehensive education reform agenda attends to in-school and out-of-school stakeholders at multiple levels.

1 CERE-NC is comprised of qualitative and quantitative researchers from the Friday Institute for Educational Innovation at North Carolina State University, the Education Policy Initiative at Carolina at the University of North Carolina-Chapel Hill, and the SERVE Center at the University of North Carolina-Greensboro.
2 All evaluation reports for each initiative are available at https://www.fi.ncsu.edu/projects/evaluation-of-race-to-the-top/
3 The neighboring states comparison includes Virginia, West Virginia, Kentucky, Alabama, Mississippi, Louisiana, and Arkansas. These states were chosen because they were not RttT recipients.
4 The comparison group of states with similar scores on outcomes was chosen by selecting the 10 states which did not receive RttT funds that had scores closest to North Carolina on each outcome in 2010 or in 2009 for NAEP scores.

View Resource