Race to the Top Evaluation: STEM Affinity Network—Third-Year Report

Executive Summary

Overview of the Evaluation and Progress Made since the Last Report

This third annual report on the North Carolina Race to the Top (RttT) STEM initiative documents ongoing implementation of the initiative in participating anchor and affinity schools and assesses intermediate outcomes for students and staff in the third year of implementation.

Two research questions guide the evaluation:

  1. To what extent have the four key elements of the network of STEM anchor and affinity schools (network structure, professional development, curriculum, and partnerships) been implemented as intended?
  2. What are the intermediate outcomes for students and staff in network schools in the third year of implementation?

In addition, this report notes recommendations from the Year 2 evaluation report that were addressed during the second year of implementation. In particular, the Implementation Team:

  • Created additional opportunities for staff in network schools to interact online through Edmodo professional development networks and Twitter chats; and
  • Added a number of online professional development opportunities, such as webinars for principals and counselors, and virtual sessions for Secondary Lenses on Learning course.

Findings and Recommendations[1]

Based on analyses of RttT STEM initiative activities to date, the Evaluation Team concluded that structures for networking, professional development, curriculum development, and partnerships are in place to support anchor and affinity schools, as intended. While all four areas of implementation have received attention from the Implementation Team, some progressed more than others. In many cases, there are components within each area that moved faster or slower relative to other components. Additionally, staff and student surveys reveal that after one year of implementation, a subset of the affinity schools (the comprehensive schools) lag behind the anchor schools and the other affinity schools (small new schools and STEM Academies) in all four areas of implementation, as well as in intended student outcomes.

I. Structure of the Network of STEM Anchor and Affinity Schools

  • The greatest amount of activity on the online networking platform Edmodo was produced by the two new content-focused networks (created by NC New Schools staff in fall 2012) and two school networks composed of teachers and students.
  • Twitter chats, a newly-introduced mode of communication among schools, have been successful in attracting participants and fostering discussion of STEM-related topics.
  • Staff in two Local Education Agencies (LEAs) that adopted the RttT STEM initiative for their high schools LEA-wide saw increased collaboration among those schools.
  • Overall, one year into the project, teacher surveys indicate that the rates for both face-to-face and online collaboration among schools were not high. Networking among RttT STEM schools lagged behind implementation of other initiative components.
  • The extent of face-to-face networking and collaboration was higher than the extent of online networking.
  • While teachers and principals found face-to-face networking at various professional development events very valuable, they also found these encounters too brief to provide a basis for the subsequent ongoing collaboration.

Recommendations:

  • To increase networking among school staff, the Implementation Team should continue to explore various online modes of communication—such as online follow-up sessions to professional development events and online groups created by instructional coaches—for follow-up after face-to-face visits.
  • To increase collaboration among school staff, the Implementation Team should consider facilitating cross-school working groups that have a common goal of creating products for the participants’ schools, such as unit plans, common authentic assessments, or planning extra-curricular STEM activities.
  • To address geographical isolation of rural schools, the Implementation Team should consider facilitating collaboration within themed networks by combining face-to-face and online modes of collaboration.

II. Professional Development

  • Schools are continuing to receive the professional development and coaching services outlined in the RttT scope of work. Based on staff surveys, 86% of staff participated in workshops or professional development offered by NC New Schools Project, and 70% of staff participated in instructional coaching provided by STEM network coaches.
  • In most cases, both participants and outside observers perceived the professional development to be relevant and in the medium to high range of quality (3.4 on a 5-point scale). In many cases, the Scaling STEM Conference and on-site professional development at one particular school lowered the average observational quality ratings.
  • Sessions at the Scaling STEM Conference received the lowest average ratings (2.67) among all observed professional development events on three of the four indicators for STEM content quality, as well as on the quality of design, implementation, and culture of professional development.
  • Instructional coaching in STEM schools mostly focused on the Design Principles and Common Instructional Framework (CIF), with much less attention paid to student project work or to developing each school’s STEM theme.
  • Challenges and barriers related to professional development included:
    • Providing effective professional development to all teachers;
    • Limited high-quality coaching time; and
    • Lack of sufficient time for teachers to effectively implement and explore new instructional strategies in the classroom.

Recommendations:

  • The Implementation Team should consider expanding opportunities for participants to engage more actively during sessions at future STEM Conferences.
  • The Implementation Team should consider increasing the flexibility of the coaching content based on each school’s specific needs, as well as either increasing the amount of STEM-specific coaching or combining CIF with STEM-specific coaching.
  • The Implementation Team should consider new ways of bringing the most essential professional development events (such as the New Teacher Institute, Critical Friends Group, or Common Practices Symposium) to all teachers in each school. For example, the Team could increase coaching efficiency by coaching teachers in teams in addition to individually.
  • The Implementation Team should consider devoting more coaching time to lesson- and project-planning that incorporates new instructional strategies.

III. Development and Implementation of Project-Based Curricula

  • Curriculum design work was completed in accordance with the contract awarded to the North Carolina School of Science and Mathematics (NCSSM) by the North Carolina Department of Public Instruction (NCDPI).
  • Reviews by non-participating STEM teachers of the newly-developed STEM curricula point to many strengths as well as areas for improvement of the materials.
  • A number of anchor and small new schools are developing integrated STEM courses that parallel those being developed by NCSSM.
  • Implementation of the newly-developed curricula presents the following challenges:
    • Finding opportunities to pilot test STEM courses with targeted population of students and teachers and to fine-tune them to better serve the needs of the users.
    • Finding teachers with the right background knowledge to teach Career and Technical Education courses with the intended levels of rigor and depth of STEM content coverage.
    • Finding ways to provide necessary professional development for teachers of STEM courses to familiarize them with the new themed content and the new teaching strategies these materials call for.
  • NC New Schools devoted a number of sessions in professional development events to the four STEM themes, to project design, and to project-based learning. NC New Schools also conducted a special event to bring together STEM curriculum developers and users.
  • One year into the initiative, the majority of teachers have not yet participated in the development of STEM cross-curricular projects. Participation was much higher in the anchor and small new schools.

Recommendations:

  • To address concerns about the level of rigor in math and science covered in the integrated courses, the Implementation Team should consider including in the Teacher Guide a clear guidance to schools on the level of rigor in STEM content areas teachers of these courses are expected to provide.
  • The Implementation Team should consider finding opportunities to conduct pilot testing of the integrated STEM courses and curriculum revisions before making the courses available for broader use.
  • To address the issue of finding qualified teachers with sufficient background knowledge for teaching integrated courses, the Implementation Team should consider including in the Teacher Guide a clear description of the background teacher knowledge desirable for teaching each of the courses to provide guidance to schools implementing these courses. Additionally, the Implementation Team should consider offering multiple ways to implement these courses (e.g., via team-teaching).
  • The Implementation Team should consider finding ways to provide necessary professional development for prospective teachers of STEM courses to familiarize them with the new themed content and the new teaching strategies these materials call for.
  • To address the duplication of efforts by the various organizations and schools that are developing integrated STEM curricula, the Implementation Team should consider conducting regular meetings of curriculum developers to encourage collaboration.

IV. Partnerships

  • Industry Innovation Councils for each of the four themes met quarterly to plan and provide support for the networks; additionally, various business partners also supported teachers and students in a variety of ways (e.g., student internships, teacher summer externships, etc.).
  • One year into the program, most staff in the 20 affinity schools have not yet participated in a collaborative activity with partners, such as collaboratively designing a unit or project for the classroom. Anchor schools participated in partnerships at a higher rate than did other schools.
  • Three of the affinity schools visited by the Evaluation Team established their own partnerships with local businesses and colleges. The initiative strengthens the focus and breadth of these partnerships.

Recommendations:

  • To increase levels of participation in partnerships with businesses, the Implementation Team should consider helping schools to set up certain goals and foci for such collaborations and to facilitate sharing of successful stories of collaborations with staff in every participating STEM school. Webinars or online chats may be conducive to engaging a wider audience.

V. Student and Staff Responses to Implementation and Outcomes Surveys

Survey responses from students in 20 STEM network schools suggest that one year into program implementation:

  • Many students placed a very high value on learning in general and on learning mathematics in particular. Students had a moderately high level of confidence in their ability to be successful in their studies of mathematics, science, and technology. Areas with the most room for improvement include: enjoyment of learning (in general, and of STEM subjects in particular) and student engagement in the engineering aspects of STEM (from initial exposure to development of confidence in learning about engineering).
  • The quality of classroom instruction and school culture was mixed. Students generally reported high expectations and care from their teachers and high levels of meaningful use of technology. At the same time, a number of desired activities were not reported as frequent, such as student engagement in cross-curricular or real-life projects, in rigorous instructional practices, or in STEM-related activities supported by the school.
  • Student outcomes, the quality of the classroom instruction, and the school culture differed among the types of schools, with students in comprehensive schools consistently reporting lower levels of desired features than did students in anchor schools, small schools, and academies.

Survey responses from staff in 20 STEM network schools suggest that one year into the program implementation:

  • Many teachers felt that they were comfortable with many of the target instructional strategies, and implemented them fairly frequently. Additionally, many teachers reported having positive relationships with students.
  • There were few extra-curricular STEM activities, additional STEM courses, cross-curricular projects for students, or staff meetings devoted to STEM issues. Staff also reported that not everyone at their school understood what it meant to be a STEM school. At the same time, two-thirds of respondents reported that their schools were focused on a STEM-related goal for students and that their schools emphasized their STEM theme in a number of different ways.
  • Staff responses differed among types of schools on many of the dimensions, with staff in comprehensive schools typically giving lower ratings than did their peers on staff-student relationships, meetings about STEM issues, using technology, extra-curricular STEM activities, and STEM vision.

VI. Site Visits to Affinity Schools

  • In all of the affinity schools visited by the Evaluation Team, the STEM initiative is in the beginning stages of implementation.
  • In all four of these schools, a common vision for the STEM initiative focuses on instructional improvement that includes the following elements:
    • Incorporation of project-based learning; and
    • A focus on the development of students’ critical thinking, understanding, problem solving, and communication skills.
  • In two schools, staff exhibited a high degree of initiative buy-in. In the other two schools, buy-in is still an area for improvement.
  • The primary focus of implementation is on changing instruction to incorporate Common Instructional Framework (CIF) across subject areas; a secondary focus is on development of STEM projects and themes.
  • Regarding technology use, a common goal among schools is to shift away from use by teachers and to increase technology use by students.
  • The single most notable impact of the initiative on students reported across all schools was an increase in student engagement.
  • The challenges for implementation fall into five main categories: 1) logistical, time, and resource challenges; 2) student, faculty, and community buy-in; 3) implementation of STEM curriculum and instruction; 4) sustainability; and 5) relationships with the wider community.

Recommendations:

  • To address logistical problems faced by some schools related to transportation, lack of resources for technology and projects, geographical isolation, and lack of time for planning, the Implementation Team should encourage other schools to share their best practices in solving these problems either face-to-face or online.
  • To help schools define what this initiative means for the school and to get community, faculty, and student buy-in, the Implementation Team should consider identifying schools and communities with those issues and providing them with more opportunities for visiting model STEM schools.
  • To address schools’ concerns about the sustainability of funding for program components post-RttT, the Implementation Team should consider including discussions about sustainability in professional development events.

Next Steps

  • Continue qualitative data collection and analyses.
  • Analyze responses to staff and student surveys that will be collected in fall 2013 and compare them to earlier survey responses.
  • Provide a summative evaluation of the initiative’s components and intermediate outcomes, as well as recommendations related to the sustainability of the initiative.

1A response from North Carolina New Schools to these findings and recommendations is included in Appendix J of the main report.

View Resource