Consortium for
Educational
Research and
Evaluation–
North
Carolina
The Golden LEAF STEM Initiative
Evaluation
Year Two Report
Malinda Faber, Meredith Walton, Sherry Booth, Brandy Parker,
and Jeni Corn, Friday Institute for Educational Innovation
Eric Howard, SERVE Center at UNCG
May 2013
We would like to thank the leaders and participants of The Golden LEAF STEM Initiative
Grants for their collaboration and partnership.
This project received support from The Golden LEAF Foundation.
The Golden LEAF Foundation
301 N. Winstead Avenue
Rocky Mount, NC 27804
(252) 442-7474
www.goldenleaf.org
Golden LEAF STEM Initiative
May 2013
Table of Contents
Table of Contents ............................................................................................................................ 1
Executive Summary ........................................................................................................................ 3
Introduction ................................................................................................................................... 11
The Golden LEAF Foundation STEM Initiative ....................................................................... 11
The Structure and Purpose of the Evaluation of the Golden LEAF STEM Initiative ............... 12
Structure of this Report.............................................................................................................. 14
I. Data Sources and Analyses........................................................................................................ 15
Grant Coordinator Interviews .................................................................................................... 15
Site Visits................................................................................................................................... 15
Focus Groups with Participating Teachers ................................................................................ 16
Classroom Visits ........................................................................................................................ 16
Teacher Efficacy and Attitudes toward STEM (T-STEM) Surveys.......................................... 17
Student Attitudes toward STEM (S-STEM) Surveys ................................................................ 19
Pilot Leadership for STEM Self-Assessment ............................................................................ 21
The Golden LEAF STEM Implementation Rubrics .................................................................. 21
Administrative Data on Student Performance ........................................................................... 23
II. Findings .................................................................................................................................... 24
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Student Attitudes toward STEM Education?............................. 24
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Student STEM Learning? .......................................................... 37
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Teachers’ Instructional Practice? ............................................... 41
Additional Findings ................................................................................................................... 63
III. Capacity-Building Activities .................................................................................................. 67
Initiative- and Grant-Level Survey Results Reports.................................................................. 67
Golden LEAF STEM Implementation Rubric Grant-Level Results ......................................... 68
Summer STEM Evaluation Institute 2013................................................................................. 68
The Golden LEAF STEM Initiative Wiki ................................................................................. 68
IV. Recommendations................................................................................................................... 69
Continue to Implement Hands-On, Problem-Based STEM Curricula and Activities and
Increase Emphasis on Rigor ...................................................................................................... 69
1
Golden LEAF STEM Initiative
May 2013
Continue to Raise Student Awareness of STEM Careers and Increase Opportunities for
Students and Teachers to Engage with STEM Industries; Focus on Females in Engineering;
Further Relationships between Schools and Industry................................................................ 70
Continue Providing Opportunities for STEM Teachers and Others to Collaborate and Focus on
Ways to Support Cross-Curricular Integration .......................................................................... 71
Increase Professional Development Opportunities that are Hands-On, Content-Specific, GradeLevel Specific, Led by Lead Teachers, and that Offer Immediate Classroom Solutions; Provide
More Time for Teachers to Plan, Experiment, and Implement ................................................. 71
Find Ways to Have Safe, Professional Conversations about Teaching Philosophies and Beliefs
................................................................................................................................................... 72
Continue to Invest in Sustainability Planning; Continue to Collect Data about the Progress of
Programs and Use them to Strategically Plan for the Future .................................................... 72
V. Next Steps ................................................................................................................................ 73
References ..................................................................................................................................... 75
2
Golden LEAF STEM Initiative
May 2013
THE GOLDEN LEAF STEM INITIATIVE EVALUATION YEAR TWO REPORT
Executive Summary
Student success in the core content areas of science, technology, engineering, and mathematics
(STEM) has emerged as an essential component in the development of an American workforce
that can compete in the global, 21st century economy. In response to this critical need states
across the country, including North Carolina, have developed K–12 public school initiatives
designed to inspire and prepare the next generation of scientists, mathematicians, and engineers.
In North Carolina the Golden LEAF Foundation (Golden LEAF) is a leader in these efforts to
promote and sustain high quality STEM education in public schools. In 2010 the Foundation
launched a STEM Initiative to support “successful models that increase STEM education for
students in grades four through nine in rural, economically distressed, and/or tobacco-dependent
counties of North Carolina.” The Foundation awarded grants to projects that:

Were evidence-based and represented systemic approaches to STEM education, including
in-school, out-of-school, or extended day and support programs providing assistance to
students transitioning from elementary to middle and middle to high school.

Represented collaborations among public schools and higher education, community, and
relevant industry partners.

Targeted improved preparation for and academic performance in advanced STEM
curricula by minorities, females, and students from limited-resource families.

Served students in 4th through 9th grades, placing priority on curricular approaches that
were integrated, used project- and inquiry-based learning concepts, and/or prepared
students for successful completion of Algebra 1 by 8th or 9th grade – a gateway to
participation in advanced placement courses.

Included strategies that incorporated content-specific professional development for
teachers, and provided relevant career and work connections for teachers and students.
In the spring of 2011, fourteen grantees were selected and funded up to $750,000 for a three-year
period. In total, these grants impact 43 public school districts in North Carolina, 225 schools,
approximately 1,190 teachers, and approximately 31,890 students.
The Golden LEAF STEM Initiative Evaluation
In 2011 the Golden LEAF STEM Initiative evaluation team was charged with completing a
formative and summative evaluation and acting as a resource for the participating grantees who
would be conducting some evaluation of their own. The evaluation of the Golden LEAF STEM
Initiative would take place over the three-year grant implementation period, from 2011 through
2014. The research is now being conducted by the Consortium for Educational Research and
3
Golden LEAF STEM Initiative
May 2013
Evaluation–North Carolina (CERE–NC), a partnership of the SERVE Center at the University of
North Carolina at Greensboro, the Carolina Institute for Public Policy at the University of North
Carolina at Chapel Hill, and the Friday Institute for Educational Innovation at North Carolina
State University. The evaluation does not separately examine the activities and outcomes of
individual grants, but rather, it operates at the initiative-level, focusing on the overall
commonalities of the 14 grants’ activities and observing their common outcomes. The two
primary objectives of the Golden LEAF STEM Initiative evaluation are described below.
Evaluation Objective 1: Describe the Overall Effectiveness of the Initiative
The evaluation team’s first objective is to describe the overall effectiveness of the Golden LEAF
STEM Initiative in achieving its goal of improving STEM education outcomes for 4th through
9th graders in rural North Carolina. For this purpose quantitative and qualitative data are being
collected from multiple sources. Data are collected in order to answer four, primary evaluation
questions. These are, “To what degree or in what ways were the Golden LEAF STEM Initiative
grantees as a whole:
1. Faithful in implementing their STEM program’s criteria and goals?
2. Effective in changing student STEM attitudes?
3. Effective in changing student STEM learning?
4. Effective in changing teachers’ instructional practices?”
Results from three, annual periods of data collection are synthesized and compared annually. The
goals of these analyses are to provide useful information about the impact of the initiative as a
whole to Golden LEAF and to the grantees as they continuously build and improve their
programs.
Evaluation Objective 2: Evaluation Capacity-Building
The second objective of the Golden LEAF STEM Initiative evaluation is to provide technical
assistance on program evaluation strategies to the grantees as they work to continually improve
their own individual programs. The evaluation team assists each of the grantees to:

Develop and apply knowledge about education program evaluation; and

Collect, interpret, and use formative data to improve their STEM programs.
Over the course of the three-year initiative various capacity-building events and activities take
place: annual evaluation institutes, semi-annual webinars, the ongoing provision of formative
data, access to online surveys, and access to one-on-one technical assistance from members of
the evaluation team.
4
Golden LEAF STEM Initiative
May 2013
Report Structure
This report summarizes all data and results for Year Two of the evaluation, collected from
September 2012 through February 2013. Similar to the August 2012 report, this paper addresses
evaluation questions 2-4 by summarizing results from the following data sources: interviews
with grant coordinators; focus groups with participating teachers; surveys administered to
students; surveys administered to teachers; surveys administered to principals; classroom
observations; and a program implementation rubric completed in Year Two by principals. Taken
as a whole these results address the first evaluation question, regarding the faithfulness of the
implementation of the initiative. The report is divided into five sections: Data Sources and
Analyses, Findings, Capacity-Building Activities, Recommendations, and Next Steps.
Evaluation Results
To what degree or in what ways were the Golden LEAF STEM Initiative grantees as a whole
effective in changing student STEM attitudes?
The Golden LEAF STEM Initiative grantees all share the common objective of improving
student attitudes toward STEM subjects and increasing their interest in STEM careers.

Student engagement: Consistent with findings from Year One, teachers reported that
overall student engagement in STEM content was very high as a result of the hands-on,
problem-based learning opportunities provided through the Golden LEAF STEM
Initiative. Hands-on, problem-based activities specifically engaged students with a
variety of learning styles, including visual and mechanically-inclined learners, and they
also had a noticeable impact on engagement for struggling students and English as
Second Language (ESL) students. Results from classroom visits also support the overall
finding that the hands-on, inquiry-based STEM activities lead to higher student
engagement compared to other lessons.

Student awareness of industry: Findings from the 14 grant-coordinator interviews and the
focus groups with participating teachers indicate that opportunities for students to visit
STEM industries or tour facilities increased student awareness and interest in STEM
industries.

Student self-confidence in knowledge of STEM content: Findings from surveys indicate
that, on average, students feel somewhat neutral or slightly agree with statements such as,
“I feel good about myself when I do science” and “I am interested in what makes
machines work.” The survey results indicate no significant change in student attitudes
toward STEM between Year One and Year Two. Variation between students at different
school-levels was slight. Upper elementary school students reported slightly higher
confidence and interest toward mathematics, science, and engineering and technology,
while high school students reported the least positive attitudes.

Student interest in STEM careers: Student survey data indicate that students overall have
moderate levels of interest in STEM careers. On average, across 12 STEM career areas,
5
Golden LEAF STEM Initiative
May 2013
41.6% of students reported that they were “interested” or “very interested” in such
professional work. The greatest proportion of students indicated that they were
“interested” or “very interested” in veterinary work (51.1%), while the smallest
proportion of students reported that they were interested or very interested in careers in
physics (32.1%). Interest levels in computer science were higher in Year Two than in
Year One, jumping from 37.4% interested or very interested in fall 2011 to 42.1% in fall
2012. Findings show that female students have slightly lower interest in STEM careers
than males overall, including large differences in areas such as engineering, energy, and
computer science. Female students’ low interest in engineering correlates with the low
levels of confidence and interest they reported elsewhere on the surveys. The differences
in levels of interest in STEM careers between students of different races/ethnicities are
smaller than the differences between male and female students, which is consistent with
Year One findings. When comparing career interest by grade level, like in Year One,
upper elementary school students reported higher levels of interest across all STEM
career areas on average (49.9%) than both middle school students (38.7%) and high
school students (35.8%).

Student attitudes toward 21st century learning skills. Student attitudes toward 21st
century skills remain consistent at a 4.0 mean composite score from Year One to Year
Two. The survey data show that, also like in Year One, there is almost no variation
among the students’ attitudes toward 21st century skills when the learners are compared
by gender, race/ethnicity, or school-level.
To what degree or in what ways were the Golden LEAF STEM Initiative grantees as a whole
effective in changing student STEM learning?

Students’ problem-solving skills increase. The strongest focus group results related to
student learning in Year Two indicate that students’ problem-solving skills are increasing.
Teachers in almost every focus group described how the authentic, hands-on, inquirybased lessons were teaching students problem-solving skills that the young people had
never developed before. Many teachers described how the inquiry-based, hands-on
activities were giving way to higher-quality learning for students. Teachers described
how these challenging, problem-based instructional strategies were also building students’
confidence.

New materials and instruction better address mechanical and visual learners. The second
strongest finding from the 14 focus groups with participating teachers suggests that the
hands-on, inquiry-based STEM activities address a wider variety of learning styles
among students. Teachers implementing labs, experiments, and computerized simulations
through the Golden LEAF STEM Initiative described how these hands-on, problembased lessons not only favored most learners, but they especially connected with the
mechanical learners and strongly visual learners – students who learn best from practical,
applied experiences.

Students continue to develop communication and collaboration skills. Findings across
multiple data sources suggest that students participating in the Golden LEAF STEM
Initiative continue to have frequent opportunities to work together on meaningful tasks
6
Golden LEAF STEM Initiative
May 2013
and develop communication skills. Almost all of the STEM education kits, labs,
investigations, and curricula incorporate small group collaboration and team work.

Students improve their reading skills and willingness to read more challenging STEM
material. When asked whether they had noticed any changes in student learning as a
result of the new STEM activities and/or instructional strategies, a number of teachers
remarked that students’ literacy skills were improving. Students were also gaining
interest and seeking out new information on their own.

Students expect to do well; about half reported plans to take advanced mathematics and
science. Overall, the vast majority of students felt that they would do at least “okay”
(50.1 – 43.1%) if not “very well” (43.5 – 47.1%) in their ELA, math, and science courses.
Survey findings indicate that students’ performance expectations did not vary much by
gender, ethnicity, or school-level. Regarding mathematics specifically, results show that
overall 48.2% of students intended to take advanced classes in mathematics, with slightly
more females (50.0%) reporting that they had such plans than males (46.5%). When
asked whether or not they intended to take advanced classes in science, overall 42.6% of
students indicated that they would.
To what degree or in what ways were the Golden LEAF STEM Initiative grantees as a whole
effective in changing teachers’ instructional practice?

Teachers use hands-on, inquiry-based teaching strategies. Results from multiple data
sources indicate that the curricular materials, lab materials, technology, professional
development, and other instructional supports provided by the 14 Golden LEAF STEM
Initiative grants are helping increase the frequency with which teachers use hands-on,
inquiry-based, student-centered teaching strategies. The three most commonly used
STEM instructional activities by teachers who responded to the surveys were: (1)
“Students work in small groups” – 64.0% of teachers reported this happens “Usually” or
“Every Time” during instructional meetings; (2) “Students engage in content-driven
dialogue” – 62.1%; and (3) “Students complete activities with a real-world context” –
53.5%. Principals estimated that their faculties as a whole used project-based instruction
almost monthly.

Teachers integrate subjects; need opportunities to integrate more. In focus groups
teachers described that while some progress was being made with regard to the
integration subjects during instructional meetings, many subjects were still taught
separately without reference to each other. They explained that too many teachers work
in isolation from many of their colleagues and lack information about other curricula. On
average, principals across school-levels reported that roughly 25% of teachers made
explicit efforts to integrate science, technology, engineering, and mathematics. The
teachers want to be able to integrate more, but need some additional resources and/or
time.

Teachers benefit from time to collaborate; need more. In focus groups many teachers
described how they consider time with each other one of their most valuable resources
and most beneficial professional activities. One grant convened all middle and high
7
Golden LEAF STEM Initiative
May 2013
school mathematics teachers in the district for four, half-day collaborative planning and
professional development days. The mathematics teachers had opportunities to share
content and instructional strategies both horizontally across subjects and vertically across
grade- and school-levels.

Professional development is generally of high quality; need more time to evolve.
Principals report that teachers participated in roughly 15 hours per year of STEM-related
professional development which addressed integrated content, community/industry
partnerships, connections with postsecondary education, pedagogy, and/or digital
learning. Teachers reported that the most beneficial professional development sessions
either allowed the educators to conduct the STEM activity or lab as a student, were led by
other teachers, or described in deep detail how a particular tool or strategy can work in a
classroom. Many teachers face implementation fatigue with regard to professional
development. Several groups of participating teachers raised this issue. The educators
explained that while they were thankful for the professional development, they were also
concerned about the lack of sufficient time for them to implement the content or new
tools.

Some teachers benefit from visits to STEM industries. Several Golden LEAF STEM
Initiative grants used their funds to provide participating teachers with unique
opportunities to visit local STEM industry facilities and meet STEM industry
professionals. Results suggest that these experiences are very beneficial to STEM
educators. The teachers gain new and deeper understandings of the types of jobs and
competencies demanded in today’s workforce. This better equips them to share this
information with students and teach these skills. Results from the STEM Program
Implementation Rubric indicate that even though teachers participating in some of the
Golden LEAF STEM Initiative grant activities had opportunities to go on study trips,
most teachers, in general, did not – on average some teachers (approaching 50% of their
faculty) participate in an applied learning experience about once every two years.
Findings from the T-STEM Surveys suggest that only about half (46.9%-52.4%) of
participating teachers had general knowledge about STEM careers.

Teachers’ feel confident in their own teaching abilities, but are divided on whether the
classroom efforts of teachers, in general, impact student learning. Year Two results from
the T-STEM Surveys show that when asked about aspects of their instructional practice,
educators participating in the Golden LEAF STEM Initiative had a strong sense of
confidence and self-efficacy (on average 82.4% of all teachers “agreed” or “strongly
agreed” with each item). At the same time, results from items about outcome expectancy
indicate that 48.5% of participating teachers “agree” or “strongly agree” that the efforts
of educators make a difference for student learning.
Additional Findings
In addition to findings from across data sources related to changes in student attitudes, student
learning, and teacher instructional practices, other results emerged from data collection.
8
Golden LEAF STEM Initiative
May 2013

Students’ postsecondary plans. Overall 86.7% of students participating in the Golden
LEAF STEM Initiative who responded to the survey indicated that they planned to attend
college. Of those, 22.7% reported that they planned to attend a community college first
and 77.3% a four-year college or university.

Principals’ leadership for STEM. Pilot findings suggest that on average principals of
schools participating in the Golden LEAF STEM Initiative believe that they focus most
on STEM professional development, both for their STEM teachers specifically and/or for
their entire faculty (91.4% “agreed” or “strongly agreed”). Results indicate that
principals also focus somewhat heavily on maintaining technical infrastructure to support
STEM teaching (84.2% agreed or strongly agreed). Participating principals believe that
they spend the least time and energy working on advocacy and networking related to
STEM (57.3% agreed or strongly agreed).
Capacity-Building Activities
The second of the two objectives of the Golden LEAF STEM Initiative evaluation is to provide
technical assistance to increase the capacity of schools and districts for data-informed decisionmaking. In order to accomplish this goal the evaluation team has carried-out several activities:
hosted annual face-to-face institutes; held semi-annual webinars; created initiative-level and
grantee-level survey results reports; provided one-on-one reference support; built the foundation
for a Golden LEAF STEM Initiative evaluation online community of practice; and engaged
national and state education leaders in discussions about the on-going evaluation and capacitybuilding work for the Golden LEAF STEM Initiative.
Discussion
Summary of Findings
The data collected for this report demonstrate that the Golden LEAF STEM Initiative, consisting
of the individual work of the 14 grants across North Carolina, made significant progress toward
its goals in Year Two. Findings from all data sources taken together suggest that, compared to
Year One:

Student engagement in STEM learning was roughly as high;

Students’ problem-solving skills increased;

Student development of collaboration skills was roughly as high;

Students had more opportunities to visit various STEM industry facilities;

Teachers increased their use of hands-on, inquiry-based instruction;

Teachers integrated STEM subjects at roughly the same frequency;
9
Golden LEAF STEM Initiative
May 2013

Teachers had meaningful opportunities to collaborate with one another and beneficial
professional development opportunities at roughly the same frequency; and

School communities’ awareness and commitment to STEM education increased.
Recommendations

Continue to implement hands-on, problem-based STEM curricula and activities; increase
instructional emphasis on rigor.

Continue to raise student awareness of STEM careers; increase opportunities for students
and teachers to engage with STEM industries; further relationships between schools and
industry (education and work); focus on females in engineering.

Continue providing opportunities for STEM teachers and other teachers to collaborate
both within departments and across grade-levels; focus on ways to support crosscurricular integration.

Increase professional development opportunities that are hands-on, content-specific,
grade-level specific, facilitated by lead teachers, and that provide immediate classroom
solutions; provide more time for teachers to plan, experiment, and implement what
they’ve learned.

Find ways to have safe, professional conversations about teaching philosophies and
beliefs; address differing outcome expectancies.

Continue to invest in sustainability planning; continue to collect data about the progress
of programs and use them to strategically plan for the future.
CERE–NC looks forward to continuing its investigation of the impacts of Golden LEAFsupported initiatives on STEM outcomes in North Carolina schools.
10
Golden LEAF STEM Initiative
May 2013
Introduction
The Golden LEAF Foundation STEM Initiative
The Golden LEAF Foundation (Golden LEAF) STEM Initiative, launched in 2010, is designed
to help prepare North Carolina’s youth for careers requiring skills in science, technology,
engineering, and math (STEM), addressing the growing need for these skills in industries across
the state. The Initiative supports successful education models that increase education outcomes
for students in grades four through nine in rural, economically distressed, and/or tobaccodependent counties of North Carolina.1 A special program of the Golden LEAF Board of
Directors, the STEM Initiative targets projects that:

Are evidence-based and represent systemic approaches to STEM education; these include
in-school, out-of-school, or extended day and support programs that provide assistance to
students transitioning from elementary to middle and middle to high school.

Represent collaborations among public schools and higher education, community, and
regionally relevant industry partners.

Target improved preparation for and academic performance in advanced STEM curricula
by minorities, females, and students from limited-resource families.

Place priority on curricular approaches that are integrated, use project- and inquiry-based
learning concepts, and/or prepare students for successful completion of Algebra 1 by 8th
or 9th grade.

Include strategies that incorporate content specific professional development for teachers.

Provide relevant career and work connections for teachers and students.
Golden LEAF’s STEM Initiative is an important part of a recent state and national focus on
improving STEM education outcomes. Discoveries in engineering, science, and technology
fields drove huge advancements in human society in the 20th century, and experts anticipate a
similar trend in the coming decades (Carnevale, Smith, & Melton 2011; National Academy of
Engineering, 2008). In North Carolina many current and future jobs will require employees with
knowledge and abilities in STEM, in addition to an advanced set of general skills in criticalthinking, communication, and collaboration – often referred to as “21st century skills” (North
Carolina Commission on Workforce Development, 2011; Partnership for 21st Century Skills,
2004). This work in North Carolina and around the nation is also important for those populations
of students who historically have been under-represented in STEM areas, including females,
students of color, and students living in poverty (Beede et al., 2011; Griffith, 2010; Leggon,
2006).
In the spring of 2011 The Golden LEAF Foundation’s board made awards totaling $5 million to
14 grantees. These three-year projects each received varying amounts of funding. The smallest
1
Retrieved August 15, 2012: http://www.goldenleaf.org/STEMinitiative.html
11
Golden LEAF STEM Initiative
May 2013
grant was $100,000 to support a single-district project and the largest award was $600,000 to aid
a regional collaboration.
Each grant implements different strategies to increase student attitudes toward STEM, student
learning in STEM, and teacher instructional knowledge and skills in STEM. The strategies do
not vary widely in substance, but they are often different by content-area and execution. Grant
activities may include, for example: installing SmartBoard technologies in district Algebra 1
classrooms and training teachers to use them; distributing to district middle schools year-long
inquiry-based science curricula and lab materials; or building an elective class in which students
learn the biology of human performance and the engineering of racecars before building their
own model car and competing in a “pit crew challenge.” Many grants are implementing STEM
activities in core classes and elective classes and reaching all students in a grade-level or in a
school. At the same time a few projects are offering in- and out-of-school opportunities
specifically for students in underrepresented groups, including some special elective classes,
activities in targeted schools, and summer camps for girls or students struggling in math and
science.
In total, including a few additions and reductions in participating schools between Year One and
Year Two, the grants currently impact 230 schools, roughly 1,190 teachers, and 31,890 students
in 43 public school districts in North Carolina (see Figure 1). Brief descriptions of the grants can
be found in Appendix A.
Figure 1. Map of Golden LEAF STEM Initiative Participating Schools
The Structure and Purpose of the Evaluation of the Golden LEAF STEM Initiative
The evaluation of the Golden LEAF STEM Initiative takes place over the three-year grant
implementation period, from 2011 through 2014. The research is being conducted by the
12
Golden LEAF STEM Initiative
May 2013
Consortium for Educational Research and Evaluation–North Carolina (CERE–NC), a partnership
of the SERVE Center at the University of North Carolina at Greensboro, the Carolina Institute
for Public Policy at the University of North Carolina at Chapel Hill, and the Friday Institute for
Educational Innovation at North Carolina State University. The evaluation team does not
separately examine the activities and outcomes of individual grants, but rather operates at the
initiative-level, focusing on the overall commonalities of the 14 grants’ activities and observing
the common outcomes.
Evaluation Objective 1: Describe the Overall Effectiveness of the Initiative
The evaluation team’s first objective is to describe the overall effectiveness of the Golden LEAF
STEM Initiative in achieving its goal of improving STEM education outcomes for 4th through
9th graders in rural North Carolina. For this purpose quantitative and qualitative data are being
collected from multiple sources in three separate time periods: September 2011 through April
2012 (“Year One” – completed), September 2012 through February 2013 (“Year Two” –
described in this report), and September 2013 through February 2014 (“Year 3” – report
forthcoming in April 2014).
Data are collected in order to answer four, primary evaluation questions. These are, “To what
degree or in what ways were the Golden LEAF STEM Initiative grantees as a whole:
1. Faithful in implementing their STEM program’s criteria and goals?
2. Effective in changing student STEM attitudes?
3. Effective in changing student STEM learning?
4. Effective in changing teachers’ instructional practices?”
The results from the three periods of data collection are analyzed and synthesized annually. The
main purposes of these data analyses are to provide useful information about the impact of the
initiative as a whole to Golden LEAF and to provide useful information to the grantees as they
continuously build and improve their programs.
Evaluation Objective 2: Evaluation Capacity-Building
The second objective of the Golden LEAF STEM Initiative evaluation is to provide programevaluation technical assistance to the grantees as they work to continually improve their own
individual programs. The evaluation team assists each of the grantees to:

Develop and apply knowledge about education program evaluation; and

Collect, interpret, and use formative data to improve their STEM programs.
Over the course of the three-year initiative various capacity-building events and activities take
place: annual evaluation institutes, semi-annual webinars, the ongoing provision of formative
13
Golden LEAF STEM Initiative
May 2013
data, access to online surveys, and access to one-on-one technical assistance from members of
the evaluation team.
Structure of this Report
This evaluation report is the third full report; it follows a series of two baseline data reports
describing Year One. The first baseline report was completed in April 2012 and contained an
analysis of administrative data on the participating schools and formative results from evaluation
data collected from September 2011 through February 2012 (Corn et al., April 2012). In that
report, analysis of a North Carolina administrative dataset for the 2009-10 school year
(administrative data for the 2010-11 school year was not available at the time of the report
writing) revealed that grantee schools have lower minority populations and higher poverty rates
compared to all other schools in the state. Formative results from grant coordinator interviews
and focus groups with teachers indicated that the grant activities were supporting teachers to start
changing and improving their STEM instruction. Results showed that participating teachers were
starting to collaborate with each other in these efforts as well. The report also contains
descriptions of the evaluation capacity-building activities that had been provided up to that point
in time.
In August 2012 the second baseline report was completed; it summarizes all data collected
during Year One and all evaluation capacity-building activities completed (Corn et al., August
2012). The report addresses evaluation questions 2-4 and summarizes the results from: surveys
administered to students; surveys administered to teachers; focus groups with participating
teachers; classroom observations; and a program implementation rubric completed by grant
coordinators. Findings showed that, overall, the initiative had a very successful first year. The
hands-on, inquiry-based STEM education activities were having positive impacts on levels of
student engagement, including noticeably for female students. Students were also starting to
learn new collaboration skills. Results indicated that the resources were helping teachers to
integrate multiple content areas in their lessons and to find new reasons and ways to collaborate
professionally. Potential opportunities for improvement were also identified, including increased
student confidence and interest in STEM, content integration in teachers’ lessons, time for
teachers to collaborate, and teacher awareness of STEM industries and activities.
This third report summarizes all data and results for Year Two, collected from September 2012
through February 2013. Similar to the recent August 2012 report, this paper addresses evaluation
questions 2-4 by summarizing results from: interviews with grant coordinators; focus groups
with participating teachers; surveys administered to students; surveys administered to teachers;
surveys administered to principals; classroom observations; and a program implementation
rubric completed, in Year Two, by participating principals. These results, taken as a whole,
address the first evaluation question regarding the faithfulness of the implementation of the
initiative. The report is divided into five sections: Data Sources and Analyses, Findings,
Capacity-Building Activities, Recommendations, and Next Steps.
14
Golden LEAF STEM Initiative
May 2013
I. Data Sources and Analyses
Grant Coordinator Interviews
Conducting grant coordinator interviews enables the evaluation team to hear the perspectives of
the grant coordinators and to learn about their experiences with the Golden LEAF STEM
Initiative. The interviews were conducted with each grant coordinator team, which ranged in size
from one to three individuals, during October and November of 2012. The conversations were
held over the phone and lasted for 30 minutes each. In total 14 grant coordinator interviews were
conducted.
The original grant coordinator interview protocol used in Year One was developed by the
evaluation team based upon the goals of the Golden LEAF STEM Initiative and the
implementation plans of grantees. The same protocol, with some minor adjustments, was used
for the Year Two grant coordinator interviews. Questions were written in an open-ended style
and asked grant coordinators about:

The Golden LEAF STEM Initiative activities their grant had most recently completed;

Key successes reflecting back on Year One;

Key challenges reflecting back on Year One;

Any changes they had been noticing in teachers’ instructional practice;

Any changes they had been noticing in students’ engagement in STEM; and

Any changes they had been noticing in students’ learning in STEM.
The complete protocol can be found in Appendix B. Conversations were digitally recorded,
transcribed, and imported into Atlas.ti software for further analysis. Results were analyzed using
a grounded theory method for analysis, extracting themes or “codes” from the text and grouping
them into categories (Glaser & Strauss, 1967).
Site Visits
From November 2012 through February 2013 the evaluation team made 14, single-day site visits
to either one or two participating schools in each of the 14 Golden LEAF STEM Initiative grants.
During these site visits evaluation team members visited classrooms of teachers participating in
the initiative, conducted a focus group with participating teachers, and, when possible, had
informal conversations with grant coordinators about their project’s progress. Grant coordinators
planned the logistics of the site visit activities and all activities were carried-out according to
these pre-arranged schedules.
15
Golden LEAF STEM Initiative
May 2013
Focus Groups with Participating Teachers
Conducting focus groups enables the evaluation team to hear the perspectives of participating
teachers and to learn about their experiences with their Golden LEAF STEM Initiative grant
activities. The conversations were held during the in-person site visits and lasted for
approximately one hour each. The sizes of the focus groups ranged from one to nine teachers
selected by the grant coordinator team, with an average of six teachers per group. In total 14
focus groups were conducted with 74 teachers. Grant coordinators chose the focus group
participants from either one school or from multiple schools participating in their Golden LEAF
STEM Initiative grant; the arrangement depended on the size of the grant, the geographic
locations of the schools, and the site visit schedule.
The original focus group protocol, used in Year One, was developed by the evaluation team
based upon the goals of the Golden LEAF STEM Initiative and the implementation plans of
grantees. The same protocol was used for the Year Two focus groups with some minor
adjustments. Questions were written in an open-ended style and asked teachers about:

The Golden LEAF STEM Initiative activities and other STEM activities in which they
were involved;

Any changes they noticed in their students’ engagement in STEM;

Any changes they noticed in their students’ learning in STEM;

STEM education professional development in which they had participated; and

Any changes in their instructional practice as a result of their participation.
The complete focus group protocol can be found in Appendix C. Conversations were digitally
recorded, transcribed, and imported into Atlas.ti software for further analysis. Results were
analyzed using themes or “codes” from the Year One analysis, in addition to new codes
identified in the new data. Both Year One and Year Two analyses of focus group data applied a
grounded theory method for analysis, extracting codes from the text and grouping them into
categories.
Classroom Visits
Classroom visits enable evaluation team members to gather general information about the types
of STEM activities, curricula, and instructional methods that were taking place in participating
schools. Each site visit included approximately two to four hours of attendance in classrooms of
participating teachers by evaluation team members. Attendance by evaluation team members was
held according to a pre-arranged schedule, with grant coordinators selecting the specific
classrooms and class periods. Typically two participating teachers’ rooms were visited for about
one to two hours each. If two evaluation team members were present each team member visited a
single teacher’s classroom; if only one team member was present, they would observe both
rooms. Evaluation team members took descriptive notes on the classroom agenda and used the
16
Golden LEAF STEM Initiative
May 2013
Classroom Assessment Scoring SystemTM (CLASS) instrument in order to collect similar data
across all classrooms in the initiative.2 The CLASS protocol measures a general set of observed
classroom behaviors and activities on a seven-point scale and is completed in 15 minute intervals.
Evaluation team members completed a total of 82 observation protocols during visits to 31
different classrooms. Descriptive notes and CLASS results were downloaded into the Microsoft
Excel spreadsheet program and were analyzed for frequencies, general themes, and patterns.
Teacher Efficacy and Attitudes toward STEM (T-STEM) Surveys
Description of T-STEM Surveys
The Science, Technology, Engineering, Mathematics, and Elementary Teacher Efficacy and
Attitudes toward STEM (T-STEM) Surveys are five, subject-specific versions of the same
survey. (The survey items are identical except when a teachers’ subject-area is referenced, in
which case the survey item names either “science,” “technology,” “engineering,” or
“mathematics.”) The T-STEM Surveys contain two validated, reliable scales, or sets of items
which most confidently describe a single characteristic of the survey-taker when calculated as a
single, composite result.3 In addition the surveys contain five other sections. The first scale, the
Personal STEM Teaching Efficacy and Beliefs Scale (PSTEBS), consists of 11 Likert-scale
questions which ask the respondent about their confidence in their teaching skills.4 The PSTEBS
asks respondents to indicate their level of agreement with statements such as, “I am continually
improving my [content area] practice,” and “I am confident that I can answer students’ [content
area] questions.” The second scale, the STEM Teaching Outcome Expectancy Scale (STOES),
consists of nine questions and asks the respondent about the degree to which they believe student
learning can be impacted by effective teaching. The STOES uses a Likert-scale and asks
respondents to indicate their level of agreement with statements such as, “The inadequacy of a
student’s [content area] background can be overcome by good teaching,” and “The teacher is
generally responsible for students’ learning in [content area].” Finally, the other five survey
sections address the following topics: student technology use, STEM instruction, attitudes
toward 21st century learning, attitudes toward teacher leadership, and STEM career awareness.
The full versions of the Science, Technology, Engineering, Mathematics, and Elementary TSTEM Surveys can be found in Appendix D.
2
This protocol is being used in multiple, national-level, education studies including the Bill & Melinda Gates
Foundation’s Measures of Effective Teaching (MET) project (Kane & Staiger, 2012). Golden LEAF STEM
Initiative evaluation team members received training on the use of the instrument and have received certification of
their reliability. For more information on the CLASS instrument see: http://www.teachstone.org/about-the-class/
3
The Elementary T-STEM Survey contains four scales for elementary teachers who indicate that they teach both
science and mathematics. The respondents answer the two Science T-SSTEM scales and two Math T-STEM scales.
4
Likert-scale survey items ask respondents to report the degree to which they agree or disagree with a given
statement. Both the PSTEBS and STOES scales ask respondents to rate their level of agreement on a five-point
Likert-scale: “strongly disagree” (1), “disagree” (2), “neither agree nor disagree” (3), “agree” (4), and “strongly
agree (5).
17
Golden LEAF STEM Initiative
May 2013
Development of the T-STEM Surveys
From spring through December of 2011, pilot versions of the Science, Technology, Engineering,
Mathematics, and Elementary T-STEM Surveys were developed by Friday Institute staff
working on the Maximizing the Impact of STEM Outreach (MISO) research project and the
Golden LEAF STEM Initiative evaluation team.5 These pilot versions contained only the two
scales, PSTEBS and STOES, which were adapted from the original work of Enochs and Riggs
(1990). From December 2011 through February 2012 (Year One) the Golden LEAF STEM
Initiative grantees administered the pilot surveys to participating teachers along with an openended question asking the respondents for suggestions on how the survey could be improved.
From March through August 2012 these pilot administration results, along with feedback gained
from other STEM education researchers, were used by the Golden LEAF STEM Initiative
evaluation team and the MISO Project staff to revise and finalize the scales. Validity and
reliability analyses were conducted to determine: if the individual survey items behaved as they
were intended; if the items added to the explanatory power of the scales; if the scales actually
functioned as single units; if the scales functioned similarly across different types of teachers;
and overall what, if any, edits were needed to improve the accuracy and consistency with which
the surveys measure STEM teachers’ confidence and beliefs about effective teaching. Overall the
results were very positive and showed that the scales were strong and clear with high reliability
after dropping just a few items. At this time, the researchers added five additional sections
measuring frequency of student technology use, frequency of STEM instruction, attitudes toward
21st century learning, attitudes toward teacher leadership, and STEM career awareness.
Year Two T-STEM Survey Administration and Analysis
From September 2012 through December 2012 the coordinators of the 14 Golden LEAF STEM
Initiative grants administered online the Science, Technology, Engineering, Mathematics, and
Elementary T-STEM Surveys to teachers impacted by their program during the 2012-13 school
year. A single URL for all five versions of the T-STEM Survey was used. The homepage for the
survey contained an initial question sorting respondents by their self-identified content-area. The
grant coordinators either sent the link directly to participating teachers or sent the URL to
principals who then administered the surveys to the participating teachers. Due to the time frame
of the initiative-wide administration not all grants were able to administer the T-STEM Surveys
at an ideal time in their implementation process. The initiative-level results, therefore, should be
interpreted with some caution.
Table 1 shows the initiative’s response rates by survey. Survey results were analyzed at the
scale-level and item-level using descriptive statistics and independent comparisons by subject
area, school-level, and teacher years of experience.
5
For more information visit the MISO Project homepage, http://miso.ncsu.edu. See the Golden LEAF STEM
Initiative Evaluation Baseline Report (Corn et al., April 2012) for a more complete description of the survey
development process.
18
Golden LEAF STEM Initiative
May 2013
Table 1
T-STEM Response Rates, September – December 2012
T-STEM Survey
Elementary
Science
Technology
Engineering
Math
TOTAL
Number of
Responses
Estimated
Teachers
Impacted
in 2012-13
Estimated
Response
Rate
246
149
42
9
98
544
565
96.3%
Note. The estimated number of teachers impacted in 2012-13 and the estimated response rate are based on a sum of
actual and estimated impact figures provided by the 14 grant coordinator teams.
Student Attitudes toward STEM (S-STEM) Surveys
The S-STEM Surveys
The Upper Elementary School (4-5th) and the Middle/High School (6-12th) Student Attitudes
toward STEM (S-STEM) Surveys are two, grade-level specific versions of the same survey. (The
survey items are written at different reading-levels specific to the ages of respondents, but are
intended to measure the same perspectives of the students.) The S-STEM Surveys contain four
validated, reliable scales and one additional section. The first scale measures student attitudes
toward mathematics. It consists of eight Likert-scale questions which ask the respondent about
their confidence and interest in mathematics, including questions such as “I am the type of
student who does well in math,” and “When I’m older, I might choose a job that uses math.”6
The second, third, and fourth scales measure student attitudes toward science, technology and
engineering, and 21st century skills such as communication and collaboration. The final section
of the survey asks students about their levels of interest in 12 STEM career areas. Full versions
of the Upper Elementary School and Middle/High School S-STEM Surveys can be found in
Appendix E.
Development of the S-STEM Surveys
From the spring through December of 2011 (Year One) the pilot versions of the Upper
Elementary School and Middle/High School S-STEM Surveys were developed by Friday
Institute staff working on the MISO Project and the Golden LEAF STEM Initiative evaluation
team. These pilot versions contained the same four scales and one section that remain in the final
S-STEM Surveys. From December 2011 through February 2012 the Golden LEAF STEM
6
The S-STEM scales ask student respondents to rate their level of agreement on a five-point Likert-scale: “strongly
disagree” (1), “disagree” (2), “neither agree nor disagree” (3), “agree” (4), and “strongly agree (5).
19
Golden LEAF STEM Initiative
May 2013
Initiative grant coordinator teams administered the pilot surveys to participating students along
with an open-ended question asking the respondents for suggestions on how the survey could be
improved.
From March through August of 2012 these pilot administration results, along with feedback
gained from other STEM education researchers, were used by the Golden LEAF STEM Initiative
evaluation team and the MISO Project staff to finalize the scales. The researchers also finalized
the career-interest section. Validity and reliability analyses were conducted and, like the pilot TSTEM Surveys, the results from these analyses showed that the scales in the student surveys
were strong and clear with high reliability after dropping just a few items (Faber et al., 2013).
Year Two S-STEM Survey Administration and Analysis
From September through December 2012 the coordinators of the 14 Golden LEAF STEM
Initiative grants administered online the Upper Elementary School and Middle/High School SSTEM Surveys to students impacted by their program during the 2012-13 school year. A single
URL for both versions of the S-STEM Survey was used, with the survey homepage containing
an initial question sorting respondents by their self-identified grade-level. The coordinators
received the URL and each managed the administration for their grant. They either sent the link
directly to participating teachers to share with students, or sent the URL to principals who then
shared it with the participating teachers.
Due to the time frame of the initiative-wide administration not all grants were able to administer
the S-STEM Surveys at an ideal time in their implementation process. The initiative-level
results, therefore, should be interpreted with some caution. Table 2 shows the initiative’s
response rates by survey.
As with the teacher survey findings, results from the student surveys were analyzed at the scalelevel and item-level using descriptive statistics and independent comparisons by gender,
race/ethnicity, and school-level.
Table 2
S-STEM Response Rates, September – December 2012
S-STEM Survey
Upper Elementary School (4-5th)
Middle and High School (6-12th)
TOTAL
Number of
Responses
Estimated
Students
Impacted
in 2011-12
Estimated
Response
Rate
3,433
8,404
11,837
16,933
69.9%
Note. The estimated number of estimated students impacted in 2011-12 and the estimated response rate are based on
a sum of actual and estimated impact figures provided by the 14 grantee coordinator teams.
20
Golden LEAF STEM Initiative
May 2013
Pilot Leadership for STEM Self-Assessment
From mid-December 2012 through late February 2013 the grant coordinator teams administered
online the Pilot Leadership for STEM Self-Assessment to principals of participating schools.
Collecting information about principal leadership for STEM enables the evaluation team to better
understand the school-level context for initiative implementation. The pilot self-assessment is
intended to measure principal’s leadership for STEM education along six dimensions: vision,
infrastructure, professional development, shared decision-making, advocacy, and evaluation. The
self-assessment uses a five-point Likert scale for responses and contains items such as,
“Regarding the Golden LEAF STEM project/work at my school, I have articulated a vision for
the STEM project,” or, “Regarding the Golden LEAF STEM project/work at my school, I make
sure teachers have access to technology tools that facilitate their work.”7 The evaluation team
developed the survey based on a similar instrument created by The Friday Institute to measure
principal leadership for one-to-one laptop initiatives (The William and Ida Friday Institute, 2011).
In total 107 principals completed the Pilot Leadership for STEM Self-Assessment, for an
initiative response rate of 46.5%. In the spring and summer of 2013 the evaluation team will use
this data to conduct validity and reliability analyses on the survey and further develop the
instrument. See Appendix F for a full version of the self-assessment.
The Golden LEAF STEM Implementation Rubrics
The Rubrics
The Elementary/Middle and High School Golden LEAF STEM Implementation Rubrics are
diagnostic tools for leaders to reflect on the depth of implementation of school-level STEM
education programs. Most items on the two rubrics are identical and are intended to measure the
same aspects of STEM programs, but occasionally wording varies slightly to reflect the specific
context of an elementary, middle, or high school. The rubrics aim to articulate a common
language for STEM program implementation strategies and to establish a continuum describing
good-to-great STEM programs.
The elementary/middle school rubric’s framework consists of 10 overarching “attributes” of a
successful STEM program. The high school rubric’s framework consists of the same 10
attributes and an additional 11th attribute, containing items measuring postsecondary alignment.
The attributes describe a wide range of qualities of successful STEM programs, from the
application of project-based learning across all STEM subjects to the communication of a STEM
education plan to local education, business, and civic communities. These attributes were
identified by the North Carolina Department of Public Instruction and adopted by the North
Carolina State Board of Education in the fall of 2011 as part of a statewide STEM Education
7
The Pilot Leadership for STEM Self-Assessment asks principal respondents to rate their level of agreement on a
five-point Likert-scale: “strongly disagree” (1), “disagree” (2), “neither agree nor disagree” (3), “agree” (4), and
“strongly agree (5).
21
Golden LEAF STEM Initiative
May 2013
Strategy.8 Represented within each attribute are three to five “key elements,” calibrated along a
four-item scale from “early” to “developing” to “prepared” to “model.” The elementary/middle
rubric contains 34 key elements and the high school rubric contains 40 key elements.
Respondents identify where on the four-point continuum they believe their STEM program is
operating for each key element. The rubric pertains to school-wide programs, so for users
reflecting on programs that are not school-wide, not all key elements will be valid measures of
their implementation. In these cases, however, the key elements can be useful descriptions of the
program’s larger school environment.
While the rubrics serve as a reflective resource for the grantees as they plan, evaluate, and adjust
their own STEM education programs, they also act as useful measurement instruments for the
evaluation of the Golden LEAF STEM Initiative overall. Collecting program information using
the rubric enables the evaluation team to better understand both the implementation of initiative
activities and the broader, school-level context for implementation. The rubrics have also shown
promise in aiding philanthropic organizations in evaluating proposed projects and allocating
scarce resources by identifying projects that demonstrate “readiness” to implement with fidelity.
The full version of the High School Golden LEAF STEM Implementation Rubric can be found
in Appendix G.
Development of the Golden LEAF STEM Program Implementation Rubric
In the fall of 2011 the Golden LEAF STEM Initiative evaluation team developed a pilot version
of the Golden LEAF STEM Implementation Rubric.9 It contained the same 11 attributes and
most of same key elements as the final high school version. The evaluation team gained feedback
from STEM education leaders in North Carolina and incorporated these recommendations in the
first draft.
This pilot rubric was administered to the Golden LEAF STEM Initiative grant coordinators in
Year One. They completed their program assessment and also used space in the rubric to submit
feedback about the instrument. The evaluation team used this feedback, along with edits from
experts at North Carolina State University’s College of Education and recommendations
gathered by the North Carolina Department of Public Instruction, to further develop and refine
the version of the rubric administered in Year Two of the Golden LEAF STEM Initiative.
Year Two Golden LEAF STEM Implementation Rubric Administration and Analysis
From mid-December 2012 through late February 2013 the grant coordinators administered
online the Elementary/Middle School and High School STEM Implementation Rubrics to
principals of participating schools.10 The data collection strategy had been changed from
administering hard copies to only the 14 grant coordinator teams in Year One, to an online
8
For more information see http://www.ncpublicschools.org/stem/
See the Golden LEAF STEM Initiative Evaluation Baseline Report (Corn et al., April 2012) for a more complete
description of the rubric development process.
10
Two grants administered the rubrics on paper; the data were then entered by hand by the evaluation team into the
online format.
9
22
Golden LEAF STEM Initiative
May 2013
version to all principals in Year Two. This was done in order to capture a more fine-grained
picture of both the actual implementation and the context for implementation of the initiative
activities. Table 3 shows the initiative’s response rates by rubric.
In the analysis of the Elementary/Middle School and High School Golden LEAF STEM
Implementation Rubric results each level of the implementation scale was assigned a rating (1 =
early to 4 = model) and data were summarized at the key-element level and attribute level. Data
were analyzed using descriptive statistics and independent comparisons by school-level. Full
results from the rubrics can be found in Appendix R.
Table 3
Golden LEAF STEM Implementation Rubric Response Rates, December 2012 – February 2013
Golden LEAF STEM Implementation Rubric
Elementary/Middle School
High School
TOTAL
Number of
Responses
Number
Schools
Impacted
in 2012-13
Estimated
Response
Rate
76
20
96
230
41.7%
Administrative Data on Student Performance
Often state-level standardized tests are not sensitive enough to measure changes in learning that
result from a single change in a student’s total experience in a short period of time. For these
reasons, the Golden LEAF STEM Initiative evaluation plan has scheduled to collect standardized
test results in Year One and Year Three of the initiative. The Golden LEAF STEM Initiative
Evaluation Baseline Report (Corn et al., April 2012) summarizes Year One’s administrative data
on student achievement from the 2009-10 school year (data from the 2010-11 school year were
not available at the time of the writing of this report). The paper reported mostly school-level
percent proficiency on standardized tests for schools participating in the Golden LEAF STEM
Initiative. Similar data will be collected in 2014 at the conclusion of the evaluation and will be
compared against the baseline results. Administrative data on student performance were not
collected for this Year Two report.
A Note about the Reporting of Numbers of Responses
For all remaining report tables showing aggregated results, in almost every case not all
respondents actually answered every item on the particular instrument. Most items are missing at
least a few, if not several, responses from survey-takers. The total numbers of respondents, or
“Ns,” therefore, reported in these tables are the maximums of these narrow, item-level ranges. In
a few cases the ranges of the total numbers of respondents, Ns, vary enough that the entire range
is reported.
23
Golden LEAF STEM Initiative
May 2013
II. Findings
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Student Attitudes toward STEM Education?
Student Characteristics in Year Two
Items on the pilot Upper Elementary and Middle/High School S-STEM Surveys ask students to
share information about their background, including gender, race/ethnicity, and awareness of
adults in STEM careers (Tables 4-5). Analyses of the demographic characteristics of the studentrespondents show that the proportions of participating students by both gender and race/ethnicity
were roughly similar to Year One, and to students in Golden LEAF STEM Initiative schools
overall (Corn et al., April 2012; Corn et al., August 2012). A smaller percentage of American
Indian/Alaska Native students responded to the student attitudes survey this year (3.8%) than last
year (6.9%). A larger percentage of Black/African American students responded to the student
attitudes survey this year (13.4%) than last year (9.9%). Also a relatively smaller proportion of
White/Caucasian students completed the student attitudes survey this year (64.5%) than last year
(70.3%). Finally, more Hispanic/Latino students took the survey this year (12.2%) as compared
to last year (8.9%).
Due to the relatively small number of Native Hawaiian/Pacific Islander students who completed
the S-STEM Surveys (29), results from this subgroup are not included in the remaining analyses
in this report. Demographic comparison and item-level results from the Upper Elementary and
Middle/High School S-STEM Surveys, including all Hawaiian/Other Pacific Islander
respondents, can be found in Appendices I-K.
Table 4
Upper Elementary, Middle, and High School Student Demographic Characteristics
Proportion of Respondents
Demographic
Characteristic
Gender
Male
Female
Race/Ethnicity
American Indian/Alaska Native
Asian
Black/African American
Hawaiian/Other Pacific Islander
White/Caucasian
Hispanic/Latino
Multiracial
Upper
Elementary
(N=3,433)
Middle
(N=7,080)
High
(N=1,324)
TOTAL
(N=11,837)
48.8%
51.2%
49.3%
50.8%
49.0%
51.0%
49.1%
50.9%
4.3%
0.7%
17.5%
0.5%
61.3%
11.2%
4.6%
3.8%
2.2%
10.6%
0.1%
66.1%
13.2%
3.9%
2.2%
0.8%
18.1%
0.3%
64.3%
9.2%
5.2%
3.8%
1.6%
13.4%
0.3%
64.5%
12.2%
4.2%
24
Golden LEAF STEM Initiative
May 2013
Note: Upper elementary results include students in grades 4-5; middle school results include students in grades 6-8;
and high school results include students in grades 9-12.
The S-STEM Survey gathered some background information from students related to their
awareness of adults in STEM careers (Table 5). When asked whether or not they know an adult
who works as a scientist, engineer, mathematician, or technologist, Asian and Hispanic/Latino
students were consistently the least likely to report that they do.
Table 5
Student Awareness of Adults with STEM Careers by Race/Ethnicity
Percentage of Respondents
Yes, I know an
American
adult who works as
Indian/
AK Native
a/an …
Asian
(N=187)
Black/
African
American
(N=812)
White/
Caucasian
(N=6,360)
Hispanic/
Latino
(N=750)
MultiRacial
(N=780)
20.1%
56.3%
27.6%
37.9%
25.1%
56.8%
44.3%
48.9%
25.1%
62.8%
37.5%
44.0%
21.0%
55.4%
35.3%
43.1%
24.2%
60.2%
38.8%
49.4%
(N=481)
Scientist
Engineer
Mathematician
Technologist
29.2%
65.4%
38.0%
46.5%
Student Attitudes toward STEM in Year Two
The Golden LEAF STEM Initiative grantees all share the common objective of improving
student attitudes toward STEM subjects. This aligns well with national experts on the President’s
Committee of Advisors on Science and Technology who contend that improving student interest
and attitudes toward STEM is as important as increasing the overall level of academic
proficiency in STEM (PCAST, 2010). Attitudes are commonly understood to be a psychological
state of favor or disfavor towards something; in educational psychology attitudes can include
sub-concepts such as student motivation, positive learning values, enthusiasm, and interest. An
important, related educational psychology concept is student engagement. While still debated
among researchers, “engagement” generally refers to some combination of behavioral
engagement (e.g. levels participation), emotional engagement (e.g. positive or negative
emotional reactions), and/or cognitive engagement (e.g. levels of investment in learning). When
students are engaged they are involved in the classroom activities, they persist despite challenges,
and they take delight in the outcomes. Researchers have found that, over time, increased student
attitudes and engagement have been associated with increased student learning outcomes
(Fredericks, Blumenfeld, & Paris, 2004; Marks, 2000). As such, grantees anticipate that some
changes in student attitudes, engagement, and/or interest will eventually lead to an improvement
in student performance in STEM, increased participation in advanced STEM courses, and in time
higher participation in STEM career pathways. The Golden LEAF STEM evaluation team aims
to gain a broad understanding of participating students’ attitudes toward STEM by measuring not
only their attitudes toward STEM subjects, but also their engagement in STEM activities and
awareness and interest in STEM careers.
25
Golden LEAF STEM Initiative
May 2013
As part of the constellation of efforts to increase student attitudes toward STEM, every grant
program includes strategies to provide students more opportunities for authentic, hands-on
learning in STEM subjects. These opportunities include, for example: lab materials for students
to experiment with battery-powered model cars and acceleration; digital probes to collect data on
changing temperatures in liquids; electrical sensors and other materials with which students build
robots; manipulatives and high-quality curricula to teach problem-based math; and/or field trips
to local STEM industry facilities. Every grant’s main strategies also include professional
development activities to support teachers to use these materials and teach with inquiry-based
and/or problem-based instruction.
Teachers reported that student engagement increased in classrooms. Consistent with teacher
focus group findings from Year One, the strongest finding overall from Year Two indicates that
hands-on lessons delivered with inquiry-based instruction increase student engagement in
comparison to lecture or other more “traditional” instructional methods. Teachers in every focus
group reported that student engagement in STEM content increased as a result of the materials
and professional development opportunities they received through the Golden LEAF STEM
Initiative. Some direct quotes from teachers illustrate this point:
I can tell a huge difference when we’re doing an inquiry-based [lesson] … It’s a totally
different class. They’re all discussing. They’re all working together. They’re trying to
solve the problems … You don’t see the type of off-task behaviors in class, because they
are engaged.
It’s a lot different than the traditional classroom that you observe. You see that increased
level of engagement, and the kids are excited about it.
I think this is the highest level of student engagement I’ve ever experienced, as far as
teaching.
Because the data belongs to a car they created, that they studied about … when they use
it in math and statistics, they care. It means something; they have ownership. Today we
talked about mean and absolute deviation and they were asking each other, “What’s
yours? What’s yours?” And they just wouldn’t have been that interested in mean or
absolute deviation otherwise.
A few teachers described how some students had become so excited that they were looking for
opportunities to explore concepts outside of class.
For whatever subject we’re investigating there’s not a book left in the media center
because all my kids are in there checking them out. So it’s actually helping with the
reading.
Hands-on – it makes a difference. Kids always want to know so much more, and they go
look on the Internet to try to find the answers.
26
Golden LEAF STEM Initiative
May 2013
Results from classroom visits support the finding that the hands-on, inquiry-based STEM
activities lead to higher student engagement compared to other lessons. The evaluation team
observed student behaviors that demonstrate high levels of engagement during all but two, 15minute observation periods out of 82 conducted across the initiative (see Appendix H for results
from classroom observation protocols).
Teachers reported that engagement also increased for struggling students. Focus group results
suggest that the new curricula and materials had a noticeable effect on students who struggled
academically and/or behaviorally in school (often referred to as “Exceptional Children” in North
Carolina). These students did not have successful records in traditional math or science classes,
but the teachers explained that the students were frequently having very positive experiences
when participating in these hands-on STEM activities. Some teachers noted:
I’ve seen it make a big difference with our Exceptional Children population, and give
them more confidence. They’re becoming more engaged ... so that’s been very, very
positive and helpful … And those Exceptional Children students become confident
enough to take leadership roles - they are teaching our other regular education or even
our academically or intellectually gifted students.
What has been really neat, that I’ve enjoyed, is my at-risk students are very much driven
forward. Some of my better thinkers during science are those that would struggle in the
other areas, but they are loving it and they are really engaged.
I have one in particular, he’s in trouble all the time, he can’t read, but he is excelling in
my class. Now that we’re using that [3D design software], he is teaching everybody else
how to do it. He is ahead of the curve. We’ve gotten him into something that he can do
and he’s really motivated. He’s coming in the mornings as soon as he gets here. And even
when he’s in “in school suspension” he comes to my room to find out what he’s going to
miss that day. And that’s never happened before.
A number of teachers also described how English as a Second Language (ESL) students were
more engaged in STEM activities than in other courses. They explained that these students
struggle with reading and writing since they are learning the English language. These same
students, however, were engaged and learning in their science and technical classes.
Our Hispanic/Latino population loves the hands-on activities. They are very
manipulative learners. A lot of them are English as a Second Language learners, so if
you hand them a science book and say, “Read this,” they can only stare at the page.
The learning of my Hispanic/Latino group has increased, and their engagement is
absolutely fantastic.
Informal STEM learning opportunities increased student engagement. Results from both grant
coordinator interviews and teacher focus groups suggest that various summer or weekend STEM
camps increased levels of student participation and excitement in STEM – both indicators of
27
Golden LEAF STEM Initiative
May 2013
behavioral and emotional engagement. Several Golden LEAF STEM Initiative grants, for
example, used their funds to either create their own summer STEM camps or to send groups of
students to already established camps, like robotics camps or others. A couple grant coordinators
stated:
The students really enjoyed that they were provided the opportunity to go the robotics
camps in the summer, even though it was a limited number of students for the size of our
county.
For those who attend the STEM camp, they are excited that they have a better
understanding of the curriculum because they had exposure to it during the summer.
Students reported moderately positive attitudes toward mathematics, science, engineering and
technology overall. Despite the high levels of engagement reported by teachers and observed by
the evaluation team, the combined baseline data from the Upper Elementary School and
Middle/High School S-STEM Surveys show that students still only slightly agreed that they felt
confident and were interested in STEM content (Table 6). When presented with positively
worded questions like, “I feel good about myself when I do science” and “I am interested in what
makes machines work,” on average students felt somewhat neutral or slightly agreed. The survey
results also indicate no significant change in student attitudes toward STEM between Year One
and Year Two. While variation between students at different school-levels was very slight, upper
elementary school students reported the highest confidence and interest toward mathematics,
science, and engineering and technology, and high school students reported the lowest.
Table 6
Mean Composite Scores of Student STEM Attitudes by School-Level
STEM Attitudes
Math Attitudes
Science Attitudes
Engineering and Technology Attitudes
Upper
Elementary
(N=785)
3.7
3.6
3.5
Middle
School
(N=7,698)
3.6
3.4
3.4
High
School
(N=926)
3.4
3.4
3.3
All
students
(N=9,409)
3.6
3.4
3.4
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5). Upper elementary results include students in grades 4-5;
middle school results include students in grades 6-8; and high school results include students in grades 9-12.
Female students reported similar attitudes toward science and math as males, but somewhat
lower attitudes toward engineering (Table 7).
28
Golden LEAF STEM Initiative
May 2013
Table 7
Mean Composite Scores of Student STEM Attitudes by Gender
Female
(N=5,809)
3.5
3.4
3.2
STEM Attitudes
Math Attitudes
Science Attitudes
Engineering and Technology Attitudes
Male
(N=6,021)
3.6
3.5
3.7
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
Results from the student attitudes surveys show that students did not vary considerably in their
confidence and interest in STEM subjects when compared by race/ethnicity. Complete
demographic comparison and item-level results on student attitudes toward math, science, and
engineering and technology can be found in Appendices I-K.
Students had positive attitudes toward 21st century learning overall. Results from the Upper
Elementary School and Middle/High School S-STEM Surveys suggest that, like in Year One,
students continued to have more positive attitudes toward communication and collaboration
skills, or 21st century learning, than toward science, math, or engineering and technology (Table
8). Student attitudes toward 21st century learning remained consistent at an average 4.0 mean
composite score from Year One to Year Two. The survey data show that, also like in Year One,
there was almost no variation between the students’ attitudes toward 21st century skills when the
learners are compared by gender, race/ethnicity, or school-level. See Appendices I-K for full
results and comparisons of students 21st century learning attitudes.
Table 8
Student Attitudes toward 21st Century Learning Compared to other STEM Attitudes
Mean Composite Score
All Students
(N=11,843)
21st century
Learning
Attitudes
4.0
Math Attitudes
Science
Attitudes
Engineering
and Technology
Attitudes
3.5
3.4
3.4
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
While survey results have the most explanatory power when considered as entire scales, a subset
of the items in the student attitudes toward 21st century learning scale directly ask students about
their communication and collaboration skills. Analyzed independently these items provide some
detailed information. Findings indicate that overall, like in Year One, upper elementary school
29
Golden LEAF STEM Initiative
May 2013
students and middle and high school students reported similar attitudes toward communication
and collaboration skills (see Table 9).
Table 9
Student Confidence in 21st Century Skills by School-Level
Survey Item
I can lead others to reach a goal.
I like to help others do their best.
I respect all children my age even if they are different
from me.
I try to help other children my age.
When I make decisions, I think about what is good
for other people.
When things do not go how I want, I can change my
actions for the better.
I can work well with all students, even if they are
different from me.
Proportion “Agree/Strongly Agree”
Upper
Middle and
Elementary
High
(N=3,332)
(N=8,122)
73.9%
71.3%
85.5%
77.3%
81.6%
81.5%
84.3%
78.9%
69.7%
75.2%
69.5%
74.6%
75.9%
74.2%
Note: The wording of the survey items was taken from the Upper Elementary S-STEM Survey. The items are
written at a slightly higher reading level in the Middle and High School Student Attitudes toward STEM Survey.
Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree nor
disagree” (3), “agree” (4), and “strongly agree” (5).
The Science, Technology, Mathematics, and Elementary T-STEM Surveys asked teachers about
their own attitudes toward the importance of teaching the same 21st century learning skills on
which the students were surveyed. Results indicate that, overall, teachers participating in the
Golden LEAF STEM Initiative had very positive attitudes toward teaching those same 21st
century skills. The mean composite score on the T-STEM attitudes toward 21st century learning
scale for all teachers was a very high 4.4, suggesting that many teachers “agreed” or “strongly
agreed” that teaching those skills is important. Demographic comparisons and item-level results
for teachers’ attitudes toward 21st century skills can be found in Appendices L-Q.
Students expected to do well in classes; about half reported plans to take advanced mathematics
and science. Students’ self-perceived likelihood of success in class and their plans for advanced
mathematics and science coursework were measured to understand better the students’
enthusiasm and interest in STEM. The Upper Elementary School and Middle/High School SSTEM Surveys asked students to report on how well they think they will do in their science,
mathematics, and English classes (Table 10). Findings suggest that students’ expectations did not
vary much from Year One to Year Two for any subject. Overall the vast majority of students felt
that they would do at least “okay” if not “very well” in their courses.
30
Golden LEAF STEM Initiative
May 2013
Table 10
Student Performance Expectations by Subject Area
How well do you expect to
do this year in your …
English/Language Arts Class
Math Class
Science Class
Percentage of Respondents
(N=11,153)
Not Very Well
OK/Pretty Well
Very Well
6.4%
11.0%
8.0%
50.1%
43.1%
44.9%
43.5%
45.9%
47.1%
While females and males overall had very similar performance expectations, survey findings do
suggest that female students have slightly more positive expectations (95.2% responded
“OK/Pretty well” or “Very Well”) than males (92.1%) for their performance in English/language
arts class. Survey findings indicate that students across different race/ethnicities and different
school-levels have very similar performance expectations. The largest variation between the
most positive and least positive performance expectations between students of different
race/ethnicities for any subject area was 4.9 percentage points – 93.3% of White/Caucasian
students expected to do “Ok/Pretty Well” or “Very Well” in science class while 88.4% of
Black/African American students did. The largest variation between the most positive and least
positive performance expectations between students of different school-levels for any subject
area was 3.1 percentage points – 90.2% of upper elementary school students expected to do
“Ok/Pretty Well” or “Very Well” in math class while 87.1% of high school students did. See
Appendices I-K for complete student survey results.
The Middle/High School S-STEM Survey also asked 6-12th grade students to share whether or
not they plan to take advanced classes in mathematics and in science. (The Upper Elementary
School S-STEM Survey does not contain this item since the students have had little experience
with advanced mathematics and science course offerings.) Results show that 48.2% of students
intended to take advanced classes in mathematics, with slightly more females (50.0%) reporting
that they had such plans than males (46.5%). When compared by race/ethnicity, Black/African
American students were the most likely to report that they intended to do advanced mathematics
work (54.1%) and multiracial and American Indian/Alaska Native students were the least likely
(44.1% and 44.4% respectively; see Table 11.) When compared by school-level, middle school
students were more likely to report that they had plans to take advanced courses in mathematics
(49.0% reported “yes” and only 18.7% confirmed “no”) than high school students (44.1%
reported “yes” and a full 27.6% confirmed “no”).
31
Golden LEAF STEM Initiative
May 2013
Table 11
Middle and High School Students’ Intentions to take Advanced Math Classes by Race/Ethnicity
In the future,
do you plan to
take advanced
classes in
mathematics?
Yes
No
Not Sure
American
Indian/
AK Native
(N=261)
44.4%
25.7%
29.9%
Proportion of Respondents
Black/
White/
Hispanic/
Asian
African
Caucasian
Latino
(N=157) American
(N=5,043)
(N=923)
(N=872)
53.5%
54.1%
47.7%
49.7%
12.1%
20.6%
20.1%
16.5%
34.4%
25.2%
32.3%
33.8%
Multiracial
(N=306)
44.1%
24.2%
31.7%
Fewer students had plans to take advanced science classes than advanced mathematics classes.
When asked whether or not they intended to take advanced classes in science, 42.6% of students
indicated that they would, 20.5% confirmed that they would not, and 36.9% reported that they
were “not sure.” Female and male students’ planned to take advanced science classes at similar
rates, with survey data indicating that 41.8% of females and 43.5% of males reported “yes.”
Students of different racial/ethnic groups were more similar in their intentions to take advanced
science than in their intentions to take advanced mathematics. White/Caucasian students were
most likely to indicate that they planned to take advanced science classes (44.5%) while
Hispanic/Latino students and Black/African American students were the least likely, by just
under 5 percentage points (39.1% and 39.3% respectively). Responses varied little by schoollevel.
Student awareness of STEM careers was low; teachers and grant coordinators reported that
activities connecting students with industry increased this. As part of the larger goal of
improving student attitudes toward STEM subjects, the Golden LEAF STEM Initiative aims to
improve student interest in STEM career possibilities as well. Golden LEAF and the grantees
anticipate that increased interest in STEM careers will result in improved student motivation and
attitudes toward STEM subjects.
Findings from the 14 grant coordinator interviews and the focus groups with participating
teachers indicate that many students lacked an awareness of STEM professions and their
characteristics. The teachers and grant leaders also reported, however, that opportunities for
students to visit STEM industries or tour facilities increased these students’ STEM career
awareness. Several of the Golden LEAF STEM Initiative grantees took students to visit local
manufacturing facilities, research and development offices, or other STEM industry sites. A few
grants brought students to local community colleges, colleges, or universities to observe
postsecondary education and training in STEM. In reflecting on their students’ experiences,
teachers commented:
I’ve had some kids that have really realized that they have some specific skills that they
didn’t appreciate before - they have some talents in STEM areas that they can actually
apply to something. … They think, "Wow I could do this as a job someday or I could
apply this to something else that I want to do." It’s been really great to see that.
32
Golden LEAF STEM Initiative
May 2013
Having been one of the teachers that went on [the field trip to visit several STEM
industries], I think the kids realized, "This is why I’m learning what I’m learning in the
classroom."
The field trips have just been amazing for some of the students that don’t have those
opportunities.
After the visit to [a local community college] I had about 10 kids come to me excited
saying, "Yeah, I really want to do this.” I had three kids who came to me and said, "I
know what I want to be now." That's very exciting.
Students have moderate interest in STEM career fields. While interview and focus group results
show that the opportunities provided for some students to visit STEM industries and colleges
began to increase student STEM career awareness, the S-STEM results suggest that participating
students’ interests in STEM careers remained at about the same level as last year (Table 12). On
average, across 12 STEM career areas, 41.6% of students reported that they were “interested” or
“very interested” in professional work in STEM, compared to 40.7% in Year One. It should be
noted, however, that some of the grant-level impacts of visits to STEM industries and colleges as
measured by the S-STEM may have been diminished by the averaging of all grants’ results – not
every grant implemented such visits.
The S-STEM findings on students’ interest in specific STEM career areas also repeated some
results from Year One. The greatest proportion of students indicated that they were “interested”
or “very interested” in veterinary work (51.1%), while the smallest proportion of students
reported that they were interested or very interested in careers in physics (32.1%). Unlike Year
One results, however, interest levels in computer science were higher in Year Two, jumping
from 37.4% interested or very interested in fall 2011 to 42.1% in fall 2012. This 4.7 percentage
point increase in student interest is the largest difference between Year One and Year Two
findings for all students in any career area. Except for biology, courses in the five career areas in
which students expressed the most interest in Year Two (veterinary work, engineering, biology
and zoology, medicine, and this year computer science) are rarely offered in public schools.
Table 12
Student Interest in STEM Careers in Year One and Year Two by Gender and All Students
Proportion “Interested/Very Interested”
Female
Male
All Students
Career
N
Veterinary work
Engineering
Biology &
zoology
Medicine
Yr 1
Yr 2
Yr 1
Yr 2
Yr 1
Yr 2
4,685
66.6%
28.1%
5,812
63.9%
31.1%
4,723
36.4%
71.2%
6,025
38.7%
67.4%
9,412
51.4%
49.7%
11,837
51.1%
49.6%
54.6%
53.4%
41.3%
43.3%
47.9%
48.2%
61.6%
57.1%
38.8%
37.5%
50.2%
47.1%
33
Golden LEAF STEM Initiative
May 2013
Career
N
Computer
science
Medical science
Chemistry
Earth science
Mathematics
Environmental
work
Energy
Physics
Average
Proportion “Interested/Very Interested”
Female
Male
All Students
Yr 1
Yr 2
Yr 1
Yr 2
Yr 1
Yr 2
4,685
5,812
4,723
6,025
9,412
11,837
25.5%
31.5%
49.2%
52.2%
37.4%
42.1%
50.3%
34.2%
33.0%
33.9%
46.6%
36.6%
35.3%
34.5%
34.1%
41.2%
37.2%
39.8%
34.6%
44.1%
41.8%
39.5%
42.2%
37.7%
35.1%
36.8%
40.5%
40.4%
38.6%
37.0%
38.5%
36.9%
36.5%
36.9%
37.5%
36.9%
19.1%
21.3%
38.9%
22.6%
24.5%
39.5%
47.0%
38.2%
42.6%
49.0%
39.4%
43.7%
33.1%
29.8%
40.7%
36.0%
32.1%
41.6%
Note: Bold percentages indicate differences between Year One and Year Two equal to or greater than 5 percentage
points. Responses were recorded on a four-point Likert scale: “not at all interested” (1), “somewhat interested” (2),
“interested” (3), and “very interested” (4).
Female students had the most interest in veterinary work (63.9% interested or very interested),
medicine (57.1%), and biology and zoology (53.4%), and the least interest in energy (22.6%),
physics (24.5%), and engineering (31.1%). Low female interest in engineering careers correlates
with the low confidence and interest female students reported on the S-STEM engineering
attitudes section. Male students expressed the most interest in engineering (67.4% interested or
very interested), computer science (52.2%), and energy (49.0%), and the least interest in medical
science (34.6%), environmental work (36.9%), and medicine (37.5%).
As in Year One, on average, female students expressed a slightly lower level of interest in STEM
careers as a whole (38.9%) than males (44.5%). Analysis of differences in interest-levels
between male and female students within individual STEM careers indicates that males and
females grew slightly more similar in their interests compared to Year One (Table 13). For
example, last year only 25.5% of female students were interested or very interested in computer
science and this year 31.5% of female students expressed such interest, moving closer to the
males’ interest level (52.2%). In all but two STEM career areas the absolute difference between
female and male student interest levels decreased this year compared to last year.
Table 13
Difference between Female and Male Student Interest in STEM Careers
Career
Engineering
Proportion “Interested/Very Interested”
Difference
Female
Male
(female – male)
(N=5,812)
(N=6,025)
(N=11,837)
31.1%
67.4%
(36.3)
34
Golden LEAF STEM Initiative
May 2013
Proportion “Interested/Very Interested”
Difference
Female
Male
(female – male)
(N=5,812)
(N=6,025)
(N=11,837)
22.6%
49.0%
(26.4)
63.9%
38.7%
25.2
31.5%
52.2%
(20.7)
57.1%
37.5%
19.6
24.5%
39.4%
(14.9)
46.6%
34.6%
12.0
53.4%
43.3%
10.1
36.6%
44.1%
(7.5)
35.3%
41.8%
(6.5)
34.5%
39.5%
(5.0)
36.9%
36.9%
0
Career
Energy
Veterinary work
Computer Science
Medicine
Physics
Medical Science
Biology & zoology
Chemistry
Earth science
Mathematics
Environmental work
Note: Responses were recorded on a four-point Likert scale: “not at all interested” (1), “somewhat interested” (2),
“interested” (3), and “very interested” (4).
The differences in levels of interest in STEM careers between students of different
races/ethnicities were smaller than the differences between male and female students. This
finding is consistent with Year One findings (Table 14). Asian students had the largest, average
level of interest in STEM careers (46.0% indicated “interested” or “very interested”) and
White/Caucasian students had the smallest average levels of interest (40.4%). The largest
differences in interest-levels were in: mathematics, in which Black/African American students
had the most interest (48.5%) and White/Caucasian students had the least interest (33.8%); earth
science, in which Asian students had the most interest (48.4%) and White/Caucasian students
had the least interest (36.7%); and veterinary work, in which American Indian/Alaska Native
students had the most interest (54.2%) and Black/African American students had the least
interest (42.8%).
Table 14
Student Interest in STEM Careers by Race/Ethnicity
Proportion
“Interested/Very Interested”
Career
Area
American
Indian/
AK Native
(N=424)
Asian
(N=182)
Black/
African
American
(N=1,518)
White/
Caucasian
(N=7,299)
Hispanic/
Latino
(N=1,380)
MultiRacial
(N=477)
Physics
Environmental work
Biology & zoology
Veterinary work
Mathematics
34.4%
38.4%
47.9%
54.2%
38.4%
37.9%
40.7%
51.6%
42.9%
41.8%
31.4%
34.5%
41.2%
42.8%
48.5%
30.6%
36.2%
49.5%
52.7%
33.8%
37.3%
38.8%
49.4%
52.2%
40.7%
33.3%
41.1%
44.4%
46.5%
36.1%
35
Golden LEAF STEM Initiative
May 2013
Proportion
“Interested/Very Interested”
Career
Area
Medicine
Earth science
Computer science
Medical science
Chemistry
Energy
Engineering
Average
American
Indian/
AK Native
(N=424)
Asian
(N=182)
Black/
African
American
(N=1,518)
White/
Caucasian
(N=7,299)
Hispanic/
Latino
(N=1,380)
MultiRacial
(N=477)
46.2%
42.0%
43.9%
41.5%
40.6%
39.2%
55.0%
43.5%
54.4%
48.4%
49.5%
43.4%
44.5%
39.6%
57.1%
46.0%
48.4%
37.9%
47.8%
41.9%
42.9%
39.2%
47.6%
42.0%
46.1%
36.7%
39.2%
39.1%
38.6%
33.6%
49.0%
40.4%
52.4%
44.2%
47.6%
46.2%
43.7%
41.6%
52.3%
45.5%
45.5%
40.0%
43.4%
38.6%
42.8%
35.6%
49.3%
41.4%
Note: Responses were recorded on a four-point Likert scale: “not at all interested” (1), “somewhat interested” (2),
“interested” (3), and “very interested” (4).
Similarly to Year One, comparisons of overall STEM career interest by school-level show that
upper elementary school students had higher levels of interest across all career areas on average
(49.9% “interested” or “very interested”) than middle school students (38.7%) and high school
students (35.8%). Upper elementary school students expressed greater interest than middle and
high school students in every career area except medicine.
Full demographic comparison and item-level results on student STEM career interest can be
found in Appendices I-K.
Grant coordinators and principals intend to provide more opportunities for students to interact
with STEM industries. Preliminary results suggest that opportunities for students to interact with
STEM industries were increasing student awareness of and excitement for STEM careers,
psychological outcomes related to student attitudes toward STEM. Several grants implemented
these activities.
At the school level, results from the Golden LEAF STEM Implementation Rubric indicate that
while efforts were underway to provide experiences for students to visit STEM industries and
meet STEM professionals, overall schools still had room to provide additional opportunities. On
average elementary, middle, and high school principals rated their schools between the “Early”
and “Developing” levels for this rubric key element (see Table 15). The principals also estimated,
on average, that their students had somewhere between one and two in-school learning
opportunities per year that facilitated exploration of work in STEM industries.
A number of grant coordinators indicated that they intend to build stronger industry connections
and partnerships in future years. The coordinators understand, and are realizing even more
through experience, that partnerships with STEM businesses and public agencies provide
valuable resources. A number of coordinators and teachers expressed a desire to implement more
36
Golden LEAF STEM Initiative
May 2013
field trips to STEM industries for students. A few also communicated concerns that their schools
lack the funding to do so – transportation for high numbers of students can be very expensive, for
example, and this is especially true for schools in particularly rural or remote areas.
Table 15
Student Experiences with STEM Industries by School-level
Average Rubric
Score
Rubric Indicators
Key Element
Early
Developing
Elem./
Middle
(N=76)
(N=20)
High
Prepared
Model
1.7
1.6
1.9
1.9
2.2 Students
and STEM
Professionals
Leaders are
creating plans to
provide
opportunities for
students to meet
STEM
professionals and
to participate in
STEM learning
environments
outside school
Direct experiences
with STEM
professionals and
STEM learning
environments
during and/or
outside school are
available to
students 2 times
throughout the
year
Direct experiences
with STEM
professionals and
STEM learning
environments
during and/or
outside school are
available to
students monthly,
and are directly
connected to inclass learning
Direct
experiences with
STEM
professionals and
STEM learning
environments
during and/or
outside school
are available to
students weekly,
and are directly
connected to inclass learning
8.1 Learning
Connected to
Industries
Program leaders
are researching
and planning inschool learning
opportunities for
students that
facilitate
exploration of
work in STEMrelated industries
1-2 in-school
learning
opportunities for
students facilitate
exploration of
work in STEMrelated industries
Several in-school
learning
opportunities for
students facilitate
exploration of
work in STEMrelated industries
In-school
learning
opportunities for
students that
facilitate
exploration of
work in STEMrelated industries
are frequent
Note: Rubrics were scored on a four-point scale: “early” (1), “developing” (2), “prepared” (3), and “model” (4).
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Student STEM Learning?
In addition to increasing student engagement and student attitudes toward STEM subjects, the
Golden LEAF STEM Initiative grants aim to increase student learning in STEM. The new
curricula and projects provided through the initiative are not only engaging to students but they
also cover challenging material. Six grants, for example, include implementation of the Project
Lead the Way (PLTW) engineering curriculum as one of their strategies. PLTW covers advanced
material in many topics, including measurement, engineering, and design, with a core strategy of
providing a “rigorous and innovative” STEM curriculum.11 Other grants are disseminating Lab11
See the Project Lead the Way website at http://www.pltw.org/about-us/who-we-are
37
Golden LEAF STEM Initiative
May 2013
Aids SEPUP science kits which require students to complete significant amounts of technical
and situational reading throughout the activities.12 Through planning and implementation all
grants have been careful to ensure that materials purchased and activities implemented align to
North Carolina’s content standards, the Common Core State Standards and the Essential
Standards for Science.
Student Learning Outcomes
Students’ problem-solving skills increased. Employers often cite problem-solving and criticalthinking abilities as key qualities they seek in their employees (MetLife and Harris Interactive,
2011; Symonds, Schwartz, & Ferguson, 2011). Problem-solving skills include, among other
proficiencies, the ability to make sense of problems, reason abstractly, reason quantitatively,
construct viable arguments, use appropriate tools, persevere in problem-solving, and attend to
precision. A promising sign, the strongest focus group results related to student learning in Year
Two indicate that students’ problem-solving skills increased through the new STEM activities.
This finding is stronger than in Year One. When asked, “What, if any, changes in student
learning have you noticed,” teachers in almost every focus group described how the authentic,
hands-on, inquiry-based lessons were teaching students problem-solving skills that these young
people had never developed before.
The students are starting to gradually shift into a frame of mind in which learning is not
about an answer any more – it’s about how you got there and what does the answer mean.
It’s been really neat to see their ability to apply what we’ve been discussing with these
investigations … better than when we've been reading articles or doing something with
paper and pencil. They see it work out and then they have “a-ha moments.”
There are fewer now who want to be spoon fed the answers; they’re more willing to think
things through.
Because of the way we are changing our instruction and teaching courses where students
plan, design, test, and re-test when something doesn't work, the students are beginning to
ask themselves, "Okay, now, what do I do?" ... The students are part of their own
learning now. I think that’s the biggest change.
Many teachers described how the inquiry-based, hands-on activities were giving way to higherquality learning for students:
I have more students that are retaining and understanding information on a deeper level.
I’ve watched lessons with the kits, and lessons without the kits. The difference is night
and day when comparing the students' abstract thoughts, connections, and transfers. The
quality of learning is totally different compared to a lesson out of a book … It's huge.
12
See the Lab Aids website at http://www.lab-aids.com/
38
Golden LEAF STEM Initiative
May 2013
The students are actually digging in there, now. It’s not superficial. It’s a deeper learning.
Their answers today during the lab – they realized that by just changing one variable,
they’re creating a fair test. Had they just opened up a text book and read that three weeks
ago, they would not have been able to give that information back, but it’s in their schema.
They know it.
Additionally, teachers described how challenging, problem-based instructional strategies were
building students’ confidence:
Self-confidence is what I’m seeing … They’re just more self-confident after they’ve
struggled … I just see their confidence level increasing the more inquiry-based that the
classes get.
When they’re actually seeing the experiments in action, that’s when the light bulb goes
on, and it does two things. Yes, they learn the knowledge and make a connection, but
what’s most important is their self-esteem goes up.
And in [providing them with problem-based learning opportunities], ultimately, it’s
improving their confidence in themselves… As the self-confidence gets built, and as we
foster it, it’s going to start spreading into other content areas too.
It seems that the more STEM activities I’m doing, the more hands-on I’m doing, then the
more students are saying, “Okay, well, if I try this, maybe this will work.” It might not be
the right thing, but at least they’re willing to try it. There’s more risk taking.
In a few focus groups educators mentioned that students were starting to use the problem-solving
skills they learned in STEM courses in other classes as well.
The opportunity to allow the students to be innovative in the inquiry-based learning
allows them to have that spill-over into the other classes … It affects how they are
learning in the other classes.
They take their problem-solving from science and they apply it to math or to whatever
else they’ve got to do. It can’t help but build them as students, that ability to solve a
problem.
The new materials and instruction connected with mechanical and visual learners. The second
strongest finding from the 14 focus groups with participating teachers suggests that the hands-on,
inquiry-based STEM activities better addressed a wider variety of learning styles among students.
Lecture-based and independent reading lessons tend to favor auditory and some visual learners
and tend to present more of a challenge to kinesthetic or mechanical learners and other visual
learners. Teachers implementing labs, experiments, and computerized simulations through the
Golden LEAF STEM Initiative described how these hands-on, problem-based lessons not only
39
Golden LEAF STEM Initiative
May 2013
favored most learners, but they especially connected with the mechanical learners and strongly
visual learners – students who learn best from practical, applied experiences. The teachers
described:
It is hands-on, so when you’ve got the learners who are oral or kinesthetic, they’re going
to get it from STEM, because the information is coming at them in all different ways. So, I
mean, it’s almost impossible for a STEM lab not to affect a child.
I’ve got some exceptional children students who, typically, if you give them a paper and
pencil activity, will struggle to get focused. But, they’ll do the same thing digitally, using
the same skills, and the interest is there … The exceptional children students are doing
better than some of the other students. It’s the different type of learning that completely
pulls them in.
The hands-on materials make a difference. You’re connecting with all those learners, all
that differentiation is right there. Even your low ones can get it. They may not be able to
write it down, but they sure can verbalize and make those connections … It’s been really
good.
I’ve noticed a change in my very low readers. Some of them are super, super at science.
They may not be able to read real well. They not be able to take good notes, but they can
become really engaged in this activity and totally understand it and explain it to
somebody else, because they’re seeing it. They’re doing it.
Students continued to develop communication and collaboration skills. The ability to
communicate and collaborate with others is another important skillset that students will need
when they enter the 21st century workplace and adult life. Employers today frequently cite these
skills as key competencies for their staff (MetLife & Harris Interactive, 2011; Symonds,
Schwartz, & Ferguson, 2011). Findings from the focus groups and Golden LEAF STEM
Implementation rubrics (see page 50) suggest that students participating in the Golden LEAF
STEM Initiative continued to have frequent opportunities to work together on meaningful tasks
and develop communication skills – almost all of the STEM education kits, labs, investigations,
and curricula incorporate small group collaboration and team work.
Results from the 14 focus groups with participating teachers clearly indicate that students
increased their collaboration skills in Year Two. When asked whether or not they had noticed
changes in student learning, several focus groups described how the STEM activities put students
in small groups and gave the learners opportunities to grow their teamwork skills.
The students learn to look at things from different points of view. They have the ability to
not sit in a row, but to look across the table and ask, “What do you think,” or to be put in
different groups that they’re not comfortable with and figure it out, even though it’s not
their best friend … I think it’s made them stronger. They’ve even actually gotten to where
they will pair with people they normally wouldn’t pair with.
40
Golden LEAF STEM Initiative
May 2013
The activities are helping students develop a skill set, an underlying skill set, that
business and industry have said is missing from students who have graduated over the
last 10 years. That’s the soft skills, being able to come together in a team to discuss a
problem, come up with a formal solution, try it out, and see how that solution works …
They have to work together in this class.
Students’ reading skills and willingness to read more challenging STEM material increased.
When asked whether they had noticed any changes in student learning as a result of the new
STEM activities or instructional strategies, a number of teachers remarked that students’ literacy
skills were improving. The informational and technical materials that students had to read and
comprehend in their science, technology, engineering, or mathematics classes impacted their
overall literacy. Students were also gaining interest and seeking out new information on their
own. A few teachers explained:
I’ve realized that the kits are actually building a huge skill set for the students with
reading, because the labs are presenting information embedded in stories. The students
have to have a whole different set of reading skills to come away with anything … I think
what we’re doing in class is good and is going to contribute to the school’s reading
scores this year.
It’s improving their literacy without them realizing it … When a student is not the highest
reader, they feel discouraged when they know they’ve got to read something … Now, if
they’ve done an experiment, they don’t shy away from the hard books. They think, “I’ve
done the experiment. I can do this. I want to learn more about it, so I’m going to go get a
hard-level book to read.”
To What Degree or in What Ways Were the Golden LEAF STEM Initiative Grantees as a
Whole Effective in Changing Teachers’ Instructional Practice?
The goals of the Golden LEAF STEM Initiative include supporting the ongoing development of
STEM teachers and their use of high quality instructional practices. Specifically, the initiative
seeks to support projects that:

Place priority on curricular approaches that are integrated, use project- and inquiry-based
learning concepts, and/or prepare students for successful completion of Algebra 1 by 8th
or 9th grade.

Include strategies that are comprehensive, incorporating content specific professional
development for teachers and providing relevant career and work connections for
teachers and students.
One component of the complex work of high-quality instruction is inquiry-based teaching. These
strategies focus on asking students chains of questions and giving students opportunities to
discover concepts on their own. Exemplary teaching also provides opportunities for students to
hypothesize, experiment, analyze, reason, and engage in other elements of problem-solving. All
41
Golden LEAF STEM Initiative
May 2013
of this is done while achieving the delivery of content that is challenging to the learner. If
resources are available, good instruction makes use of instructional technology. Additionally,
high quality STEM instruction pays special attention to the integration of multiple subjects, both
highlighting where this integration is naturally occurring and making explicit efforts to
incorporate multiple subjects when applicable.
Teacher characteristics in Year Two
The Science, Technology, Engineering, Mathematics, and Elementary T-STEM Surveys asked
teachers to share basic information about their background (Table 16). Due to the relatively
small number of engineering teachers who completed the Engineering T-STEM Survey (9),
results from this subgroup are not included in the remaining analyses. Full demographic
comparison and item-level results from the teacher surveys, including all respondents to the pilot
engineering teacher survey, can be found in Appendices L-Q.
Analysis of the demographic characteristics of the teacher survey respondents show that, as in
Year One, female teachers outnumbered males both overall and within every subject-specific
teacher survey. Females outnumbered males by far among elementary school teachers (93.1%
and 6.9% respectively). Also, similar to Year One, there was very little racial/ethnic diversity
among the teacher respondents. There were no American Indian/Alaska Native or Native
Hawaiian/Other Pacific Islander teachers, and very few Asian, Black/African American,
Hispanic/Latino, or Multiracial teachers among those who completed the T-STEM Surveys. Of
those participating teachers who hold National Board Certification, the largest proportion was
among the elementary teachers (22.9%) and the smallest proportion among the technology
teachers (14.3%).
Table 16
Teacher Demographic Characteristics and National Board Certification
Demographic
Characteristic
Gender
Female
Male
Race/Ethnicity
American Indian/
Alaska Native
Asian
Black/African
American
Percentage of Respondents
Science
(N=149)
Technology
(N=42)
Math
(N=98)
Elementary
(N=246)
All
Teachers
(N=535)
72.9%
27.1%
66.7%
33.3%
76.5%
23.5%
93.1%
6.9%
81.5%
18.5%
0.0%
0.0%
0.0%
0.0%
0.0%
1.4%
0.0%
0.0%
0.0%
0.4%
1.4%
7.3%
6.1%
3.7%
4.1%
42
Golden LEAF STEM Initiative
May 2013
Percentage of Respondents
Demographic
Characteristic
Native Hawaiian/
Other Pacific
Islander
White/Caucasian
Hispanic/Latino
Multiracial
National Board
Certification
Science
(N=149)
Technology
(N=42)
Math
(N=98)
Elementary
(N=246)
All
Teachers
(N=535)
0.0%
0.0%
0.0%
0.0%
0.0%
95.0%
1.4%
0.0%
92.7%
0.0%
0.0%
92.9%
0.0%
1.0%
93.1%
2.4%
0.4%
93.2%
1.5%
0.4%
18.6%
14.3%
21.4%
22.9%
20.4%
Note: Science, Mathematics, and Technology teachers are all instructors from grades 6-12.
Data collected on the teachers’ years of instructional experience show that elementary teachers
had the least experience overall and technology teachers had the most (Table 17). This trend
differs somewhat from what was observed in Year One, when technology teachers had the
lowest average years of teaching experience. An exact comparison between the two years cannot
be drawn, however, because the Pilot T-STEM Surveys asked teachers to report their exact years
of teaching experience, while the final T-STEM Surveys administered in Year Two provided
teachers three ranges from which to choose.
Table 17
Teacher Years of Experience
Years of Experience
0-3
4-10
11 or more
Science
(N=156)
Technology
(N=27)
Math
(N=94)
Elementary
(N=168)
All
Teachers
(N=445)
12.9%
28.6%
4.8%
35.7%
59.5%
13.3%
30.6%
56.1%
15.5%
31.0%
53.5%
13.7%
30.9%
55.4%
58.6%
Teacher leadership characteristics. Results from the T-STEM Surveys show that participating
teachers overall exhibited very positive attitudes toward teacher leadership and the basic
responsibilities of all educators. When asked if they think it is important that teachers
communicate vision to students, use a variety of assessment data to evaluate progress, use a
variety of data to plan and set goals, establish a safe and orderly environment, and empower
students, 98.5% of teachers on average “agreed” or “strongly agreed”. Less positive
comparatively was the teachers’ perspective on educators’ responsibility for student learning.
When asked if they think it is important that teachers take responsibility for all student learning,
81.3% of respondents agreed or strongly agreed.
43
Golden LEAF STEM Initiative
May 2013
Instructional Outcomes
The activities of the 14 initiative grants address the Golden LEAF Foundation’s goals to develop
STEM teachers and increase the implementation of high quality teaching practices. Every grant
provides curricular material, technological tools, and/or funds for STEM-specific professional
development to support the increased implementation of integrated, project- and inquiry-based
instruction. Some grants, for example, provided professional development for teachers in concert
with the provision of materials and guides for science experiments. A few grants took teacher
leaders, along with district staff and others, to a week-long STEM education planning and
instruction institute in Washington, DC. Still other grants purchased substitute-teacher hours and
gave their participating teachers opportunities to study and practice various STEM instructional
strategies together. Finally, some projects supported their teachers to attend professional
conferences related to STEM teaching and learning.
Implementation of the Common Core State Standards affected the instructional context. While
not directly related to the Golden LEAF STEM Initiative, it is important to note a significant
contextual change that took place during Year Two. In 2011-12 North Carolina, like many states
across the country, changed its mathematics and English Language Arts educational standards
across all grade-levels to the nationally benchmarked Common Core State Standards (Common
Core). The Common Core is different enough from North Carolina’s current standards that the
instructional changes required to meet these new measures have been significant.
The standards for mathematics emphasize deep mathematical understanding for students. They
demand, for example, that students not simply know how to plug numbers into a formula, but
that they understand when to use the formula, why to use the formula, and the very nature and
mathematical derivation of the formula itself. The Common Core standards for mathematics
emphasize that educators at all levels should seek to develop important processes and
proficiencies in their students, including problem solving, reasoning and proof, communication,
representation, and connections. Students are supposed to be encouraged to see math as useful
and worthwhile, and to believe in their own diligence and ability.13
While the Common Core standards for English/Language Arts are equally detailed, they
specifically overlap with STEM education in that they emphasize literacy in all subject areas,
including technical subjects. The standards aim to guide the education system to ensure that
students are able to read, write, speak, and listen across a wide variety of disciplines. The
Common Core standards for English/Language Arts, for example, demand that teachers renew a
focus on teaching students to read informational texts while building their abilities to reason and
cite evidence.
Districts across North Carolina have been providing professional development to teachers to
support them during their first year of instructing to these new standards. It is impossible to say
exactly how this system-wide change impacted the implementation of the Golden LEAF STEM
Initiative, but it is clear that there were connections. Reference to the Common Core transition
13
For more information see http://www.corestandards.org/
44
Golden LEAF STEM Initiative
May 2013
was made at least once if not multiple times in many focus groups and in the interviews with
grant coordinators. The teachers and grant leaders commented on how the change to the
Common Core was taking up a lot of the teachers’ time, energy, and focus relative to other new
activities. At the same time, it was mentioned in a number of focus groups and interviews that
the hands-on, inquiry-based STEM activities of the initiative aligned well with the instructional
changes demanded by the Common Core.
Teachers used hands-on, inquiry-based teaching strategies. Evaluation results from the
interviews and focus groups suggest that the curricular material, lab materials, technology,
professional development, and other instructional supports provided by the initiative grants have
increased teachers’ use of hands-on, inquiry-based, student-centered teaching strategies. One of
the strongest findings, reported in almost all grant coordinator interviews, relates to the changes
the grant leaders saw in teachers’ instructional strategies. Grant coordinators explained:
I am seeing more hands-on activities. I’m seeing more problem-based learning.
I think the teachers are making intentional efforts to use more inquiry-based teaching. I
think that the projects have helped us out with that, and the kits by their nature have
helped.
Two of my 9th grade teachers are now embracing project-based learning, and they would
never have done that two or three years ago. I even have my P.E. teachers doing projectbased learning … In fact, we’ve had buy-in from people who aren’t even qualified to be
in the grant. The excitement about the whole endeavor has just spread.
Before our initiative I would have to say that people liked to define STEM as just what
the acronym stands for, but now this is impacting us by helping people really understand
how STEM will apply to our kids and their lives. It’s equipping teachers. It’s helping
teachers learn to use hands-on, project-based learning.
Clear findings related to changes in instructional strategies also came from the focus groups.
Teachers in almost every conversation described how they and their colleagues had been
changing their teaching. The educators described how they were using more inquiry-based
teaching strategies and letting students explore and discover concepts without direct delivery of
content. Teachers described:
I think STEM has helped a lot with using inquiry-based teaching strategies. It’s making
us really think. The leadership conferences and different things we’ve been going to are
also stressing that. Then we pass it on to our colleagues, and it’s making everybody think,
“Okay, what is it that we really need to say and how do we need to approach this.”
I’m just more open. For example, this year I’ve allowed the kids to create their own
experiment. I would have never done that before my grant’s training. So I’ve become
more open in those inquiry-based values.
45
Golden LEAF STEM Initiative
May 2013
Participating teachers also explained that, relatedly, the teaching and learning in their classrooms
was becoming more student-centered. They commented:
My instructional practices have totally changed from being at the front of the classroom
giving whole group instruction to sitting down and work with one or two children at once
… You are facilitating knowledge. You’re not up there preaching. You’re helping them
investigate and you learn together.
One of the other teachers implementing the kits is almost like a different teacher. She has
really gone from a teacher-led classroom to really taking that step back and letting it be
student-centered, and that is huge.
For us, it’s largely a difference in the way content is delivered. We haven’t so much
gotten any new equipment, per se, but we did get new standards and the school switched
to a project-based learning format, so the way we are delivering that same content is
changing dramatically … The students really are in charge of what’s happening in the
classroom and the teacher really is there as a facilitator.
In doing the investigation, it was really exciting for me as a teacher. It was so good to
hear them making connections … I could teach like that every day of the year. It’s
exciting for the kids. It’s exciting for me … You’re more of a facilitator, you’re
empowering them.
The Science, Technology, Engineering, Mathematics, and Elementary T-STEM Surveys ask
participating teachers to report on the frequency with which they use instructional strategies that
relate to STEM education. The surveys use a five-point response scale of “never,” “occasionally,”
“about half the time,” “usually,” and “every time.” The Golden LEAF STEM Initiative T-STEM
Survey findings correlate with the results from the grant coordinator interviews and focus groups,
and suggest that teachers frequently used instructional strategies specific to STEM (Table 18).
When the results are grouped into three frequency categories instead of five,
“Never/Occasionally,” “About Half the Time,” and “Usually/Every Time,” analysis reveals key
trends in more and less commonly used instructional strategies. (It is important to consider when
interpreting these results that not all of these activities can be done in every class period, all of
the time.)
The three most commonly used STEM instructional activities across all respondents were:
1. “Students work in small groups” – 64.0% teachers reported this happens “Usually” or
“Every Time” during instructional meetings;
2. “Students engage in content-driven dialogue” – 62.1%; and
3. “Students complete activities with a real-world context” – 53.5%.
Importantly, however, the teachers also reported that the three least common STEM instructional
activities were:
46
Golden LEAF STEM Initiative
May 2013
1. “Students learn about careers related to instructional content” – 48.4% teachers reported
this happens “Never” or “Occasionally”;
2. “Students critique the reasoning of others” – 40.2%; and
3. “Students reason abstractly” – 33.6%.
Table 18
Teacher Use of Instructional Strategies Related to STEM Education
During instructional meetings
(e.g. class periods, after school
activities, days of summer
camp, etc.), how often do your
students...
Develop problem-solving skills
through investigations (e.g.
scientific, design, or theoretical
investigations)
Work in small groups
Make predictions that can be tested
Make careful observations or
measurements
Use tools to gather data (e.g.
calculators, computers, computer
programs, scales, rulers, compasses,
etc.)
Recognize patterns in data
Create reasonable explanations of
results of an experiment or
investigation
Choose the most appropriate methods
to express results (e.g., drawings,
models, charts, graphs, technical
language, etc.)
Complete activities with a real-world
context
Engage in content-driven dialogue
Reason abstractly
Reason quantitatively
Critique the reasoning of others
Learn about careers related to the
instructional content
Never
Proportion of Respondents
(N=511)
About
OccasionHalf the
Usually
ally
Time
Every
Time
2.7%
26.7%
23.7%
40.8%
6.1%
1.0%
3.7%
10.6%
28.2%
24.5%
25.4%
51.8%
37.5%
12.2%
5.1%
2.6%
29.9%
22.4%
38.9%
6.3%
2.0
23.1%
25.0%
40.1%
9.8%
3.7%
22.8%
26.4%
42.9%
4.1%
3.5%
22.4%
27.7%
38.7%
7.7%
3.1%
24.1%
25.7%
38.8%
8.2%
1.4%
18.7%
26.4%
44.6%
8.9%
2.0%
3.4%
2.8%
5.1%
15.4%
30.2%
24.8%
35.1%
20.5%
29.8%
30.1%
27.1%
47.7%
33.5%
39.0%
28.6%
14.4%
3.2%
3.4%
4.1%
4.7%
43.7%
22.9%
25.9%
2.8%
Collapsed into the same three categories, “Never/Occasionally,” “About Half the Time,” and
“Usually/Every Time,” and averaged across the entire survey section, findings indicate that the
technology teachers used STEM instructional strategies slightly less frequently than science and
47
Golden LEAF STEM Initiative
May 2013
mathematics teachers (Table 19). See Appendices L-Q for full T-STEM results on teachers’
STEM instructional strategies.
Table 19
Teacher Use of Instructional Strategies Related to STEM Education by Subject Area
Frequency of STEM
Instruction
Never/Occasionally
About Half the Time
Usually/Every Time
Science
(N=325)
28.2%
26.0%
45.8%
Proportion of Respondents
Technology
Mathematics
(N=42)
(N=246)
38.1%
30.3%
15.4%
25.7%
46.6%
44.0%
All Teachers
(N=511)
28.4%
25.5%
46.1%
Note: Science and mathematics teacher results include the elementary teachers who responded only to the sciencespecific T-STEM sections, to the mathematics-specific T-STEM sections, and to both the science- and mathematicsspecific T-STEM sections. A composite score was created for the elementary teachers who instruct both subjects
when calculating the “all teachers” results.
Results from the Elementary/Middle School and High School STEM Program Implementation
Rubrics also suggest that teachers, both those participating in Golden LEAF STEM Initiative
activities and their colleagues, were using project-based instructional strategies that focused on
inquiry-based instruction approximately monthly (Table 20). Project-based learning activities are
unique because they engage students in an extended process of inquiry. Rigorous projects help
students learn key academic content, practice collaboration, communication, and critical thinking,
and often lead to student creation of products or presentations.14 Principals estimated that their
faculties as a whole used project-based instruction almost monthly.
Table 20
Frequency of Project-Based Learning by School-Level
Average Rubric
Score
Rubric Indicators
Key Element
Early
(1)
1.1
Frequency
of ProjectBased
Learning
14
Project-based
learning is used
rarely in all
subject areas
Developing
(2)
Prepared
(3)
Model
(4)
Elem./
Middle
(n=76)
(n=20)
Project-based
learning is used
monthly in all
subject areas
Project-based
learning is infused
throughout all
subject areas,
which includes all
4 STEM content
areas as well as
additional subjects
Project-based
learning is used as
an interdisciplinary
teaching strategy in
all subject areas,
which includes all
4 STEM content
areas as well as
additional subjects
1.7
2.0
High
For more information visit The Buck Institute at www.bie.org.
48
Golden LEAF STEM Initiative
May 2013
Finally, findings from visits to a small sample of participating teachers’ classrooms conducted by
evaluation team members also correlated with interview, focus group, and survey results. The
data suggest that, as in Year One, teachers not only provided positive, organized classroom
environments, but they also continued to provide sound instructional support to students in
STEM classes (Table 21). Comparing Year Two to Year One results shows that educators
transmitted content knowledge and provided students with opportunities for analysis and
problem-solving with slightly more regularity in Year Two. The educators significantly
increased the frequency with which they provided positive, one-on-one, verbal feedback to
students in Year Two (81.7%) compared to Year One (62.1%). Results show a slight decrease,
however, in the regularity with which participating teachers facilitated content-driven, whole
class dialogue in Year Two (52.4%) compared to Year One (55.2%). On average the sample
classroom observations measured a 3.9 on the seven-point CLASS observation scale for the total
instructional support dimension (see Appendix H for complete results).
Table 21
Instructional Support by Percentage of CLASS Protocol Dimensions Scored at a 3 or Greater
Instructional Support Dimension in CLASS Protocol
n
Transmitting content knowledge
Providing opportunities for analysis and problem-solving
Observation Protocols
Yr 1
Yr 2
58
82
82.8%
84.1%
72.4%
75.6%
Providing positive, one-on-one, verbal feedback
Facilitating content-driven, whole class dialogue
62.1%
55.2%
81.7%
52.4%
Note: The official names for the dimensions used in CLASS are slightly abbreviated from what is shown in the table.
The CLASS protocol uses a seven-point frequency rating scale, ranging from 1 to 7. All evaluation team members
have received training and are certified users of the protocol. For more information about CLASS TM, see:
http://www.teachstone.org/about-the-class/
Teachers provided students opportunities to work and learn in teams. Regarding opportunities
for students to meaningfully collaborate, measured changes in teacher behavior correlated with
measured changes in student behavior. As has already been mentioned, results from the T-STEM
Survey suggest that the most commonly implemented STEM instructional strategy was small
group work (64.0% teachers reported this happens “Usually” or “Every Time” during
instructional meetings). In addition, the STEM Program Implementation Rubric measures how
often students in STEM programs have opportunities to collaborate. Elementary, middle, and
high school principals, when asked how often students learn in teams to frame problems and test
solutions (Key Element 8.2), on average rated their schools between the “Developing” and
“Prepared” implementation levels (Table 22). This finding suggests that teachers provided
students opportunities to work in teams more than “occasionally” but less than weekly. (It is
important to note, again, that the rubric asks principals to reflect on the entire operations of their
school, not only events occurring as a result of their participation in a Golden LEAF STEM
49
Golden LEAF STEM Initiative
May 2013
Initiative grant. In this way the rubric results capture the activities of the entire school faculty
and the broader context STEM education at each school.)
Table 22
Frequency of Students Learning in Teams by School-Level
Average Rubric
Score
Rubric Indicators
Key Element
8.2 Students
Work in
Teams
Early
(1)
Developing
(2)
Prepared
(3)
Model
(4)
Elem./
Middle
(N=73)
(N=20)
Students rarely
learn in teams to
frame problems
and test solutions
that incorporate
STEM content
and/or apply
STEM skills
Students
occasionally learn
in teams to frame
problems and test
solutions that
incorporate STEM
content and/or
apply STEM skills,
with clearly
defined individual
and team
expectations
Students weekly
learn in teams to
frame STEMrelated problems
and test solutions
that incorporate
STEM content
and/or apply
STEM skills, with
clearly defined
individual and
team expectations
Students regularly
learn in teams to
frame problems
and test solutions
that incorporate
STEM content
and/or apply
STEM skills, with
clearly defined
individual and
team expectations
2.3
2.5
High
Teachers integrated subjects and desire opportunities to integrate more. Employers in the 21st
century job market are seeking workers with analytical skills and the ability to synthesize
different kinds of information. The focus group protocol asked participating teachers how, if at
all, STEM activities had impacted the frequency or quality of content integration in their
instruction. The findings were mixed. A number of teachers explained that they and their
colleagues were integrating multiple subjects more often in their lessons.
It has put me in the mindset of using a lot more content integration, actually in all of my
other science classes … I think just having been through the grant program has put me
more in that mindset, so I look for those opportunities where I may not have looked for
them before … it’s been a big boost for me and the kids that I teach.
I had no idea at times how well science can be passed through the other subjects …
Integration wasn’t as big of a push before our STEM initiative.
Teachers are working collaboratively to integrate different subjects … I had a student
last week that said, “You don’t sound like a science teacher anymore,” then he said,
“You sound like a social studies teacher.”
At the same time, a number of teachers described how many subjects were still taught in
isolation from each other. They explained that too many teachers have been working in isolation
from many of their colleagues and have been lacking information about other curricula. These
50
Golden LEAF STEM Initiative
May 2013
teachers explained that they would like to be able to integrate more, but need some additional
resources and/or time.
I’m a math a teacher, I’m not a science person. As far as knowing the science curriculum
and some of the stuff that they do, I still don’t know it … If I could open up a book or
something that said, “The science teacher is teaching this and is going to use this math
skill,” it would be a snap. I need that document that takes all of my math skills and
integrates it into the science curriculum.
I don’t get to go in and collaborate with science teachers. I get to collaborate with other
math teachers. I can’t make that connection, and the science teachers can’t make that
connection … There’s just too much disconnect between the subject areas at the high
school level.
Teachers benefited from time to collaborate and need more. The opportunity for meaningful
collaboration among teachers is an important working condition which can lead to growth in
instructional skills of teachers (Jackson & Brueggman, 2009). Collaboration can catalyze high
quality integration as well. Results from interviews with grant coordinators and focus groups
with teachers indicate that the collaborative planning time implemented by several grants has
been beneficial. One grant convened all middle and high school mathematics teachers in the
district for four, half-day collaborative planning and professional development days. The
mathematics teachers had opportunities to share content and strategies horizontally across
schools and vertically across grade- and school-levels. Another grant contracted with a science
education specialist to lead professional learning communities among science and math teachers.
Still other grants capitalized on already structured time for the teachers to collaborate and used
that time to get together entire grade-levels or departments. Some grant coordinators remarked:
I am really proud of how the teachers have been working as a district professional
learning community. When we’ve brought the middle and high school teachers together
there really isn’t a distinction between the middle school and high school teachers,
because they’ve been together so much.
The professional learning teams have really been a foundation for the work because
they’ve put together not just science teachers, but teachers from other areas of the
curriculum.
At the end of last year we made the staff present their lessons to their peers and we did a
critical friends circle exercise … And now I am seeing totally different engagement with
our teachers. I’m seeing cross-curricular people working with each other, people who
they never thought they’d work with.
In focus groups many teachers described how they consider time with each other one of their
most valuable resources and most beneficial professional activities.
51
Golden LEAF STEM Initiative
May 2013
Hands down, the times when we get to actually just work together and hammer things out
have been the times when we have done the most developing units. Hopefully this projectbased idea continues. There’s nothing more powerful than having multiple people,
because everybody brings something different.
We feel very secure when we have it planned. We can do a good job. I think one huge
beneficial thing was the time we were given to collaborate, to really bounce ideas off
each other, to make it the best it could be.
Several focus groups also expressed how they need more collaboration time, and how it would
help them to implement high quality content integration within and across their courses.
The most beneficial thing that could be done is to give us time to work together. We need
time to work together to share some of this stuff.
If we could just have, you know, more time. Our planning time is eaten up. We don’t get
to really work together as much as we need to.
We need to get that time. The magic wand is collaboration. Give us that time, and we’re
good. We need the time. We need the time. Protect our time.
When the curriculum is broad you’ve got to be able to go next door and say, “What can
we do to work together?” And where is the time to do that?
Findings from the STEM Program Implementation Rubric reveal additional information about
the frequency with which teachers integrated STEM subjects and collaborated to plan STEM
activities and lessons (Table 23). The principals reflected on the full range of their school
operations, not only their Golden LEAF STEM program activities, and results show that
elementary school teachers integrated content slightly more frequently than middle and high
school teachers. This is not entirely surprising since often individual elementary educators teach
multiple subjects. On average, however, principals across school-levels reported that roughly
25% of teachers made explicit efforts to integrate science, technology, engineering, and
mathematics. Regarding time to collaborate, results from the STEM Program Implementation
Rubric show that somewhere between quarterly and monthly teachers shared STEM activities or
ideas and planned learning outcomes through professional learning community meetings or
common planning time.
52
Golden LEAF STEM Initiative
May 2013
Table 23
Frequency of STEM Integration and Formal Teacher Collaboration by School-Level
Average Rubric
Score
Rubric Indicators
Key Element
Early
(1)
Developing
(2)
Prepared
(3)
Model
(4)
Elem./
Middle
(N=76)
(N=20)
1.2
Frequency
of STEM
Integration
Up to 25% of
teachers make
explicit efforts to
integrate science,
technology,
engineering and
math, requiring
students to
organize
knowledge across
disciplines
25-50% of teachers
make explicit
efforts to integrate
science,
technology,
engineering and
math, requiring
students to
organize
knowledge across
disciplines
50-75% of teachers
make explicit
efforts to integrate
science,
technology,
engineering and
math, requiring
students to
organize
knowledge across
disciplines
Over 75% of
teachers make
explicit efforts to
integrate science,
technology,
engineering and
math, requiring
students to
organize
knowledge across
disciplines
2.2
1.8
1.3
Collaborative PLCs
Biannually,
teachers share
STEM activities
or ideas and plan
learning outcomes
through
professional
learning
community
meetings and
common planning
time
Quarterly, teachers
share STEM
activities or ideas
and plan learning
outcomes through
professional
learning
community
meetings and
common planning
time
Monthly, teachers
share STEM
activities or ideas
and plan learning
outcomes through
professional
learning
community
meetings and
common planning
time
Weekly, teachers
share or co-create
STEM activities or
ideas and plan
learning outcomes
through
professional
learning
community
meetings and
common planning
time
2.3
2.5
High
Most grant-provided professional development was high quality and more is needed.
Professional development opportunities are a key support structure for teachers and an important
tool for schools to continuously improve their faculty’s instructional practice. The focus group
protocol asked participating teachers to recall the STEM-related professional development they
had received and to reflect on the aspects of it that they found helpful. The most common
response described professional development opportunities in which the teachers themselves
were able to conduct the relevant STEM activity or lab. Teachers in a few focus groups stated
this, explaining that the professional development provided to them through their Golden LEAF
STEM Initiative grant was beneficial because it provided just these kinds of opportunities.
Educators commented:
Being able to physically do an activity helps you understand how you could pull this off
in your classroom and what things you may have to tweak along the way.
Any time that you can get engaged at a staff development such that you can take the role
of the student, to see it from their eyes is beneficial. This way you know, “Okay, this is a
problem with the directions, with my own understanding, so I’m going to have to make
sure to address that.”
53
Golden LEAF STEM Initiative
May 2013
The second most common response from teachers described professional development led by
master teachers or colleagues as being particularly beneficial. Professional development
facilitators who have teaching experience are often best able to describe in detail how a
particular tool or strategy can work in a classroom setting. They are often the most equipped to
address questions related to the complex, dynamic process of teaching.
The most valuable professional development that I’ve gotten is when someone who has
actually taught this stuff comes in and shares things that they’ve actually done
successfully in their classroom … We went to a week-long staff development last summer,
led by two people from [a North Carolina school district]. They showed us what they
actually do in integrated math, because they’ve been doing it for several years. That was
the most helpful professional development that I’ve done in the past 15 years.
A few teachers mentioned that professional development is especially helpful when it provides
teachers with something to implement in their classrooms immediately. Additionally, a number
of teacher explained that opportunities to attend professional conferences were very beneficial –
being exposed to a broad range of ideas and colleagues from across the state and nation greatly
influenced them.
Finally, results from focus groups suggest that teachers believed they would benefit from more
professional development that is specific to their work. Frequently professional development is
delivered to large audiences and is not tailored to teachers of particular subject-areas or gradelevels. Educators, therefore, must find extra time to teach themselves how to use the new tool or
strategy in their particular work. Due to teachers’ full schedules this type of unstructured
professional learning and exploration often becomes secondary to more immediate tasks. One
teacher described:
I want something specific in my area of concentration. A lot of times we have these staff
development things where it’s a broad thing with everyone in there – social studies
teachers, science teachers, math teachers, technology teachers – and then you have to
sort it out for yourself on how you’re going to use it.
Reflecting on some of the professional development provided through their school district’s
Golden LEAF STEM Initiative grant, a teacher commented:
My first training with the [technology tool] was truly one of the best trainings on a
technology piece I had ever been to because it was geared toward math. It was the first
time ever I’d been to a staff development and they told a math teacher how to use
something. Most of the time we’re told, “Well, I don’t know how to do it in math. Can you
figure it out yourself?”
Findings on professional development from the STEM Program Implementation Rubric align
with the findings from the focus groups, and show that professional development specific to
54
Golden LEAF STEM Initiative
May 2013
teachers’ practice was infrequent (Table 24). Principals reported about the context for STEM
professional development at their schools, indicating that, on average:

Teachers participated in large group professional development sessions focusing on
STEM teaching skills;

A job-embedded or practice-based approach to professional development for STEM was
used roughly three times during the school year;

On an annual basis STEM professional development resources for teachers focused on
strategies for teaching specific content to specific types of learners; and

Teachers participated in roughly 15 hours per year of STEM-related professional
development which addressed integrated content, community/industry partnerships,
connections with postsecondary education, pedagogy, or digital learning.
Full results for the STEM Program Implementation Rubric can be found in Appendix R.
Table 24
Frequency and Quality of STEM Professional Development (PD) by School-Level
Key Element
5.1 Individualized PD
5.2 Job-Embedded PD
5.3 Specific to Teachers and
Students
5.4 Frequency of PD
Elementary/
Middle School
(N=74)
1.9
1.6
2.0
1.5
High School
(N=20)
1.8
1.4
1.9
1.4
Note: Responses were recorded on a four-point scale: “early” (1), “developing” (2), “prepared” (3), and “model” (4).
Visits to STEM industries increased teachers’ awareness of real world application and they need
more opportunities. Teachers are a critical tool for increasing student awareness of STEM
careers. They can pass on their knowledge and understanding of the types of skills needed and
jobs available in today’s job market to students. If they have the knowledge, educators can
increase the authenticity of lessons by making real world connections. All of these activities have
the potential to improve student attitudes, engagement, and learning in STEM subjects.
Several Golden LEAF STEM Initiative grants used their funds to provide participating teachers
with unique opportunities to visit local STEM industry facilities and meet STEM industry
professionals. For example, one grant provided teachers the opportunity to visit a local phosphate
plant. Another grant provided a chance to visit a medical research facility, and a third led
students and teachers on an all-day tour of multiple companies and plants. Results from a few
grant coordinator interviews and several focus groups suggest that these experiences were very
beneficial to the STEM educators. The teachers gained new and deeper understandings of the
types of jobs and competencies demanded in today’s workforce. This better equipped them to
55
Golden LEAF STEM Initiative
May 2013
share this information with students and to relate materials with industry-related skills. One grant
coordinator described:
The feedback on the tour from the teachers was really good. They had no idea what
businesses were in our community. They’re busy doing their job. They don’t know what’s
going on through no fault of their own. So that was the hugest impact with the tour.
Teachers commented:
For me the visit was a very important avenue for integration, because we actually talk
about bio-engineering in 8th grade science and the tour allowed me to see more
application – why it’s important to learn about bioengineering, what’s out there in their
future, the different kinds of jobs that are available for bioengineering, and why STEM,
the whole idea of STEM is very, very important.
I remember when we returned from the STEM tour last year that I think the teachers
learned as much as the kids about what the businesses do on a daily basis, and the skills
that they need, and what the kids are going to have to do when they go into those
workplaces … the teachers learned a lot that day.
Results from the STEM Program Implementation Rubric indicate that even though teachers
participating in some of the Golden LEAF STEM Initiative grant activities had opportunities to
go on study trips, most teachers in these schools overall did not (Table 25). Principals reported
that on average some teachers (approaching 50% of their faculty) participated in an applied
learning experience to increase STEM content and career knowledge about once every two years.
Table 25
Frequency of Applied Learning for STEM Teachers
Average Rubric
Score
Rubric Indicators
Key Element
8.3 Applied
Learning
for STEM
Teachers
Early
(1)
Developing
(2)
Prepared
(3)
Model
(4)
Elem./
Middle
(N=76)
(N=20)
Very few teachers
participate in
customized,
applied learning
experiences to
increase their
STEM content or
career knowledge
As many as 50% of
teachers participate
every-other-year in
at least 1
customized,
applied learning
experience to
increase their
STEM content or
career knowledge
As much as 75% of
teachers participate
every-other-year in
at least one
customized,
applied learning
experience to
increase their
STEM content or
career knowledge
All teachers
participate
annually in at least
one customized,
applied learning
experience to
increase their
STEM content or
career knowledge
1.7
1.8
High
Note: “Applied learning” refers to study trips, fellowships, externships, etc.; durations of these experiences could
vary from 1 day to 1 year.
56
Golden LEAF STEM Initiative
May 2013
Findings from the Science, Technology, Mathematics, and Elementary T-STEM Surveys
correlate with results from the interviews, focus groups, and the rubric. They suggest that while
some participating teachers had general knowledge about STEM careers, others did not (Table
26). Middle school teachers were more likely to indicate that they know about STEM careers and
where to learn more about them than high school teachers, and much more likely than
elementary school teachers.
Table 26
Teacher Awareness of STEM Careers and Resources by School-Level
I know …
About current STEM careers.
Where to go to learn more
about STEM careers.
Where to find resources for
teaching students about STEM
careers.
Where to direct students or
parents to find information
about STEM careers.
Proportion “Agree” or “Strongly Agree”
Elementary
Middle
High
All Teachers
(N=233)
(N=189)
(N=75)
(N=506)
41.1%
60.8%
57.3%
51.4%
39.4%
66.0%
55.6%
52.4%
36.9%
62.4%
53.3%
49.6%
33.8%
59.3%
54.1%
46.9%
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
Teachers experienced implementation fatigue. While teachers reported benefiting from particular
kinds of professional development, focus group results revealed that many teachers also faced
implementation fatigue with regard to trainings. Several groups of educators raised this issue.
They explained that while they have been thankful for the professional development, they have
also been concerned about the lack of sufficient time to implement the content of it. Furthermore,
the teachers raised concerns about the level and frequency with which they were asked to change
fundamental aspects of their teaching. Some educators reflected:
It’s wonderful to be able to have the professional development, but you need the time also
to implement those things, instead of having to manage something new coming out.
They’re doing something different every week.
That’s half the problem we have with professional development. You’re given something
and then you don’t have the time to really, fully learn how to do it, to put it into practice.
If you need a follow-up session, there’s never that time built in for it – we just don’t have
the time.
57
Golden LEAF STEM Initiative
May 2013
It takes time to implement and integrate those things into your lessons and what you’re
doing – you have to try some different strategies and some different places to make it
come together for you, and then for the children … You don’t have time to evolve.
Teachers’ felt confident in their own teaching abilities, but divided on whether the classroom
efforts of teachers, in general, impact student learning. The Science, Technology, Mathematics,
and Elementary T-STEM Surveys asked teachers broad questions about their practice. The
surveys asked respondents to reflect on their level of confidence in their own teaching (PSTEBS)
and on their belief that the classroom efforts of teachers, in general, can have an impact on
student learning overall (STOES).
Teachers’ levels of confidence and perspectives on the degree to which educators’ efforts impact
student learning were almost identical from Year One to Year Two. In Year Two, results from
the T-STEM Surveys show that when asked about aspects of their instructional practice,
educators participating in the Golden LEAF STEM Initiative had a strong sense of confidence
and self-efficacy (4.0 scale-level mean composite score on PSTEBS; see Table 27). At the same
time, results also show that the teachers had mixed expectations that these efforts in the
classroom can significantly impact student achievement (3.4 scale-level mean composite score
on STOES). Demographic comparison and item-level results on PSTEBS and STOES can be
found in Appendices L-Q.
Table 27
Teacher Self-Efficacy and Beliefs (PSTEBS) and Outcome Expectancy (STOES) by Subject-Area
Scale
Personal STEM Teaching
Efficacy and Beliefs Scale
(PSTEBS)
STEM Teaching Outcome
Expectancy Scale (STOES)
Science
(N=351)
Mean Composite Score
Technology
Math
(N=42)
(N=261)
All Teachers
(N=535)
3.9
4.0
4.0
4.0
3.4
3.3
3.5
3.4
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
Survey results are the most powerful and reliable when they are considered at the scale-level, or
when a respondent’s answers to multiple, similar questions are synthesized and treated as a
single finding. At the same time, examining individual survey-item results can reveal some
interesting patterns and identify possible areas for further investigation. Table 28 and Table 29
show item-level results for the PSTEBS and STOES scales.
The PSTEBS item-level data on teachers’ confidence and self-efficacy (Table 28) reveal that
overall most teachers felt confident in their own teaching abilities. On average 82.4% of all
teachers “agreed” or “strongly agreed” with each item. PSTEBS findings show that mathematics
teachers had slightly higher confidence (on average 84.8% agreed or strongly agreed with each
58
Golden LEAF STEM Initiative
May 2013
item) than technology teachers (82.0%) and science teachers (80.3%). Results show that all
teachers most strongly agreed that:

They are “continually improving their teaching practice” (97.7% of all teachers agreed or
strongly agreed), and

When teaching their content area, they are “confident enough to welcome student
questions” (94.4%).
Results show that the fewest teachers agreed that:

They “know what to do to increase student interest” in their content area (76.5% of all
teachers agreed or strongly agreed), and

If given a choice, they “would invite a colleague to evaluate their [content area] teaching”
(77.8%).
The largest PSTEBS item-level differences between teachers of various subject areas were in the
following items:

“I know what to do to increase student interest in [content area]” – 83.3% of science
teachers agreed or strongly agreed, while only 65.9% of technology teachers did;

“I know the steps necessary to teach [content area] effectively” – 100.0% of technology
teachers agreed or strongly agreed, 86.5% of science teachers did; and

“I am confident that I can answer students’ [content area] questions” – 94.6% of math
teachers agreed or strongly agreed, 83.9% of science teachers did.
Table 28
STEM Teacher Self-Efficacy and Beliefs (PSTEBS) by Subject-Area
Personal STEM Teaching Efficacy
Beliefs Scale (PSTEBS)
I am continually improving my
[content area] teaching practice.
I know the steps necessary to teach
[content area] effectively.
I am confident that I can explain to
students why [content area]
experiments work.
I am confident that I can teach
[content area] effectively.
I wonder if I have the necessary
skills to teach [content area].
Proportion “Agree/Strongly Agree”
Science
Technology
Math
All Teachers
(N=341)
(N=41)
(N=260)
(N=535)
96.5%
100.0%
96.5%
97.7%
86.5%
100.0%
93.5%
93.3%
84.5%
82.9%
90.4%
85.9%
88.2%
92.7%
93.8%
91.6%
21.1%
12.2%
17.3%
16.9%
59
Golden LEAF STEM Initiative
May 2013
Personal STEM Teaching Efficacy
and Beliefs Scale (PSTEBS)
I understand [content area] concepts
well enough to be effective in
teaching [content area].
Given a choice, I would invite a
colleague to evaluate my [content
area] teaching
I am confident that I can answer
students’ [content area] questions.
When a student has difficulty
understanding a [content area]
concept, I am confident that I
know how to help the student
understand it better.
When teaching [content area], I am
confident enough to welcome
student questions.
I know what to do to increase
student interest in [content area].
Average
Proportion “Agree/Strongly Agree”
Science
Technology
Math
All Teachers
(N=341)
(N=41)
(N=260)
(N=535)
87.7%
97.6%
95.0%
93.4%
74.2%
78.0%
81.1%
77.8%
83.9%
85.4%
94.6%
88.0%
85.9%
92.7%
93.8%
90.8%
91.5%
95.1%
96.5%
94.4%
83.3%
65.9%
80.4%
76.5%
80.3%
82.0%
84.8%
82.4%
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
The item-level findings regarding teachers’ perspectives on the degree to which educators can
impact student learning (STOES) suggest substantial disagreement (Table 29). On average,
across all items, 48.5% of all teachers reported that they “agreed” or “strongly agreed” that the
efforts of educators make a difference for student learning. This means that 50.5% of teachers
reported that they “neither agreed nor disagreed,” “disagreed,” or “strongly disagreed” that the
efforts of educators make a difference for student learning. Findings show that math teachers had
slightly higher expectations that the efforts of educators can impact student learning (on average
52.9% “agreed” or “strongly agreed” with each item) than science teachers (50.3%), and
substantially higher expectations than technology teachers (42.3%).
The respondents were more likely to agree that educators’ actions can cause higher-thanexpected student learning as compared to lower-than-expected student learning:

“The inadequacy of a student’s [content area] background can be overcome by good
teaching” (69.4% of all teachers “agreed” or “strongly agreed”);

“If parents comment that their child is showing more interest in [content area] at school,
it is probably due to the performance of the child’s teacher” (59.4%); and
60
Golden LEAF STEM Initiative
May 2013

“When a student’s learning in [content area] is greater than expected, it is most often due
to their teacher having found a more effective teaching approach” (59.2%).
Results show that the fewest teachers agreed that:

“If students’ learning in [content area] is less than expected, it is most likely due to
ineffective [content area] teaching” (23.1% of all teachers “agreed” or “strongly agreed”),
and

“Minimal student learning in [content area] can generally be attributed to their teachers”
(26.6%).
The largest differences between teachers of various subject areas were consistent with the trend
that mathematics teachers had the highest outcome expectancy and technology teachers had the
lowest:

“The teacher is generally responsible for students’ learning in [content area]” – 60.2% of
math teachers “agreed” or “strongly agreed,” while only 39.0% of technology teachers
did;

“When a student does better than usual in [content area], it is often because the teacher
exerted a little extra effort” – 55.9% of math teachers “agreed” or “strongly agreed,”
36.6% of technology teachers did; and

“When a low achieving child progresses more than expected in [content area], it is
usually due to extra attention given by the teacher” – 62.5% of math teachers “agreed” or
“strongly agreed,” 48.8% of science teachers did.
Table 29
STEM Teaching Outcome Expectancy (STOES) by Subject-Area
STEM Teaching Outcome
Expectancy Scale (STOES)
When a student does better than
usual in [content area], it is often
because the teacher exerted a little
extra effort.
The inadequacy of a student’s
[content area] background can be
overcome by good teaching.
When a student’s learning in
[content area] is greater than
expected, it is most often due to
their teacher having found a more
effective teaching approach.
Proportion “Agree/Strongly Agree”
Science
Technology
Math
All Teachers
(N=341)
(N=41)
(N=260)
(N=35)
47.9%
36.6%
55.9%
46.8%
74.0%
70.7%
63.5%
69.4%
62.1%
51.2%
64.3%
59.2%
61
Golden LEAF STEM Initiative
May 2013
STEM Teaching Outcome
Expectancy Scale (STOES)
The teacher is generally responsible
for students’ learning in [content
area].
If students’ learning in [content area]
is less than expected, it is most
likely due to ineffective [content
area] teaching.
Students’ learning in [content area]
is directly related to their teacher’s
effectiveness in [content area]
teaching.
When a low achieving child
progresses more than expected in
[content area], it is usually due to
extra attention given by the
teacher.
If parents comment that their child is
showing more interest in [content
area] at school, it is probably due
to the performance of the child’s
teacher.
Minimal student learning in [content
area] can generally be attributed to
their teachers.
Average
Proportion “Agree/Strongly Agree”
Science
Technology
Math
All Teachers
(N=341)
(N=41)
(N=260)
(N=35)
52.1%
39.0%
60.2%
50.4%
26.0%
17.1%
26.3%
23.1%
46.4%
43.9%
50.4%
46.9%
52.1%
48.8%
62.5%
54.5%
63.3%
51.2%
63.7%
59.4%
28.7%
22.0%
29.3%
26.6%
50.3%
42.3%
52.9%
48.5%
Note: Responses were recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree
nor disagree” (3), “agree” (4), and “strongly agree” (5).
In general, the participating teachers’ sense of self-efficacy and their expectations for the degree
to which educators can impact student learning did not vary when the teachers were sorted by
other characteristics. Independent comparisons show that teachers instructing in elementary,
middle, or high school had similar levels of confidence and outcome expectancy, as did teachers
teaching in different school-levels within subject-areas (e.g. a middle school science teacher
compared to a high school science teacher), and teachers with varying years of experience. These
results were also found in Year One. See Appendix L for complete demographic comparison
findings.
62
Golden LEAF STEM Initiative
May 2013
Additional Findings
In addition to findings regarding changes in student attitudes, student learning, and teacher
instructional practices, some other results emerged from data collection.
Most students reported plans to attend a college or university after high school. The
Middle/High School S-STEM Survey asked students about their postsecondary plans. Overall,
86.7% of students who responded to the survey indicated that they planned to attend college.
Female students were more likely to report that they had plans to attend college (91.2%) than
males (82.3%). When compared by race/ethnicity, Black/African American students were the
most likely to indicate that they intended to go to college (90.1%), followed by White/Caucasian
students (88.1%), and Multiracial students (86.6%). See tables 30 and 31.
Table 30
Middle and High School Student Plans to Attend College by Gender
Do you plan to go to
college?
Yes
No
Not Sure
Female
(N=3,913)
91.2%
0.8%
7.8%
Male
(N=4,014)
82.3%
3.0%
14.7%
All Students
(N=7,934)
86.7%
1.9%
11.4%
Table 31
Middle and High School Student Plans to Attend College by Race/Ethnicity
Do you plan to
go to college?
Yes
No
Not Sure
American
Indian/
AK Native
(N=265)
84.9%
3.8%
11.3%
Asian
(N=155)
78.7%
0%
20.0%
Black/
African
American
(N=887)
90.1%
1.9%
8.0%
White/
Caucasian
(N=5,078)
Hispanic/
Latino
(N=936)
Multiracial
(N=307)
88.1%
1.7%
10.2%
78.9%
1.7%
19.4%
86.6%
3.6%
9.8%
Of those students who intended to go to college, 22.7% reported that they planned to attend a
community college first, and 77.3% a four-year college or university first. Females were slightly
more likely to indicate that they planned to go to a community college (24.2% reported
“community college” and 75.8% reported “four-year college/university”) than males (21.1% and
78.9%).
When compared by race/ethnicity, Black/African American students were least likely to report
that they planned to attend a community college first (17.1%) and most likely to indicate that
they planned to attend a four-year college or university first (82.9%). Conversely, American
Indian/Alaska Native students were most likely to report that they planned to attend a community
college first (33.9%) and least likely to report that they intended to go to a four-year college or
university first (66.1%). See Table 32 for complete results.
63
Golden LEAF STEM Initiative
May 2013
Table 32
Middle and High School Student Plans to Attend First Either a Community College or Four-Year
College/University by Race/Ethnicity
Postsecondary
Institution
Community
College
Four-Year
College/
University
American
Indian/
AK Native
(N=218)
Asian
(N=118)
Black/
African
American
(N=762)
White/
Caucasian
(N=4,359)
Hispanic/
Latino
(N=707)
Multiracial
(N=254)
33.9%
20.3%
17.1%
21.4%
31.7%
21.7%
66.1%
79.7%
82.9%
78.6%
68.3%
78.4%
When compared by school-level, results show that slightly less middle school students planned
to attend a community college first (22.0%) than high school students (26.4%). Described
another way, slightly more middle school students planned to attend a four-year college or
university first (78.0%) than high school students (73.6%). Full demographic comparisons and
item-level results regarding students’ postsecondary intentions can be found in Appendices I-K.
Leadership for STEM
To collect more information about school leadership and the implementation context for the
Golden LEAF STEM Initiative, the Pilot Leadership for STEM Self-Assessment was
administered to principals of participating schools in January and February of 2013. The results
summarized by survey section below are reported with some caution, since the survey was in the
pilot phase (Table 33). Pilot findings suggest that, on average, principals of schools participating
in the Golden LEAF STEM Initiative believed that they focused most on STEM professional
development (91.4% “agreed” or “strongly agreed”). Results also indicate that principals focused
somewhat heavily on maintaining technical infrastructure to support STEM teaching (84.2%
“agreed” or “strongly agreed”). Principals believed that they spent the least time and energy
working on advocacy and networking related to STEM (57.3% “agreed” or “strongly agreed”).
Comparisons between principals with varying amounts of administrative experience suggest that
less-experienced principals more frequently agreed that they focused on advocacy work related
to STEM (63.5% “agreed” or “strongly agreed” – compared to 53.3% for administrators with 1014 years of experience and 52.3% for those with 15 or more years of experience). Principals with
7-9 years of experience also most frequently agreed that they carried-out a variety of leadership
activities related to STEM education (on average 82.2% “agreed” or “strongly agreed” across all
survey items), while principals with 15 or more years of experience least frequently agreed
(78.0%). Complete item-level results from the Pilot Leadership for STEM Self-Assessment can
be found in Appendix S.
64
Golden LEAF STEM Initiative
May 2013
Table 33
Principal Leadership for STEM by Years of Experience
Pilot
Leadership for
STEM Survey
Section
Vision
Infrastructure
Professional
Development
Shared Decisionmaking
Advocacy
Evaluation
Average
7-9 Years
(N=29)
75.9%
89.7%
Proportion “Agree” or “Strongly Agree”
15 or More
10-14 Years
All Principals
Years
(N=35)
(N=107)
(N=26)
73.4%
79.8%
76.4%
76.0%
86.5%
84.2%
95.6%
86.6%
90.7%
91.4%
82.8%
78.8%
81.2%
83.1%
63.5%
85.7%
82.2%
53.3%
80.0%
74.7%
52.3%
77.7%
78.0%
57.3%
81.5%
79.0%
Note: The number of respondents with 0-6 years of experiences (17) was too small to report. Responses were
recorded on a five-point Likert scale: “strongly disagree” (1), “disagree” (2), “neither agree nor disagree” (3), “agree”
(4), and “strongly agree” (5).
Grant Coordinators’ Reflections on Successes and Challenges
Cultures of STEM education emerged. During interviews, the grant coordinators were asked to
reflect on key successes in Year One. In response to this, several grant coordinators described
how a culture embracing the value of STEM education had begun to emerge in their schools or
communities.
Last year we didn’t see children going home and talking about robotics and being excited,
or hear people in our community get excited about our kids talking about robotics. This
year we do.
It really seems like a project that has snowballed, because there is so much interest and
excitement about it. I don’t want to say you can’t fail with it, but there are so many good
things you could do to precipitate more excitement and more engagement and more
partnership, it just seems like an endless round of success … It’s almost more than I can
even keep in a notebook, the things that are going on in the schools, because everybody is
so excited about it.
Even a few teachers brought up in focus groups the topic of emergent STEM education cultures.
One explained:
The students can talk about math and science and they can get up and explain it, and it’s
all okay. People don’t judge. It used to be that some kids made fun of those who tried to
get up and explain, and I don’t see that as much anymore, because everybody is going
65
Golden LEAF STEM Initiative
May 2013
this way. Kids, teachers, the whole culture is changing, and I think that’s probably the
most beneficial thing that I’ve seen.
Launching programs in Year One was a key accomplishment. Additionally, when asked to reflect
on Year One successes, just over half of all grant coordinator teams described that actually
launching their programs was one of their biggest successes. A few grant coordinators mentioned
that early implementation in Year Two was proceeding a little more smoothly than in Year One
because participants and key stakeholders had bought-in to the programs more. A number of
leaders explained that at some point, however, logistical efficiency and smooth execution cannot
accelerate instructional change any faster. Fundamental changes in pedagogy take time for
teachers and fundamental changes in learning take time for students.
Teacher turnover was a significant challenge. When asked to reflect on their key challenges in
Year One, several grant coordinators and other program leaders identified teacher turnover and
leadership turnover. Many grants experienced the loss of teachers who had participated in the
first year of grant activities, including some who were in key positions. The grant coordinators
recognized these events were out of their control, but still acknowledged that it was difficult to
lose the human resources and capacity for STEM education that they had spent a year building.
Grant coordinator teams anticipated that insufficient resources would be a future challenge and
used data analysis and sustainability planning to begin to prepare. Results from both grant
coordinator interviews and teacher focus groups suggest that a lack of resources for high quality
STEM education raised concerns for the future. A number of teachers remarked that having more
materials for hands-on learning would help them and their colleagues deliver higher quality
STEM education. They expressed gratitude for the materials provided by the Golden LEAF
Foundation, while at the same time acknowledged that they still lacked sufficient resources to
accomplish fully the vision.
Several grant coordinator teams described the sustainability planning they had begun to
undertake, preparing for the conclusion of the grant cycle and the loss of current resources. The
planning included both financial and human resource strategies. Some leaders, for example, had
begun searching for future funding sources. Others described how strategic support for particular
teacher leaders or professional development strategies was intended to build capacity for the
future, when fewer resources would be available. These teacher leaders would be able to share
information gained during the Golden LEAF STEM Initiative with new teachers and students.
Grant coordinators used new information for data-driven decision-making. Finally, a large
majority of grant leaders reported that they had used new data sources for decision-making. Most
grant coordinators had used data collected by the evaluation team to reflect on their own grant,
especially results from the S-STEM Surveys, T-STEM Surveys, and the Golden LEAF STEM
Implementation Rubric. The grant leaders, district leadership, and a few teachers used this data
in making strategic, collaborative decisions for the future. Still other grants created and used
their own measurement tools, including at least two new surveys and one new classroom
observation protocol.
66
Golden LEAF STEM Initiative
May 2013
III. Capacity-Building Activities
The second, main objective of the Golden LEAF STEM Initiative evaluation, besides evaluating
the impact on students and teachers, is to provide technical assistance to the grantees. More
specifically, the goal is to increase the capacity of schools and districts for collecting and using a
variety of data for decision-making. Recent school improvement research has demonstrated that
“capacity problems are too often the barrier rather than the core focus of many reform efforts”
(Roderick, Easton, & Sebring, 2009, p. 16). Additional research finds that consistent and formal
data-informed policies can lead to improvements in education programs overall (Bryk, Gomez,
& Grunow, 2011). For these reasons each grantee is required to take part in several evaluation
capacity-building activities as part of their agreement with the Golden LEAF Foundation.
To support each of the grantees in building capacity for data-informed decision making, the
evaluation team focuses on achieving two, interrelated goals: (1) helping grantees develop and
apply knowledge about education program evaluation; and (2) providing technical assistance to
grantees as they collect, interpret, and use formative data to improve their STEM programs. The
technical assistance also aims to provide grantees with a framework and some common
instruments with which to make these decisions, increasing program coherence across the entire
initiative (Bryk et al., 2011; Honig & Hatch, 2004; Newmann et al., 2001). By the end of the
grant period each grantee will have experienced using traditional and new types of STEM
education data for continuous improvement, explored what types of data are of optimal use, and
used the findings to design and improve programs.
In order to accomplish these goals the evaluation team has carried-out several activities thus far.
The team has: hosted annual face-to-face institutes; held semi-annual webinars; created
initiative-level and grantee-level survey results reports; provided grant-level rubric results;
provided one-on-one reference support; built a wiki for the Golden LEAF STEM Initiative
evaluation; and engaged national and state education leaders in discussions about the on-going
evaluation and capacity-building work for the Golden LEAF STEM Initiative. Capacity-building
activities that have taken place since the writing of the Golden LEAF STEM Initiative baseline
report in August 2012 are described below.
Initiative- and Grant-Level Survey Results Reports
The Science, Technology, Engineering, Mathematics, and Elementary T-STEM Surveys and
Upper Elementary School and Middle/High School S-STEM Surveys serve dual purposes for the
Golden LEAF STEM Evaluation. First, these surveys are measurement tools for the initiativelevel evaluation which seeks to describe the overall impacts of the 14 grants on participating
teachers and students (see Section II). Secondly, grant-level results from these surveys can be a
tool for continuous improvement efforts of individual grantees. By providing grant-specific
survey results back to the coordinators, the evaluation team aims to support the programs in their
decision-making processes.
67
Golden LEAF STEM Initiative
May 2013
On April 1, 2012, after the initiative-wide administration of the S-STEM and T-STEM Surveys,
the evaluation team provided surveys results back to the grant leaders. First, the team sent two
reports summarizing initiative-level findings for the S-STEM and T-STEM Surveys. Data were
presented at the item-level, showing percentages of responses to all survey items in tables and
bar graphs. Demographic summaries were also provided in order to report basic characteristics of
the teachers and students who took the survey. Next, the evaluation team provided two reports to
each individual grant coordinator team summarizing their particular grant’s S-STEM and TSTEM survey results. Like the initiative-level reports, item-level results were reported in tables
and bar charts. Basic demographic data was not provided, however, as an added layer of
protection for the privacy of respondents. The evaluation team encouraged grant leadership
teams to reflect on these survey results, determine how the results could be useful, and then
possibly use the data to inform their future decisions about program design. Grant coordinators
were also encouraged to consider using these reports to reflect on potentially new data collection
strategies in the coming year.
Golden LEAF STEM Implementation Rubric Grant-Level Results
Like the surveys, the Elementary/Middle and High School Golden LEAF STEM Implementation
Rubrics serve dual purposes for the Golden LEAF STEM Evaluation. The rubrics are
measurement tools for the initiative-level evaluation and help to identify the school-level context
for STEM education. At the same time grant-level rubric results can be a tool for continuous
improvement efforts of individual grantees. The evaluation team provided anonymous, grantlevel rubric results back to the individual leadership teams in May 2012, with the aim to support
the programs in their decision-making processes.
Summer STEM Evaluation Institute 2013
The annual, face-to-face summer institute series provides opportunities for the evaluation team
and grantee-leadership teams to discuss the initiative work, share information, and interact over
the course of an entire day. Each summer institute has been held twice, once in Raleigh and once
in Asheville, in order to reduce the travel burden on grantee teams. The final institute in June of
2013, however, will be held only once in Raleigh. This way the initiative culminates in a final
meeting of all participating grant leadership teams. The institute agenda will consist of time for
grant coordinator teams to plan their final year of implementation and evaluation, and time for
the teams to share with each other about their work.
The Golden LEAF STEM Initiative Wiki
In the summer of 2011, the Golden LEAF STEM Initiative evaluation team created a wiki to
organize evaluation materials and initiative resources. A wiki is a website within which users can
add, modify, or delete content using simple editing tools. The web page was created using a
68
Golden LEAF STEM Initiative
May 2013
popular and free service provided by Wikispaces.com. The Golden LEAF STEM Initiative
evaluation wiki is password protected and private so that only users given permission by the
evaluation team may view or edit the page and its content. All 2011 and 2012 Summer STEM
Institute participants used their Wikispaces.com accounts or opened new, free accounts to join
the private web page. The evaluation team uses the wiki to share information about each of the
grant projects, archive materials from institutes and webinars, house STEM education resources,
and manage evaluation activities, including administrations of the rubric and surveys. The wiki is
available at http://glfstem.wikispaces.com/.
IV. Recommendations
The data collected for this report demonstrate that the Golden LEAF STEM Initiative, consisting
of the individual work of 14 grants across North Carolina, made significant progress toward its
goals in Year Two. Findings from all data sources taken together suggest that, compared to Year
One:

Student engagement in STEM learning was roughly as high;

Students’ problem-solving skills increased;

Student development of collaboration skills was roughly as high;

Students had more opportunities to visit various STEM industry facilities;

Teachers increased their use of hands-on, inquiry-based instruction;

Teachers integrated STEM subjects at roughly the same frequency;

Teachers had meaningful opportunities to collaborate with one another and beneficial
professional development opportunities at roughly the same frequency; and

School communities’ awareness and commitment to STEM education increased.
With a final year remaining for project implementation, the initiative has the potential to produce
more positive results and accomplishments. Findings from the data collected for this report point
to some activities which the grantees should continue to prioritize and others which grantees
might consider adding to their implementation plans.
Continue to Implement Hands-On, Problem-Based STEM Curricula and Activities and
Increase Emphasis on Rigor
Results from the focus groups indicate that the hands-on, problem-based STEM curricula,
investigative labs, kits, and other activities implemented by the initiative grantees continued to
have significant, positive impacts on levels of student engagement. The activities also impacted
student learning – students developed stronger problem-solving skills, gained deeper levels of
content understanding, built their confidence, and continued to gain collaboration skills. The
69
Golden LEAF STEM Initiative
May 2013
hands-on, inquiry-based activities had particularly positive impacts on engagement and learning
with kinesthetic learners and with some struggling students. Moving forward, Golden LEAF
STEM Initiative grants should to continue to support teachers to implement these kinds of
activities and instruction. Grants and participating schools could explore possible connections
between students who struggle academically and or behaviorally and their learning styles.
Findings suggest that kinesthetic learners may lack opportunities to learn and excel in ways that
match their strengths.
While the initiative had success with student engagement and student learning, findings from the
pilot student attitudes toward STEM surveys indicate that students had moderate levels of
confidence and interest in mathematics, science, and engineering and technology. Interest and
confidence in these specific STEM subjects may take longer to increase than students’ general
confidence, excitement for hand-on activities, and problem-solving skills. In the future grants
could stay particularly attuned to increases not only in students’ general learning, but also in their
specific learning in these content areas. Continued or new focus could be given to supporting the
provision of rigor when implementing these hands-on activities – supporting students to develop
even deeper understanding of content that is complex, ambiguous, and provocative may
encourage these students to feel more confident in their STEM knowledge and skills.
Continue to Raise Student Awareness of STEM Careers and Increase Opportunities for
Students and Teachers to Engage with STEM Industries; Focus on Females in Engineering;
Further Relationships between Schools and Industry
Interview and focus group findings indicate that both students and teachers benefited from the
grants’ field trips and tours to STEM industry facilities, research centers, etc. S-STEM Survey
results show that students had moderate levels of interest in STEM careers like in Year One.
Female students continued to report slightly lower interest in STEM careers than males,
especially in engineering. Relatedly, findings from T-STEM Surveys and focus groups indicate
that teachers did not spend a lot of time teaching students about careers. Results from the Golden
LEAF STEM Implementation Rubric, grant coordinator interviews, and focus groups suggest
that school faculty and staff knew there was room for growth in the provision of opportunities for
students to engage directly with STEM industries and professionals.
Project leaders should consider the possibility of providing a few more opportunities for students
and teachers to engage with a variety of STEM industries. These activities could include: mini
projects solving industry problems using industry data and equipment; tours of local and regional
STEM industry facilities; visits to local community colleges or colleges; speakers and presenters
at school; or even opportunities for low-level internships for students or externships for teachers,
to name a few. Pursuing relationships with local STEM companies, government agencies, and
colleges or universities, which all also have an interest in well-educated young people, may
create opportunities for partnerships. These relationships could increase the number of such
activities. If possible, grants should consider implementing additional activities that support
female students to develop their interest in engineering and other STEM fields, for example
morning meetings, or lunch or after school clubs. Additionally, to the extent possible, grant
70
Golden LEAF STEM Initiative
May 2013
leaders could encourage districts and schools to offer a wider variety of STEM content. With the
exception of biology, content related to the top five career areas in which students expressed the
most interest – veterinary work, engineering, biology and zoology, medicine, and computer
science – is taught either occasionally or rarely.
Continue Providing Opportunities for STEM Teachers and Others to Collaborate and Focus
on Ways to Support Cross-Curricular Integration
Data collected from the focus groups with participating teachers suggest that the time they spent
in meaningful collaboration with colleagues was an extremely beneficial professional activity.
The teachers were able to help each other improve their lessons. Also, importantly for STEM
education, this collaboration enabled them to integrate their different content areas better. The
teachers were able to build off each other’s content expertise and identify ways their material
connected. At the same time, however, focus group results clearly showed that teachers believed
they lacked sufficient time to collaborate. They also reported that time dedicated to cooperation
sometimes became monopolized by administrative tasks or other duties, instead of collaborative
lesson planning. Other times events bringing together teachers from different departments or
grade-levels lacked sufficient facilitation and valuable opportunities to collaborate meaningfully
were lost.
Golden LEAF STEM Initiative grant leaders could make the case for more cross-subject
collaborative time to those who have the ability to change school schedules. Time is so precious
that instead of looking for new blocks of time for teachers to collaborate, schools could consider
coordinating small groups of teachers from across subjects to design integrated curriculum. Also,
schools could leverage technology tools and programs for online networking, socializing, and
collaboration among faculty. Individual teachers could share live, online documents
communicating their personal curricula and pacing plans. Finally, schools and districts could
ensure that materials identifying where and how different curricula overlap, align, and interact
are available to teachers. Furthermore, all professional development activities could provide time
for participants to discuss and network.
Increase Professional Development Opportunities that are Hands-On, Content-Specific,
Grade-Level Specific, Led by Lead Teachers, and that Offer Immediate Classroom Solutions;
Provide More Time for Teachers to Plan, Experiment, and Implement
In general teachers participating in the Golden LEAF STEM Initiative focus groups were
appreciative of the professional development they received. They benefited the most from STEM
professional development opportunities that were hands-on, content-area and grade-level specific,
led by qualified teachers, and provided resources which could be immediately implemented in a
classroom. This kind of support helped the educators to continually learn and accomplish the
complex, long-term task of changing instruction. A number of educators commented that they
learned a lot from attending professional conferences as well.
71
Golden LEAF STEM Initiative
May 2013
The focus group results clearly indicate that teachers’ lack of time to plan and implement
instructional changes presented the greatest challenge to successful professional development.
The educators very clearly communicated that they were experiencing extreme implementation
fatigue. Teaching is a complex activity and changes to both content and especially pedagogy take
time to successfully implement. Moving forward, grant coordinators could consider focusing
professional development on review and reexamination of content or strategies already addressed
in Year One or Year Two. Teachers will likely benefit greatly from time to revisit the new
curricula and new instructional strategies already introduced, reflecting on their successes and
challenges and adjusting plans for the future. Additionally, grants could consider providing
professional development that exhibits the qualities participating STEM teachers clearly
identified as most beneficial: hands-on, content-area and grade-level specific, led by qualified
teachers, and providing resources which could be immediately implemented.
Find Ways to Have Safe, Professional Conversations about Teaching Philosophies and Beliefs
Results from the T-STEM Surveys suggest that teachers have significantly differing perspectives
on the degree to which instructional efforts impact student learning. Some teachers agreed or
strongly agreed that educators can impact students, in general. Others disagreed. Collectively,
teachers were more likely to accept that instructional efforts contribute to higher-than-expected
student learning than lower-than-expected student learning. When comparing different groups of
educators, findings indicate that technology teachers had lower outcome expectancies than other
teachers.
Grant coordinators and other school leaders could consider using these identified differences in
teaching philosophies and beliefs as a way to build teamwork and a positive, professional culture.
Over time these efforts may help enable educators to have straightforward conversations about
instructional impacts on students that they may not have had otherwise.
Continue to Invest in Sustainability Planning; Continue to Collect Data about the Progress of
Programs and Use them to Strategically Plan for the Future
The grant coordinators explained that they had begun looking towards the future and their ability
to sustain the work after the Golden LEAF STEM Initiative funding ends. In a development
which may support this sustainability work, interview results indicate that several grant
coordinators believed that a culture of interest and excitement for STEM education was building
in some schools and even some business and civic communities. Additionally, a number of grant
coordinators described how they had begun strategically building capacity in order to maximize
current resources for longer term benefits. For example, some grant coordinators had begun
investing in teacher leaders who would be able to share instructional strategies and lead in-house
professional development. Others described how they used rubric and survey data collected
during the evaluation process to have reflective conversations with the participants, building
collective knowledge and planning capacity for the future. Grant coordinators could continue this
72
Golden LEAF STEM Initiative
May 2013
reflective work and strategic planning throughout the final year of the initiative in order to
capitalize on current resources and position themselves for efficient use of resources in the future.
V. Next Steps
The evaluation will continue into the spring of 2014 in an effort to understand the
implementation and impact of the Golden LEAF STEM Initiative and to provide evaluation
capacity-building support to grantees. Table 34 presents evaluation data collection activities and
events that are planned for the summer and fall of 2013 and winter of 2014.
Table 34
Upcoming Evaluation Activities and Events – Summer and Fall 2013, Winter 2014
Event
Spring capacity-building
webinar
Summer STEM Evaluation
Institute 2013
Administration of S-STEM
and T-STEM Surveys
Grant coordinator
interviews
Site visits to grants
Administration of
Leadership for STEM
Self-Assessment and
Golden LEAF STEM
Implementation Rubric
Topics
Evaluation team presents findings
from Year Two
Grant teams plan for final year of
implementation and evaluation; networking
and sharing about grant activities
Participating students and teachers
complete online surveys
about their attitudes toward STEM
30 minute phone interviews with grant
coordinators to discuss early Year 2
implementation
Members of Golden LEAF STEM Initiative
evaluation team visit classrooms and conduct
focus groups with participating teachers
Principals of participating schools complete
both self-assessment and program
implementation rubric
Date
TBD
June 26, 2013
September –
December 2013
October 2013
November 2013 –
February 2014
December 2013 –
February 2014
The evaluation team has several upcoming deliverables as well (see Table 35).
73
Golden LEAF STEM Initiative
May 2013
Table 35
Golden LEAF STEM Initiative Evaluation Deliverables, 2013-14
Deliverable
Golden LEAF STEM Initiative
Evaluation – Year Two
Annual Report
Interim Progress Report # 5
Golden LEAF STEM Initiative
Evaluation – Final Report
Final Progress Report
Period covered
Due date
September 2012 –
February 2013
April 15, 2013
February 2013 –
July 2013
September 2013 –
February 2014
August 2013 –
March 2014
August 2, 2013
April 15, 2014
March 31 –
May 31, 2014
The evaluation team looks forward to continuing its investigation of the impacts of the Golden
LEAF STEM Initiative on STEM education outcomes in North Carolina schools.
74
Golden LEAF STEM Initiative
May 2013
References
Beede, D., Julian, T., Langdon, D., McKittrick, G., Khan, B., & Doms, M. (2011). Women in
STEM: A Gender Gap to Innovation. Economics and Statistics Administration Issue Brief, 4(11).
Bryk, A. S., Gomez, L. M., & Grunow, A. (2011). Getting ideas into action: Building networked
improvement communities in education. In M. T. Hallinan (Ed.), Frontiers in sociology of
education (pp. 127-162). Dordrecht: Springer.
Carnevale, A. P., Smith, N. & Melton, M. (2011). STEM: Science, Technology, Engineering,
Mathematics. Georgetown University Center on Education and the Workforce: Washington, DC.
Corn, J., Faber, M., Howard, E., Lauen, D., & Gaddis, M. (April 2012). Golden LEAF STEM
initiative evaluation: Baseline report. (Golden LEAF Evaluation Report). Raleigh, NC: Friday
Institute for Educational Innovation, North Carolina State University. Available from
http://cerenc.org.
Corn, J., Faber, M., Howard, E., & Walton, M. (August 2012). Golden LEAF STEM initiative
evaluation: Descriptive Data Report. (Golden LEAF Evaluation Report). Raleigh, NC: Friday
Institute for Educational Innovation, North Carolina State University. Available from
http://cerenc.org.
Carlson, D., Borman, G. D., & Robinson, M. (2011). A multistate district-level cluster
randomized trial of the impact of data-driven reform on reading and mathematics achievement.
Educational Evaluation and Policy Analysis, 33(3), 378-398.
Enochs, L.G. & Riggs, I. (1990). Further development of an elementary science teaching
efficacy belief instrument: A preservice elementary scale. School Science and Mathematics,
100(4), 194-202.
Faber, M., Unfried, A., Wiebe, E. N., Corn, J., Townsend., T., & Collins, T. (2013). Student
Attitudes toward STEM: The Development of Upper Elementary School and Middle/High
School Student Surveys. Proceedings of the 2013 American Society for Engineering Education
Annual Conference & Exposition. Washington, DC: ASEE.
Fredricks, J. A., Blumenfeld, P. C., and Paris, A. (2004). School engagement: potential of the
concept: state of the evidence. Review of Educational Research, 74, 59–119.
Glaser, B, & Strauss, A.L. (1967). The discovery of grounded theory. Chicago: Aldine.
Honig, M. I., & Hatch, T. C. (2004). Crafting coherence: How schools strategically manage
multiple, external demands. Educational Researcher, 33(8), 16-30.
Jackson, C. K. & Bruegmann, E. (2009). Teaching students and teaching each other: The
importance of peer learning for teachers. NBER Working Paper 15202. Washington, DC:
National Bureau of Economic Research.
75
Golden LEAF STEM Initiative
May 2013
Kane, T. J., & Staiger, D. O. (2012). Gathering feedback for teaching: Combining high-quality
observations with student surveys and achievement gains (MET Project Research Paper). Seattle,
WA: Bill & Melinda Gates Foundation. Retrieved April 5, 2012 from
http://metproject.org/downloads/MET_Gathering_Feedback_Research_Paper.pdf
Marks, H. M. (2000). Student engagement in instructional activity: patterns in the elementary,
middle, and high school years. American Educational Research Journal, 37, 153–184.
MetLife & Harris Interactive (2011). The MetLife Survey of the American Teacher: Preparing
Students for College and Careers. Retrieved August 15, 2012 from
http://www.metlife.com/assets/cao/contributions/foundation/americanteacher/MetLife_Teacher_Survey_2010.pdf.
National Academy of Engineering. (2008). Grand challenges for engineering. Washington, DC:
The National Academies Press. United States Department of Commerce. (2012). The
Competitiveness and Innovative Capacity of the United States. Washington, DC: United States
Department of Commerce.
Newmann, F. M., Smith, B., Allensworth, E., & Bryk, A. S. (2001). Instructional program
coherence: What it is and why it should guide school improvement policy. Educational
Evaluation and Policy Analysis, 23(4), 297-321.
North Carolina Commission on Workforce Development. (2011). State of the North Carolina
Workforce 2011-2020. Raleigh, NC: Author.
Partnership for 21st Century Skills. (2004). Homepage. Retrieved March, 2006, from
http://www.21stcenturyskills.org/index.php
PCAST, President’s Committee of Advisors on Science and Technology. (2010). Prepare and
Inspire: K-12 Education in Science, Technology, Engineering, and Math (STEM) for America’s
Future. Washington, DC: Executive Office of the President.
Roderick, M., Easton, J. Q., & Sebring, P. B. (2009). Consortium on Chicago School Research:
A new model for the role of research in supporting urban school reform. Chicago: Consortium
on Chicago School Research. Retrieved April 5, 2012 from
http://ccsr.uchicago.edu/publications/CCSR%20Model%20Report-final.pdf
Symonds, W. C., Schwartz, R. B., & Ferguson, R. (2011). Pathways to Prosperity: Meeting the
Challenge of Preparing Young Americans for the 21st Century. Report issued by the Pathways to
Prosperity Project, Harvard Graduate School of Education. Retrieved August 15, 2012 from
http://www.gse.harvard.edu/news_events/features/2011/Pathways_to_Prosperity_Feb2011.pdf.
The William and Ida Friday Institute for Educational Innovation. (2011). Governor Perdue’s
North Carolina Student Learning Conditions Survey (SLCS): Survey Implementation Study.
Raleigh, NC: Author.
76