i2 Learning –
Boston City Package
Evaluation Final Report
Education Development Center, Inc.
Waltham, MA
August 10, 2016
Jackie DeLisi, Research Scientist
Lisa Marco-Bujosa, Research Associate
Tracy McMahon, Research Associate
EDC | 43 Foundry Avenue, Waltham, MA 02453 USA
tel: 617-969-7100
edc.org
i2 Learning – Boston City Package Evaluation Final Report
Contents
Executive Summary ...................................................................................................................................... 3
I.
Introduction.......................................................................................................................................... 5
II.
Evaluation Questions ........................................................................................................................... 5
III. Methodology ........................................................................................................................................ 5
A.
Sample............................................................................................................................................... 5
B.
Measures/ Instruments..................................................................................................................... 6
IV. Results ................................................................................................................................................ 10
V.
A.
Outcomes ........................................................................................................................................ 11
B.
Program Characteristics .................................................................................................................. 13
Discussion and Conclusion ................................................................................................................. 23
VI. Recommendations ............................................................................................................................. 24
Appendix A: Logic Model ............................................................................................................................ 25
Appendix B: Characteristics of Student Survey Respondents ..................................................................... 26
Appendix C: Teacher Survey ....................................................................................................................... 27
Appendix D: Characteristics of teacher survey respondents ...................................................................... 29
Appendix E: Teacher Interview Guide......................................................................................................... 31
Appendix F: Student Focus Group Guide .................................................................................................... 33
Appendix G: Observation Protocol Guide ................................................................................................... 34
Appendix H: Student Survey Responses by Gender and Grade Level ........................................................ 35
Appendix I: Results by model ...................................................................................................................... 38
2
i2 Learning – Boston City Package Evaluation Final Report
Executive Summary
Overview
Education Development Center, Inc. (EDC) conducted an evaluation of the i2 Learning Boston City
Package, a program implemented in Boston-area schools that provides teacher professional
development (PD) and project-based engineering activities for upper elementary and middle school
students. By training teachers and supporting their school year implementation of STEM activities, i2
Learning seeks to address the challenges teachers often face when implementing authentic inquiry- and
engineering design-based activities, and thereby support students’ interest and engagement in science
and engineering. The goal of the evaluation was to provide formative feedback to the project team by
documenting variation in program implementation across schools and developing an understanding of
progress toward meeting desired student outcomes. The evaluation questions focus on first measuring
perceived outcomes, and then on exploring and identifying potential program characteristics that may
be contributing to outcomes.
Methods
The i2 Learning Boston City Package was implemented in 14 schools during the 2015-2016 school year. A
total of 9 schools provided data for the evaluation, and 6 “focus schools” were selected to provide more
in depth data regarding implementation. Student surveys were provided to all schools in order to
document student outcomes. Complementary data collection efforts at the focus schools were intended
to provide qualitative evidence of student outcomes and reveal how and why the program may have
impacted student interest and engagement in engineering. This combination of data sources is intended
to describe not only the student outcomes related to the i2 Learning Week (Evaluation Question #1), but
also to inform i2 Learning Program staff about the aspects of the i2 Learning Week that might contribute
to student outcomes (Evaluation Question #2).
Results
Outcomes
Evaluation Question 1: Are there changes in students’ engagement with and interest in STEM and STEM
careers before and after participation in the school year i2 Learning events?
Overall, across schools, our data reveal that students and teachers found the i2 Learning Week activities
to be both engaging and interesting. Students reported an increase in their interest in engineering,
desire to learn engineering, and their understanding of the importance of engineering, and the
connection between engineering and society. On surveys and in focus groups, students reported science
and engineering is more interesting than they had thought before the program.
Program Characteristics
Evaluation Question 2: In what ways might the i2 Learning experience in schools promote students’
interest and engagement in STEM?
 Which aspects of the i2 Learning experience may be most successful in supporting these outcomes?
 Which aspects of the program, if any, might have additional potential for influencing student
outcomes?
Changes in student engagement and interest in STEM and STEM careers were found across all schools
and across different models of implementation, whether as an intensive week-long experience as
intended by i2 Learning, or a modified/shortened version. These outcomes may be due to numerous
factors, including differences between the i2 Learning Week and the way science is typically taught in
that school. Data indicate i2 Learning activities were more focused on problem-solving, hands-on
manipulation of materials, and collaboration than students traditionally experience in science class.
3
i2 Learning – Boston City Package Evaluation Final Report
Several factors also supported teachers’ implementation of the i2 Learning Week. Teachers, and
especially non-science teachers, reported high levels of comfort and support for teaching the activities.
The PD, especially the opportunity to engage with the module activities, was reported as particularly
beneficial for helping teachers to understand the materials and what students would need to do.
Each school also had a lead teacher who had taught summer school and participated in the PD, managed
logistics within the school, planned the activities within the school’s schedule, and collaborated with
other teachers within their school to plan lessons.
Data revealed some implementation challenges, with most related to logistics, highlighting the need for
consideration of school factors in any potential future scale up. For example, teachers needed additional
support in managing materials especially given limited classroom space for storage and set up. In
addition, teachers stated that additional adult presence was or could be helpful at times for managing
students’ attention and behavior especially during hands on activities. Teachers across schools and
implementation models also described time as a challenge, particularly as it related to materials, and to
time for students to process what they were learning and to redesign and retest products. Finally,
overall, the evaluation noted a tension between the camp format of the week and “school science”;
teachers were challenged to consider ways of embedding the week long activities into their curriculum
in a coherent fashion. Most teachers considered the integration of the i2 Learning Week within their
curriculum to be among their biggest challenges. Some chose to address this challenge by implementing
it as a separate set of activities at the end of the school year, while others chose to further modify the
lessons in order to teach it within the scope of their classroom and curriculum.
Recommendations
Based on the evaluation findings, the i2 Learning Boston City Package appears to be successful in
engaging students in STEM content and encouraging them to “invent & inspire, imagine & innovate,
investigate & inquire, and initiate & improvise” (See i2 Camp’s Mission at http://www.i2camp.org/
about.html). The recommendations below provide general suggestions to improve the teaching of i2
Learning Week, make it more easily adaptable to the school year, and increase the possibility for
successfully scaling up and building capacity as i2 develops plans for expanding i2 Learning Week in the
future.



Increase support for teachers to embed i2 Learning Week into their existing practice, possibly
including connections to other curricula, or guidance for teachers regarding scheduling and
connecting elements of the i2 Learning Week to their instructional practice.
Provide additional support for materials management, including consideration of ways in which
teachers can manage materials and time for set up within the confines of their school buildings.
Provide guidance for schools about the possible benefits for implementing the i2 Learning Week
at different times during the school year, or different methods for implementation.
4
i2 Learning – Boston City Package Evaluation Final Report
I.
Introduction
Education Development Center, Inc. (EDC) conducted an evaluation of the i2 Learning Boston City
Package, a program implemented in Boston-area public, charter, and parochial schools that provides
teacher professional development (PD) and project-based engineering activities for upper elementary
and middle school students. During the summer of 2015 and the 2015-16 school year, i2 Learning
expanded Boston-area programming to include PD for classroom teachers, an opportunity for teachers
to prepare for and practice implementation in a summer camp setting, and a week-long school based
experience (the i2 Learning Week). By training teachers and supporting their school year
implementation of STEM activities, i2 Learning seeks to address the challenges teachers often face when
implementing authentic inquiry- and engineering design-based activities, and thereby support students’
interest and engagement in science and engineering. EDC has now completed an initial formative
evaluation of the first year of the Boston City Package. The goal of the evaluation was to provide
formative feedback to the project team by documenting variation in program implementation across
schools and developing an understanding of progress toward meeting desired student outcomes.
II.
Evaluation Questions
In order to explicitly understand project goals and desired outcomes the EDC evaluators developed an
initial draft of a program logic model (see Appendix A). The logic model was reviewed by i2 Learning
staff and, based on their input, was used to refine the evaluation questions and develop instruments
and measures throughout the design of the evaluation.
The goals of the evaluation were to provide descriptive information to the i2 Learning team about
program implementation in Boston area schools, specifically the most promising elements of the i2
Learning model, including those elements that may be most closely related to any measured or
perceived student outcomes. An understanding of progress toward meeting student outcomes and
potentially promising elements of the program could help inform program improvements and potential
scale up. In order to achieve these goals, the evaluation employed quantitative retrospective student
surveys, teacher surveys, and detailed qualitative measures designed to illuminate participants’
experiences and perspectives. The evaluation questions focus on first measuring perceived outcomes,
and then on exploring and identifying potential program characteristics that may be contributing to
outcomes. The evaluation questions addressed were:
1. Are there changes in students’ engagement with and interest in STEM and STEM careers before
and after participation in the school year i2 Learning events?
2. In what ways might the i2 Learning experience in schools promote students’ interest and
engagement in STEM?
a. Which aspects of the i2 Learning experience may be most successful in supporting these
outcomes?
b. Which aspects of the program, if any, might have additional potential for influencing
student outcomes?
III.
Methodology
A. Sample
The i2 Learning Boston City Package was implemented in 14 schools during the 2015-2016 school year.
All 14 schools were contacted to provide student data in the form of a student survey, described in
further detail below. A total of 9 schools provided quantitative and/or qualitative data for the
5
i2 Learning – Boston City Package Evaluation Final Report
evaluation. Although an effort was made to collect data from all 14 schools, in some cases the school’s
schedule for implementation and district approval for conducting research determined whether schools
were eligible to participate in the evaluation. In addition, a subset of six schools, “focus schools,” were
selected to provide more in depth data regarding implementation and to offer potential insights into
why particular student outcomes were observed in the survey. These six schools were selected because
they represented a range of school types, including public, parochial, and charter schools, as well as a
variety of implementation models for i2 Learning Week that were of interest to i2 Learning staff.
Additional evaluation data collection efforts in focus schools included observations of i2 Learning Week
activities, student focus groups, teacher interviews, and a teacher survey. Table 1 below provides a
summary of the data collected by school.
Table 1. Evaluation data collected from i2 Learning Boston City Package schools
Student
Student
Teacher
Schools
Observations
Focus
Survey
Interviews
Groups
Brooke East Boston
Yes
No
No
No
Brooke Mattapan
Yes
No
No
No
Brooke Roslindale*
No
Yes
Yes
Yes
Eliot K-8 Innovation School*
Yes
Yes
Yes
Yes
MATCH Community Day*
No
Yes
Yes
Yes
Mother Caroline Academy*
Yes
Yes
Yes
Yes
Sacred Heart, Roslindale*
Yes
Yes
Yes
Yes
St. John School-North End
Yes
No
No
No
UP Academy Leonard*
Yes
Yes
Yes
Yes
* Evaluation focus schools
Teacher
Survey
No
No
Yes
Yes
Yes
Yes
Yes
No
Yes
B. Measures/ Instruments
Data collected for this evaluation included both quantitative data, in the form of student and teacher
surveys, and qualitative data, in the form of classroom observations, student focus groups, and teacher
interviews. During the summer of 2015, the EDC evaluation team worked closely with the i2 Learning
program staff to consider and pilot test various measures of student outcomes.
Student surveys were provided to all schools in order to document student outcomes. Complementary
data collection efforts at the focus schools were intended to provide qualitative evidence of student
outcomes and reveal how and why the program may have impacted student interest and engagement in
engineering. This combination of data sources is intended to describe not only the student outcomes
related to the i2 Learning Week (Evaluation Question #1), but also to inform i2 Learning Program staff
about the aspects of the i2 Learning Week that might contribute to student outcomes (Evaluation
Question #2). Each of the instruments, including their development, data collection efforts, and
analysis, are described in further detail below. Table 2 above shows the alignment between the
evaluation questions and each set of instruments.
6
i2 Learning – Boston City Package Evaluation Final Report
Table 2. Alignment between evaluation questions and instruments
Instrument
Student Survey
Teacher Survey
Student Focus Group
Teacher Interview
Observations
Q #1: Changes in student
engagement and interest
X
Q #2: In what ways does i2 Learning Week promote
engagement and interest?
X
X
X
X
X
X
1. Student Survey
Students who participated in the i2 Learning Boston City Package during the 2015-16 school year, were
given the Engineering Interest and Attitudes survey developed by the Engineering is Elementary (EiE)
team at the Museum of Science, Boston1. The Engineering Interest and Attitudes instrument is
comprised of 24 Likert-scaled items representing 12 pairs of reflective pre-assessment and postassessment items. This instrument assesses student attitudes about engineering, specifically student
interest and motivation. A goal of i2 is to engage students in STEM content and encourage them to
“invent & inspire, imagine & innovate, investigate & inquire, and initiate & improvise.”2 Therefore, this
survey has the potential to measure the degree to which the i2 Learning Week influenced student
outcomes aligned with the goals of the i2 Learning Week experience. Additional post-assessment-only
items and some student characteristic data (grade, gender, etc.) were also included on the instrument.
The pairs of reflective pre- and post-assessment items asked students to respond to two statements:
“Before i2 Learning Week, I would have said…” and “Now, I would say…” on a 5-point Likert scale
(Strongly Disagree = 1, Disagree Somewhat = 2, Not Sure = 3, Agree Somewhat = 4, and Strongly Agree =
5). 18 of the 24 reflective pre- and post-assessment items comprised four scales used in past EiE
evaluations and included the sets of scales, as shown in Table 3.
Table 3. Engineering Interest and Attitudes Instrument Scales
Scale
Enjoyment/Interest in
Engineering
Importance of
Engineering
Desire to Learn
Engineering
Value of Engineering to
Society
1
2
Item
I enjoy studying engineering.
Engineering is fun.
We learn about interesting things when we do engineering in school.
When we do engineering, we us a lot of interesting materials and tools.
I am interested when we do engineering in school.
I try hard to do well in engineering.
We learn about important things when we do engineering in school.
It is important for me to understand engineering.
Engineering helps me to understand today’s world.
It is important to understand engineering in order to get a good job.
I would like to learn more about engineering.
I really want to learn engineering.
I would like to work with other engineers to solve engineering problems.
I would enjoy being an engineer when I grow up.
I know what engineers do for their jobs.
Engineering is useful in helping to solve the problems of everyday life.
Engineers help make people’s lives better.
Engineering is really important for my country.
See http://www.eie.org/sites/default/files/research_article/research_file/nasafinalreport2015_0.pdf
See i2 Camp’s Mission at http://www.i2camp.org/about.html
7
i2 Learning – Boston City Package Evaluation Final Report
Student surveys were distributed by i2 staff in paper form to each participating school. Paper responses
were collected by teachers in each school and returned to i2 staff who entered student responses into
an Excel spreadsheet, and then shared with EDC for analysis.
As shown in Table 4, a measure of internal consistency reliability (Cronbach’s Alpha coefficient) was
calculated for each scale. Measures of Cronbach’s Alpha for the post scales were appropriate as they
were near or above 0.70; and pre scale alphas were slightly less than the post with the exception of the
Value of Engineering to Society scale that was significantly lower in the pre.
Table 4. Engineering Interest and Attitudes Instrument Scale Reliability
Scale
Pre Alpha
Post Alpha
Enjoyment/Interest in Engineering
0.837
0.854
Importance of Engineering
0.655
0.700
Desire to Learn Engineering
0.784
0.694
Value of Engineering to Society
0.440
0.652
An additional scale from the Engineering Interest and Attitudes survey collected data on gender beliefs
in engineering, including the following items: 1) Boys are better at engineering than girls; 2) Girls are
better at engineering than boys; 3) Girls have a harder time understanding engineering than boys; 4)
Girls and boys are equally good at engineering; and 5) Boys have a harder time understanding
engineering than girls. One additional item was also asked pre- and post- assessment, but was not part
of any scale: Engineering is easy for me. The five gender items were also tested for internal consistency,
but alphas from both pre and post responses were significantly lower than 0.70. Therefore, all five of
these items were analyzed by item rather than as a scale.
Across the nine schools that provided student data, a total of 346 students responded to the i2 Learning
Week Student Survey. Students represented five i2 courses, with Contagion being the most represented
course with 44% of responses. Respondents ranged in grade level, with grade 6 the most represented
(43%), followed by grade 4 (19%). There was a slightly higher percentage of female respondents (52%)
than male respondents (42%). Additional information regarding the characteristics of students who
completed the surveys is provide din Appendix B.
2. Teacher Survey
Teachers participating in i2 Learning Week through the Boston City Package in the six focus schools were
administered an online survey at the end of each i2 Learning Week experience and between November,
2015 and June 2016. The purpose of this survey was to provide additional information about teachers’
background, school, and experience teaching the Learning Week activities. The survey was developed by
the EDC evaluation team based upon instruments used with teachers in similar evaluations. The survey
consisted of 25 items that addressed basic demographic information in addition to their general
teaching experience and their science teaching experience, how i2 Learning Week was implemented in
their school, the successes and challenges they encountered, and the support they received from i2. See
Appendix C for the teacher survey. Surveys were sent to teachers during the i2 week with a request to
complete the survey immediately following the conclusion of the week. Email reminders were sent to
teachers during the week following implementation, with one additional reminder sent to nonresponders.
8
i2 Learning – Boston City Package Evaluation Final Report
A total of 14 teachers responded to the i2 Learning Week Teacher Survey, with at least one response
from a teacher in each of the six focus schools, representing a 54% response rate. Teachers represented
five i2 courses, with Engineering Ice Cream the course taught most often (29%). Most teachers who
responded to the survey had a Master’s degree, were female, and had taught between 7-10 years.
Teachers represented a range of teaching experience, from first year teachers to teachers with more
than 11 years of teaching experience. Teachers also taught a variety of subjects, with slightly over half
(57%) of responding teachers reporting teaching science. Though most teachers were certified, only a
few were certified to teach science. Half of teachers indicated they had between 1 and 10 hours of
professional development related to engineering, including PD provided by i2 Learning. See Appendix D
for more information about the characteristics of teacher survey respondents.
3. Teacher Interview
An interview protocol was developed by the EDC evaluation team in collaboration with i2 Learning staff.
The goal of the interview was to describe the supports and the challenges teachers faced in
implementing the i2 Learning week, the staff, materials, and resources necessary for successful
activities, the amount of planning and preparation necessary for successful program implementation,
and teachers’ perceptions of the level of student engagement and motivation during i2 Learning Week.
The interview consisted of 17 questions falling into five main categories: (1) teaching background; (2)
implementation of the activities with their students; (3) successes and challenges; (4) planning and
preparation for Learning Week activities; and (5) resources and support they received from i2 Learning.
See Appendix E for the teacher interview guide.
All teachers who taught i2 Learning Week activities were invited to participate in the interview. Teachers
were contacted by email by a member of the evaluation team to set up a time for the interview. All
interviews were conducted at the end of the Learning Week (day 4 at the earliest) or within one week of
completing Learning Week activities. Interviews took, on average, 30 minutes to complete. Most were
conducted as one on one interviews, either in person or by phone; at two schools, interviews were
conducted as a group interview or focus group. All interviews were audio recorded and transcribed for
further analysis by a member of the evaluation team.
4. Student Focus Group
A student focus group protocol was developed by the EDC evaluation team in collaboration with i2
Learning staff. The protocol consisted of nine questions addressing students’ experience participating in
i2 Learning Week activities, their assessment of their own learning, and their interest in science and
engineering. Questions probed students’ perspectives on the similarities and differences between their
i2 experience and their regular science class. See Appendix F for the student focus group protocol.
The intent was for one student focus group to be held at each of the six focus schools; focus groups
were held at five of the six focus schools. Students in focus groups were selected by their teachers and
were intended to represent the ability and interest levels of their classes as well as ensuring
representation of boys and girls. In schools where multiple grade levels were included in i2 Learning
Week, students in the focus group typically included representatives of each class across one grade
level. Pizza was provided to students as an incentive for participation.
Student focus groups typically lasted 30 minutes and took place during the school day, at a time
convenient for students and arranged by their teachers. Focus groups took place on either the last day
of Learning Week or early the following week. Between five and eight students participated in each
9
i2 Learning – Boston City Package Evaluation Final Report
focus group. Focus group conversations were audio recorded and transcribed by a member of the
evaluation team for further analysis.
Due to scheduling constraints at the sixth school, students provided their feedback to three questions
from the focus group protocol when they completed the student survey. These questions were:
1. What did you like best about i2 Learning Week?
2. What did you like least? What was most challenging?
3. What do you think you learned or gained from this experience?
Responses were received from 37 students at this school, and were transcribed by a member of the i2
staff. This information was sent to the EDC evaluation team for analysis.
5. Observation Protocol
The observation protocol was developed by the EDC evaluation team in consultation with i2 Learning.
The evaluation team drew upon a number of sources in developing the protocol, notably the
Dimensions of Success (DoS) observation protocol3. The evaluation team, in collaboration with i2
Learning program staff, determined that field notes focusing on specific dimensions that are most
closely aligned with i2 Learning Week professional development and implementation would provide the
best description of the i2 Learning Week implementation successes and challenges. Therefore, the
evaluators focused on the following dimensions in their observations of the implementation of i2
Learning Week activities with students: 1) the structure of activities (eg. who is teaching, the activities
goals, and what students are doing); 2) the nature of activities (eg. is it teacher directed or student
directed, how students engage in the engineering design process, and the types of questions asked by
the teacher); 3) student engagement and interest; and 4) management. The protocol, therefore,
directed observers to record field notes during the observation, and then summarize afterwards
according to these four themes. (See Appendix G for the observation guide.)
A total of 15 observations were conducted in the six focus schools between November 2015 and June
2016. For each of the focus schools, two or three observations of Learning Week activities were
conducted. These included at least two observations of instructional activities, typically on the second
and fourth day of implementation, and, where applicable, the student showcase. The timing of these
observations within the i2 Learning Week were arranged with the teacher to best align with their
teaching schedule. In schools where students were in more than one classroom, observations were
conducted of the lead teachers’ classroom only if the teachers reported that they were implementing
activities according to the same schedule.
Observations lasted approximately one hour, sometimes longer depending on the activity. Three
members of the evaluation team conducted classroom observations. Initially two evaluators visited each
classroom until they reached a consensus on their observations. After a high level of consensus was
reached, evaluators independently observed classrooms.
IV.
Results
The results of the evaluation are presented below by the two questions guiding the evaluation. First, we
address changes in student overall engagement and interest in STEM after participating in i2 Learning
Week (Evaluation Question 1), followed by an exploration of the aspects of the learning experience that
3
Program in Education, Afterschool & Resiliency (PEAR). (2013). Dimensions of Success: A PEAR Observation Tool.
Harvard University: Cambridge, MA. February 25, 2013.
10
i2 Learning – Boston City Package Evaluation Final Report
may have had an impact, as well as potential areas for leveraging future impact (Evaluation Question 2).
As the results indicate, students reported interest and engagement in STEM increased during the time of
implementation of i2 Learning Week activities. This occurred regardless of how the i2 Learning Week
was implemented in the school, whether as an intensive week-long experience as intended by i2
Learning, or a modified/shortened version. Results indicate this may be due to numerous factors,
including differences between the i2 Learning Week and the way science is normally taught in that
school. Notably, participants found i2 Learning activities were more focused on problem-solving, handson manipulation of materials, and collaboration than students traditionally experience in science class.
Teachers overall felt comfortable implementing the activities given the materials and support provided
by i2 Learning. However, they encountered several challenges with the program itself and fitting the
program into their school day and structure that offer opportunities for improvement in future
implementation efforts.
A. Outcomes
Evaluation Question 1: Are there changes in students’ engagement with and interest in STEM and STEM
careers before and after participation in the school year i2 Learning events?
1. Student Survey
Pre- and Post- Items
The pre- and post-assessment items on the Student Survey were designed to determine students’
perspectives about their interest in engineering both before (retrospective) and after the i2 Learning
Week. To do so, scale scores were calculated by summing the Likert-scale values of each item with in the
scale (from “Strongly Disagree” = 1 to “Strongly Agree” = 5). The responses to these pre- and postassessment items within the four scales were compared using paired samples t-tests.
Overall, students showed statistically significant increases in summed means from pre- to post- scores
on each of the four scales (Table 5), which suggests the following:




Students reported a higher degree of enjoyment and interest in engineering after having
participated in the i2 Learning Week (n=310; p<.001).
Students felt more strongly about the importance of engineering after having participated in the
i2 Learning Week (n=308; p<.001).
Students reported a stronger desire to learn engineering after having participated in the i2
Learning Week (n=312; p<.001).
Students felt more strongly about the value of engineering to society after having participated
in the i2 Learning Week (n=324; p<.001).
Table 5. Engineering Interest and Attitudes Instrument Scale Results
Scale
N
Pre to Post
Mean Change
Enjoyment/Interest in Engineering
310
4.82
Std. Error
Mean
.27084
Importance of Engineering
308
4.75
.24556
-19.343
307
<.001*
Desire to Learn Engineering
312
3.15
.21629
-14.552
311
<.001*
Value of Engineering to Society
324
2.79
.23148
-12.053
323
<.001*
*statistically significant change from pre- to post-assessment
11
t
-17.782
df
309
p-value
<.001*
i2 Learning – Boston City Package Evaluation Final Report
When comparing results by gender, all scales showed increases from pre- to post- assessment. Scale
scores were significant for all four scales for female students and three of the four scales for male
students (all except for Value of Engineering to Society), though there are fewer males in the sample,
which could influence the results. When comparing results by grade, scale scores were significant for all
four scales for all students, regardless of grade level. Appendix H presents more details for the survey
results.
Overall Enjoyment of the Course
One item on the survey asked students about their enjoyment of the i2 Learning Week; students across
all courses reported liking the experience. Overall, 73% of the 330 student respondents reported that
they either “really liked” or “liked” i2 Learning Week (see Figure 1). Close to 80% of respondents in
grades 4, 5, and 6 reported that they either “liked” or “really liked” the experience. Positive responses
for 7th and 8th graders were lower, at 58% and 63%, respectively. Responses to this item were similar for
both genders with approximately 71% of female students and 76% of male students reporting that they
either “really liked” or “liked” the week.
100%
80%
60%
41%
40%
32%
25%
20%
2%
0%
I really liked it
I liked it
It was okay
I really didn't like it
Figure 1: Student Enjoyment of the i2 Learning Week
Qualitative Data—Interest in Engineering
Students also reported on their enjoyment of the course and their increased student interest in
engineering in focus groups. Students across different grade levels reported being more interested in
science and engineering as topics of future study or potential careers after participating in i2 Learning
Week activities. For example, when asked whether or not they thought differently about studying
engineering in the future, one 6th grade student responded, “I want to do engineering. What we’ve been
doing in i2 made me think more about engineering because I saw that creativity can be used.”
Students also noted they are considering different types of science jobs, even if they were already
interested in science, because participating in i2 Learning Week made them aware of different types of
skills and career opportunities in the sciences. For example, after participating in the Engineering Ice
Cream unit, one fourth grade student explained, “I was always interested in being a doctor, but learning
about chemicals was something I was interested in this week.” Another 6th grade student stated, “I think
of engineers like with cars and that stuff, but then I wouldn't think of giant buildings with a bunch of
plants in it. That expands my knowledge of what an actual engineer is, not just a person who makes cars
or electronics.” A potential reason for this shift may be due to the nature of Learning Week activities.
One student described students may be more interested because, “now that we know science isn't just
sitting in a lab with chemicals all day, it's about exploring things, some of us have a different thought
about what we want to do with our life.” As one fourth grade student noted, “What I learned is that
trying new projects is good for you because when I didn't do i2 I felt engineering was boring but now I
think it's really fun.”
12
i2 Learning – Boston City Package Evaluation Final Report
Qualitative Data—Value and Importance of Engineering
After participating in the i2 Learning Week activities students expressed an increased understanding of
the value and importance of engineering and the connections between engineering, other disciplines
and social problems. One student in a focus group summarized the perspectives of others when he
commented, “I learned that engineering plays a big part in life.” Some teachers also described the ways
in which the collaborative and problem-solving nature of the activities reflected the ways in which they
envision engineers working. Through the i2 activities students were exposed to ways in which science
and engineering could be used to solve important problems, like the presence of viruses, or the need for
food.
B. Program Characteristics
Evaluation Question 2: In what ways might the i2 Learning experience in schools promote students’
interest and engagement in STEM?
 Which aspects of the i2 Learning experience may be most successful in supporting these outcomes?
 Which aspects of the program, if any, might have additional potential for influencing student
outcomes?
The second evaluation question is designed to understand the aspects of the program that may have
contributed to the outcomes documented above. In this section, we explore the aspects of the i2
program that might help to explain why students reported an increased interest in engineering and
overall enjoyment of the i2 activities. We first examine the results by implementation model: an
intensive experience (model 1) or a modified implementation model (model 2); followed by a
consideration of aspects of the i2 Learning package itself, specifically: 1) the nature of the activities
(including differences from typical science instruction); 2) supports provided by i2 to increase teacher
comfort (particularly the curriculum, physical materials, and training); and 3) school-based factors that
may have impacted teachers’ implementation and, consequently, student experience of i2 Learning
Week. Areas in which schools and teachers may require additional support are also discussed.
A. Implementation Model
Schools implemented the i2 Learning Week activities in a variety of ways to accommodate their school
schedules and meet their students’ learning needs. At the onset of the project and through the summer
months, we worked closely with the i2 Learning program staff to understand and anticipate the factors
by which implementation might vary across schools. Using input from i2 Learning staff, as well as our
own interviews and observations of implementation, we determined two possible implementation
models which enabled our team to explore possible characteristics that might relate to outcomes: the
full stand-alone i2 Learning Week (model 1), and embedded i2 Learning activities (model 2). The section
below examines results of student surveys and qualitative data by these two implementation models in
order to determine any potential differences. Additional data, disaggregated by implementation model,
is presented in Appendix H.
1. Student Survey
T-tests were run for all student respondents by school implementation model. The two models were
defined as:
 Model 1 = Full, stand-alone, separate week implemented as a special event, immersive, allday/all-week experience. In some schools students were combined across classes and teachers
across disciplines may have worked together.
13
i2 Learning – Boston City Package Evaluation Final Report

Model 2 = Represents a range of implementation models from a half-day special, immersive
experience to mostly embedded in regularly scheduled science class. In these schools i2
Learning may have been presented as a separate activity, but in contrast to model 1, it was not
conducted as a full immersive experience. In some of the model 2 schools, classes may have
been combined or rearranged.
When comparing implementation models, scale scores were significant for all four scales regardless of
model type. However, students who participated in model 2 showed higher increases than those from
model 1, as shown in Figures 2 and 3 below.
0
5
10
15
20
25
16.80
Enjoyment/Interest in Engineering
20.79
16.65
Importance of Engineering
Pre
20.74
Post
11.77
14.22
Desire to Learn Engineering
14.97
17.28
Value of Engineering to Society
Figure 2: Mean Attitude Scale Scores Pre- and Post- for Model 1 Schools.
0
5
10
15
20
25
17.09
Enjoyment/Interest in Engineering
23.18
16.74
Importance of Engineering
22.53
Pre
Post
12.37
Desire to Learn Engineering
16.69
14.25
Value of Engineering to Society
17.77
Figure 3: Mean Attitude Scale Scores Pre- and Post- for Model 2 Schools.
Results from the student survey item about overall impression of the i2 Learning experience indicate
students who experienced model 2, a partial experience, were slightly more positive about i2 Learning:
70% of the 208 students who experienced model 1 reported that they either “really liked” or “liked” the
14
i2 Learning – Boston City Package Evaluation Final Report
i2 Learning Week experience, and 78% of the 122 students who experienced model 2 reported that they
either “really liked” or “liked” the experience. (see Figure 4 below)
100%
80%
60%
40% 42%
40%
30%
36%
26% 22%
20%
3%
0%
0%
I really liked it
I liked it
It was okay
Model 1
I really didn't like it
Model 2
Figure 4: Student Enjoyment of Course, by Model
2. Qualitative Data by Model
Results from teacher surveys and interviews and the student focus group indicate that while teachers
and students perceived benefits related to both implementation models, there were both advantages
and disadvantages to the full immersion week model. Regarding the full immersive week (model 1), one
fourth grade teacher explained,
There was a lot of continuity day to day. My [students] struggle more with retaining things. It was a
lot easier for them to have a consistent routine. They knew what they were doing every day and
could build on their learning from the day before. Also, for teachers it was good to only have to do all
of that materials stuff for one week because I think it would be overwhelming to do that for a lot
longer than that.
On the other hand, many students and teachers commented that the week was intense, that following
the five-day schedule may have been limiting, and perhaps focusing on one topic per day would be
preferable. For example, one teacher, who implemented the i2 Learning Week for a short period every
day (Model 2), appreciated the ability to focus on one topic each day rather than fitting in a number of
different activities and topics into a full day, as would be necessary in the immersive experience.
Another teacher, who implemented the unit during regularly scheduled science lessons, described the
benefits of a modified approach to implementing Learning Week: “I think that it was really cool for the
kids ... we kind of only tackled one thing every day. So we could tease the next thing for the following
day. I wonder if that increased engagement.”
Some students who experienced model 1 echoed this sentiment, and noted that they were
overwhelmed by all of the new information and topics that were presented in the immersive week:
We did a lot in each day and sometimes we didn't get to do all that [the teacher had planned]. They
can [Teachers should] do something so we work on something specific that day so we're not
confused about what we're doing but we're excited about doing the thing but don't have to rush to
get to the next thing. So one day we're learning about irrigation and not moving onto lights and
mirrors.
15
i2 Learning – Boston City Package Evaluation Final Report
Overall, however, teachers across schools and models described challenges of implementation as
related to the week being a separate, isolated activity not connected to the rest of the science
curriculum or other subjects. According to one teacher, “It was hard devoting all my day time with the
one class and putting my regular job … on the back burner.” When asked whether they would run the i2
Learning Week in their schools again in the future one teachers stated, “I might consider running it, if we
did it at the end of the year like we did for this one. Otherwise, a week's worth of projects don't actually
fit into our curriculum easily.” However, teachers who implemented i2 at the end of the school year also
noted that they ran out of time:
The biggest challenge I felt was we did this as a two-hour block and that was good but I always felt
like I could use 15 more minutes every time. It was good to keep them on track, but I felt like a drill
sergeant in terms of watching the clock. We also have 2 days of school left, so they were excited
about it but motivation was low.
Another teacher stated, “Under the current requirement of presenting the program in a week, the
program impacted too many other teacher schedules… I'd be happy to implement [the module] as a 4week STEM unit. I think the concepts would be internalized better and there would be more
opportunities for assessment and re-teaching.”
It is also important to note that the majority of schools chose to implement the i2 Learning week toward
the end of the school year, in May or June, at a time when teachers are typically looking for activities
that may not be related to their regular curriculum but might continue to keep students engaged. One
school implemented Learning Week in the fall but between grading periods, and another school
implemented the activities in the winter (February), but taught the activities only during regularly
scheduled science instruction over the course of two weeks. This further highlights potential challenges
teachers may have had in determining how to integrate the i2 Learning Week into their classrooms and
instruction. According to one teacher, “the entire program as a summer camp totally makes sense. I
think incorporating it into the curriculum that already exists in schools is where the challenge is. So what
we did, making it an end of the year thing where it was fun and hands-on was a good decision, but it
would have been difficult to coordinate with the rest of the year”. Another teacher summarized this
sentiment well:
I think the fact that because of the last week of school you need something hands on that doesn't
really impact anything, like a task. It just felt like a task. I wouldn't grade, I wouldn't know what to do
if it was done for a grade or earlier in the year.... with the engineering design process, I would want
something either really free and you have one goal and you're working on it for many days, or
something to incorporate the engineering design process at the end of each unit [during the school
year]. We learn the content and then do something with the engineering design process to apply
what we learned. This [module] is separate from our curriculum rather than incorporating or
extending our curriculum. When I was in PD, I had this constant feeling of why am I doing this? How
is this connecting to the teaching I'm doing the rest of the year?
B. Learning Week Activities as Different from Typical Instruction
Students and teachers described ways in which the i2 Learning Week was different than the ways in
which science is typically taught in their school. The key differences, which likely impacted student
interest and engagement in STEM, were opportunities to manipulate materials, to solve problems, and
collaborate with their peers.
16
i2 Learning – Boston City Package Evaluation Final Report
On the survey, students were asked to respond to the open-ended question, “How was learning this
week different from a typical week at school?” Responses were coded using an emergent coding
scheme, categorized, and then compared by model. The results indicate five themes regarding the ways
in which students describe the i2 Learning Week as different from their typical science classrooms: the
class composition; the content of the lessons being taught; the structure of their school day; their level
of interest; the amount of collaboration required; and the teaching style. Examples of student
statements regarding each theme are presented in Appendix I, Table 6. When examining the data by
implementation model, students in model 2 schools did not mention differences in class composition,
and very few responses indicated differences in the structure of the school day. The remaining themes
were present in responses from students across schools within both implementation model types.
Students also echoed the sentiment that Learning Week activities were more engaging than regular
science class, partially due to course content and teaching style. On the topic of engineering, another
fourth grade student commented, “It sounds boring, but when you actually do it, it's more fun. Science
is way funner than usual this week.”
1. Hands-on and Problem Solving
The qualitative data sources provided additional insight into reasons why students and teachers may
have perceived the i2 Learning Week as different from typical science class, and potentially more
engaging. For example, one recurring theme is that students and teachers attributed the success of i2
Learning Week to the opportunities for students to engage in hands-on and problem-based learning
experiences. For many students and some teachers this was perceived as different from their typical
science instruction. Specifically, students stated that they enjoyed learning about the engineering design
process and the ability to work with their hands to learn. One sixth grade student drew this as the main
distinction between the way science is normally taught in this school and i2 Learning Week:
It was like different because last year we barely got to engineer, all we had is a paper and just you
work on it. But here we did the engineering process, we actually built more things than last year, and
we actually learned a little bit more and understand more.
Similarly, in the teacher surveys and interviews, teachers noted that students were engaged and
interested in the i2 activities, sometimes more so than in regular science class. According to one
teacher, “I would say the energy level was probably consistently higher [than usual]. I try to do hands-on
activities with some frequency, but every day was really good for kids, and they were consistently
excited to know what we were doing next.” On the teacher surveys another teacher noted, “The course
was so hands-on! My students loved how learning about engineering was grounded in actual
engineering challenges.”
Observations also indicated that student engagement was highest for the activities when students were
actively engaged in the design or redesign process, as opposed to lessons when they needed to follow
specific steps, when student motivation and attention wavered. For example, during observations of the
Prosthetics unit, students were less engaged, as indicated by off-task behavior, in the lesson that
required following steps, such as setting up the base hand model, as opposed to building the tower. In
the hands-on lesson, students followed directions at different rates, and student attention wavered,
with a number of students not participating in the activity with their team during the observed time.
Teachers also described some of the specific activities and commented the most successful activities
were the ones in which students were directly engaged in the design process, and with materials that
17
i2 Learning – Boston City Package Evaluation Final Report
were easy to manipulate. This was particularly true for the younger grades, markedly teachers who
implemented the unit with the fourth grade. Below are several quotes from teachers describing student
engagement.
 “They were engaged in anything they were designing the process or making the actual thing.”
 “They had criteria but they could be as creative as they wanted with turns and different
elevations. Starting with the marble could be a product start to finish. They could do it over and
over again. They could mistake and go and fix it. They never gave up, they really enjoyed that
one.”
 "I think it was the freedom to design. The other things were following step by step directions
and there wasn't a lot of creativity involved. In this, it was, ‘here are materials, figure out how
it's going to work.’ So they were engaged in build, test, redesign process most."
During focus groups, students also described reasons why specific activities may have been more
engaging: “Making the marble run because I like how you can go creative and add and subtract
whatever you want and you don't get instructions.” Another student highlighted the importance of
struggling through challenges; “I really liked when me and my partner were struggling to make the wind
turbine and we sort of made it.”
Additionally, students believed that they were able to learn more science content with this hands-on
approach than in their traditional science class. According to one student, “We got to experience what
we were learning about, so when we learned about irrigation, we actually got to do it. I feel like in
science class if we were to learn about that it would be easier to forget, but since we got to experience
it, it was easier to remember and know exactly what it is.” Related to the problem-based nature of the
activities, another student said, “I think that was cool how we got to learn stuff on our own, but also the
teacher was there just in case. It was cool she gave us an opportunity to learn stuff by ourselves and
learn from our mistakes.”
Although teachers and students clearly appreciated the hands-on and problem solving nature of the
activities, some commented that they needed more time for discussion, redesign or retesting, and
assessing student thinking. For example, one teacher stated, “I think a challenge we faced together is
that I don't think I always left enough time for the processing.... I think one big difference from the
summer camp is that in class it's harder to test the team groups.” In one of the focus groups, a student
stated, “That [activity] was kind of hard because you had limited time for redoing the assignment that
you were supposed to do, and so, it was like, it made us rush and to think if something worked, and if it
doesn't work, what we were going to do next."
2. Reaching all students
Teachers also reported that the hands-on nature of the activities were important for engaging all of the
students. “There were different types of opportunities and experiences to engage with. The variety that
was provided, there was something for everybody to feel successful and happy about. It was hands-on,
it was fun… so it was a big change, which they were very excited about.” According to another teacher,
“I could see a lot of students who are not normally engaged in science class who were really interested,”
And another teacher described new roles that students took on through the i2 Learning Week activities.
“Some kids surprised me how much they stepped up in terms of being leaders, in the problem solving
and redesign process. Some kids I thought would be interested, and they weren't. Some kids I thought
would be frustrated, and they weren't. Some kids got frustrated, and it was interesting to see them
power through.”
18
i2 Learning – Boston City Package Evaluation Final Report
3. Collaboration
Students appreciated the ability to collaborate with their peers during Learning Week activities and
noted this aspect as different from typical science class. Many students commented that their science
class normally involved independent seat work. Therefore, the ability to work in groups to solve
problems was novel and exciting.
From the focus groups: Students described enjoying the opportunity to work with their peers.
 "Sometimes we don't get to express all our ideas in school, so if we are in groups, we get to
express our ideas to the small group and feel comfortable with the people we're with"
 “I liked that we were with other people that we haven't spent much time with and we listened
to other people's ideas and how to mix up the ideas that everyone approved of”
In the interviews, teachers also commented on the collaborative nature of the activities. They noted that
students were able to collaborate with their classmates during the i2 Learning Week more often and
more intensely than in regular science class.
 “The kids worked well together in small groups. I mixed them up a little more than usual. I did
notice them all taking more risks, working together more than they would. A lot of kids who
were quicker became teachers or helpers… I think that experience in itself led them to be
helpers to get one another through this.”
 “They were for the most part engaged in constructing it and figuring it out, talking about what
was the best way to do things and set it up. For the most part, they were able to have good
constructive conversations about it. So, they did work well together.”
4. Opportunities for interdisciplinary connections
Some teachers also noted the interdisciplinary nature of the activities, though this varied by school.
Teachers in the schools with the full stand-alone implementation model (model 1) were more likely to
describe the connections between i2 Learning Week activities and other non-science disciplines.
According to one teacher:
Even though it seems like it is primarily a science week, the fact that they're writing PSAs, they're
writing directions, it really draws in skills they are going to need in other subjects in a way that it
doesn't feel like this is what you have to do for English, ok, now let's go back to science, now it's
off to math. They don't realize that it's, not everything is separated, everything is really
integrated in the real world and this is a better way of being shown it without it explicitly being
said. Because it's a science class doesn't mean art doesn't exist in science or English doesn't exist
in science.
C. Teacher Preparation and Support
In addition to understanding the ways in which i2 Learning Week was implemented, we examined the
ways in which teachers’ preparation for teaching i2 Learning Week and engineering may have
contributed to perceived successes and challenges. On the teacher survey, respondents were asked
about their comfort teaching science and engineering (see Table 6 below). Nearly all teachers reported
feeling comfortable teaching science (93%), but fewer teachers felt comfortable teaching engineering to
their students (79%).
19
i2 Learning – Boston City Package Evaluation Final Report
Table 6. Teachers’ ratings of their level of comfort leading science and engineering lessons.
Response
1 Not very comfortable
2
3
4
5 Very comfortable
Total (N=14)
Science %
7%
0%
0%
57%
36%
100%
Engineering
%
0%
7%
14%
64%
14%
100%
Yet, teachers reported feeling comfortable implementing i2 Learning Week activities. Through the
teacher survey and teacher interviews, teachers attributed this comfort teaching i2 Learning Week
activities to supports from i2 Learning, particularly the materials, as well as training (professional
development and summer teaching), and the curriculum itself.
On the survey, teachers were asked to describe any supports that i2 Learning staff provided while
implementing the i2 Learning Week at their school. Teachers appreciated that materials were provided,
were prepared by i2 Learning staff, and were easy to use. Teachers also appreciated that check-ins with
i2 staff were available, though in the interviews, teachers commented that they were prepared to teach
and did not feel the need to reach out to i2 during Learning Week for instructional concerns. However,
they appreciated knowing assistance could be there when needed.
Interviews with teachers offer additional insights into the aspects of support that were most helpful to
teachers. In particular, teachers found the PD helpful because they were able to engage in the activities
from the student perspective and were able to see a finished product. One teacher explained, “Just the
experience of going through all of the projects was helpful because I knew what the kids should be doing
and what potential trouble they would go through.” Another teacher stated that the summer teaching
experience was crucial: “That made all the difference. Teaching this week, I needed that summer to
really know what it looked like. And that professional development, doing it was also big... It really lets
you see the level of organization you need to have in preparation and let you do some time
management yourself.”
D. Curriculum and Materials
Curriculum
A number of teachers commented on the curriculum itself, noting that the curriculum guides made
planning and preparation relatively smooth and easy. Teachers commented that the curriculum guide
was easy to read and understand, with tips for teachers, guidelines, and scripts where needed. This
sentiment was prevalent for teachers who do not normally teach science, particularly at the lower grade
levels. As one teacher noted, "Some of the science, I haven't taught that kind of science in a long time or
ever as a teacher. The lessons were very scripted so they were easy to read and know exactly how to
explain it to the kids."
However, teachers did encounter some challenges with instruction that could be supported by i2. In
survey responses teachers discussed issues with the curriculum, materials, and classroom management.
For example, with respect to the curriculum, teachers described needing additional support for some
students: “Differentiating different guiding questions for students who either didn't understand the
curriculum questions or for students who needed information re-phrased multiple times.”
20
i2 Learning – Boston City Package Evaluation Final Report
Teachers also provided additional information in the survey and interviews about ways in which they
needed to adapt or modify the curriculum to make it fit into their school and classroom. For example:
 “We added some pieces, including an emphasis on writing "Engineering Reports" and using chants
to reinforce the engineering design process. I'd be happy to share those resources!”
 “We cut out the non-essential activities because we were worried about running i2 all day long. I
would do the same thing in the future.”
Showcase
Additionally, schools varied with respect to their implementation of the showcase. The showcase, a
central component of the summer camp programming was intended to be a celebration of student work
during the week by providing students with an opportunity to present and display their work during the
week. Two model 1 schools included the showcase during the i2 Learning Week, and most model 2
schools did not include the showcase. Some schools that did include the showcase invited the entire
school to view student presentations. Teachers and students from these schools appreciated the
opportunity to display the work and accomplishments. For example, a student from one school thought
the showcase was a great way to engage other grade levels in engineering. According to one student,
“When we did the presentation the second graders [and] third graders came up to look at the projects
and out of all the people in the school, the second graders were really inspired.”
Though valuable, students and teachers from another school who implemented Learning Week i found
the showcase to be stressful and chaotic and took valuable time away from doing the activities.
Teachers at one school described needing the full fifth day of the week to plan and conduct the
showcase, which for some felt rushed. According to one student:
At the end of the week, teachers were focused on the showcase rather than making sure it was
fun for us. To make sure it was understandable and they didn't give us any time to rehearse. We
still had to follow the guidelines.
Time was a factor influencing the decisions of some schools that did not include the showcase. Two of
the teachers we spoke with at these schools discussed scheduling challenges, including the “time
crunch” at the end of the school year. Another school mentioned that a previously planned STREM night
would take the place of the showcase. Finally, it is interesting to note that across all teachers, the
showcase was not identified as a central component of the Learning Week experience in any of the
schools by students or teachers.
Materials
Although teachers appreciated the ways in which the activities were organized and the helpful tips, they
described some materials as particularly challenging. For example, teachers noted difficulty with faulty
equipment, such as flashlights and water pumps in the Vertical Farms unit, or materials that did not
function for the purpose of the model, such as the tape used for the model hand that kept popping off
in the Prosthetics unit.
Teachers also noted that students often had challenges with materials, though they acknowledged that
the material challenges were potentially beneficial opportunities for students to engage in problem
solving. For example, one teacher implementing the Prosthetics unit remarked, "I also think for some
kids, frustration with materials [was a challenge], ugh, this isn't working! That sort of stuff. They were
having a hard time following through with other solutions. We ended up talking a lot about this isn't
working for you, big picture, what would you change if you could?” Another teacher stated,
21
i2 Learning – Boston City Package Evaluation Final Report
They don't understand that failure is kind of a necessary step to any scientific advancement because,
you know, in school the expectation is you either get it right, and if you've done something bad you
fail, they don't quite have that understanding that failure is a good thing, persistence is something
that is part of scientific progress and innovation…
Furthermore, though teachers appreciated the materials i2 provided, the delivery, preparation of
materials for instruction, and management of materials during instruction was described by some as a
daunting task. Teachers reported having too many materials, which contributed to challenges related to
materials management. According to one teacher, “Just organizing it and figuring out which materials
went into each bin and why, that was a challenge, because we were also doing regular instruction in the
afternoon and doing reading at the same time.” Other teachers noted limited time for materials
management, both during the school year, and at the end of the year.
E. Structure for Implementation
Point person for logistics
All schools, to varying degrees, had a point person who took the lead on preparing for instruction,
notably establishing planning meetings with other teachers about a month prior to implementation,
managing materials, and ensuring they are prepared. In some cases, teachers needed to work with
administrators to make logistical decisions such as the use of school space and even the time of year for
implementation. In one school, a school administrator managed the logistics and worked with i2 to
ensure the school was prepared to teach, but one teacher was the “point person” who worked with the
other teachers to plan instruction, ensure that the instruction flowed smoothly, and that the schedule
was arranged in a way that made sense for teaching the curriculum.
More than 1 adult
Most schools implemented the unit in a traditional school format, with one adult in a classroom.
However, all teachers who implemented the unit alone in their classroom commented that this was
difficult, and additional adults would be helpful. Teachers who were able to team teach, or had an
assistant in the room, were appreciative of the extra adult assistance and believed this was an essential
support. According to one teacher, “It would be pretty essential to have two adults in the classroom at
least. More would be better. A big difference between the summer and the classroom was that there
were more adults and fewer students over the summer, and that felt like a pretty good ratio. During
school, there are many more students and fewer adults. The more adults the better, and I think you
really need two adults there.”
Classroom space
The space that teachers had for implementing activities emerged as another theme. Several teachers
noted that they did not have storage space for all of the materials for the number of students in their
classroom, that they needed additional outlets for activities requiring electricity or sinks for activities
requiring water. Teachers also commented that additional space was required to stage and prepare the
materials for student access during lessons. Our observations confirmed the variation in space
availability, and some teachers suggested that arranging alternative space, such as the cafeteria, would
have been helpful. One school was able to implement activities in a shared space I where students were
able to spread out, materials were easily accessible during lessons, and student artifacts were displayed
throughout the week. Several of the units also proved to be challenging for teachers in a traditional
school building. For example, teachers who taught the Engineering Ice Cream unit, noted the challenge
of inadequate freezer space for the ice cream.
22
i2 Learning – Boston City Package Evaluation Final Report
V.
Discussion and Conclusion
Overall, across schools, our data reveal that students and teachers found the i2 Learning Week activities
to be both engaging and interesting. Students reported an increase in their interest in engineering,
desire to learn engineering, and their understanding of the importance of engineering, and the
connection between engineering and society. On surveys and in focus groups, students reported science
and engineering is more interesting than they had thought before the program.
It is important to note that limitations in evaluation methods, specifically the convenience sample,
retrospective nature of the questionnaire, and lack of a comparison group limit our ability to make
causal claims. Future evaluation efforts would need to first determine the appropriate comparison
group, including understanding the vision of where i2 seeks to add value to the school. For example, in
order to determine an appropriate comparison and means of determining added value, it will be
important for i2 staff to determine whether the i2 Program seeks to expand already existing curricula
through infusing these new hands-on problem solving i2 Learning activities, or whether the pure model
of the i2 Learning week as a set of stand-alone activities could offer the most promise for engaging
students. Although the current evaluation was exploratory in nature, our data indicate several factors
related to the implementation of i2 Learning Week may contribute to students’ and teachers’
perspectives of the success of the program. The i2 Learning Week provided a hands-on, problem-based,
collaborative, and interdisciplinary opportunity for students across the Boston area. Students enjoyed
learning about and engaging with the materials and the engineering design process, and in many cases
enjoyed the opportunity for more science class than they would have received during a typical week of
school. In addition, teachers described the i2 Learning Week as a great way to engage all students,
including those who may be less engaged or interested in science during the rest of the school year.
Several factors also supported teachers’ implementation of the i2 Learning Week. Implementation in
most schools consisted of a week-long course where students spent anywhere from 2 hours to entire
school days on i2 Learning activities each day. Teachers had flexibility in their ability to modify the
lessons, modules, or their own schedules in order to successfully embed the activities into the school.
However, most teachers considered the integration of the i2 Learning Week within their curriculum to
be among their biggest challenge. Some chose to address this challenge by implementing it as a
separate set of activities at the end of the school year, while others chose to further modify the lessons
in order to teach it within the scope of their classroom and curriculum. Despite the variation in
implementation, teachers, and especially non-science teachers, felt comfortable and well supported to
teach the curriculum. The PD, especially the opportunity to engage with the module activities, was
beneficial for helping teachers to understand the materials and what students would need to do.
Each school also had a lead teacher who had taught summer school and participated in the PD, managed
logistics within the school, planned the activities within the school’s schedule, and collaborated with
other teachers within their school to plan lessons.
Although teachers and students alike perceived benefits to participating in i2 Learning Week, there were
some challenges to implementation, most related to logistics. For example, teachers needed additional
support in managing materials, although some were able to translate instances when students struggled
with materials into problem solving challenges. However, management of materials posed a challenge
for some, especially given limited classroom space for storage and set up. In addition, teachers stated
that additional adult presence was or could be helpful at times for managing students’ attention and
behavior especially during hands on activities. Teachers across schools and implementation models also
described time as a challenge, particularly as it related to materials, and to time for students to process
23
i2 Learning – Boston City Package Evaluation Final Report
what they were learning and to redesign and retest products. Finally, overall, the evaluation noted a
tension between the camp format of the week and “school science”; teachers were challenged to
consider ways of embedding the week long activities into their curriculum in a coherent fashion.
VI.
Recommendations
Based on the evaluation findings, the i2 Learning Boston City Package appears to be successful in
engaging students in STEM content and encouraging them to “invent & inspire, imagine & innovate,
investigate & inquire, and initiate & improvise” (See i2 Camp’s Mission at http://www.i2camp.org/
about.html). The recommendations below provide general suggestions to improve the teaching of i2
Learning Week, make it more easily adaptable to the school year, and increase the possibility for
successfully scaling up and building capacity as i2 develops plans for expanding i2 Learning Week in the
future.
1. Increase support for teachers to embed i2 Learning Week into their existing curriculum.
 Add explicit alignment of the units to the Next Generation Science Standards or state
content and performance standards.
 Provide time during PD and periodically throughout the school year for teachers to meet in
teams to consider how to embed the i2 Learning Week into their curricula.
 Provide support for ways in which teachers could embed elements of i2 activities into units
they already teach. Identify key content ideas and skills present within the i2 Learning Week
activities and offer teachers guidance and/or tips for how to integrate these content and
skills into curricula and teaching frameworks throughout the school year.
 Provide additional content support for those who do not teach science or engineering
throughout the school year.
 Increase guidance for ways in which teachers could increase the interdisciplinary nature of
the experience, including structured teaching teams.
2. Provide logistical and structural guidance to schools planning to implement the i2 Learning Week.
 Clarify the benefits and reasons that schools may choose to implement the i2 Learning
Week at different time points within the school year.
 Ensure all materials are sufficient for the activities and function well in the intended design;
offer troubleshooting tips to teachers if they or their students encounter challenges.
 Encourage schools to designate a point person for communication with i2 program staff and
coordination across teachers.
 Encourage teachers to collaborate with colleagues across disciplines.
 Schools could provide extra adult support (eg, teacher assistants) during instruction.
 Schools should consider possibilities for extra space, possibly including extra electrical
supply, refrigeration/freezer space, sinks, etc, and space for material set up and storage.
 Provide guidance and more explicit goals for the showcase, with explicit consideration of the
possible benefits and alignment of the showcase with intended skills, as well as guidance for
how to implement for those schools choosing to embed i2 Learning lessons within a more
typical week.
24
i2 Learning – Boston City Package Evaluation Final Report
Appendix A: Logic Model
25
i2 Learning – Boston City Package Evaluation Final Report
Appendix B: Characteristics of Student Survey Respondents
Table B-1. Number and percent of student survey responses, by school
School
# of Surveys
%
Brooke – Mattapan
57
16%
Brooke - East Boston
33
10%
Mother Caroline Academy
22
6%
Sacred Heart
30
9%
St. John
70
20%
Eliot
56
16%
UP Leonard
78
23%
Total
346
100%
Table B-2. Number and percent of student survey responses, by course
Course
# of Surveys
%
Contagion: Pandemic Response
152
44%
Engineering Ice Cream
35
10%
Prosthetic Devices
71
21%
Vertical Farming
50
14%
Kinetic Sculpture
38
11%
Total
346
100%
Table B-3. Number and percent of student survey responses, by grade level
Grade level
# of Surveys
%
4th
65
19%
5th
35
10%
6th
150
43%
7th
54
16%
8th
38
11%
No response
4
1%
Total
346
100%
Table B-4. Number and percent of student survey responses, by gender
Course
# of Surveys
%
Female
180
52%
Male
147
42%
No response
19
5%
Total
346
100%
26
i2 Learning – Boston City Package Evaluation Final Report
Appendix C: Teacher Survey
Dear i2 Educator,
Welcome! This survey is being conducted by researchers at the Education Development Center, Inc.
(EDC) as part of an effort to learn about teachers' and students' experiences during the i2 Learning
Week in their schools this year. The questions on this survey will ask about your participation in i2
professional development activities, your use of i2 courses with students, and your background. Your
completion of this survey will inform the development of the activities and professional development so
that teachers may be better supported in using the activities with their students.
We estimate that this survey should take approximately 10 minutes.
Your participation in this survey is voluntary. Individual answers to the survey questions will be kept in
strict confidence and will not be disclosed or released beyond the EDC project staff. If you have any
questions about this survey, feel free to contact Jackie DeLisi, Evaluation project director, at
[email protected].
Thank you for your time and input!
Please click the "I agree" button to indicate your understanding of the above information and consent to
participate in this research study.
Implementation
The questions in this section ask you about your experience with using i2 Learning course material.
1.
2.
3.
4.
5.
Which i2 course(s) did you use with your students?
Did you teach this course with other teachers?
Did you plan for this course with other teachers?
How many students were in your i2 course this week?
What was the grade level of the students in your i2 course?
The questions in this section address how you implemented the i2 Learning Week activities in your
school, and the successes and challenges you experienced during implementation.
6.
7.
8.
9.
How many days did you use i2 Learning activities?
What went well for you in implementing i2 Learning Week in your school?
What challenges did you encounter in implementing i2 Learning Week in your school?
What support, if any, did you receive from your principal, school, or district that was most
helpful for successfully implementing the i2 Learning Week with your students?
10. What support, if any, did you receive from your i2 Learning program staff that was most helpful
for successfully implementing the i2 Learning Week with your students?
27
i2 Learning – Boston City Package Evaluation Final Report
11. What additional support or resources would you need in order to improve the i2 Learning Week
experience for your students?
12. Would you run an i2 Learning Week again in the future?
Why or why not?
13. Is there anything else we should know about how you implemented this program in your
school?
About You
14.
15.
16.
17.
18.
19.
20.
21.
22.
23.
24.
25.
What is the name of your school?
For how many years have you taught as a school classroom teacher?
How comfortable are you leading science lessons with your students?
Please explain which aspects you are most and least comfortable with.
How comfortable are you leading engineering lessons with your students?
Please explain which aspects you are most and least comfortable with.
How many hours of professional development related to engineering (including i2 Learning PD)
have you attended?
What teaching certifications do you currently hold?
What are the grade level(s) of the students that you teach? (Select all that apply)
What subject(s) do you teach? (Select all that apply)
What is your highest degree earned?
What is your gender?
Thank you for taking our survey! Please press the "Done" button below to exit the page.
28
i2 Learning – Boston City Package Evaluation Final Report
Appendix D: Characteristics of teacher survey respondents
Table D-1. Number and percent of teacher survey responses, by school
School
# of Surveys
%
Brooke - Roslindale
2
14%
Eliot
4
29%
Match Community Day
3
21%
Mother Caroline Academy
2
14%
Sacred Heart
2
14%
UP Leonard
1
7%
Total
14
100%
Table D-2. Number and percent of teacher survey responses, by course
Course
# of Surveys
%
Contagion: Pandemic Response
3
21%
Engineering Ice Cream
4
29%
Kinetic Sculptures
2
14%
Prosthetic Devices
3
21%
Vertical Farming
2
14%
Total
14
100%
Table D-3. Number and percent of teacher survey responses, by highest degree
Highest degree
# of Surveys
%
Bachelors
5
36%
Masters
9
64%
Total
14
100%
Table D-4. Number and percent of teacher survey responses, by gender
Gender
# of Surveys
%
Female
12
86%
Male
2
14%
Total
14
100%
Table D-5 Number and percent of teacher survey responses, by grade taught
Grade
# of Surveys
%
Kindergarten
2
14%
Grades 1-3
3
21%
Grades 4-5
9
64%
Grades 6-8/Middle school
8
57%
Grades 9-12/High school
0
0%
Note: these numbers to not tally to 100% because respondents could select more than one
response option.
Table D-6. Number and percent of teacher survey responses, by subject taught
Subject
# of Surveys
%
Math
9
64%
Science
8
57%
English
7
50%
29
i2 Learning – Boston City Package Evaluation Final Report
Social Studies
6
43%
Other*
4
29%
*Other responses included: STEM; Technology; Special Education; and Writing.
Note: these numbers to not tally to 100% because respondents could select more than one
response option.
Table D-7. Number and percent of teacher survey responses, by years taught
Subject
# of Surveys
%
Zero
1
7%
1 to 3
3
21%
11 or more
3
21%
4 to 6
3
21%
7 to 10
4
29%
Total
14
100%
Table D-8. Number and percent of teacher survey responses, by hours of PD
Subject
# of Surveys
%
Zero
1
7%
1 to 10
7
50%
11 to 20
4
29%
more than 20
2
14%
Total
14
100%
30
i2 Learning – Boston City Package Evaluation Final Report
Appendix E: Teacher Interview Guide
Hello! Thanks so much for coming! My name is [Lisa Marco-Bujosa/Jackie DeLisi/Tracy McMahon]. I
work for the Education Development Center, which is an education non-profit in Waltham, MA. My
colleagues, [Jackie/Lisa/Tracy], and I are conducting an evaluation of the i2 Learning Week program in
Boston area schools. As part of this study, we are talking with teachers who implemented the Learning
Week to learn about how the program went for you and your students.
i2 Learning is specifically interested in learning from teachers about:




The supports and the challenges faced in implementing the i2 Learning week
The staff, materials, and resources necessary for successful activities
The plans and preparation necessary for successful program implementation
The level of engagement and motivation you observed in your students this week
Through this interview and other information collection efforts, we intend to document how the i2
Learning week was implemented in schools to better understand what worked and what did not work.
The results of this evaluation are intended to ultimately inform future decisions about program
requirements and supports.
I expect that our discussion will last about 30-45 minutes. I will be taking notes during our conversation
to ensure accuracy and we would like to audio-tape this conversation, with your permission. No
individuals will be identified by name. If you have any further questions that we may not be able to
answer about this evaluation or this conversation, please contact Jackie DeLisi, the director of this
evaluation.
1. Background
 What grade do you teach?
 What is your teaching experience? What is your science teaching experience?
 If you teach science, what does a typical science lesson look like in your classroom during
the regular year?
PROBE: What are you doing? What are students doing?
 Which i2 Learning course(s) did you teach this week?
2.
Student Activities
 What do you consider to be the most successful classroom experience from this week?
o What activities were most interesting and engaging to students?
o Why do you think that is?
 What were some changes you noticed in your students this week so far?
o Probe: any changes in interest, motivation, learning?
o What about the week might have contributed to any changes?
 What were some challenges your students faced this week?
o Why did they experience these challenges? What might have made it
different/better?
 What were some of the challenges you faced in implementing this week?
31
i2 Learning – Boston City Package Evaluation Final Report
3.
Planning/preparation
 How did you plan and prepare for this week?
 Did you work with other teachers? Attend i2 Learning PD? Other?
 What challenges have you faced in planning and preparing for instruction this week?
 How did you address these challenges?
 Do you have any suggestions to improve the planning and preparation for learning week?
4. Resources and support
 Did you have adequate resources and support to provide i2 experiences to your students?
o Support from other teachers? Principal? i2 Learning?
 To what extent did the PD prepare you to teach this week?
 Did you receive support from i2 during the school year?
o If so, what support did they provide? Was it helpful?
 What other resources and support would have been helpful to you?
5. Closing
 Would you run an i2 Learning week again?
o If so, what aspects would you keep the same? What would you change?
o What benefits do you see in implementing this across a week [or 2 weeks, depending on
the school]?
o Did you have a choice in the way the Learning week was implemented?
 Do you have any additional thoughts about your experience teaching the i2 Learning week that
we have not discussed?
32
i2 Learning – Boston City Package Evaluation Final Report
Appendix F: Student Focus Group Guide
Hello! Thanks so much for coming! My name is [Jackie/Lisa/Tracy]. I work for EDC, which is in Waltham,
MA. You may have seen me in your school a lot this week. We are conducting a research project on the i2
Learning week that your school was part of. As part of this study, we are talking with students to learn
about how the week went for you and what you think you learned. The results of our study will help
improve the experience for future students.
What we are doing today is called a focus group. Have you ever participated in a focus group? What this
means is that I will ask you a set of questions, and will ask each of you to respond. This will be much like
any other discussion—you should feel free to respond to what others say. There are no right or wrong
answers here, so if you agree or disagree with someone, or if your experience is different, that’s ok and I
would want to hear about that.
With that in mind, also please remember that everything you say here is confidential—your teachers are
not here and we will not tell them what any one of you said. I am not asking for any of your names, and
anything we report to the camp will be summarized across all of your comments. But please do not bring
what your classmates say here outside of this room.
We expect that our discussion will last about 30 minutes. To make sure I capture all of your thoughts I
would like to audio-record this conversation, with your permission. Is this ok with you?
Introduction
What i2 Learning course/activities did you participate in this week?
What have been some of the best parts of your i2 Learning Week? What was fun about it?
What was the hardest part of the week so far?
How does your experience this week compare to how science is normally taught in your school? What
was better? What was not as good?
Student learning
Do you think you have learned anything this week? If so, what? If not, why not?
Have you learned how to do anything?
What was it about the activities that helped you learn?
Do you think you learned more or less than you would have in your normal science classes? Why?
Interest in science
Did you think any differently about science and/or engineering after this experience? In what way?
What did you think before, and what do you think now?
Do you have any interest in studying science in high school, or after high school?
Do you think differently about studying science or engineering in the future, after this week?
What about the experience this week might have made you think differently about science or about
studying science in the future?
33
i2 Learning – Boston City Package Evaluation Final Report
Appendix G: Observation Protocol Guide
I2 Observation Guidelines
Date:
School:
Teacher:
Class/Observation Start time:
Class/Observation End time:
Number of students present:
Number of teachers present:
Instructions to observer:
 During class time, record field notes, with particular attention to each of the elements described
below.
 Every 5-10 minutes make note of the time and what students are doing.
 Record what you see and hear, and try to avoid inferences. For example, instead of recording
“students are disengaged”, write down the evidence of disengagement—what students are doing,
how many, etc, as well as some of the context around them (e.g., what should they be doing, what
are the teachers doing, etc.). We will be able to make inferences later.
 After observing, write a brief summary of the lesson that addresses the questions below.
Elements of instruction to make note of:
 What is the structure of activities?
o Who is teaching (eg. Team teaching? Individual teacher?)
o What are the lesson/activity goals?
o What does the teacher do?
o What are the students doing?
 What is the nature of the activities?
o Is it more student-directed or teacher-directed?
o What types of questions do teachers ask? (ISIOP, eg. fact or open-ended)
o How do students engage in the engineering design process? Are all of the students
engaged? What does their engagement look like?
o How is this different from typical school activities?
 Any indication of the contextual elements (classroom environment, number of teachers, set up of
the lesson, …) influencing….
o Student engagement
o Student interest
 Management
o Describe materials management/storage
o Describe any indications of the teachers’ organization and preparation for the lesson.
34
i2 Learning – Boston City Package Evaluation Final Report
Appendix H: Student Survey Responses by Gender and Grade Level
Table H-1. Engineering Interest and Attitudes Instrument Scale Results – Female Students
Scale
N
Pre to Post
Mean Change
Enjoyment/Interest in
Engineering
Importance of Engineering
Desire to Learn Engineering
Value of Engineering to
Society
160
5.82
5.49
3.57
163
164
172
Std.
Error
Mean
.39987
t
-14.552
df
159
p-value
<.001*
.30452
.32423
.34263
-18.031
-11.021
-8.518
162
163
171
<.001*
<.001*
<.001*
2.92
*statistically significant change from pre- to post-assessment
Figure H-1. i2 Learning Week: Attitude Scales Pre- and Post- Mean Scores - Female
Female
0
5
10
15
20
25
Enjoyment/Interest in Engineering
Importance of Engineering
Pre
Post
Desire to Learn Engineering
Value of Engineering to Society
Table H-2. Engineering Interest and Attitudes Instrument Scale Results – Male Students
Scale
N
Enjoyment/Interest in
Engineering
17
Pre to Post Mean
Change
4.65
Importance of Engineering
15
3.87
.97525
.97525
187
0.001*
Desire to Learn Engineering
17
3.29
.88186
.88186
195
0.002*
Value of Engineering to Society
16
1.69
1.68503
1.68503
195
.332
*statistically significant change from pre- to post-assessment
35
Std. Error
Mean
1.40362
t
1.40362
df
187
pvalue
0.004*
i2 Learning – Boston City Package Evaluation Final Report
Figure H-2. i2 Learning Week: Attitude Scales Pre- and Post- Mean Scores – Male
Male
0
5
10
15
20
25
Enjoyment/Interest in Engineering
Importance of Engineering
Pre
Post
Desire to Learn Engineering
Value of Engineering to Society
Table H-3. Engineering Interest and Attitudes Instrument Scale Results – Grade 4 Students
Scale
N
Pre to Post Mean
Change
Enjoyment/Interest in Engineering
59
-4.88136
.65501
-7.452
58
pvalue
<.001*
Importance of Engineering
55
-4.76364
.60911
-7.821
54
<.001*
Desire to Learn Engineering
53
-3.13208
.57464
-5.450
52
<.001*
Value of Engineering to Society
60
-2.18333
.80832
-2.701
59
.009*
Std. Error Mean
t
df
*statistically significant change from pre- to post-assessment
Table H-4. Engineering Interest and Attitudes Instrument Scale Results – Grade 5 Students
Scale
N
Pre to Post Mean
Change
Std. Error
Mean
Enjoyment/Interest in Engineering
32
-4.25000
.55721
-7.627
31
pvalue
<.001*
Importance of Engineering
31
-3.09677
1.11138
-2.786
30
.009*
Desire to Learn Engineering
30
-3.16667
.54474
-5.813
29
<.001*
Value of Engineering to Society
31
-2.64516
.59306
-4.460
30
<.001*
t
df
*statistically significant change from pre- to post-assessment
Table H-5. Engineering Interest and Attitudes Instrument Scale Results – Grade 6 Students
Scale
Enjoyment/Interest in Engineering
136
-5.72059
.40039
-14.287
135
pvalue
<.001*
Importance of Engineering
137
-5.51825
.33446
-16.499
136
<.001*
139
-3.87770
.35949
-10.787
138
<.001*
143
-3.17483
.33203
-9.562
142
<.001*
Desire to Learn Engineering
Value of Engineering to Society
N
Pre to Post Mean
Change
*statistically significant change from pre- to post-assessment
36
Std. Error
Mean
t
df
i2 Learning – Boston City Package Evaluation Final Report
Table H-6. Engineering Interest and Attitudes Instrument Scale Results – Grade 7 Students
Scale
N
Pre to Post Mean Change
Std. Error Mean
t
df
Enjoyment/Interest in Engineering
46
-3.73913
.71656
-5.218
45
p-value
<.001*
Importance of Engineering
47
-3.82979
.55861
-6.856
46
<.001*
Desire to Learn Engineering
51
-2.09804
.39133
-5.361
50
<.001*
Value of Engineering to Society
51
-2.47059
.42320
-5.838
50
<.001*
*statistically significant change from pre- to post-assessment
Table H-7. Engineering Interest and Attitudes Instrument Scale Results – Grade 8 Students
Scale
N
Pre to Post Mean Change
Std. Error Mean
t
df
Enjoyment/Interest in Engineering
34
-3.20588
.90825
-3.530
33
p-value
.001*
Importance of Engineering
36
-4.41667
.60733
-7.272
35
<.001*
Desire to Learn Engineering
36
-2.00000
.49119
-4.072
35
<.001*
Value of Engineering to Society
38
-2.92105
.39083
-7.474
37
<.001*
*statistically significant change from pre- to post-assessment
37
i2 Learning – Boston City Package Evaluation Final Report
Appendix I: Results by model
Table I-1. Change in means for “Gender issues around engineering” items – Model 1
Item
N
Mean
Change
Pre
Post
Boys are better at engineering than girls.^
208
1.72
1.60
-0.12
Girls are better at engineering than boys.^
207
2.23
2.23
0
Girls have a harder time understanding engineering
202
1.80
1.72
-0.08
than boys.^
Girls and boys are equally good at engineering.
206
4.11
4.29
0.18
Boys have a harder time understanding engineering
than girls.^
204
1.89
1.84
-0.05
Significance
(2-tailed)
0.070
0.943
0.173
<.001*
0.319
*statistically significant
^reverse-scored: desirable result is negative
Table I-2. Change in means for “Gender issues around engineering” items – Model 2
Item
N
Mean
Chan
ge
Pre
Post
Significance
(2-tailed)
Boys are better at engineering than girls. ^
132
2.27
1.89
-0.38
<.001*
Girls are better at engineering than boys. ^
Girls have a harder time understanding engineering than
boys.^
Girls and boys are equally good at engineering.
130
125
2.26
2.07
2.24
1.82
-0.02
-0.25
0.803
0.024
128
4.42
5.06
0.64
0.259
Boys have a harder time understanding engineering than
girls. ^
127
1.99
1.97
-0.02
0.827
*statistically significant
^reverse-scored: desirable result is negative
Table I-3. Change in means for “Gender issues around engineering” items – by Grade
Item
Boys are better at engineering than girls.
Girls are better at engineering than boys. ^
Girls have a harder time understanding engineering
than boys. ^
Girls and boys are equally good at engineering.
grade level
4th
5th
6th
7th
8th
4th
5th
6th
7th
8th
4th
5th
6th
7th
8th
4th
38
N
62
34
148
54
38
62
33
147
53
38
60
32
143
52
37
62
Pre
2.08
1.82
2.14
1.52
1.61
2.73
2.18
2.15
2.09
2.08
2.02
1.75
2.09
1.58
1.59
3.92
Post
2.03
1.65
1.75
1.50
1.42
2.73
2.27
2.10
2.15
2.05
1.88
1.78
1.87
1.50
1.49
4.47
Change
-0.05
-0.18
-0.39
-0.02
-0.18
0.00
0.09
-0.05
0.06
-0.03
-0.13
0.03
-0.22
-0.08
-0.11
0.55
Sig
.684
.245
<.001*
.868
.051
1.000
.540
.571
.705
.856
.381
.662
.025*
.420
.210
<.001*
i2 Learning – Boston City Package Evaluation Final Report
Boys have a harder time understanding
engineering than girls. ^
5th
6th
7th
8th
4th
5th
6th
7th
8th
33
146
52
38
62
33
143
52
38
4.45
4.06
3.87
4.29
2.15
2.03
1.90
1.77
1.74
4.58
4.86
4.08
4.39
1.92
2.03
1.97
1.60
1.74
0.12
0.80
0.21
0.11
-0.23
0.00
0.06
-0.17
0.00
.254
.024*
.004*
.160
.118
1.000
.477
.095
1.000
*statistically significant
^reverse-scored: desirable result is negative
Table I-4. Change in means for “Gender issues around engineering” items – Female respondents
Item
N
Mean
Change
Significance
(2-tailed)
Pre
Post
.056
Boys are better at engineering than girls. ^
177
1.53
1.40
-0.13
.303
Girls are better at engineering than boys. ^
175
2.66
2.58
-0.08
.017*
Girls have a harder time understanding engineering
170
1.66
1.51
-0.15
than boys. ^
<.001*
Girls and boys are equally good at engineering.
175
4.09
4.47
0.38
Boys have a harder time understanding engineering
than girls. ^
174
2.10
1.94
-0.16
.042
*statistically significant
^reverse-scored: desirable result is negative
Table I-5. Change in means for “Gender issues around engineering” items – Male respondents
Item
N
Mean
Change
Significance
(2-tailed)
Pre
Post
.004*
Boys are better at engineering than girls. ^
144
2.41
2.12
-0.29
.554
Girls are better at engineering than boys. ^
143
1.76
1.80
0.04
.178
Girls have a harder time understanding engineering
141
2.20
2.07
-0.13
than boys. ^
.067
Girls and boys are equally good at engineering.
141
4.09
4.76
0.67
Boys have a harder time understanding engineering
than girls. ^
140
1.69
1.84
0.15
.050
*statistically significant
^reverse-scored: desirable result is negative
39
i2 Learning – Boston City Package Evaluation Final Report
Table I-6. What went well for you in implementing i2 Learning Week in your school?
Model 1 (n=4)
Model 2 (n=10)
Materials Preparation
Materials Preparation
 Having the materials already waiting for us,
 We got everything on time. There was good
with enough for teacher demos and "do-overs"
materials.
for student errors.
 videos and building the prosthetics
 Planning was very easy on our end as the
 Having most of the materials, having all of the
materials were already prepared and the
handouts and the notebooks ready, having a clear
activities were written in great detail.
sense of the goals for the week.
 Program materials and teacher guide were well  The lesson plans were easy to adapt and use.
prepared.
Student Engagement
Student Engagement
 The scholars were very engaged with the
 High engagement, curiosity, and exploration
materials and activities.
 Great engaging activities for grade 4. This course
 Students were engaged and having fun with the
was perfect for the grade level! The final products
activities.
were fun and kids were very collaborative.
 The students really enjoyed it! They seemed to
internalize the engineering design process as well.
 The students loved having the chance to experiment
(and eat ice cream) every day.
Hands-on Work
 The course was so hands-on! My students loved how
learning about engineering was grounded in actual
engineering challenges.
 The best part of this i2 learning week was when the
students were able to make marble runs. Students
were very successful and creative during this time
and it was great to see them thinking about the
different types of energy that go into these runs.
Table I-7. What support, if any, did you receive from your i2 Learning program staff that was most helpful for
successfully implementing the i2 Learning Week with your students?
Model 1 (n=4)
Model 2 (n=10)
Materials
Materials
 Program staff made sure that all supplies were
 Communicating with us throughout the week
delivered early and were ready to use.
about what materials we needed - thanks!
 well organized materials
 Ordering supplies.
 they got us what we needed.
Communication
 Materials!
 Having Kristin and Phil stop by several times to
both observe and check in if we needed anything
Communication
or needed suggestions on how to approach a time  Check-ins were available
crunch with activities was great!
 Check ins with teachers and staff were available
 their presence
 Teacher's guide and very detailed instructions

Multiple conversations with Luisa, our contact
person
Training
 Training
40
i2 Learning – Boston City Package Evaluation Final Report




Trainings beforehand on curriculum
I was able to meet with Melissa Higgins before
teaching the course, to gain insight into how the
course aligned with the new Mass. draft science
standards.
The PD we took over the summer was great,
mostly because we got to actually create all the
prosthetics that the kids would be making as well.
Effective professional development prior to
teaching the course.
Evaluation
 Observations
 Jackie came to observe
 Jackie came in to observe/ work with us for a few
days.
Other
 None
41
i2 Learning – Boston City Package Evaluation Final Report
Table I-8. What challenges did you encounter in implementing i2 Learning Week in your school?
Model 1 (n=4)
Model 2 (n=10)
Curriculum
Curriculum
 Differentiating different guiding questions for
 dead URL links in the teacher manual
students who either didn't understand the
curriculum questions or for students who
Materials
needed information re-phrased multiple times
 The materials management was a challenge. Every
day there were different materials, and they weren't
Materials
always packaged in a way that made it easy to get
 It was especially challenging when the
them ready for class.
equipment provided did not work. (flashlights
 Not having enough supplies.
and water pumps)
 Some of the materials were difficult to work with.
Specifically, the gears!
Time
 LOTS of materials to manage
 It was hard devoting all my day time with the
 Materials management was tricky.
one class and putting my regular job as
 Materials prep; too many materials; would have
technology integration specialist on the back
liked more adults to manage all materials and set-up
burner.
 materials
 Trying to fit in as many activities as I could; the
 We were unable to make a marble lift. The materials
time limits suggested were too short; Vertical
were not made to allow the students to be
Schedule impacted other teacher schedules and
successful.
specials; needed to put together more formal
assessments
Logistics
 Thousands of tiny logistical problems! Plastic bags
Classroom Management
leaking, so salty water got mixed into the ice cream.
 It was hard to shift the focus from group work
Not having easy access to a sink to do all the dishes
to independent work when we needed to.
every day, and not having any dishwashing
detergent or sponges. Plugging in all four burners to
make applesauce and blowing the fuses to my
classroom. The video for one day suddenly
unavailable online. The stress of having to manage
when the groceries were delivered, storing them in
student lockers and the shared staff fridge, and then
sharing them fairly between three classrooms.
 we found it difficult to create the models for
students. Would have loved to have some models
made for us or pictures for us to go off of!
Classroom Management
 Behavior
 Because I'm not the students' normal classroom
teacher, behavior management was difficult at
times.
 Time management
42