BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
B.Y.O.D and its Effects on Standardized Test Scores
Jude Pascarelli
ED 501
Dr. Bonnie Rabe
1
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
Abstract: The purpose of this research is to determine what, if any effect that B.Y.O.D.
programs have on standardized test scores. It focuses on two Connecticut districts, New
Canaan and Bethel, and compares their scores with districts sharing similar
demographics. The result is that there is no significant effect on scores within the first
two years of implementing the technology programs. However, due to the value of their
ability to empower students and make them digital citizens, more districts should
consider the programs.
2
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
3
No User Left Behind!
With the increasing weight attached to students learning 21st century technology skills,
the high significance placed on standardized testing, an influx of English Language Learners
(ELL’s) and the inclusion of exceptional learners in the general education classroom, teachers are
becoming more and more like waiters at busy restaurants balancing all of these trays and trying
not to drop them all crashing to the floor. But with great challenges mankind responds with great
solutions.
There are a variety of recent strategies aimed at relieving some of these challenges.
Flipping the classroom, inquiry, e-learning and Web 2.0 tools are all gaining popularity as
teachers report on their effectiveness in terms of learning and student engagement. They all focus
on tiering students’ cognitive skills into higher levels of Bloom’s Taxonomy and instilling in
them the technological savvy they will need in their adult lives. Another similar technique is
B.Y.O.D., or Bring Your Own Device.
Bring Your Own Device, interchangeably referred to as Bring Your Own Technology
(B.Y.O.T.), is the name of the strategy that many businesses like IBM and Colgate implement to
allow employees to bring their own smart phones, tablets and laptops. The idea behind it is that
you know how to use your own device better than any other, and many users prefer their iPhones
over the Blackberry’s that companies typically issue to employees. The term B.Y.O.D. was first
used in a 2004 research paper by Ballagas et. al. Although the research focused on large public
display screens that passersby could interact with using their phones, the B.Y.O.D. acronym was
later generalized to other arenas public and private.
Around 2009, amid concerns of funding and after the observation of how adroitly
students used their phones and laptops to complete homework assignments, several public school
districts around the United States began pilot programs to test the feasibility of Bring Your Own
Device programs. Soon after positive reports of student achievement and engagement came out to
ameliorate teachers’ concerns over issues like security, keeping students on-task and cyber
bullying and more districts implemented BYOD programs. Now in 2013 there are dozens of
school districts across the country that are BYOD, including schools in Virginia, Ohio, New
Jersey, Texas, California, Florida and more.
To determine if an innovative technology program like Bring Your Own Device should
be considered, district administrators must decide if the benefits supplant the costs. In addition to
the crucial digital literacy skills that using their mobile devices brings pupils, it also increases
student collaboration time as students share with each other useful tools and shortcuts that they
encounter (“BYOD: One Year Later”, 2013). Scot A. Graden, superintendent of Saline Area
Schools in Michigan, found that with B.Y.O.D.:
“Students naturally teamed up and sought out students who didn't have their own
technology to bring to class-kind of a no classmate left behind scenario. Students
have really enjoyed the role of content producers versus passive consumers. We
considered offering a help desk for students so that classroom teachers weren't
burdened but there was no need” (“BYOD: One Year Later”, 2013).
Additionally, Bring Your Own Device bridges the communication gap, not only between
instructors and students but also between instructors and parents. Gmail and Edmodo provide
constant contact between students and teachers. Differentiated instruction and assessments
become much more organic and simple due to the multimodal and interactive nature of digital
tools (“B.Y.O.D.: One Year Later”, 2013). Perhaps most importantly of all, permitting the use of
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
4
digital devices in school can show students to love learning. According to J.D. Ferris-Rowe, the
Director of Faculty Development in the Brebeuf Jesuit Preparatory School in Indianapolis:
“Students are empowered to use their tools the way they want to support their
learning. When teachers allow students to drive technology, they are much more
focused on their content specialties than on teaching how to insert a picture in
PowerPoint. Collaborative work has increased as well” (“B.Y.O.D.: One Year
Later”, 2013).
Some of the drawbacks that administrators need to consider when creating a Bring
Your Own Device policy are lack of appropriate materials, partial teacher participation, and
placing too many restrictions on device usage. Teachers must now allocate e-textbooks, and it can
be a difficult of not impossible task to find quality e-books that also align with the new Common
Core Standards. Some staff members are also unwilling to or unsure of how to utilize the
student-owned technology and are leery of students making poor choices with them (“B.Y.O.D.:
One Year Later”, 2013).
Students need a certain degree of freedom to maximize the effectiveness of an open
technology program. Some of the fears that cause districts to over-moderate student activities are
the possibility of inappropriate content; distractions from games, texting and social media; and
digitally-enhanced cheating (Quillen, 2011). According to Ackerman and Krupp, the five
components to consider for B.Y.O.D. are:
1) Security – Both user information and devices need protection
2) Stakeholders need to be vested – Teachers, principals, students, parents and IT
workers must cooperate for optimal success.
3) Policies – Students and parents must sign Acceptable Use Policies to prevent
inappropriate activities.
4) Professional development – Teachers need to be familiar with techniques that use
the devices to promote inquiry-based, constructivist activities.
5) Financial plan for sustainability – Funding must be re-allocated from textbooks to
network infrastructure, devices and applications (Ackerman & Krupp, 2011).
School networks place blocks on inappropriate content, but the problem is compounded
when students bring in their cell phones and tablets. Most have access to 3G and 4G networks, so
rules restricting students to solely utilize the school’s wireless Internet network must be enforced
to ensure that classrooms are acting in accordance with the federal Children’s Internet Protection
Act (Quillen, 2011).
In addition to the promotion of digital citizenship, active engagement, and collaboration,
for a sweeping policy change like B.Y.O.D. to be considered effective it must also promote
student achievement. One of the best measures of achievement, and one that is less biased than
alphabetical grades, is content-based standardized testing. The purpose of this research is to shed
some light on the relationship between B.Y.O.D. and standardized assessment scores.
Number 2 pencil? Is that like, an app. or something?
Moving from the total cell phone bans of the ‘90’s to requiring students to bring in
devices is a monumental leap. It begs the question, does Bring Your Own Device take valuable
time away from teaching standards, or will higher levels of enthusiasm and understanding
increase content area-specific learning? In other words, will standardized scores go up or down?
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
5
My hypothesis was that scores would initially decrease slightly due to the implementation
curve but would in time increase as teachers discover what works and what doesn’t and students
gain more experience from lower grade levels on. The statistic I chose to test the hypothesis is the
percent proficiency of all the students assessed because districts view it as more vital to their
grading and goal setting than the mean score.
While many school systems across the nation have already put their B.Y.O.D. programs
into action, I chose to focus my research on Connecticut schools. The reason behind this was an
aim at preserving internal validity. The strongest correlating factor to a school district’s
proficiency rate is its socioeconomic status (SES). The Department of Education in Connecticut,
unlike many other states, groups districts into a hierarchy called District Reference Groups
(DRG) based on their SES. There are nine groups from A to I with A being the most
affluent/lowest need districts. This allows us to create control groups that ensure that we are
comparing apples to apples.
New Canaan, for instance, is a B.Y.O.D. district in southwestern Connecticut and also
happens to be one of the wealthiest towns in the nation. It wouldn’t be accurate to compare it to
Bridgeport, a city that ranks as an I on the DRG. Instead, we can use the scores of all the A
districts besides New Canaan as a control group (none of them have an open use policy).
The other district that champions students bringing their own devices is Bethel. Like New
Canaan, Bethel is a town located in Fairfield County in western Connecticut. Although it was
ranked as a B district in 1996, changes within the demographics resulted in its being classified as
a D district in 2006. As a result, the control group for Bethel is comprised of the 23 other districts
that share a D ranking on the DRG. Sampling all of the districts from each ranking for the control
increases external validity because it ensures that results can be generalized at least to other areas
of the country with similar SES.
The test I focused my attention on is the Connecticut Academic Performance Test, or
C.A.P.T. for short. It is administered to students during their sophomore year and they must
retake it if they do not pass. The content areas it assesses are mathematics, science, reading and
writing. Below are the data collected from the C.A.P.T. score database:
Mathematics
District
New
Canaan
Bethel
DRG
Year
#
Tested
A
2009
2010
2011
2012
2013
2009
2010
2011
2012
2013
312
332
349
333
309
244
234
229
237
219
D
#
Proficient
%
Proficient
Control
Total
Control
Proficient
Control
%
Proficient
Observed
- Control
301
326
343
327
302
215
208
209
207
188
96.5
98.2
98.3
98.2
97.7
88.1
88.9
91.3
87.3
85.8
1944
1908
1954
1973
1954
5313
5084
5008
4942
4870
1887
1859
1913
1919
1897
4644
4479
4441
4378
4297
97.07
97.43
97.90
97.26
97.08
87.41
88.10
88.68
88.59
88.23
-0.57
0.77
0.40
0.94
0.62
0.69
0.80
2.62
-1.29
-2.43
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
6
Science
District
New
Canaan
Bethel
DRG
Year
#
Tested
A
2009
2010
2011
2012
2013
2009
2010
2011
2012
2013
313
334
352
336
311
249
236
231
239
223
D
#
Proficient
%
Proficient
Control
Total
Control
Proficient
Control
%
Proficient
Observed
- Control
303
328
341
328
301
214
217
210
207
194
96.8
98.2
96.9
97.6
96.8
85.9
91.9
90.9
86.6
87.0
1957
1912
1965
1981
1959
6542
6223
6186
6105
5949
1886
1866
1914
1938
1913
5666
5595
5541
5465
5387
96.37
97.59
97.40
97.83
97.65
86.61
89.91
89.57
89.52
90.55
0.43
0.61
-0.50
-0.23
-0.85
-0.71
1.99
1.33
-2.92
-3.55
Reading
District
New
Canaan
Bethel
DRG
Year
#
Tested
A
2009
2010
2011
2012
2013
2009
2010
2011
2012
2013
310
332
350
334
309
244
232
229
237
219
D
#
Proficient
%
Proficient
Control
Total
Control
Proficient
Control
%
Proficient
Observed
- Control
308
329
345
329
300
226
219
213
214
195
99.4
99.1
98.6
98.5
97.1
92.6
94.4
93.0
90.3
89.0
1945
1909
1954
1977
1957
6452
6118
6066
6000
5802
1899
1870
1930
1930
1911
5718
5481
5409
5324
5206
97.63
97.96
98.77
97.62
97.65
88.62
89.59
89.17
88.73
89.73
1.77
1.14
-0.17
0.88
-0.55
3.98
4.81
3.83
1.57
-0.73
Writing
District
New
Canaan
Bethel
DRG
Year
#
Tested
A
2009
2010
2011
2012
2013
2009
2010
2011
2012
2013
312
334
352
335
311
247
235
232
239
223
D
#
Proficient
%
Proficient
Control
Total
Control
Proficient
Control
%
Proficient
Observed
- Control
309
331
349
330
306
234
227
223
234
210
99.0
99.1
99.1
98.5
98.4
94.7
96.6
96.1
97.9
94.2
1952
1915
1967
1986
1962
6539
6178
6152
6088
5926
1922
1889
1946
1966
1938
5988
5618
5724
5702
5511
98.46
98.64
98.93
98.99
98.78
91.57
90.94
93.04
93.66
93.00
0.54
0.46
0.17
-0.49
-0.38
3.13
5.66
3.06
4.24
1.20
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
7
How much better or worse a district performed in comparison to its control is listed in the
far right column of each chart. If the district performed better then the cell is highlighted green;
red if it performed less well. However, not all years listed are relevant for computing a statistical
analysis. Both towns weren’t B.Y.O.D. district-wide until a few months before the 2012 testing,
so only the 2012 and 2013 scores can tell us if there is a correlation between 1:1 student devices
and test scores. However, including the proficiency numbers for the three years before can help
us analyze the statistics and further increase internal validity. Does a certain district consistently
out-perform other districts with similar socioeconomic status, even before it implemented a
B.Y.O.D. program? Including the years before can illuminate that.
The following line charts offer a more clear visual representation of the data. The line
represents when BYOD was instituted.
Bethel
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
8
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
New Canaan
Mathematics
% Proficiency
98.5
98
97.5
New Canaan
97
Control
96.5
96
95.5
2009 2010 2011 2012 2013
Science
98.5
% Proficiency
98.0
97.5
97.0
New Canaan
96.5
Control
96.0
95.5
95.0
2009
2010
2011
2012
2013
9
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
10
% Proficiency
Reading
100.0
99.5
99.0
98.5
98.0
97.5
97.0
96.5
96.0
95.5
New Canaan
Control
2009
2010
2011
2012
2013
Writing
% Proficiency
99.2
99.0
98.8
New Canaan
98.6
Control
98.4
98.2
98.0
2009
2010
2011
2012
2013
Analysis
As evidenced by the line graphs, New Canaan students scored very closely to
their control peers for all five years studied. Bethel, on the other hand, scored fairly
closely to the control group for reading and writing but something interesting happens in
2012, the same year Bring Your Own Device was implemented. Math and Science scores
seemed to take a sudden downturn in contrast to the scores of comparable non-BYOD
regions. Is this enough to suggest causation?
The statistical analysis most appropriate for this data is a two proportion Z-test. A
t-test may also be used, but because of the large sample size (there were always more
than 200 students tested) a Z-test is more fitting. Using a 95% confidence level there
were only two areas where critical values were discovered. In 2012, Bethel’s writing
scores yield a p-value of less than 0.4%, which indicates that there is less than a 1%
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
11
chance that the difference between Bethel’s writing scores and those of the control group
occurred by chance. In 2013, Bethel’s science scores yield a p-value of about 3.8%,
indicating about a 4% chance that the difference between Bethel’s percent proficiency
and that of the control group occurred by chance.
Conclusions
Although some significant differences were found, there is no clear indication that
they were caused by B.Y.O.D. If say, blogging in Bethel enhanced the writing skills of
Bethel students, why were scores only critically higher in 2012 and not 2013? Similarly,
why would science scores be lower exclusively in 2013? And why were there no
significant differences in New Canaan scores?
In summary, there is no evidence to accept the hypothesis that standardized test
scores are decreased when Bring Your Own Device is introduced. However, a
longitudinal study would be beneficial to observe how scores change over time. If there is
in fact an implementation curve, then scores will likely increase over time. Some
additional information that we need to obtain is student and teacher attitudes towards the
programs as well as surveys on digital literacy skills improvement.
So what does this mean for school systems that are considering open use policies?
Being that there aren’t any considerably increased costs for districts over time,
achievement scores aren’t lowered, and because 21st century skills are developed and
students are empowered, more schools should absolutely jump on the BYOD bandwagon.
The most important reason to jump on, I think, is because we as educators like children
and they genuinely enjoy using their devices.
Sources
Ackerman, A. S., & Krupp, M. L. (2012). Five Components to Consider for
BYOT/BYOD. International Association For Development Of The Information
Society
Ballagas, R., Rohs, M., Sheridan, J.G., Borchers, J. (2004). BYOD: Bring your own
device. In Proceedings of the Workshop on Ubiquitous Display Environments,
Ubicomp. Retrieved from:
http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.58.9939
BYOD: One Year Later (2013). Technology & Learning, 33(7), 36-39
Canny, Priscilla (2006). District Reference Groups (DRGs) Formerly Educational Reference
Groups (ERGs). Connecticut Voices for Children. Retrieved from:
http://www.ctvoices.org/publications/district-reference-groups-drgs-formerlyeducational-reference-groups-ergs
Quillen, I. (2011). Crafting Your BYOT Policy. Digital Directions, 22-23
BYOD AND ITS EFFECTS ON STANDARDIZED TEST SCORES
Schools Plug Into BYOD: Mobile Devices Transform Learning at Katy ISD. Cisco
Newsletter. Retrieved from:
http://www.cisco.com/web/strategy/docs/gov/ciscoedukatysdc_cs.pdf
12