Research Report:
New Assessment Booklet Layout Usability Study
Prepared for EQAO by
Rhona Shulman, MA
N OV E M B E R 2 0 1 4
Background
EQAO is in the midst of a multi-year project to move its provincial assessments online. The purpose of
this move is to bring the provincial assessments in line with the digital world and the digital classroom
and to make the assessments more engaging for students.
EQAO is committed to ensuring the move to online assessments does not in any way compromise the
technical quality of the provincial assessment program. The agency is conducting a variety of research
studies over the course of the transition period to ensure that its online assessments continue to have
best-in-class quality and offer valid, reliable evidence of student learning that can be compared across
the province and from year to year.
In addition to moving to online assessments, EQAO is moving to an online scoring model, which will be
implemented in 2015. To support the move to online scoring, the layout of EQAO’s paper-based
assessment booklets for the 2015 administration needed to be modified so that they could be more
easily scanned. To maximize scanning efficiency without compromising the student experience, EQAO
devised a plan to modify the layout of its assessment booklets in the following ways:
Assessment
Primarydivision
assessment
Juniordivision
assessment
New Booklet Layout for 2015
Previous Booklet Layout
Language
Separate reading selection booklet and
answer booklets
Combined question and answer
booklets
Math
Combined question and answer booklet
Combined question and answer
booklet
Language
Separate reading selection and answer
booklets
Math
Combined question and answer booklet
Combined question and answer
booklets with Student Answer Sheet
for multiple-choice responses
Combined question and answer
booklet with Student Answer Sheet
for multiple-choice responses
Combined question and answer
booklets, with Student Answer Sheet
for multiple-choice responses
Combined question and answer
booklets, with Student Answer Sheet
for multiple-choice responses
Grade 9 Assessment of
Mathematics
(academic and applied)
OSSLT
Separate question and answer booklets
Separate question and answer booklets
To determine the usability of the new layout of the booklets, EQAO undertook a pilot test with Ontario
students and teachers in the fall of 2014. This report highlights the findings from this study and EQAO’s
conclusion.
What did we do?
The usability study involved pilot testing the proposed new layout of booklets with Ontario students and
teachers. Questions on the pilot test were taken from the 2013–2014 assessments. Based on external
psychometric advice, EQAO selected 10 representative English- and French-language schools to
participate in the pilot test. Teachers were provided with instructions and guidance on how to
1
administer the assessment. Upon completion of the assessment, students and teachers were asked to
respond to feedback surveys—a student interview survey, a teacher interview survey and an
observation survey.
The surveys included both multiple-choice and
open-response questions. The responses were
reviewed, systematically coded and analyzed
for dominant themes. All responses were
tabulated and are summarized in this report.
EQAO representatives were present in the
schools to observe the study.
Students in Grades 3 and 6 in English-language
schools responded as a class group to survey
questions as teachers read them aloud.
Students in Grades 9 and 10 responded to the
surveys individually. All students in Frenchlanguage schools responded as a class group.
Who participated in the study?
Students and teachers from a selection of 10 representative English- and French-language schools
participated in the pilot testing. The table below shows a breakdown of participation by grade.
Grade 3
English
French
1
2
4
2
4
2
classes
2
classes
Teacher
Interview
Teacher
Observation
Student
Interview
Grade 6
English French
Grade 10
(OSSLT)
Grade 9
TOTAL
English
French
English
French
2
7
3
3
2
24
1
4
4
5
4
2
26
1
class
1
class
86
students
2
classes
50
students
1
class
9 classes,
136
students
What did we find?
1. The majority of students and teachers responded favourably to questions about the new
layout of the booklets.
Overall, did you like the answer booklet layout? [Students]
Grade 3 classes
All classes (2 English- and 2 French-language) responded “Yes”
Grade 6 classes
Both of the English- and French-language classes responded “Yes”
Grade 9
Academic
students/class
Grade 9 Applied
students/class
OSSLT
students/class
85% (44 of 52) of English-language students and the 1 French-language class
responded “Yes”
65% (22 of 34) of English-language students and the 1 French-language class
responded “Yes”
76% (38 of 50) of English-language students and the 1 French-language class
responded “Yes”
2
Overall, how would you say the answer booklet layout worked? [Teachers]
All grades
75% (18 of 24) of teachers responded “Very Well”
21% (5 of 24) of teachers responded “Satisfactorily”
Did the students appear to be comfortable with the layout of the answer booklets? [Teachers]
All grades
77% (20 of 26) of teachers responded “Yes”
12% (3 of 26) of teachers did not respond
12% (3 of 26) of teachers responded “No”
Positive comments about the new layout




Well-organized; easy and clear to follow; less flipping of pages
Good amount of space to do work and enough space for open-response answers
Easy to keep track of where they were and to check answers; helped them to be more organized
Layout of separate question and answer booklets was familiar to students from other assessments
Comments about difficulties with the new layout


Too many visual distractions on the page (e.g., banners, dotted edges and bar codes)
Challenges for some students in aligning their answers for multiple-choice questions (this difficulty
was also experienced on assessments in the previous layout)
Suggested solutions


Consider other ways to format multiple-choice bubbles
Remove as many unnecessary elements as possible from the page
2. Most students responded that there was no difficulty having a separate booklet in which to
write answers.
Did you have any difficulty with there being a separate booklet in which to write your answers?*
[Students]
3 of 4 classes responded “No Difficulty”
Grade 3 classes
* applied only to language component of the primary-division assessment
1 of 2 classes responded “No Difficulty”
Grade 6 classes
* applied only to the math component of the junior-division assessment
Grade 9
65% (34 of 52) of English-language students and the 1 French-language class
Academic
responded “No Difficulty”
students/class
Grade 9 Applied 59% (20 of 34) of English-language students and the 1 French-language class
students/class
responded “No Difficulty”
OSSLT
students/class
86% (43 of 50) of English-language students and the 1 French-language class
responded “No Difficulty”
3
Positive comments about the new layout


Like having the reading sections separated from the answer sections
Like the clustering of multiple-choice and open-response items in both the question and answer
booklets
Comments about difficulties with the new layout




Difficult to differentiate between the question and answer booklets
Difficult to identify and match different sections; headers not consistent and identifiers not clear;
titles not obvious enough (particularly for Grade 9 and OSSLT students)
Difficult to organize two booklets and other materials (e.g., calculators and rulers) on a small desk
Time-consuming to go back and forth between the two booklets (particularly for Grade 9 and OSSLT
students)
Suggested solutions



Make the question and answer booklets more easily identifiable and visibly different
Clarify booklet labels, section headers and titles
Consider desk space and arrangements when configuring rooms for the assessments; fold booklets
to maximize space; place booklets one under the other; include these ideas in the written
instructions to teachers
3. The majority of students and teachers indicated that the instructions for students were clear.
Did you find the instructions clear and easy to understand? [Students]
Grade 3 classes
Both French-language classes responded “Yes”
Both English-language classes suggested improvements
Grade 6 classes
Both the English- and French-language classes responded “Yes”
Grade 9
Academic
students/class
Grade 9 Applied
students/class
OSSLT students
71% (37 of 52) of English-language students and the 1 French-language class
responded “Yes”
71% (24 of 34) of English-language students and the 1 French-language class
responded “Yes”
88% (44 of 50) of English-language students responded “Yes”
The French-language class did not respond
How clear were the instructions for students? [Teachers]
All grades
75% (18 of 24) of teachers responded “Very Clear”
21% (5 of 24) of teachers responded “Needs Improvement”
Comments about difficulties with the new layout



Some students wrote answers in the question booklet
Some students in the primary grades didn’t stop after each reading selection to answer questions
Some instructions were said to be too long and repetitive
4
Suggested solutions




Clarify in the booklet design and instructions the importance of responding in the answer booklet
Instruct students to open both booklets at the beginning; explain why there are two booklets
Clarify that the reading booklet can remain open
Add a prompt after each reading selection to instruct students to stop and answer questions in the
answer booklet
4. The majority of teachers indicated that the instructions for teachers were clear.
How clear were the instructions for teachers?
All grades
83% (20 of 24) of teachers responded “Very Clear”
4% (1 of 24) of teachers did not respond
13% (3 of 24) of teachers responded “Needs Improvement” (these were Grade 9
teachers)
Positive comments about the new layout

Teachers were well prepared to administer the new layout
Comments about difficulties with the new layout

Teachers were not clear about whether to read the “key words” to the students
Suggested solutions

Clarify the instructions for the teachers
Summary and Conclusion
Overall, the new layout of the assessments was well received by both students and teachers. The
difficulties that some students and teachers identified were related to
 a lack of clarity about some instructions, headers and labels,
 practical difficulties going back and forth between two booklets,
 visual distractions on the page and
 inadequate desk space.
Based on these findings, EQAO is confident in moving forward with the new layout of its
assessment booklets. This change will facilitate scanning and online scoring without negatively
impacting the student experience. The agency will make the modifications necessary to address
the difficulties identified in this pilot test.
In addition, in 2015, a sample of students across Ontario will complete each EQAO assessment using
the old booklet layout so that EQAO can equate the results to account for any differences in student
achievement between the two layouts. This will ensure that achievement results can be reliably
compared across the two layouts and with previous provincial assessments.
5