Concord University Website Usability Evaluation, April 2010
Jamie Boggs
2
Table of Contents
I.
Introduction
II.
Procedure
III.
Participants
IV.
Results & Recommendations
A.
Terminology
B.
Task Observation
C.
SU Scores
V.
Conclusion
VI.
References
VII.
Appendix A: Test Introduction
VIII.
Appendix B: Background Questionnaire
IX.
Appendix C: Follow-up Survey
3
This evaluation tested the user interface of the Concord University website,
http://www.concord.edu, for its effectiveness and adherence to best practices and usability
standards. MyCU, a website subsystem that provides access to e-mail, courses, financial aid
awards, and other self-service functions, was also tested during this evaluation. It is the
objective of this evaluation to provide useful data that can be used by the University to improve
the overall user experience for all core audiences.
Procedure
The test environment was configured in a way that best represents the technology used by
the website’s users. A computer was configured with Windows XP, Internet Explorer 8, and a
resolution of 1024 pixels by 768 pixels. The computer was connected to the Internet via an
average speed broadband connection. This configuration was wholly based upon the
University’s Google Analytics data; see Table 2 for details.
Table 2. Statistical information from Google Analytics displaying the top three configurations for each
section (as of April 5th, 2010)
BROWSERS
VISITS
Internet Explorer, all versions
Internet Explorer 8
Internet Explorer 7
MozillaFirefox, all versions
Version 3.5+
Safari
OPERATING SYSTEMS
Windows
Windows XP
Windows Vista
Macintosh
Intel-based, version 10.5
64,906
33,124
29,041
14,608
10,719
6,928
VISITS
81,705
43,955
31,289
6,976
3,473
% OF TOTAL
VISITS
72.11
50.31
44.11
16.23
73.38
7.70
% OF TOTAL
VISITS 90.77
52.98
37.71
7.75
49.37
4
Intel-based, version 106
iPhone
SCREEN RESOLUTIONS
1024 x 768
1280 x 800
1280 x 1024
BANDWIDTH
T1
Cable
DSL
2,043
465
VISITS
24,075
21,072
10,640
VISITS
41,375
28,762
12,895
29.18
0.52
% OF TOTAL
VISITS 26.75
23.41
11.82
% OF TOTAL
VISITS 45.97
31.95
14.33
Participants were chosen at random from each of the core audiences: current students,
prospective students, faculty, staff, and alumni.
Each subject was initially asked to complete a 15-question background questionnaire that
queried them on their age group, language, computer and browsing skills, length of daily
computer and Internet use, and their experiences using websites of higher education. This
information allows the study to segment users based on these particular criteria so that future
improvements can easily be targeted to those users. This segmentation makes any demographic
issues apparent so that proper measures can be put in place to improve accessibility to those
particular audiences.
After completing the background questionnaire, the user was verbally presented with a
set of 11 terms and asked what information or content they would expect to find if the term
represented a link to a section on the website. Most of the set of terms were derived from actual
sections of content that currently exist on the website while other terms were added as possible
section additions that could be added to the website, if found by this evaluation to be useful.
5
The third part of the test involved the subject completing a set of tasks that were
generated based on common actions that particular subject would do on a regular basis. The
tasks were verbally presented to subjects while the evaluator monitored their activity and
recorded data. Completion time, errors made, amount of unsolicited comments (positive and
negative), and comments (subject’s and evaluator’s) or suggestions for improvements were all
collected during this process. There were a few tasks that were given to all participants for
comparative analysis.
The last part of the test involved the participant completing a survey based on the System
Usability Scale or SUS (Brooke). The SUS is a small 10-question survey that provides a single
score that represents the website’s overall usability. The survey was modified to replace all
instances of the word “system” with “website.” Additional questions were also added to provide
feedback on negative or positive aspects the subject may have after using the website during the
test.
Participants
This section details the subjects involved in this evaluation and the reason they
were chosen. A random subject was chosen from the 5 largest sets of users. Each set of users
utilize the website in their own unique way to accomplish tasks. Students check admission
requirements and deadlines, staff fill out online leave requests, alumni find and connect with
classmates, and faculty manage courses.
It was important that this evaluation evaluated the website in a manner that reflects the
6
institution’s strategic plan, which includes an initiative to better represent international cultures
on the campus by 2014 (“Concord University Strategic Plan 2009 – 2014”, 2010). Because of
this, this evaluation chose a random international student to play the role of the prospective
international student so that this evaluation’s goals could be met given the circumstances of
limited access of candidates to meet this criteria.
Table 3 lists the information obtained from the background questionnaire. In general,
analysis of this data show that most of the subjects are well experienced with using computers
and the Internet and should not have many problems when using a website that meets usability
standards. However, subject 1 speaks English as a secondary language and could take longer to
complete tasks due to processing information.
Table 3. Participant Profiles
Subject 1
(prospective
student)
Subject 2
(staff)
Subject 3
(alumni)
Subject 4
(current
student)
Subject 5
(faculty)
Primary
Language
Job/Profession
Age Group
Gender
General
Computer
Experience
(years)
Vietnamese
English
English
English
English
NA
18-28
M
4-6
IT
29-39
F
10+
Homemaker
18-28
F
10+
Student
18-28
M
4-6
Professor
40-50
F
10+
Experience
Browsing the
Web (years)
4-6
10+
10+
4-6
10+
5+
2-4
5+
5+
2-4
2-4
5+
2-4
8
8
9
8
Avg. Daily Hours 5+
on Computer
Avg. Daily Hours 2-4
Browsing Web
7
General
Computer Skill
Level
7
Skill Level
Browsing the
Web
Education
7
8
10
10
8
High School
4 Years of
College
4 Years of
College
2 Years of
College
Professional,
Ph.D.
Software
Training
None
None
None
Adobe
Software,
Basics of
HTML
Undergraduate
classes,
various
workshops
from
conferences
Results & Recommendations
This section details the results of this usability evaluation and provides recommendation
on improvement based on feedback and usability standards. Results and recommendations are
presented in the same order that subjects experienced during the tests.
Terminology
The first sets of results were gathered from the questions on terminology. Subject’s were
asked to describe what they think would be found in potential links or sections on the Concord
University website. The terminology questions were asked before the hands-on test so that a list
of expectations could be compiled that weren’t changed due to exposure to the actual published
sections. It is important to get an idea of the participant’s perception to what should be included
in a section as opposed to what is currently located there; this will make navigating the website
easier to learn since user perception of terminology and actual content are properly combined.
Table 4.1 shows a list of potential sections or links that were presented to each participant
8
along with their perceptions that have been distilled into a better representation of popular
feedback. 9 of the 11 terms are current links to content on the Concord website.
Table 4.1. Terminology and their distilled perceptions
TERMINOLOGY
Academics
DISTILLED PERCEPTIONS



Distance Learning
Student Life
Advancement
Research
Technology






















International Students

Current Students
Administration









Degree Programs, their entrance requirements and course requirements with
course schedules
Academic rules and regulations, academic catalog
Divisions and their offerings, faculty information
Classes available from a distance
Explanation of Distance Education, the format and the expectations of
students
Effectiveness of Distance Education
Steps to enroll
Activities
Cafeteria
Dorm Information
All student organizations
Services offered to students
Career Services/Job Placement
Graduate program information
Latest Alumni-related projects and highlights and the donors involved
University Relationships
Focus on faculty & student research, highlight faculty activities (4)
How to get involved with projects
Human Subject Review Board
Grant opportunities
Research and Development Corporation
Library and Interlibrary loan information
Help section for faculty/staff, section for students. Hours and contact
information for assistance
Services, and access to them
Technology offerings and incoming freshman requirements
List of countries represented on campus with photos, testimonials, interviews
and information on cultures
Requirements and Admission Steps
International Office, message and contact information
Student account information: grades, schedule, financial aid.
Student Life& Organizations
Technology and Distance Education
President’s information and message to visitors
Cabinet information with links to respective departments and their functions
Board of Governors
Organizational Structure
9
Admissions
About Concord








Contact Information
Deadlines
Admission Requirements
Easy to access application for admission and financial aid
Forms
Costs of Attendance
History
Environment and Location
Analysis of each subject’s perceptions of terminology mostly point to a proper use of
terminology on the current version of the Concord University website. One exception, as seen in
Table 4.1, seems to be the Advancement section.
Concord currently uses the Advancement section of the website to highlight alumni,
provide means for online giving and memberships, planning events, publications, and career
opportunities. One recurring perception by subjects during the evaluations pointed to the
inclusion of graduate program information. Three out of the five subjects perceived
Advancement to be a place for graduate information.
There are two options to resolve this problem. The first could be that Concord adds
graduate information under Advancement, however this solution isn’t recommended since the
functional area of Advancement really doesn’t have anything to do with graduate study. The
second and recommended option would be to have a main menu item for graduate programs.
For example, if a prospective graduate student looking for graduate entrance requirements
notices both the Advancement and Graduate links when scanning the main menu then they
would probably be inclined to click the link that best describes the information they are looking
for, in this case, the Graduate link.
Additional terminology such as, Distance Learning, Research, and Technology were
10
included in the set of questions as possible additions to the website. Research received a
tremendous amount of ideas for new content that could give website visitors insight into
activities of the Concord University faculty and students. This new section could also provide
new research and grant opportunities for the university and the community. Both Distance
Learning and Technology would provide new content that would be helpful to visitors but would
also provide easier access to information that already exists in other areas.
Feedback for ideas for new content within the International Students section was also
abundant. Many expected to see photos, testimonials, and countries represented to offer visitors
a welcoming image of international diversity. Currently, clicking International Students
immediately routes the user to a form requesting contact information that, upon completion,
redirects them to the international admission requirements. There is fine print to bypass the
requirements but, based on this evaluation’s observations, the visitor will navigate backwards
and find another way instead of proceeding forward. Based on the amount of feedback for ideas
on improving this section, it is recommended to enhance the international section by providing
more multimedia that reflects campus diversity with the form only accessed by clicking a button
within this section.
Task Observation
Each participant was asked to complete a set of 7-10 tasks that they would perform using
the website on a regular basis. Each subject was asked to complete the same first three tasks for
comparative analysis. The average completion time for the first task was 19.4 seconds. The
second task took an average of 48.6 seconds. The third task took an average of 111 seconds to
11
complete. See Table 4.2 for details.
Table 4.2. Results for Common Tasks
Common Task
Subject Completi
on Time
(seconds)
1. Use the website
to discover when
the University was
founded.
1
2
3
4
5
14s
23s
19.8s
25s
15s
2. Use the website
to find driving
directions to the
campus in Athens,
WV from
Charleston, WV
1
2
3
4
5
9s
11s
19s
159s
45s
3. Use the website
to find the
University’s
strategic plan
1
2
3
4
5
240s
16s
17s
217s
65s
Completion
Time
Standard
Deviation
4.8
63.4
109.4
# of
Errors
Made
# of
Hints
Given
# of
Unsolicited
Positive
Comments
# of
Unsolicited
Negative
Comments
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
3
1
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
4
0
0
6
1
1
0
0
1
0
0
0
0
0
0
0
0
0
1
0
Analyses of the task completion times for the same tasks show that task 1 was completed
successfully and quickly by all subjects. Task 2 was completed successfully by all subjects but
subject 4 and 5 had a difficult time. During the observation, subject 5 looked for the driving
directions under Visit CU which is under Prospective Students. Logically, this makes sense and
12
adding the driving directions under this section is recommended to improve accessibility.
Task 3 had a very large standard deviation. Both subject 1 and subject 4 had difficulties
completing this task, with 10 errors made between them. Interestingly enough, both subjects
represent students, either prospective or current. It is possible that the other subjects have
accessed this information before or have a better understanding of the functions of the
administration, and in turn, the Administration link on the website. Students also aren’t as much
concerned with the strategic plan and the administration as much as they are with the academic
programs, student life activities, and quality of education.
Table 4.3 shows the average completion time for all tasks as well as the overall amount
of errors encountered and hints given throughout the test in an effort to better understand the
values in this context.
Table 4.3. Overall Task Results
Subject 1
Avg. Completion 84.3s
(out of 10 tasks)
Time
# of Errors Made
# of Hints Given
Throughout
9
1
Subject 2
56.5s
(out of 7 tasks)
Subject 3
47.18s
(out of 10 tasks)
Subject 4
85.2s
(out of 10 tasks)
Subject 5
40.9s
(out of 8 tasks,
1 of 9 not
completed)
2
2
5
1
16
1
2
0
SU Scores
The last part of the website usability evaluation involved the subject completing the SUS.
Table 4.4 shows the average responses for each of the 10 questions and provides the individual
13
SU scores along with the adjective association presented by Bangor, Kortum, & Miller, 2009.
Table 4.4. Average Responses and Individual SUS scores
Question
# of Responses*
3
4
1
1
1
1
2
0
1
0
4
1
0
0
1
1
0
1
2
1
1
2
0
5. I found the various functions in this website were
well integrated.
1
0
0
2
2
6. I thought there was too much inconsistency in this
website.
3
2
0
0
0
7. I would imagine that most people would learn to
use this website very quickly.
0
1
0
1
3
8. I found the website very cumbersome to use.
2
2
1
0
0
9. I felt very confident using the website.
0
0
0
1
4
10. I needed to learn a lot of things before I could get
going with this website.
4
1
0
0
0
1. I think that I would like to use this website
frequently
2. I found the website unnecessarily complex
3. I thought the website was easy to use.
4. I think that I would need the support of a technical
person to be able to use this website.
5
2
*Ranges 1-5 (1=Strongly Disagree and 5=Strongly Agree)
Individual SU Scores
Score out of 100
Corresponding
Adjective
Subject 1
(prospective
student)
50
Subject 2
(staff)
Subject 3
(alumni)
Subject 5
(faculty)
Average.
40
Subject 4
(current
student)
52.5
50
45
47.5
OK
OK
POOR
OK
POOR
POOR
Analysis of the average responses from the SUS survey give an idea of what the website
is doing well and what aspects it needs to improve upon.
From the survey, the website obviously is an integral part of the regular functions of each
14
core audience. There is a strong feeling that the website will be used frequently to complete
tasks. The exception seems to be subject 1, the prospective international student, who may be
communication with the campus community via phone, e-mail, and person to person interaction
due to the language barrier. This is fine for current students but better effort needs to be made
for international student recruitment. Enhancing accessibility to the website’s translation feature
may be a good way to get international students to use the website as a tool to gather
information.
The complexity of the website is split, with half of the users finding the website to be too
complex and the other finding it not to be. The half that found the website to be complex, the
international prospective student and the faculty member, both have considerable computer and
browsing skills and have a lengthy past experience using these technologies. The faculty subject
frequently complained about the tediousness of accessing information within MyCU and, along
with 3 of the other subjects, felt that leaving MyCU to return to the Concord home page was too
difficult. Worth noting is the fact that 4 out of 5 subjects felt that they wouldn’t need the help of
a technical person to complete tasks. This indicates that as far as complexity is concerned, the
website lies somewhere between too complex to easily navigate but not complex enough to need
technical assistance.
Consistency in navigation and design are very important in keeping a user interface
simple and easy to learn. Most subjects rated the website well in this area and most felt that it
would be quick for others to learn to use the interface. It is the conclusion of this evaluation that
this area is well designed in its current state.
15
To measure user satisfaction, participants were asked to list the most negative and
positive aspects of the website. Table 4.5 lists the most common responses from subjects.
Table 4.5. User Satisfaction
Negative Responses







Positive Responses







Categories too specific
MyCU links vague and cluttered
MyCU too complex and not self explanatory
No web tour
PDF’s loaded in same window, no way back
home
Small search toolbar
Some links difficult to access
Effective search function
Content well organized
Clean design
Easy to use
Academics provide easy access to division
websites
Easy to find general school information
Good, informative pages for students
Based on the user satisfaction responses from Table 4.5 and the observations and
comments received during the task completion stage of the test, it is recommended that MyCU
should use consistent terminology with the other portions of the website so that it is easier to
navigate. It is also recommended that a method be put in place for users to easily return to the
University homepage. Many have claimed that the current method on the CU Links menu is
ineffective or too time consuming to use. The amount of clicks to get to information should also
be addressed by simplifying access. This is particularly important due to the fact that MyCU
isn’t normally accessed directly, but via the Concord University homepage. This offers yet
another layer between the user and their content.
16
Another observation during testing was the frustration users encountered when links to
PDF documents open in the same window rather than a new window. It is recommended that
PDF documents always open in a new window so that the window, and the Concord website,
aren’t closed by accident.
A final observation involves the use of microsites to provide a web presence to the
academic divisions of Concord (Nielson, 2009). Sometimes the microsites fail to use the same
terminology or branding and make traveling from the main website to the microsite a confusing
experience. The transition needs to be seamless so that users can feel confident that they know
how the interface works and how it should function.
Conclusion
In conclusion, the average of the SU scores show that the website has a borderline poor
usability rating that is the result of many factors pinpointed in this evaluation. Overall, users
seemed satisfied with the website and find it to be a useful tool to retrieve information. Users
claimed that the website had a clean design, was easy to use, and provided informative pages for
students. It is recommended that iterative future evaluations take place to ensure optimal
usability and to maintain a website that best reflects the needs of each of its users.
17
References
Bangor, A., Kortum, P., Miller, J. (2008). An Empirical Evaluation of the System Usability
Scale, International Journal of Human-Computer Interaction. Pages 574-594
doi:10.1080/10447310802205776
Bangor, A., Kortum, P., Miller, J. (2009). Determining What Individual SUS Scores Mean:
Adding an Adjective Rating Scale. pp.114-123.
Brooke, J. "SUS - A quick and dirty usability scale,"
http://www.usabilitynet.org/trump/documents/Suschapt.doc.
Concord University Strategic Plan 2009 – 2014. Retreived April 5, 2010. From
http://www.concord.edu/administration/presidents-office/strategic-plan-2009-2014
Nielson, J. (2009) Top 10 Information Architecture Mistakes. Retreived April 6, 2010. From
http://www.useit.com/alertbox/ia-mistakes.html
18
Appendix A: Test Introduction
Research Introduction for Concord University Website Usability Evaluation
Thank for you agreeing to participate in this study. It is this study’s goal to obtain useful
data that can be used to enhance the user experience when accessing the website of
Concord University.
The study will begin with a background questionnaire, in which you will complete on your
own. After completing the questionnaire, you will be ask for your input on online content
terminology.
Once done, I will ask you to complete a task using the Concord University website while I
monitor your actions. After you have completed the tasks you will place your hands in your
lap and say “task completed.” I will then query you on your experience completing the task.
We will repeat this procedure until all tasks are complete. Please be honest about your
feedback. It is helpful if you think aloud as you are processing tasks.
Lastly, I will ask that you complete a follow-up survey about your overall experience using
the Concord University website. Remember, this evaluation measures the effectiveness of
the website’s user interface; it is in no way measuring you or your abilities.
The test will take approximately 90 minutes to complete. You may ask to stop the test at
any time without penalty.
If you don’t have any questions, we will start the test now.
19
Appendix B: Background Questionnaire
CONCORD UNIVERSITY: WEBSITE USABILITY QUESTIONAIRRE
Please complete this form before beginning the usability evaluation. Your information will
remain confidential; your alias will be shown in place of your name.
Evaluation Alias: ______________________ Date:_____________
Please answer the questions below.
1. Please indicate your primary language:
2. Please indicate your Job/Profession:
3. Please indicate your age:
___ 18-28
___ 29-39
___ 40-50
___ 51-61
___ 62-72
___ 72+
4. Please indicate your gender:
____ MALE
____ FEMALE
5. Please indicate the length of your experience (in years) with using computers:
___ 1-3
___ 4-6
___ 7-9
___ 10+
20
Please continue to the next page…
21
6. Please indicate the length of your experience (in years) with browsing the
web:
___ 1-3
___ 4-6
___ 7-9
___ 10+
7. On average, how many hours do you spend on a computer on a daily basis?
___ 0-1
___ 2-4
___ 5+
8. On average, how many hours do you spend browsing the web on a daily basis?
___ 0-1
___ 2-4
___ 5+
9. Circle the number that best describes your skill level with using a computer, in
general:
1
2
3
4
5
6
7
8
9
10
(BEGINNER)
(EXPERIENCED)
(EXPERT)
10. Circle the number that best describes your skill level with browsing the
internet:
1
2
3
4
5
6
7
8
9
10
(BEGINNER)
(EXPERIENCED)
(EXPERT)
Please continue to the next page…
22
11. When is the last time that you have visited a website for a higher education
institution? (choose the most recent only)
_____ WITHIN THE PAST WEEK
_____ WITHIN THE PAST MONTH
_____ WITHIN THE PAST 6 MONTHS
_____ WITHIN THE PAST 2 YEARS
_____ NEVER
a. If applicable, what is the primary reason(s) for visiting the website?
(check all that apply)
____
UNDERGRADUATE ACADEMIC INFORMATION (classes available,
programs offered, etc.)
____
GRADUATE ACADEMIC INFORMATION (classes available, programs
offered, etc.)
____
RESEARCH
____
CAREER OPPORTUNITIES
____
CONNECT WITH ALUMNUS
____
CONNECT WITH COLLEAGUES (FORMER OR CURRENT)
____
LEARN ABOUT THE INSTITUTIONS HISTORY
____
CHECK FINANCIAL AID
____
REGISTER FOR CLASSES
____
PAY TUITION
____
APPLY FOR ADMISSION
____
OTHER, Please describe below:
12. Do you consider yourself to have any disabilities (vision or hearing
impairment, reduced motor skills, etc)? If so, please describe your disability
below:
13. Please indicate your highest level of education completed:
_____ HIGH SCHOOL OR GED_____ 2-YEARS OF COLLEGE
_____ 4-YEARS OF COLLEGE _____ MASTERS
23
_____ PROFESSIONAL
14. Have you ever been formally trained on any specific software applications?
NO:_______
YES:_______
If so, please describe the training with regards to format, software subject,
length of training, etc.
24
Appendix C: Follow-up Survey
Concord University Website Usability Evaluation: Follow-up survey
Please answer the following questions by marking the appropriate box for each question.
Make sure that your selection is clearly distinguished.
QUESTIONS
Strongly
disagree
1
1. I think that I would like to use this
website frequently.
2. I found the website unnecessarily
complex.
3. I thought the website was easy to use.
4. I think that I would need the support of
a technical person to be able to use this
website.
5. I found the various functions in this
website were well integrated.
6. I thought there was too much
inconsistency in this website.
7. I would imagine that most people would
learn to use this website very quickly.
8. I found the website very cumbersome to
use.
9. I felt very confident using the website.
10. I needed to learn a lot of things before I
could get going with this website.
PLEASE CONTINUE TO THE NEXT PAGE
Strongly
agree
2
3
4
5
25
List the most negative aspect(s) of the website:
List the most positive aspect(s) of the website: