2009
Grumpy
James Ma
[email protected]
YuChor Lam
[email protected]
Peter Park
[email protected]
Adenike Adebayo
[email protected]
Andrew Duong
[email protected]
TA: David Dearman
[
RESULTS OF EVALUATIONS &
EVALUATIONS OF RESULTS
Assignment G4 of CSC318H1S2009
]
1
Table of Contents
Description of the Evaluation Techniques, Tasks and Participants .................................... 2
1. Think-aloud ................................................................................................................. 2
2. Cognitive Walkthough ................................................................................................. 4
3. Heuristic Evaluation .................................................................................................... 6
Rationale ........................................................................................................................... 12
Results of the Study .......................................................................................................... 13
Think Aloud ................................................................................................................... 13
Cognitive Walkthroughs................................................................................................ 13
Heuristic Evaluation ...................................................................................................... 14
Discussion and Analysis .................................................................................................... 16
General Implications ......................................................................................................... 18
Potential Improvements ................................................................................................... 19
Very High Priority Changes ........................................................................................... 19
High Priority Changes .................................................................................................... 20
Medium - Low Priority Changes.................................................................................... 21
Critiques of the Evaluation Plan........................................................................................ 22
Appendix ........................................................................................................................... 23
Who Did What............................................................................................................... 23
2
Evaluation of CLAS
Description of the Evaluation Techniques, Tasks and
Participants
1. Think-aloud
Introduction:
Think aloud involve participants thinking aloud as they are performing a set of
specified tasks. Users are asked to say whatever they are looking at, thinking, doing, and
feeling, as they go about their task. This enables observers to see first-hand the process
of task completion (rather than only its final product). Observers at such a test are asked
to objectively take notes of everything that user’s say, without attempting to interpret
their actions and words. Test sessions are often audio and video taped so that observers
can go back and refer to what participants did, and how they reacted.
Purpose:
The purpose of the think-aloud is to identify unforeseen usability failures in the
design of the UI. Using the think-aloud, we will be able to identify areas in the software
where the UI may be ambiguous or unclear to the average user.
Participants:
Subjects can be either male or female, but must lie within the age range of 18 –
35 years old. Subjects do not have to have prior domain knowledge. Subjects do not
have to be HCI experts. Repeat this experiment for 8 – 10 people.
Task:
Ask the subject to perform the following task: “Please search for a blue dress shirt from
Harry Rosen”
3
Evaluation Procedure:
1. The following think-aloud evaluation requires two evaluators. The first evaluator
will act as the observer; he/she will take notes on the subject as they execute the
task. The second evaluator will act as the main speaker and prototype driver. This
evaluator will be in charge of “changing” the screens on the prototype as the user
“clicks” on buttons.
2. Prepare a recording device (cassette recorder, mp3 recorder, laptop recorder,) that
can record audio input from a microphone.
3. Slide in the page labeled “Main Menu” into the page receptacle of the physical
mockup.
4.
Present the physical mockup to the subject.
5. The main speaker shall notify the subject of the following: “This is a think-aloud
evaluation of the CLAS, Clothing Advice Service, and mobile application. The CLAS
mobile application is a program that is designed to help enhance the everyday
shopping experience. Using CLAS, you will be able to search for and navigate to
stores, look up real-time store stock, and find sales within the store. For the
purposes of this evaluation, we are asking you to perform a task using the CLAS
mobile application. You will be required to speak out your actions and your current
thought process while executing the task”
6. The main speaker shall explain the task to the subject: “Please search for a blue
dress shirt from Harry Rosen”
7.
The observer should now begin recording the subject
8. As the user “clicks” and “selects” buttons on the mockup, the main speaker is now
in charge of switching in pages from the functional prototype that lead to the
corresponding page of the “clicked” button.
4
9. Meanwhile, the observer should observe the subject as he/she navigates through
the menus. Using paper and pencil make short form notes about how the user
progresses through the application. Make notes on tasks that the subject
completed quickly, and tasks that the subject took longer to complete than
expected. Use the short form “e.” for easy completion of task, and the short form
“h.” for tasks that were more difficult for the user. Take a quick note of which
screen the subject was on when he/she encountered the difficulty, and what task
the subject was trying to perform.
10. a. If the subject finishes the task, skip to step 11.
b. If the subject becomes frustrated or stuck, offer some hints on how to proceed.
If they are still stuck and cannot precede any further, stop the experiment. Begin
discussing where and why the user got stuck, and what he/she was performing
when he/she got stuck.
11. End the recording.
12. End of evaluation.
2. Cognitive Walkthough
Introduction:
A cognitive walkthrough starts with a task analysis that specifies the sequence of
steps or actions required by a user to accomplish a task, and the system responses to
those actions. The designers and developers of the software then walkthrough the steps
as a group, asking themselves a set of questions at each step. Data is gathered during
the walkthrough, and afterwards a report of potential issues is compiled. Finally the
software is redesigned to address the issues identified.
5
Purpose:
The Cognitive walkthrough method is a usability inspection method used to
identify usability issues in a piece of software, focusing on how easy it is for new users
to accomplish tasks with the system. The method is rooted in the notion that users
typically prefer to learn a system by using it to accomplish tasks, rather than, for
example, studying a manual.
Users:
Select a team of 4 – 5 HCI experts from the CSC318 class. If not enough people
are available, select experts from the U of T DCS who have taken CSC318 in previous
years.
Identify Users:
1. The user is already familiar with Smart phone and other GPS-aware and
internet enabled devices.
2. The user is looking for a way to quickly find objects/clothing items of interest
3. The user is using this application “on-the-go”, so the responsiveness (load
times) and clarity of the UI will be important.
Prototype:
The prototype was made with the intention of allowing the user to simulate
using this application as they would with a real mobile device. The physical prototype
itself is not a definition of our software, CLAS. The physical form prototype is simply a
box made to the dimensions of a mobile device (in this case an iPhone) with proper
pictures on each side. This prototype is intended to be used with the functional
prototype to simulate how the end product would work. The users will be able to hold it
in their hands and use the touch screen/keyboard as they would with a real iPhone to
interact with the functional prototype. As they perform their intended actions, a group
member will change the pictures from the functional prototype accordingly.
6
Task:
Chloe is looking for a new green sweater. She has already entered some of her
favorite stores (H&M, Lululemon, Coach, etc.) and is looking for a store that has
something she might like.
Interaction Action Sequence:
Assume that CLAS has been loaded
1. Select Finder
2. Enter “Green Sweater” into the search bar
3. Select the sweater of interest
4. Looking at the item information, select the store name
5. From the listed option, select option “1. Find on Map”
Believability Story:
1. Will the user be trying to produce whatever effect the action has?
2. Will the user be able to notice that the correct action is available?
3. Once the user finds the correct action at the interface, will she know that it is the
right one for the effect she is trying to produce?
4. After the action is taken, will the user understand the feedback given?
3. Heuristic Evaluation
Introduction:
Heuristic evaluation is a discount usability engineering method for quick, cheap,
and easy evaluation of a user interface design. It specifically involves evaluators
examining the interface and judging its compliance with recognized usability principles
(the "heuristics").
7
Purpose:
UIs are often designed in a short space of time on a budget that may restrict the
amount of money available to provide for other types of interface testing. Heuristic
evaluations are one of the most informal methods of usability inspection in the field of
human-computer interaction. The goal of heuristic evaluation is to find the usability
problems in the design so that they can be attended to as part of an iterative design
process.
Used Heuristics:
1. Aesthetics and Minimal Design
Description:
Dialogs should not contain information that is irrelevant or rarely needed.
Menu items should not be cluttered and the text should be readable (within the
context of a smart-phone)
Reason of its significance:
The product must be visually appealing while having the simplest possible
design. This prevents users from getting confused on what is what or becoming
overloaded with features or dialogs. The idea is also coined from the phrase “less is
more” in saying that, by providing less competing features and dialogs, more can be
achieved quicker and faster. We wanted to test to make sure that our product does
satisfy this need in providing the end users with a simple enough interface to
minimize any possible confusion and allow them to proceed with their intended
actions.
2.
User Control and Freedom
Description:
8
Users often choose system functions by mistake and will need a clearly
marked "emergency exit" to leave the unwanted state without having to go
through an extended dialogue. Support undo and redo
Reason of its significance:
The program is designed to work across various mobile platforms and devices,
so there should be providing features like undo, redo, or back allows them to go
through the process more smoothly without having to go through heavy dialog
again just to change one thing. When such options are not provided it would likely
lead to frustration for the user or cause incorrect outcomes.
3.
Recognition Rather Than Recall
Description:
Minimize the user's memory load by making objects, actions, and options
visible. The user should not have to remember information from one part of the
dialogue to another. Instructions for use of the system should be visible or easily
retrievable whenever appropriate.
Reason of its significance:
This is a mobile application that is meant to be used “on-the-go”, so we need
to focus on the program's ease of use.Also, the instructions should be easily laid out
with visual aid to make it accessible. Again, this allows the users to go through the
menus and options much more quickly and with ease.
4.
Flexibility and Efficiency of Use
Description:
9
Accelerators -- unseen by the novice user -- may often speed up the
interaction for the expert user such that the system can cater to both
inexperienced and experienced users. Allow users to tailor frequent actions.
Reason of its significance:
While the product should be usable by everyone, when used by an expert
(anyone who has used the product before), there should be accelerators that would
allow them to speed up the interactions. Therefore, the frequently used actions can
be tailored towards the user specifically, once again, allowing them to access it in a
quicker manner.
5.
Match between System and Real World
Description:
The system should speak the users' language, with words, phrases and
concepts familiar to the user, rather than system-oriented terms. Follow real-world
conventions, making information appear in a natural and logical order.
Reason of its significance:
Programmers often find themselves naming features or items that are familiar
to them. These terms however may not be familiar to the users. Every item should
be in terms which the users can understand and use concepts familiar to the users.
The program should follow conventions that are frequently used in the real world
and allowing things to appear in a natural and logical order. This allows the users to
logically know where things should be found rather than having to go through all
the possible options just to access a certain feature.
10
Severity Rating:
Use the following rules for assessing the severity rating:
Based on:
• Frequency
• Impact
• Persistence
• Market impact
Severity Rating Scale
0 = Not a usability problem
1 = Cosmetic problem (fix if extra time)
2 = Minor usability problem (low priority to fix)
3 = Major usability problem (high priority to fix)
4 = Usability catastrophe (must be fixed before release)
Evaluators:
The evaluators consisted of three classmates of the course CSC318H1S in spring
of 2009 at University of Toronto. They are ideal evaluators for this project as they have
proficient knowledge in the subject, and thus deemed to be experts of HCI in this
specific research. The evaluators consist of 2 male and 1 female participants whom also
fit into the ideally targeted end user criteria of between ages of 18~35. To maintain
anonymity, they will be referred to as participants A, B and C throughout the usability
findings section below.
11
Task:
1. Finding Sweater on Store Search
The evaluators were asked to find sweater sold by Holt Renfrew using the
application (CLAS). Under the ideal circumstance, this process can be completed by
going to Finder from the main screen and typing “Holt Renfrew, Sweater” (in any order,
with or without punctuation and capitalization), then select one of the items on the list.
By doing so, the application will recognize the store’s name and search for sweater
under that store specific search. Alternatively, this process can also be completed by
typing “Holt Renfrew” first, select “Holt Renfew” from the store search result, select
Find Clothing, then search for “Sweater” under their store specific search. This task goes
through one of the critical part of the application which also may contain numerous
steps.
2. Changing the Settings
There were two main features in settings to be explored.
First, much more simple setting change request of turning the sound on/off was
asked of the evaluators. The process involves in the following steps: Click on Settings
from the main screen, click on Sound, then, once again Sound (ON), followed by the last
step of selecting the off on the pop-up window. Only this task was chosen to be tested
for out of many other similar functions since they should share the same usability
findings as they have nearly identical interface.
The second settings-modification setting would be to add their favourite brand or
store to the favourites list in to the application. This process can be completed by once
again, going into Settings, Set Favourites, type in the name, then enter. Once this task
has been completed, they were also asked to remove the one they just added. This can
be accomplished by simply pressing the “X” button on the same screen.
12
Rationale
The three evaluation techniques used are think-aloud, heuristic evaluations, and
cognitive walkthroughs. Think-aloud was chosen as a quick way to get insight on the
user thought process and sources of confusion as they perform tasks on the system.
Heuristic evaluations were chosen to acquire criticism in areas where we felt were most
important (by choosing heuristics accordingly). Lastly, the cognitive walkthrough was
done to get feedback on the flow of the system and to root out problems in the actions
and feedback of the system.
In all three of the evaluations techniques used, the participants were asked to
perform a search for a particular piece of clothing from a named store. More specifically
they were asked to search for a blue dress shirt from Harry Rosen for the think aloud
evaluations and search for a green sweater for the cognitive walkthrough. The
application was created with the sole purpose of saving time during the shopping
process for its users. This type of task was chosen since it is a strong representation of
this goal.
The materials provided and used for the evaluations are the physical and
functional prototypes (the physical mock up of the iPhone along with a set of paper
slides depicting the user interface). These were used to simulate the effect of actually
using a mobile device with the application. No other materials were provided, but a
general description of the system was given to the evaluators. The application was
intended to be simple and easy to use, and providing a large amount of documentation
would have gone against this.
For the heuristic evaluations, the evaluators were also asked to perform actions
pertaining to the other screens. They were asked to test the settings portion of the
application. This was done to get feedback on the layout and familiarity of the settings
screens. To do this, the same set of heuristics for the search task were used, but greater
13
emphasis was put on the heuristics ‘recognition rather than recall’ and ‘match between
system and real world’.
Results of the Study
Think Aloud
Five of the six subjects were able to select the proper menu item, “Finder”, from
the main menu. Subject #3 appeared to be clicking on every button in the menu, but
was quickly able to return to the main menu to select Finder.
Only two out of the six subjects directly searched for a variation of “blue shirt” in
the search bar once they reached the Finder page. The other four subjects selected
Harry Rosen directly, as it was listed as a result under the “Previous Searches” listed
(which is shown by default when the user goes into the Finder menu).
Of the two users who searched for the shirts, both had trouble identifying what
store a particular item belonged to. Subject #1 eventually figured it out when the
subject said: “Oh I see, the stores are in blue”.
Of the four users who selected the Harry Rosen store directly from the Finder
page, all four eventually used the Find Clothing option in the Store Information page to
proceed through the search.
Subject #3 mis-clicked on Find Clothing in the Store Information page and
selected Find on Map. This took the user to the Maps page, but there is no button from
there to proceed “back” in the UI.
Cognitive Walkthroughs
The evaluators commented on the use of icons for menu items on the main page
(CLAS) and the Store Information page.
14
Evaluators commented on the layout of initial Finder page. For the first step of
the task, Select Finder, evaluators came up with a failure story to question #3: will the
user know that it is the right action for the effect she is trying to produce. Evaluators
made note of the Previous Searches items took up a majority of the screen space, which
may mislead users into thinking that they did not select the proper menu item.
Evaluators commenting on the way lists of items are presented. It is not
immediately obvious that the search results are clickable.
When looking at the detailed Clothing Information page, evaluators were unable
to find store information. That was because this information did not exist in the
prototype. A button for Store Information was missing
Heuristic Evaluation
The following are comments made by evaluators when performing the heuristic
evaluations

Reversing the position of the Search bar and the Title bar was awkward for
evaluators. Typically the Title bar is at the top of the screen, and the Search bar is at
the bottom. This ordering is switched in our mobile app

There were comments about the fact that there is no “obvious” way to jump back to
the main home page. Searching for a store and finding a particular piece of clothing
requires a lot of steps and windows. It would be a pain to “Back” out of all of those
menus

Having “commonly searched” items listed by default was confusing to certain users.
This was because there were items being listed by nothing has been searched for yet

Users could not tell the difference between looking for clothes and looking for
stores.

Search results were often ambiguous and unclear. One evaluator instantly backed
out of the search results after the evaluator completed their first search
15

Evaluators found the search bar to be ambiguous, they could not differentiate
between “store search” and “clothing search”

Evaluators wanted a larger preview, or at least a zoom, into the previewed clothing
item

Lack of error messages and warning screens for invalid inputs

Evaluators also noticed that we did not make use of the iPhone’s tilt function (where
the screen would change according to the orientation of the iPhone itself)

Evaluators also noticed that the “Favorites” menu was ambiguous. It lets you save
stores and designers, but does it let you save malls or plazas?

Evaluators did not notice the “sort by” option for search results

Many evaluators did not know that the red-highlighted price indicated a SALE item

Evaluators liked the fact that the information page indicated what sizes were in
stock and out of stock
16
Discussion and Analysis
Talk Aloud
Talk Aloud evaluations showed that users did not follow expected path through
the software. Four out of the six users immediately selected the store name from
history instead of entering items through the search bar. One possible explanation for
this is the way the prototype is set up in combination with the task we asked the
subjects to perform. We asked the user to find an item from the store Harry Rosen, and
it just so happened that our prototype showed Harry Rosen as a “Previously Searched”
item. This might have biased the user towards the path of least resistance, which was to
immediately select the store as opposed to searching for the clothing item directly.
In addition, once the users completed their search, the user was unable to
identify which store the item of clothing belonged to. This is an indication that it is too
cluttered or unclear, so it may be necessary to change the font or the colour scheme to
increase visibility.
Subject #3’s mistake also indicates that an entire mode of the application is
missing a “Back” button.
Cognitive Walkthrough
Again, as we can see in the comments made by the evaluators, we see that the
previous search results’ listing poses a problem for the user. The prominence of the list
causes confusion to the user.
Also, the evaluators commented on the fact that it might not be obvious to
everyone that the listed items are clickable. In order to mitigate that, it might be wise
to highlight menu items as the user has them selected, to indicate that they can actually
select them to proceed to another page.
17
One of the larger problems with our layout was in the Clothing Information Page.
We were missing key functionality in the prototype that we left out in the design phase.
Store information must be re-incorporated back into the design.
Heuristic Evaluation
As part of the I4 assignment, heuristic evaluation was done first of the three
method. Though the use of the heuristic evaluation, we found several critical problems
with our application which played a large role in both letting us know of the critical
problems and enlightening us to see that our program really is not perfect as we had
initially imagined. Of course, it also told us what and where the flaws are which allowed
us to have the ability to correct them.
Since it was the first of the three experiments, the finding of the flaws also
allowed us how to find more errors not only by ourselves but with the other two
experiments above efficiently.
18
General Implications
Each of the problems implies poor usability of our program. Lack of help and
proper guidance in each action causes the users to become confused on what they need
to do and how to go about doing what they were trying to do. Eventually, frustrating
outcomes will cause users to stop using the software. Providing help
The results of the evaluations showed key items missing from the design. This
includes a home button on every screen and in some cases a link from screen to screen
(such as a link to the map screen).
In order to conserve space in the interface, our design combined interface
objects of similar functionality together (for example: the search bar). This choice
generally caused confusion among the evaluators. They had trouble grasping that these
functionalities were put into one object, and spent time looking for something more
specific to their task. In particular, the main problems stemmed from trying to find a
store search bar and assuming the general search bar is for clothing search and vice
versa.
A similar problem showed itself when the previous search results are
automatically loaded when clicking the find button. This design choice also caused
confusion, in that the evaluators expected the search area to be empty upon entering
the find section of the application.
The results also showed a poor choice in design layout. The evaluators ran the
tests on the physical mock-up and functional prototype, and in this case it was an
iPhone. Some of them are familiar with iPhones and had a preconceived notion about
iPhone application layouts and commented that the current design was out of the norm.
Apart from missing some buttons, the main design problem of this system was
with familiarity and recognition. More details are explored along with their
recommendations in the next section.
19
Potential Improvements
The evaluation process showed many flaws in the application’s designs and numerous
areas of improvement exist.
Very High Priority Changes
Help and About CLAS
First and foremost, our application needs an about
section and help button so that the users can clearly
know what they can do, should do and know how to
do it. The help button should be available on every
page that can give a small pop-up window with descriptions possibly aided by images if
possible. Doing so should limit the number of confusion for the users as to why they are
at a certain page and to tell them how to use it. The help button can consist of a small
question mark with a circle around it
to indicate that it is a button and it can be
placed on the top right corner of the screen. Additionally, an about section can be
added on the front screen. This button will open a new page that simply describes what
this application does. That is, it is used for searching for clothing stores of their interest.
It can also give a brief note on features of the application.
Missing Button (Store Information)
Our application also missed another key item that
is crucial to our application. A button to go back to
store information must be added at the clothing information screen and another at the
store specific clothing search screen. The current prototype does not allow such travel
under certain cases of how they arrived there. This means that in order to go to the
store information, the user must restart from the beginning to search for that store
which will cause a lot of confusion and frustration. The image above shows the possible
change style that incorporates a permanent menu bar with things like help, home
button and etc.
20
High Priority Changes
Missing Button (Home)
An addition of a button to go back to initial screen,
called “home” is necessary. This button should be
available during the search process screens up to
the clothing information screen. It will provide a convenient and simple access to the
initial screen instead of requiring to press back repeatedly or to restart the application.
Since pressing the home button may cause unwanted results of losing data, it should
provide a warning screen on exit that the search results will be lost and ask if they really
want to exit.
Separation of Store Search and Clothing Search
The current layout
allows the users to
search for both
clothing and stores
both on the same search bar similar to a web’s search
engine such as Google®. This process allows expert
users to make a quick selection with ease, but proved to
cause confusion for the first time users. To avoid this
without causing inconvenience for expert users, we can
add an advanced search button where it allows the
users to type in each type of keywords separately. This
should allow novice users to know exactly where to type
what. The new functions however should be reevaluated.
21
Medium - Low Priority Changes
Moving the Previous Search Results
The previous search result was initially made for
easy re-access, but it is to be moved to a separate
page accessible by a new button on the finder.
Reorganizing the Layout
Some users found our layout of the application
inconsistent with the iPhone®’s usual look and feel.
Therefore, the title bar has been moved to the top of
screen. The permanent menu is fixated to the bottom of
the screen to always provide the users the way to go
back and to go home.
Other Cosmetic
Add-Ons
Functions like zoom can
be added to the clothing
information screen for a
close look at the article
of clothing. Once an image is clicked, a larger version of
the image can be displayed if available. Everything else
would be faded away and pressing any area will close
the zoomed view. This feature however, is very low on
the priority.
22
Critiques of the Evaluation Plan
Looking back at the results of the evaluation we carried out and knowing what
we do now and assuming that we have more resources available, there are several
things we could have done better.
During the think-aloud evaluations, we did not encourage our users to literally
“think aloud” enough. That is, the evaluators were too frequent in simply nothing saying
enough. These evaluations’ results were not enough to draw up heavy or precise
conclusions. Also, there were times when reviewing the results from notes, we felt there
were some pieces we thought we heard but were not there. We should have had more
than one person take notes or have an additional tape or video recorder at the scene.
Then nothing would have been missed and not leave us guessing.
During the heuristic evaluations, we were not fully prepared. This actually
caused some of our results to produce wrong results. For instance, we gave the
evaluators the wrong screen after an event, throwing much of the proper interactions
off. Second, the images we had to display the functional prototype were for various uses
and were inconsistent. Some were about store Harry Rosen, another about Holt
Renfrew and the product searched again were various items. With proper preparation,
we could have trained ourselves in knowing what screen comes next more thoroughly
as well as to draw up a consistent screen sets. This would have prevented us from
requiring to constantly explaining what it was supposed to display rather than simply
letting them see it.
We should have asked more questions during the cognitive walkthrough for
more data to be collected.
However, overall, we do not believe that there would have been any changes
that would have caused a significantly different result. Even though we’ve made some
mistakes, problems were ironed out through numerous repetitions with different
evaluators and participants that allowed us to retrieve number of useful data.
23
Appendix
Who Did What
Adenike Adebayo – Critiques
Andrew Duong – Design Rationale, Implications
James Ma – Think Aloud & Cognitive Data (Gathering), Results Summary & Analysis
YuChor Lam – Evaluation Techniques
Peter Park – Analysis, Implications, Improvements, Critiques, Final compilation