Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
A Cognitive Walkthrough of Wunderground.com
Executive Summary
Site evaluated:
http://www.wunderground.com/
User:
A casual internet user who is traveling to Seattle, WA and wants to
know more about seasonal rainfall patterns to plan a trip.
Evaluators:
The evaluation team consisted of four graduate school students who are
taking the Usability of Digital Information class at Pratt Institute. One
student served as a guide through the interface and each of the four
students took notes on paper worksheets.
Task:
Determine which month of the year has the lowest amount of rainfall
on average in Seattle, WA.
Action Sequence:
Step 1: Search for Seattle, WA.
Step 2: Scroll down to the “History & Almanac” box.
Step 3: Click “Seasonal Weather Averages” link.
Step 4: Scroll down to the “Rainfall” chart.
Step 5: Uncheck the “Show Records” checkbox.
Step 6: Note which month has the lowest average rainfall.
General Recommendations:
Specific design problems and detailed solutions are included in the recommendations section in
the body of this report. Generally, we recommend:





Reviewing content for better grouping of data sets.
Reviewing labels to ensure that they reflect the appropriate data being measured and the
location being described.
Taking care with advertising banners so they do not serve as a cognitive barrier.
Considering how people tend to read charts and design the charts so the information is
readable and understandable.
Repeating helpful chart behavior, such as displaying underlying values while hovering the
mouse over a data line, so that what a user experiences while reviewing one chart will be
anticipated and experienced in all charts.
1
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
A Cognitive Walkthrough of Wunderground.com
Table of Contents
Introduction
3
Methodology
Input Definition
User Definition
Evaluators
Task
Action Sequence
3
2
2
2
3
3
Findings and Recommendations
3
Limitations
4
Conclusion
5
Detailed Findings
Appendix A
2
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Introduction
We performed a cognitive walkthrough of weather website, Wunderground.com, to determine
the usability of the site while navigating and interpreting data available on rainfall. A cognitive
walkthrough is a usability inspection method in which a team of peers, usually designers,
evaluate an interface in the context of one or more typical tasks that a user can be expected to
complete. (Wharton, Cathleen, John Rieman, Clayton Lewis and Peter Polson. (1994) Cognitive
Walkthrough: Practitioners Guide. Boulder, CO: University of Colorado at Boulder, p. 2).
This study is a cognitive walkthrough of the weather website, Wunderground.com. In our
experience, most travel sites include information on which months to expect the most rain, the
most snow, or the highest/lowest temperatures, in other words, basic, almanac-type
information that helps a person plan a trip. We expected that a weather website would contain
this information and determined that Wunderground.com and similar sites, like The Weather
Channel (weather.com) and the National Weather Service (weather.gov), in fact, do contain this
information. Our goal was to test whether this information is easily retrievable by an average
user.
Methodology
During the cognitive walkthrough, the evaluators discuss each step within an action sequence,
telling a story of how a user might approach each step, the ease of use and understanding of
each step and where they may make a wrong move or fail to complete a task. Components of
the story include the following criteria, described by Wharton et al:




Will they try to achieve the right effect?
Will they notice that the correct action is available?
Will the user associate the correct action with the effect they are trying to achieve?
If the correct action is performed, will the user see that progress is being made toward
solution of their task?
Success with all four criteria indicates a success for the task defined in the cognitive
walkthrough. Failure to satisfy two or more of these four criteria, unless a single criteria failure
is severe, indicates a failure which should be corrected in the interface design. The severity of
the failure depends on a number of factors including how many of the criteria failed and
whether the difficulties associated with the task prevent the user from completing the entire
sequence. For example, if a user takes a long time to notice that the correct action is available or
that it is associated with the effect they want to achieve, but eventually completes the intended
action, this would be less severe than if the correct action is never found or if the user fails to
recover from an incorrect action.
Inputs
Website tested:
We selected Wunderground.com for the evaluation, because it is a popular weather website
that contains information about specific areas, based on data collected at National Weather
Service stations nationwide. Because we wanted to test a user’s ability to discover rainfall data,
3
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
we chose an area, Seattle, WA, that we know receives a significant amount of rainfall.
Users:
The target user is a casual internet user and, specifically, a traveler who is generally unfamiliar
with seasonal weather patterns in Seattle, WA. We assume that the user is familiar with World
Wide Web browser interfaces, including the function of hyperlinks and scrolling, and
understands the use of a computer, keyboard and mouse.
Evaluators:
The evaluation team consisted of four graduate school students who are taking the Usability of
Digital Information class at Pratt Institute. One student served as a guide through the interface
and each of the four students took notes on paper worksheets.
Task:
The predefined task was to find out which month of the year has the lowest amount of rainfall
on average in Seattle, WA. A series of steps outlined the most direct method of discovering
average monthly rainfall available on the Wunderground.com website. Starting at the home
page for Wunderground.com, the evaluators were asked to complete the following action
sequence and make notes on the four criteria described above:
Action Sequence
Step 1: Search for Seattle, WA.
Step 2: Scroll down to the “History & Almanac” box.
Step 3: Click “Seasonal Weather Averages” link.
Step 4: Scroll down to the “Rainfall” chart.
Step 5: Uncheck the “Show Records” checkbox.
Step 6: Note which month has the lowest average rainfall.
4
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Findings and Recommendations
Findings
Step 1: Search for Seattle, WA
The evaluators found that it is easy to locate current weather data for a given city on
Wunderground.com (Figure 1). The search box is prominently displayed and the label, “Enter a
Location,” is understandable. Located in the center of the page, the search box tells the user to
indicate a location to look up. An autofill function predicts a location based on what the user is
starting to type, providing an additional confirmation that a location input is expected. When
activated, the resulting page displays good feedback, including the page title, “Seattle, WA” and
additional weather information for that location, including temperature, 5-day forecast,
sunrise/sunset, pressure, humidity, rainfall, etc. This step was a Success.
Figure 1: Wunderground Home Page
Step 2: Scroll down to the “History & Almanac” box.
Locating historical weather information was difficult, because of the enormous amount of
information on the page and the non-uniform placement and organization of historical
information. For example, historical information about average temperature and rainfall for a
specific date was not located in the same place as historical information about average
temperature and rainfall over a period of time.
5
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
The evaluators located links and tabs whose labels created an expectation that they would lead
to the required information. Tabbed navigation at the top of the page, labeled “Climate” was
thought to contain almanac information; however this section only contained data for the
location that was saved as the local area, in our case New York, NY. Although the Local Climate
subpage did contain information related to average precipitation
over time, and did allow you to switch location, it showed daily
averages over time, not seasonal average for a given month.
Similarly, tabbed navigation in the middle of the Seattle charts
labeled “Today’s Almanac” only showed averages for the current
date. A link at the bottom of this chart, labeled “View more
history data,” did not contain seasonal average data. Each time
the user was forced to return to the home page to look for
another path to seasonal weather averages.
A box labeled History & Almanac information, located midway
down a long page (Figure 2), contains a link to “Seasonal
Weather Averages.” The evaluators felt that most users would
not necessarily know to scroll that far down the page. Banner
advertisements beneath the current data tables give the
impression that the information beyond it is of low importance,
and the organization of the additional information was cluttered
and confusing, which made it difficult to find the information we
were seeking. A section labeled “Nearby” included the History &
Almanac box for Seattle, in addition to information about
locations near Seattle, including coastal water temperatures,
earthquake activity and snow depth and “Astronomy”
information.
The History & Almanac box is the only content block that links to
the seasonal averages charts. Unfortunately, the History &
Almanac box does not contain rainfall data. It does contain a
“Seasonal Weather Averages” link to get to seasonal rainfall data.
This is a critical problem. Unless the user is aware that rainfall is
one of the data points described in the seasonal weather
averages page, he may not know to click on this link. Therefore,
the evaluators felt that a user would not have any reason to
assume that the History & Almanac box would be any better than
these other information leads. Information scent, or the ability to
know that one is getting closer to the information he needs, was
too faint. This step was considered to be the most critical Failure
in the action sequence.
History &
Almanac Box is
way down here
6
Figure 2: A Very Long Page
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Step 3: Click “Seasonal Weather Averages”
link.
Like the location search box, the label
“Seasonal Weather Averages” is a good
indication of the information that should
appear on the resulting page. Also, it looks
and behaves like a link: it is underlined, it is a
different color from the rest of the text, and
it changes color when a mouse is hovering
over it. This was an example of a very
straightforward link with predictable results.
This step was considered a Success.
Figure 3: History & Almanac Table
Step 4: Scroll down to the “Rainfall” chart.
Because the chart labeled rainfall is
partially visible, the evaluators felt
that the user will attempt to scroll.
On our rather large monitor, the
rainfall chart was visible right away.
In fact, one doesn’t have to scroll far
to find the rainfall chart, even if it is
not immediately visible on smaller
screens. Scrolling causes the screen
to move, which is feedback that the
action is successful. This step was a
Success.
Rainfall Chart
Figure 4: Rainfall Chart
7
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Step 5: Uncheck the “Show Records” checkbox.
The rainfall chart contains a historical average data line as well as record high and low
precipitation data lines. Figure 5 below shows the Rainfall chart with the Records data visible
and not visible. All of our evaluators noted that they didn’t see the check box labeled “Show
Records” until after scrolling to and reading the chart, and even then, only one evaluator
noticed that the most prominent line on the chart was in fact a record line. Only then did the
evaluators spot the check box that allows the user to remove the record data.
All evaluators agreed that the user would associate the correct action with the effect to be
achieved, but only when they realized the chart is best read without the record data. The
evaluators also agreed that only a skilled chart reader would immediately understand the
difference between average and record data and that few users would know to remove the
record line to obtain a better reading of the chart data. Once the Show Records check box was
located, however, all evaluators noted that the graph changes, providing feedback that the
records data was removed from the chart. This step was considered a Failure.
Figure 5: Comparing Checked and Unchecked States
Note: the blue line in the left chart is the Average Rainfall, which is difficult to read when the
Record Rainfall is displayed. The right chart shows Average Rainfall without the Record Rainfall
data.
8
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Step 6: Note which month has the lowest average rainfall.
This is an entirely cognitive task: the user reads the chart. So the question to ask to determine if
the user can note the correct month is, “Will the user be able to interpret the chart correctly?”
Most of the evaluators assumed that the gridlines that are aligned with each month name
indicated the middle of that month, when in fact it indicated the beginning of the month. A
closer inspection of the chart shows that data for the month of January begins at the leftmost
gridline of the chart and that data for the month of December ends at the rightmost gridline.
Beginning of
January
End of
January
Beginning of
August
Figure 6: Seasonal Weather Averages
Three out of four evaluators said that, even though the criteria for associating the correct action
with the required effect are met, users would probably not be able to interpret the chart,
because it is not clear where the months begin and end, and because it is also not clear whether
to read the records line or the averages line, since the records line was much more prominent.
Only one evaluator said users would be able to make a good guess. The rest felt the user would
be confused or simply make the wrong guess.
Also, based on the behavior of the rainfall chart within the Current Data tab on the main Seattle,
WA page, users might expect to be able to hover over a data line to see actual data values on
the chart. This particular chart does not indicate a value on hover. Therefore the only real
9
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
feedback to determine if you read the chart correctly is the position of the data points in the
month that appears to have the lowest rainfall. This step was considered a Failure.
Recommendations
The home page is cluttered and difficult to understand. There is too much information and like
information is not always grouped together.

We recommend evaluating home page content for better grouping of like information, and
either display that information together in a more condensed format or create new pages
that aggregate like data.
We also noticed that in the case of seasonal averages, there were many red herrings where it
appeared that historical averages might be available but in fact, none were displayed.




The blue tab, labeled “Climate,” at the far right of the top navigation, could be relabeled
“My Climate” to indicate that it is set to the user preferences, not necessarily the locality
currently being displayed.
The gray tab marked “Today’s Averages,” located in the local navigation within “Seattle
Weather at a Glance,” includes data on rainfall and a link to “View more history data.”
Provide a link to seasonal historical averages from the “View more history data” page.
The chart on the Current Data tab, located in the local navigation within “Seattle Weather
at a Glance,” also shows rainfall data and a small chart of the day’s accumulation. Provide a
link to seasonal averages, as well.
Review the website content for other information related to historical data, and provide
links to seasonal averages. Also, where possible, link from seasonal historical averages to
current average data to further associate this information.
The evaluators noted that banner advertising created a cognitive barrier between sections of
information on the website. While we accept that advertising is necessary to generate revenue,
the placement of advertising should not interfere with the interpretation of site content.


Reconsider the placement of banner advertisements that give the impression that the end
of the page has been reached. Do not use banner advertisements as section breaks. Keep
ads in the right column, top and bottom of page, so they do not separate content sections.
Alternatively, arrange banner ads within content groups as part of, rather than a barrier to,
site content.
The “Show Records” checkbox was missed by all of the evaluators and only after struggling to
read the chart did they realize that it needed to be unchecked, so that the average data could be
read clearly.

Make the unchecked state the default. This makes sense semantically since the user would
then associate this check box with the action, “Show Record,” rather than to stop showing
the record.
10
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
All of the evaluators agreed that the chart was difficult to read. They could see which areas of
the line appeared to contain the lowest values, but were unable to determine immediately if the
data was for July or August.


Consider aligning the month labels between the two gridlines that encompass data for that
month. This will provide more confidence that the action (identifying the correct month)
was achieved and the task was completed.
Consider allowing the user to discover data values by hovering over a chart line, similar to
how data is revealed on hover on the Current Data chart on the main Seattle, WA, page or
any local weather page.
Limitations
One of the limitations of a Cognitive Walkthrough is the focus on ease of learning (Anderson, J.
R. (1987) “Skill acquisition: Compilation of weak-method solutions.” Psychological Review, 94,
192-211). While the evaluators supplied suggestions for improving the display of graphical
information for better understandability, recommendations about content within data charts, as
opposed to features that are specific to the interface, are somewhat outside the scope of a
cognitive evaluation.
Another limitation is related to the “narrow focus” of cognitive walkthroughs (Wharton et al, p.
4). The action sequence that was selected was determined to be the “correct sequence,” i.e.,
the quickest and best path to retrieving the required information. The predefined “correct”
sequence may not be what the user thinks is the correct sequence, based on his or her
experience with similar websites. This is another limitation of the cognitive walkthrough
method. Multiple red herrings in the interface throw off the user by providing false information
scent. Therefore it is helpful that a complete evaluation combines a review of the “correct
sequence” as indicated by the cognitive walkthrough, with other evaluation methods that
explore alternative action sequences, including those that may or may not result in the correct
result.
Conclusions
Because the Wunderground.com website contains a large amount of data, it is imperative that
the data be organized in a logical way. The current website offers a number of ways to locate
historical information; however, many content areas relating historical data fail to provide links
to related data. A review of the website with the goal of better grouping of historical content, as
well as linking historical data to seasonal averages where possible, would be helpful in
improving access to this data.
11
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Appendix A: Detailed Findings
Step 1: Search for Seattle, WA
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
Yes
Success/
Failure
Success
2. Will the user notice if
the correct action is
available?
Yes
Success
3. Will the user associate
the correct action with
the effect trying to be
achieved?
Yes
Success
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task?
Yes
Success
Comments
E1: Yes. User will look for a way to enter a location, such as a Search box.
E2: Yes.
E3: Yes.
E4. Yes.
E1: Yes. “Enter a Location” box in center.
E2: Yes.
E3: Yes. Autofill
E4: Yes.
E1: Yes. User will associate clicking the Search button with the correct
action.
E2: Yes.
E3: The autofill = correct state
E4: Yes.
E1: Yes. When the user clicks “Search,” the Seattle, WA page opens.
E2: Yes.
E3: Yes. The entire page changes to a new location.
E4: Goes quickly to the next screen.
Step 2: Scroll down to the History & Almanac box
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
No
Success/
Failure
Failure
2. Will the user notice if
the correct action is
available?
No
Failure
3. Will the user associate
the correct action with
the effect trying to be
achieved?
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task?
Yes
Success
No
Failure
Comments
E1: No. User does not know to scroll all the way down.
E2: No. I would never think to scroll so far down.
E3: No. User has to scroll significant amount to find “Almanac” near
bottom of page. “Today’s almanac” only returns current information.
E4: No. Not obvious you would have to scroll to the bottom. A lot of
other data and text makes it hard to focus.
E1: No. Link to “today’s Almanac” seems logical but you can’t get
information about historical averages there. The correct link is too far
down to scroll
E2: No. Not only would I not scroll down, the interface is very busy and
“History & Almanac” is not visible.
E3: No.
E4: No.
E1: Yes. I scrolled until I found the box.
E2: Yes.
E3: Yes. I scrolled because I couldn’t find it at the beginning.
E4: Yes.
E1: Probably not; Maybe if they see the “History & Almanac” box, but
there is too much clutter, text banner ads, etc that they may miss it.
E2: No.
E3: No real feedback other than finding box mentioned – doesn’t show
rainfall.
E4: No. Not really as no mention of rainfall in that box.
12
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Step 3: Click “Seasonal Weather Averages” link.
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
Yes
Success/
Failure
Success
2. Will the user notice if
the correct action is
available?
Yes
Success
3. Will the user associate
the correct action with
the effect trying to be
achieved?
Yes
Success
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task?
Yes
Success
Comments
E1: Yes. It is labeled “Seasonal Weather Averages”
E2: Yes.
E3: Yes.
E4: Yes.
E1: Yes. It is labeled correctly.
E2. Yes.
E3. Yes.
E4. Yes.
E1: Yes. It is a link.
E2. Yes.
E3: Yes. Looks like a link, changes appearance when mouse hovers over
wording.
E4: Yes. Looks like a link and looks clickable. Highlights and is underlined.
E1: The page that opens is titled “Seasonal Weather Averages.”
E2. Yes.
E3: Web page changes - charts are available.
E4: Yes.
Step 4: Scroll down to the Rainfall Chart
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
Yes
Success/
Failure
Success
2. Will the user notice if
the correct action is
available?
3. Will the user associate
the correct action with
the effect trying to be
achieved?
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task?
Yes
Success
Yes
Success
Yes
Success
Comments
E1: Yes. Chart labeled Rainfall is partially visible, so user will attempt to
scroll.
E2. Yes.
E3. Yes.
E4. Yes.
E1-4: Yes. Evaluators generally agreed that a user would attempt to
scroll.
E1: Yes. Screen moves.
E2. Yes.
E3. Yes.
E4: Screen moves.
E1: Yes. As user reads chart, user can determine that rainfall is indicated
and can find the lowest average month.
E2. Yes.
E3. Yes.
E4: Don’t have to scroll far to find it. You see it right away.
13
Evaluation 1: Cognitive Walkthrough
Noreen Whysel
LIS 644-01
Step 5: Uncheck the “Show Records” checkbox
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
No
Success/
Failure
Failure
2. Will the user notice if
the correct action is
available?
No
Failure
3. Will the user associate
the correct action with
the effect trying to be
achieved?
Yes
Success
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task?
Yes
Success
Comments
E1: No. Not visible.
E2: No. Not apparent.
E3: No answer recorded.
E4: No.
E1-E4: In discussion, all evaluators said no, not at first. You need to
evaluate the chart before realizing you need to remove the Records line.
E1: No. Not clear. We attend to the chart first.
E2. No.
E3: No.
E4: No.
E1-4: All evaluators noted that they didn’t see the check box until after
scrolling to the chart.
E1: Yes. They may have to scroll back up first, but clicking on the box
does have the correct effect.
E2: No.
E3: Yes. The graph changes.
E4: Yes.
E1: Yes. Graph changes.
E2: Yes.
E3: Yes. Because the graph significantly changes.
E4: Yes. Graph changes right away.
Step 6: Note which month has the lowest average rainfall.
Criterion
Yes/No
1. Will the user try to
achieve the right effect?
Yes
Success/Fail
ure
Success
2. Will the user notice if
the correct action is
available?
Yes
Success
3. Will the user associate
the correct action with
the effect trying to be
achieved?
Yes
Success
4. If the correct action is
performed, will the user
see that progress is being
made toward solution of
the task
No
Failure
Comments
E1: Yes. Read chart.
E2: Yes.
E3: Yes.Reading. Hard to determine actual value of average and where
months begin and end.
E4: Yes.
E1: Yes. Read is the only thing you can do.
E2: Yes.
E3: Yes.
E4: Yes.
E1: Yes, but it is hard to determine the actual value and where months
begin and end.
E2: No. Difficult to read, no indication of how to accurately read and
select data.
E3: Yes, but reading it is hard to determine the actual value of average
and where month begin and end.
E4: Yes, but if looking for exact value it is difficult to see. Bad affordance
as can hover over some lines but not for the rainfall chart.
E1: No. There is no value to ensure that you picked the correct month.
Hovering the mouse on the line doesn’t show data like it did on the main
page rainfall chart.
E2: No. No hover over the month
E3: No. No values – can’t find actual average of rainfall.
E4: No.
14