Putting Evidence-based Practice to Work:
Usability Testing at Northwestern
University Library
Frank Cervone
AUL for Information Technology
Monday October 29, 2007
Internet Librarian 2007
Overview




The problem of web site design
Using more formal methods
Formal methods in practice
Outcomes
The problems of web site
design

Web development/HCI is an intricate mix of
technology and design
– While there are exceptions, the majority of
librarians haven’t been trained in HCI
– Gaps in understanding the significant differences
between the on-line and in-person experience

Once a site is done, most people are not
eager to have others suggest adjustments or
changes, particularly to those things they
“know” are right
Liberally adapted from Problems with in-house website development by “The IT Guy”
http://www.newfangled.com/contentmgr/showdetails.php/id/7441
Web site design and
organizational responsiveness


The more an organization depends upon its
publics for achieving its mission, the more it
should employ dialogic features into its Web
site design
The challenge is to move the tasks
associated with Web site design and
maintenance away from being a "B-list" job
to being an imperative for the survival of
highly stakeholder-dependent organizations
Kent, M. L., Taylor, M., and White, W. J. (2003). The relationship between Web site design and
organizational responsiveness to stakeholders. Public Relations Review, 29(1), pp. 63-77.
Which leads to …

Evidence-based information practice
– “…an approach to information practice that
promotes the collection, interpretation, and
integration of valid, important, and applicable
user-reported, librarian-observed, and researchderived evidence”
Andrew Booth
Senior Lecturer in Evidence-based Healthcare Information
School of Health and Related Research
University of Sheffield
What is evidence-based
librarianship?

Data provides the primary evidence for
making decisions
– Not anecdotal stories
– Not “common sense”
– Evaluation occurs early in the process

How is this different from what often
happens now?
– Decisions are made based on

Beliefs of what is needed
– Frequently, these beliefs are inherently biased

Assumptions, anecdotal evidence, and preferences
– Evaluation, if it occurs, happens afterward
Evidence-based professional
practice


Derived from the evidence-based model of
medicine
Fundamental precepts
– Study the phenomenon
– Contrast results to other studies of the same or
related phenomena
– Combine results to better understand the
phenomenon at hand
Evidence-based practice
process
Define
the problem
Redefine
the problem
Evaluate change
Find the
evidence
Apply results of the
evaluation
Evaluate the
evidence
Defining the problem


Very similar to concepts used in “user-centered
design”
Five points
– Yes, there’s an acronym - SPICE
– SETTING

Where is this being used; what is the context?
– POPULATION

Who are the (potential) users?
– INTERVENTION

What is being done to/for them?
– COMPARISON

What are the alternatives?
– EVALUATION

What does success mean?
Levels of evidence
1.
2.
Systematic review of a wide-body of rigorous studies
Systematic review of a wide-body of less rigorous
studies
3. Randomized controlled trials
4. Controlled-comparison studies
5. Cohort studies
6. Descriptive surveys
7. Case studies
8. Decision analysis
9. Qualitative research (focus groups, ethnography,
historic, Delphi techniques, interviews)
10. Surveys, audits
Adapted from Eldredge, J. (2000). Evidence-based librarianship: An overview. Bulletin of the Medical Library
Association, 88(4). pp: 289-302. Online at http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=35250
Which brings us to the
Northwestern story…


First usability test in 2001
Focused in on “Electronic Resources”
– A home-grown resource finder


Investigation and questions based on
anecdotal evidence with ER interface
Learning experience
– Provided model for organization

Whetted people’s appetite for usability
testing
The second test – Searching in
NUcat


Began summer 2002
How are people using Voyager?
– Are there discernable patterns?
– Most importantly are there patterns of errors?
– Can we do something to fix that?

What we did
– Looked at searching patterns in our Voyager
system
– Used a more rigorous research approach
Based on grounded theory
 Simple data mining techniques

ISBN
Subject Browse
Keyword Phrase
Command
Call Number
Advanced Keyword
Subject Heading
Builder
Guided Search
Keyword
Guided Keyword
Subject
Journal Title
Author
Title
And in the process…
Searches by type
1400000
1200000
1000000
800000
600000
400000
200000
0
ISSN
ISBN
Keyword Phrase
Call Number
Command
Advanced Keyword
Guided Search
Builder
Subject Heading
Guided Keyword
Keyword
Subject
Journal Title
Author
Title
…dispelled some myths
No hits within searches by type
500000
450000
400000
350000
300000
250000
200000
150000
100000
50000
0
Which lead to our tackling a
bigger problem

Existing site did not work especially well for
the Library's various user communities
– Both students and faculty expressed frustration
with the web site
Ability to navigate the site
 Overall look and feel

– Problem areas determined by gathering
information in focus groups, online surveys,
interaction with librarians and staff
Hours and locations
 Collections
 Electronic resources

Moving to a more evidencebased process


Restructuring of the web development process
Creation of a active web group
– Library Web Advisory Group

Representation from all divisions of the Library
–
–
–
–
–
–
–
–

Reference/Instruction
Collection management
Bibliographers
Branch libraries
Technical Services
Library Public Relations
Digital Library Systems Development
Library Information Technology Support Services
Web Communications Group at University Relations
Induction process

Required reading list
– Extensive list of supplemental readings

Required training
– In usability
– Conducting a usability test
– Other “soft skills”


Evidence-based procedures
Required participation in design and testing
–
–
–
–
Usability test moderator
Usability test observer
Transcriber
Missionary/representative
Current results

Overall, site usability has improved
– And we can provide it with statistical measures!
– People find resources more quickly and easily


But there is still an unfortunate amount of confusion
Debates are less rancorous about how to
proceed
– We can always go back to the data
– We can always go back to the studies

Easier to develop a strategy for incremental
improvements over time
– No longer locked into a tight academic schedule
Remaining issues

Jargon
– I don’t know what
– Metasearch
– E-journals
– Virtual reference
– Electronic resources
means!

Why should I go here? I can find everything
in Google (or Google Scholar)
– To some degree, this is true…
The big issue

Designing for a world where they don’t even
come to the library web site
– When the day comes that Google Scholar or
Microsoft Academic Live do provide a “universal
index”
– Pluggable modules for the University portal
Thanks

Frank Cervone
Assistant University Librarian for Information Technology
Northwestern University Library
1970 Campus Drive
Evanston, IL 60208
email: [email protected]
AIM/YAHOO: hfcervone
Skype: frank.cervone