End-User Research in the Informatics Process
An Overview and Demonstration n Don Rickert, Ph.D. n 4/8/03
What do I mean by “Research?”
n
Any set of activities where data are collected and
analyzed for the purpose of making generalizations
beyond the study (Hughes, 1999)
2
What Is User Research in the Technology
Development Context?
n
The discipline that is concerned with…
n
n
Translating end user or other consumer insights into the
development of new systems (or improving legacy systems)
Who does this research?—much interdisciplinary
cross-fertilization, especially among—
n
Psychology
n
Human Factors
n
Information & Decision Systems
n
Market/Consumer Research
n
Industrial Design
n
Anthropology
3
Fundamental Foci of End-User Research
in a Technology Context
n
Figuring out who the target users are
n
Understanding the motivations of these target users
n
Uncovering their unmet needs and desires (what is
appealing?)
n
Using the insights gained to support intelligent
business decisions
4
Typical Pragmatic Research Questions
n
Will anybody care about this product?
n
Why should they care?
n
Who are these people who will care?
n
What should the product look like?
n
How should it behave?
n
Will people (the ones who care) pay enough for the
product for us to make money?
n
Bottom line: Should we build it at all?
5
Faith in Reason
n
Historically, the practice of Research involving
people tends to be characterized by—
n
A rational view of human motivation
n
Faith in linguistic measures (the “Voice of the Consumer”)
6
Risks to Gaining Useful Insights From
People
n
Asking irrelevant questions
n
Not asking the right questions
n
Asking questions in such a way as to create
attitudes and opinions where none previously
existed
n
Assuming that what happens in a formal research
setting applies in the “real world”
n
Confusing what people are willing to say in a group
interview with what they really think
7
More Risks
n
Observing people doing things that they would not
really do
n
NOT observing them doing the things that they really
would do
n
Confounding people’s remembered or imagined
beliefs, attitudes, intentions or experiences with
actual beliefs, attitudes, intentions and experiences.
n
Talking to, surveying or observing the wrong people
8
Reality Check
n
Beliefs, attitudes and intensions expressed by
people (i.e. “Voice of the Consumer”) are notoriously
unsuccessful at predicting actual behavior.
n
What people say and what they do are very often
different.
9
Why Is This So?
n
People are not consciously aware of all of their
beliefs, attitudes and intensions.
n
People don't really know what they want or need or
how to articulate it.
n
Motivation is not completely rational.
n
n
Our thought process is different in different emotional states.
People’s memory about an experience is a VERY
different thing than the experience itself.
10
The Role of Science
n
Science is—
n
n
•
Discover what is so
•
Explain it
Science myths
n
n
A form of inquiry that is capable of revealing to us the structure of the
world and explaining what the world is really like. It seeks to—
It IS possible to measure the all of the key variables and thus explain
people’s behavior
n
Science is always deductive and involves hypothesis testing
n
Good science is always quantitative
n
Statistical significance is the same as importance
Beyond that, there are MAJOR debates on what makes for
inquiry that can be suitably called Science.
11
The Common Concerns on Which Most
Scientists Agree
n
The notion of Error and the importance of avoiding it
n
Reliability
n
Validity
12
Error
n
Type I Error
n
n
Type II Error
n
n
Failing to observe something that really is true
Concluding that something is true when it really isn’t
Type III Error
n
Asking the wrong questions
13
Reliability
n
Are your measurements good?
n
E.g. can you repeat your study and consistently get the same results?
14
Validity
n
Are you measuring what you think you are measuring?
n
Internal Validity
n
n
Is the effect you observe due to I stimulus that you have identified or
something else entirely (that you don’t know about)?
External Validity
n
Do the conclusions based on your observations apply in the real world?
15
Good Science boils down to …
n
n
Avoiding error; i.e.—
n
Failing to observe something that really is true
n
Concluding that something is true when it really isn’t
n
Asking the wrong questions
By…
n
Developing reliable measures
n
Observing the effects of stimuli on outcomes
n
Ruling out threats to validity
16
All of the methodology stuff that researchers
argue about…
n
…is really about tactics for controlling error
furthering reliability and validity, such as—
n
Representative samples
n
Qualitative vs. Quantitative
n
Response bias
n
Degrees of freedom
n
Experimental control
n
Situational biases
n
Repeatability
17
“Convergence of Method”
n
No research method is perfect.
n
No method, by itself, can ensure reliability and validity
n
Scientists always employ multiple methods, using
the strengths of one or more methods to
compensate for the weaknesses of each.
n
The process is called—
n
“Convergence of method”
n
Or “Triangulation
18
The Most Common Methods Employed by
End-User Researchers
n
Questionnaires
n
Focus Groups
n
Interviews
n
Usability Testing
n
Real World Observation, such as…
n
Ethnography
n
Analysis of Artifacts
19
Questionnaires
n
Key Points
n
n
Acceptable for “getting a pulse”
on gross opinions
•
“I like this.”
•
“I hate that.”
Easy way to collect data for
quantitative analysis
20
Questionnaires
n
Typical purpose
n
n
n
n
n
To measure people’s experiences to assess their satisfaction with
experiences of products
To obtain “factual” information about people, such as their income or age.
Why researchers use them
n
Easy way to collect data for quantitative analysis
n
Easy to get large sample sizes for “statistical significance”
The risks
n
n
To measure people’s beliefs, attitudes, intentions in order to make
inferences about their behavior
They measure people’s remembered or imagined beliefs, attitudes,
intentions or experiences
The possibility
n
A questionnaire with a well-documented relationship with actual behavior
outcomes makes powerful quantitative analyses possible.
21
Focus Groups
n
Typical purpose
n
n
Same as questionnaires: measure beliefs, attitudes, intentions and
experiences
Why researchers use them
n
n
An assumption that, somehow, group discussion of attitudes, beliefs,
intentions and experiences “draws out” the truth.
Convenience
•
n
n
They look rigorous
The risks
n
n
Focus groups are a big business and there are lots of facilities.
They measure what people are willing to say in a group about their
remembered or imagined beliefs, attitudes, intentions or experiences
The possibility
n
The discussion that occurs in focus groups can give researchers many
insights to better explain actual consumer behavior that they observe in
companion studies.
22
Interviews
n
Key Points
n
n
Can be…
•
In a facility or
•
In the field or
•
Over the telephone
Do not generate easy to analyze
data.
23
Interviews
n
Typical purpose
n
n
Why researchers use them
n
n
n
Probing by a good interview can draw out much more information than
questionnaires
Convenience—can even be done over the telephone or online
The risks
n
n
n
You guessed it—measure beliefs, attitudes, intentions and experiences
Same as questionnaires and focus groups, plus
Asking questions can create attitudes and opinions where none existed
before.
The possibility
n
Hearing what consumers say can add to insights gained from actual
observation of behavior.
24
Usability Testing
n
Key Points
n
n
n
What people do in a
controlled setting
When you are asking the
questions
Relatively unobstrusive
observation of semi-realistic
behaviors
25
Usability Testing
n
Typical things that researchers want to measure with
usability tests
n
n
Why researchers use them
n
n
n
n
Problems with ease of use and other conventional quality problems
A notion of scientific rigor—controlled setting with well-defined usage
scenarios
They can allow relatively unobtrusive observation of semi-realistic
behaviors
They do a good job of uncovering egregious problems with conventional
quality
The risks
n
“Type III” error
•
n
the type of error arising from asking the wrong questions
Observing people doing things that they would not really do and NOT
observing them doing the things that they really would do.
26
The possibility for usability tests
n
When the right research questions are identified,
usability testing really can help in identifying
unforseen problems with a product.
n
When combined with other methods such as
questionnaires, interviews, real-world observation,
etc. usability testing provides compelling data.
27
Ethnography
n
Key Points
n
n
n
n
n
n
On people’s own turf
Observation of what they
really do
Often supplemented by field
interviews and quantitative
Expensive
Data analysis is timeconsuming.
High external validity
(generalizable conclusions)
28
Analysis of Artifacts
n
Usually—
n
Customer Service logs
n
Complaint emails
n
Provide good context for understanding actual
observations of customer behaviors
n
Can lead to wrong conclusions if interpreted ‘standalone’ (not in the context of actual observation).
n
I.e. should be done in concert with ethnography or other
observational research
29
Ethics in End-User Research
n
The following slides are exerpted from a refereed
paper entitled The Ethics Problem in Technology
Companies and What Can Be Done About It
presented by Don Rickert and Alycen Whiddon at
the—
n
The eighth annual Ethics & Technology Conference, June 24-25,
2005 at Saint Louis University
•
http://205.240.10.101/denver.ethicstech/past_conference.html
30
Happy workplaces are a prerequisite for
becoming ethical workplaces.
n
This is all about creating workplaces that are
conducive to the type of reflection and responsibility
needed on the part of every individual in the making
of an ethics-driven company.
n
See Mihaly Csikszentmihalyi’s recent book, Good Business:
Leadership, Flow and the Making of Meaning (2003)
31
Good workplaces don’t have…
n
Periodic staff reductions, especially when outsourcing is
used to fill the deleted positions
n
Mergers (culture clashes and staff reductions are almost
always a result)
n
Too few staff combined with aggressive development
schedules
n
Vague and ever changing goals
n
Frequent reorganizations and reshuffling of staff
n
Dysfunctional politics
32
Human-Centered Product Design
n
Human-centered design is all about attempting to
meet the real needs of consumers, not the
development of new and improved manufactured
desires.
n
What is important is that all companies, especially
technology companies, need an organized and
empowered group, whose duty it is to focus on the
actual needs of customers.
33
The Belmont Report
n
Prescribes the essential responsibilities of
researchers working with human beings, including
informed consent, assessment of risks and benefits
and selection of research participants.
n
Three key ethical principles
n
n
Respect for Persons
n
Beneficence
n
Justice
A copy of the Belmont Report is inclduded with this
presentation
34
Focus Responsibility (Making ethical
considerations everyone’s problem)
n
While the Belmont Report is a tool for focusing
ethical decision-making for researchers, The Social
Impact Statement helps engineers and others
involved in the design of new products.
35
The Social Impact Statement
n
Attributed to Ben Shneiderman
n
Its explicit purpose is to provide a framework for designers to investigate the
social impact of the systems they are responsible for.
n
Every product development team should prepare a detailed Social Impact
Statement for whatever it is they produce. The report includes the following
sections—
n
Executive summary
n
Description of the system
n
Analysis of the ethical issues (stakeholders, principles, risks, etc.)
n
Recommendations for actions, with analysis of the possible outcomes
n
Reader's guide to literature on the issues in more depth
n
Appendix that describes the methods used to collect data and prepare the analysis.
36
Education
n
What we need is true education in ethical principles.
n
Such education happens in university programs, not
in two-day workshops.
n
The real hope is with the next generations.
n
Beacons of light
n
Online Ethics Center for Engineering and Science at Case
Western Reserve (http://onlineethics.org/).
37