How to Write
Better Questionnaires
From the author of the new book:
The Complete Guide to Writing Questionnaires:
How to Get Better Information for Better Decisions
David F. Harris
3 October 2014
| 1
Agenda
• Why I Wrote This Book (5 minutes)
• Important Experiments on Questionnaire Design (15 minutes)
• The Framework for Writing Questionnaires (5 minutes)
• The 8 Key Themes for Writing Questionnaires (20 minutes)
• Q&A (15 minutes)
| 2
If I Am Successful, You Will Have:
• Better sense of the enormous amount of measurement error from
our questionnaires
• Questionnaire design needs to be taken much more seriously
• You will have eight key themes for how to write better questionnaires
1.
2.
3.
4.
5.
6.
7.
8.
| 3
Eight Key Themes
Plan Research to Support Decisions
Conduct Qualitative Research
Write a Questionnaire Plan
Make Questions Clear
Make Questions Answerable
Make Questions Easy
Make Questions Unbiased
Pretest the Questionnaire
The Problems I Want to Help Solve
•
•
•
•
Research rarely organized to support decisions
Often no qualitative research
Questionnaires much too long
Many well-intentioned questions did not make sense to
respondents
-
Unclear, unanswerable, too complex, and biased
• Too little real pretesting (cognitive interviewing)
• Questionnaire writing has not been thought of as a serious
discipline
| 4
Writing Questionnaires is Arguably One of the
Most Difficult Forms of Writing
• It is a “conversation” between you and hundreds, and
sometimes thousands, of diverse respondents
• The conversation may take 15, 30, or even 45 minutes
• Every single question has to be written so that every one of
these diverse respondents understands each question exactly
the same way
-
Can recall the information you are asking about
Will answer without bias
Is willing to answer each question
• Unlike a live conversation, however, you are not there to clarify
ambiguities
| 5
The Problems I Want to Help Solve
• Respondents will only give you so much time and effort
• They understand questions on their terms, not yours
• If you don’t communicate well with them on their terms, they
will drop out or make up answers to get through the survey
Measurement
error
| 6
Inaccurate or
biased
information
Misunderstandings
&
Bad Decisions
Q: How many cars do you
you own?
own
| 7
1. Did you, yourself
yourself take an aspirin within the past 7 days?
2. How many vitamin bottles are in your household
household?
| 8
I Decided to Write this Book Because
• Nearly every questionnaire I have seen has questions that are biased,
scales that are unbalanced, concepts that are unclear, questions that
are double-barreled, and words and phrases that respondents don’t
use.
• Slight changes in how questions are worded have a significant impact
on the results
• If the questionnaire is too long, too difficult, or asks questions that
do not make sense to respondents, they drop out or just make up
answers without much effort
• There needs to be a comprehensive user-friendly guide on how to
write questionnaires
• I want to help us write better questionnaires to get better
information to support better decision-making
| 9
There is a Huge Volume of Knowledge on How to Write
Questionnaires – Largely Unknown to Private Industry
• The University of Michigan’s Institute for Social Research
• University of Chicago - NORC
• Agencies within the United States Federal Government
•
•
•
The National Center for Health Statistics
The US Census Bureau
The Bureau of Labor Statistics
• Research Organizations
•
•
•
National Opinion Research Center
Research Triangle Institute (RTI)
Westat
• Other International Agencies
• The American Association of Public Opinion Research (AAPOR)
| 10
| 11
| 12
Experiment in 1950
People were randomly assigned to answer the following question:
Group A
Group B
Do you think the United States
should let communist newspaper
reporters from other countries come
in here and send back to their papers
the news as they see it?
Do you think the United States
should let communist newspaper
reporters from other countries come
in here and send back to their papers
the news as they see it?
36% agree
73% agree
| 13
Order Bias
Group A
Group B
Q1. Do you think the United States should
let communist newspaper reporters from
other countries come in here and send back
to their papers the news as they see it?
Q1. Do you think a communist country like
Russia should let American newspaper
reporters in and send back to their papers the
news as they see it?
90% agree
36% agree
Q2. Do you think a communist country like
Russia should let American newspaper
reporters in and send back to their papers the
news as they see it?
Q2. Do you think the United States should let
communist newspaper reporters from other
countries come in here and send back to
their papers the news as they see it?
73% agree
66% agree
The first question has a substantial impact on the second question
| 14
Source, Hyman & Sheatsley, 1950
Question Wording
When people were asked whether they would:
“Favor or oppose taking military
action in Iraq to end Saddam
Hussein’s rule.”
68% favored military action
25% opposed military action
“Favor or oppose taking military
action in Iraq to end Saddam
Hussein’s rule even if it meant that
U.S. forces might suffer thousands
of casualties.”
43% favored military action
48% opposed military action
The introduction of U.S. casualties altered the context of the question and
influenced whether people favored or opposed military action in Iraq
Ref: January 2003 Pew Research survey
| 15
Question Wording
What about these terms? It makes a big difference.
“…making it legal for doctors to give
terminally ill patients the means to
end their lives.”
“…making it legal for doctors to assist
terminally ill patients in committing
suicide.”
“Welfare”
“Assistance to the poor”
“Easy”
“Convenient”
Ref: January 2003 Pew Research survey
| 16
Open or Closed-ended Questions?
Q: What one issue mattered most to you in deciding how you voted for president?
The economy
The war in Iraq
Health care
Terrorism
Energy policy
Other
Candidate mentions
Moral values/social issues
Taxes/dist. of income
Other issues
Other political mentions
Change
Other
Don’t know
Closed-ended
Open-ended
58%
10
8
8
6
8
2
100
35%
5
4
6
*
43
9
7
7
5
3
3
9
7
100
Data from Pew Research Nov ’08 Post-election survey
| 17
Open or Closed-ended Questions?
Q: There have been a lot of national and world events and changes over the past 50
years…Would you select/mention one or two such events or changes that seem to you
to have been especially important?
World War II
Exploration in space
Assassination of J. F. Kennedy
Invention of the computer
The Vietnam War
All other responses
Don’t know
Closed-ended
Open-ended
23%
14%
16
7
12
5
30
1
14
10
5
52
0
11
100
100
Sources: University of Michigan Survey of Consumer Attitudes, July and August, 1986
Schuman, Method and Meaning in Polls and Surveys, 2008
| 18
Open or Closed-ended Questions
Q: What is the most important thing for children to prepare them for life?
Closed-ended
Open-ended
61.5% selected “To think for
themselves”
4.6% gave an answer that could be
interpreted as “To think for
themselves”
Source: Pew Research Center, October 1999
| 19
Question Form & Acquiescence
Bias
Agree / Disagree format
Forced Choice
The best way to ensure peace is
through military strength
The best way to ensure peace is
through military strength
55% agree
(33% agree)
42% disagree
OR
Diplomacy is the best way to ensure
peace
(55% agree)
Source: Pew Research Center, October 1999
| 20
Selecting Numeric Categories from a List
The question asked students how many hours they studied per day
Low Range of Answer Choices
High Range of Answer Choices
Less than .5 hours per day
.5-1 hour
1-1.5 hours
1-2 hours
2-2.5 hours
More than 2.5 hours
Less than 2.5 hours per day
2.5-3 hours
3-3.5 hours
3.5-4 hours
4-4.5 hours
More than 4.5 hours
23% indicated they studied more
than 2.5 hours per day
69% indicated they studied more
than 2.5 hours per day
Source: Rockwood, Sangster, Dillman (1997)
| 21
The Framework
Plan Research To
Support DecisionMaking
Conduct Qualitative
Research
Plan the
Questionnaire
Make Questions
Clear
Make Questions
Answerable
Presenting
Lists
Make Questions
Easy
Presenting
Rating Scales
Pretesting the
Questionnaire
| 22
Make Questions
Unbiased
Asking Open-Ended
Questions
First 3 Themes for Writing
Questionnaires
1. Plan Research to Support Decision-Making
2. Do Qualitative Research First
3. Write a Questionnaire Plan
| 23
1. Plan Research to
Support Decisions
• Most decisions are made based on 3-7
things
• Planning research to support decisions
is the way to:
-
Do the right “type” of study
Get the right data
Get accurate data
Keep the questionnaire shorter
We will decide _______________________________________
The information we need to make this decision is ___________
The criteria for the decision is ___________________________
| 24
2. Conduct Qualitative
Research First
On a basic level:
• To know what to ask, and how to ask it
• To figure out who to ask – so you survey the right people!
On a deeper level:
• In order to understand a topic, you need the insights that
only qualitative research can provide
| 25
3. Write a Questionnaire Plan
•
•
•
•
State decisions
Information needed
Order of information needed
How to analyze the data
Decisions: We will decide ____________
Information needed
How to ask the question
Screener
Message Reception
Purchase History
Purchase Intention
| 26
Analysis Plan
3. Write a Questionnaire Plan: ATU example
Decisions:
We will decide how to adjust our messages.
Information needed
How to ask the question
Analysis plan
Message Reception
Are customers receiving
our brand messages?
Purchase History
Who has purchased our
brand, and now much?
See if message reception
is correlated with
purchase or intent to
purchase
What is correlation
between message
reception and purchasing?
Purchase Intention
Who intends to
purchase our brand?
| 27
Is purchase intention
correlated with message
reception?
Order Misunderstanding
1. How many headaches have you had in the past 30 days?
2. Did you take any medicine for your most recent headache?
3. Did you see a doctor for your most recent headache?
4. How many of the [INSERT Q1] headaches you had in the past 30
days were mild, moderate, and/or severe?
| 28
3. Write a Questionnaire Plan
Distinguish between information needed and how to ask the question.
Q: How would you rate the technical skills of the technician who repaired your
computer?
Q: How interested are you in this new product?
Q: Is your dog a vegetarian?
| 29
First 3 Themes for Writing
Questionnaires
1. Write a Research Plan
We will decide ______
The information needed is ______
2. Do Qualitative Research First
To know what to ask, and how to ask it
To determine who to survey
To develop a complete understanding of your subject, your respondents
3. Write a Questionnaire Plan
Organize order of information needed
Work with clients on information needed, not how to ask questions!
| 30
Writing Questions
Plan Research To
Support DecisionMaking
Conduct Qualitative
Research
Plan the
Questionnaire
Make Questions
Clear
Make Questions
Answerable
Presenting
Lists
Make Questions
Easy
Presenting
Rating Scales
Pretesting the
Questionnaire
| 31
Make Questions
Unbiased
Asking Open-Ended
Questions
Writing Questions
Make Questions Clear
1.
2.
3.
4.
5.
6.
7.
8.
State the unit of measurement.
Use the vocabulary of respondents.
Use precise words and phrases.
When using the word “you,” make sure respondents know to whom you
are referring.
Make sure the question is really asking only one question.
When asking for percentages, make sure the base is clear.
Make sure the question stem and the answer choices match each other.
Use bold, underlining, italics, and/or capitalization to highlight key words
and phrases.
| 32
4. Make Questions Clear
Q: How long was your baby in the hospital?
State the unit of measurement (Clear #1)
Improved question(s):
How many days was your baby in the hospital?
Or
How many inches was your baby at birth?
| 33
4. Make Questions Clear
How frequently do you go to the grocery store?
__ Very frequently
__ Frequently
__ Neither frequently nor infrequently
__ Infrequently
__ Very infrequently
Fred:
I go there once a week. That is pretty
frequently, so I’ll say frequently.
Joe:
I go there once a week, which isn’t that
much, so I’ll say infrequently.
State the unit of measurement (Clear #1)
In the past 30 days, how many times have you, yourself, gone to the grocery
store?
__ # times gone to grocery store
| 34
4. Make Questions Clear
Thinking of your typical depression patient, for what percentage
do you prescribe Prozac?
__ % prescribed Prozac
Use the vocabulary of respondents (Clear #2)
Use precise terms and phrases (Clear #3)
| 35
5. Make Questions Answerable
Q: Of the last 100 patients you have personally seen for weight control, how
many are currently following the American Heart Association (AHA) exercise
program?
Don’t Ask People for Information They Simply Don’t Have (Answerable #3)
Improved Question:
Thinking of the last 10 patients you have personally saw for weight control,
for how many of these 10 patients did you recommend the American Heart
Association (AHA) exercise program, if any?
| 36
5. Make Questions Answerable
Q: Thinking of the past 12 months, how often have you baked a cake?
__ Less than once a month
__ 1 to 3 times a month
__ More than 3 times a month
Sarah:
I bake cakes when there is a birthday
party. We’ve had three in my family this
year, all in November. Does that mean
three times a month?
Jamie:
I bake one or two cakes every month
during the school year. I don’t bake any
during the summer. I do pies during the
holidays. I am not sure how to answer
the question.
Don’t assume regularity of behavior. (Answerable #2)
| 37
4. Make Questions Clear
5. Make Questions Answerable
Do you regularly read the newspaper?
__ Yes
__ No
Do you generally exercise in the morning?
__ Yes
__ No
Avoid Vague Modifiers
State the unit of measurement (Clear #1)
State a time frame in which people can recall the information you need.
(Answerable #2)
| 38
6. Make Questions Easy
Keep the question stem under twenty-five words. (Easy #1)
Limit the length of the questionnaire. (Easy #3)
Don’t ask for more detail than you really need. (Easy #4)
Don’t ask questions in the form of complex grids. (Easy #6)
| 39
7. Make Questions Unbiased
Q: Considering the extremely high price of gasoline, for your next vacation are you most
likely to: (choose one)
__ Fly __ Drive __ Take a train __ Take a bus __ Stay at home __Other
Jack:
Wow, they are right! Gas will be through the roof, so we will probably just stay
at home.
Do not introduce ideas or opinions that will influence responses. (Unbiased #1)
Improved Question
Q: For your next vacation, are you more likely to: (choose one)
__ Fly __ Drive __ Take a train __ Take a bus __ Stay at home __Other
| 40
7. Make Questions Unbiased
Q: Will you finish your project on budget?
To get sensitive information, consider disguising the question or
shifting the focus away from the respondent. (Unbiased #5)
Improved question(s):
Do you foresee any circumstances that might prevent you from finishing
your project on budget?
(IF YES)
Please describe the circumstances and provide suggestions about how
we might overcome them:
| 41
7. Make Questions Unbiased
Q: How would you rate your health?
__ Excellent
__ Very good
__ Good
__ Fair
__ Poor
Make sure bipolar scales are balanced. (Scaling #9)
Improved option A:
Q: How would you rate your health?
__ Very good
__ Good
__ Fair
__ Poor
__ Very Poor
| 42
Improved option B:
Q: How would you rate your health?
__ Excellent
__ Very good
__ Good
__ Fair
__ Poor
__ Very Poor
__ Terrible
7. Make Questions Unbiased
Q: Please indicate how much you agree or disagree with the following statements.
Strongly disagree 1 2 3 4 5 Strongly agree
a. Our company values its people……………………………………………………1 2 3 4 5
b. The company gives me the opportunity to train and develop..…...1 2 3 4 5
c. I get adequate feedback from my manager……………………..………….1 2 3 4 5
Replace agree/disagree scales with direct questions about what you really
want to measure. (Scales #10)
| 43
Scaling – The Scale Toolbox
1. What do you want to measure (e.g., likelihood, interest,
satisfaction, or something else)? Consider how people think about
the topic.
2. Should you use a unipolar or bipolar scale? Think about whether
the dimension you are scaling has a natural opposite.
3. What will be the length of the scale (e.g., 5 points, 7 points, or some
other length)? Think about how many levels people actually have in
their heads.
4. If you have a bipolar scale, should you provide a midpoint? Does it
make sense to force respondents toward one side of the scale or the
other?
5. Will you label only the endpoints or also all of the middle points
between the endpoints?
6. What will you name the endpoints (e.g., very satisfied and very
dissatisfied, or extremely satisfied and extremely dissatisfied)? If you
name the |middle
points, what will you name them?
44
8. Pretesting the Questionnaire
• Ask respondents to tell you in their own words what they
think each question is asking. This way you will find out if the
questions are working as intended.
• Ask respondents how they came up with their answers.
• Find out what comes to mind when respondents think about
certain terms and phrases
– “What does the term ‘family member’ mean to you?”
– “What does the term, ‘own’ mean to you?”
• Often we find that respondents have different definitions of
terms and phrases than we do.
| 45
8. Pretesting the Questionnaire
Q: Do you think young people who smoke cigarettes have more friends?
__ Definitely yes
__ Probably yes
__ Probably not
__ Definitely not
Problem with term, “young people”
Most respondents believe smoking does not affect number of friends.
They also thought that the answer choices “Probably not” and “Definitely not”
meant that young people who smoke have fewer friends.
So they thought they did not have an answer choice that reflected their views.
Ref: Gordon Willis, Cognitive Interviewing, 2005
| 46
8. Pretesting the Questionnaire
Revised Question
Q: Which of the following statements do you agree with the most for people
your age?
__ People my age who smoke cigarettes have more friends
__ People my age who do NOT smoke cigarettes have more friends
__ People my age who smoke cigarettes and those who do NOT smoke
cigarettes have the SAME NUMBER of friends
NOTE: Pretest this revised question to see if it needs to be revised
Ref: Gordon Willis, Cognitive Interviewing, 2005
| 47
8. Pretesting the Questionnaire:
A Few More Questions
Q: In your lifetime, how many bones have you broken?
“Well, three. I broke my arm in two places while skateboarding, and I
am pretty sure I broke the goalie’s nose in middle school soccer.”
| 48
Summary
• Writing a quality questionnaire
enormously challenging and needs to be
taken much more seriously
• There are resources that can help us
1.
2.
3.
4.
5.
6.
7.
8.
Eight Key Themes
Plan Research to Support Decisions
Conduct Qualitative Research
Write a Questionnaire Plan
Make Questions Clear
Make Questions Answerable
Make Questions Easy
Make Questions Unbiased
Pretest the Questionnaire
| 49
Summary
Plan Research To
Support DecisionMaking
Conduct Qualitative
Research
Plan the
Questionnaire
Make Questions
Clear
Make Questions
Answerable
Presenting
Lists
Make Questions
Easy
Presenting
Rating Scales
Pretesting the
Questionnaire
| 50
Make Questions
Unbiased
Asking Open-Ended
Questions
Contact Information
If you have any questions or ideas you want to share,
don’t hesitate to contact me.
email: DavidFHarris.com
website: www.davidfharris.com
phone: 919.451.0786
| 51