ONLINE TECHNOLOGY FOR RESEARCH AND INSIGHT
BRIEFING:
Avoiding Bias in Online Survey Scripting
Beware of ambiguity when wording questions in your online surveys. Our Insight
expert, Gina Lee-Young, offers these tips on avoiding bias and misinterpretation.
Keep it objective and avoid leading questions
When designing a survey, correct wording of your questions is crucial to ensure that no bias is
present. A well written survey will ask objective questions rather than leading ones. It is all too easy
to presume you know how respondents are feeling based on your own views, but the key is to
phrase questions in an objective manner and not second guess what their answers may be.
Beware of leading respondents in a certain direction with your question wording. An example
would be: Do you strongly oppose the changes that have been implemented? You are assuming that
many respondents feel strongly about the changes when this may not be the case. The question
could be reworded to: How do you feel about the changes that have been implemented? The
answers could then be a series of options ranging from strongly oppose to strongly support.
A more subtle example would be: Do you suffer from asthma? This assumes that anyone with
asthma is “suffering”. This may not be the case and some respondents who have asthma may tick
“no” because they don’t feel they are suffering with it. It would be better to reword the question to
simply read: Do you have asthma? This rewording ensures that the question can’t be misinterpreted.
Consider all possibilities and rotate answer lists
There are other basic things to think about to ensure you’re getting unbiased responses. Primary
among these is to ensure you give respondents a range of answer options which include all
possibilities. Take this question for example:
What do you think of our new website design?
 I love it
 I like it
 It’s ok
In this example you are not allowing for any negative feedback. Respondents are forced to give one
of three positive answers which may not be what they really think. So, to ensure you get genuine
feedback, whether good or bad, you would need to add in further options, such as:
 I don’t like it
 I hate it
You might also consider rotating answer options in single or multiple choice questions. Online
surveys make this easy, as each iteration of the survey served can re-order the answers in a list. This
will eliminate any bias to the first answer options – some respondents may subconsciously be drawn
to the first options they see, and may be more likely to tick the ones nearer the top. Rotating answer
options will eliminate any bias to the top answers.
www.demographix.com
[email protected]
+44 (0)20 7112 5111
Page 1 of 2
ONLINE TECHNOLOGY FOR RESEARCH AND INSIGHT
Avoid combining questions, jargon and absolutes
A top tip is to watch out for combining several questions into one. You should avoid convoluted
questions that should really be separated into distinct components. An example of this might be:
Do you think brand x has the best quality product and the best price? This type of question can easily
frustrate respondents and you are not giving them the option to answer the question accurately if
for example they think brand x has the best quality product but not the best price. So make sure you
split this into two questions:
 How do you rate Brand X on quality?
 How do you rate Brand X on price?
Don’t assume that all respondents are as literate as yourself. Always try to avoid using jargon,
abbreviations or technical terms unless you are 100% sure that all of your audience will understand
them. If there is a chance that some of your respondents may not understand the terms used in a
question, clarify what you mean to avoid any potential misinterpretation.
Where possible, avoid absolute words - such as “every”, “always”, and “all” - as this may make it
harder for people to answer truthfully. For example: Do you always catch the train to work? allows
little scope for variation in a respondent’s routine - someone may catch the train most days but cycle
to work once a month or when the sun is out. A much better alternative would be: Which mode of
transport do you use most often to get to work?
Don’t be vague, define what you mean, and be unambiguous
The flipside of the coin is to avoid vague words - such as “regularly” or “occasionally” - wherever
possible. The question Do you regularly watch the evening news? will mean different things to
different people. Once a day could be regular to some; once a week to others. If you do want to use
a word such as regularly, it would be prudent to define what you mean - e.g. “By regularly we mean
at least 4 nights a week”. Better still, rephrase the question to read How often do you watch the
news? And give frequency options for them to choose from.
Questions should always be easy to understand and not flummox respondents. To this end, it is
advisable not to use double-negative questions/statements as they can be confusing. For example:
Do you agree that Company X should not stop selling popcorn? Y/N. This statement can easily be
answered incorrectly as the respondent has to think about what it means first. This is better: Should
Company X stop selling popcorn? Y/N.
Where ambiguity exists or you are asking leading questions, your results are likely to suffer as
questions will be answered incorrectly or subjectively. You also run the risk of frustrating
respondents which may lead to a high drop-out rate. So keep your questions simple and easy to
understand and always remember that your results are only as good as the questions you ask. And
remember the best way to ensure your surveys make sense to most people is to circulate them to a
small group of colleagues in advance for them to take a look, and give you feedback!
If you want any further information on any of the scripting tips mentioned in this briefing,
please send us an email at [email protected].
www.demographix.com
[email protected]
+44 (0)20 7112 5111
Page 2 of 2