"Be-ing" a Quality Survey Researcher
Online survey software has opened the door to more research and data use and more users who want more timely and
accurate information than any other innovation since the advent of the computer. Online survey software and data collection is now the default mode of survey research. As users are empowered to conduct their own research, survey research
quality is more important than ever. Qualtrics strives to be at the forefront of online survey innovation both in feature
innovation and quality. Increasing question-building skills is perhaps the lowest hanging of the fruit that can significantly
reduce measurement error and increase the validity of research results.
As we have worked with tens of thousands of researchers, Qualtricians have found that five simple “Be’s” help refine
the art of questionnaire building and avoid the common pitfalls we often see in “bad questionnaires.” Quality in survey
research is a state of “Be-ing” that helps you create great questionnaires that greatly reduce “measurement error”.
1. BE WELL THOUGHT OUT
Respondents love context: avoid placing survey questions out of order or out of context. In general, a funnel approach is advised. Place broad and general questions at the beginning of the questionnaire as a warm-up,
followed by more specific questions, followed by more general easy to answer questions (like demographics) at the end of the questionnaire. All should follow the flow of a normal thought process.
2. BE UNBIASED
Slight wording changes can produce great differences in survey results. “Could”, “Should”, and “Might” all sound almost the same, but may produce a 20% difference in agreement to a question: DNA modification of hu
man food supplies could.. should.. might.. be allowed in the United States.
Strong words that represent control or action, (such as “prohibit”) produce similar differences in survey results: “Do you believe that congress should prohibit gasoline refiners from raising prices?”
In speech, intonation could make the following a positive statement: “You wouldn’t want to go to Rudolpho’s
Restaurant for the company’s annual party would you?” However, in written form a negative bias would result.
3. BE SPECIFIC IN WHAT YOU ASK
Unclear survey questions produce answers that lack meaning. The question “How much do you like orange juice?” produces a less than meaningful answer. The respondent answers the question using a numeric scale, but what are they liking?... the taste, texture, nutritional content, vitamin C, the current price, concentrated, or fresh squeezed?
"Be-ing" a Quality Survey Researcher | 1
Questions must be specific. Though intended as a question about taste, the question “What suggestions do you have for improving energy drinks?” may produce suggestions about texture, the type of can or bottle, mixing juices, additives, or something related to use as a mixer or in recipes.
Even the question “Do you watch TV regularly?” begs for a definition of “regularly”. Detail in asking questions produces detailed understanding.
4. BE SINGULAR IN FOCUS
Avoid Double Barreled Questions. The question “What is the fastest and most economical Internet service for you?” is problematic because the fastest is certainly not the most economical. Two questions should not be asked in one question.
Avoid Overly Compressed Answers. Make sure answers choices are independent and cover the landscape of
possible answers. For example the question “Do you think of basketball players as being independent agents or as employees of their team?” is incorrect because some believe that basketball players are both.
Response categories for multiple choice survey questions should be mutually exclusive so that clear choices can be made. Likewise, questions that do not provide all acceptable or meaningful answers will frustrate the
respondent and make interpretation of results difficult at best. If you are unsure, conduct a pretest using the “Other (please specify)” option. Then revise the question making sure that you cover at least 90% of the
respondent answers.
5. BE CONSIDERATE
Avoid intrusive questions. Do not ask for information that respondents may not have access to, remember, or will not provide. Privacy is an increasingly important issue in survey research. Questions about finances, income,
occupation, family life, personal hygiene and beliefs (personal, political, religious) can be too intrusive and rejected by the respondent. Strong justification or sponsorship is helpful in securing response cooperation.
Avoid techno-babble. Caloric content, bits, bytes, MBS, and other industry specific jargon and acronyms produce confusion. Your audience must understand your language and terminology in the question being asked.
Avoid impossible questions on future intentions. Yogi Berra (famous New York Yankees baseball player) once said that making predictions is difficult, especially when they are about the future. Predictions are rarely accurate more than a few weeks or in some case months ahead.
Avoid long questions. Multiple choice questions are the longest and most complex to write, but easiest to answer. Free text answers are the shortest and easiest to answer, but most difficult to
decipher. Questionnaire building is a series of trade-offs, so remember that increasing the length of questions and surveys decreases the chance of receiving a completed response.
"Be-ing" a Quality Survey Researcher | 2
These very important rules open the door to quality interactions with customers, employees, and valued partners. Qualtrics supports quality research through advanced survey software that is “easy enough for an intern, but sophisticated
enough for a Ph.D.”
We’re Here to Help!
Qualtrics.com provides the most advanced online survey building, data collection (via panels or corporate / personal contacts), real-time view of survey
results, and advanced “dashboard reporting tools”.
If you are interested in learning more about how the Qualtrics professional
services team can help you with a conjoint analysis research project, contact
us at [email protected].
"Be-ing" a Quality Survey Researcher | 3