Survey
Project
Mario Sandri
www.mariosandri.it
Index
Survey Project
Part I ................................................................................................................................................................................................. 4
Part II ................................................................................................................................................................................................ 4
Part III ............................................................................................................................................................................................... 4
Part IV .............................................................................................................................................................................................. 4
Part V ............................................................................................................................................................................................... 4
Part VI .............................................................................................................................................................................................. 5
Survey Project Scoring Rubric ...................................................................................................................................................... 6
How to do a survey
Survey Says ... ............................................................................................................................................................................... 10
Four Steps ...................................................................................................................................................................................... 10
Step One: Create the Questions .............................................................................................................................................. 10
Step Two: Asking the Questions ................................................................................................................................................ 11
Step Three: Tally the Results ....................................................................................................................................................... 12
Step Four: Presenting the Results............................................................................................................................................... 12
Question survey
Questions ...................................................................................................................................................................................... 14
Types of Questions ....................................................................................................................................................................... 15
Question Sequence .................................................................................................................................................................... 16
Neutral Questions ........................................................................................................................................................................ 17
Possible Answers .......................................................................................................................................................................... 17
How Will I Gather the Answers? ................................................................................................................................................ 19
How Will I Show the Results? ...................................................................................................................................................... 19
Test It Out ...................................................................................................................................................................................... 19
Your Original Objective .............................................................................................................................................................. 19
Sampling Info
The Beauty of Sampling.............................................................................................................................................................. 20
The Margin of Error....................................................................................................................................................................... 20
Advantages of a Sample Survey over a Census ................................................................................................................... 20
Bias ................................................................................................................................................................................................. 20
pag. 2
Sampling Methods
Choosing a Simple Random Sample ....................................................................................................................................... 21
Stratified Random Sampling ...................................................................................................................................................... 21
Cluster Sampling .......................................................................................................................................................................... 21
Systematic Sampling ................................................................................................................................................................... 22
Multistage Sampling.................................................................................................................................................................... 22
Difficulties and Disasters in Sampling
Using the Wrong Sampling Frame ............................................................................................................................................ 23
Not Reaching the Individuals Selected ................................................................................................................................... 23
Nonresponse or Volunteer Response ....................................................................................................................................... 24
Disasters in Sampling ................................................................................................................................................................... 24
How to Ask Survey Questions
Possible Sources of Response Bias in Surveys .......................................................................................................................... 25
Deliberate Bias ............................................................................................................................................................................. 25
Unintentional Bias ........................................................................................................................................................................ 25
Desire to Please ........................................................................................................................................................................... 25
Asking the Uninformed ............................................................................................................................................................... 25
Unnecessary Complexity............................................................................................................................................................ 26
Ordering of Questions ................................................................................................................................................................. 26
Confidentiality and Anonymity ................................................................................................................................................. 26
Be Sure You Understand What Was Measured ...................................................................................................................... 26
pag. 3
Survey Project
Part I
Team members brainstorm possible survey topics on issues of interest
Part II
Each team submits a typed proposal describing:
•
Topic/question of interest
•
Background motivation for selecting topic/question
•
Questions to be included in the survey
•
Methodology

The type of sampling procedure you intend to use (i.e. stratified, cluster, simple random sample,
etc.)

Precise description of your randomization

When, where, and how you will administer the survey
Part III
Select your sample and administer your survey
Part IV
Organize, summarize, and analyze your data
Part V
Prepare a written report that documents your survey. Follow these guidelines:
Your written report should include each of the sections described below. The finished product will be evaluated
according to the attached rubric, so read it carefully.
•
Topic/Question: Should be descriptive and eye catching
•
Background: Why did you decide to investigate this topic/question? Why is it interesting and important?
•
Methodology: This should be clear enough so that anyone who reads your description could replicate the
survey effortlessly.
Describe and defend your chosen sampling procedure. Detail your randomization process. Carefully explain when,
where, and how you administered the survey. Provide a copy of your survey!
•
Data: Organize your data

Analysis: Include appropriate graphical and numerical summaries (i.e. bar graphs, pie charts,
etc.)
pag. 4

Interpretation: Discuss what the data tells you about the topic/question you chose.

What generalizations might you draw about the population from which the sample was drawn?
•
Pitfalls and extensions: Share any difficulties you experienced during the survey project. What might you
do differently if you were to repeat the survey? Are there any possible extensions of this survey project that
might prove interesting?
Part VI
Class Presentation: a 5 minute opportunity for you to share the critical aspects of your survey project with your
classmates! Make it interesting!! (See the scoring rubric.)
pag. 5
Methodology:
Randomization
Methodology: Sampling Procedure
Topic/Question and Background
Survey Project Scoring Rubric
pag. 6
4
The topic/question
selected is clearly
stated, is ofinterest
to the school
community, andis
appropriate. The
background
provided gives
strong motivation
for the team’s
choice of this
topic/question and
represents its
relevance
4
The chosen sample
procedure is
appropriate for
addressing the
selected
topic/question, is
described
accurately, and is
implemented
according to the
stated plan
4
The randomization
process includes a
clear description of
the selection
process and the
results of that
randomization (i.e.
the subjects
chosen). The
randomization
process matches
the chosen
sampling procedure
3
The
topic/question
selected is
clearly stated, is
of interest to
the school
community,
and is
appropriate.
The
background
provided gives
considerable
support for the
team’s choice
of this
topic/question
and some
attempt is
made to show
its relevance
3
The chosen
sample
procedure
satisfies two of
the three
criteria from (4),
but is weak in
the other area
3
The
randomization
process
matches the
chosen
sampling
procedure, but
clarity of
communication
would prevent
easy replication
of this
randomization
2
The
topic/question
is flawed in one
of the areas:
clearly stated,
of interest to
the school
community,
appropriate OR
the
background
provided fails in
either its
support for the
chosen
topic/question
or the
relevance
1
Both the
topic/question
and the
background
provided are
flawed in at
least one area.
However, one
or both
satisfactorily
address at
least half of the
areas specified
0
Neither the
topic/question
nor the
background
provided
satisfactorily
address at
least half of
the specified
areas
2
The chosen
sample
procedure
satisfies two of
the three
criteria from(4)
completely,
and does not
satisfy the third
requirement OR
the chosen
sample
procedure
satisfies one of
the three
criteria
completely
andthe other
twopartially
2
The
randomization
process does
not match the
chosen
sampling
procedure
1
The chosen
sample
procedure
satisfies one of
the three
criteriafrom (4)
completely,
and one of
theothertwo
criteria partially
0
The chosen
sample
procedure
does not
satisfy any of
the three
criteria from
(4) completely
1
There is some
flaw in the
randomization
procedure
itself
0
The
randomization
is flawed in all
areas or no
randomization
was used
Methodology: Administration of Survey
Methodology:
The Survey
Data Recording and
Summarization
pag. 7
4
The survey is
administered
according to the
stated plan. All
those selected in
the randomization
process actually
complete the survey
successfully. No
evidence of bias is
present
3
The survey is
administered
almost entirely
according to
the stated plan.
Nearly all of
those selected
in the
randomization
process
complete the
survey
successfully. No
evidence of
bias
4
Survey questions
have all been
pretested and
refined. They are
clear and unbiased
3
Survey
questions have
all been
pretested and
refined. They
are unbiased,
but somewhat
unclear
3
Original data
provided and
summarized in
appropriate
form, but with a
minor error or
sloppy
presentation
4
Original data
provided and
summarized in an
appropriate form.
Neat and accurate
2
The survey is
administered
almost entirely
according to
the stated plan,
and nearly all
of those
selected
actually
complete the
survey. Some
evidence of
bias is present
OR the survey
administered
deviates from
the stated plan
in some way
that does not
introduce bias,
but that might
impact who
completes the
survey
2
Survey
questions have
all been
pretested, but
not completely
refined or show
some bias
2
Either original
data is omitted,
but the data is
summarized
neatly and
accurately OR
the original
data is
presented, and
there is a major
flaw in the
presentation of
the data
1
Bias has
impacted the
survey
administration
to a great
extent, but the
stated plan
was generally
followed OR
the
administration
procedure
deviated
markedly from
the stated
plan, with
some bias
0
The
administration
process
deviates
markedly from
the stated
plan and bias
is noticeable
1
Survey
questions have
not been
pretested, but
are somewhat
clear and
relatively
unbiased
1
Original data is
provided, but is
not
appropriately
summarized
OR the original
data is omitted
but the data is
summarized
partially
correctly
0
Survey
questions are
unclear and
show distinct
bias
0
Original data
is not
provided and
the data is not
summarized
appropriately
3
The meanings
of the data
summaries are
interpreted.
Any
generalizations
that may be
drawn about
the population
from which the
sample was
drawn are
identified
2
The meanings
of the data
summaries are
interpreted but
some
errors/omissions
are made.
Generalizations
about the
target
population are
identified
Graphical and
Numerical
Summaries
4
The data is correctly
summarized using
bar graphs/pie
charts/etc. Graphs
and calculations
are neat and
accurate
Pitfalls and Extensions
4
All pitfalls
encountered are
articulated, with a
clear explanation of
how the group
dealt with each of
these obstacles. At
least one plausible
extension of the
survey project is
shared
3
The data is
correctly
summarized
using bar
graphs/pie
charts/etc.
Minor error in
either
computation or
graphing
3
All pitfalls
encountered
are articulated,
but explanation
of how group
dealt with each
of these
obstacles is
unclear. At
least one
plausible
extension of the
survey project is
shared
2
The data is
correctly
summarized
using bar
graphs/pie
charts/etc.
Major error in
either
computation or
graphing
2
Some pitfalls
encountered
are articulated,
with an
explanation of
how the group
dealt with each
of these
obstacles. At
least one
plausible
extension of the
survey project is
shared
3
Writing is
grammatically
correct, is
punctuated
properly, and
flows logically
from one point
to the next,
except for
minor flaws in
one of those
categories. No
spelling
mistakes
2
Significant
errors in one of
the areas:
grammar,
punctuation,
spelling, flow
OR minor errors
in two of the
areas
English Mechanics
Interpretation
4
The meanings of the
data summaries are
thoroughly and
accurately
interpreted. Any
generalizations that
may be drawn
about the
population from
which the sample
was drawn are
identified
pag. 8
4
Writing is
grammatically
correct, is
punctuated
properly, and flows
logically from one
point to the next. No
spelling mistakes
1
An attempt is
made to
interpret the
meanings of
the data
summaries are
but
errors/omissions
are made.
Generalizations
about the
target
population
might be
omitted
1
Appropriate
techniques are
used to
summarize the
data but has
not executed
the techniques
correctly
0
The
interpretation
is inadequate
1
Some pitfalls
encountered
are
articulated,
but no
explanation of
how the group
dealt with
each of these
obstacles or
does not share
at least one
plausible
extension of
the survey
project
1
Writing is
flawed in two
or three of the
areas:
grammar,
punctuation,
spelling, flow
0
The obstacles
encountered
are not
articulated
0
Neither the
graphical nor
the numerical
summary is
appropriate
0
Writing is
deficient in all
four areas:
grammar,
punctuation,
spelling, flow
Oral Presentation: Context
Oral
Presentation:
Delivery
pag. 9
6
The group
accurately presents
all key aspects of its
survey project,
including
topic/question,
background,
methodology, data
summary,
graphical/numerical
analysis,
interpretation, and
possible
pitfalls/extensions.
Correct terminology
is used throughout
and all members
participate
6
The presentation is
clear, wellorganized
and fluent. Visual
aids are used to
assist the audience
in understanding
important points
4
Minor errors or
omissions are
made in
discussing the
key aspects of
the survey
project OR the
group fails to
use
appropriate
terminology
throughout OR
imbalanced
contributions
2
Major error in
content OR
complete
failure to use
appropriate
terminology
throughout OR
a one person
show
0
A complete
disaster Flaws
in content and
terminology
4
The
presentation is
generally clear,
well organized,
and fluent.
Visual aids are
not used
adequately
2
The delivery is
not especially
clear, well
organized, or
fluent and/or
visual aids are
lackin
0
Delivery
destroyed
content
How to do a survey
Survey Says ...
Turn on the television, radio or open a newspaper and you will often see the results from a survey.
•
Gathering information is an important way to help people make decisions about topics of interest.
•
Surveys can help decide what needs changing, where money should be spent, what products to
purchase, what problems there might be, or lots of other questions you may have at any time.
•
The best part about surveys is that they can be used to answer any question about any topic.
You can survey people (through questionnaires, opinion polls, etc) or things (like pollution levels in a river, or traffic
flow).
Four Steps
Here are four steps to a successful survey:
•
Step one: create the questions
•
Step two: ask the questions
•
Step three: tally the results
•
Step four: present the results
Let us look at those steps in more detail ...
Step One: Create the Questions
The first thing is to decide is What questions do you want answered?
Sometimes these may be simple questions like:
•
"What is your favorite color?"
Other times the questions may be quite complex such as:
•
Which roads have the worst traffic conditions
Simple Surveys
If you are doing a simple survey, you could use tally marks to
represent each person’s answer:
Sometimes, it is helpful to be creative in how the people can
respond. It makes it more fun for both you and your
respondents (the people answering the question).
pag. 10
Example:What is your favorite color?
Have them write down their favorite color on a
piece of paper and drop it in a fish bowl.
Then, put all of the pieces of paper into piles and
count them.
Step Two: Asking the Questions
Now you have your questions, go out and ask them! But who to ask?
•
If you survey a small group you can ask everybody (called a Census)
•
If you want to survey a large group, you may not be able to ask everybody so you should ask a sample
of the population (called a Sample)
If your are Sampling you should be careful who you ask.
To be a good sample, each person should be chosen randomly
•
If you only ask people who look friendly, you will only know what friendly people think!
•
If you went to the swimming pool and asked people "Can you swim?" you will get a biased answer ...
maybe even 100% will say "Yes"
Note: the surveys where people are asked to ring a number to vote are not very accurate, because only certain
types of people actually ring up!
So be careful not to bias your survey. Try to choose randomly.
Example: You want to know the favorite colors for
people at your school, but don't have the time to
ask everyone.
Solution: Choose 50 people at random:
- stand at the gate and choose "the next person to
arrive" each time
- or choose people randomly from a list and then
go and find them!
- or you could choose every 5th person
Your results will hopefully be nearly as good as if
you asked everyone.
pag. 11
If you choose a person and they do not want to answer, just record "no answer" on the survey form and mention
how many people did not answer in your report.
After completing a sampling survey you can use the information to make a prediction as to how the rest of the
population would respond.
The more people you have asked, the better your result will be.
Example: nationwide opinion polls survey up to
2,000 people, and the results are nearly as good
(within about 1%) as asking everyone.
Step Three: Tally the Results
Now you have finished asking questions it is time to tally the results.
By "tally" I mean add up. This usually involves lots of paperwork and computer work (spreadsheets are useful!)
Example: For "favorite
colors of my class" you
can simply write tally
marks like this (every fifth
mark crosses the previous
4 marks, so you can
easily see groups of 5)
Step Four: Presenting the Results
Now you have your results, you will want to show them to other people in the best possible way.
We have written a special page called Showing the Results of a Survey, but here is a quick summary:
Tables
Sometimes, you can simply report the information in a table.
A table is a very simple way to show others the results. A table should have a title, so those looking at it
understand what results the table shows:
Yellow
4
pag. 12
Red
5
Table: The Favorite Colors of My Class
Blue
Green
6
1
Pink
4
Statistics
You can also summarize the results using statistics, such as mean or standard deviation
Example: you have lots of information about how
long it takes people to get to school but it may be
simpler just to present a summary such as:
- Shortest Journey: 3 minutes
- Average Journey: 22 minutes
- Longest Journey: 58 minutes
Graphs
But nothing makes a report look better than a nice graph or chart.
Use Data Graphs (Bar, Line and Pie) to make them.
Example Survey
Question: What
is your favorite
color?
pag. 13
Question survey
How to make a good Questionnaire!
The first question is one you should ask yourself:
"What do I hope to learn from asking the questions?"
This defines your objective (the purpose, or why you are conducting the survey).
Example: you want to clean up the local river. You
feel that with some help and some money you
could make it really beautiful again. You want to
survey your local community to find out:
- Are other people also worried about the river.
- Would they be willing to donate their time or
money to help.
Questions
Now you know why you are doing a survey, start writing down the questions you will ask!
Just write down any questions you think may be useful. Don't worry about quality at this stage, we will improve
your list of questions later.
Example: Questions you could ask for the river
survey:
- Does pollution worry you?
- Do you ever go down to the river?
- Can you spare some money to help the river?
- Have you noticed the pollution in the river?
- Would you be happy to volunteer for river
cleanup?
- When would you be available to help?
pag. 14
- How should we clean up the river?
- etc...
You can also ask the person about themselves (not too personal!), such as approximate age, male or female,
etc, so that you know the kind of people that you have been surveying.
Types of Questions
A survey question can be:
•
Open-ended (the person can answer in any way they want), or
•
Closed-ended (the person chooses from one of several options)
Closed ended questions are much easier to total up later on, but may stop people giving an answer they really
want.
Example: "What is your favorite color?"
- Open-ended: Someone may answer "dark
fuchsia", in which case you will need to have a
category "dark fuchsia" in your results.
- Closed-ended: With a choice of only 12 colors
your work will be easier, but they may not be able
to pick their exact favorite color.
Example: "What do you think is the best way to
clean up the river?"
Make it Open-ended: the answers won't be easy
to put in a table or graph, but you may get some
good ideas, and there may be some good quotes
for your report.
Example: "How often do you visit the river?"
Make it Closed-ended with the following options:
- Nearly every day
- At least 5 times a year
pag. 15
- 1 to 4 times a year
- Almost never
You will be able to present this data in a neat bar
graph.
Question Sequence
It is important that the questions don't "lead" people to the answer
Example: people may say "yes" to donate money
if you ask the questions this way
- Do you love nature?
- Will you donate money to help the river?
But probably will say "no" if you ask the questions
this way:
- Is lack of money a problem for you?
- Will you donate money to help the river?
To avoid this kind of thing, try to have your questions go:
•
from the least sensitive to the most sensitive
•
from the more general to the more specific
•
from questions about facts to questions about opinions
Example: I will ask people how often they visit the
river (a fact) before I ask them what they feel
about pollution (an opinion)
I will ask people their general feelings about the
environment before I ask them their feelings about
the river.
pag. 16
Neutral Questions
Your questions should also be neutral ... allowing the person to think their own thoughts about the question.
In the example above we had the question "Do you love nature?" ... that is a bad question because it is almost
forcing the person to say "Yes, of course."
Try rewording it to be more neutral, for example:
Example: "How important is the natural
environment to you?"
- Not Important
- Some Importance
- Very Important
But you can also make statements and see if people agree.
Possible Answers
For each "closed-ended" question try to think:
What are the possible answers to this question?
Make sure you have most of the common answer available.
If you are not sure what people might answer, you could always try a small open ended survey (maybe ask your
friends or people in the street) to find common answers.
Trick: try to avoid neutral answers (such as "don't care") because people may choose them so they don't have to
think about the answer!
It is also helpful to have an “other” category in case none of the choices are satisfactory for the person answering
the question.
Example: What is your favorite color?
Red, blue, green, yellow, purple, black, brown,
orange, other
pag. 17
. Scaled Answers
Sometimes you could have a scale on which they can rate their feelings about the question.
Have "opposite" words at either end and a scale in between like this:
Examples:
The river is ...
Polluted :_____:_____:_____:_____:_____:_____: Clean
Cleaning up the river is ...
Easy :_____:_____:_____:_____:_____:_____: Difficult
. Rated Items
For this type of answer the person gets to rate or rank each option.
Don't have too many items though, as that makes it too hard to answer.
Example: Please rank the following activities from 1
to 5, putting 1 next to your favorite through to 5 for
your least favorite.
___ Fishing
___ Football
___ Golf
___ Shopping
___ Sleeping
. Number Answers
You can also just ask for a number
Example: "How many times did you visit the river
during the past year?"
____ times
pag. 18
How Will I Gather the Answers?
Try to make life easier by thinking how you will gather the answers before you ask the questions
It is important to make the process simple, for both yourself and those responding.
The Questionnaire
You are going to want a neat form that makes it easy to answer the questions AND easy to total up the answers
later on.
How Will I Show the Results?
Go over each of the questions and think how you want the answers to go into your report:
•
in a table,
•
a bar graph,
•
a pie chart,
•
or just explained in words.
Example: you decide to have six options for "How
many times do you visit the river" so the bar graph
looks best.
Test It Out
You should test your questionnaire on a few people.
•
was each question clear and easy to understand?
•
were they happy with the options?
It is also a good idea to time how long it takes so you will be able to tell people "this survey only takes 2 minutes"
(or however long it takes).
Your Original Objective
Lastly, look back at your original objectives for this survey ...
•
will the questions really help you find out what you want to know?
•
are there some questions you can remove? (smaller surveys are easier!)
pag. 19
Sampling Info
The Beauty of Sampling
In a sample survey, a subgroup of a large population is questioned on a set of topics. The results from a sample
survey are used as if they are representative of the larger population, which they will be if the sample is chosen
correctly and if those selected cooperate in responding. It also takes less time to conduct a sample survey than it
does to conduct a census, in which everyone in the population is measured.
The Margin of Error
The Accuracy of Sample Surveys Sample surveys often are used to estimate the proportion or percentage of people
who have a certain trait or opinion. This measure of accuracy in sample surveys is a number called the margin or
error.
Advantages of a Sample Survey over a Census
•
When a Census Isn't Possible
•
Speed
•
Accuracy
Bias
How Surveys Can Go Wrong Not all surveys produce trustworthy results. While it is unlikely that the sample value will
equal the population value precisely, the goal of a good survey is to get an unbiased sample value. Results based
on a survey are biased if the method used to obtain those results would consistently produce values that are either
too high or too low.
There are three common types of bias that might occur in surveys:
•
Selection bias occurs if the method for selecting the participants produces a sample that does not
represent the population of interest.
•
Nonresponse bias occurs when a representative of the sample is chosen for a survey, but a subset cannot
be contacted or does not respond.
•
Response bias occurs when participants respond differently from how they truly feel. The way questions
are worded, the way the interviewer behaves, as well as many other factors might lead an individual to
provide false information.
pag. 20
Sampling Methods
The ability of a relatively small sample to accurately reflect a huge population does not happen haphazardly. It
only happens if proper sampling methods are used. The basic idea is that everyone in the population must have a
specified chance of making it into the sample, and methods with this characteristic are called probability sampling
plans. The most basic method to accomplish this goal is to use a simple random sample.
Choosing a Simple Random Sample
To actually produce a simple random sample, you need only two things.
•
First, you need a list of the units in the population
•
Second, you need a source of random numbers.
For large populations it may not be practical to take a simple random sample because it may be difficult to get a
numbered list of the units.
Stratified Random Sampling
Sometimes the population of units falls into natural groups, called strata. A stratified random sample is collected by
first dividing the population of units into groups (strata) and then taking a simple random sample from each one.
You can often recognize this type of sampling when you read the results of a survey because the results will be
listed separately for each of the strata.
Stratified sampling has other advantages in addition to the fact that results are available separately by strata.
•
One is that different interviewers may work best with different people.
•
It also may be more convenient to stratify before sampling.
The accuracy with which we can estimate the average depends on the natural variability among the
measurements. The less variable they are, the more precisely we can assess the population average on the basis
of the sample values.
Cluster Sampling
Cluster sampling is often confused with stratified sampling, but actually it is a radically different concept and can
be much easier to accomplish. The population units are again divided into groups, called clusters, but rather than
sampling within each group, we select a random sample of clusters and measure only those clusters.
In most applications of stratified sampling, the population is divided into a few large strata, and a small subset within
each one is randomly sampled. In most applications of cluster sampling, the population is divided into small clusters,
a large number of clusters are randomly sampled, and everyone in those clusters is measured.
One obvious advantage of cluster sampling is that you need only a list of clusters instead of a list of all individual
units.
If cluster sampling is used, the analysis must proceed differently because there may be similarities among the
members of the clusters that must be taken into account.
pag. 21
Systematic Sampling
Suppose you have a list of 5000 names and telephone numbers from which you want to select a sample of 100
individuals. That means you would want to select 1 out of every 50 people on the list. You would not want to simply
choose the first 100 names because, depending on how the list was ordered, they might not constitute a
representative sample.
An idea that would work in most cases is to choose every 50th name on the list. If you did so, you would be using a
systematic sampling plan. With this plan, you divide the list into as many consecutive segments as you need,
randomly choose a starting point in the first segment, then sample at that same point in each segment.
In our example, you would randomly choose a starting point in the first 50 names, and then sample every 50th
name after that. When you were finished, you would have selected one person from each of the 100 segments,
equally spaced throughout the list.
Systematic sampling is often a good alternative to random sampling. In a few instances it can lead to a biased
sample, and common sense must be used to avoid those.
Multistage Sampling
Many large surveys use a combination of the methods we have discussed. The survey designers might stratify the
population by region of the country, then stratify by urban, suburban, or rural, and then choose a random sample
of communities within those strata. They would then divide those communities into city blocks or fixed areas, as
clusters, and sample some of those. Everyone on the block or within the fixed area may then be sampled. This is
called a multistage sampling plan.
pag. 22
Difficulties and Disasters in
Sampling
Difficulties that can occur in practice need to be considered when you evaluate a study. If a proper sampling plan
is never implemented, this can lead to very misleading and inaccurate conclusions. Some problems can occur
even when a sampling plan has been well designed. Here is a list of possible problems:
•
Using the wrong sampling frame
•
Not reaching the individuals selected
•
Nonresponse or volunteer response
•
Self-selected sample
•
Convenience or haphazard sample
Using the Wrong Sampling Frame
The sampling frame is the list of units from which the sample is selected. This list may or may not be the same as the
list of all units in the desired "target" population. Sometimes a sampling frame either will include unwanted units or
exclude desired units. For example, if we sample from a list of registered voters in order to predict election outcomes,
we will include individuals who are not likely to vote as well as those who are likely to do so. Using a telephone
directory to survey the general population excludes those who move often, those with unlisted phone numbers (like
many physicians and teachers), and those who do not have a telephone. Using the wrong sampling frame is one
way to create selection bias.
The sampling frame doesn't cover the target population, leading to selection bias
target population = all U.S. adults
↓
no telephone or unlisted
↓
never sampled
↓
sampling frame = telephone white pages
↓
random sample of n adults
Not Reaching the Individuals Selected
Even if a proper sample of units is selected, the desired units may not be reached.
It is very important once a sample has been selected that those individuals are the ones who are actually measured.
It is better to put resources into getting a smaller sample than to get one that has been biased by moving on to the
next person on the list when someone is initially unavailable. Failing to contact or measure the individuals who were
selected in the sampling plan leads to nonresponse bias.
pag. 23
Nonresponse or Volunteer Response
Even the best surveys are not able to contact everyone on the list, and not everyone contacted will respond.
Remember that the lower the response rate, the less the results can be generalized to the population as a whole.
Responding to a survey (or not) is voluntary, and those who respond are likely to have stronger opinions than those
who do not respond. As mentioned earlier, this type of nonresponse bias can lead to systematically over- or
underestimating the truth about a population.
Disasters in Sampling
Basing a sample survey on a self-selected sample (also called a volunteer sample or a convenience sample) is
usually so problematic that the results cannot be extended to anyone beyond the sample. This warning applies
only to sample surveys and to some observational studies.
•
Self-Selected Sample
Those who have a strong opinion about the question are more likely to respond than those who do not
have a strong opinion.
•
Convenience or Haphazard Sample
Another worthless sampling technique for surveys is to use the most convenient group available or to
decide haphazardly on the spot who to sample. Rarely do the responses from a convenience sample or
haphazard sample represent any larger population for the question of interest.
You have seen the proper way to collect a sample and have been warned about the many difficulties and dangers
inherent in the process.
pag. 24
How to Ask Survey
Questions
You may be surprised at how much the answers to questions can change based on simple changes in wording.
The wording and presentation of questions can significantly influence the results of a survey.
Possible Sources of Response Bias in Surveys
Many pitfalls can be encountered when asking questions in a survey or experiment. Here are some of them:
•
Deliberate bias
•
Unintentional bias
•
Desire to please
•
Asking the uninformed
•
Unnecessary complexity
•
Ordering of questions
•
Confidentiality and anonymity
Deliberate Bias
Sometimes, if a survey is being conducted to support a certain cause, questions are deliberately worded in a biased
manner. Be careful about survey questions that begin with phrases like "Do you agree that...?" Most people want
to be agreeable and will be inclined to answer "yes" unless they have strong feelings the other way.
Unintentional Bias
Sometimes questions are worded in such a way that the meaning is misinterpreted by a larger percentage of the
respondents.
If you were to ask people to recall the most important date in their life, you would need to clarify if you meant the
most important calendar date or the most important social engagement with a potential partner. (It is unlikely that
anyone would mistake the question as being about the shriveled fruit, but you can see that the same word can
have multiple meanings.)
Desire to Please
Most survey respondents have a desire to please the person who is asking the question.
They tend to understate their responses about undesirable social habits and opinions, and vice versa.
Asking the Uninformed
People do not like to admit that they don't know what you are talking about when you ask them a question.
pag. 25
Political pollsters, who are interested in surveying only those who will actually vote, learned long ago that it is useless
to simply ask people if they plan to vote. Most of them will say "yes." Instead, they ask questions that establish a
history of voting, such as "Where did you go to vote in the last election?"
Unnecessary Complexity
If questions are to be understood, they must be kept simple. A question like "Shouldn't former drug dealers not be
allowed to work in hospitals after they are released from prison?" is sure to lead to confusion. Does a "yes" answer
mean they should or should not be allowed to work in hospitals? It would take a few readings to figure that out.
Another way in which a question can be unnecessarily complex is by actually asking more than one question at
once. An example would be a question like "Do you support the President's Health Care Plan, since it would ensure
that all Americans receive health coverage?" If you agree with the idea that all Americans should receive health
coverage but disagree with the remainder of the plan, do you answer "yes" or "no"? Or what if you support the
President's plan, but not for that reason?
Ordering of Questions
If one question requires respondents to think about something that they may not have otherwise considered, then
the order in which questions are presented can change the results.
Confidentiality and Anonymity
People will often answer questions differently based on the degree to which they believe they are anonymous.
Because researchers often need to perform follow-up surveys, it is easier to try to ensure confidentiality that
anonymity.
In ensuring confidentiality, the researcher promises not to release identifying information about respondents. In an
anonymous survey, the researcher does not know the identity of the respondents.
Be Sure You Understand What Was Measured
Sometimes words mean different things to different people. When you read about survey results, you should get a
precise definition of what was actually asked or measured. Sometimes even common terminology may mean
different things to different people.
pag. 26