Intention
Crash Course in Managing
Big and Small Surveys
•
•
•
•
•
•
These slides are meant to be an introduction
And an overview of practicalities
And, self explanatory to a large extent
Orginally for field work far away from home
And, also for big survyes with hired staff.
No specific litterature is offered here. Search
the web, the library etc.
• There is a lot out there !
Planning and Operation
of Fieldwork
Interview and/or Questionnarie Based
Linköping University Feb 9, 2011.
Interviews and Questionnares 2011
1
•
•
•
•
•
Observation
Surveys
Secondary data
Qualitative research
Can employ one or more of these in any single
study.
• Each has distinct approaches
• Some triangulation may be necessary – use
more than one method of data collection to
test the same hypothesis.
• Surveys today, in the developed world,
can also be done through the internet.
• The responded is asked in some way to
fill in a questionnare on the web, instead
of doing on paper.
• It os possible to buy ready made
solutions for this.
3
Interviews and Questionnares 2011
Observation Methods
4
Survey Research
• Participant observation – a form of qualitative
research
• Used in informal settings
• Decide what is to be observed – have a
checklist
• Record what you see/hear without influencing
• Must be all eyes/ears
• Usually no talking to subjects
• Disadvantage: can miss reasons for certain
actions/occurrences
Interviews and Questionnares 2011
2
Data Collection Methods
Not Mentioned Today
Interviews and Questionnares 2011
Interviews and Questionnares 2011
• Types of information sought; Behavioural; Personal;
Social/Economic; Other
• Must decide on
• Method of data collection that will be used:
–
–
–
–
–
–
5
Personal interviews,
Telephone interviews
Self administrated (postal or hand delivered)
Procedures to be followed in approaching the subjects
Order/sequence of questions
Type of questions: Direct vs. Indirect; structured vs. Nonstructured questions
Interviews and Questionnares 2011
6
1
Mail Questionnaire
Field Planning and Operations - yes
• An Impersonal Survey Method
• Advantages
• SURVEY MANAGEMENT: Essential for
achieving speed and quality of data.
• AIM:
AIM: To speed up the execution of field work,
and ensuring quality of output.
• People to coordinate:
– Low cost compared to other methods
– Reduced biasing because respondents are not influenced be the
researcher
– Provides a high degree of anonymity to respondents
– Respondents have time to think about their responses
– Can achieve greater geographic spread at low cost
– Field coordinator/Survey Manager
– Data Analyst/Computer specialist
– Field Officers, Research Assistants & Data
enumerators
The success/failure depends on the skills of these
people, in particular on the SURVEY MANAGER –
usually yourself.
• Disadvantages
–
–
–
–
Requires simple easy to understand questions/instructions
No opportunity for probing to clarify answers.
Researchers have no control over who fills in the questionnaire
Response rates tend to be (very) low
Interviews and Questionnares 2011
7
THE SURVEY MANAGER EXCEUTES VARIOUS
CRITICAL ROLES:
Develops the questionnaire & Instructions
Conducts/Evaluates pilot study
Adjusts/Revises the survey questionnaire
Recruits/Trains field staff
Designs the sample and estimating procedures
Determines data collection procedures
• If things go wrong in the areas above,
the survey results will be subject to
errors
• Basically two types of errors:
Sampling Error – You probably did not spend
enough time on constructing a sampling
method
Non-sampling Error – Probably sloppy
planning
9
Sampling Error
Interviews and Questionnares 2011
10
Non-sampling errors
• Check if the sampling method worked.
• Is the actual sample displaying some
known the characteristics of the
population.
• 50/50 Man/women, age structure,
employed, unemployed, etc?
• If not matching? Compare answers.
• Resampling? Pick out groups?
Interviews and Questionnares 2011
8
TYPES OF ERRORS
THE SURVEY MANAGER
Interviews and Questionnares 2011
Interviews and Questionnares 2011
All the other errors – bad management
and supervision.
Wrong timing
Badly trained interviews
Etc.
Jeopardize the study.
Is the sample ok? Is the survey ok?
11
Interviews and Questionnares 2011
12
2
Interviews
• Personal interviews:
Interview Based Data
Collection
– Schedule-Structured
– The focused
– The Non-directive
Organising the Interviews
Different types of
Interviews and Questionnares 2011
• Questionnares – how to for interviewer
or mail, internet etc.
13
• A fact to face interpersonal role situation in
which the interviewer ask the respondent
questions to elicit answers pertinent to the
research hypothesis.
• The questions, their wording and sequence
define the structure of the interview are
important.
• Person interviews – test interviews!
• Three types of personal interviews:
It is the least flexible
The number of questions, sequencing and
wording is identical for all respondents
The interviewer should not reword questions or
provide explanations, even if asked to do so
Variations between responses are then not
coming from variations in the interview.
15
2) The Focused Interview
Interviews and Questionnares 2011
16
3) The Non-directive Interview
Interviewer is familiar with the ”experience”
I.e. often referring to situations analysed prior to the
interview
Proceeds on the basis of an interview guide specifying
topics related to the research
It seeks the respondent’s own experience regarding
the issues under study
Respondents are given considerable liberty regarding
their definition of the situations presented to them
Also, permits the researcher to obtain details of
personal reactions, interpretations etc.
The interviewer, knowing the ’experience’ before is
sensitive to inconsistencies and omissions of data that
may be needed for clarification!
Interviews and Questionnares 2011
14
1) The Scheduled-structured
interview
The Personal Interview
Interviews and Questionnares 2011
Interviews and Questionnares 2011
The most flexible form of interviewing
The researcher
Respondents are allowed to relate their
experiences freely, etc.
The interviewer has hugechoice of probe
various aspects and to raise queries
during the interview
17
Interviews and Questionnares 2011
18
3
Principles of Interviewing ...
Personal Interviews +/Advantages
• Flexibility
• Control over interview situation
• High response can be achieved
• Collation of supplementary data possible
Disadvantages
• Higher costs
• Interviewer bias
• Lack of anonymity
Interviews and Questionnares 2011
First step is getting the respondent to cooperate
and provide the desired information.
– Make the respondent feel that their interaction
with the interviewer will be pleasant and satisfying
– Make respondent see the study as worthwhile
– Overcoming barriers to the interview in the
respondents mind, such as misconceptions,
suspicions, etc., by explaining to the responded, in a
friendly manner, the purpose of the study, how they
were identified as potential respondents and the
confidentiality of the interview.
19
After the introduction:
Follow this simple sequence to introduce
yourself:
1. Tell the person who you are and who you
represent
2. Tell the respondent what you are doing in a
way that will simulate his or her interest
3. Tell the respondent how he or she was chosen
4. Try to create a relationship of confidence and
understanding between you and the respondent
21
1. Follow the questionnaire, but you can use it informally -if you
already know the questions and their sequence
2. Conduct the interview in an informal and relaxed atmosphere;
avoid creating the impression that what is going on is a crossexamination or quizzing
3. Ask the questions exactly as worded in the questionnaire
4. Read each question slowly to allow the respondent enough time to
assimilate the question before answering
5. Present the questions in the same order as they appear in the
questionnaire
6. Ask every question specified in the questionnaire
7. If a question is misinterpreted or misunderstood, repeat it just to
clarify, but do not reword it. (Unless you have decided on that
before … se above)
Interviews and Questionnares 2011
22
Questionnarie Construction ..
Questionnarie Construction ...
Issues to address first:
• Basics:
• A questionnaire has three
components:
– The cover letter, ”sell your study” why
is it important!
– Main body
– Instructions, clear and short
Interviews and Questionnares 2011
20
Principles of Interviewing .
Principles of Interviewing ..
Interviews and Questionnares 2011
Interviews and Questionnares 2011
23
In the questionnaire the most appropriate
method of data collection for this study?
What type of questions should I use?
What type of responses do i anticipate?
What language (wording) do I use?
How many questions should I include (length)
How will ethics and objectivity be observed
in the study?
Interviews and Questionnares 2011
24
4
The Cover Letter
Questionnarie Construction .
THE COVER LETTER:
Introduces the researcher and the research issue
Motivates the respondent to participate
Assures anonymity and confidentiality
FORMAT:
Explains the objectives and significance of the study
Familiarises the respondent with the research
institution’
Explains the reason for asking the respondent to
complete the questionnaire
Offers assurance of anonymity and confidentiality
Includes instructions about how to answer each
question (optional)
The questionnaire is the main instrument.
Issues to address before designing the
questionnaire.
Content of questions?
Type of questions?
Sequencing of questions?
Possible bias on wording questions; Leading
questions, double barrelled, or threathing
questions
Cover letter and instructions
Interviews and Questionnares 2011
25
Interviews and Questionnares 2011
The Main Body
Lenght of Questionnaire?
Main body includes the questions
Have to address the key aspects:
Format refers to how the questions are placed in the
context of the topic – the logical order that allows
passage/transition from one topic to the next – avoid
distortion and confusion.
Questions should be related to each other in a logical
and interesting manner. Be observant of bias from
sequencing.
The order and presentation should make the
respondents feel comfortable.
Be prepared to change the ordering and split the
sample.
Interviews and Questionnares 2011
Depends on focus of the research or survey
The objectives of the study
Type of respondents, and their time
constraints.
Available resources (Time)
GOLDEN RULE: Should include as many questions
as necessary and as few as possible. If a
question seeks information you are unlikely to
use, omit it. Be critical. Think in terms of
relevant information. THUS: KEEP IT SHORT
AND SIMPLE!!
27
Interviews and Questionnares 2011
28
And the Steps are ?
The Layout
1)
2)
3)
4)
5)
6)
Preparation
First draft
Self critique
External critique
Re-examining and revision. Pretesting/piloting
Review the performance and the feelings of
the respondents. Did the questions capture
the information? Are there vague questions?
7) Revision
8) Final Questionnaire
o Questions must be well presented, easy to
read/follow
o Questions and response categories must be
easy to identify and distinguishable from
others
o Leave enough space between questions
o Include clear instructions regarding how to
answer the questions – be precise
o Where necessary, leave sufficient space for
remarks and observations
Interviews and Questionnares 2011
26
29
Interviews and Questionnares 2011
30
5
Contents of Questions
Content of Questions ..
FACTUAL QUESTIONS
Designed to elicit objective information from the respondents
regarding their backgrounds, environment, habits, etc.
Background questions seek to provide information that can be
used to classify respondents, e.g. sex, age martial status,
education, income, etc.
Other questions are designed to provide information about the
respondents social environment, means of transport, leisure
activities
How accurately they are answered depends on how they are being
asked
Respondents can fail to give accurate answers either because:
Questions can be concerned with:
• Facts – factual questions
• Opinions
• Attitudes
• Motivation
• Level of familiarity with certain issues, etc.
Boils down to two categories:
• Factual questions
• Questions about subjective experiences or
assignments
Interviews and Questionnares 2011
31
Interviews and Questionnares 2011
Contents of Questions .
SUBJECTIVE EXPERIENCES/ASSESSMENT QUESTIONS
o Designed to elicit information about the respondents’ beliefs,
attitudes, feelings and opinions
o Attitudes, commonly surveyed in social sciences, are inclinations
that people tend to have when confronted with certain stimuli
o Attitudes may be described by their content (about what?), their
direction (positive or negative) and their intensity
o Questions about opinions and attitudes present more problems in
surveys than questions about facts
o The answers to subjective questions are more sensitive to
changes in wording, emphasis and sequence than answers to
factual questions
o Presented in different ways, the same question can elicit
different answers! This can be i) a solution, ii) sensitivity check.
Interviews and Questionnares 2011
32
Questionnaire Construction
TYPES OF QUESTIONS
• Question can fall into three types
– Closed end questions
– Open ended questions
– Contingenty questions
33
Interviews and Questionnares 2011
Closed End Questions
34
Open Ended Questions
Could be factual or subjective
Respondents are offered a finite set of answers and
are asked to choose the one that closely reflects their
understanding
Advantage:
– Are easy to ask and answer
– Require little writing by interviewer and/or respondent
– Anaysis is straithforward, in particular statistics/computer
procesesing
•
•
•
•
•
•
•
•
Disadvatage:
Can introduce bias. E.g. By forcing the respondend to choose
from a list they feel unsure about or offering them
alternatives they may never have thought about on their own.
Interviews and Questionnares 2011
Do not know the information
Cannot recall the information
Do not understand the question
They simply do not want to answer
•
35
The opposite to Closed Ended ...
Could also be factual or subjective
Respondents are not offered any type of choice
The respondents’ answers are recorded in full verbatim
Do not force the respondent to adopt to a set of
preconceived answers
Enable respondents to express to express their
opinions freely and spontaneously
Probing is possible where answers are not clear
Enable the interviewer to clear misunderstandings (if
desired)
Drawback: Difficult to answer, and even more difficult
to analyse. Need to design a coding frame in order to
classify the answers. Details might be lost.
Interviews and Questionnares 2011
36
6
Contingency Questions ?
Open Ended or Closed Ended? Or Mix?
Depends:
• On the objective of the questionnaire
• The respondent’s level of information about the
topic
• The extent to which the topic has been thought
through by the respondent
• The extent to which respondents can
communicate the content of the answer or the
extent to which they are motivated to answer
Interviews and Questionnares 2011
37
• Are a special-case of closed ended questions –
applying only to a sub-group of respondents: Do
you have children, yes or no?
• The researcher determines the relevance of
the question to the respondent by asking a
preceding filter question
• Only those who answered positively to the
contingency question will find the question
relevant
Interviews and Questionnares 2011
38
Questionnaire Construction ...
Questionnaire Construction ...
FORMAT OF QUESTION
• Several techniques can be used to structure the
response categories of closed ended questions
• A general format is to present all possible answers and
have the respondents choose the appropriate category
by either circling, writing the number of the answer in
a box or blank space, or check the correct answer box
• The least recommended is the use of blanks because
respondents may check in between the blanks, making
it difficult to discern which answer was intended
• Circling code numbers is most preferred for ease of
analysis.
SEQUENCE OF QUESTIONS
• Studies have shown that the order in which questions
are presented affects the type of responses given
• Generally, the questions presented first should put the
respondent at ease, serving to create a rapport
between the respondent and the interviewer
• They should be easy to answer, interesting and noncontroversial. Leave open ended question till late
• There are two broad patterns of question sequencing:
Interviews and Questionnares 2011
Interviews and Questionnares 2011
39
– The funnel sequence
– The inverted funnel sequence
40
Sequence of Questions ..
Sequence of Questions
FUNNEL SEQUENCE
• Each successive question is related to the previous one
and has a progressively narrower scope
• The method is recommended in studies where the
purpose of the survey is to obtain detailed information
and when the respondent is motivated to supply that
information
• By asking the broadest questions first,< the researcher
avoids imposing a frame of reference before obtaining
the respondent’s perspective
• If the objective is to discover unanticipated
responses, use the funnel method.
THE INVERTED FUNNEL SEQUENCE
• In this approach narrower questions are
followed by broader ones
• Recommended when the survey topic does not
strongly motivate respondents to communicate
– either because the topic is not important to
them or their experiences are not recent
enough to be vivid in their memories
• In this case, narrower questions are easier to
answer or serve to establish specify facts
before a question requiring overall judgement
is asked.
Interviews and Questionnares 2011
41
Interviews and Questionnares 2011
42
7
Questionnaire Construction ......
Questionnaire Construction .....
• PITFALLS IN QUESTIONNARIE
CONSTRUCTION – AVOIDING BIASES
• A) WORDING
• Questions must be worded so that the
respondent understands them
• Use the words that they understand
• Avoid or qualify words that are open to
different interpretations
• Questions should be worded such that the
responded understands their meaning and so
that the questions has the same meaning to
each respondent
• PITFALLS IN QUESTIONNARE
CONSTRUCTION – AVOID BIASES
• B) RESPONSE SETS
• A response set is the tendency to answer all
questions in a specific direction, regardless of
their content
• problem when a set of questions are presented
together with the same format, and especially
when all refer to the same topic
• Can avoid creating a response set bias by
changing the forma of the questions.
Interviews and Questionnares 2011
43
Questionnaire Construction ....
PITFALLS IN QUESTIONNARE
CONSTRUCTION – AVOID BIASES
• C) LEADING QUESTIONS
• Are the questions phrased in such a as to lead
the respondent to think that the researcher
expects a certain answer, e.g., you wouldn’t say
you favour a new president in the next
election? Avoid such phrasing.
Interviews and Questionnares 2011
44
Questionnaire Construction ...
PITFALLS IN QUESTIONNARE CONSTRUCTION –
AVOID BIASES
• D) THREATHENING/INTIMIDATING QUESTIONS
• Are the questions touching on topics that are anathema
to the respondents – they find the embarrassing and
thus difficult to answer.
• Are there anxiety arousing questions?
• May concern illegal or contra-normative or about
behaviours though to be socially deviant, or not usually
discussed in public without arousing tension, e.g. A
respondents drinking habits, gambling, hygiene, sexual
preferences, tax evasion.
45
Interviews and Questionnares 2011
46
Questionnaire Construction.
Questionnaire Construction ..
PITFALLS IN QUESTIONNARE CONSTRUCTION –
AVOID BIASES - THREATHENING/INTIMIDATING
QUESTIONS cont.
• There is evidence that respondents, if presented with
threatening questions tend to deny the aspect under
discussion
• The researcher can check whether a question
respondent feels threatened by trying to determine
how comfortable the latter feels talking about the
subject
• Can then use a long introduction to the question rather
than a short straight question
Interviews and Questionnares 2011
Interviews and Questionnares 2011
47
PITFALLS IN QUESTIONNARE CONSTRUCTION –
AVOID BIASES –
E) DOUBLE BARRELED QUESTIONS
• These combine two or more questions in one
• Poor political and economic governance are the two
most serious problems facing our country today:
Agree
Not sure
Disagree
Strongly disagree
• Such questions confuse the respondent
• Most questions that contain ’and’ are likely to be
double barrelled
• Can use ’and’ if the dimensions separated by ’and’ are
mutually exclusive
Interviews and Questionnares 2011
48
8
Data Collection Methods
Questionnarie and Interviews
• Before performing, and during construction,
think about how to analyse the data?
• Statistical methods (if enough data)?
– Descriptive statistics, of answers and respondents?
– Hypotesis testing?
– Correlation, causality, probability of answering,
linear regressions?
– Sampling corrections?
Interviews and Questionnares 2011
49
QUALITATIVE RESEARCH
• Qualitative versus quantitative data
• Qualitative research attempts to understand
behaviour and institutions getting to know the
persons involved, their values, rituals, symbols,
believes and emotions
• Uses a number of approaches:
Field research
Participant observation
Interviews and Questionnares 2011
50
Qualitative Research .....
Qualitative Research ....
FIELD RESEARCH
This is the study of people acting in their
natural courses of their daily lives
The researcher ventures into the worlds of
others to learn first hand about how they live,
how they talk and behave, and what captivates
them, and what distress them etc.
It is carried out in natural settings
If is a way of empathising and understanding
the subjective meanings of the the people
being studied.
PARTICIPATING OBSERVATION
• The participant attempts to obtain
membership to or close attachment to the
group being studied
• He/she adopts the perspectives of the people
in the situation being observed
• Systematically shares, insofar as
circumstances allow, the life and activities, and
on occasion, in their interests and effects
• Direct observation may entail learning the
language, habits, work patterns, leisure
activities and other aspects of daily living.
Interviews and Questionnares 2011
51
Interviews and Questionnares 2011
52
Qualitative Research ...
Qualitative Research ..
PARTICIPANT OBSERVATION cont.
In a complete participant role,
The observer is wholly concealed (Walraf method))
The research objectives are unknown to the observed,
and
The researcher attempts to become a member of the
group under observation
The approach has certain methodological problems;
problems of role playing/true identity, difficult to
deciding what to observe since one cannot ask certain
questions without arousing suspicions; recording
observations or taking notes on the spot is impossible.
• PARTICIPATING OBSERVATION ...
• In a participant-as-observer role:
• The observer informs the group being studied about
the research agenda; thus the research objectives are
known to the observed
• The researcher becomes a participant in the activities
of the group
• The observer must establish close relations with the
group members who subsequently serve as both
informants and respondents
• In this approach, the researcher gains deeper
appreciation of the group’s way of life by actually
participating rather than only observing.
Interviews and Questionnares 2011
53
Interviews and Questionnares 2011
54
9
Qualitative Research.
Data Collection: Baseline Data
THE PRACTICE OF FIELD OPERATIONS – THE
STEPS:
Selecting the research topic
Choosing a site and gaining access
Establishing relations
Finding resourceful and reliable informants
Leaving the field –exit strategy
Recording observations
Analysing the data – an ongoing process.
Interviews and Questionnares 2011
• Ecord initial observations before an
experien, a stimule, a change of some
kind, in order to compare before and
after.
55
Data Collection Methods
57
Secondary Data Analysis .....
WHY SECONDARY DATA ANALYSIS?
Conceptual –substantive reasons
Methodological reasons
Economic reasons of course: $ £ € SKr!
Conceptual – substantive reasons
– It may be the only data available for certain research
problems, e.g. historical data
– It facilitates greater geographic and depth for contemporary
issues than is possible using primary data
– Enlarges the scope for comparison since the data/the study
can be used for comparisons between nations, societies, socioeconomic groups etc. Enlarging the scope for generalisation.
Interviews and Questionnares 2011
58
Secondary Data Analysis ....
WHY USE?
• Methodological Reasons:
– If data is reliable, opportunities for replication
– Availability of data over long periods of time allows the
researcher to employ longitudinal study designs, using baseline
measurements in studies done in the past and similar data
collected more recently – temporal comparisons
– The approach may improve measurement by expanding the
scope of independent variables employed in the
operationalisation of concepts
– Increase in sample size, its representativeness thus allowing
more encompassing generalisations
– Triangulation reasons: to increase the validity of research
findings obtained with primary data.
Interviews and Questionnares 2011
56
Secondary Data Analysis ......
SECONDARY DATA ANALYSIS
• Methods discussed above concerns primary data the
researcher personally or with assistants of others
collects the data directly.
• But can also use data collected by other investigators,
usually for purposes that differ from their own
research objectives; or data can routinely be collected
by the government and other agencies for
administrative and policy purposes: census, special
surveys, archival data, etc.
• Common in economics: data used to investigate
household structure, income, etc.
• Secondary data analysis refers to research findings
based on data collected by others.
Interviews and Questionnares 2011
Interviews and Questionnares 2011
59
LIMITATIONS OF SECONDARY DATA
o Secondary data can only approximate the data
that you would like to employ for testing
hypothesis. Differences in sample size,
question wording and sequence, details of
interview schedules and techniques, etc.
o Access problems: It may be difficult to find
the studies with the variables of interest to
the researcher
o The recearcher may have insufficient
information about how the data was collected.
Some secondary data is very poor.
Interviews and Questionnares 2011
60
10
Secondary Data Analysis ...
Secondary data Analysis ..
SEARCHING FOR SECONDARY DATA ... THE
STEPS:
Specify your needs, examine the subject indexes of archive
holdings and identify key-words
Do you need permits ?
Familiarise yourself with sources: Search the guides, catalogues
and data archives of organisation with the data you need.
Make initial contacts – with people familiar with those catalogues
or data archives
Make secondary contacts – where possible, use professional
people to verify the information and learn how to formally request
for the data
Check accessibility – obtain information on possible problems from
people who have used the data
Analysis and supplemental analysis – obtain additional data if
necessary.
Interviews and Questionnares 2011
• COLLECTING SECONDARY DATA ... THE
METHODS
• Unobtrusive methods/non-reactive measures:
Have no effect on the conditions under which
the data are generated/collected e.g.,
– Perusing public records
– Simple observations
– Archival records
Public: actuarial, judicial and other official records,
government and quasi-governmental documents, mass media
reports
Private: autobiographies, diaries, letters. Problem –
authenticity.
61
Interviews and Questionnares 2011
Secodary Data Analysis.
Some methods
ANALYSING SECONDARY DATA
Social science analyse secondary data using the method
of content analysis
Content analysis is any technique for making inferences
by systematically and objectively identifying specified
characteristics
Economists have, in addition, other methods that they
use to analyse secondary data
The secondary data available to economists could be
Cross sectional
Time series
Panel data.
Interviews and Questionnares 2011
62
63
• Linear regression – CLR, OLS
– Principal component
• System estimation
• Panel data methods
– Variance analysis
• Tobit models
– Censored data
– Count data
• Probit – Logic
• Duration
• Event studies
Interviews and Questionnares 2011
64
The times they are a’ changing
• Quantitative methods are getting more ad more
important.
• The demands for what is considered best practice is
increasing rapidly.
• The amount of research in quantitative methods are
exploding
• Marketing is taken over by people with solid training in
micro and macro economics and econometrics.
• Finance has been taken over since long, expanding into
organisation and accounting.
• Tons of more data, costless storage, computing cost
are almost zero – there are no excuses left.
• You have been warned!
Interviews and Questionnares 2011
65
11