The Contributions of Mixing
Methods to Survey Research
Danish Survey Research Society
February 2015, Copenhagen
Jennifer C. Greene
Overview
•  A conversation
•  Introductions
•  What is survey research?
–  Strengths, challenges, limitations?
•  Mixing methods – key concepts
•  Mixed methods survey research,
examples
•  Ongoing discussion
Introductions
•  JCG
•  Survey researchers
What is survey research?
The relatively accurate assessment (or
estimate) of the frequency, incidence,
magnitude, and dispersion of selfreported characteristics, attitudes,
behaviors, and other human practices
among a designated population or set of
populations.
•  Additions, corrections?
What does survey research
do well?
•  Character of survey research
–  “Proven” methodology
–  Strong potential for representative samples
–  Ability to develop good items and
psychometrically strong scales for measuring
higher-order constructs ! highly reliable
inferences with some degree of validity
•  Character of survey research (continued)
–  Ability to inter-relate demographics with
attitudes and behaviors, with some complexity
and nuance (multi-level modeling)
–  Ever-increasing methodological sophistication
(e.g., HLM, data analytics)
•  What else?
•  Contributions of survey research
–  Defensible inferences related to multiple
constructs
–  Detailed snapshots of groups of people
–  Complex patterns of association among
various clusters of variables
–  Relational patterns across discipline and fields
of study
•  What else?
What are key challenges of
survey research?
•  Defer to the experts
What are key limitations of
survey research?
•  Limited conceptualization and definition
of especially complex constructs
•  Insufficient attention to context
•  View the complexity of the social world
from just one angle, just one perspective
•  Correlational not explanatory
•  What else?
What can mixed methods
thinking contribute to survey
research?
• 
• 
• 
• 
MM definition
MM purposes
MM designs
Examples
“Definition” of MM
•  The intentional, and connected or linked,
use of more than one social science
tradition, methodology, and/or method in
service of better understanding
–  Tradition = philosophical paradigms and
assumptions, logics of justification, privileged
questions, ways of knowing
•  Examples: postpositivism, interpretivism,
constructivism, feminisms, critical social science
–  Methodology = inquiry logic, including
questions, design, sampling, method choice,
analysis, quality criteria, and defensible forms
of writing
•  Examples: experimentation, survey research,
ethnography, case study, narrative inquiry
–  Method = a technique or tool for data
gathering
•  Examples:
Ask ~ questionnaire, interview, assessment
Watch ~ observation
Find traces ~ unobtrusive measures
–  Can also mix theories or conceptual lenses
•  The intentional, and connected or linked,
use of more than one social science
tradition, methodology, and/or method in
service of better understanding
•  A study is a mixed methods study when
there is some connection or linkage among
the various methods and data sets at one
or more stages of inquiry.
•  Statement of vision
Purposes for Mixing, various
forms of “better understanding”
•  Example: Study of the character and
(in)stability of the educational pathways
of K-12 children when their families/
parents are homeless
! Toward revised educational policies for
children in temporarily-homeless
families
• 
Triangulation is the use of different
methods to generate findings that
(hopefully) converge in their assessment of
the same phenomenon, toward increased
validity and defensibility of inquiry
inferences.
• 
EX: Mix methods to better understand
phenomenon of school attendance patterns.
Mix analysis of school attendance records
with family (parent and child) interviews
•  Complementarity is the use of different methods
to assess overlapping phenomena or multiple
facets of the same phenomenon, whereby the
results from one method are used to enhance,
augment, clarify the results of the other, toward
more comprehensive understanding.
•  EX: Mix methodologies to better understand the
role or place of schooling amidst a homelessness
crisis.
Mix surveys of parents, children, and homeless
service staff with mini-case studies of selected
families
•  Development is the sequential use of different
methods to assess the same phenomenon, where
the results of the first method are used to inform
the development of the second.
•  EX: To address questions regarding how
schooling patterns differ for families that are
homeless for different reasons, use results of
surveys or case records to purposefully select
diverse sample of families for further interviews.
(Could be a mix of methods or methodologies.)
•  Expansion is the use of different methods to
assess different phenomena in order to
expand the breadth and scope of a study,
again toward more comprehensive
understanding.
•  EX: An analysis of school data on student
academic and behavioral performance is
added to this otherwise-experiential study of
the place of schooling within the
phenomenon of homelessness (method mix)
•  Initiation is the intentional use of different
methods to seek paradox or fresh conceptual
insight for a given phenomenon or a set of
phenomena, or serves as a placeholder for the
legitimization of dissonance in results.
(Initiation, continued)
•  EX: Mix ‘paradigms’ (and methodologies
and methods) to possibly invoke fresh
insights into the educative dimensions of the
homeless experience itself for children and
youth.
Mix constructivist narrative elicitation of
stories with a structured assessment of
possible knowledge and skills gained during
the homeless episode.
MM purposes ! MM design
Key design dimensions (so far)
1. 
Assess the same or different phenomena with the
different methods
2.  Methods of equal status, or one dominant and the
other less-dominant
3.  Sequential or concurrent implementation
4.  Mix at the end or mix throughout the study
•  Component designs
•  Integrated designs
Design
What
phenomena?
Status of
methods?
Sequence of
implem’tion?
Linking task?
COMPONENT
DESIGNS
* Triangulation
Same
Equal
Concurrent
Comparing results
* Expansion
Different
Variable
Variable
Connecting inferences
INTEGRATED
DESIGNS
* Development
Same
Preferably
equal
Sequential
Representing data in new
form
* Complementarity, Initiation
Same
Preferably
equal
Concurrent
Jointly analyzing data and
generating integrated
inferences
* Complementarity, Initiation
Same
One method
primary
Concurrent
Jointly analyzing data and
generating integrated
inferences
•  Comments, questions, challenges??
Example 1
Ungar, M., & Liebenberg, L. (2011).
Assessing resilience across cultures using
mixed methods: Construction of the Child
and Youth Resilience Measure. Journal of
Mixed Methods Research, 5(2), 126-149.
Study purpose
•  Focus – develop a structured, crossculturally respectful measure of resilience
in children and youth
•  Resilience viewed as not only an individual
characteristic, but also as qualities of the
environment
•  Aim – use measure to study internal and
external assets most influential in positive
developmental pathways
Study design
•  Mixed methods approach and rationale
–  Legitimize multiple standpoints
–  Honor tensions between homogeneity and
heterogeneity, between context and
generality, between standardization and
localization
–  Iterative, collaborative, involving
international team and local researchers
Study implementation
and selected results
•  Initial development of resilience measure
included:
–  Research team ideas (and theories)
–  Focus groups with youth and adults in each
participating country
–  Expert view ! local views ! integrated view
•  Pilot testing of structured instrument
combined with interviews with youth =
conversation about what it means to be
“doing well despite adversity”
•  Analysis of pilot and interview data
–  Bronfenbrenner’s framework not useful
–  Developed 7 aspects of resilience from
interviews ! scales in quantitative measure
•  Analyzed scales further ! four portraits of
resilience, reflecting four groupings of
youth
–  Minority world boys and girls
–  Majority world girls
–  Majority world boys in contexts with high
social cohesion
–  Majority world boys in contexts with low
social cohesion
•  Finalized instrument
Example 2
Jang, E.E., McDougall, D.E., Pollon, D.,
Herbert, M., & Russell, P. (2008).
Integrative mixed methods data analytic
strategies in research on school success in
challenging circumstances. Journal of Mixed
Methods Research, 2(3), 221-247.
•  Focus – developing a multidimensional
understanding of school success for
schools serving low-income immigrant
families and children (Ontario, Canada)
•  Concurrent MM design, n=20 schools
(purposefully selected)
–  Interviews and surveys with teachers,
principals
–  Focus groups with students and parents
–  Purpose of complementarity
•  Descriptive and reductive analyses
–  Interview data analyzed ! 11 themes
associated with school improvement
–  Survey data analyzed ! 9 factors associated
with school improvement
•  Integrative analyses
–  Transformed survey results to narrative form
–  Compared to qualitative themes
•  Next, created new survey ‘scales’ from
interview themes
–  Of 75 survey items, 63 were judged to relate to
the interview themes
–  3 interview themes not present in survey
items
–  Assigned the 63 items to one of the remaining
8 themes
–  That is, used the structure of meaning in the
interview data to ‘rescale’ and then reanalyze
the survey data
•  New ‘blended’ scales showed more
variation than original survey factors
•  Further analyzed blended themes
•  For each theme, identified schools with a
mean score significantly different from
overall mean (extreme cases)
•  Returned to qualitative data to provide
contextually rich narrative of the nature
and contours of the theme at the selected
schools
•  Wrote narrative case profiles by theme !
understanding the contextual meanings of
‘high’ and ‘low’ for that theme
•  Example for ‘parent involvement’ theme
–  ‘High’ school
•  Community with 25 different languages
•  School active in multiple parent programs, some in
partnership with local service agencies
•  One teacher serves as community liaison with
parents and families
•  Principal walks around community getting to know
families
•  Principal personally visits parents of children
placed ‘at risk’ and generates with them a ‘game
plan for their child
•  Parents perceived school as welcoming and ‘on
their side’
–  ‘Low’ school
•  Similar demographics as ‘high’ school, more
central city
•  Same recent history of academic success
•  Recently, principal and teachers have concentrated
on school safety
•  Principals recognizes importance of strong
parental involvement; school needs to turn
energies to this domain
Example 3
Kling, J.R., Liebman, J.B., & Katz, L.F.
(2005). Bullets don’t got no name:
Consequences of fear in the ghetto. In T.S.
Weisner (ed.), Discovering successful
pathways in children’s development (pp.
243-282). Chicago: University of Chicago
Press.
•  “Moving to Opportunity” housing
demonstration project, experimental
study, US 1990s
•  Key question: How does a substantial
change in neighborhood affect low-income
families?
•  Original design
–  Baseline survey
–  Two-year survey
–  Administrative data analysis
–  Focus on: job training and job earnings,
children’s school progress
•  Addition of a “qualitative” component
–  To better understand program “uptake”, gangs
and crime, and to listen well to issues not yet
on the table
–  Conducted observations of program contexts
and activities, consulted program counselors,
interviewed program participants
–  This account – 12 interviews with participants
(both experimental and control groups)
Four contributions to study
Interview data:
1.  Informed survey
2.  Generated conceptual framework re
mechanisms of change
3.  Provided “deep institutional
understanding” of MTO on the ground
4.  Contributed to lessons for housing policy
And in MM language …
What MM purposes invoked?
•  Development
•  Complementarity
•  Initiation
•  Comments, critiques, conversations?
… Thank you