August 12, 2006 (excerpts)
Multiple-Case Study on Evidence Use in the Sex Education Debates:
The Interacting Roles of Values, Beliefs, and Collateral Information
Principal Investigator: Norman A. Constantine, Co-Principal Investigator: Carmen R. Nevarez
3. Specific Hypotheses or Questions to be Addressed
As described in section 1, this study is to address the primary research question of (1) how do policy
influentials use evidence related to the effectiveness of school-based sexuality education and other
adolescent sexual health promotion programs? To address this question, we further ask (2) how do
social values, ideology, prior beliefs, and collateral information impact and interact with evidence
appraisal and use? Finally, we ask a more exploratory question, (3) how can deliberative values
inquiry methods help disentangle intertwined social values and research evidence arguments?
evidence
evidence-based policy arguments
theory
values and
ideology
prior
collateral
beliefs information
Figure 1. Hypothesized Theoretical Model of Research Evidence Use
A hypothesized theoretical model (Figure 1) of research evidence use in the sex education policy
debates will be assessed and further developed. Based on previous theory and research (described in
section 2 above) from the sociological study of research use in policy making, and the cognitive
laboratory study of causal and scientific reasoning, this model involves nine specific propositions:
1. Research evidence as argument predominates over research evidence as data or as ideas in
the sex education policy debates.
2. Political use of research predominates over conceptual use and instrumental use.
3. Much of the evidence appraisal process among policy influentials involves the preconscious,
implicit, and often heuristic reasoning system rather than the conscious explicitly logical
system.
4. Policy influentials rely on considerations of theory (including plausible mechanisms and
explanations for potential causal relationships) when appraising evidence.
5. Considerations of theory and evidence are fundamentally interdependent.
1
6. Policy influentials’ social values and ideology influence responses to evidence, and are
themselves influenced by evidence.
7. Policy influentials’ prior beliefs and working hypotheses influence responses to evidence,
and are themselves influenced by evidence.
8. Collateral information, such as plausible explanatory mechanisms, information that yields
anomalous predictions, information that resolves anomalies, information that yields
congruent predictions, and plausible rival alternative explanations, influence responses to
evidence and is itself influenced by evidence.
9. The collateral information that policy influentials search for is limited by the collateral
information that is initially available to them, and by the social context of their reference
group.
Using the rigorous multiple-case-study methods described in the next section, we will compare each
of the propositions from our hypothesized theoretical model to the less complex plausible rival
hypotheses that are implicit in each proposition. For example, the plausible rival hypothesis to
proposition 1 is the use of research evidence as data or ideas rather than as argument, the plausible
alternative to proposition 2 is instrumental and conceptual research use rather than political use, etc.
It could be argued that the existing evidence in favor of some of the nine propositions is so strong
that the rival alternatives are not really plausible. Yet for many of these propositions, the rival
alternatives are commonly accepted or implicitly assumed among many evidence-based policy
proponents who have not explicitly studied the research use and evidence appraisal process.
Furthermore, these propositions have not previously been studied in this particular policy context,
nor within the integrative theoretical framework we have proposed.
Additional more specific propositions and rival hypotheses will be developed during the course of
the study. For example, we have not hypothesized the precise role of confirmation bias in the sex
education policy debates, and several good arguments could be made in various directions, e.g., no
role, consistent role, conditional role, etc. At this point in our understanding of the phenomenon, we
believe it is prudent to first look for evidence for and against the roles of prior beliefs more
generally (proposition 7), and then address the most promising specifics as our data collection and
analyses proceed and our theory develops. We believe that it is a strength of qualitative research to
allow this iterative development.
As is standard in this type of approach, we will seek convergence across data sources within cases,
and replication across cases and sub-cases. As accumulating data corroborate or challenge our
propositions, individual propositions and their extended networks of implications will be modified
as appropriate and retested across additional cases. Although it is possible that every proposition
will be corroborated, or alternatively that every proposition disconfirmed in respect to a plausible
rival, we expect to find a combination of corroboration, disconfirmation, modification, and
clarification of individual propositions across cases, yielding in the end a further developed and
robust theoretical model.
4. Research Methods and Data Analysis Plan
More and more I have come to the conclusion that the core of the scientific method is not
experimentation per se but rather the strategy connoted by the phrase "plausible rival
hypotheses." This strategy may start its puzzle solving with evidence, or it may start with
2
hypothesis. Rather than presenting this hypothesis or evidence in the context-independent
manner of positivistic confirmation (or even of postpositivistic corroboration), it is
presented instead in extended networks of implications that (although never complete) are
nonetheless crucial to its scientific evaluation. – Donald T. Campbell (from the Foreword to
Yin’s Case Study Research, 2003a, originally 1989)
The study will be employ a rigorous explanatory multiple-case-study methodology (Yin, 2003a, see
also 1998, 2000, 2003b). This methodology was developed to emulate procedures from the natural
sciences in pursuing inquiry. Among the working assumptions are that “theory-driven inquiries are
to be preferred, and multiple-case studies are best designed around the same replication logic that
underlies the design of multiple scientific experiments” (Yin, 1998, pp. 229-230). Another central
assumption is that explanatory case studies must involve “a constant awareness and testing of
(plausible) rival hypotheses” (Yin, 2000, p. 242).
We are employing a multiple case embedded design (Yin, 2003a, pp. 40-55). This type of design
employs two or more primary cases, with additional embedded case units and subunits located
within each primary case. As described in more detail below, we employ four primary cases, with a
total of eight embedded cases, and four additional subunits (see Figure 2).
Sampling
Two overlapping strategies for selection of cases and other study elements (such as documents and
persons) are employed: Patton’s (2002) purposeful sampling strategies, and Yin’s (2003)
replication logic.
Patton’s purposeful sampling strategies consist of 16 specific sampling strategies for qualitative
research, to be used either individually or in combination. Five of Patton’s 16 strategies will be
employed in this study. Intensity sampling is defined as sampling for information-rich cases that
manifest the phenomenon of interest intensely. With theory-based sampling, the researcher samples
cases and other units on the basis of their potential manifestation or representation of important
theoretical constructs. Criterion sampling involves the selection of all cases that meet some
predetermined criterion of importance. Confirming/disconfirming case sampling involves selecting
cases predicted to confirm or disconfirm the researchers theoretical explanation for the phenomenon
under study. According to Patton, confirming cases can “confirm and elaborate the findings, adding
richness, depth, and credibility,” while disconfirming cases can be “a sources of rival explanations
as well as a way of placing boundaries around the findings” (2002: 239). Finally, snowball sampling
is primarily a logistical methods of obtaining additional information-rich case informants through
referrals from other informants.
Yin (2003a) views case selection less in terms a sampling logic, but rather as a type of replication
logic where each selected case is viewed as an attempted replication of other cases. This view is
similar to the way one would view multiple experiments conducted by different research groups, as
in laboratory sciences (Herson & Barlow, 1976). Yin’s replication logic is similar to Patton’s
confirmation/disconfirmation case sampling. Using Yin’s terminology, literal replication involves
selecting cases to predict expected results (similar to Patton’s confirmation sampling), and
theoretical replication involves selecting cases to predict contrasting results (similar to Patton’s
disconfirming sampling).
3
At an overarching level, it can be useful to think of the specific research policy area, school-based
sexuality education, as an intensity sample from the population of all potential policy area topics
that might be relevant to a research investigation on the interacting roles of values, ideologies,
beliefs, and collateral information on evidence use in policy debates. Arguably, the sex education
debates are among the most value-laden areas of policy deliberation within the fields of education
and youth development, and as we noted earlier, are seen by some as foundational to broader
political and ideological conflicts (Irvine, 2003).
Case selection
Our four primary cases represent sex education policy debate categories, and have been selected
according to Yin’s replication logic as literal replications, in other words, we expect to find similar
results across the four selected debate types. These are debates about: (1) federal legislation, (2)
state legislation, (3) local district policy, and (4) research study interpretation. Using Patton’s
terminology, this could be considered confirming case sampling. However, as Koslowski and
Maqueda (1993) have pointed out in a broader context, the same information can be seen as
confirming or disconfirming, depending on the expected result. As such, each of these four cases
will function as either confirming or disconfirming cases to the extent that they do or do not
replicate as expected.
Our embedded cases have also been selected according to replication logic, but here combined with
the strategy of intensity sampling, i.e., sampling for information-rich cases that manifest the
phenomenon of interest intensely. As illustrated in Figure 2, we have selected two Level 2
embedded cases within each of the four primary cases. Within the federal legislation debates
primary case we have selected prototypical cases to represent both abstinence-only, and
comprehensive sexuality education. Within the state legislative and local district policy primary
cases, we have selected embedded cases in each of two states -- California and Minnesota. Finally,
within the research study interpretation debates primary case, we have selected two types of studies
– sex education public support surveys, and influential evaluations of sex education programs. Two
additional Level 3 sub-unit cases have been selected within each of the research study embedded
cases. These have been selected to represent both abstinence-only supportive findings and
comprehensive sexuality education supportive findings within the sex education support survey
case, and within the sex education program evaluation case.
To summarize, as illustrated in Figure 2, we have selected four primary cases, eight level 2
embedded cases, and four additional level 3 subunit cases. This diversity of cases will facilitate the
iterative testing and revision of our theoretical model and propositions, and our attempted
replication of results across primary cases, embedded cases, and subunits.
4
Figure 2. Multiple embedded case design (large square boxes represent primary cases, rectangular boxes represent level
2 embedded cases, small square boxes represent level 3 embedded sub units).
Data sources and data collection
As described below, each case will be studied via several types of data sources, including archival
documents, research reports and articles, open-ended in-depth individual and group interviews,
evidence-review individual interviews, and values clarification group interviews.
As is standard practice in qualitative and case study research, data collection, analysis, and theory
development will be highly iterative, with initial data and ongoing analyses suggesting theoretical
specifications for subsequent data sources and interview questions. In this study, we plan to
organize the data collection and analysis activities into two phases, as diagramed below. Phase 1
will involve an inter-dependent series of archival document analyses, open-ended individual
interviews, and open-ended group interviews. Phase 2 will build on the findings of Phase 1, with
more structured individual interviews and groups.
5
Phase 1
Phase 2
Archival
document
analysis
Openended
individual
interviews
Openended
focus
groups
Evidence
review
individual
interviews
Values
inquiry
groups
Figure 3. Data collection
1. Archival documents. Phase 1 case analysis will begin with a deep and thorough review of
archival documents associated with each of the ten embedded cases and subunits, including
legislative documents (such as legislative bills and revisions, legislative analysis reports, and
transcripts of committee discussions and testimony), research reports (including both mainstream
articles or reports and advocate-sponsored or -conducted research), advocacy materials (including
press release and research summaries, and other advocates materials), media coverage (primarily
newspaper and magazine articles and television news transcripts), and other relevant materials that
emerge. (See Exhibit 1, Archival Documents Sources by Case Table.) Archival documents will
be selected for analysis by a combination of criterion and theoretical sampling. Following a
comprehensive search, all relevant documents identified within each category will be obtained and
analysis will begin. For case-by-document-type combinations with a limited number of documents
available, all will be analyzed. For combinations where there are many documents available,
theoretical considerations, i.e., relevance to the hypothesized theoretical model as it evolves, will
inform the decision of which documents to analyze.
Archival document will be analyzed using retrospective content analysis methods to identify and
investigate, within each case, the following issues and topics.
(a) key arguments and counterarguments made in support or opposition to a position,
(b) values and ideologies implicitly or explicitly employed to support arguments,
(c) research and evaluation-based evidence employed to support arguments,
(d) any criteria employed implicitly or explicitly to demonstrate the validity of the research,
(e) the processes and trajectories of evidence use,
(f) the clustering and consistency of value- and evidence-based arguments,
(g) other interactions among values, ideology, and evidence,
6
(h) if one or more concrete policy decisions resulted, how any of the above were presented
to justify the policy decision after adoption.
2. Individual and group interviews
A total of 286 relevant individuals (90 individual interview participants, and 196 group interview
participants) will be selected using stratified and theoretical sampling. Individuals will be stratified
by case and by role. Case stratification will be by the eight Level 2 embedded cases, or for some
interviews, the ten combined Level 2 and 3 cases and subunits will be used (see Figure 2). Role
stratification will be by our four primary policy influential roles, selected to provide a maximum
range of literal and theoretical replication opportunities: policy makers (including legislators,
legislative staff, school board members, and school superintendents), abstinence-only sex education
advocates, comprehensive sex education advocates, and media reporters.
The initial plan, as summarized in Exhibit 2, pre-specifies the numbers of participants to be
selected within each case/role combination. However, as data collection and analysis progress and
concepts and theoretical explanations are developed and tested, sampling allocations across cases
and roles will be modified and rearranged as necessary according to emerging theoretical sampling
considerations. This is a fundamental feature of most case study research (Yin, 2003a), and many
other types qualitative research (e.g., Miles & Huberman, 1994; Strauss & Corbin, 1998), intended
to provide theoretically-rich cases to use in probing the developing theoretical constructs and
relations, and seeking confirmatory and disconfirmatory challenges. Sampling at the individual
person level within specific case/role combinations will continue until theoretical saturation is
reached, whereby all categories related to the evolving theory are well developed and their
relationships well established and validated, such that the collection of additional data would be
redundant. In some cases this might be fewer individuals than estimated, in others more individuals
might be necessary, however the total number of interviews will remain as proposed.
Individual and group interviews will be conducted in California and in Minnesota, depending on the
geographic context of the case. In-person interview and focus group facilities will be available at
the Public Health Institute’s Oakland and Sacramento, California offices and at the University of
Minnesota Prevention Research Center offices in Minneapolis. Additional facilities outside of these
areas will be rented as necessary.
We expect to conduct 40 of the 90 individual interviews, and 12 of the 28 focus group interviews by
telephone, primarily during the Phase 1 open-ended interviews. All others will be conducted in
person. Use of telephone interviews and focus groups will help us to recruit busy policy makers and
other policy influentials by requiring smaller and more flexible demands on their time than would
be required by in-person interviews, and also will allow us to extend the geographic reach of our
data collection. Although not yet well known in academic research, telephone focus groups have
been used for more than 25 years in market research, and increasingly have been used successfully
in public health research (e.g., White & Thompson, 1995; Appleton et al., 2000). We have
previously employed telephone focus groups successfully in the community support study discussed
under preliminary studies.
Open-ended individual interviews. Also as part of Phase 1 data collection, between three and five
in-depth individual interviews will be conducted for each of the eight embedded cases, for a total of
approximately 32 open-ended individual interviews. Potential participants will be identified through
professional networks, the archival document reviews, and additional “snowball” referrals requested
7
of interview participants and other key informants. Interview participants will span the full range of
policy influentials relevant to each case. (See Exhibit 2, Interview Participants by Type Table).
Initial open-ended interview questions will be designed to further probe the areas investigated in the
archival document review (a-h above), as well as to focus on individual’s evidence appraisal
strategies.
1.
2.
3.
4.
5.
What is your view on …? (probe)
Why do you feel this way? (probe)
What evidence or other information have you considered that informs your view? (probe)
How did you decide if this evidence was believable? relevant? important? (probe)
What if this evidence were reversed, how would that affect your feelings about …? (probe)
Additional initial interview questions will be selected and adapted from existing protocols of the
Constantine and colleagues (2004, under review) community support for sex education study, and
Koslowski’s hot cognition studies described above under preliminary research (copies of these
protocols are on file and available on request). As is standard practice in qualitative research, once
data collection begins, the initial interview questions will evolve based on theoretical sampling
considerations to best address the developing theoretical framework.
Open-ended focus groups. Also as part of Phase 1 data collection, two open-ended focus groups
(i.e., group interviews) will be conducted for each embedded case, for a total of 16 groups. These
groups will employ similar questions to the individual interviews. The individual and group
interviews will be interspersed over time to provide mutually informing data, and potential
convergence across personal and social perspectives.
Evidence review individual interviews. As part of Phase 2 data collection, individual interview
participants will review research summaries based on actually case relevant research, and will be
asked a series of questions regarding the evidence presented in the summary and their appraisal of
this evidence. The questions will be modeled after questions employed in Koslowski’s ongoing hotcognition research (available on request, and similar to interview protocols that appear in
Koslowski, 1996 and Koslowski, et al, 1989, Appendix A), but extended to include specifics
appropriate to the relevant studies in each case. These Phase 2 interview questions will also be
substantially informed by the findings of the Phase 1 open-ended interviews and archival documents
analysis.
Comprehensive values inquiry groups. Also as part of Phase 2 data collection, a series of 12
comprehensive values inquiry groups (Mark, Henry, & Julnes, 2000) will be conducted. This will
involve group interviews to explore, probe, and clarify the explicit and implicit criteria by which the
success and value of school-based sex education programs are judged. Rather than conducting
separate group interviews for each embedded case, each group will be linked to one of the four
primary cases (three groups per primary case) and specifically to the issues and values most
commonly identified within that set. The goal of this inquiry will be to further elucidate the relevant
values involved across different clusters of stakeholders, and to explore the relationships among
values, ideology, and evidence appraisal and use. The groups will involve two strategies -- focus
group discussion questions, and simulated deliberative process exercises (Mark et al., 2000).
Elements of deliberative democratic evaluation processes (House & Howe, 1999), together with
frame-reflective deliberative processes (Fischer, 2003; Schön & Rein; 1994) will also inform the
development of these group exercises. Participants will be selected to represent a wide range of
8
policy making and policy shaping roles, ideological perspectives, and racial/ethnic and geographic
diversity. Both heterogeneous and homogenous groups will be used.
Data management and analysis
Modified grounded theory qualitative analysis methods (Strauss and Corbin, 1998) will be used to
help develop, test, and refine, as the data collection progresses, hypothesized concepts and relations
within an evolving theoretical framework. The primary modification to traditional grounded theory
will be the incorporation of an initial hypothesized theoretical model at the beginning of the analysis
process, rather than developing one solely based on the newly collected data (as described by
Dey,1999). Comparisons across cases also will be facilitated through the use of the cross-case
qualitative analysis and data display methods of Miles and Huberman (1994).
Intensive data management and analysis will begin with the first obtained archival documents and
completed interviews and continue in coordination with data collection throughout the course of the
study. Primary activities will consist of: (1) interview transcription, (2) open, axial, and selective
coding, and (3) validation of the theoretical scheme (all described below.) Throughout this process,
the specialized tools of microanalysis and memos and diagrams will be used extensively.
Microanalysis involves detailed line-by-line coding of interview transcript data. Analytic memos
are aids in exploring and documenting the dimensions and assumptions of coding processes, while
diagrams are visual representations of the emerging concepts and categories and the relationships
among them. Data management and analysis will be enhanced by the use of the ATLAS.ti (Muhr,
1997) qualitative analysis software system, described below.
1. Interview transcription. An interview transcription service with experience working with
confidential data and a long track record with the Public Health Institute and this study’s
investigators will be used to transcribe all audio tape recordings of interviews and focus groups. The
resulting transcripts will be provided pre-formatted for immediate entry into the ATLAS.ti software
system. The senior research associate will be responsible for interview tape and transcript tracking
and security, as well as monitoring transcription quality control. (Archival documents also will be
transcribed when necessary, but we anticipate obtaining most of these documents in electronic
versions, e.g., through Lexus/Nexus, which will then be reformatted and loaded in the ATLAS.ti
system.)
2. Coding. Coding is the fundamental analytic process to be employed. Three basic strategies of
grounded theory coding will be employed (Strauss & Corbin, 1998). Open coding is the process to
be used to label and categorize phenomena as indicated by the data. Conceptual labels will be
applied to segments of data, and properties and dimensions of the resulting concepts will be
explored as informed by but not limited to concepts and propositions from the hypothesized
theoretical model. These concepts will be compared, and similar concepts grouped together under a
common label, or category. As the base of concepts and categories grows, both within an individual
interview transcript and across the database of all conducted interviews, axial coding, or coding
around the axis of categories, will begin. This will involve using our hypothesized theoretical
framework as the starting point for relating the categories and their subcategories theoretically, in
order to adapt, contextualize, or expand on this framework in light of the data collected.
Subsequently, and in some cases simultaneously, selective coding will be performed. This will
involve linking and integrating categories around one or more central categories, and further
developing and refining the theoretical framework and its propositions.
9
The primary coding responsibilities will be assumed by the principal investigator and senior
research associate, with input, regular discussion, and review by other research team members and
consultants. A coding manual will be developed initially based on the hypothesized theoretical
model and propositions, and regularly updated with supplemental concepts and definitions as they
emerge in the analyses. Inter-coder agreement will be assessed monthly by having the both primary
coders code the same transcript and then discuss and resolve differences. Consultants also will
review transcripts and coding results and provide analytic critiques on a regular basis.
3. Theory validation. Several strategies for validation of the emerging theoretical scheme will be
employed. First, bi-weekly in-person meetings between the principal investigator and senior
research associate (and the co-principal investigator when available) will be held to review
interview transcriptions and coding activities. The PI will have primary responsibility for setting
and maintaining uniform coding standards as well as reviewing and monitoring all coding, however,
the senior research associate will independently collect and code her own data, then during the biweekly meetings will discuss and cross-validate interpretations with the PI. Second, the emerging
theory and interpretations of the data will be regularly presented to the co-PI and the three
consulting advisors for feedback and advice. Finally, the research literature on evidence appraisal
and policy use of research will be regularly consulted, and the application of these existing research
findings to our evolving theoretical model will be revisited. This literature comparison phase
(Pandit, 1996) will involve reviewing the emerging theory in relationship to findings presented in
the literature, and asking what is similar, what is different, and why.
Data management and analysis software
ATLAS.ti (Muhr, 1997) is a qualitative analysis software systems designed specifically for theory
building and testing. ATLAS.ti will be used to assist in managing the copious amounts of interview
transcript data that will be generated, and in extracting concepts and relationships from the raw data
and organizing those concepts into a logical, explanatory theoretical scheme. PHI maintains a
multiple-user license and an internal users group for ATLAS.ti. While strongly supporting data
management and analysis activities, use of ATLAS.ti will at the same time add credibility and
confirmability to this study. Raw and analyzed data will be systematically organized and archived.
The process of testing and refining the hypothesized theoretical model will be well documented, and
the research audit trail easy to follow.
10
Legislative Documents
CASE
Bills and
revisions
Legislative
analysis
reports
A Generic Debate
B1.Section 510 Debate
B2. FLEA/ REAL Debate
C3. Cal. SB71 Debate
C4. Minn. SB581 Debate
D5. Mt Diablo USD Debate
D6. Minn. school Debate
E7a. Zogby Survey
E7b. Kaiser Survey
E8c. Virginity Pledge Eval
E8d. Mathematica Eval
E9d. Minn. ENABL Eval
Exhibit 1. Archival data sources by case
Committee
discussion
and
testimony
transcripts
Research Reports
Mainstream
article or
research
report
Researcher
press
releases &
summaries
Advocacy Materials
Advocates
article or
research
report
Advocates
press
releases
and media
advocacy
Advocates
other
materials
Media Coverage
National
media
State or
local media
Other
Other
media
Other
relevant
materials