2017 Vanderbilt Conference
”Questions of Validity and Reliability
for Qualitative Research”
Lessons Learned from Quantitative Methodology
Scott D. Churchill Ph.D.
University of Dallas
Peabody College, Vanderbilt University
March 24, 2017
Overview of Issues to be Addressed
•“Internal Validity”
(reframed for qualitative studies)
A fidelity to the targeted experiences
Avoiding Researcher Bias
•“External Validity”
•Crossing the threshold from the idiographic to
nomothetic
•“Reliability”
•Finding convergences and divergences
among different researcher’s perspectives
I. Internal Validity Redefined for
Qualitative Research:
“Finding our way to the evidence in a way that reflects a
fidelity to the targeted experience at the individual
level”
So, the question of internal validity – the validity of our findings
with regard to the sample studied – has two components:
•
first, finding participants whose experiences actually
represent the experience under investigation;
•
and secondly, developing a method of reflection that results
in findings that correspond faithfully to the experiences
reported.
(A) The Validity of Qualitative Evidence
To achieve a genuinely “phenomenological”
understanding, the aim would be for the
phenomenologist to allow the phenomenon to appear
concretely in its own self-givenness….
... and then to proceed to reflect upon and describe
its appearance as best one can without the
mediation of theories or hypotheses.
•To avoid the charge of subjectivism – where researchers base
their “findings” on their own private experiences, the founders
of phenomenological method in psychology decided to
“outsource” the original description of lived experience.
• When we ask patients or research subjects to describe their
experiences, they do so with many of the same operative
defenses and self-deceptions that were inherent in the
experiences they described.
• Thus if empirically-based phenomenological research is going to aim
to achieve fidelity with respect to an original experience using
narrative data and narrative methods of analysis, then out of
concern for the validity of narration one must consider the
possibility of "distortion" in the reflexive movement from the data to
the findings.
• In practice there will always be a precarious relationship between
an experience and its description, for a multitude of reasons.
Averill (1983) has proposed one understanding of why
verbal reports (protocols) have fallen under suspicion
and "searching criticism" by empirical psychologists:
•What a person says is under conscious, voluntary control, and
hence is subject to dissimulation and conformity to social
expectations. And even when a person is not dissimulating in an
attempt to present a positive self-image, she or he may not have
the ability to report accurately what is taking place internally (p.
1154).
The latter charge was the substance of Nisbett & Wilson's (1977)
oft-cited critique of verbal reports as data. As they see it, selfreports amount to "telling more than we can know."
If verbal reports do not accurately reveal the
cognitive mechanisms underlying human
experience, this is not to say that they are not
revelatory of the meaning or "intentionality" of
human experience.
Moreover, we will see that to the extent that
verbal explanations are themselves human
behaviors, they constitute a legitimate realm of
investigation in their own right
(B) The Question of Validity in Regards
to the Researcher’s Process
•The participant’s description functions as a
medium…. through which, as meanings of the
subject's experience begin to resonate within the
researcher's own experience, the researcher gains
access to the world of the subject…. and at the
same time grasps this world as a function of the
subject's presence, or intentionality.
•“Intentionality” is the central phenomenon of
interest to the phenomenological psychologist
• The phenomenological approach requires that
the researcher enter into direct, personal, living
contact with the psychological event being
studied.
• The words of the research subject open up a
world of experience, and it is this world that the
researcher directly and vividly intuits through
the self-report data.
• The task of the researcher is essentially to
imagine the subject’s intentional relationship to
his or her world of experience through
empathy
Empathie, seule attitude requise pour comprendre.
-- J-P Sartre, L’Idiot de la familie
II. External Validity:
The Generalizability of Qualitative Findings
If “empathy” is used to “open up” the meaninghorizons of qualitative data at the individual level….
“free variation in the imagination” is utilized to attain
external reliability so that findings at the individual
level can begin to be generalized to wider contexts
The move towards universal structures of conscious
experiencing requires that the phenomenologist
shift the attention away from the individual
experience grasped for its own sake, and toward
the ‘category’ or class of experiences of which the
individual experience is now taken as merely an
example.
• In quantitative psychology, this would be the move
from the sample to the population.
• For the phenomenologist, it would be from the
'instance' to the 'category' – from “this” experience
of anger, to the category of “all experiences of
anger.”
III. The Question of Reliability
"Reliability refers to the consistency of a measuring
procedure or instrument.
A method of measurement is reliable if it always
produces the same result under the same
conditions" (Lewin, 1979).
• Reliability thus assumes that one can establish an equivalence
of measurement.
• In the modern sciences, "measurement" has come to mean
quantification according to an established standard or scale.
“Experiential Methodology”
The original meaning of the term “measurement” was
“description”
When qualitative researchers conduct research, their
“descriptive findings” consist of their own
verbalizations of their experience of the participants’
own descriptions of their experience.
hence reference to “experiential method”, which refers
both to the experience described by our informants,
and to the researcher’s empathic experience of their
participant’s experience.
Within my own situation, that of the [other] whom I am
questioning makes its appearance and, in this bipolar
phenomenon, I learn to know both myself and others.
… it is not a question … of reducing his experiences to
mine, or coinciding with him, or sticking to my own
point of view, but of making explicit my experience,
and also his experience as it is conveyed to me in my
own … and to understand one through the other”
(Merleau-Ponty, 1945/1962, p. 338).
So: how do we establish the “reliability” of a method that requires the
researcher to place herself or himself into the position of the
participant, by means of the other’s description of their experience?
In a comparative study of several qualitative researchers analyzing the
same data, what we noticed were “convergences” – as well as
divergences – among the individual researcher’s thematizations.
The points of convergence, in which researchers appeared to be making
the same observation while using slightly different words, established a
basis for considering this (phenomenological) method of observation as
‘reliable’.
Divergences were reconciled by asking each member of our team of
researchers if they could “see” what the other researchers were seeing,
by putting themselves into the position of the other researcher’s
standpoint. Each qualitative researcher brings his or her own unique
perspectives to bear - and the question is thus not whether on first
reading we all see the same thing.
Thus, the chief point to be remembered with this type of
research is not so much whether another position could be
adopted (this point is granted beforehand) but [rather]
whether a reader, adopting the same viewpoints as [those]
articulated by the researcher, can also see what the
researcher saw, whether or not he/she agrees with it. That is
the key criterion for qualitative research. (Giorgi, 1975, p.
96)
All research discloses only a limited truth, that is, a
truth limited by the researcher’s procedures and
perspective.
Phenomenological researchers attempt to articulate
those limits reflectively and honestly, and additional
limits may be discerned by others whose scholarship
and reflections bring additional perspectives and
procedures to bear.
The validity of research findings – as well as the
reliability of our method - is therefore not contingent
on whether they are entirely similar to those of other
viewpoints.
According to the phenomenological approach, it is not
possible to exhaustively know any phenomenon, and
different viewpoints can be valid (Churchill, Lowery, McNally,
& Rao, 1998; Wertz, 1986).
In other words, other perspectives, perhaps rooted in
different research interests, and their corresponding
intuitions, always are possible and contribute in a
complementary manner to our knowledge of “the whole.”
In the end, the value of the findings depends on their ability
to help others gain some insights into what has been lived
unreflectively. Other insights from different viewpoints may
then supplement, and thereby extend and possibly even
radically decenter, what always is essentially a partial
knowledge of human life.
To quote Jacques Derrida (1985, p. 4):
(everything comes down to the ear you are able to hear me with.)