EMPIRICAL REPORT
Evaluating Scientific Research
in the Context of Prior Belief:
Hindsight Bias or
Confirmation Bias?
Amy M. Masnick, PhD
Hofstra University
Corinne Zimmerman, PhD
Illinois State University
When there is a mismatch between new evidence and prior beliefs, do people reject the conclusions from
this evidence because of confirmation bias or do they support them because of hindsight bias? Ninety-four
participants expressed a belief about a study’s outcome before reading a research report. When belief was
confirmed, the study’s methodology was subsequently rated more positively and findings (whether presented with or without an explanation) were rated as more obvious, important, and interesting than when
beliefs were disconfirmed. However, the presence of an explanation for the reported findings affected
ratings of obviousness and interestingness but not of methodology. These results indicate that judging a
research finding to be obvious involves more than a simple hindsight bias.
Keywords: confirmation bias; hindsight bias; evidence evaluation; reasoning; obviousness
M
ost people have conceptions about how the
natural and social world works. These beliefs may be held with varying degrees of
conviction. At the same time, people encounter new
information every day that may call into question
some of these pre-existing beliefs. When new knowledge and prior beliefs conflict, how do individuals
evaluate and reconcile the two? Reconciling existing
beliefs with new evidence is one of the key components of scientific reasoning (across the lifespan), but
it is also necessary for reasoning about the scientific
findings individuals are exposed to regularly through
a variety of media sources. Therefore, the answer to
how individuals reconcile conflicting beliefs and evidence is important not only for understanding the
cognitive processes involved in scientific reasoning,
but it also has practical implications for understanding scientific literacy and reasoning about science.
Personal, professional, and public policy decisions
may be informed by evaluating research findings
(Miller, 2004; Shapin, 1992), and these evaluations
may include assessing empirical findings, theoretical explanations, or both. However, past research in
cognitive science and social cognition leads to different conclusions about how people evaluate new
information. On one hand, there is evidence that
people are reluctant to consider evidence that disagrees with their beliefs and have a bias to look for
information that confirms prior beliefs. (In this article, we consider confirmation bias as a bias toward
one’s initial beliefs rather than as a hypothesis-testing
strategy [e.g., Klayman & Ha, 1987].) On the other
hand, there is evidence that people are susceptible to
hindsight bias; that is, they agree with newly learned
conclusions and claim to have held these beliefs all
along.
Journal of Psychology of Science and Technology, Volume 2, Number 1, 2009 © Springer Publishing Company
DOI: 10.1891/1939-7054.2.1.29
29
30
Numerous studies of scientific reasoning have
shown that it is difficult to integrate evidence disconfirming prior belief (see Zimmerman, 2000, 2007, for
reviews). In particular, when new evidence challenges
an existing causal belief, that belief is more resistant
to change than when a noncausal belief is challenged
(e.g., Kanari & Millar, 2004). Additionally, people are
particularly likely to disbelieve new evidence when a
theoretical reason or plausible causal mechanism to
maintain the current belief is present, and are more
likely to take seriously an implausible covariation between two factors when there is a plausible causal
explanation (Ahn, Kalish, Medin, & Gelman, 1995;
Koslowski, 1996).
Similar types of studies on attitude change have
also looked at how individuals evaluate evidence in
the context of prior belief. Participants in these studies evaluate evidence about controversial topics for
which they often have strong prior beliefs (e.g., gun
control, capital punishment). For example, Munro,
Leary, and Lasane (2004) found that participants
whose beliefs were disconfirmed by a fictitious study
about the relationship between homosexuality and
mental illness were more critical of the study’s methodology than those whose beliefs were confirmed. In
general, biased assimilation and attitude polarization
are common when evaluating evidence (e.g., Lord,
Ross, & Lepper, 1979). That is, research is evaluated
more favorably if it supports initial attitudes (biased
assimilation), and rather than becoming more moderate in the face of disconfirming evidence, attitudes
often become more extreme (attitude polarization;
see MacCoun, 1998, for a review).
In contrast to research showing that prior beliefs
are resistant to change, other research evaluation
studies have focused on the perception that, independent of prior belief, social science and educational
research findings are considered “obvious” or “common sense” (e.g., Gage, 1991; Yates, 2005). In these
studies, participants are shown either true findings
or their foils (i.e., false findings). When presented
with research findings from the areas of personality
(Barnett, 1986), developmental psychology (Barnett,
Knust, McMillan, Kaufman, & Sinisi, 1988), and
social psychology (Richard, Bond, & Stokes-Zoota,
2001), accuracy in distinguishing true findings from
foils ranged from 66%–75% (chance is 50%). Using
a similar methodology, Wong (1995) found that
MASNICK AND ZIMMERMAN
when presented with both a true finding and a foil
about educational research, participants were equally
likely to select either version as the actual finding.
Moreover, ratings of obviousness were equivalent for
findings and nonfindings. Results presented with an
explanation for the finding, however, were rated as
significantly more obvious than those without an explanation, for both actual findings and foils.
These findings are consistent with research on
hindsight bias (e.g., Guilbault, Bryant, Brockway,
& Posavac, 2004). That is, when individuals have
knowledge of an outcome, they tend to overestimate
their knowledge in advance of knowing that outcome
(i.e., the “I knew it all along” phenomenon). Descriptions of hindsight typically include two components:
(a) overestimation of one’s ability to predict an outcome after the fact, and (b) the belief that one is not
in fact influenced by the knowledge of the outcome
(Hawkins & Hastie, 1990). More recent discussions,
however, distinguish several subtypes of hindsight
bias (Blank, Nestler, von Collani, & Fischer, 2008).
Here we concentrate on a hindsight bias, focusing on
one’s ability to assess the likelihood of an outcome
after knowing the outcome, as measured by ratings
of how obvious research findings are after one has
read them. Wong’s findings suggest that this type of
hindsight bias may be even stronger in the presence
of a plausible explanation for the reported finding.
Most previous research examining the obviousness of findings has not specifically examined the
effects of prior expectations about the findings. Past
research on scientific reasoning and social cognition
typically uses topics for which individuals have prior
expectations, due either to ingrained or politicized
beliefs or to fairly robust science misconceptions. The
current study explored whether findings from newly
reported research would be subject to a “knew-itall-along” hindsight bias when a prior belief is expressed. We also explored whether the presence of
an explanation for a finding influences evaluations of
obviousness of a study after having one’s belief either
confirmed or disconfirmed.
We created a task that includes characteristics used by researchers studying reasoning, attitude
change, and perceptions of the obviousness, importance, and interestingness of research findings.
We explored the process of evaluating educational research findings because individuals have been shown
EVALUATING RESEARCH
to be less accurate in discriminating true findings
and foils in this domain (Wong, 1995). If evaluating
research after expressing a belief about the outcome
leads to the confirmation bias found when evaluating evidence that challenges or confirms political attitudes (e.g., Lord et al., 1979; Munro et al., 2004),
then evaluations should differ as a function of prior
belief. Specifically, evaluations about the methodology used to conduct the study should be more positive after learning that the result confirms prior belief
(and negative when results disconfirm prior belief ).
Overall evaluations of the research would be more
positive by those whose beliefs are confirmed by
a study’s findings because participants would be
allowing prior beliefs to bias interpretation of new
information.
Moreover, the presence of an explanation may
lead to different cognitive processing based on prior
belief. Explanations for confirmed beliefs may result
in positive evaluations, whereas explanations for disconfirmed beliefs may result in negative evaluations,
as predicted by confirmation bias. Alternatively, if
simply reading about a research result can make a
finding seem obvious in hindsight, then participants’
evaluations about study quality and the obviousness
of results should be similar regardless of prior expectations about the findings (e.g., Richard et al., 2001;
Wong, 1995). In addition, the presence of an explanation for the findings may affect evaluations even
when a belief is challenged. Investigations of scientific reasoning have shown that the ability to reconcile beliefs and evidence is often mediated by the
presence of (or ability to generate) plausible causal
mechanisms for the pattern of evidence/data (e.g.,
Ahn et al., 1995; Koslowski, 1996; Wong, 1995).
When reasoning about scientific research, it is reasonable to assume that individuals are also considering theoretical explanations or causal mechanisms
(e.g., Chinn & Brewer, 2001). That is, hindsight bias
may be more evident when explanations are present,
such that an explained finding may seem even more
obvious regardless of prior belief.
In sum, the present study explored two factors
that could influence the evaluation of a research
study: (a) having one’s belief about an educational
research question confirmed or disconfirmed by evidence, and (b) the presence or absence of a causal
explanation for the finding.
31
METHOD
Participants
Ninety-four undergraduate students (82 women,
12 men) participated for course credit. The average age was 22.1 (SD = 4.1). They had an average of
3.8 years of college-level instruction (SD = .74),
with an average of 1.8 (SD = 1.1) methods/statistics
courses. The racial composition of the sample was
73% White, 12% African American, and 7% each
Hispanic and Asian. Most participants indicated
they were majoring in psychology (37%), education
(30%), or psychology and education (13%).
Materials and Procedure
A questionnaire was administered in small groups.
The first page briefly described an issue in early science education. Participants were told that researchers have been studying the relative efficacy of direct
instruction (being taught information explicitly) as
compared to discovery learning (being given a goal
and some materials and told to explore on one’s own).
Participants were asked to indicate which method
they believed was the most effective (i.e., a forcedchoice question). The procedure of requiring a forced
choice between the two sides of this issue allows participants to be later classified as having their belief
confirmed or disconfirmed by the result of a study
(Munro et al., 2004).1 All participants then read the
same one-page description of an experimental study,
including a brief introduction (background and research questions) and method section. Participants
then evaluated the appropriateness of the methods,
design, participants, and measures on a 7-point Likert scale (from strongly disagree to strongly agree that
each was appropriate).
After evaluating the study’s methodology, participants were randomly assigned to read one of four
versions reporting the study’s findings. In two versions, direct instruction was described as the more effective intervention. One version asserted this finding
without an explanation. The other version included
the explanation that direct instruction was more effective because learning is facilitated by the teacher’s
organized presentation and focus on relevant concepts and procedures. In the other two versions, discovery learning was described as the more effective
32
intervention. One version included the explanation
that discovery learning is more effective because of
students’ active involvement in the learning process,
which makes the information more meaningful and
likely to be remembered.
After reading the findings (with or without an
explanation) participants rated characteristics of
the conclusions of the study using a 7-point Likert
scale. Participants rated how strongly they disagreed
or agreed with statements about the conclusions
with respect to whether they were (a) obvious,
(b) important, and (c) interesting. Participants then
completed a second evaluation of the methods, design, participants, and measures. Participants were
debriefed at the end of the study about the study’s
purpose.
RESULTS
Based on the initial forced-choice question, one-third
of the students believed that direct instruction was
more effective and two-thirds believed that discovery
learning was more effective. Because of random assignment to which results were seen, approximately
equivalent groups had initial belief confirmed (n = 46)
and disconfirmed (n = 48). To assess the equivalence
of versions, ratings were initially analyzed disregarding prior belief. Ratings of the obviousness of the reported findings were similar for those who read the
discovery-learning conclusion (M = 4.9, SD = 1.4;
n = 46) and the direct-instruction conclusion (M = 4.6,
SD = 1.4; n = 48). In addition, no differences were
found for the presence of an explanation or in initial
or postfinding evaluations of methodology based on
which version participants read (all Fs ≈ 1, ps > 0.10).
Thus, the content of the conclusion participants read
(discovery learning or direct instruction) did not influence participants’ reasoning about the study. We
therefore focused on whether participants’ beliefs
were confirmed or disconfirmed, disregarding prior
belief. The seven ratings participants gave (method,
design, participants, measures, obviousness, importance, and interestingness) were entered into a multivariate 2 (belief: confirmed vs. disconfirmed) × 2
(explanation: present vs. absent) analysis of variance
(ANOVA).
MASNICK AND ZIMMERMAN
Judgments About Methodology
No differences in initial ratings of appropriateness of
the method, design, participants, or measures were
found as a function of either belief or explanation
(Fs ≈ 1, ps > 0.10) so analyses were conducted on
change scores (second minus first assessment). Ratings of the appropriateness of the methods increased
for those whose belief was confirmed (M = .50;
SD = 1.1) but decreased for those whose belief was
disconfirmed (M = −.56; SD = 1.6), F(1, 90) = 13.3,
p < .001, partial η2 = .13 (Figure 1). No differences
were found for the presence of an explanation, and
there was no interaction between explanation and belief confirmation (Fs ≈ 1, ps > 0.10).
Participants’ ratings of the design used by researchers increased when beliefs were confirmed
(M = .74, SD = 1.1) but decreased (M = −.35, SD = 1.1)
when beliefs were disconfirmed, F(1,90) = 13.9,
p < .001, partial η2 = .13. No differences were found
for the presence of an explanation, and there was no
interaction (Fs ≈ 1, ps > 0.10). The pattern was similar to that illustrated in Figure 1. There were small
positive changes in ratings about the appropriateness
of the participants and measures but no differences
among groups (all Fs ≈ 1, ps > 0.10).
Judgments About the Findings
and Conclusions
Participants’ ratings of the obviousness of the conclusion were greater when initial belief was confirmed (M =
5.1, SD = 1.4) than when initial belief was disconfirmed
(M = 4.4, SD = 1.4), F(1,90) = 5.78, p = .018, partial
η2 = .06. Although there was no main effect of explanation, there was an interaction between belief confirmation and explanation, F(1,90) = 4.26, p = .042, partial
η2 = .05 (Figure 2). When there was no explanation for
the finding, ratings were similar for those whose beliefs
were confirmed or disconfirmed. Ratings of obviousness
were much higher for those whose belief was confirmed
compared to those whose belief was disconfirmed,
F(1,90) = 9.70, p = .002, partial η2 = .10. That is, when
findings were presented with an explanation, there was
a large difference in perceived obviousness.
Participants’ ratings of the importance of the research were also greater when beliefs were confirmed
EVALUATING RESEARCH
FIGURE 1. Mean change in appropriateness of methods rating (before and after reading
the results of the study) as a function of whether one’s belief was confirmed or
disconfirmed by the findings, and whether or not an explanation for the findings
was presented.
FIGURE 2. Mean ratings of the obviousness of the findings/conclusions as a function
of whether one’s belief was confirmed or disconfirmed by the findings, and
whether or not an explanation for the findings was presented.
33
34
(M = 6.1, SD = 1.3) than when beliefs were disconfirmed (M = 5.4, SD = 1.4), F(1,90) = 7.94, p =
.006, partial η2 = .08. The presence of an explanation resulted in higher ratings of the findings as important, F(1,90) = 4.06, p = .047, partial η2 = .04,
but the presence of an explanation did not interact
with belief. The same pattern was found for ratings of
interestingness, with higher ratings from those whose
belief was confirmed, F(1,90) = 12.09, p = .001, partial η2 = .19, and for those who read an explanation
for the finding, F(1,90) = 7.80, p = .006, partial η2 =
.08, with no interaction.
DISCUSSION
The results demonstrate that when evaluating reports
of social science research, the tendency toward hindsight bias is trumped by beliefs expressed before evaluating the findings. Evidence that contradicts initial
belief is assessed differently from evidence that supports it. Consistent with predictions from research on
attitude change and scientific reasoning, there was a
strong effect of confirmation on ratings of methodology, design, and judgments of several qualities of
the research, such as its obviousness. This finding
indicates that when stating an expectation about the
finding prior to reading the outcome, participants
are more likely to indicate a confirmation bias than
a hindsight bias.
Both the methods and the findings were rated
more positively after a belief was confirmed and more
negatively after a belief was disconfirmed. Recall that
both findings were judged to be equivalent, independent of prior belief (i.e., the discovery learning and
direct instruction versions were rated equally obvious), and methodology ratings were equivalent prior
to reading the findings. Thus, it was not the specific
content that affected evaluation but rather the confirmation or disconfirmation of prior beliefs. Hindsight
bias was not in evidence: people remained tethered
to their initial beliefs more than the newly presented
conclusions, as indicated by the effect of confirmation
on the change in ratings of the appropriateness of the
methodology. This finding suggests that working to
change someone’s view simply based on content may
be unsuccessful without considering initial views on
the topic. Evaluating the methodological soundness
MASNICK AND ZIMMERMAN
of a study may also be influenced by pre-existing beliefs. One difference between the current study and
many other studies of hindsight bias (e.g., Hawkins
& Hastie, 1990) is that research findings can be assessed both on the nature of the conclusions and
on methods by which the conclusions were drawn
(unlike, for example, the assessment of a historical
event). This distinction may be one reason we found
little evidence of hindsight bias.
Assessments of the appropriateness of participants
(i.e., generalizabililty to other populations) and measures (i.e., construct validity) were not influenced by
prior belief. Participants may have been able to evaluate these features of the study more objectively, or
they may not have believed these characteristics were
as important in evaluating a study’s validity. Having
one’s belief confirmed also resulted in ratings of the
reported findings as more obvious, interesting, and
important.
In contrast to previous research showing a type
of hindsight bias in the evaluation of research findings, when prior belief was taken into consideration,
ratings of obviousness varied as a function of that
prior belief. If hindsight bias were at work, we would
expect equivalent (and positive) obviousness ratings
regardless of whether a prior belief was confirmed or
challenged. Although lower ratings of obviousness
do not necessarily indicate participants are discounting or discrediting the study, the differences in ratings
are still changing based on confirmation.
Two possible explanations seem plausible for this
effect. First, although we presume participants’ initial
opinions on the topic (i.e., educational interventions)
were formed quickly and on the spot, most participants were psychology or education majors (80% of
the sample), and so they may have had occasion to
consider such topics prior to participation. If so, then
challenging or confirming those beliefs would be expected to result in behavior similar to that observed
when other strongly held attitudes are challenged
(e.g., Munro et al., 2004). Second, it is also possible
that being forced to state an opinion early, even if
chosen somewhat arbitrarily, leads to a commitment
to the position and consistency within the task. To
explore this possibility further, it will be necessary
to replicate this finding with a group that does not
express an opinion prior to reading the methods and
findings.
EVALUATING RESEARCH
The presence of an explanation for a finding led
to higher ratings of how important and interesting
that finding was. These results are consistent with
past work indicating that explanations are a critical part of reasoning about science (e.g., Ahn et al.,
1995; Koslowski, 1996; Wong, 1995). Individuals
are more likely to believe an empirical finding if there
is a theory or explanation for that finding. Thus, it is
unsurprising that the presence of explanatory information would increase perceptions of how important
and interesting a topic is.
Yet the explanation effect was not completely
straightforward. An unexpected finding was the interaction of explanation and belief confirmation on
ratings of obviousness. Wong (1995) found that the
presence of an explanation increased feelings of obviousness for both true findings and foils. We found
similar results, but only under certain conditions. In
the absence of an explanation, findings were rated as
equally obvious regardless of prior belief. However,
when an explanation was provided, a confirmed belief
was rated as much more obvious than a disconfirmed
belief. Thus, the absence of an explanation actually
eliminated the effect of prior belief. It is possible
that participants generate their own explanations for
the results and therefore rate the findings as equally
obvious. In addition, with disconfirming evidence,
participants may become more critical and discredit
both the finding and the explanation for that finding
to maintain their belief, leading to a lower rating of
obviousness (Klaczynski & Narasimham, 1998). If a
belief has just been disconfirmed, then the accompanying explanation is presumably one that has either
not been considered earlier or has been considered
and dismissed.
In summary, the methodology developed in the
current research can be productive for examining the
process of evidence evaluation and for making links
among several research literatures. The findings add
to the literature on scientific reasoning by showing
that the presence of an explanation does not always
affect evaluation of scientific reports in a uniform
way. Consistent with previous research on confirmation bias, we found that prior belief exerts a powerful
influence, but our results suggest this effect is not diminished by the presence of a plausible explanation.
The current findings also contribute to the hindsight
bias literature by demonstrating that although partici-
35
pants often consider research findings to be obvious,
there are some boundary conditions for such judgments (e.g., belief confirmation or disconfirmation
in conjunction with the presence of an explanation).
The judgment of a research finding to be obvious
clearly involves more than a simple hindsight bias.
Additional research is needed to further explore the
complexities of evaluating the kinds of research evidence commonly reported in the media that may be
used to inform personal and policy decisions.
NOTE
1. Munro et al. (2004) allowed individuals to indicate
ambivalence by selecting a midpoint on a Likert scale when
expressing views about homosexuality. This procedure resulted in removing only a small minority of individuals
from their analyses (4 of 92).
REFERENCES
Ahn, W., Kalish, C. W., Medin, D. L., & Gelman, S. A. (1995).
The role of covariation versus mechanism information
in causal attribution. Cognition, 54, 299–352.
Barnett, M. A. (1986). Commonsense and research findings
in personality. Teaching of Psychology, 13, 62–64.
Barnett, M. A., Knust, J., McMillan, T., Kaufman, J., & Sinisi,
C. (1988). Research findings in developmental psychology: Common sense revisited. Teaching of Psychology,
15, 195–197.
Blank, H., Nestler, S., von Collani, G., & Fischer, V. (2008).
How many hindsight biases are there? Cognition, 106,
1408–1440.
Gage, N. L. (1991). The obviousness of social and educational research results. Educational Researcher, 20, 10–16.
Guilbault, R. L., Bryant, F. B., Brockway, J. H., & Posavac,
E. J. (2004). A meta-analysis of research on hindsight
bias. Basic and Applied Social Psychology, 26, 103–117.
Hawkins, S. A., & Hastie, R. (1990). Hindsight: Biased judgments of past events after the outcomes are known.
Psychological Bulletin, 107, 311–332.
Kanari, Z., & Millar, R. (2004). Reasoning from data: How
students collect and interpret data in science investigations. Journal of Research in Science Teaching, 41,
748–769.
Klaczynski, P. A., & Narasimham, G. (1998). Development
of scientific reasoning biases: Cognitive versus egoprotective explanations. Developmental Psychology, 34,
175–187.
Klayman, J., & Ha, Y. (1987). Confirmation, disconfirmation, and information in hypothesis-testing. Psychological Review, 94, 211–228.
36
Koslowski, B. (1996). Theory and evidence: The development
of scientific reasoning. Cambridge, MA: MIT Press.
Lord, C. G., Ross, L., & Lepper, M. R. (1979). Biased assimilation and attitude polarization: The effects of prior
theories on subsequently considered evidence. Journal
of Personality and Social Psychology, 37, 2098–2109.
MacCoun, R. J. (1998). Biases in the interpretation and
use of research results. Annual Review of Psychology, 49,
259–287.
Miller, J. D. (2004). Public understanding of, and attitudes
toward, scientific research: What we know and what
we need to know. Public Understanding of Science, 13,
273–294.
Munro, G. D., Leary, S. P., & Lasane, T. P. (2004). Between
a rock and a hard place: Biased assimilation of scientific information in the face of commitment. North American Journal of Psychology, 6, 431–444.
Richard, F. D., Bond, C. F., & Stokes-Zoota, J. J. (2001).
“That’s completely obvious . . . and important”: Lay
judgments of social psychological findings. Personality
and Social Psychology Bulletin, 27, 497–505.
MASNICK AND ZIMMERMAN
Shapin, S. (1992). Why the public ought to understand
science-in-the-making. Public Understanding of Science,
1, 21–30.
Wong, L. Y. (1995). Research on teaching: Process-product
research findings and the feeling of obviousness. Journal of Educational Psychology, 87, 504–511.
Yates, G. C. R. (2005). “How obvious”: Personal reflections on the database of educational psychology and
effective teaching research. Educational Psychology, 25,
681–700.
Zimmerman, C. (2000). The development of scientific reasoning skills. Developmental Review, 20, 99–149.
Zimmerman, C. (2007). The development of scientific thinking skills in elementary and middle school.
Developmental Review, 27, 172–223.
Correspondence regarding this article should be directed
to Amy M. Masnick, Hofstra University, Hauser Hall, Psychology Department, Hofstra University, Hempstead, NY,
11549. E-mail: [email protected]