Confirmation Bias
Critical Thinking
Among our critical thinking questions were:
Does the evidence really support the claim?
Is there other evidence that I should consider?
Sufficient Conditions
Consider the following claim: “you can get to
CUHK from Lingnan using public transportation.”
The claim is true if the MTR goes from Lingnan
to CUHK.
The claim is also true if there is a bus from
Lingnan to CUHK.
In fact, the claim is true if there is any way to get
from Lingnan to CUHK.
Sufficient Conditions
A claim X is a sufficient condition for the claim
“you can get to CUHK from Lingnan” when if X is
true, “you can get to CUHK from Lingnan” is
true. It’s true that if “the MTR goes from
Lingnan to CUHK” is true, then “you can get to
CUHK from Lingnan” is true.
Therefore, “the MTR goes from Lingnan to
CUHK” is a sufficient condition for “you can get
to CUHK from Lingnan.”
Sufficient Conditions
A helpful way of thinking about this:
Whenever “If A then B” is true:
A is a sufficient condition for B.
Examples
“If today is Thursday, then we have class” is true.
So, today’s being Thursday is a sufficient
condition for our having class.
“If you are convicted of murder, you will go to
jail” is true.
So, being convicted of murder is a sufficient
condition for going to jail.
Examples
But the following are not true:
“If we have class today, then today is Thursday.”
(we also have class on Monday)
“If you go to jail, then you were convicted of
murder”
(Some people in jail were convicted of other crimes)
Necessary Conditions
Suppose that “if A, then B” is true. We know
that this means that A is a sufficient condition
for B.
It also means that B is a necessary condition for
A.
Requirements for Office
Consider the requirements for Chief Executive in
Hong Kong:
• Must be at least 40 years old.
• Must be a Hong Kong permanent resident.
• Must not have right of abode in a foreign
country.
• Must have resided in Hong Kong for 20
consecutive years.
Necessary Conditions
Requirements for office are things that are
necessary for office– things that must be true
for you to hold office.
So we know that if someone is the Chief
Executive, then that person is at least 40 years
old.
In other words, being at least 40 (being a
permanent resident, etc.) is a necessary
condition for being Chief executive
“I know horoscopes can predict the future,
because I’ve seen it happen.”
“Positive thinking can cure cancer, because I
know someone who used it, and got better.”
“Everyone knows you do worse in your second
year in college, you see it all the time.”
Necessary and Sufficient Evidence
If claims like these are true, for example, if it’s
true that horoscopes predict the future, then
the evidence in question is necessary.
If horoscopes predict the future, then there
must be cases where a horoscope predicted
something, and then it happened.
Necessary and Sufficient Evidence
But such evidence is not sufficient for
establishing the truth of these claims.
When a horoscope predicts that X will happen,
and then X happens, that doesn’t prove
anything. Were there times horoscopes
predicted things that didn’t happen? Were there
things that horoscopes should have predicted,
but never happened?
Contingency Tables
Contingency Tables
A contingency table is used to plot predictions of
the form “If A happens, then B will happen.”
In the upper left corner of the table (Prediction
= Yes, Observation = Yes) are cases where A
happens and B happens. These are “true
positives” and they confirm the prediction.
Contingency Tables
A contingency table is used to plot predictions of
the form “If A happens, then B will happen.”
In the lower right corner (Prediction = No,
Observation = No) are cases where A does not
happen, and B does not happen. These are “true
negatives” and also confirm the claim in
question.
Contingency Tables
A contingency table is used to plot predictions of
the form “If A happens, then B will happen.”
Two sorts of cases disconfirm the prediction.
First there are false positives in the upper right
corner, where the predicted outcome
(Prediction = Yes) does not match the observed
outcome (Observation = No).
Contingency Tables
A contingency table is used to plot predictions of
the form “If A happens, then B will happen.”
Second, there are “false negative” cases. These
are also outcomes where the prediction
(Prediction = No) fails to match the observation
(Observation = Yes).
Perfect Correlation Claims
For “absolute” claims (If A happens, B will
always happen, A and B are perfectly positively
correlated), all cases must fall in the upper left
or lower right corners.
The prediction must always match the outcome.
Imperfect Correlation Claims
For probabilistic claims (A and B are imperfectly
positively correlated), you must compare the
proportion of true positives to predicted
positives to the proportion of false positives to
predicted negatives.
(How you should compare them depends on the
base rates…)
For example…
Suppose I claim that students are more likely to
get an A on the final if they take notes in class
than if they don’t take notes.
I point out 5 students who took notes and got
an A, and 18 students who didn’t take notes and
didn’t get an A as evidence for this claim. But
whether the claim is true depends on more than
just these true positives and true negatives…
I’m Right
Get an A
Take notes
5
Don’t get an Totals
A
10
15
Don’t take
notes
Totals
2
18
20
7
28
35
I’m Right
Here, it looks like taking notes matters. 5 out of
15 note-takers get A’s (33%) and only 2 out of 20
students who don’t take notes get A’s (10%).
However, with the same number of true
positives, I could also be very wrong. Let’s add
10 students to the “false positive” and “false
negative” squares…
I’m Wrong
Get an A
Take notes
5
Don’t get an Totals
A
20
25
Don’t take
notes
Totals
12
18
30
17
38
55
I’m Wrong
Even though the number of true positives and
true negatives stayed the same, we now see that
a measly 5 out of 25 note-takers got A’s (down
to 20% from 33%), but a full 12 out of 30
students who didn’t take notes (40% up from
10%) got A’s. Students who didn’t take notes are
more likely to get A’s, in this scenario.
Main Idea
The basic point is that you cannot use examples
that agree with a claim to show that that claim
is true.
You must also look at examples that disagree
with the claim. If there are enough of these,
then the claim might be false, even if there are
lots of examples that support it.
CONFIRMATION BIAS
Confirmation Bias
Even though evaluating predictions requires
looking at both the rates of true positives among
predicted positives, and the rates of false
positives among predicted negatives, human
beings have a tendency to only consider true
positives (and to a lesser extent, true negatives)
when evaluating predictions or other claims of
imperfect correlation.
Wason Selection Task
Around ½ of people studied say “D” and “5”.
About 1/3 say just “D”.
Only about 1/20 get the right answer: “D” and
“2”!
Searching for Confirmation
People have a preference for positive answers
that confirm their theories, even though
negative answers that disconfirm their theories
might give the same amount of information.
For example, suppose A picks a number
between one and ten, and you’re supposed to
guess what it is. I suggest that the number is 3.
The psychological research shows that you’d be
more likely to ask “is the number odd?” than “is
the number even?”– even though both answers
are equally informative.
In this case, the preference for confirmation
does not matter:
Asking the positive question and the negative
question give you the same information. If
people prefer the positive question, that doesn’t
harm them at all. But the preference can harm
them if the positive question gives less info.
A Strange Example
Americans were asked:
Which of these pairs of countries are more
similar to one another?
1. West Germany, East Germany
2. Sri Lanka, Nepal
They said (1), West Germany and East Germany.
Others (Americans) were also asked:
Which of these pairs of countries are more
different from one another?
1. West Germany, East Germany
2. Sri Lanka, Nepal
They also said (1).
Americans thought that West Germany and East
Germany were both more similar to each other
than Sri Lanka and Nepal and less similar to each
other than Sri Lanka and Nepal.
How is that possible?
First, when considering the question ‘which are
more similar?’ the subjects looked for all the
positive evidence that West Germany and East
Germany were similar, and all the positive
evidence that Sri Lanka and Nepal were similar.
Since Americans know nothing about Asian
countries, they had no positive reason to think
Sri Lanka and Nepal were similar.
Similarly, when asked ‘which are more
different?’ the subjects considered the positive
evidence that West Germany and East Germany
were different and the positive evidence that Sri
Lanka and Nepal were different.
Again, having no knowledge of Sri Lanka or
Nepal, Americans chose (1), because of all the
positive evidence in its favor.
But it cannot be true that East Germany and
West Germany are both more similar and more
different than Sri Lanka and Nepal.
What the subjects did not do is consider the
relevant negative evidence that would
disconfirm their hypotheses.
The Problem of Absent Data
Sometimes it’s not just that we only look for or
evaluate the positive evidence, but that there is
no negative evidence. This can lead us to think
we have very well-confirmed beliefs when we do
not.
Hiring Job Applicants
Suppose you’re hiring job applicants for your
shoe company. You think people who haven’t
studied a musical instrument would not be good
employees.
Who do you hire? The people who have studied
music, of course! And if they’re successful at
your company, do you have good reason to
believe that you were right?
No! You have positive evidence– people you
predicted would be successful, who are
successful– but you have no negative evidence.
What about all the people you didn’t hire, the
ones who didn’t study music? They might have
been successful too. They could’ve been more
successful. You don’t know.
Absent evidence is all around us.
Suppose you decide to major in accounting
instead of philosophy. You find that you are very
happy studying accounting. Did you make the
right choice? You can’t know. You could have
been more happy studying philosophy. There’s
just no evidence.
Self-Fulfilling Prophecies
The Prisoner’s Dilemma
There are two general strategies to playing the
prisoner’s dilemma. You can view the game as
one where the goal is for everyone to do well,
and thus play “cooperatively” or you can view
the game as one where the goal is for you to do
better than your opponent, and thus play
“competitively”.
The Prisoner’s Dilemma
Believing that the goal is selfish, to win more for
yourself, is a self-fulfilling prophecy.
If you play against other selfish players, they will of
course play competitively.
But even if you play against cooperators, they will
have to play competitively, or get 0 every round. So
it will seem as if there are no cooperators.
Another Example
Suppose you think I’m not a very nice person, so
you avoid me.
If you avoid me, then you’ll never have a chance
to correct your initial impression.
So if I’m a nice person, you’ll never find out if
you start out thinking I’m not.