The CRT: Roles of Intuition and Calculation
Aleksandr Sinayev & Ellen Peters
The Ohio State University
Abstract
Scores on the Cognitive Reflection Test (CRT) have been
linked with dual-system theory and normative decisionmaking patterns. In particular, the CRT is thought to measure
monitoring of System 1 intuitions such that, if Cognitive
Reflection is high enough, intuitive errors will be detected and
the problem resolved. However, CRT items also require
numeric ability to be answered correctly and it is unclear how
much numeracy vs. Cognitive Reflection contributes to better
decision making. In two studies, CRT responses were used
to calculate Cognitive Reflection (proportion of non-intuitive
CRT responses) and Calculation (proportion correct
responses out of non-intuitive responses). CRT Calculation
was a better predictor than Cognitive Reflection of decision
biases and financial outcomes. A standard numeracy scale
accounted for the findings of both CRT components. These
findings indicate that correlations with the CRT are
insufficient evidence to implicate overriding intuitions in
decision making biases and that numeric skills play a
substantial role in good decision making.
Study 1
Frame Inconsistency
Understanding America Study panel (n=859)
5 CRT items (Toplak et al., 2014), 6 numeracy items (Weller et al., 2012)
Risk consistency subscale of the ADMC (good internal, external validity; Bruine de Bruin, Parker &
Fischhoff, 2007). Chosen because it correlates with numeracy (Del Missier et al., 2012) and it
includes conjunction effects which correlate with the CRT. Divided into:
•
•
•
Frame consistency (e.g., probability of getting into a car accident, probability of driving accident free)
Conjunction subset vs. superset. (e.g., going to a dentist to get a cavity fixed, going to a dentist for any
reason)
Conjunction time. (e.g., Going to a dentist in the next year, going to a dentist in the next 5 years.) Did not
correlate with Cognitive Reflection or numeracy, so it is excluded
Cognitive Reflection and Calculation were modeled using IRTrees, but were calculated for this
presentation as follows:
# 𝐶𝐶𝐶𝐶𝐶𝐶𝐶
# 𝑁𝑁𝑁 𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼
𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 =
𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶𝐶 𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅𝑅 =
#𝑁𝑁𝑁 𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼𝐼
𝑇𝑇𝑇𝑇𝑇 𝑃𝑃𝑃𝑃𝑃𝑃𝑃𝑃
As expected, Cognitive Reflection correlated with frame consistency and conjunction (subset
vs. superset). Also as expected, numeric ability accounted for these effects (see right).
Without
With
Without
With
Numeracy Numeracy Numeracy Numeracy
0.96
Intercept
(0.07)
Cognitive
-0.25
Reflection (0.12)
-0.25
Calculation
(0.09)
-Numeracy
F
Adj. R2
American Life Panel (n=1478, varies by measure)
• 3 CRT, 6 non-CRT numeracy items
• All measures from Study 1
• Under/overconfidence (deviation of accuracy from confidence). Expected to correlate with
Cognitive Reflection independentof numeracy. (Hoppe & Kusterer, 2011; Del Missier et al, 2012)
• Incentivized risky choice (choose between a gamble with higher maximum payoff and a gamble with
higher minimum payoff; Holt & Laury, 2002)
Risk aversion= # higher minimum payoff gambles chosen
Inconsistency= # choices to be reversed to make preferences consistent
Incentivized intertemporal choice (payment for survey now or 110% of earnings in 2 weeks)
•
• Financial outcomes
Avoiding predatory lending, being denied credit, late payments on loans, paying credit cards on time and
in full
Expected to correlate with Cognitive Reflection because limiting excessive spending requires selfregulation (Vohs & Faber, 2003)
Expected to correlate with numeracy because avoiding debt, making savings requires understanding of
exponential growth
• But the CRT also measures mathematical ability. Its items load
on one factor with numeracy items in 4 of 5 published factor
analyses (Låg et al., 2014; Baron et al., 2014, Liberali et al., 2012)
• CRT items involve a two-step process:
A bat and a ball cost $1.10. The bat costs $1.00 more than the
ball. How much does the ball cost?
12.9
0.03
0.74
(0.05)
-0.02
(0.09)
-0.10
(0.07)
-0.51
(0.10)
17.1
0.05
Study 2
• Many biases are thought to arise because intuitions go
unchecked.
• Cognitive Reflection, as tested by the CRT, is the ability to
check intuitions and is thought to explain why the CRT
correlates with good decision making.
0.61
(0.05)
-0.19
(0.08)
-0.22
(0.06)
--
parentheses of regressions predicting frame inconsistency and conjunction
errors. Numeracy scored as proportion correct.
Methods
• In one dual-system explanation, System 1 first summons
Intuitions, then System 2 checks and sometimes engages more
processing (Kahneman & Frederick, 2012; Kahneman, 2013).
9.06
0.02
1.06
(0.08)
-0.13
(0.13)
-0.17
(0.10)
-0.35
(0.15)
8.00
0.02
Table 1. Unstandardized beta coefficients with standard errors in
Study 2 examines real financial outcomes and decision tasks that have correlated with CRT
scores in past studies.
Introduction
Conjunction
(subset vs. superset)
• Composites were calculated for decision biases and financial outcomes
Results
•
Study 1 results replicated.
• When biases were considered separately, only inconsistency correlated with Cognitive
Reflection (r=-0.14), but the effect was accounted for by numeracy (r=-0.26) in multiple
regression. We did not find a bias that correlated with Cognitive Reflection independently.
• Numeracy and Calculation are both significant independent predictors of a decision bias
composite. Effects of Cognitive Reflection were accounted for by IQ.
• Financial outcomes correlated with Cognitive Reflection (r=0.11), Calculation (r=0.10) and
numeracy (r=0.14). Of these, only numeracy was a significant independent predictor
(Bockenholt, 2012;
Intercept
Education
Income
Bracket
Age
(Decades)
Sex
(1=Male)
IQ
Cognitive
Reflection
Calculation
Numeracy
F
2
Adj. R
Decision Bias
Composite
Without
With
Numeracy Numeracy
0.19
0.09
(0.15)
(0.15)
0.01
0.01
(0.01)
(0.01)
-0.01
-0.01
(0.01)
(0.01)
0.01
0.01
(0.01)
(0.01)
0.02
0.00
(0.04)
(0.04)
-0.15
-0.12
(0.04)
(0.04)
-0.03
0.01
(0.06)
(0.06)
-0.20
-0.15
(0.05)
(0.05)
-0.31
-(0.11)
9.0
8.9
0.06
0.07
Financial Outcome
Composite
Without
With
Numeracy Numeracy
-0.57
0.54
(0.05)
(0.05)
0.005
0.003
(0.003)
(0.003)
0.004
0.004
(0.002)
(0.002)
0.028
0.029
(0.004)
(0.004)
-0.02
-0.02
(0.01)
(0.01)
0.005
0.006
(0.002)
(0.003)
0.02
0.01
(0.02)
(0.02)
0.02
-0.00
(0.02)
(0.02)
0.09
-(0.04)
13.05
12.39
0.75
0.069
Table 2. Stepwise regression results, predicting Decision bias composite and
financial outcomes composite from demographic variables, IQ and our target
variables. We started with demographic variables in step 1, added IQ in step 2,
Cognitive Reflection in step 3, Calculation in step 4 and finally, numeracy in step 5.
Steps where the new variable was not significant are omitted.
Campitelli & Gerrans, 2014)
• Numeric skill may underlie performance in many decision
making tasks (Peters et al., 2006)
• We examine whether the power of the CRT in predicting
decision making and financial outcomes is due to the ability to
check intuitions or make calculations.
Contact Aleksandr Sinayev: [email protected]
Conclusions
Cognitive Reflection predicted poorly
CRT results depend on numeracy
CRT remains an important scale
Other scales to consider
Checking Intuitions may not be important for the
tasks examined
Possibly because some tasks were incentivized,
participants resemble the general population, not
highly selective university students
Calculation may measure facets of numeric
ability not covered by other numeracy items OR
Calculation on CRT may simply be more difficult
and effects of numeracy may be curvilinear
Generates interesting research questions
Impressive validity for a short test
Dual-system theory describes performance
on this task well
Baron et al (2014) suggest syllogisms to
measure Cognitive Reflection separate
from Calculation
Inhibition can be measured by executive
tasks like Stroop, Go-no-Go, etc.
This material is based upon work supported by the National Science Foundation under Grant No. SES-1047757 and by the Behavioral Decision Making Initiative. Any opinions, findings, and
conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF or the BDM.