FUR XII, LUISS, Roma, June 24, 2006
Financially Stimulated Effort Hits
Individual Cognitive Constraints:
Evidence From a Forecasting Task with Varying
Working Memory Load
by Ondrej Rydval, CERGE-EI, Prague
Acknowledgements: invaluable comments especially from Andreas
Ortmann and Nat Wilcox, financial support from Grant Agency of the Czech
Republic and Hlavka Foundation
COMMENTS WELCOME!!!
MOTIVATION
I examine how performance-contingent financial incentives interact with
intrinsic motivation and cognitive constraints in determining individual
differences in cognitive performance.
Camerer and Hogarth (1999, JRU) propose a capital-labor framework
describing how financial incentives may interact in non-trivial ways with
intrinsic motivation to induce cognitive effort (labor), and how cognitive
effort productivity may vary across individuals due to their different cognitive
constraints (capital).
Even if salient financial incentives induce high effort, both financial and
cognitive resources may be wasted for individuals whose cognitive
constraints inhibit performance improvements.
This prediction, if warranted, calls for attention to individual cognitive
constraints in designing efficient incentive schemes in firms, experimental
settings, and elsewhere.
The next slide shows the main blocks of the capital-labor framework…
Here is how one can think of the capital-labor framework.
I briefly outline the literature and the main research questions:
financial
incentives
Camerer & Hogarth (1999)
Labor
Cognitive
theory
Capital-Labor
Framework
of
Production
cognition
Crowding out?
cognitive
performance
intrinsic
motivation
Benabou & Tirole trilogy,
Cacioppo et al. (1996)
cognitive
effort
Labor theory of cognition:
Smith & Walker (1993),
Wilcox (1993)
cognitive
capital
Degree of
complementarity?
Experimental psychology:
Engle & Kane (2004)
DESIGN
I provide an initial empirical test of the capital-labor framework, focusing
on the complementarity of cognitive capital and effort.
To impose theoretical structure on the framework, one can broadly think of
cognitive constraints as a vector composed of general cognitive capital and
task-specific capital.
Drawing on contemporary cognitive psychology, I measure individual
differences in general cognitive capital by a working memory span test –
a strong and robust predictor of general fluid intelligence as well as
performance in cognitive tasks requiring controlled information processing.
Since pre-existing task-specific capital (think of expertise) is vital for
performance in many field cognitive tasks but is hard to measure,
I intentionally minimize its potential relevance by designing a controlled
laboratory experiment where working memory is itself the main component of
task-specific capital, aside expertise acquired endogenously through on-task
learning.
The next slide shows what working memory is and why it is a useful measure of
individual differences in cognitive capital…
Working memory is a domain-general ability to control and rapidly
reallocate attention among competing cognitive uses, over and
beyond domain-specific short-term memory capacity. People with high
working memory are better able to code and store a limited amount of taskrelevant information and keep this information accessible during the
execution of complex, information-interfering cognitive and behavioral
tasks.
A typical working memory span test has two interacting components:
- processing component: e.g. calculating simple equations “(9/3)-2=?”
- memory component: e.g. memorizing a sequence of letters
1. In the test, subjects observe several sequences with alternating
processing and memory components (sequences have various length).
2. After a given sequence, subjects must recall the memory components in
correct order (e.g. letters).
3. Throughout the test, subjects must also maintain accuracy/speed on the
processing component.
4. Subjects’ working memory score depends on the number of memory
components recalled in correct order.
Note: A typical short-term memory test only has the memory component.
DESIGN cont.
To identify the impact of working memory on cognitive performance, I
supply “external memory” to subjects as a treatment:
In a computerized time-series forecasting task, two screens with forecastrelevant information are presented either concurrently or sequentially =>
two between-subject treatments.
HYPOTHESIS to be tested: Since the Sequential (Concurrent) treatment
offers less (more) “external memory” to subjects and hence features a higher
(lower) working memory load, working memory should be a stronger
(weaker) determinant of forecasting performance, after controlling for other
between-subject cognitive, motivational and personality differences.
The next several slides show how I experimentally implemented the time-series
forecasting task and in particular the Sequential and Concurrent treatments…
Period
Signal
Repeating
pattern
Error
Omega
Time-series forecasting task (a la Klayman, 1988)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
40
20
30
10
20
40
30
10
20
10
40
10
30
20
10
30
20
30
10
40
46
34
18
46
34
18
46
34
18
46
34
18
46
34
18
46
34
18
46
34
4
-8
8
0
4
0
-4
8
8
0
-4
-8
0
4
-4
-8
4
8
0
-4
90
46
56
56
58
58
72
52
46
56
70
20
76
58
24
68
58
56
56
70
Subjects repeatedly forecast Omega, a sum of
Signal, Repeating pattern and Error.
They observe 8-period history windows of Signal and
Omega (see the green columns).
Subjects are told that to accurately forecast Omega,
they need to discover the Repeating (seasonal)
pattern from successive values of Omega and
Signal by subtracting them. The unobserved,
random Error makes this task harder but subjects
know the Error distribution (uniform discrete).
Subjects forecast next-period value of Omega. To
be able to do that (as Signal is unpredictable), they
are shown next-period value of Signal.
Concurrent treatment subjects observe Signal
and Omega on one screen.
Sequential treatment subjects observe Signal and
Omega on two successive screens.
Financial incentives to forecast accurately are high:
subjects can earn up to 70-80 PPP Dollars.
Here is what Concurrent treatment subjects observe on a typical screen (for period 15):
subjects combine Signal and Omega values to forecast period-16 Omega.
Current period
15 of 100
Time remaining 15
Signal
Omega
Period 8
10
64
Period 9
20
50
Period 10
10
24
Period 11
40
90
Period 12
10
36
Period 13
30
52
Period 14
20
66
Current period 15
10
48
Next period 16
30
?
By contrast, Sequential treatment subjects first observe a screen with Signal values only: they
memorize Signal values and wait for the corresponding Omega screen…
Current period
15 of 100
Time remaining 10
Signal
Period 8
10
Period 9
20
Period 10
10
Period 11
40
Period 12
10
Period 13
30
Period 14
20
Current period 15
10
Next period 16
30
…and once the corresponding Omega screen appears, Sequential treatment subjects combine
the previously memorized Signal values with the observed Omega values to forecast period-16
Omega.
Current period
15 of 100
Time remaining 10
Omega
Period 8
64
Period 9
50
Period 10
24
Period 11
90
Period 12
36
Period 13
52
Period 14
66
Current period 15
48
Next period 16
?
ma12error1performance – absolute forecast
ma12error2
Forecasting
errors
20
The graph below shows 12-period moving
averages of absolute forecast errors, averaged
across subjects in each forecasting period,
separately for the Sequential and Concurrent
treatment.
absolute forecast errors
15
10
Sequential (average)
5
Concurrent (average)
0
10
20
30
EARLY
40
50
60
Period
70
80
90
LATE
100
ma12error1performance – absolute forecast
ma12error2
Forecasting
errors
20
The Concurrent treatment (with lower working
memory load) has lower absolute forecast errors (on
average) throughout the task…
…but statistically significant learning occurs in both
treatments between EARLY and LATE stages of the
task.
absolute forecast errors
15
10
Sequential (average)
5
Concurrent (average)
0
10
20
30
EARLY
40
50
60
Period
70
80
90
LATE
100
Heterogeneity in forecasting performance
Same graph, with 10th and 90th percentiles added to
averages, reveals substantial across-subject
heterogeneity in absolute forecast errors in both
treatments.
As hypothesized,
ma12error2 working memory should better
explainp90ma12error1
the heterogeneity in the Sequential
treatment (with higher working memory load). I
focus on the LATE stage.
ma12error1
p10ma12error1
absolute forecast errors
20
90th percentiles for
Concurrent (o) and
Sequential (+)
15
10
Sequential (average)
5
Concurrent (average)
10th percentiles for
Concurrent (o) and
Sequential (+)
0
10
20
30
EARLY
40
50
60
Period
70
80
90
LATE
100
Correlations between forecasting performance and working memory
Concurrent treatment:
spearman = -0.0006 (p=0.997)
pearson = -0.2208 (p=0.155)
20
absolute
LATEMedlate
forecast error
15
As hypothesized, working memory is
much stronger negatively correlated with
subjects’ LATE absolute forecast errors
in the Sequential treatment (with higher
working memory load).
10
5
0
20
30
40
50
OspanTotal
60
70
80
working memory
20
absolute
LATEMedlate
forecast error
15
By contrast, other measured cognitive,
motivational, personality and
demographic individual differences
cannot explain between-subject
performance variation in the Sequential
treatment. I nevertheless control for these
in the formal analysis that follows…
10
5
Sequential treatment:
spearman = -0.3028 (p=0.048)
pearson = -0.4540 (p=0.002)
0
20
30
40
50
OspanTotal
working memory
60
70
80
Testing the hypothesis that working memory is a stronger determinant of
forecasting performance in the Sequential treatment
• I regress LATE absolute forecast error on working memory and other potential determinants of forecasting
performance: short term memory, math ability, intrinsic motivation, etc. The figure below shows only several
selected specifications as explained by the labels.
• As some subjects’ performance is top-bounded, I use Censored Least Absolute Deviations (CLAD) estimator. So
far I have 43 observations per treatment (students from Prague non-selective universities).
• The bars are coefficient estimates for working memory. The estimates for the Sequential treatment generally
have economically meaningful magnitude. A yellow bar indicates that working memory has a significant impact on
forecasting performance (at 10% level). A green bar indicates that, in addition, the impact of working memory is
significantly larger in the Sequential treatment (at 10% level).
Estimation with working memory only
…with short term memory and math ability added
…with intrinsic motivation further added
…with EARLY performance added as a proxy for intrinsic
forecasting ability (covariates partialled out of the proxy)
…same, but with an alternative working memory measure
that has short term memory and math ability partialled out
Estimate for the hardest forecasting
season only, with covariates added
…same, but with EARLY
performance proxy added
Sequential
Concurrent
Testing the hypothesis that working memory is a stronger determinant of
forecasting performance in the Sequential treatment
• Conclusion: The impact of working memory (cognitive capital) on forecasting performance is clearly
stronger in the Sequential than in the Concurrent treatment…
•…but establishing this result more robustly may require more observations.
Estimation with working memory only
…with short term memory and math ability added
…with intrinsic motivation further added
…with EARLY performance added as a proxy for intrinsic
forecasting ability (covariates partialled out of the proxy)
…same, but with an alternative working memory measure
that has short term memory and math ability partialled out
Estimate for the hardest forecasting
season only, with covariates added
…same, but with EARLY
performance proxy added
Sequential
Concurrent
CONCLUSION / ROAD AHEAD
Working memory, or rather lack thereof, clearly presents a cognitive constraint on
forecasting performance.
In additional treatments (not completed), the working memory constraint is interacted with
variation in financial incentives by offering subjects to purchase “external memory” at
different relative prices: subjects start in the harder Sequential treatment but can purchase
switching to the easier Concurrent treatment.
What individual characteristics, beside working memory, will determine buying behavior?
Will more external memory be bought under higher financial incentives?
In each period, subjects also bet on the quality of their forecasts, prior to placing a forecast.
Financial return to betting is decreasing in subjects’ absolute forecast error.
Especially psychologists have argued that performance in cognitive tasks is likely to be also
affected by people’s confidence in their abilities. Bets provide a measure of confidence in
one’s forecasting abilities. Initial results suggest that working memory affects how quickly
bets respond to improvements in forecasting accuracy.
I will next investigate a two-equation system where both bets and performance are treated
as a result of dynamic learning processes, using exogenous variation in the quality of
forecasting feedback to identify the impact of bets on performance.