Latent Problem Solving
Analysis (LPSA): A
computational theory of
representation in complex,
dynamic problem-solving
tasks
José Quesada
Complex problem solving (CPS)
definition
• dynamic, because early actions determine the
environment in which subsequent decision must
be made, and features of the task environment
may change independently of the solver’s
actions;
• time-dependent, because decisions must be
made at the correct moment in relation to
environmental demands; and
• complex, in the sense that most variables are
not related to each other in one-to-one manner
2
‘Despite 10 years of research in the area, there
is neither a clearly formulated specific theory
nor is there an agreement on how to proceed
with respect to the research philosophy. Even
worse, no stable phenomena have been
observed’
(Funke, 1992, p. 25)
3
"How similar are two participant's
solutions?"
For CPS there is no common, explicit theory to explain
why a complex, dynamic situation is similar to any other
situation or how two slices of performance taken from a
problem solving task can possibly be compared
quantitatively
This lack of formalized, analytical models is slowing
down the development of theory in the field
4
Example of a complex, dynamic task:
Firechief (Omodei and Wearing 1995)
5
No. Command/
Event
Wind Change
Mature Fire
Mature Fire
Mature Fire
Mature Fire
1 Move
2 Move
3 Drop Water
4 Move
5 Move
6 Drop Water
7 Move
8 Move
9 Control Fire
10 Control Fire
11 Move
12 Drop Water
13 Move
14 Move
15 Move
16 Move
17 Drop Water
18 Move
19 Move
20 Move
21 Drop Water
22 Drop Water
23 Drop Water
Gen
0
0
0
0
0
17
31
38
54
70
77
99
113
122
131
152
177
187
222
236
267
273
296
319
341
347
352
361
Perf
App. App.
Position Destination/ Landscape/
Code Type
Upper Left
Lower Right
100.00 Wind Strength = 6
Wind Direction = East
100.00
(10, 10)
100.00
(6, 9)
100.00
(6, 8)
100.00
(9, 10)
100.00 4 Copter (11, 4)
(11, 9)
Forest
100.00 2 Truck
(4, 11)
(17, 7)
Clearing
100.00 4 Copter (11, 9)
Forest
100.00 3 Copter
(8, 6)
(10, 11)
Forest
99.77 1 Truck
(4, 14)
(18, 10)
Forest
99.42 3 Copter (10, 11)
Forest
99.18 4 Copter (11, 9)
(21, 8)
Dam
99.18 3 Copter (10, 11)
(12, 14)
Dam
98.95 2 Truck
(17, 7)
Clearing
98.95 1 Truck
(18, 10)
Forest
98.95 4 Copter (21, 8)
(12, 10)
Clearing
98.71 4 Copter (12, 10)
Clearing
98.71 3 Copter (12, 14)
(11, 11)
Clearing
98.48 4 Copter (12, 10)
(21, 8)
Dam
98.48 3 Copter (11, 11)
(12, 14)
Dam
98.25 3 Copter (12, 14)
(10, 12)
Forest
98.01 3 Copter (10, 12)
Forest
98.01 2 Truck
(17, 7)
(8, 5)
Forest
96.85 4 Copter (21, 8)
(7, 6)
Forest
96.61 3 Copter (10, 12)
(12, 7)
Forest
96.50 4 Copter
(7, 6)
Forest
96.50 2 Truck
(8, 5)
Forest
96.26 3 Copter (12, 7)
Forest 6
Problems with the classic
'problem space’ approach!
Most of the theories about cognitive skill
acquisition and procedural learning are
based in two principles:
– The problem space hypothesis
– Representation of procedures as productions
7
Problems with the classic
'problem space’ approach!
1. The problem with the ‘generation’ of the problem
space
2. The utility of the state space representation for tasks
with inner dynamics is reduced because in most CPS
environments it is not possible to undo the actions,
and prepare a different strategy:
8
Problems with the classic
'problem space’ approach!
3. The classic problem solving theory used mainly verbal
protocols as data. However, TALK ALOUD
INTERFERES PERFORMANCE IN COMPLEX
DYNAMIC TASKS (Dickson, McLennan & Omodei,
2000)
4. Independence (or very short-term dependences) of
actions/states is assumed in some of the methods for
representing performance. That is, the features that
represent performance are local
9
Objectives of the dissertation
1. Solve methodological problems on
microworld performance assessment
2. Propose an alternative theory of problem
solving and representation
3. Present LPSA as a theory of expertise
4. Develop a landing technique automatic
assessment system
10
LPSA as a theory of
representation in CPS tasks
(5) Applications:
Automatic landing
technique
assessment
(3) Expertise effects of amount of practice
(4) Expertise effects of amount of
environmental structure
(1) human similarity judgments
(2) ‘Strategy’ changes
11
What LPSA is and how it
relates to other theories
Latent Problem Solving Analysis
(LPSA)
• Assumptions:
– Similarity-based theory of representation
– The problem space is a vector space
– It can be generated from experience
automatically (corpus-based)
– Search and movement in this problem space
consists of vector operations
13
LSA
LPSA
The problem space is a metric space, where states and trials are represented as vectors
Cow
calf
Cheetah
14
Approaches to complexity: The
ant and the beach parable
(Simon, 1967,1981)
15
Approaches to complexity: The
ant and the beach parable
(Simon, 1967,1981)
16
Approaches to complexity: The
ant and the beach parable
(Simon, 1967,1981)
17
Approaches to complexity: The
ant and the beach parable
(Simon, 1967,1981)
?
18
Latent Problem Solving Analysis
(LPSA)
• Unsupervised learning
• Empirical adjustment of a problem space
• Definition of a productivity mechanism and
a similarity measure.
• LPSA: addition and cosine.
19
Latent Problem Solving Analysis
(LPSA)
• m(trial) = f{m(sa1), m(sa2),….. m(san), context}
• Simplifying assumptions:
m(trial1) = m(sa11) + m(sa21) +….. + m(san1)
m(trial2) = m(sa12) + m(sa22) +….. + m(san2)
….
m(trialk) = m(sa1k) + m(sa2k) +….. + m(sank)
• Where sa is a ‘state or action’
20
Latent Problem Solving Analysis
(LPSA)
• Complexity reduction: Reducing the number of
dimensions in the space reduces the noise
21
LPSA solutions for the problems with
the classic 'problem space’ approach
1. The
problem
witha the
‘generation’
of the problem
space
LPSA
proposes
mechanism
to generate
automatically
the problem space
2. LPSA
The utility
thepropose
state space
representation
forthis
tasks
doesofnot
a specific
solution for
with innerbut
dynamics
is reduced
because to
in represent
most CPS
problem,
it enables
the experimenter
environments
it is not possible
undo tasks
the actions,
very
different complex
problem to
solving
using aand
prepare a different strategy:
common formalism that could implement, as additional
assumptions, the irreversibility of some actions
22
LPSA solutions for the problems with
the classic 'problem space’ approach
3. The
classic
problem
solving
theory
used mainly
verbal
LPSA
uses log
files and
human
judgments
as data,
but
protocols
as data.
However,
TALK ALOUD
not concurrent
verbal
protocols
INTERFERES PERFORMANCE IN COMPLEX
DYNAMIC TASKS (Dickson, McLennan & Omodei,
2000)
does not (or
assume
independence
or short of
4. LPSA
Independence
very short-term
dependences)
dependences
states/actions.
Indeed,
it uses
actions/statesbetween
is assumed
in some of the
methods
for
representing
performance.
Thatsimultaneously
is, the featurestothat
the
dependences
of all of them
represent
performance
areThe
local
derive
the problem
space.
features that represent
performance are global
23
Theoretical surroundings of
Latent Problem Solving
Analysis
Mental representations
Perceptual
symbols
Propositions (Amodal
symbols)
Similarity based (varying in the
Rules and theories
amount of structure represented)
Continuous features
25
1. Encoding processes
2. Processes of
internal
transformation
3. Decoding processes
26
LPSA applied to model human
judgments
Actions :
Move_4_Copter_11_4_11_9_Forest_
Participants’
trials
1 Move_4_Copter_11_4_11_9_Forest_
2 Move_2_Truck_4_11_17_7_Clearing_
3 Drop_Water_4_Copter_11_9_Forest___
4 Move_3_Copter_8_6_10_11_Forest_
5 Move_1_Truck_4_14_18_10_Forest_
6 Drop_Water_3_Copter_10_11_Forest___
7 Move_4_Copter_11_9_21_8_Dam_
8 Move_3_Copter_10_11_12_14_Dam_
9 Control_Fire_2_Truck_17_7_Clearing___
10 Control_Fire_1_Truck_18_10_Forest___
11 Move_4_Copter_21_8_12_10_Clearing_
( . . . )
Words
Docs
28
Firechief corpus
• Data from the experiments described in
experiments 1 and 2 in Quesada et al.
(2000), and Canas et al. (2003).
• Total: 3441 trials, 75.575 different actions
• The first 300 dimensions where used
29
Trial 1
Trial 2
Trial 3
log files containing series of actions
Action 1
Action 2
3400 log files
actions
57000 actions
30
Human Judgment correlation
• if LPSA captures similarity between complex
problem solving performances in a meaningful
way, any person with experience on the task
could be used as a validation
• To test our assertions about LPSA, we recruited
15 persons and exposed them to the same
amount of practice as our experimental
participants, so they could learn the constraints
of the task.
31
Human Judgment correlation
• Replay trials, with different similarities
0.06
0.12
0.53 0.60
0.75
0.90
G
F
D
A
B
E
• People watched a randomly ordered series of
trials, in a different order for each participant,
which were selected as a function of the LPSA
cosines
32
Human Judgment correlation
FULL-SCREEN
REPLAY OF THE
TRIAL SELECTED, 8
TIMES FASTER THAN
NORMAL SPEED
33
Human Judgment correlation: Results
Human Judgment
Correlation: .948
LPSA
34
Human Judgment correlation :Discussion
• Applied: LSA is an automatic way of generating
a problem space and compare slices of
performance in complex tasks. It scales up very
well and does not depend on a-priori task
analyses
• Theoretical: LSA proposes that problem spaces
are metric spaces that are derived from
experience. Actions or States that are
functionally related are represented in similar
regions of the space. In this sense, problem
solving is unified with theories of object
recognition and semantics.
35
LPSA as a theory of
expertise in problem solving
• Ebbinghaus approach: manipulating previous
knowledge by eliminating it. Random
assignment of participants to groups.
• Chase and Simon approach (expert –
novice), manipulating previous knowledge by
pre - selecting participants (no random
assignment of participants to groups)
• Move complexity to the lab, and manipulate
previous knowledge (exactly = amount of
practice and experience for all participants)
37
• Ebbinghaus approach: manipulating previous
knowledge by eliminating it. Random
assignment of participants to groups.
• Chase and Simon approach (expert –
novice), manipulating previous knowledge by
pre - selecting participants (no random
assignment of participants to groups)
• Move complexity to the lab, and manipulate
previous knowledge (exactly = amount of
practice and experience for all participants)
38
Move complexity to the lab
• To simulate expertise environments in
labs, we need tasks more complex than
the standard ones:
– More representative
– Long learning curve
– Interesting enough to keep the motivation for
a long period of time
39
40
DURESS
• Christoffersen Hunter, & Vicente (1996,
1997, 1998) 6-month long longitudinal
experiment using Duress II. 225 trials, with
different goals values. Every participant
received exactly the same kind of trials.
• However, analysis mostly qualitative. Not
without a good reason…
41
34 variables, governed by mass and energy conservation laws
Example DURESS protocol
VAR1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
VAR2
0
3
6
9
12
15
18
21
24
27
30
33
36
39
42
45
48
51
54
57
60
63
66
69
72
75
78
81
84
87
90
93
96
99
102
105
108
111
114
117
120
123
126
129
132
135
138
141
144
147
150
0
0
0.273689
1.418777
2.496949
3.336244
3.936109
4.312084
4.586229
4.743035
4.842909
4.905937
4.944792
4.969547
4.979459
4.989136
4.765001
3.649077
2.691963
1.908286
1.386679
1.077631
0.897433
0.795081
0.738594
0.768093
0.94002
0.964371
0.881871
0.812034
0.820305
0.825
0.842689
0.85
0.854809
0.850205
0.873514
0.909544
0.947416
0.98365
1.01317
1.02949
1.03
1.03
1.017387
1.005
1.004015
1.0008
1.00075
0.993036
0.99
VAR3
0
0
0
0
0
0
0
0
0.013575
0.324127
0.514431
0.734883
0.909096
1.003641
0.948248
0.913023
0.890729
0.88084
0.877833
0.872994
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
0.87
VAR4
0
0
0.000138
0.000712
0.001255
0.001677
0.001977
0.002166
0.002303
0.002383
0.002434
0.002465
0.002484
0.002496
0.002503
0.002506
0.002393
0.001834
0.001352
0.000959
0.000697
0.000542
0.00045
0.000399
0.00037
0.000385
0.000473
0.000484
0.000443
0.000408
0.000413
0.000415
0.000424
0.000428
0.000429
0.000428
0.000439
0.000458
0.000476
0.000494
0.000509
0.000516
0.000518
0.000518
0.000512
0.000506
0.000506
0.000504
0.000503
0.0005
0.000497
VAR5
0
0
0
0
0
0
0
0.000842
0.001652
0.002292
0.00282
0.003252
0.003607
0.003904
0.00414
0.004334
0.004494
0.004622
0.004727
0.004814
0.004886
0.004944
0.004993
0.005032
0.005064
0.00509
0.005111
0.005129
0.005143
0.005155
0.005164
0.005172
0.005178
0.004716
0.004098
0.003578
0.003179
0.002842
0.00256
0.002336
0.002146
0.001994
0.001729
0.00141
0.001155
0.000946
0.000773
0.000631
0.000519
0.000424
0.000347
VAR6
0
0
0
0
0
0
0
0
8.05E-06
0.000216
0.000378
0.000584
0.000772
0.000904
0.000895
0.000901
0.000917
0.000944
0.000983
0.001029
0.001082
0.00114
0.001199
0.001261
0.001324
0.001387
0.001447
0.001506
0.001564
0.001623
0.001682
0.00174
0.001797
0.001851
0.001895
0.001931
0.001958
0.001979
0.001994
0.002005
0.002011
0.002015
0.002014
0.00201
0.002002
0.001991
0.001977
0.001962
0.001945
0.001927
0.001909
VAR7
0
0
0
0.00261
0.020982
0.03198
0.06
0.081715
0.108101
0.137773
0.162776
0.189602
0.209861
0.239094
0.258917
0.287408
0.308542
0.328321
0.344334
0.35
0.353689
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.35041
0.35
0.357386
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.36
0.363928
0.37
VAR8
0
0
0.000188
0.001449
0.004418
0.008833
0.014306
0.022185
0.032371
0.045039
0.059069
0.074055
0.089675
0.105762
0.122595
0.140073
0.157957
0.175024
0.190718
0.205373
0.219138
0.232322
0.24515
0.257705
0.270064
0.282277
0.294581
0.306915
0.319035
0.330913
0.342611
0.35417
0.365602
0.376412
0.385152
0.392332
0.397659
0.402016
0.405565
0.408281
0.410506
0.412195
0.413206
0.413484
0.41281
0.411474
0.409614
0.407348
0.404689
0.401805
0.398712
VAR9
0
0
0.073746
0.385738
0.696833
0.942067
1.11746
1.227147
1.307358
1.35428
1.383688
1.401696
1.412798
1.419871
1.422703
1.425468
1.492649
1.811692
1.965049
1.927847
1.784063
1.631448
1.504779
1.414729
1.353304
1.295564
1.216285
1.190239
1.172993
1.135694
1.080892
1.038963
1.005498
0.984395
0.97192
0.964579
0.976187
1.000421
1.024947
1.049267
1.070121
1.081089
1.082857
1.082857
1.074449
1.059373
1.040043
1.026229
1.016964
1.011174
1.008056
VAR10
0
0
0
0
0
0
0
0
0
0.146906
0.491208
0.671362
0.789265
0.872862
0.913472
0.936597
0.978125
1.003555
1.017191
1.025149
1.029625
1.031909
1.031062
1.01644
1.008371
1.004906
1.002201
1.001429
1.0002
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
1
NEWVAR11
NEWVAR12
NEWVAR13
NEWVAR14
NEWVAR15
0
0
0
0
0
0
0
0
0
0
7.38E-05
0
0
0 0.000174
0.000388
0
0
0.00261 0.001359
0.000701
0
0 0.020982 0.004232
0.000947
0
0
0.03198 0.008573
0.001122
0
0
0.06 0.013998
0.001234 0.000157
0 0.081715 0.020715
0.001314 0.000394
0 0.108101 0.028693
0.001361 0.000581
0.00017 0.133886
0.03799
0.00139 0.000736 0.000591 0.152776
0.04648
0.001408 0.000862 0.000847 0.173068 0.054147
0.001419 0.000966 0.001042
0.18493 0.061339
0.001426 0.001053 0.001201 0.199547 0.068199
0.00143 0.001122 0.001307 0.209459 0.074773
0.001432 0.001178 0.001386 0.219136 0.081218
0.0015 0.001225 0.001492
0.23 0.087532
0.001821 0.001263 0.001562 0.248321 0.094802
0.001974 0.001293 0.001601 0.263001 0.103304
0.001937 0.001319 0.001628 0.288022 0.111984
0.001793
0.00134 0.001652 0.303689 0.120116
0.001639 0.001357 0.001677 0.313366 0.127414
0.001512 0.001371
0.0017
0.33
0.13397
0.001421 0.001382 0.001703
0.33082 0.139923
0.001359 0.001392 0.001718 0.340703
0.14549
0.001302 0.001399 0.001738
0.35 0.150708
0.001222 0.001405 0.001763 0.357298 0.155422
0.001196 0.001411 0.001789
0.36 0.159807
0.001179 0.001415 0.001815
0.36 0.163983
0.001141 0.001418 0.001841 0.368281 0.167887
0.001086 0.001421 0.001868
0.37 0.171408
0.001044 0.001423 0.001896
0.37
0.17456
0.00101 0.001425 0.001924
0.37
0.1774
0.000989 0.001427 0.001952
0.37
0.17994
0.000976 0.001263 0.001975
0.37 0.181783
0.000969 0.001124
0.00199
0.37
0.18274
0.000981 0.001018 0.001996
0.37 0.182972
0.001005 0.000976 0.001996
0.37 0.182914
0.00103 0.000955 0.001994
0.37 0.182846
0.001055 0.000938
0.00199
0.37 0.182835
0.001075 0.000924 0.001983
0.37 0.182882
0.001086 0.000913 0.001976
0.37 0.182987
0.001088 0.000924 0.001969
0.37 0.183176
0.001088 0.000933 0.001962 0.378123 0.183439
0.00108 0.000941 0.001957
0.38 0.183769
0.001064 0.000948 0.001953
0.38
0.18409
0.001045 0.000953 0.001951
0.38 0.184367
0.001031 0.000957 0.001949
0.38 0.184595
0.001022 0.000961 0.001949
0.38
0.18478
0.001017 0.000964 0.001949
0.38 0.184947
0.001013 0.000966
0.00195
0.38 0.185104
42
Trial 1
Trial 2
Trial 3
log files containing series of States
State 1
State 2
1151 log files
States
57000 States
43
Current theories of expertise
Long Term
Working
Memory
(LTWM)
Ericsson and
Kintsch
(1995)
EPAM IV
(e.g., Gobet,
Richman,
Staszewski
and Simon,
1997)
Constraint
Attunement
Hypothesis
(CAH)
Vicente and
Wang
(1998)
44
Current theories of expertise
Long Term
Working
Memory
(LTWM)
Ericsson and
Kintsch
(1995)
EPAM IV
(e.g., Gobet,
Richman,
Staszewski
and Simon,
1997)
PROCESS THEORIES
Constraint
Attunement
Hypothesis
(CAH)
Vicente and
Wang
(1998)
PRODUCT
THEORY
45
LTWM (Ericsson and Kintsch,
1995)
• STM accounts for working memory in unfamiliar
activities but does not appear to provide
sufficient storage capacity for working memory in
skilled complex activities (p.220)
• LTWM is acquired in particular domains to meet
specific demands imposed by a given activity on
storage and retrieval. LTWM is task specific.
46
LTWM (Ericsson and Kintsch,
1995)
• Intense practice in a domain creates retrieval
structures: associations between the current
context and some parts of LTM that can be
retrieved almost immediately without effort
(example: SF and digits).
• LTWM permits rapid and reliable reinstantiation
of a context after interruption without a decrease
in performance.
47
CAH (Vicente and Wang, 1998)
•
Contrary to what process theories maintain,
Constrain Attunement Hypothesis (CAH) does
not commit to a particular psychological
mechanism to explain the phenomenon of
expertise.
1. How should one represent the constrains that
the environment (i.e., the problem domain)
places on expertise?
2. Under what conditions will there be an
expertise advantage?
3. What factors determine how large the
advantage can be?
48
CAH (Vicente and Wang, 1998)
• Describing the constraints in the
environment is the task of an expertise
theory.
from Shanteau (2001)
Higest level of performance
Lowest level of performance
Aided decisions
competent
restricted
Random
Wheather forecasters
astronomers
test pilots
insurance analysts
Physicists
chess masters
livestock judges
grain inspectors
photo interpreters
soil judges
parole officers
psychiatrists
student admissions
intelligence analists
polygrahers
managers
stock forecasters
parole officers
court judges
49
CAH (Vicente and Wang, 1998):
the Abstraction Hierarchy
FUNCTIONAL
ABSTRACT
GENERALIZED
PHYSICAL
Overall system goals (how much
water each reservoir is outputting,
and at which temperature)
'D1','D2','T1','T2'
conservation of mass and energy
for each reservoir (how much
mass & energy is entering and
leaving the reservoir).
'MI1', 'MO1',
'EI1', 'EO1',
'M1', 'E1',…
Flows and storage of heat
'FA','FA1','FA2','HTR1‘…
Settings of valves, pumps,
'PA','PB','VA','VA1','VA2‘,…
and heaters
50
Continuum of abstraction, means- ends relationship between levels
LTWM vs. CAH
• LTWM claims that the magnitude of expertise effects
is “related to the level of attained skill and to the
amount of relevant prior experience”
• CAH argues that this claim is incomplete. Expertise
effects in memory recall are also determined by the
amount of structure in the domain (and by active
attunement to that structure)
• LPSA is sensible both to ‘relevant previous practice’
and to ‘amount of structure in the domain’
51
3/4
1/4
?
52
3/4
1/4
?
53
54
Predictions
• Only huge amounts of experience with the
system would enable the actor (human or
model) to make accurate predictions of the last
quarter of the trial
• Sparse practice should clearly lead to poor
prediction
• Only structured environments should show the
expertise advantage. Following CAH, the expert
(human or model) should not do well in a
55
completely unstructured environment
Expertise results: Three years of experience with
DURESS
Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest Neighbors
When the three first quarters are used to retrieve the neighbors
10 nearest neighbors
10 random trials
1
0.9
Average cosine
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Functional
Abstract
Generalized
Physical
56
Expertise results: Six months of experience
with DURESS
Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest Neighbors
When the three first quarters are used to retrieve the neighbors
1
10 nearest neighbors
10 random trials
Average cosine
0.8
0.6
0.4
0.2
0
Functional
Abstract
generalized
physical
-0.2
57
Expertise results: Three year of experience in a
DURESS with no constraints (random states)
Average cosine between the fourth quarter of a target trial and the fourth quarter of the 10 nearest Neighbors
When the three first quarters are used to retrieve the neighbors
10 nearest neighbors
10 random trials
1
0.9
Average cosine
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0
Functional
Abstract
generalized
physical
58
Expertise results: Discussion
• LPSA can explain both LTWM and CAH main
assertions
– LTWM claims that the magnitude of expertise effects
is related to the level of attained skill and to the
amount of relevant prior experience
– CAH claims that expertise effects in memory recall
are also determined by the amount of structure in the
domain (and by active attunement to that structure)
• Better yet, LPSA proposes both processes and
representational structures
59
Automatic Landing
Technique Assessment
using Latent Problem
Solving Analysis (LPSA)
The problem
• There is currently no methodology to automatically
assess landing technique in a commercial aircraft or a
flying simulator. Instructors are a significant cost for
training and evaluation of pilots, and the use of
instructors also incorporates a subjective component
that may vary from pilot to pilot.
• The advantages of automatic landing technique
evaluation are many:
1.
2.
3.
4.
5.
6.
Reduced cost of the evaluation.
Increased objectivity in the evaluation.
Decrease the influence of the instructor.
Perfect Test-retest reliability.
It is always available and can be triggered by the trainee at will.
61
The model can rate as many landings as time enables, etc.
A solution: Latent Problem
Solving Analysis (LPSA)
• In this application of LPSA to landing technique
evaluation, we assume that an expert uses her
past knowledge to emit landing ratings by
comparing the current situation to the past ones,
and generates an expanded representation of
the environment by composing the past
situations that are most similar to the current
one.
62
63
Complex, dynamic tasks are intractable
when considered as a whole:
64
Complex, dynamic tasks are intractable
when considered as a whole:
• We need to perform complexity reduction,
in a mostly automatic way
– The triangulation technique
– Dimensionality reduction (LPSA)
65
The triangulation technique:
C: Modeling
of the expert
with restricted
information:
Expert with partial
access to the
variables (having
to make a
A: Modeling of
the whole
selection)
situation:
MODEL
Accessible
B: Selection
(part of the
theory)
Expert with
complete
access to any
variable in the
Inaccessible
66
Complexity reduction (I): variable selection using differently
informed experts
Vertical acceleration
Radio altitude
Thrust
Two experts graded the same landings, with
different information; the reduced information
expert selected a set of variables and plotted them
in a computer screen. The complete information
expert sat in the copilot seat, and has access to all
the variables, exactly as the pilot. We trained the
model with only the variables selected by the
reduced information expert.
The reduced information expert
plotted the variables that he believed
were sufficient to rate the landing
The complete information expert had
access to all possible variables (visual,
67
proprioceptive, etc)
Complexity reduction (II) : Using SVD, the problem space is a
vector space
68
Using the LPSA problem space any known landing is represented as a vector. We can
approximate humans ratings by retrieving from memory the nearest neighbors of the vector
formed for any new landing, and averaging the neighbors’ ratings.
Past landing: too low, correct (…), 5
Past landing: correct, too fast (…), 4
New landing: unknown ratings
Past landing: too low, correct (…), 5
New landing: average of neighbors
Correlate?
New landing: human judgments
Past landing: correct, too fast (…), 4
Model selection
1. Number of dimensions (100, 150, 200, 250, 300, 350, and maximum
dimensionality, 400)
2. and the number of nearest neighbors used to estimate the landing ratings (from
69
1 to 10).
3. The model with the best fit used 200 dimensions and 5 nearest neighbors.
Results
70
Results: no-constraints corpus
agreements human - human
agreements human reduced information - LPSA
human complete information - LPSA
Polychoric correlation
1
0.8
0.6
0.45
0.4
0.2
0.17
0.12
0.020.03
0.06
0
-0.2
-0.4
-0.12
flare
initiation
Flare
initiation
altitude
altitude
-0.06
thrust
Thurst
reduction
pitch
angle
Pitch
angle
-0.01
-0.04
ov. Landing
performance
average
agreement
71
Conclusions
Objectives
of the dissertation
1. Solve methodological problems on
microworld performance assessment
2. Propose an alternative theory of problem
solving and representation
3. Present LPSA as a theory of expertise
4. Develop a landing technique automatic
assessment system
72
Conclusions
1.
2.
Solve
methodological
problems on
microworld
performance
assessment
Propose an
alternative
theory of
problem solving
and
representation
• LPSA reduces to a minimum the task/specific,
a priori assumptions
• Generalizability
1. Wide variety of systems
2. systems are described in terms of nominal
(discrete) or continuous variables
3. Actions or states as units
3.
Present LPSA
as a theory of
expertise
4.
Develop a
landing
technique
automatic
assessment
system
73
Conclusions
1.
2.
Solve
methodological
problems on
microworld
performance
assessment
Propose an
alternative
theory of
problem solving
and
representation
3.
Present LPSA
as a theory of
expertise
4.
Develop a
landing
technique
automatic
assessment
system
The problem space hypothesis: all intelligent
behavior takes place in a problem space (Newell,
1980)
Problem space
Problem space
Problem space
But the question of where the problem space came from
in the first place remains unanswered.
The generation of the problem space is considered
intelligent behavior and thus, takes place as well in a74
problem space!
Conclusions
1.
Solve
methodological
problems on
microworld
performance
assessment
• Integration between molecular and molar levels
• Explains both the ‘amount of experience’ and
‘amount of environmental structure’ effects,
characteristic of LTWM and CAH simultaneously
2.
Propose an
alternative
theory of
problem solving
and
representation
• Explains both processes and representations.
Happy marriage of process and product theories of
expertise
3.
Present LPSA
as a theory of
expertise
4.
Develop a
landing
technique
automatic
assessment
system
• Well-specified. In LTWM’s original formulation the
retrieval structures were under-specified. In LPSA,
the basic mechanisms postulated are defined
computationally. In CAH’s original formulation, the
representation of the environmental constraints (its
most central assertion) where under-specified too.
LPSA proposes an automatic mechanism to
represent the statistical regularities of the
75
environment
Conclusions
1.
Solve
methodological
problems on
microworld
performance
assessment
2.
Propose an
alternative
theory of
problem solving
and
representation
3.
Present LPSA
as a theory of
expertise
4.
Develop a
landing
technique
automatic
assessment
system
• GENERALITY: the fact that the same
mechanism, with the very same underlying
assumptions, can be used for language and
Problem Solving is interesting per-se: In
LTWM, the retrieval structures for chess are
different compared to the ones proposed for
text comprehension; In CAH, two AH for two
different tasks are different too; In LPSA, any
space for any task is a vector space.
76
Conclusions
1.
2.
Solve
methodological
problems on
microworld
performance
assessment
Propose an
alternative
theory of
problem solving
and
representation
3.
Present LPSA
as a theory of
expertise
4.
Develop a
landing
technique
automatic
assessment
system
• LPSA is not limited to laboratory tasks
• Complex tasks can be explained without recurring to
constructs such as problem solving, mental models
or reasoning
• The triangulation technique
• Practical advantages:
1. Reduced cost of the evaluation.
2. Increased objectivity in the
evaluation.
3. Decrease the influence of the
instructor.
4. Perfect Test-retest reliability.
5. It is always available and can be
triggered by the trainee at will. 77
-END• Acknowledgments
–
–
–
–
–
–
–
–
–
–
Tom Landauer
Kim Vicente
John Hajdukiewicz
Anders Ericsson
Simon Dennis
Alex Rutten
Adri Marksman
Nancy Mann
Yumiko Abe
Melanie Haupt
• Funding
– Grant EIA – 0121201 from
the National Science
Foundation
– European Community Access to Research
Infrastructure action of the
Improving Human Potential
Program under contract
number HPRI-CT-199900105 with the National
Aerospace Laboratory,
NLR
78
79
Three examples of performance
• 8 first actions in a trial
1
RELATED
2
3
80
0
1
1
CONTROL
FIRE
2
3
4
5
6
7
8
9 10 11
12 13 14 15
CONTROL
FIRE
0
1
2
3
4
5
6
7
8
9
10 11
12 13 14 15 16 17
18
19 20 21 22 23 24
81
0
2
1
2
3
4
5
6
7
8
9 10 11
DROP
WATER
12 13 14 15
CONTROL
CONTROL
FIRE
FIRE
0
1
2
3
4
5
6
7
8
9
10 11
12 13 14 15 16 17
18
19 20 21 22 23 24
82
3
0
CONTROL
FIRE
1
2
3
CONTROL
FIRE
4
5
6
7
8
9 10 11
12 13 14 15
0
1
2
3
4
5
6
7
8
9
10 11
12 13 14 15 16 17
18
19 20 21 22 23 24
83
comparisons
method
Subjective similarity
Example 1 to
Example 2
Example 1 to
Example 3
Example 2 to
Example 3
high
low
Low
LSA Cosine
0.7219
0.0567
0.0711
Exact matching
0.125
0
0
Transitions between actions
0.971
1
0.971
84
Possible way of comparison: Exact
matching of actions
• Exact matching: count the number of
common actions in two files. The higher
this number, the more similar they are
85
Example 1
Example 2
Move_2_truck_4_11_13_3_forest
move_2_truck_4_11_12_15_forest
Move_1_truck_4_14_16_14_forest
move_1_truck_4_14_13_5_forest
Move_3_copter_8_6_11_12_forest
move_4_copter_11_4_11_9_forest
move_4_copter_11_4_11_9_forest
drop_water_4_copter_11_9_forest
control_fire_2_truck_13_3_forest
move_4_copter_11_9_13_8_forest
control_fire_1_truck_16_14_forest
control_fire_2_truck_12_15_forest
move_2_truck_13_3_17_7_clearing
move_2_truck_12_15_13_14_forest
move_1_truck_16_14_20_12_forest
control_fire_2_truck_13_14_forest
Example 1
Example 3
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_2_2_pasture
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_0_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_8_6_8_4_clearing
move_4_copter_11_4_11_9_forest
move_3_copter_8_6_8_10_clearing
control_fire_2_truck_13_3_forest
control_fire_2_truck_2_2_pasture
control_fire_1_truck_16_14_forest
control_fire_1_truck_0_5_forest
move_2_truck_13_3_17_7_clearing
move_4_copter_8_4_4_2_forest
move_1_truck_16_14_20_12_forest
move_3_copter_8_10_2_3_clearing
86
comparisons
method
Subjective similarity
Example 1 to
Example 2
Example 1 to
Example 3
Example 2 to
Example 3
high
low
Low
Exact matching
0.125
0
0
Exact matching
0.125
0
0
Transitions between actions
0.971
1
0.971
87
Possible way of comparison:
Transitions between actions
• count the number transitions between actions
in two files. Create matrices, and correlate
them
88
Example 1
Example 2
Move_2_truck_4_11_13_3_forest
move_2_truck_4_11_12_15_forest
Move_1_truck_4_14_16_14_forest
move_1_truck_4_14_13_5_forest
Move_3_copter_8_6_11_12_forest
move_4_copter_11_4_11_9_forest
move_4_copter_11_4_11_9_forest
drop_water_4_copter_11_9_forest
control_fire_2_truck_13_3_forest
move_4_copter_11_9_13_8_forest
control_fire_1_truck_16_14_forest
control_fire_2_truck_12_15_forest
move_2_truck_13_3_17_7_clearing
move_2_truck_12_15_13_14_forest
move_1_truck_16_14_20_12_forest
control_fire_2_truck_13_14_forest
Example 1
Example 3
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_2_2_pasture
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_0_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_8_6_8_4_clearing
move_4_copter_11_4_11_9_forest
move_3_copter_8_6_8_10_clearing
control_fire_2_truck_13_3_forest
control_fire_2_truck_2_2_pasture
control_fire_1_truck_16_14_forest
control_fire_1_truck_0_5_forest
move_2_truck_13_3_17_7_clearing
move_4_copter_8_4_4_2_forest
move_1_truck_16_14_20_12_forest
move_3_copter_8_10_2_3_clearing
89
Example 1
Example 2
Move_2_truck_4_11_13_3_forest
move_2_truck_4_11_12_15_forest
Move_1_truck_4_14_16_14_forest
move_1_truck_4_14_13_5_forest
Move_3_copter_8_6_11_12_forest
move_4_copter_11_4_11_9_forest
move_4_copter_11_4_11_9_forest
drop_water_4_copter_11_9_forest
control_fire_2_truck_13_3_forest
move_4_copter_11_9_13_8_forest
control_fire_1_truck_16_14_forest
control_fire_2_truck_12_15_forest
move_2_truck_13_3_17_7_clearing
move_2_truck_12_15_13_14_forest
move_1_truck_16_14_20_12_forest
control_fire_2_truck_13_14_forest
Example 1
Example 3
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_2_2_pasture
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_0_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_8_6_8_4_clearing
move_4_copter_11_4_11_9_forest
move_3_copter_8_6_8_10_clearing
control_fire_2_truck_13_3_forest
control_fire_2_truck_2_2_pasture
control_fire_1_truck_16_14_forest
control_fire_1_truck_0_5_forest
move_2_truck_13_3_17_7_clearing
move_4_copter_8_4_4_2_forest
move_1_truck_16_14_20_12_forest
move_3_copter_8_10_2_3_clearing
90
Example 1
Example 2
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_12_15_forest
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_13_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_11_4_11_9_forest
move_4_copter_11_4_11_9_forest
drop_water_4_copter_11_9_forest
control_fire_2_truck_13_3_forest
move_4_copter_11_9_13_8_forest
control_fire_1_truck_16_14_forest
control_fire_2_truck_12_15_forest
move_2_truck_13_3_17_7_clearing
move_2_truck_12_15_13_14_forest
move_1_truck_16_14_20_12_forest
control_fire_2_truck_13_14_forest
Example 1
Example 3
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_2_2_pasture
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_0_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_8_6_8_4_clearing
move_4_copter_11_4_11_9_forest
move_3_copter_8_6_8_10_clearing
control_fire_2_truck_13_3_forest
control_fire_2_truck_2_2_pasture
control_fire_1_truck_16_14_forest
control_fire_1_truck_0_5_forest
move_2_truck_13_3_17_7_clearing
move_4_copter_8_4_4_2_forest
move_1_truck_16_14_20_12_forest
move_3_copter_8_10_2_3_clearing
91
Example 1
Example 2
Move_2_truck_4_11_13_3_forest
move_2_truck_4_11_12_15_forest
Move_1_truck_4_14_16_14_forest
move_1_truck_4_14_13_5_forest
Move_3_copter_8_6_11_12_forest
move_4_copter_11_4_11_9_forest
move_4_copter_11_4_11_9_forest
drop_water_4_copter_11_9_forest
control_fire_2_truck_13_3_forest
move_4_copter_11_9_13_8_forest
control_fire_1_truck_16_14_forest
control_fire_2_truck_12_15_forest
move_2_truck_13_3_17_7_clearing
move_2_truck_12_15_13_14_forest
move_1_truck_16_14_20_12_forest
control_fire_2_truck_13_14_forest
Example 1
Example 3
move_2_truck_4_11_13_3_forest
move_2_truck_4_11_2_2_pasture
move_1_truck_4_14_16_14_forest
move_1_truck_4_14_0_5_forest
move_3_copter_8_6_11_12_forest
move_4_copter_8_6_8_4_clearing
move_4_copter_11_4_11_9_forest
move_3_copter_8_6_8_10_clearing
control_fire_2_truck_13_3_forest
control_fire_2_truck_2_2_pasture
control_fire_1_truck_16_14_forest
control_fire_1_truck_0_5_forest
move_2_truck_13_3_17_7_clearing
move_4_copter_8_4_4_2_forest
move_1_truck_16_14_20_12_forest
move_3_copter_8_10_2_3_clearing
92
Possible way of comparison:
Transitions between actions
(a)
(b)
Example 1
drop
move
Example 2
control
drop
Move
Control
drop
0
0
0
drop
0
1
0
move
0
4
1
move
1
2
2
control
0
1
1
control
0
1
0
(c)
(c)
Example 3
drop
move
control
drop
0
0
0
move
0
4
1
control
0
1
1
93
comparisons
method
Subjective similarity
Example 1 to
Example 2
Example 1 to
Example 3
Example 2 to
Example 3
high
low
Low
Exact matching
0.125
0
0
Transitions between actions
0.971
1
0.971
Transitions between actions
0.971
1
0.971
94
comparisons
method
Subjective similarity
Example 1 to
Example 2
Example 1 to
Example 3
Example 2 to
Example 3
high
low
Low
Exact matching
0.125
0
0
Transitions between actions
0.971
1
0.971
LSA Cosine
0.7219
0.0567
0.0711
•Exact matching is not sensitive to similarity differences
(exigent criterion).
Since Transitions between actions is blind to most of the information
in the logs, it fails because declares as similar
performances that are not
•LSA has correctly inferred that the remaining actions,
although different, are functionally related
95