Using Expectations to Drive
Cognitive Behavior
Unmesh Kurup
Christian Lebiere, Tony Stentz, Martial Hebert
Carnegie Mellon University
Cognitive Decision Cycle
• Cognition is driven by Expectations/Predictions.
Prediction
World
Action
High-level
Cognition
Calculate
Mismatch
Prediction
Action
Prediction
Retrieve
Response Action
World
Cognition
t-1
t
t+1
Pedestrian Tracking & Behavior Classification
Goals:
• Investigate use of expectations
• Integrate with perception
• Run both offline & real-time
Integrated System
Partial Matching & Blending
+retrieval>
isa locationchunk
id person1
nextx 300
Chunk1
isa locationchunk
id person2
nextx 255
nexty 100
Declarative Memory
Chunk2
isa locationchunk
id person2
nextx 1010
nexty 500
Partial Matches
Chunk1
isa locationchunk
id person1
nextx 255
nexty 100
Chunk4
isa locationchunk
id person1
nextx 299
nexty 100
Chunk3
isa locationchunk
id person3
nextx 187
nexty 313
Chunk4
isa locationchunk
id person1
nextx 299
nexty 100
Blended result
Chunk5
isa locationchunk
id person1
nextx 293.91
nexty 100
Using Expectations: Tracking
Chunk-type visual-location
id X Y
Dx Dy Nextx
Nexty
Foreach Object o:
+blending>
isa visual-location
id o
compare to (x,y)s from perception
pick thresholded closest match,
calculate newdx, newdy, newx, newy
+imaginal>
isa visual-location
id o
…
Features
Features:
straight1
straight2
detour
left
straight3
veer
Behavior
Features
Behavior
Features
Normal – Straight
straight1, straight2,
straight3
Detour
straight1, detour,
straight3, chk-pt
Normal – Left
straight1, straight2,
left
Veer
straight1, straight2,
left, veer, chk-pt
Peek
straight1, detour, left,
no-chk-pt
Walkback
straight1, straight2, left,
straight2, straight1, chk-pt
Using Expectations: Detecting Features
from Data
Straight & Left
Deviation from expected location indicates a point of interest
Foreach location
+blending>
isa visual-location
x =x
y =y
compare to (x,y)s from perception
if path deviates more than threshold, mismatch!
+imaginal>
isa visual-location
id o
…
Cluster points into regions
Detected Features
Data
• Combined Arms Collective Training
Facility(CACTF) at Fort Indiantown Gap, PA.
• 4 examples. 3/1 split.
• Multiple behavior set
– 10 behaviors.
Behaviors
Straight & Left
Detour
Peek
Veer
Walkback
Results
Learning Model
Learning Model
(Single Behavior Set)
(Multiple Behavior Set)
Made
86.1%
Made
82.4%
Correct
68%
Correct
43.8%
Incorrect
18.1%
Incorrect
38.6%
Hand-coded Model
Hand-coded Model
(Single Behavior Set)
(Multiple Behavior Set)
Made
99.3%
Made
46.5%
Correct
99.15%
Correct
30.2%
Incorrect
0.15%
Incorrect
16.3%
Future Work – Semantic Labels
Future Work – Using Semantic Labels
Behavior
Features (Spatial)
Features (Semantic)
Normal – Straight
straight1, straight2,
straight3
Sidewalk, Pavement
Normal – Left
straight1, straight2,
left
Sidewalk
Peek
straight1, detour, left,
no-chk-pt
Pavement, Sidewalk
Detour
straight1, detour,
straight3, chk-pt
Pavement
Veer
straight1, straight2,
left, veer, chk-pt
Sidewalk, Pavement
Walkback
straight1, straight2, left,
straight2, straight1, chkpt
Sidewalk
Future Work
• Generic model of monitoring using
expectations
• Learn expectations
• Monitor for deviations from expectations
– Signal failure
– Provide for recovery
Collaborators
Max Bajracharya, JPL
Bob Dean, GDRS
Brad Stuart, GDRS
FMS lab, CMU