Measuring Team Mental Models
J. Alberto Espinosa
PhD Candidate, Information Systems
Graduate School of Industrial Administration
[email protected]
Prof. Kathleen M. Carley
Dept. of Social and Decision Sciences
Carnegie Mellon University
Academy of Management Conference 2001
Washington, D.C., August 8, 2001
Introduction
Motivation:
• Team coordination studies: needed SMM measures
– Simulated management decision teams (done)
– Large-scale software developers (in progress)
• Empirical work lags theory
•
Not much agreement on measures
[Mohammed & Dumville 2001]
Outline
• Theoretical foundations: coordination & SMMs
• Propose SMM measures: SMMTask and SMMTeam
• Preliminary empirical validation results
8/8/2001
A. Espinosa, AoM 2001
2
Coordination: and Old Problem
Explicit coordination mechanisms
• Coord by "programming"
Team/Task
Programming
[March & Simon 1958; Thompson 1967]
• Impersonal mechanisms
[VanDeVen & Delvecq 1976]
More routine
aspects of the task
Management of interdependencies among
Coordination = members, sub-tasks & resources
[Malone & Crowston 1994]
Less routine
aspects of the task
Team
Communication
• Coord by "feedback", "mutual adjustment"
[March & Simon 1958; Thompson 1967]
• “Personal” mechanisms
[VanDeVen & Delvecq 1976]
• How teams communicate matters
[Kraut & Streeter 1995; Sproull & Kiesler 1991]
8/8/2001
A. Espinosa, AoM 2001
3
Coordination: Newer Concepts
Implicit coordination mechanisms
Team/Task
Programming
Implicit coordination through:
• Team mental models
[Cannon-Bowers et. al. 1993,
Klimoski et. al. 1994]
Implicit
Coordination
Mechanisms
• Team situation awareness
[Endsley 1995; Wellens 1993]
Coordination
• Transactive memory
[Wegner, 1986, 1995;
Liang et. al. 1995]
Team
Communication
8/8/2001
• Group mind [Weick 1990; 1993],
distributed cognition, schema
similarities, etc.
A. Espinosa, AoM 2001
4
Team/Shared Mental Models
• Mental Models
Organized knowledge structures that help individuals interact
with their environment (i.e., describe, analyze and anticipate)
[Johnson-Laird 1983; Rouse & Morris 1986]
• Team/Shared Mental Models (SMMs)
Organized knowledge shared by team members that enable
them to form accurate explanations and expectations about
the task, team members, etc.
[Orasanu et. al. 1993; Cannon-Bowers et. al. 1993; Klimosky et. al. 1994]
Will use "shared" & "team" mental models interchangeably
• Main Types
About taskwork & teamwork
[Klimosky et. al. 1994; Cooke et. al. 2000]
8/8/2001
A. Espinosa, AoM 2001
5
Previous Measures Used for SMMs
•
All methods are based on some form of intra-team knowledge
similarity measure [Cooke et. al 2000; Mohammed et. al. 2001]
Similarities in word sequences [Carley 1997]
Correlation between individual mental models [Mathieu et. al. 2000]
Within-team response similarities [Levesque et. al. 2001; James et. al. 1984]
Multidimensional scaling [Rentsch et. al. 2001]
8/8/2001
A. Espinosa, AoM 2001
6
Proposed SMM Measures
•
•
•
•
•
•
•
Also based on knowledge similarities
At the dyad level [Klimosky et. al. 1994]
Network analysis methods: ideal to study dyadic relationships
Sociomatrices: facilitate computation of SMM measures
Distribution of shared knowledge: centralities, isolates, cliques, etc.
Analyze SMMs at different levels of abstraction
Sociograms: visual representation
Method:
• Knowledge similarity sociomatrices KSt(nxn)
• One for each task aspect or area t
• One row and one column for each of the n team members
• Cell kstij contains knowledge similarity in task area t between
members i and j
• Aggregate across dyads and task areas
8/8/2001
A. Espinosa, AoM 2001
7
SMM Measures Proposed
SMMTask
Knowledge similarity within the team about the task =
Average task knowledge similarity among all dyads
• From task knowledge similarity (TKS) sociomatrices
SMMTeam
Knowledge similarity within the team about each other =
Average team knowledge similarity among all dyads
• From member similarity (MS) sociomatrices
8/8/2001
A. Espinosa, AoM 2001
8
SMMTask Measure
a) When member's task knowledge can be evaluated
Knowledge Similarity: Finance
tkstij =
Mbr 1
2
3
4
5
6
1
2.0 2.0 2.0 2.0 2.0
min(kit,kjt)
2 2.0
2.1 2.7 2.7 2.7
[Cooke et. al 2000] 3 2.0 2.1
2.1 2.1 2.1
4 2.0 2.7 2.1
3.2 3.2
5 2.0 2.7 2.1 3.2
5.8
6 2.0 2.7 2.1 3.2 5.8
K(nxt)
Knowledge Matrix
Mbr Fin Prd Mkt
1 2.0 2.2 5.5
2 2.7 4.0 6.5
3 2.1 6.2 4.7
4 3.2 2.1 7.0
5 6.0 6.6 6.8
6 5.8 6.0 6.4
Knowledge Similarity: Production
Mbr 1
2
3
4
5
6
1
2.2 2.2 2.1 2.2 2.2
2 2.2
4.0 2.1 4.0 4.0
3 2.2 4.0
2.1 6.2 6.0
4 2.1 2.1 2.1
2.1 2.1
5 2.2 4.0 6.2 2.1
6.0
6 2.2 4.0 6.0 2.1 6.0
Knowledge Similarity: Marketing
Mbr 1
2
3
4
5
6
1
5.5 4.7 5.5 5.5 5.5
2 5.5
4.7 6.5 6.5 6.4
3 4.7 4.7
4.7 4.7 4.7
4 5.5 6.5 4.7
6.8 6.4
5 5.5 6.5 4.7 6.8
6.4
6 5.5 6.4 4.7 6.4 6.4
TKSt
TKS =  TKSt
Knowledge Similarity: All Tasks
Mbr 1
2
3
4
5
6
1
9.7 8.9 9.6 9.7 9.7
2
9.7
10.8 11.3 13.2 13.1
3
8.9 10.8
8.9 13.0 12.8
4
9.6 11.3 8.9
12.1 11.7
5
9.7 13.2 13.0 12.1
18.2
6
9.7 13.1 12.8 11.7 18.2
Scale
Min value =
Max value =
Tasks =
Members =
1
7
3
6
SMMTask = 0.64
Visual Representation:
SMMTask Sociograms
1
1
1
1
2
3
2
3
2
3
2
3
4
5
4
5
4
5
4
5
6
6
6
6
Finance
Production
Marketing
Aggregate
Cutoff x*=4
Cutoff x*=4
Cutoff x*=4
Cutoff x*=12
8/8/2001
A. Espinosa, AoM 2001
10
SMMTask Measure
b) Member's task knowledge cannot be evaluated
•
•
•
•
•
•
•
•
•
Instead of having knowledge ratings in T task areas
Need to ask Q task-relevant questions [Levesque et. al. 2001]
Use an ordinal rating scale for the answers
Use similar method to a) but instead of task areas
Compute distance (i.e., dissimilarity) of responses
dqij = |rqi – rqj| for each dyad (i,j) & question q
Similarity (reverse scale) = scale range – distance
Alternatively: compute similarities using correlation in responses
Then model all dyadic values into TKSq matrices
Aggregate (and normalize to 0-1) into TKS
8/8/2001
A. Espinosa, AoM 2001
11
SMMTeam Measure
Response Matrices Rq
Mbr
1
2
3
4
5
6
1
5
1
1
6
2
2
Q1: Mbr's Fin Task Knowlg
2
3
4
5
4
5
2
1
2
1
1
2
1
1
1
1
6
6
6
6
2
2
3
3
2
2
3
3
6
6
1
1
6
3
3
Q2: Mbr's Prod Task Knowlg
1
2
3
4
5
6
5
4
5
2
5
5
1
2
2
1
2
1
1
1
1
1
1
1
6
6
6
6
6
6
3
4
2
5
6
6
3
4
2
5
6
6
Q3: Mbr's Mktg Task Knowlg
1
2
3
4
5
6
6
4
5
2
1
5
2
2
1
1
1
2
1
1
1
1
1
1
6
6
6
6
6
6
3
3
2
2
1
2
3
3
2
2
1
2
mdqij = avg(|rqi- rqj|) = average distance (dissimilarity) on question q
between members i and j on their knowledge ratings of all members
Member Distance Matrices MDq
Mbr
1
2
3
4
5
6
Q1: Mbr's Fin Task Knowlg
1
2
3
4
5
6
2.83 2.83 2.17 2.33 2.33
2.83
0.33 4.67 1.17 1.17
2.83 0.33
5.00 1.50 1.50
2.17 4.67 5.00
3.50 3.50
2.33 1.17 1.50 3.50
0.00
2.33 1.17 1.50 3.50 0.00
Q2: Mbr's Prod Task Knowlg
1
2
3
4
5
6
2.83 3.33 1.67 1.67 1.67
2.83
0.50 4.50 2.83 2.83
3.33 0.50
5.00 3.33 3.33
1.67 4.50 5.00
1.67 1.67
1.67 2.83 3.33 1.67
0.00
1.67 2.83 3.33 1.67 0.00
Q3: Mbr's Mktg Task Knowlg
1
2
3
4
5
6
2.33 2.83 2.17 1.67 1.67
2.33
0.50 4.50 0.67 0.67
2.83 0.50
5.00 1.17 1.17
2.17 4.50 5.00
3.83 3.83
1.67 0.67 1.17 3.83
0.00
1.67 0.67 1.17 3.83 0.00
Method: SMMTeam Measure (cont'd.)
msqij = scale range - mdqij  max/0 dist = 0/max similarity
Member Similarity Matrices MSQ
Mbr
1
2
3
4
5
6
Q1: Mbr's Fin Task Knowlg
1
2
3
4
5
6
2.17 2.17 2.83 2.67 2.67
2.17
4.67 0.33 3.83 3.83
2.17 4.67
0.00 3.50 3.50
2.83 0.33 0.00
1.50 1.50
2.67 3.83 3.50 1.50
5.00
2.67 3.83 3.50 1.50 5.00
Q2: Mbr's Prod Task Knowlg
1
2
3
4
5
6
2.17 1.67 3.33 3.33 3.33
2.17
4.50 0.50 2.17 2.17
1.67 4.50
0.00 1.67 1.67
3.33 0.50 0.00
3.33 3.33
3.33 2.17 1.67 3.33
5.00
3.33 2.17 1.67 3.33 5.00
MS = Avg(MSq)
Q3: Mbr's Mktg Task Knowlg
1
2
3
4
5
6
2.67 2.17 2.83 3.33 3.33
2.67
4.50 0.50 4.33 4.33
2.17 4.50
0.00 3.83 3.83
2.83 0.50 0.00
1.17 1.17
3.33 4.33 3.83 1.17
5.00
3.33 4.33 3.83 1.17 5.00
Alternative: similarities
based on correlation values
Overall Member Similarity Matrix MS
Mbr
1
2
3
4
5
6
1
2.33
2.00
3.00
3.11
3.11
Un-Normalized
2
3
4
2.33 2.00 3.00
4.56 0.44
4.56
0.00
0.44 0.00
3.44 3.00 2.00
3.44 3.00 2.00
5
3.11
3.44
3.00
2.00
5.00
6
3.11
3.44
3.00
2.00
5.00
1
0.47
0.40
0.60
0.62
0.62
Normalized for Scale
2
3
4
5
0.47 0.40 0.60 0.62
0.91 0.09 0.69
0.91
0.00 0.60
0.09 0.00
0.40
0.69 0.60 0.40
0.69 0.60 0.40 1.00
6
0.62
0.69
0.60
0.40
1.00
Scale:
Min = 1
Max= 6
Questions = 3
SMMTeam =
0.54
Visual Representation:
SMMTeam Sociograms
1
2
3
4
5
6
Average member rating distance of 2 scale points or less
Average member rating distance of 1 scale point or less
8/8/2001
A. Espinosa, AoM 2001
14
Preliminary Internal Validity Testing
Data
• 57 teams from CMU's Management Game Course (n=4-6)
• Teams manage simulated companies for 10 weeks +
• No lectures in course, just team competition via simulation
• Teams report to a board of directors (external)
• 3 surveys + financial performance data + 3 board evaluations
Validity
• Convergent and concurrent validity [Ghiselli et. al. 1981]
8/8/2001
A. Espinosa, AoM 2001
15
Convergent Validity Results
It measures what we wish to measure [Ghiselli et. al. 1981]
1. SMM's should increase over time
through team interaction
.8
99
.9
.7
.8
SMMTask, F=50.902, p<0.001
SMMTeam, n.s., marginally T1-T2
SMM of the Team
.6
SMM of the Task
[Cannon-Bowers et. al. 1993;
Klimosky et. al. 1994]
1.0
.9
.5
.4
.3
N =
48
47
32
1
2
3
SurveyNo
.7
5
.6
N =
48
47
32
1
2
3
SurveyNo
Team interaction: indiv comm frequency rating w/each member
SMMTask, =0.58, p<0.001
SMMTeam, =0.27, p=0.002
2. Stronger SMM's should be associated with more knowledge overlap
3 questionnaire items on perceived knowledge overlap, =0.75
SMMTask, =0.51, p<0.001
SMMTeam, =0.22, p=0.011
8/8/2001
A. Espinosa, AoM 2001
16
Concurrent Validity
Correlation with variables SMM should affect [Ghiselli et. al. 1981]
SMMs should affect performance by improving team process
(e.g., strategy and task coordination) [Klimoski et. al. 1994]
Cohesive Strategy: 6 questionnaire items, =0.84
SMMTask, =0.59, p<0.001
SMMTeam, =0.22, p=0.012
Task Coordination: 9 questionnaire items, =0.79
SMMTask, =0.40, p<0.001
SMMTeam, =0.21, p=0.020
Performance: BOD evaluations, 11 questions, =0.97
Cohesive Strategy, =0.373, p<0.001 (more visible to BOD)
Task Coordination, =0.228, p<0.010
8/8/2001
A. Espinosa, AoM 2001
17
Conclusions
Measures proposed:
(1) Computationally simple
(2) Can be used with correlation, distance or overlap metrics
(3) Model SMM at different levels of detail
(4) Visual representation
(5) Some internal validity
SMMTask has better properties than SMMTeam, possibly:
(1) Not enough time in task for SMMTeam to develop
(2) SMMTeam not as important for this type of task
(3) SMMTeam is strong, but not accurate
Limitations:
(1) Need more thorough validity and mediation testing
(2) Need to test in other contexts
(3) Only two types of SMMs explored
(4) Knowledge (not structure) similarity only
Questions ?
8/8/2001
A. Espinosa, AoM 2001
19