Method and Application of
Protocol Analysis
Anguo Dong
Computer Science
Department, University
of Calgary
Topics covered




What are protocol and protocol analysis
History of protocol analysis
Three types of protocol analysis and their
applications
A comprehensive protocol analysis software
and WBG (Web Behavior Graph)
What are protocol and protocol
analysis?


Definition 1: Protocols are participants'
accounts of their thought processes during
the performance of a specific task [4].
Definition 2: Protocol analysis is a method
of inferring how humans reason and solve
problems, using the protocols collected as
data[1].
Topics Covered




What are protocol and protocol analysis
History of protocol analysis
Three types of protocol analysis and their
applications
A comprehensive protocol analysis software
and WBG (Web Behavior Graph)
Topics Covered




What are protocol and protocol analysis
History of protocol analysis
Three types of protocol analysis and their
applications
A comprehensive protocol analysis software
and WBG (Web Behavior Graph)
Three types of protocol analysis and
their applications



Verbal protocol analysis
Discrete action protocol analysis
Eye-movement protocol analysis
Verbal Protocol Analysis (I)



Think out aloud -- obtaining data from verbal
utterances
Two basic approaches -- the concurrent or
“think aloud” method and the retrospective
method
Time delay occurring in retrospective
protocol
Verbal Protocol Analysis (II)

Summary of the steps of verbal protocol
analysis
1.
2.
3.
4.
5.
Taping
Transcription
Code development or encoding vocabulary
Segmenting and encoding process
Analysis
Verbal Protocol Analysis (III)
After censoring the tape, the following
transcriptions are produced.
… I want to store the important phone
number 210-1234 in memory. Em, First, I
need to press the mem function key. Then
press the number 2101234 in sequence.
Ok, next press the mem function key
again. Finally, enter the memory location. I
will choose 6 because the memory
location has to be between 0-9 and the
first 6 have been occupied. …”
Verbal Protocol Analysis (IV)
11. … I want to store the important
phone number 210-1234 in memory.
Store(2101234, memory)
12. Em, First, I need to press the mem
function key.
Press(mem)
13. Then press the number 2101234 in
sequence.
Press(1), Press(2), …,
Press(4)
Press(mem)
14. Ok, next press the mem function
key again.
15. Finally, enter the memory location. I
will choose 6
Goal(2101234, memLoc)
16. because the memory location has
to be between 0-9 and the first 6 have
been occupied
Because[Between(memL
oc; 0, 9) & Occupied(0, 5)]
Enter(6)
Three types of protocol analysis and
their applications



Verbal protocol analysis
Discrete action protocol analysis
Eye-movement protocol analysis
Discrete Action Protocol Analysis


Discrete Action Protocol Analysis collects the
keystrokes, mouse clicks, or other actions
that people exhibits as they engage in
interactive behavior [3].
ACT-PRO -- A tool for analyzing discrete
action protocols
Basic Structure of ACT-PRO [3]
Discrete Action
Protocol
ACT-PRO
Grammars
of action
patterns
Manual
modification
based on
results
Grouping
program
Grouping
program
Tracing
program
Grouping
program
Hierarchical
structure
Manual modification
based on results
The grouping and tracing programs of ACT-PRO. Solid lines represent
direct input; dotted lines represent manual modification based on the
results of the program outputs.
Task-relevant Hierarchical Goal
Structure of a VCR Interface [3]
PROGRAM-A-SHOW
SET-START-TIME
SETSTAR
THOU
R
Starthour
button
SETSTART
10MIN
Start10min
button
SET-END-TIME
SETSTAR
TMIN
Startmin
button
SETENDHOU
R
Endhour
button
SETEND10MI
N
End10min
button
SETCHANNEL
SET-DAY-OFWEEK
SETENDMIN
Endmin
button
Channe
l button
Dayofweek
button
A Syntactic Representation of Action
Pattern – Grammar
SET-CHANNEL: [Object1][Object2][Object3 ]
(1)
[Object1] → channel
(2)
[Object2] → up-arrow
(3)
[Object2] → down-arrow
(4)
[Object2] → up-arrow [Object2]
(5)
[Object2] → down-arrow [Object2]
(6)
[Object2] → enter [Object2]
(7)
[Object3] → enter
(8)
Parsing the Action Protocol by Using
the Grammars


When a sequence of actions matches all the
grammar rules in a grammar in the specified
order, the sequence of actions is “captured”
by the grammar [3].
Example: if the sequence of actions channel,
up-arrow, down-arrow, enter is matched to
the grammars.
Three types of protocol analysis and
their applications



Verbal protocol analysis
Discrete action protocol analysis
Eye-movement protocol analysis
Eye-movement Protocol Analysis (I)
By studying eye movements,
researchers can understand
user behavior in basic interface
tasks to reveal how users
encode and process information
and to infer user intent in realtime interfaces [4]
Eye-movement Protocol Analysis (II)

Pros
–
–
–

Can provide a wealth of information
Convenient
Subjects need little instruction and training
Cons
–
–
–
Generate enormous sets of eye-movement data
Include a great deal of equipment noise
High degree of individual variability
Topics covered




What are protocol and protocol analysis
History of protocol analysis
Three types of protocol analysis and their
applications
A comprehensive protocol analysis software
and WBG (Web Behavior Graph)
A Comprehensive Protocol Analysis
System -- SNIF-ACT [5]
WBG -- Web Behavior Graph [5]
References






[1] Fu, W.-T., & Gray, W. D. (2000). Implications of rational analysis for
interface design: Observations in a natural learning environment.
Manuscript submitted for publication.
[2] D.Anders Ericsson & Herbert A. Simon (1993). Protocol Analysis:
Verbal Report as Data. Massachusetts Institute of Technology
[3] WAI-TAT FU ACT-PRO action protocol analyzer: A tool for
analyzing discrete action protocols. Behavior Research Methods,
Instruments, & Computers 2001, 33 (2), 149-158
[4] Dario D. Salvucci & John R. Anderson Automated Eye-Movement
Protocol Analysis. Carnegie Mellon University, Cambridge Basic
Research
[5] User Interface Research at PARC
http://www2.parc.com/istl/projects/uir/projects/snif-act/
[6] Ericsson Protocol Analysis and Verbal Reports on Thinking – An
Updated and Extracted Version From Ericsson(2002).
Question and Answer