Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
Application of ThinkLets to Team Cognitive Task Analysis
Robert J. Harder
US Army Research Laboratory
[email protected]
Abstract
As was seen in the 2003 Gulf War, the U.S. Army is
migrating to a lighter, more mobile and more lethal
fighting force. In support of this desired paradigm, the
major U.S. Army’s battle laboratories are performing key
experiments to determine how the projected mix of
personnel, materiel, and doctrine can be interwoven into
a desired structure for the year 2015. In February 2003,
the U.S. Army’s Battle Command Battle Laboratory
(BCBL-L) at Fort Leavenworth, Kansas appraised,
utilizing a more intuitive decision-making process, a
simulated futuristic two thousand-man force structure
with its anticipated equipment. There were several stated,
and a few unstated, objectives to this experiment,
necessitating the use of numerous observers and several
data collection instruments. Of the data collection
instruments, the principal instrument for Team Cognitive
Task Analysis functions was the Wagon Wheel interview
methodology. Of the tools, the principal tool employed to
collaborate, consolidate and sustain the data collection
events was a Group Support System. This paper will first
explore how the selected Group Support System tools
were utilized to automate the Wagon Wheel process from
a one-on-one manual process to an automated system that
enabled simultaneous data collection from 20 individuals.
This paper will then examine how the concept of
ThinkLets was used to define the Wagon Wheel process.
Lastly, an exchange of ideas will be provided, talking
about the strong and weak points of using a Group
Support System in the experiment, the problems that arose
and the solutions employed, and some thoughts for using
a Group Support System in the follow-on experiments.
Howard Higley
US Army Research Laboratory
[email protected]
determine how the anticipated combination of personnel,
materiel, and doctrine could be assembled into a working,
functional concept. And as a secondary goal, it was
important to determine whether the desired results were
indeed desirable. The following subsections describe the
experiment, the tools used, and some background on the
methodologies employed.
1.1. Experiment Overview
In February 2003, the U.S. Army Battle Command
Battle Laboratory at Fort Leavenworth (BCBL-L)
conducted its first experiment to explore the functions and
composition of a 2015 “brigade equivalent” staff. The
experiment was preceded by two major workshops to
determine functions and explore decision-making
processes [2]. The experiment was staged over a twoweek period using an abbreviated brigade level staff (25%
of normal) and several one-person cells representing
subordinate units. Experimental issues included a new
brigade force structure, a more intuitive decision-making
process, and futuristic equipment. The first week of the
experiment dealt with training on these issues. The
second week comprised three simulations driven battle
“runs,” followed by a battle run without simulation [3].
Over a dozen data collection instruments and techniques
were exercised. A Group Support System (GSS) was
employed to enable most of those instruments and
techniques [4].
1.2. Group Support Systems
1. Introduction.
The Group Support Systems (GSS) used in the
experiment was GroupSystems.
GroupSystems is
comprised of several automated tools that allow
participants to provide input in a simultaneous and
anonymous nature. The tools allow a group to assimilate,
brainstorm, organize, evaluate, and prioritize information.
Over the past few years, the threats to the United
States of America have changed considerably. To counter
these threats, the U.S. Army is aggressively engaged in a
major effort to evolve to a lighter, more mobile and more
lethal fighting force. This futuristic force, to be in place
in the year 2015, is called the Objective Force [1]. In
support of this vision, the U.S. Army’s battle laboratories
were given the mission to perform several experiments to
The GroupSystems tool used to assimilate the data
collection effort described in this paper is called
Categorizer. Categorizer allows for ideas to be generated
and organized into discrete storage areas called “buckets.”
In each bucket an idea can have a series of comments that
provides additional information. Also, the comments can
be further annotated with small electronic yellow
annotations allowing for the quick identification of key
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
1
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
It is important to note a key attribute of a GSS –
anonymity - was turned off. Identifiable labels were
attached to each response during the workshops and most
of the experiment. Only one activity was performed using
anonymity.
1.3. Team Cognitive Task Analysis
Cognitive Task Analysis (CTA) is a collection of
methods used to describe the cognition necessary for task
performance. CTA is normally focused on individual
cognition in task performance as can be seen in Klinger
[6]. Team CTA methods view the team as an intelligent
entity and attempt to identify the cognitive processes
required by team dependent tasks. “To capture the
cognitive processes of a team, you must focus on the way
the team coordinates the understanding of the different
members and synthesizes the task elements.
This
represents the starting point for Team CTA” [6].
Furthermore, there are numerous techniques available for
Team CTA. The three techniques used for the BCBL-L
experiment were the Team Audit, the Wagon Wheel, and
the Decision Requirements Exercise [3]. All three of the
techniques require almost 2 hours to complete. All are
interview-based procedures. The Team Audit can be done
individually or in a group. The Wagon Wheel is
traditionally an individual interview and the Decision
Requirements Exercise is a group effort. Other than tape
recording devices, these methods are not usually
supported with automation.
1.4. ThinkLets
The existing basic and applied research in Group
Support Systems (GSS) has shown mixed results [7].
Consequently, a concept called a ThinkLet was developed
to provide an underlying mechanism for documenting and
describing group processes. “A ThinkLet is the smallest
unit of intellectual capital required to create one
repeatable, predictable pattern of thinking among people
working toward a goal” [7]. These small units of
intellectual capital are combined in a logical sequence to
form a “pattern.” As such, repeatable patterns can be
established and run. Accordingly, any desired change to
the pattern can be easily affected by modifying the
appropriate ThinkLet.
On closer examination, a ThinkLet can be broken
down into three components: a tool, a configuration, and
a script.
The tool is the hardware and software
technology employed to create the pattern of thinking; the
configuration consists of the specified settings of the
hardware and software; and, the script consists of the
instructions and sequence of events used by the facilitator
and given to the group to create the pattern of desired
thinking [7]. shows the notation for a ThinkLet [8],
where the top line gives the name of the ThinkLet; the
middle section provides the name and description of the
activity; and the text on the left describes the pattern of
collaboration.
Within the ThinkLet paradigm, there are five patterns
of collaboration: diverge, converge, organize, evaluate,
and build consensus. Within the diverge pattern, the most
common application of a divergence ThinkLet is
LeafHopper, which is simply a list of items for
brainstorming by a group [7]. LeafHopper is employed
when the group must brainstorm on several topics at one
time and that different participants may have different
levels of interest or expertise in the different topics.
Activity
Decision
thinkLet
Pattern of
Collaboration
material [5].
All three levels of identification or
separation were utilized.
Activity name
Flow direction
'decision outcome'
Figure 1. ThinkLet notation
2. Traditional Wagon Wheel
As established earlier, a key tool in Team Cognitive
Task Analysis is the Wagon Wheel [6]. In an ideal
environment, the Wagon Wheel process would be a
highly effective method for determining the various
communication characteristics among personnel. For this
experiment, the conditions were anything but ideal, so an
alternative was required. Nonetheless, to understand the
alternative, it is critical to understand from where the
alternative was derived – the traditional Wagon Wheel
[6].
The principal goal of the traditional Wagon Wheel
process is to determine the main communication channels
for each position on the team. Once determined, a
secondary goal can be pursued - the investigation of the
nature of those communications. The Wagon Wheel is
usually performed manually in a one-on-one environment
during a one-time, two-hour session. The session starts
with the interviewee given a blank, paper copy Wagon
Wheel template and instructed on how to complete the
template. Furthermore, a white board or picture is
typically used as a reference device between the
interviewer and the interviewee. The Wagon Wheel
process is achieved in three simple steps, as summarized
in Klinger [6] and illustrated in Figure2.
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
2
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
x
x
is longer, but the quality of the information is more
than acceptable;
interviewing each member in a separate location.
“When several members are interviewed together, we
find that they often talk themselves into how
important, or unimportant, a link is;” [6] and,
using more than one interviewer.
2.3. Step Three – Analyze the Results.
Figure 2. Wagon Wheel
2.1. Step One – Determine Relationships.
The interviewee’s name is placed in the center of the
board as shown in Figure2. The interviewer asks
questions concerning the other team members that the
interviewee communicates with on team tasks. The
names or roles are placed around the center of the board
in the outside circles and arrows used to indicate the
communication direction. The final process is to number
the relationships in ranked order of volume/frequency of
contact. In the example in , persons 3 and 6 “talk” to the
interviewee, but not vice versa. This first snapshot can
provide a great deal of information concerning team
interactions. The next step is to determine more details of
the interactions.
2.2. Step Two – Probing the Interactions.
The interviewer probes for more information
regarding the connections in the Wagon Wheel by asking
questions, such as: what type of information was
exchange, where was that information obtained, how did
the interviewee use the information, what decisions did
the information affect, and how critical was the
information. The nature and number of questions depends
upon the goal of the Team CTA analysis. For example, if
the goal of the analysis were to redesign the command
post, the interviewer would ask questions regarding faceto-face communications and the ability of radio
communications to support the desired relationships.
Some additional aspects of the interview include:
x
x
interviewing team members individually;
establishing a good repartee by asking general, nonintrusive questions to start. In this manner and over
time, the interviewer can become conscience of the
interviewee’s body language and other signals, and
can adjust the interview technique accordingly. For
this and other reasons, the time required to interview
For better data analysis, Klinger recommends placing
the various Wagon Wheels side by side for better
comparison [6].
The missing and/or inaccurate
information can be identified in this manner and the
process started anew, the information reconciled with the
individuals involved, or the data can be adjusted. The
only drawback to this process is the additional time
required to perform this function.
2.4. Why use GSS with the Wagon Wheel.
For the battle lab experiment, executing the Wagon
Wheel in the traditional manner was deemed intractable.
The time constraints of the experiment forced the
interviews to be concurrent with other activities and there
were insufficient interviewers to conduct twenty
simultaneous interviews. This served as the principle
motivation to semi-automate the Wagon Wheel process
using GSS tools. Accordingly, several planning sessions
were held with the creators of the traditional Wagon
Wheel process. Pros and cons were discussed. For
example, it was noted early on that an automated
technique would lose the interactive ability to probe for
more information. However, the consensus was that we
could get the basic relationship information in this
manner. Also, due to time constraints, the normal 2-hour
interview had to be shortened to three sessions and each
session was limited to thirty minutes. Unfortunately, each
of the three sessions was at the end of one of the battle
runs, so the relationships under study were still being
formed.
Additionally, the space allocated for the
experiment did not allow for a meeting room of twenty
users. Instead, a small room with twelve machines and
another room with a group of eight machines had to be
used for data collection. This required, at a minimum,
that a GSS facilitator and one other person familiar with
the Wagon Wheel process be present.
3. Methodology and ThinkLets Employed
ThinkLets are not dependent on the technology that is
employed [8]. ThinkLets can be prepared using manual
techniques or supported by automation. Since the GSS
personnel were unfamiliar with the Wagon Wheel
technique, they employed ThinkLet concepts in the design
to better understand the Wagon Wheel concept and to
better transition from a manual process to a semiautomated one. Figure 3 shows the pattern of thinking
used for the Wagon Wheel building process, with
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
3
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
ThinkLets being “strung” together to yield a desired
result. Breaking down Figure 3, subsections 3.1 through
3.4 describe each ThinkLet is used in the Wagon Wheel
process. The generic script and the general configuration
applicable to the process are incorporated as subsections
3.5 and 3.6.
The final subsection explains those
additional areas of technical support that were required to
run the Wagon Wheel using a GSS.
Preliminaries
3.1. Preliminaries
The first phase of the process was to introduce the
objective of the Wagon Wheel. Based on discussions
with the traditional Wagon Wheel experts, participants
were given a Wagon Wheel outline on a piece of paper
and asked to manually fill the information in as depicted
in Figure 2. The feeling was that using pen and paper at
the beginning of the Wagon Wheel would help the
participants visualize the wheel and, thus, understand the
“wheel metaphor” being used.
Other
3.2. ExpertChoice ThinkLet
Introduce objective
Develop basic
Wagon Wheel
This next step in the process was to translate the
completed wheel into the GSS. To execute this task, a
modified ExpertChoice was used. ExpertChoice is an
organizing ThinkLet and is used when there is someone in
the group best qualified to organize a set of ideas. In this
situation, the best-qualified person is the person in the
center of each wheel. To start the GSS supported process,
each player was given a “personal bucket” in the GSS tool
that listed all the players, as can be seen in Figure 4.
Organize
ExpertChoice
Select players;
Delineate
Relationships from
master list
Multiple
LeafHoppers
with different
scripts
Diverge
Diverge
LeafHopper
LeafHopper
Obtain information
on relationships,
media, and
decisions
Do comments
need
clarification?
Yes
No
Converge
Pin the Tail
Facilitators request
additional
information
Consensus
Point Counter Point
Players provide
input on
relationships with
other players
Proposed
Other
Final Activities
Final questions
and concluding
activities
Figure 4. Starting template
Next, the participants were asked to go into their bucket
and move all the players they named on their paper
diagram above the “===”, or cut line, and place them in
rank order. After that, they were to put in “arrows” on the
player line indicating the TO/FROM relationship as
shown in Figure 5. Several players were unable to attend
the first day and started their wheel on the second day. To
the surprise and the delight of the observers and
researchers, the late participants found it unnecessary to
do the Wagon Wheel paper drawing first. As will be
discussed later, this ThinkLet alone provided some of the
major insights used in the overall experiment report [3].
Figure 3. Wagon Wheel in Thinklet notation
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
4
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
Figure 5. Typical wheel after ExpertChoice
Figure 6. LeafHopper after all iterations
3.3. LeafHopper ThinkLet
3.4. Pin the Tail ThinkLet
The majority of the data normally obtained from the
interview process was obtained by asking the group to
provide additional information exchanged for each person
noted above the cut line. This was envisioned as a series
of LeafHopper ThinkLets with different scripts as
depicted in Figure 3. The LeafHopper ThinkLet is used
when the group needs to brainstorm on multiple topics.
Again, the true LeafHopper ThinkLet was modified in the
sense that each person was only “hopping” within his or
her own personal set of relationships. The first iteration
of the modified LeafHopper was to ask the players to list
information exchanged with each person. An example is
shown in the white comment screen in Figure 6. The
subsequent iteration asked them to list the mechanism
(audio, video, text, etc) used to obtain the information.
The third iteration on the second day was to add the
decisions that were being influenced by the receipt of this
information. An example of this process can be seen in
the white comment screen in Figure 7. Combined in
order, the three iterations were the basic flow of the
traditional interview process. However, by the third
iteration of the LeafHopper, the group grumbled that they
could have easily answered all three questions at one
time. The following is a suggested script based on their
input:
Unfortunately, the quality and quantity of
information varied considerably at the end of the first day.
Some variability was not unexpected as in an interview
process if the interviewee provides something unexpected,
a clarifying question can be made and the data modified.
A good example of these differences is shown in Figure 7
in the white comment screen.
For each interaction, please list the information you
exchanged with the individual and for each distinct
piece of information, list the mechanisms you used,
and the decisions that that information informed.
The first two bullets address the three previous
questions (data, mechanism, and decision) and the last
bullet addresses the final request, which was to describe
the overall flow of the relationship. In retrospect, the
information in Figure 7 was clearer and easier to
understand than the more cryptic information shown in
Figure 6.
The value of this combined script was verified when
those individuals unable to attend the first day easily
addressed all the questions at one time and didn’t need the
paper diagram to begin with. The lesson learned is that
the sequential script pattern of the interview may have to
be significantly modified to adapt to a more GSS oriented
script.
Figure 7. LeafHopper – final iteration
Although the facilitators and the Wagon Wheel
experts were “in the meeting”, looking at input, there was
not sufficient time to analyze the entire set of responses
for quality control. Some of these differences were noted
and in the worst cases, a variation of the PinTheTail on
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
5
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
the Donkey ThinkLet was used to try to illicit more
information. PinTheTail is usually used to build a shared
understanding within a group on comments. In this case,
PinTheTail was used to flag comments and request the
participant provide additional information.
Not all
participants were “pinned” and those that were joked
about being “red lined” as they felt they were in class
again and getting notes from the teacher on their papers.
An example of this technique is shown in Figure 8. The
“Pin” is the small yellow sticky in the center of the white
comment page. The original comment has been modified
based on the request of the “pin” – shown in the lower
left. In hindsight, the PinTheTail ThinkLet should have
been employed concurrently with the other ThinkLets.
x
x
x
x
x
x
x
x
DO: Assure each person is in a unique bucket – maybe put in test
comment.
EXPLAIN:
Move most important person above the cut line
<> notation – Use F5 edit the heading
Add information exchanged for that person
For each piece of information, explain how they got it.
Audio, text, face to face, lecture/one-way, graphic, email,
DO: Repeat last step for each person – one at a time
DO: End day; collect sheets
Day Two
x
x
x
x
x
DO: Have players update diagram; then reprioritize, etc
EXPLAIN:
Modify each information item with decisions
supported
DO: Watch time and content
DO: CHECKPOINT for content
DO: End day; collect sheets
x
x
x
x
x
DO: Add questions to the list
EXPLAIN: Questions – one at a time.
DO: Have players update diagram
EXPLAIN: Questions – do one at a time
DO: End day; collect sheets
Day Three
Figure 9. Wagon Wheel checklist
3.6. Configuration
Figure 8. PinTheTail thinkLet
3.5. Scripts
The ThinkLets in Figure 3 were all executed within
one tool in GroupSystems. As such, the configuration
remained relatively stable. This allowed the participants
to be given modify privileges to the headings and to all
comments.
Alas, such privileges also gave the
participants the ability to delete everything in the activity.
To insure disaster did not strike, all of the users were
cautioned to use the delete key very sparingly and backup
procedures were performed frequently. However, this did
indicate to the researchers of the need to include any
technical “dos and don’ts” within the script for the
“preliminaries.”
3.7. Technical Support
Because the Wagon Wheel sessions were split into
two distinct and separate areas, it was necessary to try to
synchronize the facilitation of the two rooms. Figure 9
shows the checklist that was developed. With the
exception of the individuals who were playing catch-up,
this served as the master sheet for the scripts for each
ThinkLet. Although sufficient for the execution, this
technique does not clearly document the script for each
ThinkLet.
For example, the scripts for the three
LeafHopper ThinkLets are actually under the last
EXPLAIN bullet in Day One shown in Figure9.
There were several data collection endeavors being
performed during the experiment, with most data
collection efforts relying on GroupSystems for their needs
[4]. Consequently, only one systems administration
person was needed, and was on call, during the entire
experiment.
Had several software systems been
employed to collect data and observations, continuous onsite and/or more than one system administrator may have
been considered necessary. As it were, other than the
occasional laptop misbehaving, no GSS technical
assistance was required.
Day One
x
x
x
x
x
x
x
x
x
x
DO: Handout
EXPLAIN:
Write name in center
Write positions you deal with in outer circles
Add more circles than the drawing if needed
Provide direction of information exchanged by using arrows
Can be two-way
Rank the links by volume (1, 2,3 …)
DO: Wait for players to get mostly through with the diagram
DO: Open Wagon Wheel Activity
EXPLAIN: Personal Bucket
An important feature of any technical support
component is to insure all the information produced is
preserved. For the GSS component in this experiment,
assistance by the system administrator was not needed.
Complete backups were conducted each night by simply
saving the GSS meeting to a hard drive on a laptop. A
facilitator of GSS can and did accomplish this task. In
fact, anyone with the correct privileges could accomplish
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
6
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
this task because the procedures were posted on a bulletin
board in front of the primary facilitator’s machine.
Additionally, after each session, the activity was
duplicated and one duplicate renamed for use the next
day. This procedure accomplished two functions. First,
the information was preserved, as it was possible for a
player to change information from day to day. Second, it
was thought that it might be important in later analysis to
see the evolution of the material. As it turned out, players
made only minor changes and additions to their original
order and listings of individuals from their first day.
4. Results
For this experiment, three days were allocated to
collecting Wagon Wheel data. Since the exercises were
similar in construct on days two and three, as it was for
day one, the players were given the opportunity to revise
their wheel, adding and/or updating players, information,
and relationships.
Figure 10. Additional questions
On the final day, all the players were asked to look
over their input for consistency and to answer four
additional questions. The four questions are shown in
Figure 10, below the cut line. Also notice, by this time in
the process, the players not talked to frequently have been
eliminated from the list. In Figure 10, only nine names
remain on the list, down from a start of twenty-eight.
Another observed result was that time was saved
using the GSS enabled Wagon Wheel. Based on an
estimated 2 hours for a one-on-one interview using the
traditional Wagon Wheel process, the GSS enabled
Wagon Wheel methodology reduced player input time by
at least 25% from 2 hours to 1.5 hours (approximately ½
hour per day over three days). Indeed, by the end of the
third day, a portion of the ½ hour allocated for the Wagon
Wheel was actually being used for collecting experiment
feedback.
One more time and effort savings was in the area of
interviewer’s time. The required interviewer’s time was
reduced by at least eighty per cent. Twenty interviewers
were reduced to 2 GSS facilitators and 2 experienced
Wagon Wheel personnel in the GSS enabled Wagon
Wheel process. In fact, by the end of the third day, the
GSS facilitators were running the sessions without
assistance from the Wagon Wheel experts; the Wagon
Wheel experts were released to conduct reconciliation of
other observation data. So in actuality, the interviewer
time reduction was higher than eighty per cent.
One expected negative finding was that the GSS
enabled Wagon Wheel procedure did not allow for the indepth probing that the traditional Wagon Wheel provides.
The in-depth probing also forms a quality check on
information so that any information not understood is
immediately clarified.
The final data did show
inconsistencies in the formatting and in the depth of the
answers. Consequently, additional quality control may
require one or several dedicated interviewer to interact
with the interviewees during the sessions [4]. This is
further discussed in the Future Efforts chapter.
Lastly, although it was not the purpose of this paper
to discuss the consequences of the experiment, some of
the experimental results caused rethinking of the GSS
enabled Wagon Wheel process resulting in the need for
additional ThinkLets. The observation causing the most
consternation was when Player A would list Player B as
an important part of their wheel, while Player B would not
list Player A. As declared earlier, Klinger [6] suggests
visually looking at the diagrams. In the GSS version, this
is very difficult to do. During the analysis phase after the
experiment was concluded, a spider diagram was
developed and did prove useful to illustrate the various
disconnects [3]. A ThinkLet for correcting this dilemma
is also suggested in the Future Efforts chapter along with
using organizational visualization techniques.
5. Future Efforts.
The use of automation, let alone GSS, for the Wagon
Wheel process was new to the cognitive task analysis
experts. Positive feedback was received and based on this
first experience, several suggestions were developed for
future uses.
5.1. RedLine ThinkLet
The RedLine ThinkLet is proposed as an enhanced
version of the PinTheTail ThinkLet. The RedLine
ThinkLet should be chosen when individual comments
need clarification and when adjustments are necessary in
order to normalize them with other comments.
The
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
7
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
inputs are a set of comments and the outputs are a set of
notations on specific comments followed by an improved
revised set of comments. The major steps would be to
determine the final state of the comments and the typical
things that needed to be done to the comments. RedLine
could be considered a form of quality control. For
example, if a group was gathered to build a document,
each paragraph may need to have certain information.
The annotations would be used to flag those portions of
the document that needed additional material. Another
name for this ThinkLet could be QualityControl.
5.2. Addition of PointCounterPoint ThinkLet
As affirmed earlier, one of the key results was that
not all participants had the same viewpoint of their
interactions as other participants. In one case, no one
listed a given staff position as someone with whom they
frequently communicated. However, the missed player
showed three interactions on his Wagon Wheel input.
This obviously shows a perception disconnect between
the individuals. Since one of the objectives of the
experiment was to determine new staff structures, this
phenomenon is a red flag concerning the perceived value
by others in this particular position. Using the traditional
Wagon Wheel method, this information would be difficult
to determine until after the results of each interview, if
discovered at all. By using the GSS, these “disconnects”
can be seen after the process passes through the first
ThinkLet, ExpertChoice. This would allow some early
exploration of the link discrepancies. Although this early
analysis was not done in the experiment, a suggested
ThinkLet for this has been added for the purpose of
understanding the “disconnects.” This ThinkLet is a
consensus building ThinkLet called PointCounterPoint
and can be found in Figure 3 in the yellow box and is
marked “proposed.”
PointCounterPoint is a three-step activity [7, 8].
Participants first enter the strongest argument in favor of a
definable position. Then, others argue against it. Finally,
someone builds an argument to bridge between the two
positions taken by others. For example, in the Wagon
Wheel Analysis section, Player A was listed by Player B,
but not vice versa. It would then be Player’s A’s role to
try to explain why Player B was not considered. A third
player would be asked to try to resolve the discrepancy.
The key to this PointCounterPoint would be to build
consensus between the two players concerning their
relationship using others as bridge builders. Care would
be taken to avoid the pitfalls Klinger [6] mentions, as
delineated in paragraph 2.3, concerning dwelling on the
importance of the links.
5.3. Pros and Cons of Interviews Over Time
Conducting the Wagon Wheel over a three days time
period had more advantages than disadvantages. As stated
earlier, the group interviews were conducted during the
execution of three sets of scenarios, while relationships
were still being built. Players were encouraged to
continue to add to their relationship wheels and were, as a
by-product, more cognizant of those relationships.
Another advantage is that the interviewers could “course
correct” any major problems. Wagon Wheel sessions
were much smaller and placed less drain at the end of the
day on the players. Additionally, since the Wagon Wheel
exercise became the “end of day” gathering activity,
supplementary non-Wagon Wheel questions were asked
before and after the Wagon Wheel sessions. Since staff
structure exercises are similar in their execution and take
place over one or more weeks, smaller and more frequent
data collection opportunities are recommended.
5.4. Distributed version
Several staff structure experiments planned for the
future will be performed in a distributed mode. That is,
the participants will be in different locations, connected
by a computer hook-up and/or radio. This poses some
unique problems for those collecting Team CTA since the
actual data collection for the Wagon Wheel took place in
only two different rooms and was easily observed.
Consequently, only limited experience in a distributed
GSS Wagon Wheel was obtained. And although each
facilitator was given a script, it became very apparent
after the first question from the players that close
coordination would be needed to make sure that all
participants had the same set of “instructions.” A solution
to this problem could be chat rooms and video
teleconferencing (VTC). But chat rooms and VTCs are
limited in scope, e.g. despite all efforts, certain players
will be simply not available each and every session,
necessitating those players to play catch-up. Both
phenomena would be made exceedingly complex in a
distributed manner. This suggests that some persistent
method of capturing or documenting questions from the
collective group during the Wagon Wheel be established,.
This could be a log of questions or a frequently asked
question reference area embedded later within the GSS.
5.5. Enabling Other Team CTA methods
As presented in the Team Cognitive Task Analysis
paragraph, only two other Team CTA techniques were
used during this experiment. Of the two, the most
promising candidate for enhancement using GSS is the
Decision Requirements Exercise (DRE). In the DRE
activity, the entire user group is walked-through a twodimensional decision requirements table to determine the
requirements necessary to make decisions and judgments.
Input is collected from the group verbally in a one-at-atime manner, as directed by the primary interviewer, and
typed into a collective spreadsheet by a co-interviewer.
The results are then projected to a main screen. A
possible GSS enabled approach would be to restructure
the procedure using Categorizer “buckets” to represent
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
8
Proceedings of the 37th Hawaii International Conference on System Sciences - 2004
decisions and list items, the first sublevel of buckets, to
represent columns.
ThinkLets, these differences can be mitigated in future
experiments of this nature.
5.6. Organization Visualization Techniques
Although the GSS-enabled Wagon Wheel provided
useful information, organizational process analysis
techniques could have been applied to further enhance the
quality of the interviews and, thus, the information.
Organizational visualization techniques provide graphic
representations of the human communication networks
and the decision and influence networks. Armed with this
visual information, the GSS Wagon Wheel could then
have been used to probe into the relationships. An
example of this procedure can be found in Mann [9].
5.7. ThinkLets – Some Additional Thoughts
ThinkLets enable group cognition.
The
ExpertChoice and LeafHopper ThinkLets described in
this paper are single author activities and do not involve
any group dynamics. The last activities, PinTheTail and
the proposed PointCounterPoint, do facilitate group
dynamics. Although Klinger [6] cautions against group
interviews, with the correct combination of ThinkLets, the
dynamics engendered should yield much more
information with the effort invested. Instead of getting
twenty answers of the same or similar response with
twenty one-on-one interviews, the result is a delineated,
single response that is looked at from twenty different
perspectives.
6. Summary
Four major conclusions were derived from this
experiment concerning the use of GSS and ThinkLets.
First, ThinkLets and the underlying theory behind
them provided a unique method for understanding the
cognitive patterns underlying the GSS enabled Wagon
Wheel Team Cognitive Task Analysis.
1 US Army (2000). Concepts for the Objective Force. US
Army White Paper.
2 Lyles, B LTC, McWhorter C. CPT, Wilk C. Major, Steege K.
Major, & Harder R. (2002) Interim Report for Phase I of the
Commander’s Information Fusion Cell (CIFC) CEP. Battle
Command Battle Laboratory – Leavenworth.
3 Ross, K., Thunholm, P., Uehara, M., McHugh, A., Crandall,
B., Battaglia, D., Klein, G., Harder, R. (2003) Unit of Action
Battle Command: Decision-Making Process, Staff Organization,
and Collaboration. Report prepared for Battle Command Battle
Laboratory - Leavenworth.
4 Harder, R. & Higley, H. (2003). Application of a Group
Support System for Experimentation for the Objective Force. In
Proceedings of Systemics, Cybernetics, and Information
Systems Conference 2003.
5 Ventana Corporation (1999). GroupSystems Concepts Guide.
6 Klinger, D. & Hahn B. (2003) Handbook of TEAM CTA.
Contract F41624-97-C-6025 Human Systems Center, Brooks
AFB, Klein Associates.
7 Briggs, R., deVreede, G., Nunamaker, J., & Tobey, D. (2001).
ThinkLets: Achieving Predictable, Repeatable Patterns of
Group Interaction with Group Support Systems (GSS). In
Proceedings of the Thirty-fourth Annual Hawaii International
conference on System Sciences, IEEE Computer Society.
8 Briggs, deVreede, Nunamaker, & Harder (2003). Repeatable
Success with Collaborative Technology: ThinkLets and
Methodologies. Tutorial presented at Hawaii International
Conference on System Sciences (HICCS ’36), January 2003.
9 Mann, M. (2003). Technology in Corporate Design and
Governance. White Paper. Encompass Knowledge Systems,
Inc. www.encompassknowledge.com
Second, ThinkLets can afford a simple and complete
method for documenting a process, with the overall
ThinkLet methodology providing the data required for the
experiment analysis.
Third, the GSS enabled process using ThinkLets
allowed the traditional interview technique to be done in
parallel, thereby saving an inordinate amount of time of
the interviewers and a reasonable amount of time of the
interviewees. Moreover, due to resource constraints and
experimental limitations, without the GSS enabled
technique, considerably less data on perceived
interactions would have been collected.
And fourth, expected differences in the type and
quality of information did occur. But with additional
0-7695-2056-1/04 $17.00 (C) 2004 IEEE
9