International Journal of Medical Informatics (2005) 74, 797—807
Asynchronous communication among clinical
researchers: A study for systems design
John H. Gennari a,∗, Chunhua Weng a, Jacqueline Benedetti b,
David W. McDonald c
a
University of Washington, Department of Medical Education, Biomedical and Health Informatics, 1959
NE Pacific St., Box 35 72 40, Seattle, WA 98195-7240, USA
b The Southwest Oncology Group, Fred Hutchinson Cancer Research Center, USA
c The Information School, University of Washington, USA
Received 10 January 2005; accepted 22 March 2005
KEYWORDS
Asynchronous
communication;
Computer-supported
cooperative work
(CSCW);
Modern health care
Summary Group work is an essential part of modern health care. Although there
have been advances in the technology of communication, these have not necessarily
led to efficient and effective communication among collaborating health-care professionals. Instead, barriers such as varied organizational cultures, different training
backgrounds, and varied time schedules can overwhelm technological solutions and
impede efficient communication. We focus on one particular sort of collaboration,
that of group work around a clinical trial protocol, where the collaboration is asynchronous and the participants are geographically distributed. In this work setting,
we have applied a computer-supported cooperative work (CSCW) approach in two
distinct ways. First, we used observational methods to uncover and understand a
complex set of communication behaviors and needs. Second, we used participatory
design and iterative prototyping to design a system that aims to improve communication and work flow among these collaborators. We found that these methods work
well in tandem—–our observational study helped better inform our design, and our
prototyping cycles provided additional insight into the work. More specifically, we
were able to identify a set of communication problems in the work that led us to
specify a set of design desiderata for systems that support asynchronous collaboration around an evolving medical document.
© 2005 Elsevier Ireland Ltd. All rights reserved.
1. Problem statement
∗
Corresponding author. Tel.: +1 206 616 6641.
E-mail addresses: [email protected]
(J.H. Gennari), [email protected] (C. Weng),
[email protected] (J. Benedetti), [email protected]
(D.W. McDonald).
Communication behaviors among medical collaborators are complex. In many medical settings,
communication is also an essential and potentially
safety-critical part of the work. However, many
informatics systems and technology solutions avoid
1386-5056/$ — see front matter © 2005 Elsevier Ireland Ltd. All rights reserved.
doi:10.1016/j.ijmedinf.2005.03.019
798
or ignore this aspect of work. For example, information systems often retrieve information for a single
user, and decision-support system help individual
users make medical decisions. Although some systems, such as most electronic medical record systems, are designed for multiple users, they often
provide little support or acknowledgement of the
many communication tasks around medical work
and decision making. We argue that the design
of medical informatics systems should explicitly
address the needs of groups of users, who collaborate and communicate with each other to solve
the health-care problem at hand (Coeira et al. also
share this view [1]).
Communication in medical settings is complicated by a number of factors. In some settings,
members of the group cannot work at the same
time, or the same place, and therefore must
communicate over distance and/or time. Medical
problems are often best addressed by multidisciplinary teams of experts. In such groups,
communication may be difficult because of differing educational backgrounds and levels of
technical expertise. Finally, medical problems
are often time-critical: a solution or decision
must be made with a certain time frame. Thus
communication must occur efficiently, within the
time frame required by the setting.
We believe there has not been enough study
of communication among teams of health-care
providers or experts. There have been some exceptions [1—4], and some of these studies point out
that problems in communication can have a direct
impact on the quality of medical care, and can
sometimes lead to medical errors. An important
aspect of this sort of work is the need for in-depth
observational data [1]. This observational approach
is also exemplified by work in computer-supported
collaborative work (CSCW). As we argue elsewhere,
the design of medical informatics systems, especially those for group work, should be informed by
work from CSCW [5].
One of our goals is to fill this gap in knowledge
about medical communication. We have carried
out an in-depth study of a particular sort of
medical collaboration, namely that which occurs
when creating, editing, and reviewing a document. Ultimately, we aim to inform the design
of informatics systems that improve and enhance
communication and workflow around collaborative document authoring. Within the health-care
setting, there are a variety of documents that are
important for patient care, including documents
such as insurance records and billing information,
nursing logs, drug information, and demographic
or administrative documents. The patient record is
J.H. Gennari et al.
the obvious example of a document that must be
created collaboratively by a group of health-care
providers—–no single author creates this document,
and it is a synthesis of knowledge gathered from a
variety of health-care professionals. These collaborators must communicate to create the document,
and the resulting document can be viewed as a
medium for communication about the patient [6].
The collaborative work setting we have studied is
the development of large, multi-center clinical trial
protocols for cancer, as sponsored by the Southwest
Oncology Group (SWOG). These protocol documents
are central to the clinical research process; they
are detailed and comprehensive, and their accuracy, clarity, and coherency have direct influence on
the administration of treatment and on the validity
of the resulting research results. The development
of these documents is one step removed from direct
patient care—–it is at the end of the authoring process, when a protocol is approved and open for
enrollment that the document affects and drives
patient care. However, in chronic diseases without
cures such as cancer and aids, clinical research and
these protocols strongly affect the standard of care
for patients.
For large multi-site clinical trial protocols, the
development of these documents is a lengthy, difficult process involving a number of collaborating
experts and stakeholders. Unfortunately, existing
communication methods do not adequately satisfy
the needs of collaborative health-care professionals. Most medical information systems support
synchronous communication, such as pagers, wireless communication, or conference calls, but these
communication activities are hard to coordinate
[4]. In addition, Parker and Coiera argue that
asynchronous communication methods may be
better than synchronous ones for the exceptionfilled and interrupt driven medical setting. In this
setting, increasing synchronous communication
methods may just exacerbate problems by providing yet another way to interrupt work. Although
email messages are asynchronous, they do not
provide sufficient ‘‘message acknowledgement’’
which can lead to confusion and inefficiencies
for collaborative document generation [4]. In our
work setting, we have documented a number of
problems that demonstrate the inadequacies of
email communication (see Section 3).
In summary, there is evidence that communication problems in health-care settings are
significant, collaborative work around documents
is not well supported by informatics systems, and
that there are insufficient in-depth, observational
studies of real world communication behaviors
in health care. Thus, our primary goal is to carry
Asynchronous communication among clinical researchers
799
Fig. 1 An idealized view of protocol authoring. After the initial idea is approved, the document iterates through many
cycles of review-and-revise. There are a number of different roles and stakeholders that may participate in reviewing
and negotiating.
out an in-depth study of communication. As a secondary subgoal, we believe that an observational
study can fruitfully be combined with iterative,
prototyping design work. In this article, we report
on our study of collaboration and communication
around clinical trial protocol documents. In the
next section, we describe some of the details of
our work setting and collaborative tasks.
2. Clinical trial protocol authoring at
SWOG
Because large, multi-site protocols are complex and
costly to create and administer, the National Cancer Institute (NCI) in the United States has funded a
number of consortia or ‘‘umbrella’’ organizations,
known as cooperative groups in support of research
in adult cancers. These organizations’ goals are to
facilitate and coordinate the development of protocols, as well as to streamline the data collection
and analysis of on-going clinical trials. The Southwest Oncology Group (SWOG) is one of the largest
of these groups, with administrative headquarters
in San Antonio, Texas, and a statistical center in
Seattle, Washington. The SWOG organization helps
author about 30 new protocols each year.1 Each of
these protocols are developed by multidisciplinary
teams over a period of 4—9 months and the resulting documents can be 60—100 pages long. Over the
past 18 months, we have studied protocol authoring
processes both in general, and as carried out within
the SWOG organization.
At a simple level, the protocol writing process
can be described as shown in Fig. 1: (1) submit
an initial proposal or research idea; (2) generate
a first complete draft; (3) iteratively review and
1
SWOG also helps conduct and analyze data from open, ongoing clinical trials. At any point in time, SWOG manages about
120 clinical trials in various stages.
revise the draft; and (4) submit final draft to NCI for
approval. However, the details within the ‘‘review
and revise’’ cycle are quite complex, with a variety of participants. In fact, the SWOG organization
has specified in great detail what the ‘‘official’’
guideline review protocol should be, listing review
meetings, expected participants and an expected
timeline for completion [7]. An important milestone
for protocol development are the ‘‘protocol review
committee’’ (PRC) meetings that occur at the Seattle Statistical Center at least once for each protocol
under development. As we describe in Section 3.1,
these meetings generate many of the comments
and revision advice that lead to a new draft. The
official SWOG guideline for protocol review includes
a diagram (a Pert chart) that specifies the process of protocol authoring as including 27 separate
steps between idea approval and submission. (Thus,
Fig. 1 does not depict SWOG’s view of the process,
but our own, more informal view.)
The expected time between the approval of a
research idea and the submission of a final draft
protocol to NCI is six months. However, in practice,
there are very few deadlines that are enforced,
and the time needed to submit a complete draft
varies tremendously. A high-priority protocol may
manage to get through the review and revise cycle
in four months, while others may take a year or
longer. Protocols can also be approved for drafting, and then reach such significant roadblocks
or problems that they are abandoned, and never
submitted to NCI. The reviewers and participants
in this process can include clinical researchers,
biostatisticians, pharmacists, pharmaceutical company representatives, SWOG administrative personnel, and occasionally others such as patient advocates and oncology nurses. Given the variety and
number of participants, and the diversity of their
backgrounds and expertise, it should not be surprising that communication is a significant part of
the work, and significant source of problems and
delays.
800
3. Research methods
Our study was framed in two parts. First, we
studied the field with some standard qualitative
methods: interviews, observation and document
collection. Our focus here was to understand the
implications for the design of systems that might
improve communication and collaboration. Second, we used some participatory design methods
to design a system prototype. We did not simply
proceed in sequence, but rather, interleaved these
two efforts and methodologies. As we describe
below, our qualitative methods are complementary
with our design methods.
3.1. Qualitative methods: interviews,
observations, and document collection
From our initial knowledge and the official SWOG
process documents, we understood that each protocol is authored primarily by a team consisting
of (a) the primary clinical researcher that initiates
the research idea and plan; (b) the bio-statistician
who oversees protocol study design and statistical analysis issues; and (c) a SWOG staff member
in charge of both task coordination and document
editing. This last team member, known as a protocol coordinator, also applies corporate knowledge
about prior SWOG protocols that may be related to
the document under development. To further complicate the work, the biostatisticians are in Seattle, the protocol coordinators are in San Antonio
Texas, and the primary clinical researcher may be
at any SWOG-affiliated institution across the United
States. With these subjects, and across these locations, we carried out three types of observation:
semi-structured interviews, meeting observation,
and document collection. For all of this work, we
obtained human subject approval (IRB), and our
subjects signed appropriate consent forms.
3.1.1. Interviews
We began interviewing subjects in July 2002.
Initially, we selected subjects at the Seattle biostatistics center via snowball sampling, where an
interview with one participant would lead us to the
next. However, the total pool of SWOG participants
is small: for example, there are only a total of 7
staff protocol coordinators employed by SWOG. In
November of 2002, we traveled to San Antonio and
were able to interview four of these coordinators.
In total, we carried out semi-structured interviews with 11 subjects: four biostatisticians, three
clinical researchers, and four protocol coordinators
(thereby including each of the three roles we ini-
J.H. Gennari et al.
tially identified). We worked from a set of questions
that probed at the subjects’ view of the protocol
authoring process, including both their own activities in the process and those of other collaborators.
Our interviews typically lasted about an hour each,
and some sample questions included:
1. What are the major steps in the writing process?
2. Who is involved in each step and what is their
work?
3. What is the most difficult part of the writing process?
4. How do protocol writers coordinate the group
work?
5. What is the protocol review process like?
We also used two additional interview methods
to further elicit knowledge of how the collaborative writing process occurs. First, we used an actual
protocol document to prompt the user about concrete aspects of the authoring process. Because
most of our subjects were experienced SWOG members, when interviewing a subject, we were able to
find a particular SWOG protocol where the subject
was one of the authors of that protocol. Then, using
that document to prompt the subject, we could ask
very concrete, specific questions about the development of that protocol: which sections were the
subject most familiar with? Which sections in this
protocol were difficult or problematic to create?
What was typical or unusual about this protocol? By
making our questions very concrete, we can query
the subject on a specific example of collaboration,
rather than asking about generalities.
As a second method, we explicitly explored how
our subjects viewed the overall process. To do this,
we asked subjects to draw a diagram or picture of
the protocol authoring process. Participants naturally detailed aspects of the collaboration that
are most salient to them. Then, we used the subject’s specific diagram to frame follow-up questions, probing further about portions of the diagram
that the subject highlighted, or asking about sections that were omitted. For example, for portions
of the process where the participant provided great
detail, we could then ask more about the problems and strengths of that phase of the overall
process.
3.1.2. Protocol review committee meetings
Over a period of about 6 months, we observed
the SWOG protocol review committee (PRC) meetings. At the time of our study, protocols were typically reviewed twice in PRC meetings. Each meeting
covered approximately 2—3 protocols, and meetings occurred approximately weekly. Meetings were
attended primarily by statisticians, although one
Asynchronous communication among clinical researchers
protocol coordinator manager from San Antonio
also attended via conference call. Although the
meetings were not taped, we collected both our
own notes from the meetings, as well as the official
PRC notes that documented the suggested revisions
and discussion points around a particular protocol.
In total, we collected data from about 30 different
protocols.
3.1.3. Email and document collection
All possible members of the PRC were included on
a SWOG mailing list; starting in 2002, we became
members of this mailing list, and thus collected a
large volume of emails. In addition, a work product
resulting from each PRC meeting is a detailed set of
‘‘comments’’ that specify modifications, questions
or discussion points for each protocol. By collecting about five months worth of these documents,
we were able to build a database of over 1200 individual protocol comments. Of course, we also collected a large number of versions of the protocols
themselves (these were essential for understanding
the database of comments).
We collected additional email messages about
the development of individual protocols in two ways
beyond the PRC mailing list. First, we tracked the
development of a single randomly selected protocol. To carry this out, we simply asked the team
assigned to this protocol to ‘‘cc:’’ one of us (CW) on
all correspondence. (We recognize that this method
does not guarantee a truly complete sample of
emails about the protocol.) Second, SWOG leadership would occasionally forward to us email threads
that represented a ‘‘critical incident’’ or memorable or unusual exchange representing a problem
or communication gap.
3.2. Prototyping and participatory design
A weakness of observation alone (e.g., the methods
described in section 3.1) is that although it helps us
understand current work and communication practices, it does little to find out how users think
that work and communication should be improved.
Given that our long-term goal is for systems that
improve collaborative authoring, we are interested
in how users might contribute and react to ideas for
the design of such systems. Thus, we have developed an iterative set of prototype system implementations, and we have used methods from participatory design [8] to refine and improve these
prototypes. Our design is exploratory: at this stage,
we use design as a tool to probe and better understand users’ system requirements.
In traditional software engineering, the
‘‘requirements analysis’’ phase is defined as a
801
step that documents the system requirements as
stated by users prior to system design. However,
this approach often fails to elicit true user needs,
because many of these needs are hard to articulate
and may be implicit, as part of a shared understanding among users. In other words, there may
be a large gap between the set of explicitly stated
requirements, and the set of ‘‘real’’ requirements
that are necessary before a system is acceptable
and usable [9].
To avoid this problem, we used several
approaches. First, these gaps are often reduced
simply by the process of participatory design with
the users. We carried out some early brainstorming sessions with SWOG biostatisticians, and we
were able to take some of our ideas directly to
design, without first eliciting formal requirements.
Next, we developed a series of rapid prototypes and
elicited feedback and additional design ideas from
users. Over time, we expanded the sophistication
of these prototypes, beginning with simple visualizations (with Visio and/or Powerpoint), moving
to more complex implementations, and ultimately
building a web-based, nearly fully-functional prototype [10]. We also gradually grew the number of
users from whom we elicited feedback; beginning
with just the biostatisticians, then protocol coordinators, and finally including a clinical researcher.
By having frequent design iterations, we were better able to discover some of the unspoken system
requirements.
In addition, our on-going observational study of
the work helped us bridge the gap between explicit
requirements and real user needs. By participating in meetings and observing work practices, we
have come to share at least some of the unspoken
culture of the work setting. In addition, the set of
problems and issues we identified during our qualitative study sometimes led directly to design ideas
and systems requirements, rather than a tradition
user requirement elicitation.
4. Results
From a design perspective, our primary result is
that the process at SWOG is enormously complex.
There are a large number of participants, who
interact in complex ways to carry out the work.
Currently available systems for protocol authoring
[11,12] do not take this complexity into account,
and are typically focused on specific aspects of protocol design (e.g., the design-a-trial system focuses
primarily on study design issues [12]). In contrast,
we are interested in general-purpose designs that
have broad applicability. Below, we divide our
802
results into two halves: (1) we describe some results
from our observation of the work at SWOG; and
(2) we present results from our exploratory design
ideas and prototype development. In the former,
we focus on the complexity of the communication process, and the problems and delays that are
therefore introduced in the work. With the latter,
we present our ideas around improved awareness
and a model for facilitating collaborative activities
around the revision comments.
4.1. Observational results
From our study, we found that there are several characteristics of clinical trial protocol writing that affect collaboration and communication
among participants. Because we were concurrently
designing a system for improving the work process,
we necessarily focused on those characteristics of
work that led to problems or perceived problems
in communication. The results we present in this
section are not meant to be general descriptions of
social work behavior based on formal grounded theory; rather, our findings are functional categories
of problems and issues that we found to be useful in generating designs for our prototype systems.
Thus, some of the results we took at ‘‘face-value’’,
without formal methods for data analysis. For others (as noted below), we do have more complete
data and analysis that support our findings. Below
we discuss some significant characteristics of the
work, and then we list a set of perceived problems
that arise from the communication behavior among
SWOG collaborators.
4.1.1. Characteristics of the work and
communication
As we noted earlier, the collaborators that create a SWOG protocol include people with a wide
variety of training, expertise, and backgrounds.
We observed and were explicitly told that there
is a clear hierarchy across these roles. The primary clinical investigator has the highest status,
both because of their training and experience in
clinical research, and because they are ultimately
responsible for the success or failure of the clinical trial. However, the primary work of the clinical
researcher is usually patient care, and thus protocol development may take a secondary role. In
contrast, the primary job of a SWOG protocol coordinator is in administering and authoring protocols.
Thus, the protocol coordinator has more experience and ‘‘corporate knowledge’’ about protocols
in general than does the clinical researcher. Unfortunately, the differences in status make direct communication across these roles difficult. In addition,
J.H. Gennari et al.
the researcher is typically carrying out protocol
authoring work as a secondary task in the evenings
or on weekends, whereas the SWOG administrative
staff work on a more standard work-day schedule (and are often in a different geographic time
zone). For these reasons, communication across
roles is asynchronous, via email, where participants are not expected to work at the same
time.
We also found that users from different roles
fundamentally viewed the process differently. This
result we can directly see in our interview data
and sketches that subjects drew of the overall process [13]. One human characteristic is that many
participants highlighted those parts of the process
where they had a central role, and de-emphasized
other parts of the process and other participants.
For example, some researchers omitted or simplified the revision-and-reviewing cycles and meetings, such as the PRC, where they did not play
a significant role. Similarly, the protocol coordinators had a reduced perception of the roles and
activities of the statisticians. However, we found
that these differing views of the work process
did not by themselves impede protocol authoring
[13].
In part because of the diversity of roles, and
because the work is asynchronous, we characterize the collaborative work of the protocol authoring team as loosely-coupled [14]. By this term, we
means that although team members are collaborating toward a common goal, they do not communicate frequently in a direct manner. Instead, it is
expected that some will pick up the work ‘‘when
they have time’’ and ‘‘as available’’. Because the
work is geographically distributed, and because of
the loosely-coupled nature of the work, SWOG team
members may not know each other very well. (Usually the biostatistician knows the protocol coordinator, but other links are weak.)
With three to six active participants working
asynchronously, document versioning is a significant
problem. For these reasons, SWOG has imposed a
single scriber methodology [15]. After initial drafting of the idea by the clinical researcher, only the
protocol coordinator is allowed to modify the master document. Thus, the coordinator is the only
‘‘scribe’’; all communication for changes must go
through the coordinator, who makes all the changes
and then periodically re-distributes the modified
document. The majority of comments in our collection of 1200 are simple directives from collaborators to the protocol coordinator, asking for a change
of some sort to the document. The requested
change may be very minor, down to the level of fixing typographic errors, but with the single-scriber
Asynchronous communication among clinical researchers
approach, all changes require an email request to
the scribe.
4.1.2. Perceived problems
Together with our participants, we began cataloging a list of communication problems. We
do not have conclusive data that demonstrate
that these problems lead to inefficiencies or
quality deficiencies; thus, we title our results
as ‘‘perceived problems’’. In some cases, these
are communication problems that the design and
observation team identified; in other cases these
are problems identified by some of the SWOG
workers.
Although the single scribe approach avoids many
problems with version control, it introduces a number of other communication problems. First, comments and directives are communicated to the
protocol coordinator via email. This means that
each comment must include sufficient context to
allow for the change. For example, to request
that four specific words be changed, a reviewer
wrote: ‘‘. . .section 2, second paragraph, first sentence, suggest ‘over the use of single agents’
instead of ‘over the sequential use of them’ since
we decided against a crossover.’’ When a protocol coordinator is working on multiple protocols at the same time, and receiving such messages from many collaborators, the additional context information adds a significant cognitive load
to reading and understanding these emails, and
this load can lead to confusion or inefficient
response.
Furthermore, although communication primarily occurs via email, protocol coordinators must
also integrate email messages with other forms
of communication, such as telephone conversations. In addition, some information or important knowledge about a new protocol may come
from the SWOG repository of earlier, but similar protocols. Thus, the coordinator may need to
incorporate or consult with this type of information source as well. Appropriately integrating
knowledge across these various sources and channels is a considerable challenge for the protocol
coordinator.
Because all or most communication goes through
the coordinator, it may be difficult for others to
know the progress or status of the team’s work.
This lack of clarity, together with the asynchronous
nature of the work, can sometime lead to inefficiencies: person A may think that the next step
in the work may need to be done by person B,
while person B may be waiting for a response from
person A. As a more specific example, because of
the lack of knowledge about status, participants
803
may not be aware when they may ask the same
question or point out the same problems as others.
In addition, participants are not immediately
aware if their answers or suggestion conflict with
one another—–instead, these conflicts must all
be noticed and resolved (or re-broadcast) by the
protocol coordinator.
Email does not provide good feedback in a
loosely-coupled collaboration. Participants do not
know if their comments have been ignored,
resolved, or simply not yet incorporated into the
latest version of the document. There is no good
mechanism to notify participants when their questions or comments have been resolved, or to help
them be aware of specific questions that may be
addressed directly to that participant. For these
reasons, participants must generate a volume of
‘‘meta-communication’’—–message that ask about
the status of other messages. The following excerpt
of an email message from a protocol coordinator (person ‘‘B’’) demonstrates a number of these
problems and issues:
Hello A, Just an update, I had sent the draft to X
and Y as you requested. I have heard back from Y
but not X. I don’t know what the situation with X
is, I’m going to go ahead and incorporate Y’s comments and send you a draft back within the next
couple of days. Did Z have any further comments
regarding the draft? Thanks, B.
Because comments are collected by the single
scribe who creates all official versions of the
document, there is no trace of document evolution
and rationale. Occasionally, there is a need to
understand the rationale or justification for a
particular design or wording choice in a protocol.
(E.g., the NCI may ask for support for a particular
choice before granting approval.) With current
methods for communication of revision requests
it can be impossible to determine from whom a
particular suggestion originated, or in response to
what sort of question.
We focused on these particular problems and
communication challenges because they connected
with our on-going design efforts. In particular, these
problems led toward a desiderata for solutions that
would provide communication support in this document authoring environment. As we detail in the
next section, our list includes features such as group
awareness, so that collaborators are more aware of
work carried out by others, in situ communication,
so that collaborators can annotate the document
with in-place comments and messages, and an integrated work environment, so that the work of integrating information across multiple communication
channels is minimized.
804
4.2. System design requirements and
results
As described above, the clinical trial protocol writing, reviewing and revising process is complex and
fraught with communication problems and challenges. To design a system that might help ameliorate these problems, we turned to a concept
that is well-established in the computer-supported
cooperative (CSCW) work literature: that of
awareness:
Awareness is an understanding of the activities of
others. Awareness of individual and group activities is critical to successful collaboration and is
commonly supported in CSCW systems by active
information generation mechanisms separate from
the shared workspace [16].
A number of the communication problems we
observed were related to a lack of awareness,
especially across the different roles in the process.
Because current work practices do not include the
use of any awareness mechanisms, there is a need
for extra communication to establish this crossrole awareness. For example, we observed and
frequently heard of the frustration of statisticians—
–from their perspective they often did not receive
any feedback or follow-up from comments or questions that they had sent to the clinical researchers
or protocol coordinators. When new versions of
the protocol appeared from the coordinator (who
acted as the single scriber), the statisticians did not
know what comments had been incorporated into
the new version. Thus, they needed a mechanism
to better understand (1) the work status of these
other roles; and (2) the rationale behind changes
that appeared in new versions of the document.
As corroborated by Farkas, authors and editors can
develop misunderstandings since authors may not
know why editors have changed the document in a
particular way [17].
As suggested by work in CSCW, awareness should
be supported by automatic mechanisms which
reduce the need for synchronous or asynchronous
messages (e.g., telephone or email) that explicitly
ask about work status. With a large number of participants in the protocol authoring team, who are
working in different places and at different times,
these status queries can add significantly to the
communication burden. In general, we found that
the cognitive overhead for understanding email and
a variety of other communication channels to be
high, which could cause participants to drop or
lose important comments. Similarly, when two participants contradict each other, there may be a
delay before the protocol coordinator notices the
J.H. Gennari et al.
problem. We suggested an exploratory design that
might alleviate these problems by better awareness and more direct and uniform communication
channels.
Our focus on awareness aspects of collaborative
system design and our observation of the work have
led us to the following desiderata for the design of
a collaborative writing system [18]:
• Wide accessibility: Given the number of participants involved in the SWOG protocol authoring
process, and given our observations about differences in work priority, our design must allow for
easy access across platforms and locations.
• Integrated work environment: Participants
should have a single environment where a
variety of information sources can be easily
combined.
• A version control mechanism that includes
change rationale: A system that captures both
version history and the rationale for document
changes, can help group members better understand how the draft changes, and notice when
comments are incorporated rather than ignored
or forgotten.
• In situ communication and decision-making: Integrating communication and decision-making into
the context of writing artifacts can reduce the
cognitive overhead in communication between
collaborative writers. Comments should be connected ‘‘in-place’’ to the context provided by
the evolving document.
• A shared workspace that improves cross-role
awareness: Efficient group work relies on smooth
transitions or interactions between work done
by people of different roles. If information
about individuals’ work status is part of a
shared workspace, then participants will be
more likely to receive timely responses and
feedback to their own efforts, and will need
to expend less effort on communication about
status.
Considering these desiderata, we envision a
system that (a) provides a web-based shared
repository for evolving protocol drafts; and (b)
provides in-context communication among participants about the document design. Based on
prior work in CSCW, and more specifically on
research in document-centered discussion [19—21],
we propose a design that relies on in-context
annotations.
More specifically, for the SWOG work setting, we have designed a prototype system that
meets these desiderata for improved communication. As part of this design, our system includes
a set of features for an ‘‘annotation entity’’
Asynchronous communication among clinical researchers
(in the SWOG context, a comment). [18] These
include:
• Context: An annotation must include its context—
–this enables users to view annotations ‘‘in situ’’.
We envision three types of contextual information. First, context should include the document
context that provides information about the
document, including a name, date and version
information. Second, context should include
the textual context that provides anchoring
information that says where the annotation
is attached to what piece of text within the
document. Finally, context should include a
discussion context which would include pointers
to related annotations including responses. This
information would position an annotation into a
threaded discussion among collaborators.
• Annotation creator and recipient: Annotations
must include information about who created the
annotation and who is expected to receive the
annotation. Besides recording the user name,
one could include role and status information to
help prioritize annotations.
• Annotation time: Annotations should include
not only the creation time, but also some recommendations about when collaborators should
respond to the annotation.
• Status: Annotations are dynamic as the document evolves; thus, annotations should include
life cycle and status information. In collaborative
work, a transition from one state to another
helps provide cross-role awareness. Status should
indicate whether an annotation has been read,
whether changes were made to the document
as a result, or whether there are new responses
to this annotation. Specific milestones in an
annotation life cycle could vary from system
to system depending on concrete tasks. For
our prototype system, we defined four states
for an annotation: ‘‘unread’’, ‘‘responded’’,
‘‘incorporated’’, and ‘‘resolved’’.
We hope that these features will help users share
progress information within the collaborative writing group. Although many of these features are
present in other annotation systems [20,21], our
emphasis is on those features that improve awareness, and reduce the quantity of communication
(email or otherwise) about document status.
For SWOG, and with their help through participatory design, we have designed and built a prototype system, known as PCAT (Protocol Collaborative Authoring Tool) that implements the annotation entity features listed above. The details of this
prototype are described elsewhere [10]. As a prototype, this system has not yet been field-tested nor
805
formally evaluated. However, as a tool in our participatory design process, it has been used and informally evaluated by SWOG staff, including biostatisticians and protocol coordinators. Their comments
and reactions have helped both to understand additional improvement to the design, and helped us
further understand the complexities of their work
and the communication that supports that work.
5. Discussion
Our work combines qualitative, ethnographicallyinspired study methods with participatory design
and iterative prototyping to better develop a complete picture of system requirements. We have
carried out this work among a group of collaborating researchers that author clinical trial protocols. Clinical research is an important activity
in medicine, especially for diseases without cures
such as aids and cancer. The clinical trial protocols
that specify how to carry out a research experiment must be complete and coherent; errors or
ambiguities in these documents are very costly.
Clinical trial protocols are authored by a number
of organizations such as pharmaceutical companies, both within the United States and worldwide.
Although our study looked at a particular cooperative group organization, we believe that many of
our results and characterizations of the work would
hold true across many institutions that carry out
clinical research.
Our study characterized the work carried out by
the SWOG protocol authors in a number of ways
that helped direct our design. An in-depth qualitative study gave us a solid foundation from which
design ideas could be better understood and evaluated. Our extensive study of the work gave us time
to characterize and think about the problems and
bottlenecks in current communication behaviors.
To summarize, we found that:
• Participants have a wide variety of backgrounds
and expertise.
• Different roles (protocol coordinator, statistician, etc.) view the collaborative process differently.
• The work is loosely-coupled. Although there
are sometimes ambiguities and issues that need
group resolution, usually participants do not
expect rapid synchronous feedback from their
collaborators.
• To better manage version control issues, SWOG
has imposed a single-scribe methodology, where
a single participant (the coordinator) controls all
document modifications.
806
These findings may not be particularly surprising to any that have worked closely with clinical trial protocols. Nonetheless, even this level of
description is richer and shows a greater complexity than is typically assumed by tool builders. Our
study also identified a set of communication problems, especially those that related to the revision
comments:
• Revisions comments included lengthy context
information, adding to the cognitive overhead of
understanding and incorporating changes to the
document.
• Combining knowledge and revision advice from a
variety of media can be challenging.
• It was hard or impossible for participants to track
the progress of document evolution. Participants
did not receive feedback about their individual
comments.
• Without a repository of comments and their connections to document versions, evolution rationale is lost.
As should be clear, these findings are directed:
they are focused on the structure of revision comments, and on aspects of the collaborative work
that are readily affected by system design choices.
If we were to carry out a more complete and
thorough grounded-theory study of the work, it
may be that other categories and patterns of work
would come to light. However, as our goal is to
understand requirements for good system design,
we focused on aspects that had to do with email
communication and with the structure of revision
comments.
This focus was natural, given that our qualitative
field study was interleaved in time with our participatory design and iterative prototyping efforts.
In particular, some of our earliest designs focused
on in situ commenting and revising, where the
comments are viewed and placed into the context provided by the document, as well as connected to other comments by threaded links. In
turn, these prototypes led us to study the work
more closely, especially how SWOG participants
collaborated via comments. Ultimately, this study
focus led us to the rich model for annotations
that we incorporated into the design of the PCAT
system.
Thus, one of our important findings is that
our field work was complementary to our design
work. A well-known problem for design is to elicit
appropriate users’ requirements, especially when
these may shift over time. Our rapid prototyping methodology is one way to help manage this
problem. However, we also found that by observing the overall work process, we were better able
J.H. Gennari et al.
to sense which requirements are hard-and-fast,
and which ones are more likely to change over
time.
During our iterative system development we did
experience some episodes of shifting requirements.
For example, our users were initially conservative
about changing work practices, and requested that
our designs simply ‘‘computerize’’ existing work
practices, including the method of a single scriber.
However, after viewing a series of mock-ups and
early prototypes, they became more interested in
changing their work practices, and requested that
our design for the PCAT system include the ability
for users to simultaneously modify different sections of a document. Of course, such a requirements shift can cause significant change to the
design of our prototypes, but it is far less costly
to absorb these changes during requirements analysis and design than it would be during real system
development or deployment.
Finally, we found that in our collaborative work
setting, many communication problems are related
to a lack of cross-role awareness. For example, one
way to reduce the number of email messages is
to automatically inform collaborators about other
team members’ work status. Increased awareness may also reduce frustration about a lack of
response, and improve the efficiency of work that
requires collaboration across different roles. For
these reasons, our PCAT system implements a number of features that improve awareness. Perhaps
the most requested awareness feature is to make
the status of an annotation or comment visible, so
that users can understand when (or if) their annotation has had a response or has led to a change in
the document [10].
Our long-term goal is to validate our methodology by evaluating the PCAT system (or its successor) and showing improved work efficiency and/or
quality of the resulting clinical trial protocol documents. There are significant challenges to be
addressed before carrying out such a evaluative
study. First, we expect our prototype to continue
to evolve, in response to feedback. Second, evaluating groupware is inherently difficult due to the
variety of user types that participate in the collaboration. However, our results here are not about the
particular features of the PCAT system nor its evaluation, but rather about the work-oriented design
process we used to develop that system. By interleaving qualitative, observational study of the work
with iterative design prototyping, we were able to
both better understand the richness of the work
patterns and communication, and to better design
a system that can address user needs and improve
collaboration.
Asynchronous communication among clinical researchers
Acknowledgements
We wish to acknowledge the on-going support and
participation of the SWOG organization, especially
John Crowley, the director of the SWOG Statistical
Center. This work was partially funded by SWOG,
through its grants from the National Cancer Institute. Finally, we thank our reviewers for extensive
comments on an earlier draft.
[11]
[12]
[13]
References
[14]
[1] E. Coiera, V. Tombs, Communication behaviours in a hospital
setting: an observational study, BMJ 316 (1998) 673—676.
[2] G. Makoul, R.H. Curry, P.C. Tang, The use of electronic medical records: communication patterns in outpatient encounters, J. Am. Med. Inform. Assoc. 8 (6) (2001) 610—615.
[3] L.K. McKnight, P.D. Stetson, S. Bakken, C. Curran, J.J.
Cimino, Perceived information needs and communication
difficulties of inpatient physicians and nurses, in: AMIA
Annual Fall Symposium, 2001, pp. 453—457.
[4] J. Parker, E. Coiera, Improving clinical communication: a
view from psychology, J. Am. Med. Inform. Assoc. 7 (5)
(2000) 453—461.
[5] W. Pratt, M.C. Reddy, D.W. McDonald, P. Tarczy-Hornoch,
J.H. Gennari, Incorporating ideas from computer supported
cooperative work, J. Biomed. Inform. 37 (2004) 128—137.
[6] M. Berg, Implementing information systems in health care
organisations: myths and challenges, Int. J. Med. Inform.
64 (2001) 143—156.
[7] SWOG, Southwest Oncology Group’s Protocol Development
Guidelines. Internal report, San Antonio, TX, 2004.
[8] M. Muller, S. Kuhn, Participatory design, Commun. ACM 36
(4) (1993) 24—28.
[9] R.R. Young, The Requirements Engineering Handbook,
Artech House, Norwood, MA, 2004.
[10] C. Weng, D. McDonald, J.H. Gennari, A collaborative clinical trial protocol writing system, in: Proceedings of the
[15]
[16]
[17]
[18]
[19]
[20]
[21]
807
International Conference on Medical Informatics, 2004, pp.
1481—1485.
P. Fazi, D. Luzi, M. Manco, F. Ricci, G. Toffoli, M. Vignetti,
WITH: a system to write clinical trials using XML and
RDBMS., in: Proceedings of the AMIA Symposium, 2002, pp.
240—244.
J.C. Wyatt, D.G. Altman, H.A. Heathfield, CF.A. Pantin,
Development of Design-a-Trial, a knowledge-based critiquing system for authors of clinical trial protocols, Comput. Methods Programs Biomed. 43 (3—4) (1994) 283—
291.
D.W. McDonald, C. Weng, J.H. Gennari, The multiple views
of inter-organizational authoring, in: Proceedings of the
ACM Conference on Computer Supported Cooperative Work
(CSCW’04), 2004, pp. 564—573.
G. Olson, J. Olson, Distance Matters, Hum. Comput. Interact. 15 (2000) 139—178.
I. Posner, R. Baecker, How people write together, in: Proceedings of the Hawaii Conference of System Science, 1992,
pp. 127—138.
P. Dourish, V. Bellotti, Awareness and coordination in shared
workspaces, in: Proceedings of the 1992 ACM conference on
Computer-supported cooperative work, 1992.
D.K.P. Farkas, S.E. online editing, mark-up models, and the
workplace lives of editors and writers, IEEE Trans. Prof.
Commun. 38 (2) (1995) 110—117.
C. Weng, J.H. Gennari, Asynchronous collaborative writing
through annotations, in: Proceedings of the ACM Conference on Computer Supported Cooperative Work (CSCW’04),
2004, pp. 578—581.
E. Churchill, J. Trevor, S. Bly, L. Nelson, StickyChats: remote
conversations over digital documents, in: Proceedings of
the 2000 ACM conference on computer supported cooperative work, 2000.
T. Sumner, S.B. Shum, From documents to discourse: shifting conceptions of scholarly publishing, in: Proceedings of
the SIGCHI conference on Human factors in computing systems, 1998.
A.J.B. Brush, D. Bargeron, A. Gupta, J.J. Cadiz, Robust
annotation positioning in digital documents, in: Proceedings
of the SIGCHI conference on Human factors in computing
systems, 2001.