A Multilevel Theory of Effective System Use:
Insights from the Users of an Electronic Health Record System
Olga Volkoff
Beedie School of Business
Simon Fraser University, Canada
[email protected]
Andrew Burton-Jones
UQ Business School
The University of Queensland, Australia
[email protected]
October 7, 2013
Working paper, University of Queensland
Being revised for resubmission
Abstract:
Outcomes from using information systems can vary widely depending on how effectively they are
used. Nonetheless, past research offers very few insights on what it means to use an information system
effectively. What little research exists has been confined to a single level of analysis (e.g., individual or
group). We provide the first multilevel account, derived from a qualitative case study of the use of an
electronic health record system, coupled with insights from research on effective use and affordances. Our
study confirms the importance of some aspects of effective use highlighted in prior single-level theories,
such as ensuring data accuracy, but it also reveals new aspects needed in a multilevel setting, especially the
role of consistency and the role of reflective practice in achieving the ‘right’ accuracy and consistency.
Our theory contributes by providing the first multilevel account of what effective use looks like in an
organization and by extending research on effective use, affordance actualization, and multilevel theory.
Acknowledgments: The paper has benefited from presentations at National University of Singapore,
Monash University, University of British Columbia, University of Maryland College Park, University of
New South Wales, University of Sydney, Shanghai Jiao Tong University, Fudan University, and the 2012
Workshop of the AIS Special Interest Group on Grounded Theory Methodology. The research was
supported by funds from the Social Sciences and Humanities Research Council of Canada, Sauder School
of Business, UBC, Beedie School of Business, SFU, and the UQ Business School.
A Multilevel Theory of Effective System Use:
Insights from the Users of an Electronic Health Record System
1. Introduction
Organizations invest in information systems to obtain benefits. Although benefits might
sometimes accrue more-or-less automatically from using a system, benefits often accrue only if the
systems are used effectively (Pavlou and El-Sawy 2006). The importance of effective use is widely
recognized. For instance, a practitioner report estimated that “if health care in the US used [IT]…
effectively to drive efficiency and quality, the potential value … could be more than $300 billion … every
year” (Riskin 2012). Likewise, in academia, Agarwal (2011 p. 1) stated that understanding “the effective
and efficient utilization of information technology” was one of the core missions of ISR. Nonetheless,
even though past research offers many insights for how to increase peoples’ acceptance and use of
information systems—arguably the most researched topic in the IS field (Cordoba et al. 2012)—there is
very little research on what the effective use of information systems involves or how to facilitate it.
A recent review of four computer-related fields found only a handful of studies of effective use
(Burton-Jones and Grange 2013 p. 634). Two characteristics of those studies were notable: their exploratory
nature and their focus on single levels of analysis, e.g., individual (Agarwal et al. 2010), group (LeRouge et
al. 2007), or work unit (Pavlou and El-Sawy 2006). Burton-Jones and Grange (2013) addressed the first
issue, proposing a new theory of effective use at the individual level, but the second issue—providing a
multilevel account—remains. A multilevel account is key because what is effective at one level (e.g.,
individual) can be quite ineffective at another (e.g., organization) (DeShon et al. 2004; Goodman 2000).
Because an organization is inherently a multilevel system (Kozlowski and Klein 2000), our
research addresses the question: What constitutes effective use of an organizational information system?
As past research has not yet addressed it, we used grounded theory methods to learn what effective use
actually looks like in the field. We chose a context—the use of a community care electronic health record
(EHR)—for theoretical and practical reasons. From a theoretical perspective, we build on Burton-Jones
and Grange (2013) who developed their theory of effective use based on an assumption that information
2
systems provide representations. Because the core purpose of an electronic health record is to provide
representations (of clients and care practices), this context allowed us to draw on their work. From a
practical perspective, the question of how to use EHR systems effectively is one of the pressing questions
of our time in healthcare administration (Blumenthal and Tavenner 2010). Thus, our findings could be
useful for both research and practice. In terms of underlying assumptions, we take a critical realist view, as
was done in the studies we draw on regarding representations (Burton-Jones and Grange 2013) and
affordances (Volkoff and Strong 2013). This has two main implications. First, rather than view
information systems and their use as being inseparable, we view them as being closely intertwined but
distinct phenomena. Second, we assume that effectiveness is realized through a set of underlying
mechanisms—and the purpose of our work is to understand these mechanisms.
Our paper is structured as follows. In the next section, we situate our work within three emerging
streams of research and highlight the gap in research we seek to address. After that, we describe the
methods we undertook to develop our grounded theory. We then describe the case study site in sufficient
detail to form the basis of our grounded theory. Finally, we present our grounded theory, discuss how it
extends research on effective use, affordance actualization, and multilevel theory, and conclude the paper.
2. Past Research on Effective Use, Multilevel Theory, and Affordance Actualization
We seek to advance emerging research on effective use, multilevel theory, and affordances.
2.1 Effective Use
A recent trend in IS research has been to move from the study of use to the study of effective use.
Whereas ‘use’ refers to engaging with a system, ‘effective use’ refers to engaging with it in a way that
achieves a desired goal, such as improved work performance (Burton-Jones and Grange 2013). The move
from use to effective use can be seen in a small but emerging stream of research offering preliminary
conceptions of the effective use of different types of systems (Agarwal et al. 2010; Boudreau and
Seligman 2005; LeRouge et al. 2007; Pavlou et al. 2008). This trend can also be seen in a growing body
of research on the performance implications of using information systems (Ahearne et al. 2008; Barki et
al. 2007; Burton-Jones and Straub 2006; Ko and Dennis 2011; Sundaram et al. 2007).
3
While various theoretical lenses have been employed in studies of effective use, including framing
(Agarwal et al. 2010), task technology fit (Ahearne et al. 2008), and activity theory (Barki et al. 2007), a
general theory has been elusive to date. Burton-Jones and Grange (2013) proposed a candidate based on
the representation theory of information systems, which assumes that people use information systems to
obtain faithful representations of a domain that can inform action (Wand and Weber 1995; Weber 2003).
For example, representation theory would assume that clinicians use EHR systems to store accurate
information about clients that enables the provision of high quality care. Based on representation theory,
Burton-Jones and Grange (2013) proposed that effective use can be viewed in terms of how well users:
access representations in the system, which they referred to as ‘transparent interaction’; obtain a faithful
representation of the domain from the system, which they referred to as ‘representational fidelity’; and
leverage high quality representations from the system in their work, which they referred to as ‘informed
action.’ While Burton-Jones and Grange (2013) offered conceptual arguments to support their theory, they
provided no empirical support. Strong theory often emerges when theoretical ideas are contrasted with
field-based insights (Smith and Hitt 2005 pp. 572-581). As a result, while we did not wish to test their
theory per se, we wished to examine what effective use actually looks like in the field and compare it to
Burton-Jones’ and Grange’s purely theoretical conceptions.
2.2 Multilevel Theory
Burton-Jones’ and Grange’s theory was limited to the individual level of analysis alone and they
called for researchers to extend it to multiple levels. Much like the emerging stream of research on
effective use, there is also an emerging stream of multilevel research on system use. For example, BurtonJones and Gallivan (2007) proposed principles for conceptualizing system use in a multilevel fashion.
They emphasized how the use of a system by a collective (such as a group or organization) can emerge in a
shared fashion (in which users engage with a system in a similar way) or in a configural fashion (in which
different users engage with a system in a different but coordinated fashion). Recent papers have provided
empirical illustrations of these ideas, such as the enactment of shared use (Leonardi 2012), the enactment of
configural use (Kane and Borgatti 2011), and the process through which configurations emerge (Nan
4
2011). Nonetheless, there have been no multilevel studies of effective use. Our paper seeks to extend
multilevel research in IS by addressing this gap. Doing so is critical because single-level research, no
matter how rigorous, cannot provide a cohesive account of how organizations function (Goodman 2000).
As discussed later, our paper also seeks to extend multilevel research in general, by advancing the way in
which we study organizational ‘levels,’ responding to the recent call of Mathieu and Chen (2011).
2.3 Affordances
Our third theoretical building block is the concept of affordances. Originating in ecological
psychology to explain how an actor perceives objects in the environment (Gibson, 1986), affordances
focus on what the actor might do with the object, rather than on the object’s features. For example, an
actor might view a log as offering the possibility of sitting (a “sitting” affordance) rather than as an object
with features such as a hard surface, a cylindrical shape, and a certain diameter. Affordances were
recently introduced to the IS literature to help explore the relation between users and technical objects
(Markus and Silver, 2008; Zammuto et al., 2007). Defined as the potential for behaviours associated
with achieving an immediate concrete outcome and arising from the relation between an object (e.g., an
IT artifact) and a goal-oriented actor or actors (Volkoff and Strong 2013), affordances share many
elements with our initial understanding of effective use. First is the notion that IT use is purposeful;
while those purposes are not always achieved, actors are goal-oriented (Davern 2007). Second, while in
ecological psychology affordances were viewed at an individual level, in the IS literature they have been
extended to apply to collectives, from teams to organizations (Strong et al., forthcoming). Because an IT
artifact is itself a complex structure, and the actor could be an individual or a collective, the affordances
arising from the relation between them can be at many levels. For instance, at an individual level we can
see affordances such as the potential for recording data arising from the relation between a set of specific
features of the system (such as particular screens and the database) and each user, while at an
organizational level, more complex affordances such as standardizing or coordinating arise from the
relation between the system as a whole and groups of users (Strong et al. forthcoming).
Affordances are potential for action, not the action itself. Actors may or may not perceive an
5
affordance, and even once perceiving it, may or may not actualize it. Actors may actualize affordances
incompletely or poorly, thus never achieving the intended outcome. This has a clear parallel with the
notion of effective use. In building a theory of effective use, one of our contributions is to extend our
understanding of affordance actualization (Strong et al. forthcoming).
3. Method
The grounded theory method (GTM) (Glaser and Strauss 1967) was appropriate for our study
because of the lack of research on effective use and the resulting value in learning about it from the field.
Consistent with GTM principles (Birks et al. 2013), we sought to develop a mid-range theory that could
explain what effective use involved in a specific organization and why.
3.1 Field Site and Data Collection
Our study site was the community care division of a regional health authority in Canada. They
had been investing in an EHR system for a decade and were considering more investments, so they found
our research question very relevant. This system (a community care EHR that we call HITS) was used by
over 5000 users in over 100 sites and covered all community care programs, namely home care and
palliative care, public health, mental health and addictions, residential care, and the interfaces to primary
and acute care. The authority served two sub-regions, a larger one (South) that had been rolling out HITS
over a decade and a smaller one (North) that began rolling out HITS only 5-8 months before we began
our study. We studied the use of HITS in both sub-regions and all major programs (excluding residential
care, which differed markedly from other programs in relying more on external service providers who did
not use HITS). To ensure feasibility, we focused on one site in each sub-region and studied other sites as
needed. The main sites were selected to be representative and comparable in size and scope of services.
Our data collection strategy, which evolved as our understanding of the issues grew, was
designed to enable us to explore differences in use and perceptions of effective use across a variety of
dimensions. These included level of experience (i.e., new users at North, and long-term users at South),
task (i.e., different categories of health care workers including nurses, phyisiotherapists, social workers,
physicians, and clerks), domain (the different community care programs as well as IT staff) and
6
organizational level (front-line staff, clinical leads and educators, middle managers, directors, and C-suite
executives). We modeled our data collection strategy on one of the few prior studies of effective use
(Agarwal et al. 2010), using focus groups, interviews, observations of system use while shadowing front
line workers, and document analysis (see Table 1). We also attended four training sessions to better
understand the issues raised by users, four meetings with the project sponsors, and made five
presentations to staff to discuss findings and receive feedback. In the early stages of the research, our
main focus was to acculturate ourselves in the organization and build rapport with staff. We chose not to
record many of these meetings (16) because doing so would have appeared to be overly formal. Once we
were more firmly implanted in the organization, we began recording our meetings (all but six). In cases
where audio records were not taken, we took detailed notes during and after the meetings.
Table 1: Data Sources
Data source
Focus groups
Interviews
Shadowing
Documents
Other:
Description
25 sessions formally recorded and transcribed*
46 sessions formally recorded and transcribed*
22 sessions not recorded (16 during study initiation and 6 during the study proper)
16 person days, some parts recorded and transcribed*
20 Word/PDF documents, e.g., business case, terms of reference, RFPs, newspaper articles
5 PowerPoint files, e.g., project descriptions, staff workload issues
4 Spreadsheets, e.g., workload modelling, change request logs
>500 Emails: mostly administrative but some directly related to our research question
4 training sessions
4 steering committee meetings in which we received advice and input on the project
5 presentations to staff to conclude the project that allowed for comments and feedback
* In total, our transcriptions amounted to over 1600 single-spaced pages of text.
Table 2 shows our data collection over time. We initiated the study during 2010-2011, conducted
detailed fieldwork during 2011-2012, and returned to the field in 2013 to confirm our understandings of
the issues. In total, through focus groups, interviews, and shadowing, we gathered data from over 150
staff members. While our field study was carried out over several years, this was not primarily a
longitudinal study. Rather the time was spent to ensure we spoke with as many relevant people as
possible, although we did ask them to reflect on how their use evolved over time. One exception was
during shadowing, which we timed to coincide with the roll-out of a new software module (a risk
7
screener). By observing how it was used over the first month we learned how users adjusted over time.
Likewise, by returning to the field a year later, we were able to see changes in attitudes over time.
Table 2: Data Collection over Time
5/2010-2/2011
Study initiation;
study design in
South
3/2011-4/2011
Focus groups at
South; study
design in North.
12 meetings
11 meetings,
plus HITS
training
5/2011-11/2011
12/2011-2/2012
Follow-up interviews
Shadowing and
and focus groups in
final interviews.
South, and focus groups
and interviews in North. 16 person-days
shadowing, plus
32 meetings, plus risk
11 interviews
screener training
3/2012-5/2012
Study wrapup
6/2013
Follow-up,
confirmation
5 final
presentations
to staff
27 interviews
with broad
range of staff
3.2 Data Analysis and Theory Building
Consistent with grounded theory principles (Glaser and Strauss 1967), we coded our data
iteratively, with the aim of developing a mid-range theory. While we followed Strauss and Corbin’s
(1998) guidance on open, axial, and selective coding, we used their coding families as triggers for
theorizing rather than as a strict coding template. Early focus groups were coded by the two authors
independently, each using the NVivo software package. We met regularly to compare our codes and
definitions until we reached a shared understanding. After that, the transcripts for any given meeting
were coded by either one of us present, typically the first author.
Given our awareness of research on effective use, multilevel research, and affordances, we went
into the field with knowledge of these concepts and their potential applicability. But as none of this
research had studied effective use in a multilevel fashion, we remained open-minded as to what concepts
would prove relevant. As a result, our open-codes proved to be a mix of pre-existing theoretical ideas and
new ideas emerging in the field. We ended up with 9 high-level categories (affordances, challenges,
changes, effective use, enablers of effective use, fit, organizational issues, outcomes, and users) and
approximately 150 open codes. We tacked back-and-forth between open and axial coding, identifying
new concepts, relating them to higher-level categories, and recategorizing codes over time. For example,
while the high-level category of ‘effective use’ proved useful throughout our work, different subcategories emerged over time, e.g., the sub-category ‘fixing information’ emerged during the first round
8
of coding while the sub-category ‘uncertainty-in-the-moment’ emerged only a year later when we finally
realized how complex it could be to decide if something was truly effective ‘in the moment’ of use.
As our coding progressed, we wrote memos to document themes emerging from the transcripts
and we held regular coding meetings to discuss them. This allowed us to transition from axial to selective
coding as we focused on the underlying themes and mechanisms that seemed critical to the overall story
of effective use. For instance, we discovered that one category of challenges (‘shared understanding’) hid
an array of more complex phenomena, and we began a process of oncoding, creating 13 new codes for the
problem of achieving consistency-in-use. Likewise, we added three new subcategories of effective use
(conversation, professional obligations, and uncertainty-in-the-moment) as we gradually learned the
importance of the subjective practice-oriented elements of effective use and how they related to the more
objective data-quality oriented aspects of effective use that we saw at the start of the study.
As we proceeded in the selective phase of our analysis, the emerging categories from our data
helped us see new bodies of literature to read. For instance, during our oncoding of ‘shared understanding’
and ‘consistency-in-use’ we began reading papers on standardization and shared understanding (Berente
and Yoo 2012; Hanseth et al. 2006; Oborn et al. 2011) to see how our findings related to that work.
Likewise, we modified our sampling on an ongoing basis to help us test and elaborate our emerging
insights (per Glaser and Strauss 1967). For instance, as we realized the importance of consistency, we
began sampling at different levels of the organization to determine if managers had to use HITS
consistently too, not just front-line users, for the organization to be using it effectively. Naturally, some
theoretical insights emerged only after some time away from the field through mutual reflection and after
presentations of our work to colleagues. Our visit a year later (in 2013), therefore, enabled us to confirm
these emerging insights. Overall, when we were satisfied that our coding reflected our data and that our
emerging insights could contribute to the literature, we ceased making further changes to our coding and
analysis, and we concentrated on articulating our initial theory of effective use (per Urquhart et al. 2010).
4. The Implementation and Use of HITS, and Initial Themes
To provide a basis for understanding our theory, and its potential applicability to other contexts, we
9
briefly review the history of implementing and using HITS at our case site. We also highlight the initial
themes that emerged as we conducted our study, which provided the starting point of our theorizing.
The implementation of HITS was spurred by a combination of threats and opportunities. The
threats came from a “dysfunctional array of systems that are cumbersome, frail, and functionally
inadequate” (Business Case, 11/2001). The opportunities came from an awareness of the growing
importance of community and primary (versus acute) care. As we were told, “it’s the growing part of the
business” (Interview 42, C-Level executive, 12/2011). Several leaders in the organization saw the project
as an opportunity to provide a true platform for integrated community care, something they felt had rarely
been achieved anywhere in the world. Comparing HITS to another community care EHR, a director noted:
[it] was just about public health, and the system [also] needed to support home and
community care, mental health and addictions, and residential care. ‘Cause we – you
know, we believe in integrated care (Interview 33, Director, 11/2011).
The business case for HITS called for an investment of over CAD $8M and a two year
implementation. The rollout took over a decade and its success depended on who we spoke to. Some
said it was “a dream” (Focus group 5, Clinical lead, 4/2011), others “a monster” (Focus group 20, Clinical
lead, 8/2011). The official line was that it was a success. Two summary presentations in 2008 noted that
it had “enabled us to address our health care priorities” and it “realized many of the anticipated benefits
and deliverables.” These presentations touted three main benefits: facilitating care for front-line staff,
facilitating administration for managerial staff, and rationalizing the IT infrastructure for IT staff. Despite
a diversity of views, all stakeholders agreed that the implementation was hard. This was for a variety of
reasons, but mainly because it was such a major change for the organization and resources were so tight.
HITS was used at all levels of the organization, but we could broadly distinguish front-line users
from managerial users. Front-line users included clerks and clinicians who interacted with clients and
who read and recorded details of client encounters in HITS. Managerial users included a wide range of
administrators who could check individual client records (e.g., during audits or appeals) and track trends
from various reports for decision-making. Key figures in these reports were provided to the Provincial
10
and Federal Ministries. A variety of middle-level staff (educators, report writers, clinical specialists, and
clinical coordinators) supported these two groups of users and also used HITS themselves.
When we talked with users at different organizational levels about their views on what constitutes
effective use of HITS, three themes soon emerged. The first theme was simply the difficulty that
participants faced in distinguishing between an effective system and its effective use. Their comments
often melded comments on the system (its qualities), its effective use (how well or poorly people used it),
and consequences (such as client outcomes). A key reason for the difficulty of distinguishing the system
from its use was that data input into HITS became part of the system. As a manager put it: “If you don’t
fill it in properly, then you can’t get the information that you need from it” (Interview 34, 11/2011). This
dynamic was compounded by vicious and virtuous circles that arose between the system and its use. For
instance, if a clinician perceived HITS to be ineffective, this could lead him to use it ineffectively, further
harming the system (a vicious circle). As a clinician admitted (Focus Group 7, 4/2011):
“So the reports aren’t running properly yet. … That in turn leads me to put less effort into
charting my stats accurately. I just kind of throw a number in there and don’t really care.”
And when clinicians perceived HITS to be effective, they felt more motivated to use it effectively,
improving the system (a virtuous circle). As a trainer observed (Interview 11, 5/2011):
“new registration, which is where people can quickly tab through a bunch of fields… are
generally filled in quite completely.”
As noted earlier, we expected a close dynamic between an effective system and its effective use when we
began our work. Even so, we became increasingly aware of its importance during our fieldwork.
Because our research question focused on effective use (not effective systems), this issue increased our
desire to clarify the distinction for participants. We eventually found a helpful analogy, asking clinicians
to imagine visiting a client site with medical tools of varying quality. Using this analogy, participants
were generally able to distinguish between the quality of the tool and the quality of its use because doing
the best one can with the tools available is common in healthcare.
The second theme that emerged during the fieldwork was the wide variety of views on effective use.
Although some participants described effective use as we thought they would (e.g., “being complete,
11
thorough, checking the validity of the information that’s being put in,” Focus Group 30, Clinician,
10/2011), many comments challenged our conceptions of effective use. We summarized these challenges
into six unexpected categories, shown in Table 3. Taken as a whole, these six perspectives suggest that
users do not look at effective use solely as a relationship between themselves and the system (e.g., none of
them refer to mastery of any given feature), but rather as to how system use enables them to do their job
effectively or not. This perspective aligns well with the goal oriented perspective emphasized in recent
work on effective use and affordances (Burton-Jones and Grange 2013; Strong et al. forthcoming).
Table 3: Difficulties of Defining Effective Use
Illustrative quote from transcripts
Our interpretation
P102: How do you make the best use of a bad tool?.... Well I think we really need to think
about that, that’s not something that has an easy answer …
P101: I don’t think there’s a way. If you had a bad spade, a bad shovel, and you had
someone coming ‘round to do the garden and you just said keep using the bad shovel,
eventually you go, “well, I may as well use my hands.” Because it doesn’t matter which way
I hammer it and try and dent it back, I am not going to be able to dig a flower bed with this
(Focus Group 25, Clinicians, 9/2011)
Effective use could
involve not using
the system
I should be doing a case note saying I did see them, but, you know, sometimes things fall off
your desk… you have to let something fall of your desk … so that you can actually have
some contact with the actual human beings that we’re supposed to be working with....
(Focus Group 6, Clinician, 4/2011)
Effective use
involves doing less
than you should
That’s interesting because you’re saying you can pick and choose what works. … But we
can’t because if you don’t fill in all the ticky boxes and all the fields then you can’t save it
and it can’t populate into anything. …No. It’s quite rigid (Focus Group 6, Clinician, 4/2011)
Effective use
involves conforming
to a rigid system
P36: I never … received any formal training or guidance … I chart the very minimum.
P32: We’re probably like polar opposites because I dump [everything] in there
P33:…Everybody does it just a little bit differently, …. I’m never sure if the way I’ve been
taught to do it, or learned to do it, is correct… (Focus Group 7, Clinicians, 4/2011)
Effective use is
undefined,
‘anything goes’
… you could run the two reports and see, are we up…?. And if we’re not, then you think,
“OK, so why are we feeling this way?” So … it’s not gonna give you the answer, but it might
give you some information to take back to the team to start a conversation. Because …
those are only numbers, and then you have the context around the clients, … it’s just a
starting point sometimes (Interview 34, Managers, 11/2011)
Effective use is just
a start; part of a
more holistic
practice
…if I could use it any better, I’m not aware of how that would be… I don’t know if I could use
it better. I’m unsure, but I feel like I’m using it the best way I can (Focus Group 32, Clinician,
11/2011). The opportunity is for us to ...use HITS to describe our populations and the
complexity of our work…I’m not sure what that would look like…. I don’t know (Interview 35,
Director, 01/2012).
Effective use is
uncertain
12
The uncertainty reflected in some of the quotes in Table 3, particularly the last three rows, also
relates to the third theme, namely the importance of professional judgment. As our fieldwork progressed,
we found it increasingly difficult to reconcile the differing perspectives on effective use noted above and
were struck by the lack of consistency in how professionals used HITS, horizontally along the care
pathway, and vertically up organizational levels. With such differing views and inconsistent use, we
wondered how HITS was surviving, let alone achieving success in the eyes of some stakeholders. We
gradually realized that despite all the tensions pulling the organization away from a shared view of HITS
use, professional expectations helped bring the organization back to a common ground. This was evident
in senior leaders’ appreciation for the judgment of front-line workers: “That’s right [they take shortcuts].
Well, and you know, any one of us in the moment, that is the single most important thing. That’s what
we’re here for [the client]” (Interview 29, Director, 9/2011). Likewise, it was evident in front line
clinicians’ appreciation for managers’ judgments: “there are so many sort of nuances, …that it would be
hard to capture that in a ticky box, but I understand the pressure [they face] to analyze the information”
(Focus Group 30, Clinician, 9/2011). It was also evident in the way clinicians worked together to
understand each other’s use of HITS. As one team commented (Focus Group 27, Clinicians, 9/2011):
… as professionals, we meet regularly. … Where is that information going? Are you putting that
information in the case note …or … the plan…? I’ve heard [people ask], “Do your managers give
you time for things,” and my immediate professional response is, “we don’t ask for that time, we
make that time,” because … to provide good care …we are ethically bound to spend time with our
colleagues making sure our education [and] standards are good… so we … come together as
colleagues and say, “OK, … where are you putting this information?”
As we reflected on these themes—the difficulty of distinguishing between an effective system and
its effective use, the variety of views on effective use, and the importance of professional judgment—we
continued to review past literature and found that prior theories did not account for such complex settings.
For instance, Burton-Jones and Grange’s (2013) theory was restricted to the individual-level, so it was illsuited to explaining the multilevel issues we saw. Their theory also focused on objective, representationoriented aspects of effective use, whereas many issues we saw involved subjective, practice-oriented
elements. Thus, we felt that a new mid-range theory of effective use could greatly improve upon that
work. In addition, although we were challenged by the complexity of our data, we gradually found that
13
we understood the situation more fully than many stakeholders in the context. For instance, a senior
Director predicted to us that “the people who are trying to coordinate complex clients… would be some
of [HITS] biggest supporters” (Interview 33, 11/2011). As we had learned in our fieldwork, the reality
was not that simple. Clinicians with complex clients did like aspects of HITS, but they had severe
problems with other parts of it, and these other parts related directly to how ineffectively it was used to
coordinate complex cases. One clinician said flatly: “I don’t even look at other disciplines now” (Focus
Group 7, 4/2011). We therefore felt that a new mid-range theory could contribute to practice too.
5. A Multilevel Theory of Effective Use
Consistent with a grounded theory approach, we present our theory ground-up from the data. We
first describe the types of representations and affordances that HITS offered at each level of analysis.
Next, we discuss the dimensions of effective use that appeared to be related to these representations and
affordances. Finally, we put these pieces together into an initial multilevel theory of effective use.
5.1 Representations and Affordances at Different Levels
Because HITS is an electronic health record system, much of our data naturally concerned the use
of records (i.e., representations, per Burton-Jones and Grange 2013). In fact, our theorizing began with
three basic elements: the representations offered by HITS, the affordances associated with these
representations, and the organizational levels associated with these representations and affordances.
As Figure 1 shows, three types of representation were particularly prominent in our data. First,
HITS stored representations of specific clinician-client interactions. These representations served as the
record of truth, the legal representation of an interaction between a clinician and client at a point in time.
Each interaction was noted in a separate record along with the identity of the user who entered it. Over
time, many records could accumulate, e.g., when shadowing, we often saw clients with over 50 records.
To provide high quality care, clinicians could not rely on any single record. Rather, they needed a
second type of representation—a summary representation—that told them about the client’s history, the
care plan, the current status, and trends over time. It both reflected a set of specific interactions and also
informed the interpretation of any one of them (e.g., when a HITS user studied the most recent case note in
14
the context of the plan and relevant past notes). A limitation of HITS was that it did not provide a suitable
summary record, so users had to compose it themselves through their use of HITS (i.e., by finding and
piecing together the relevant records, which would vary across clients and clinicians over time) and various
workarounds (e.g., talking with other clinicians and keeping shadow paper records). It was widely agreed
that obtaining good summary representations was crucial but difficult to do through HITS:
[Previously] you could hold up one piece of paper, and you could get … an overview of what’s
been happening for that person in the last five to seven days. So you could be in the home
…looking down going, “Wow, things have really changed …,” ‘cause you had all that with you
right in front of your eyes. And it’s not there anymore…. that’s why people are so reluctant to
share clients, because there’s so much information that they’re carrying. Basically, it is on the
computer, but it’s also in their heads (Focus Group 3, Clinicians, 4/2011).
Figure 1: Modes of EHR Use and the Production and Use of Representations
Front-line workers were particularly focused on these first two representations, as they were critical
15
to serving clients. Managers would review them as well when dealing with staff audits or client appeals.
The second representation (the summary) was also a major concern for several clinically-oriented middle
managers (known at our site as clinical leads and clinical specialists), as their roles involved supervising
and improving front-line practices. These individuals had to buffer the competing demands of front-line
staff and management, often forcing them to reach difficult compromises:
Now I’ve had to design another grid area to capture people who are on pumps… And I’m, like, …
“I’m not gonna get buy-in.” … [Staff are] gonna go, “I’m just gonna put it in the case note. ….”
Yet [Management] is saying, “…you can’t capture [it that way].” I know that, but the staff doesn’t
care about that piece of it. …So now… I have no choice (Interview 40, Clinical Specialist, 2/2012).
The third representation in Figure 2 is the aggregate summary. Whereas the first two
representations concerned a single client, the aggregate summary reflected client populations, and was
reflected in various reports. Whereas the summary representation was formed by composing instances
(specific records) into a picture of a whole client, the aggregate summary went further in being formed by
classifying clients and interactions (Lackoff 1987), thereby allowing managers and directors to monitor the
types of phenomena of interest to them (e.g., learning if the number of new mothers experiencing post-natal
depression had decreased since the rollout of a particular initiative). Front-line workers knew little about
such reports; “whatever does go up, we don’t see” (Focus Group 4, Clerks, 4/2011), but they were a key
focus of managers, directors, and a group of report writers and data quality experts known as Planners. As
Figure 1 shows, these summary reports were used to meet a range of administrative responsibilities which
in turn naturally shaped the kinds of reports obtained. Like the summary representation, we show the
aggregate summary in a cloud-like image in Figure 1 because managers could not always obtain what they
needed in a single report; rather several may be needed. Also, while planners wrote many standard reports
for managers, managers could also use HITS to develop their own ad hoc reports. As one Manager told us:
…let’s say … the staff [are] saying they’re getting a lot of palliative referrals, we could go in and
run a HITS report. … and you could compare it to last year at this time, so you could run the two
reports and see, are we up in … referrals? And if we’re not, then you think, “OK, so why are we
feeling this way?” … [Now] I’m pulling data from 2005-on… “how many visits in each fiscal
period did each discipline make,” …“this is how much it’s grown, … I need some more
resources….” I can chart that all out and show that now. … But if I didn’t know how to use
HITS, I wouldn’t know how to do that, because those measures aren’t being pulled by anybody
else but me (Interview 34, Manager, 11/2011).
16
Users related to these different representations in distinctly different ways, giving rise to a range
of affordances. To understand the affordances, we turned to a recent study by Strong et al. (forthcoming)
that developed a mid-range theory of organizational EHR affordance actualization. Although we had not
set out to test their theory, we found that the affordances associated with these representations were
similar to the ones they found, as HITS allowed staff to input representations, access representations,
coordinate work, monitor operations, and make decisions. Table 4 maps these affordances and other key
phenomena in our study to each organizational level. We discuss each row of the table below.
Table 4: Mapping Key Phenomena in Our Study to Organizational Levels
Organizational
Level
Key Phenomena Across Levels
Type of Representation
Relevant Affordances
Associated Goal
Workarounds
Organization:
managers/directors
working in a pooled
interdependent
manner
Aggregate summary
(representation of
client-clinician
interaction for clients
1..n, times 1..n)
- Monitor operations
- Report data
- Make managerial
decisions
None available
To meet citizens’
health requirements
cost effectively.
Strong focus on
efficiency.
Team: a client’s
care team
(‘involved staff’),
supported by
middle-managers.
Summary
(representation of
client-clinician
interaction for client n,
times 1..n)
- Coordinate care
- Make team-level
clinical decision
To provide clients
with continuity of
care. Focus on
balancing quality
and efficiency.
Individual: frontline staff (or a
manager in an
audit/appeal role)
working
individually on a
specific client case.
Summary
(representation of
client-clinician
interaction for client n,
times 1..n)
- Make individual-level
clinical decision
Specific interaction
(representation of
client-clinician
interaction for client n,
time n)
- Input representation
- Access
representation
- Make individual-level
clinical decision
To provide high
quality client care
and meet legal
documentation
standards. Strong
focus on client
care but heedful of
need for
productivity.
Conversation
with clinicians
and clients,
and shadow
paper system
Starting at the bottom of Table 4, at the individual level, front-line staff members were the primary
users of HITS and managers and directors (serving in audit/appeal roles) were occasional users. At this
level, the main representations were the specific interaction and the summary. As Table 4 shows, HITS
17
enabled three affordances at this level: the potential to input representations, access representations, and
make clinical decisions. The goal of inputting representations was to provide a legal record of an encounter
and to guide future steps in care provision. This was the only level at which data was input into HITS; the
representations at other levels were based on these inputs. The goal of accessing representations was to
understand the care provided or required in a given case. As with data input, this was the only level where
specific interactions were accessed; representations at higher levels were formed by aggregating this data
(e.g., by composing specific interactions into a picture of a whole client at the team level and by classifying
client populations into relevant types at the organizational level). The goal of decision-making was to use
information from HITS to facilitate care. The decision-making affordance was enabled both by records of
specific interactions, such as a clinical recommendation in the client’s last case note, and by the summary
representation, which enabled clinicians to make a decision with full knowledge of the client’s context.
As Table 4 shows, we found that the guiding ethos at the individual level was to support client
care. However, HITS was not the only resource available at this level. Rather, staff had access to several
workarounds, such as talking with clinicians and clients and relying on a host of paper records:
In his view (which corroborates what I have observed) experienced nurses who are here regularly
don’t look at that info [in HITS], relying on their understanding of what is probably going on
from the calendar info [paper record] plus a case note or two [printed from HITS] and
conversations with colleagues…. (Shadowing, Notes from meeting Educator, 11/2011).
… a lot of time as I’m doing the wound care …cleaning it, I’m like, “Oh, by the way. I was
looking through your chart. When was your catheter last changed?” “Oh, I don’t have a catheter
anymore.” “Oh, really?!” “Yeah, see, I got this” [points to body] (Shadowing, Clinician, 2/2012).
At the team level, client-care teams were the primary actors. That is, clients interacted with
individuals, and individuals documented each interaction, but care was ultimately provided by a team.
Each team consisted of front-line staff (known as the client’s set of ‘involved staff’) and middle managers
(clinical leads and clinical specialists) who were available to help each team, the latter also acting as a
buffer, balancing the client-care goals of front-line staff with the efficiency goals of management. In Table
4, the individual level describes the work of each individual team member whereas the team level shows
how team members worked together. Because working together inevitably involved understanding
18
multiple aspects of a client case, the relevant representation at the team level was the summary
representation. In our data, we observed two affordances at this level: coordination and decision-making.
In terms of coordination, team members typically had different roles (nurse, physician, therapist,
counselor, clerk) and were involved at different times. Thus, in Goodman’s (2000) terms, the teams
involved both ‘horizontal links’ along the care continuum (i.e., among workers over time) and ‘vertical
links’ across levels (i.e., front-line staff supported by middle managers). Each of these links had to be
coordinated. The affordance to coordinate care was a second-order affordance that emerged when team
members activated the affordances at the individual level to input and access data. That is, because HITS
provided a history of past interactions and a repository for documenting future interactions, it became the
common reference point for a team, and thus a natural resource for coordination. It also offered specific
features, such as a notification (communication) feature, to support coordination among team members.
The second affordance at this level was for decision-making. Specifically, while clinicians
typically made clinical decisions on their own, complex cases could necessitate a meeting, known as a case
conference (Davis and Thurecht 2001), in which team members would come together with the clinical lead
and/or specialist to make a team-level decision. Once again, this was a second-order affordance that
emerged when each team member activated the affordances to input and access data at the individual level
and then brought the relevant data with them to the decision-making table. As Table 4 shows, the goal of
coordinating care and making team-level decisions was the same—to provide continuity of care, i.e., care
experienced as coherent and connected across time and providers (Haggerty et al. 2003). As Table 4 also
shows, when staff availed themselves of this affordance, they had access to the same workarounds that they
had at the individual level, i.e., conversations and a shadow paper system.
Finally, at the organizational level, senior managers and directors were the primary users. In the
community care division of the health authority we studied, most if not all managers and directors used
HITS (to varying degrees), but they did so individually. As one put it, “We are not really using it
collectively” (Interview 59, 6/2013). To be more precise, unlike teams using summary representations
(where team members were reciprocally interdependent), managers using aggregate summaries were only
19
interdependent in a pooled sense (Thompson 1967), i.e., each one reported separately on his/her own area.
At this level, the aggregate summary representations were critical, and three affordances emerged: the
ability to monitor operations (e.g., staff productivity and client wait times), submit reports (e.g., to
organizational leaders and the Government), and make decisions (e.g., to start, stop, or modify initiatives).
The overriding goal in each case was to enable the organization to meet citizens’ health requirements as
efficiently as possible.
The emphasis on efficiency at the organizational level was very evident in managers’ language
(e.g., the word “optimize” appeared 10 times in a transcript from a meeting with Planners) and their
practices (e.g., a manager referred to the popularity of one efficiency-seeking approach by noting “the
proliferation … of more Lean folks”) (Interview 12, 5/2011). Nonetheless, many leaders had clinical
backgrounds and recognized the realities of front-line care. Compared to front-line staff, however, senior
managers had no workarounds. Whereas clinicians could often supplement their use of HITS with paper
records and conversations, managers were highly reliant on HITS:
If we see an anomalous figure or number or anything that seems slightly odd… we get more
concerned. We dig deeper, because …it’s what I use now for management information decisionmaking. …I need these reports to work for me (Interview 29, Director, 9/2011).
To avoid potential confusion, we should note that in traditional multilevel research, researchers
argue that collective-level phenomena only emerge when members of a collective are interdependent
(Burton-Jones and Gallivan 2007; Morgeson and Hofmann 1999). One might question whether the
managers and directors at our site had strong enough interdependencies to constitute an organizational
level. We define them at this level, however, for two reasons. First, there were still pooled
interdependences among managers and directors and this meant that they needed to be aware of how other
managers and directors used the system (e.g., to avoid “turf wars,” Interview 59, Manager, 6/2013). Thus,
some level of interdependence was still present. Second, whereas traditional multilevel research only
considers levels in terms of aggregations of people, we believe that it is also relevant to consider
sociotechnical aggregations (i.e., not just people, but people using data from a system). For instance, the
team level in our study not only reflects a set of reciprocally interdependent individuals, it also reflects a set
20
of individuals using a composition of individual representations. Likewise, the organizational level in our
study not only reflects (pooled) interdependent individuals, it also reflects individuals using representations
that summarized entire classes of clients and interactions. These individuals also had the authority to act on
those representations at that level, making program- and region-wide decisions (i.e., they worked as
‘macro-actors’) (Goodman 2000; Mouzelis 1992). In short, we are extending multilevel principles slightly
to account for the complex reality being studied, as called for by Mathieu and Chen (2011).
5.2 Dimensions of Effective Use
The affordances in Table 4 were similar to those identified by Strong et al. (forthcoming) in their
study of how users actualized EHR-related affordances. Our study extends theirs by studying how effectively
users actualized affordances. During our fieldwork, we discovered three dimensions of effective use:
accuracy, consistency, and reflective practice. Each one reflects a characteristic associated with how users
actualized the affordances in Table 4. Figure 2 illustrates the dimensions, their different aspects, and how
they interrelate.
Effective use
forms part of
forms
part of
Accuracy
Consistency
- Truth
- Whole truth
- Nothing but the
truth
- Utilization
- Place
- Form
- Amount
- Meaning
influences
forms part of
Reflective practice
- Client
- Clinical and coworker workflow
- HITS workflow
influences
influences
Figure 2: Dimensions of Effective Use
Rather than organize this section like the prior one, by level of analysis, we organize it by
dimension of effective use. We do so because with some exceptions (discussed below), we found that each
dimension, and each aspect of each dimension, were relevant at each level, approaching what Kozlowski
and Klein (2000) call a multilevel homologous model (which has identical elements and relationships at
each level). Before discussing the dimensions, we briefly note one aspect of our analysis in case it causes
21
confusion. Specifically, multilevel researchers typically place equal emphasis on variation within and
between higher-level units, e.g., within and between teams, or within and between organizations (Hofmann
2002). However, at our case site, we mainly focused on variation within (not between) higher-level units.
This is because teams at our case site were temporary, fluid, and independent from other teams, and their
relative performance was neither measured nor rewarded. Thus, we focused mainly on mechanisms within
teams that appeared to drive their effectiveness. Likewise, the organization we studied was the sole service
provider in the region, so we could only realistically study variation within it, not between organizations.
With these clarifications in mind, we now discuss each dimension. The first dimension is
accuracy, which “refers to how well information in or derived from the data holding reflects the reality it
was designed to measure” (CIHI 2009 p. 6). Including accuracy as a dimension of effective use is
consistent with Burton-Jones and Grange’s (2013) conception of effective use (i.e., their dimension of
‘representational fidelity’). As Figure 2 shows, we found that the accuracy dimension was a function of
the truth, the whole truth, and nothing but the truth. The first aspect was obvious—clinicians avoided
entering errors of fact into HITS. The second and third aspects were more interesting. At all levels of the
organization, respondents stressed the need to understand the ‘whole picture’ in its ‘context.’ A clinician
stressed that to get the whole picture is “not easy, you have to read an awful lot” (Interview 53, 6/2013)
and a manager noted that without context, HITS data would convey “false information” (Interview 59,
6/2013). Time and again we were told that no EHR could capture the entire picture or context—the
challenge was getting as much as possible while being as clear and succinct as possible. Records should
capture the whole truth but no more. For example, case notes should be “clear…, pithy …to the point
[and] there shouldn’t be any writing that people could misconstrue” (Interview 59, 6/2013).
To further clarify the characteristics of accuracy, we briefly discuss how it relates to various
elements in Table 4. Accuracy was relevant for the effective actualization of four of the affordances,
namely inputting representations, accessing representations, and monitoring and reporting on operations.
At the individual level we saw how clinicians strove to be accurate in all three senses when inputting the
representation of a specific interaction:
22
What I noticed is how carefully she chose her wording when adding comments. For example, in
the question about aggression she wanted to indicate that the man is frustrated ...and can be very
angry, to the point of almost being verbally abusive, but that is mostly with family. She reworded
her one short phrase (trying not to be too verbose) several times until she felt it had just the right
nuance. Observation, Clinician, 11/2011.
They were motivated not only by the need to communicate accurate clinical information to future users of
the record, but also, from a legal perspective, to “cover your ass” (Shadowing, 11/2011). This meant
making sure everything was correctly captured, without going beyond the facts.
When individuals accessed representations of specific interactions, issues of accuracy arose in
several ways. In addition to the flow through effects of inaccurate input, a user might access the wrong
record (e.g., an outdated prescription), or might access the correct record but interpret it incorrectly
through lack of shared meaning between clinicians. Furthermore, even with the correct record and correct
interpretation of accurate input, the accessed representation might be inaccurate because of inadequate
context. This last problem was usually addressed by accessing multiple records to create a summary
representation, but issues of whether a sufficient number of appropriate records had been accessed and
interpreted correctly to create an accurate representation (without wasting time by accessing too many
records) still arose. In understanding accuracy of individual interaction and summary representations,
there are clearly strong interaction effects between how the input and access affordances are actualized:
So it really needs to capture what that visit is for the next person that needs to go in … If
somebody were going in that doesn’t know that client at all, they need to get an understanding of
what did you do to go back to previous notes and see what’s happened. And then what’s driving
the plan now? Interview 40, Clinical specialist, 2/2012.
… some of their counselling team will just write one or two lines … and it’s really quite cryptic,
and you just have no idea what was discussed, whereas I find at this site there’s more robust meat
of, “These are the challenges…, this is the plan.” …for multidisciplinary team working together
[the practice at my other site is] not the ideal. Interview 31, Clinician, 9/2011.
Issues of accuracy of representation did not arise for the summary representations used by teams
when actualizing the coordinating and decision making affordances. Neither of these second-order
affordances required that teams collectively input data to or access it from HITS. Individuals brought
data with them to case conference meetings, and the notifications they sent to other clinicians were
23
similarly based on individual inputs. On the other hand, accuracy was an issue with respect to aggregate
summaries as generated during the actualization by managers of both the monitoring and reporting
affordances. Once again, inaccuracies could occur because individual-level inputs were inaccurate. For
instance, if an incorrect checkbox was marked in an assessment, this would carry through to summary
reports. However, inaccuracies in reports could also occur without inaccuracies in individual records.
For instance, if different clinicians recorded data in different places or in different (but seemingly
accurate) ways arising from different interpretations of the classification scheme, the reports would no
longer represent the client population:
… when it comes to an area that I can see that when you look at the numbers and you just
compare it to the work that they’re doing, it just doesn’t match up, is child and youth. …it looks
like they’ve only seen two or three people … because they’re not putting the information in the
right places. Focus group 10, Manager, 5/2011
This type of inaccuracy is also related to the second dimension of effective use: consistency.
Consistency refers to variation among instances of a given type (CIHI 2009 p. 41). Thus, in our case,
consistency refers to variation among how staff members with the same role (e.g., nurses in elderly care,
or managers of public health) use HITS. In our data, we found five major types of consistency:
Consistency of utilization: whether staff all used a given feature of HITS, e.g.:
“In HITS there’s a form in the center index module, where you can collect a person’s
ethnicity…some programs do it, some don’t. Some staff do it, some don’t. … So you really
need the team leads to inform the staff, “we need this information to be captured so that we
can actually report it.” Focus group 8, Planners, 5/2011.
Consistency of place: whether staff all input a specific data item in the same place in HITS, e.g.:
…some nurses are very diligent … putting ...the information where it’s supposed to go…
Other nurses… don’t get the information where it is [supposed to go], or it can be in two
places or even three places, and they’re still phoning, leaving messages for the other nurse,
and leaving sticky notes….” Interview 40, Clinical specialist, 2/2012.
Consistency of form: whether staff all input their records in the same format/structure, e.g.:
“…different people chart differently, so … you have to read the whole case note to get what
you’re looking for … so you’re flipping through just to see if they happen to mention it,
‘cause it’s not highlighted.” Focus group 1, Clinical leads, 3/2011.
Consistency of amount: whether staff all input in their records to the same detail in HITS, e.g.:
24
“I know there’s tons of variance out there in …what they document and how much they
document. And in some cases, I think some nurses are over-documenting and in some cases
they are under-documenting.” Interview 38, Clinical specialist, 2/2012.
Consistency of meaning: whether staff all have the same interpretation of what HITS data and
fields refer to, and why they are there:
“because that form is accessible to multiple people from multiple programs, one will put it
in, someone else will come along, will change it to something different. Because their
interpretation could be different. Focus group 8, Planners, 5/2011
Consistency was relevant for all organizational levels and types of representation, but the saliency
of specific dimensions differed across the various affordances being actualized. Consistency of utilization
was relevant for every affordance (and hence for every organizational level and type of representation),
and reflected whether actors (be they clinicians, teams, or managers) actualized the affordance at all by
using the system features associated with it. For example, some team members used the notification
feature, associated with the coordination affordance, while others did not:
[HITS] only sort of reaches its potential if kind of everybody’s using it in that way. Like, for
example, in terms of communication, I might find that there’s a number of people affiliated with
my patient. … I might do my note, and then I might click to notify all those people, but then
months later, I might find out that that physician doesn’t even log on and use HITS there. So I’m
thinking that my progress notes have all been going to this person’s GP….And then I run into him
and he’s like, “Oh, what notes?” (Focus group 23, Clinicians, 8/2011)
Consistency of both form and amount, on the other hand, while mentioned frequently by our respondents,
only arose during the input of unstructured data, such as case notes, which only occurred when
individuals actualized the input affordance for the representation of a specific interaction. When case
notes are entered in inconsistent forms, anyone subsequently accessing an individual interaction
representation has a harder job interpreting what they see. The problem is exacerbated when a user is
using summary representations to make decisions, as it is harder to compare one record to others in the
set. Consistency of place was relevant for individuals actualizing either the input or the access
affordances, and applied to both structured and unstructured data. Low levels of consistency of place
meant it took longer for clinicians to obtain the data they needed to provide good care, and in fact they
25
might miss some of what they needed. As one clinician stressed:
“There are different places to put it, and where you look … most often isn’t necessarily where
everybody else would be looking… the information is there, but can you find it? … I don’t think
we’ve … come to consensus of where to put all the information …. [And] until everybody figures
out where their parts are, and where they should go as far as putting the information in, it’s,
“might as well just throw everything into a pot of stew, and then look for your carrot” (Focus
group 27, 9/2011).
Consistency of meaning, while applicable wherever data was input or accessed and used, was a major
concern for structured data, whether at the individual or the organizational level. Inconsistent
interpretations of tick boxes across different front-line staff could make the aggregate summary
representations, obtained and used by managers in actualizing the monitoring, reporting and decisionmaking affordances, meaningless.
While all five types of consistency were important, the goal was not complete consistency. After
all, every client was unique, and the associated representations needed to capture a lot of variety. As a
Clinical Specialist explained:
“…if we wanted everyone to document exactly the same way, then our college would say,
“Here’s your template. Fill it in.” But we don’t have that. We have, you know, “Here are
recommended guidelines…,” …There’s a lot of play in there” (Interview 38, 2/2012).
A certain level of consistency was essential, however, because it enabled clinicians to quickly
obtain the data they needed to provide good care. Problems with consistency were particularly acute for
clinicians with complex clients. HITS was supposed to support such clinicians, but the inconsistent ways
in which interactions were recorded in HITS (inconsistencies in form, amount, and meaning) were
exacerbated in such multidisciplinary cases:
“I rarely go and read Elderly Care… I do not understand them…a lot of it is in acronyms. And a
lot of it is very short. They probably understand all of it … [but] in the Mental Health profession,
I don’t know … People talk about continuity of care [but you’re] getting the other side, saying
“Well, I don’t want to look there because [I don’t understand it]” (Interview 39, Manager,
2/2012).
As Figure 2 shows, consistency influences accuracy. For instance, a clinician told us, “we [can]
26
gather great information as long as it’s being put in there … consistently … Otherwise, you’re not getting
true numbers” (Focus group 22, 8/2011). The influence of consistency on accuracy applies equally to
unstructured and structured data. With unstructured data (such as case notes on specific interactions),
inconsistent amounts and formats made it hard to find the right data and derive an accurate understanding.
With structured data (such as reports), aggregations of inconsistent inputs could be meaningless.
The third dimension of effective use is reflective practice. Whereas accuracy and consistency
were invivo codes (i.e., words used by our participants), we used “reflective practice” to reflect a range of
behaviors that we had initially coded using invivo codes such as “clinical lens” and “professional
obligations.” These behaviors were similar to notions in the literature such as mindfulness (Trudel et al.
2012; Weick et al. 1999) or heedful interrelating (Lanham et al. 2011; Weick and Roberts 1993), but also
different, e.g., they sometimes lacked the innovation associated with mindfulness or the sociality of
heedful interrelating. The common thread to the behaviors we coded as “reflective practice” was the
notion of a practice-based rationale driving the behavior, as in studies of practical rationality (Sandberg
and Tsoukas 2011), reflective conformity (Elmes et al. 2005), and reflection-in-action (Levina 2005).
Coined by Schon (1983), the notion of reflective practice has long been embraced by clinicians
(Jarvis 1992), but often simplistically (Boud 2010) and there “remains considerable scope for developing a
more sophisticated understanding” of it (Thompson and Pascal 2012 p. 311). Schon (1983) suggested that
reflective practice involves reflection-on-action (appraising past acts) and reflection-in-action (appraising
acts in production). Others added reflection-for-action (appraising future acts) (Wilson 2008). In each
case, one can consider a continuum running from infrequent, intensive reflection, to frequent, effortless
reflection. The former is triggered by rare, puzzling events (Mann et al. 2009 p. 610) and involves “taking
time to step back and to ponder the meaning of what has happened, the impact …and the direction one is
taking” (Higgins 2011 p. 584). The latter is a more every-day, tacit practice, as Thompson and Pascal
(2012 p. 316) articulated: “professional knowledge … has become so ingrained in most cases that when we
reflect in action, we are often drawing on that knowledge…without even realising [it].”
In our fieldwork, we rarely saw the first type of reflective practice, as staff rarely had time to just
27
ponder, but we often saw examples of the second type. As in past studies of reflective practice (Wilson
2008), we observed staff reflecting on the past (e.g., the history of past interactions), present (e.g., the
client’s current situation), and future (e.g., the goals for client care). However, in our review of past studies
on reflective practice, we found little in the way of guidance regarding potential objects of reflection. In
our study, we found three objects of reflection to be salient at all organizational levels:
-
Clients: Clinicians reflected on the client they served and managers reflected on the client
populations they served. For example:
[Using HITS effectively is] thinking clinically and saying, “OK, what’s, what’s important,”
… you want to communicate the important things [about the client]. Focus group 26,
Administrative users, 9/2011.
If we don't have everything on the table and look at the data it creates turf wars..., but the
hard part is that HITS is not going to give us the qualitative differences that exist between
[client] populations at different sites ... [so we] tried to come up with an agreement
collectively [to account for that] .... Interview 59, Manager, 6/2013.
-
Clinical and co-worker workflow: Clinicians reflected on their own clinical guidelines and the
practices of the other clinicians involved in the care process. Likewise, when managers obtained
and interpreted data, they reflected on the clinical workflow their staff engaged in. For example:
I choose when I should elaborate... based on the scenario. ....I put information that I know will
be relevant for that person [taking over care]. I know to do that from experience, from
feedback [from the person taking over care] and because I've worked on that side of healthcare
and I know that information would be relevant for me.... Interview 51, Clinician, 6/2013.
... the biggest hazard is when the [report] request comes from...some [manager] who doesn’t
really ... understand their own clinical process ... they know the way it should be happening,
as opposed to the way it is happening. ... it looks like a wish-list … no real link back to how
things are done, and, that can take a while to sort out. Focus group 8, Planners, 5/2011.
-
HITS workflow: Clinicians and managers both reflected on HITS data and functionality when
interacting with the system or using data from it. For example:
…a good use of HITS is using the proper …grids, making sure that if I’m going to put [data]
in …, it may take longer to initially put it in, but it saves time in the long-run because then it
carries over to all the screens [and] reports. Focus group 26, Administrative users, 9/2011.
…you have to be clever because we have to think in terms of computer logic. ...we really
have to think of what can the system pick up and what can it not, yeah. It’s tricky trying to
do some of these reports. So trying to have the outcome in mind. What is it you want to
28
look at and then looking at the system. Focus group 30, Manager, 9/2011.
Reflection often involves questioning what, how, and why (Thorpe 2004 p. 300). Accordingly, we
found that staff reflected on what to enter and obtain from HITS, how to enter and obtain it, and why. The
what dimension was closely associated with accuracy—staff focused on entering and obtaining an accurate
picture of the client/clients (hence the link from reflective practice to accuracy in Figure 2). The how
dimension was closely associated with consistency—staff focused on entering and retrieving data in a way
that would be consistent with other clinicians’ expectations (hence the link from reflective practice to
consistency in Figure 2). Finally, the why dimension provided an over-arching practice-based rationale for
the activity that not only encompassed the what and how elements, but went beyond them to consider other
relevant factors in that context (hence the direct link from reflective practice to effective use in Figure 3).
For instance, although what and how issues were particularly salient in our data, occasionally other issues
surfaced such as who best to send a notification to. A clinician engaging in reflective practice would reflect
on the best person to send it to in that context. This is just one of several such idiosyncratic scenarios we
observed and in each case, the same process of reflective practice helped facilitate the user’s response.
One point to note about reflective practice is that in the IS literature (especially quantitative studies),
IS use is traditionally viewed as a mere behavior. That is, while researchers may consider cognitive or
emotive antecedents to use, they exclude such aspects when they conceptualize use itself (Venkatesh et al.
2003). Only recently have researchers begun to consider a user’s mindset to be part of use. For instance,
Burton-Jones and Straub (2006) explained how one factor typically viewed as an antecedent to use
(cognitive absorption) could also be considered a dimension of use. Likewise, while reflective practice
could be viewed as a driver of effective use, we stress its role as a dimension of it. In fact, as a Director
explained, the design of HITS was, in fact, predicated on users’ displaying such a mindset:
Moderator: …the big complaint that comes up all the time …is, “Why do I need to see all this
stuff?” … But I’ve also seen transcripts where they say, “That’s the whole point – it’s
community care. It’s understanding the broader picture.” So … how does a clinician decide…
where to look … what’s good to look at?
P121: But I think why they’re looking [audible emphasis on why] would be my other question
back to you. … Not everyone is following up on the global picture of the person; they need all
the detail. I think both those comments from those transcripts are relevant. … If you’re focused
29
on the pieces of care that you need some information about, you do not need the whole picture,
which may involve 20 problem list areas for person. If you’re doing a follow-up assessment and
you’re trying to understand what’s been going on with the person, how they’ve changed, you’re
gonna have a more global scope …. there isn’t a one-size-fits-all, and I think the tool does give
you the ability to expand and contract that and add as much as you need to. Interview 35, 1/2012.
We found that the different dimensions of reflective practice were salient for actualizion of any of
the affordances at all levels of the organization, and for all three types of representation. For example,
reflective practice was evident in the way that front-line staff looked for and entered data about specific
interactions. Users displaying high levels of reflective practice went into the system with a filter or lens of
what was important, relevant and, most critically, why.
… what’s going on clinically for the person … tells that professional, “I think I need to put a bit
more information in this case note … so the next person looking at this understands where I was
going” Interview 35, Director, 1/2012.
For individuals and teams accessing and using summary representations, reflective practice was
similar to reflective practice at the level of a specific interaction, but required even more attention to the
set of records required and the importance of any one record in the larger set. Effective users referenced
assessments to be sure that other clinicians would see them, signaled notes that were especially crucial,
thought about which body of records were most important in light of that client’s specific priorities, and
sought the most efficient way to communicate information to other team members. As an example of the
latter, one manager explained:
… some of the staff are particularly good at sending me notifications …So they could leave a
message on my voice-mail, they could try to grab me in the hallway, but they also have another
method … where they can notify me through the system, and then it gives me a prompt to look
into the case notes, get an update quickly. So it’s very helpful … [Compared to other forms of
communication,] if the message is attached to the client record [as it is in the notification feature],
it’s a little more likely to get to a source that’s seeing the client next (Focus Group 1, 3/2011).
Finally, when creating or using aggregate summaries, senior managers reflected on the clinical
practices being undertaken by front-line staff and thought about the best way of obtaining data on them
within the constraints set by HITS. In fact, because of the importance of HITS in the health authority, this
mindset—combining a reflection on both HITS and the clinical practice—was a key differentiator of more
30
successful leaders: “leaders who are comfortable with HITS, and comfortable as clinical leaders…are
rising to the top in today’s organization” (Interview 11, Educator/Trainer, 5/2011).
Overall, compared to the comments we received about accuracy and consistency, the comments
we received about reflective practice tended to be more emotive in nature, more tied to participants’ very
identity as professionals. A good example of this occurred in our shadowing:
We moved to a little meeting room around the corner and I explained that I was interested in
exploring how a nurse decides what to record where…and when things might get recorded in
more than one place. She really responded to that – seemed important issues to her. Started by
talking about “nursing judgement” – things aren’t cut and dried, people aren’t cookie cutter
replicas, so judgement has to be used, and so it is important to express stuff very carefully
(importance of exact language). Furthermore, since records are accessible, they need to be
worded in language that is respectful, with the hope that a fellow professional could read between
the lines as required. Shadowing, Clinician, 12/2011.
This quote also highlights one final point regarding reflective practice. In Figure 2, we show that
reflective practice influences accuracy and consistency, but our data does not suggest that this influence
implies a continual increase, ad infinitum. In fact, perfectly accurate and consistent records may neither
be possible nor desirable because care processes are inherently ambiguous and uncertain and there is a
need to balance documentation needs with seeing clients. Thus, reflective practice acted more like a
valve, increasing accuracy and consistency to a level judged to be appropriate by the professionals in that
setting. In other words, even though accuracy and consistency can be judged objectively, their desired
levels were determined subjectively, and this determination was itself a product of reflective practice.
5.3 Putting the Pieces Together: A Mid-Range Theory of Effective Use
Finally we bring together the elements discussed above into a multi-level mid-range theory of
effective use of an EHR. As explained in prior sections, understanding effective use requires us to
distinguish not only between different organizational levels, but also between different representations
and affordances. Building on that structural foundation, effective use is a function of the degree of
accuracy and consistency achieved and reflective practice engaged in during the actualization of each
affordance at each level. Thus our model is constructed as a chain of eight similarly structured and interrelated components. Each component represents the actualization of a specific affordance with its related
31
Organizational level (each numbered box reflects an individual using HITS for a managerial purpose covering many staff and clients)
enables
6
Affordance
actualization
7
RDO
Affordance
actualization
enables
8
Affordance
actualization
RS
RA
MA
effective use
Affordance
A
Cu,p,m
RP
effective use
Affordance
Cost
A
RP
Cu,m
Outcome
Team level (the
numbered box
reflects a team
using HITS for an
individual client)
4
Affordance
actualization
Affordance
5
TAD
effective use
Cu
RP
Affordance
A
Cu,p,f,a,m
Cu
RP
2*
Affordance
actualization
RDO
effective use
Affordance
A
RP
Outcome
Cu,p,m
Cost
Outcome
can
inform
composition
enables
enables
Cost
COC
effective use
Cost
3*
enables
AA
Affordance
Affordance
actualization
enables
Outcome
IA
Cost
CA
classification
enables
CIR
RP
Outcome
effective use
Affordance
actualization
effective use
Cu,m
TDMA
Affordance
1
Cost
Outcome
feedback
MDM
MDMA
Cost
RP
Affordance
actualization
CAD
CDMA
Affordance
effective use
Cu,m
Outcome
Cost
RP
Outcome
feedback
Individual level (each numbered box reflects an individual using HITS to record, understand, and/or guide a specific client interaction)
Key:
Affordances & outcomes: 1. IA: Input affordance; CIR: Client interaction represented; 2. AA: Access affordance; RDO: Relevant data obtained; 3. CDMA: Clinical
decision-making affordance; CAD: Care activity determined; 4. TDMA: Team decision-making affordance; TAD: Team actions determined; 5. CA: Coordinating
affordance; COC: Continuity of care; 6. MA: Monitoring affordance; RDO: Relevant data obtained; 7. RA: Reporting affordance; RS: Report submitted; 8. MDMA:
Managerial decision-making affordance; MDM: Managerial decision made. * Note: Affordances in Boxes 2 and 3 are typically actualized by front-line staff, but
can also be actualized by managers as part of a chart audit or client appeal.
Effective use (effective affordance actualization): A: Accuracy; C: Consistency; Cu: Consistency of utilization; Cp: Consistency of place; Cf: Consistency of form;
Ca: Consistency of amount; Cm: Consistency of meaning; RP: Reflective practice.
Figure 3: Multilevel Model of Effective Use
immediate concrete outcomes, and identifies the salient dimensions of actualization for that affordance.
Understanding effective use entails not only identifying each affordance, with its associated outcomes and
critical dimensions, but also understanding the interactions among them, within and across levels. The
explanations for the individual elements within each component are embedded in the prior sections of the
paper, so for brevity’s sake we focus the discussion here on the main interactions between components.
Figure 3 illustrates the multilevel model that emerged from our data and subsequent theorizing and
presents the mechanisms through which effectiveness is achieved.
We start with three overarching observations. First, we included “cost” as part of the outcome of
all actualization activities. We did not explore cost in any depth, but include it in our model to highlight
the fact that actualization has a specific concrete outcome as its objective, but it can also generate undesired
(though not necessarily unexpected) outcomes. At the very least, actualization takes time, thereby
incurring an opportunity cost. Certainly, clinicians certainly told us about their struggles trading off the
time they spent with clients and the time they spent interacting with HITS (e.g., as shown much earlier in
row 2 of Table 3). Second, as discussed earlier, the various aspects of accuracy and reflective practice
appeared to have similar impacts on effective use across affordances and levels, whereas the different
aspects of consistency had differing effects. Thus, we only show the different dimensions of consistency in
Figure 3; we omit the different dimensions of accuracy and reflective practice from the figure to reduce
visual complexity. Third, there are aspects of temporality embedded in the model, with the outcomes of
actualization of some affordances not only enabling the actualization of other affordances (as in Strong et
al. forthcoming), but determining how effective that actualization can be. It is to this issue that we now
turn.
Effective use of an EHR begins with effective actualization of the input affordance. Thus, in
many ways, Box 1 in Figure 3 is the most critical. If inputs are inaccurate or inconsistent, front-line staff
cannot retrieve accurate representations (Box 2), nor can accurate reports be generated by management
(Box 6); each of these consequences themselves have downstream effects as well. Because, in general,
only front-line clinicians actualize this affordance, effective use by everyone depends on the effectiveness
of front-line workers, who may not always be aware of the downstream impact of their actions. Thus,
improving effective use entails not only ensuring that they understand how the various dimensions of
accuracy, consistency and reflective practice affect the record, but also how the quality of that record
affects everything else the system is intended to support, and what “quality” means to different people.
Although effective actualization of this fairly basic input affordance is necessary, it is not
sufficient for effective actualization of other affordances. For example, even if an accurate representation
of every interaction is recorded, the representation as retrieved may be inaccurate because of inconsistent
practices by staff. Thus, consistency has a parallel flow-through effect. Similarly the effective
actualization of the access affordance enables effective actualization of the coordinating affordance and
the decision making affordances at the individual and team levels. Indeed, if users fail to accurately and
consistently access records, effective coordination cannot occur, nor can effective clinical decisions be
made. The same logic applies to the other arrows indicating that the outcomes of the monitoring
affordance affect the effectiveness of the reporting and managerial decision-making affordances.
Reflective practice is the main feedback mechanism in the model. As Figure 3 shows, it not only
provides feedback in-the-moment of use (i.e., within each component), but it also provides a way for users
to learn from the consequences of their actions, e.g., updating their views on how to input data after
reflecting on data they access (the link from Box 2 to Box 1). Likewise, it provides a way for managers
who, while reflecting on their own practice, become aware of practice changes needed at the frontline (the
link from Box 6 to Box 1). This latter feedback was particularly salient because, as noted earlier in Table
4, managers relied heavily on HITS. Thus, when they retrieved and analyzed reports (actualizing the
affordance to monitor operations), if they identified problems with front-line inputs, they were triggered to
communicate with front-line staff to influence the staff’s reflective practices over inputting data. As the
Planners emphasized, “And so it’s a loop, there are a lot of loops for improvements” (Focus group 28,
9/2011). As noted before, however, while clinicians adjusted their reflective practices in response to such
influence, they did not respond in a machinelike manner. Rather, reflective practices acted like a valve;
clinicians changed their practices only to the extent they deemed professionally appropriate in that context.
34
Although the eight components (boxes) in our model each have a similar structure, the aspects of
effective use vary from box-to-box, in line with the nature of each affordance. For instance, we only show
accuracy as a dimension of effective use for affordances involving inputting or accessing data. This is
because while having accurate records facilitates coordination (Box 5) and decision making (Boxes 3, 4,
and 7), these effects flow from the preceding affordances (Boxes 1, 2, and 6). Given a particular level of
accuracy in the records, actualizing the affordances to coordinate care and make decisions depended more
on what people did with the information. Thus, consistency of utilization and reflective practice remained
relevant in such cases. A similar logic applies to the different dimensions of consistency in each box. For
instance, consistency of form and amount related mostly to data input, so they are shown only in Box 1.
Consistency of utilization and meaning, on the other hand, applied to most components of the model.
Finally, we note the different ways in which effective use emerges across levels. As Figure 3
shows, two processes were instrumental in this process: classification and composition. The link from
Box 1 to Box 6 involved classification, i.e., the identification and differentiation among types of things
according to the way that stakeholders perceive them (Lackoff 1987). For instance, managers and
directors were largely focused on particular types of interactions and clients, e.g., relating to particular
services, in particular regions, over particular periods. In fact, much managerial acumen was required to
classify client populations appropriately for a given decision, select the appropriate HITS report, and
interpret the data correctly. On the other hand, teams (the middle level of Figure 3) were focused on
providing integrated care for a patient. They were focused less on the instances (the specific records in
the EHR) and more on the composition of instances, the whole rather than the part (Hadar and Soffer
2006), mirroring their goal to care for the “whole” patient (Wilkin and Slevin 2004 p. 54). The potential
to use the EHR for team decision-making and coordination depended on team members assembling all the
relevant pieces of the client’s picture from the EHR; only then could they take part in meaningful case
conference meetings (Box 4) or know what to communicate when (Box 5). The challenge for effective
use is that two quite different types of data are required: data that is highly structured into categories for
classification, and data that is highly nuanced and unstructured for composition. Front-line clinicians,
35
who generally prefer to work with the latter as they treat individual clients, each one with unique needs,
must nonethless collect and input both types, whether they use the former or not.
Overall, this model of effective use reflects our data on the use of a particular system (an EHR) in a
specific context (community care). Although we believe the model should apply to other similar contexts,
its generalizability to other contexts is inherently limited. More generalizable is the template or approach
for theorizing. That is, while the specific representations, affordances, and levels will naturally differ
across contexts, we suggest that researchers can still account for effective use in terms of the elements we
have outlined, i.e., that (a) effective use can best be thought of as the effective actualization of affordances,
(b) complex systems will typically result in a chain of interrelated related affordances across organizational
levels, (c) in many organizational systems (especially data-intensive systems, such as EHRs), accuracy,
consistency, and reflective practice will be key dimensions of effective use, and (d) the relevance of each
dimension of effective use (and each aspect of each dimension) will depend on the specific affordances in
that context.
Before discussing the implications of our work, we briefly return to the three themes raised in
our initial fieldwork to show how our model accounts for them. The first theme was simply the difficulty
of distinguishing between an effective system and its effective use. This difficulty is understandable
when viewed through the lens of our model because distinguishing between these concepts requires one
to understand: the difference between a system and its affordances, the difference between an affordance
and affordance actualization, and the relationship between outcomes from affordance actualization and
affordances (both over time and across levels). It is no wonder that our respondents initially struggled
with our question. It also makes sense that when we found an analogy that hit upon a similar theme
(providing care with tools of varying quality), our respondents were able to draw upon their everyday
experiences of this analogous situation and articulate the complex and interconnected issues involved.
The second theme was the wide variety of views on effective use. Again, this makes sense when
we see the eight affordance-actualization-outcome components in our model. No user was involved in
the whole chain. Rather, depending on one’s role (e.g., clinicians, clinical leads and specialists, or
36
managers and directors), and the particular context of one’s work (e.g., putting information in, getting
information out, making decisions, etc.), users faced different affordances and different requirements for
effective use. Thus, they would naturally define effective use differently. It also makes sense that despite
all this variety, the notion that effective use enabled the accomplishment of one’s work was a common
view among users too. This is clearly reflected in the basic structure of each component in the model.
Finally, the third theme was the importance of professional judgment. This is reflected in our
model through reflective practice and the way it guides one’s own use (e.g., clinicians’ reflective practice
guides their levels of accuracy and consistency) and others’ use (e.g., managers’ reflective practice guides
clinicians’ reflective practice). Exercising reflective practice allowed users to socially construct a
reasonable path forward, providing a fair alternative to the impossibility of ‘perfect’ work.
6. Discussion
We believe our paper contributes by extending three bodies of work, discussed in turn below.
6.1 Extending a General Theory of Effective Use
Burton-Jones and Grange (2013) proposed the first general theory of effective use. Our theory
extends theirs in several ways. They proposed that effective use has three dimensions: how seamlessly
individuals interact with a system (‘transparent interaction’), how accurately they enter and obtain
representations from it (‘representational fidelity’), and how well they take actions based on its information
(‘informed action’). Our dimension of accuracy is similar to their notion of representational fidelity, and
our dimension of reflective practice is similar to, but broader than, their notion of informed action.1 These
similarities lend field-based support to their work. However, our theory offers no notion of transparent
interaction. Certainly, many users found HITS difficult to interact with.2 However, seamless interaction
did not appear to differentiate effective users from ineffective users in our case. That is, the difficulties in
finding information that we observed did not appear to reflect differences in navigation or interaction skills
1
Reflective practice is broader because informed action reflects the use of system outputs whereas reflective practice
covers the mindset guiding all of one’s use, which could include inputs and communications (not only outputs).
2
The following comment was typical: “It is a bit of a labyrinth, you have to go, you know, go down that path, turn
right, drop down there…there’s not really any shortcuts…” (Focus group 7, Clinicians, 4/2011).
37
per se. Rather, such difficulties appeared to be symptoms, the root cause being inconsistencies in how
different users used HITS that made it difficult to know where to put information and where to look for it.
Several clinicians also told us that good navigation was not, in and of itself, a good indicator of effective
use because untroubled navigation without clinical thinking would still be ineffective. In fact, it seemed
that reflective practice was actually a key driver of good navigation:
… an effective user is really good at sort of filtering out what do I need to fill out…. And
they seem to be very good at categorizing it by the screen. Like they go to clinical
summary, … do what they need to do in clinical summary—there’s much less bouncing
back and forth, versus an ineffective user (Interview 11, Educator/Trainer, 5/2011).
Thus, in our case, consistency of use and reflective practice were more critical dimensions than transparent
interaction and their addition to our theory represents a significant extension to Burton-Jones and Grange’s
work. The addition of reflective practice is important because Burton-Jones and Grange’s theory focuses
purely on representational aspects and does not include practice-oriented elements. Likewise, practiceoriented research typically eschews representational issues, as it “shift the focus from questions of
correspondence between descriptions and reality … to matters of practices/doings/actions” (Barad 2003 p.
802). By including both representation and practice, our account offers a more complete perspective.
6.2 Extending Affordance Actualization Theory (AAT)
Strong et al.’s (forthcoming) affordance actualization theory (AAT) offers a multilevel account of IT
effects in organizations. The study of affordances is an important trend in our field (Leonardi 2011;
Markus and Silver 2008) and AAT is the first theory to explain how multiple affordances at different levels
are actualized. Whereas AAT provides a multilevel account for how organizations use EHR systems, our
theory provides a multilevel account for how organizations do so effectively. Our theory is similar to AAT
in several ways. For instance, the affordances and goals that we observed (and the links/dependencies
among them) were similar to those in AAT, lending further support to that theory. In addition, feedback is
a key element in that theory, and in ours. AAT accounts for feedback by suggesting that users consider
outcomes arising from using their systems and, depending on how these outcomes match desired outcome,
adjust their actualization of affordances (or actualize other affordances). Our theory accounts for feedback
38
through the notion of reflective practice because to be reflective implies a willingness to look back on the
results of one’s actions and adjust. However, the notion of reflective practice allows us to have a broader
perspective on feedback than AAT, a perspective that covers feedforward and feedwithin as well:
“Feedback enables …systems… to adjust performance on the basis of information
about past results. Feedforward enables systems to … adjust performance in
anticipation of changing … circumstances. Feedwithin enables systems to monitor
internal situations and processes…and to coordinate system parts in concerted action of
the whole” (Bogart 1980 p. 237).
That is, reflective practitioners do not just respond to feedback, they also think ahead (feedforward) and
in-the-moment (feedwithin) (Schon 1983). This is reflected in our model in the way in which reflective
practice helps drive one’s effective use. The need for such a mindset was often stressed to us:
what’s going on clinically for the person … tells that professional, “I think I need to put a
bit more information in this case note … so the next person … understands” (Interview
35, Director, 1/2012).
In addition to providing a broader perspective on feedback, our use of reflective practice allows us to
extend the notion of an affordance in AAT. AAT defines affordances in terms of an immediate concrete
outcome arising from using a system. Although we viewed affordances in this way too (shown by the
outcomes in each of the eight components of our model), users with high levels of reflective practice did
not merely think of immediate outcomes related to their own work. Rather, they reflected on past
outcomes and thought ahead towards future outcomes for themselves and other users. Thus, because these
more effective users saw the potential outcomes differently—more fully—they literally saw the affordances
differently too (because affordances are defined in terms of outcomes). This is entirely consistent with
broader theories of human competence at work that suggest that more competent workers simply see their
work differently than less competent workers (Sandberg 2000; Sandberg and Pinnington 2009).
Our theory also extends AAT by accounting for top-down and bottom-up effects. AAT considers
bottom-up effects only, i.e., how an organization comes to actualize affordances at the organizational-level
through the actions of individual users. Our theory accounts for bottom-up effects (e.g., how accurate data
at higher levels depend on consistent use at lower levels), but it also accounts for top-down effects through
the way in which macro-actors influence lower-level actors (Mouzelis 1992). Specifically, by actualizing
39
the affordance for monitoring operations, high level managers could identify problems in how HITS was
used and try to influence lower-level actors’ use accordingly, through influencing their reflective practices.
Finally, our work also extends AAT by providing a more detailed account of consistency. In AAT,
consistency was a key determinant of whether affordances were being actualized at higher levels of
analysis. For instance, only if individuals entered data consistently could reports be run at higher levels of
analysis. Leonardi (2012) made a related point, arguing that for a collective to take advantage of an
affordance, they need first to have a consistent understanding of it. Our work extends these studies by
showing that consistency itself can be a complex notion. In our data, five types of consistency were salient
(utilization, place, form, amount, meaning) and they differed across levels (form and amount were relevant
for unstructured data at the instance level whereas meaning was relevant for structured data at the aggregate
level). In short, a fine-grained analysis of consistency was key to understanding effective use in our study.
6.3 Extending Multilevel Theory
Our theory extends Burton-Jones and Grange’s individual-level theory by providing a multilevel
account. Researchers have recently stressed the need to account for system use in a multilevel manner
(Burton-Jones and Gallivan 2007; Kane and Labianca 2011; Kang et al. 2012; Leonardi 2012; Strong et al.
forthcoming). Our work extends this emerging stream by providing the first multilevel account of effective
use. Compared to most multilevel studies in IS, our use of multilevel research is also novel in that we
needed to use a three-level model rather than the standard two-level model typically seen in IS research.
We believe that our work also helps to extend multilevel research in general. In traditional
multilevel research, levels are defined by aggregations of human actors, e.g., individuals within groups. In
contrast, when we defined levels, we accounted not only for the aggregation of human actors, but also the
aggregation of the representations they used in their work (specific interactions, summary representation,
and aggregate summary). As we noted earlier, the summary representation was formed through
composition—piecing together parts to form a whole picture. Composition is a well-known form of
aggregation in multilevel research (Kozlowski and Klein 2000), but compositions are typically considered
to be stable and fixed, such as a long-lived team (Mathieu and Chen 2011). In contrast, the summary
40
representations in our study were dynamic and fluid, as were the teams. The aggregate summary, on the
other hand, was formed through classification—slicing and dicing the data depending on the question. We
are not aware of any multilevel study that has examined classification. This is not surprising because in
traditional multilevel work, one wants the output of the aggregation to be a human aggregate (such as a
group of people), but the output of any classification is data, i.e., data about what is being classified. This
feature of classification means that it works well for aggregating representations, as in our study. For
instance, understanding classification enabled us to see how the individual clinicians who used data to
make clinical decisions worked at a different level of analysis from the individual managers and directors
who also used data to make managerial decisions, because the latter individuals were using summaries
(classifications of various sorts) of the clinicians’ data. The managers and directors also had the authority
to act on that data, as macro-actors (Mouzelis 1992). By accounting for both the human actors and the
representations they used—i.e., by taking a socio-technical approach—we were able to extend the way that
levels are typically thought of in multilevel research, responding to the call of Mathieu and Chen (2011) to
creatively apply multilevel principles to match the complex reality being studied.
6.4 Limitations and Future Research
Our work is an initial account and could be extended in many ways. Three areas for improvement
are particularly important. First, our theory concentrates on the effective use of a computerized system,
but the reality was that front-line staff used HITS in concert with a range of workarounds (such as talking
with other clinicians and using paper records). In our fieldwork, it was not always easy to decide on the
best unit of analysis—the effective use of HITS or the effective use of this broader and more dynamic set
of resources (HITS and its workarounds). Our theory focuses on the effective use of HITS alone, but we
recognize that this provides but a partial picture. Like researchers on cognition who reached a deeper
understanding of their subject matter when they expanded their unit of analysis to include external
representations (Hutchins 1995), it may be that we will reach a deeper understanding of effective use when
we expand our unit of analysis to consider the full set of workarounds that users have access to.
41
Second, our study focuses on the nature of effective use rather than its enablers. For instance, we
have not provided a complete perspective on what can be done to improve accuracy, consistency, and
reflective practice (e.g., through training or other initiatives). Providing such an account would be very
useful, particularly if the outcomes of any such initiative could be tracked over time, to provide a processoriented, longitudinal account of the benefits and costs of any given change.
Third, it remains to be seen how far our work can generalize to other settings. A key characteristic
of our case study context was that data accuracy was important. This assumption is reasonable in many
settings (Strong et al. 1997), but there are contexts in which organizations seem willing to live without it
(Cunha 2013). Even in community care settings, however, it is possible that our theory might not apply to
all cases.
Fourth, by accounting for both representation and practice oriented dimensions, our study builds
upon the view that these two perspectives, traditionally kept separate, are complementary (Norman 1993
pp. 1, 3). Including both also follows the tradition in affordance research to “cut across traditional subjectobject dualities” (Gaver 1996 p. 112). Even so, this idea could be developed further. For instance, on the
representation side, researchers could examine characteristics of effective representation in the literature,
such as characteristics of good classes (Parsons and Wand 2008), and determine if effective users were
better able to obtain such representations (e.g., retrieving reports with more informative classifications of
clients). Likewise, on the practice perspective, researchers could use hermeneutic ideas such as the world
of the text and the world behind the text (Lee and Dennis 2012) to understand the connection between the
world expressed in an EHR and the world of norms and practices that inform its construction.
7. Conclusion
Given how much organizations rely on information systems, it is surprising that there are no wellvalidated theories of effective use. Organizations have access to a wealth of research on how to increase
workers’ acceptance of systems, and how IT implementations can engender organizational change, but they
have very little research to help them understand how effectively they are using their systems or how they
can do so more effectively. This study is an initial attempt to develop theory in this area, based on close-up
42
insights from the field. It contributes by providing the first multilevel account of effective use and by
extending and integrating past work on effective use from a representation perspective and an affordance
perspective. Given our focus on the use of electronic health records, our work should be particularly
valuable to researchers and practitioners interested in achieving more effective use of electronic health
records. Our mid-range theory suggests that effective use is expressed in how accurately and consistently
users enter and obtain information from the system and the mindset that they adopt when using it (which
we called reflective practice). Effective use appears to evolve in a bottom-up and a top-down manner and
the middle-level of the organization appears to be an important buffering ground, balancing tensions
between front line and organizational level demands on effective use. We hope future research will
develop and test these ideas even further.
43
References
Agarwal, R. "Editorial Notes," Information Systems Research (22:1), March 2011, pp 1-4.
Agarwal, R., Angst, C. M., DesRoches, C. M., and Fischer, M. A. "Technological Viewpoints (Frames) about
Electronic Prescribing in Physicial Practices," Journal of the American Medical Informatics Association
(17) 2010, pp 425-431.
Ahearne, M., Jones, E., Rapp, A., and Mathieu, J. "High Touch Through High Tech: The Impact of Salesperson
Technology Usage on Sales Performance via Mediating Mechanisms," Management Science (54:4) 2008,
pp 671-685.
Barad, K. "Posthumanist Performativity: Toward an Understanding of How Matter Comes to Matter," Signs:
Journal of Women in Culture and Society (28:3), Spring 2003, pp 801-831.
Barki, H., Titah, R., and Boffo, C. "Information System Use-Related Activity: An Expanded Conceptualization of
Information System Use," Information Systems Research (18:2) 2007, pp 173-192.
Berente, N., and Yoo, Y. "Institutional Contradictions and Loose Coupling: Postimplementation of NASA's
Enterprise Information System," Information Systems Research (in press) 2012.
Birks, D. F., Fernandez, W., Levina, N., and Nasirin, S. "Grounded Theory Method in Information Systems
Research: Its Nature, Diversity, and Opportunities," European Journal of Information Systems (22) 2013,
pp 1-8.
Blumenthal, D., and Tavenner, M. "The "Meaningful Use" Regulation for Electronic Health Records," The New
England Journal of Medicine (363:6), August 2010, pp 501-504.
Bogart, D. H. "Feedback, Feedforward, and Feedwithin: Strategic Information in Systems," Behavioral Science
(25:4) 1980, pp 237-249.
Boud, D. "Relocating Reflection in the Context of Practice," in: Beyond Reflective Practice: New Approaches to
Professional Lifelong Learning, H. Bradbury, N. Frost, S. Kilminster and M. Zukas (eds.), Routledge,
Abingdon, UK, 2010, pp. 25-36.
Boudreau, M.-C., and Seligman, L. "Quality of Use of a Complex Technology: A Learning-Based Model," Journal
of Organizational and End User Computing (17:4) 2005, pp 1-22.
Burton-Jones, A., and Gallivan, M. J. "Towards a Deeper Understanding of System Usage in Organizations: A
Multilevel Perspective," MIS Quarterly (31:4) 2007, pp 657-679.
Burton-Jones, A., and Grange, C. "From Use to Effective Use: A Representation Theory Perspective," Information
Systems Research (24:3), Sept 2013, pp 632-658.
Burton-Jones, A., and Straub, D. "Reconceptualizing System Usage," Information Systems Research (17:1), April
2006, pp 38-60.
CIHI The CIHI Data Quality Framework Canadian Institute for Health Information, Ottawa, Canada, 2009.
Cordoba, J.-R., Pilkington, A., and Bernroider, E. W. N. "Information Systems as a Discipline in the Making:
Comparing EJIS and MISQ between 1995 and 2008," European Journal of Information Systems (21:5),
Sept 2012, pp 479-495.
Cunha, J. J. V. d. "A Dramaturgical Model of the Production of Performance Data," MIS Quarterly (in press) 2013.
Davern, M. J. "Towards a Unified Theory of Fit: Task, Technology, and Individual," in: Information Systems
Foundations: Theory, Representation, and Reality, D. Hart and S. Gregor (eds.), ANU E-Press, Acton,
Australia, 2007, pp. 49-69.
Davis, R., and Thurecht, R. "Care Planning and Case Conferencing: Building Effective Multidisciplinary Teams,"
Australian Family Physician (30:1), Jan 2001, pp 78-81.
DeShon, R. P., Kozlowski, S., Schmidt, A. M., Milner, K. R., and Wiechmann, D. "A Multiple-Goal, Multilevel
Model of Feedback Effects on the Regulation of Individual and Team Performance," Journal of Applied
Psychology (89:6) 2004, pp 1035-1056.
Elmes, M. B., Strong, D. M., and Volkoff, O. "Panoptic Empowerment and Reflective Conformity in Enterprise
Systems-Enabled Organizations," Information & Organization (15:1), Jan 2005, pp 1-37.
Gaver, W. "Affordances for Interaction: The Social is Material for Design," Ecological Psychology (8:2) 1996, pp
111-129.
Glaser, B. G., and Strauss, A. L. The Discovery of Grounded Theory Aldine, Chicago, 1967.
Goodman, P. S. Missing Organizational Linkages: Tools for Cross-Level Research Sage, CA, 2000.
Hadar, I., and Soffer, P. "Variations in Conceptual Modeling: Classification and Ontological Analysis," Journal of
the Association for Information Systems (7:8) 2006, pp 568-592.
Haggerty, J. L., Reid, R. J., Freeman, G. K., Starfield, B. H., Adair, C. E., and McKendry, R. "Continuity of Care: A
Multidisciplinary Review," British Medical Journal (327), Nov 2003, pp 1219-1221.
44
Hanseth, O., Jacucci, E., Grisot, M., and Aanestad, M. "Reflexive Standardization: Side Effects and Complexity in
Standard Making," MIS Quarterly (30:1), Aug 2006, pp 563-581.
Higgins, D. "Editorial: Why Reflect? Recognising the Link between Learning and Reflection," Reflective Practice
(12:5), Oct 2011, pp 583-584.
Hofmann, D. A. "Issues in Multilevel Research: Theory Development, Measurement, and Analysis," in: Handbook
of Research Methods in Industrial and Organizational Psychology, S.G. Rogelberg (ed.), Blackwell,
Malden, MA, 2002, pp. 247-274.
Hutchins, E. Cognition in the Wild MIT Press, Bradford, 1995.
Jarvis, P. "Reflective Practice and Nursing," Nurse Education Today (12) 1992, pp 174-181.
Kane, G. C., and Borgatti, S. P. "Using Social Network Analysis to Develop Configural Constructs: The Centrality
of IS Proficiency," MIS Quarterly (35:4), Dec 2011, pp 1063-1078.
Kane, G. C., and Labianca, G. "IS Avoidance in Healthcare Groups: A Multilevel Investigation," Information
Systems Research (22:3), Sept 2011, pp 504-522.
Kang, K., Lim, K. H., Kim, M. S., and Yang, H. D. "A Multilevel Analysis of the Effect of Group Appropriation on
Collaborative Technology Usage and Performance," Information Systems Research (23:1) 2012, pp 214230.
Ko, D.-G., and Dennis, A. R. "Profiting from Knowledge Management: The Impact of Time and Experience,"
Information Systems Research (22:1), March 2011, pp 134-152
Kozlowski, S. W. J., and Klein, K. J. "A Multilevel Approach to Theory and Research in Organizations," in:
Multilevel Theory, Research, and Methods in Organizations, K.J. Klein and S.W.J. Kozlowski (eds.),
Jossey-Bass, California, 2000, pp. 3-90.
Lackoff, G. Women, Fire, and Dangerous Things: What Categories Reveal About the Mind Chicago University
Press, Chicago, 1987.
Lanham, H. J., Leykum, L. K., and McDaniel, R. R. "Same Organization, Same Electronic Health Records (EHRs),
Different Use: Exploring the Linkage Between Practice Member Communication Patterns and EHR Use
Patterns in an Ambulatory Care Setting," Journal of the American Medical Informatics Association (19:3),
May-Jun 2011, pp 382-391.
Lee, A. S., and Dennis, A. R. "A Hermeneutic Interpretation of a Controlled Laboratory Experiment: A Case Study
of Decision Making with a Group Support System," Information Systems Journal (22:1), Jan 2012, pp 327.
Leonardi, P. M. "When Flexible Routines Meet Flexible Technologies: Affordance, Constraint, and the Imbrication
of Human and Material Agencies," MIS Quarterly (35:1), March 2011, pp 147-168.
Leonardi, P. M. "When Does Technology Use Enable Network Change in Organizations? A Comparative Study of
Feature Use and Shared Affordances," MIS Quarterly (in press) 2012.
LeRouge, C., Hevner, A. R., and Collins, R. W. "It's More than Just Use: An Exploration of Telemedicine Use
Quality," Decision Support Systems (43) 2007, pp 1287-1304.
Levina, N. "Collaborating on Multiparty Information Systems Development Projects: A Collective Reflection-inAction View," Information Systems Research (16:2), June 2005, pp 109-130.
Mann, K., Gordon, J., and MacLeod, A. "Reflection and Reflective Practice in Health Professions Education: A
Systematic Review," Advances in Health Sciences Education (14) 2009, pp 595-621.
Markus, M. L., and Silver, M. S. "A Foundation for the Study of IT Effects: A New Look at DeSanctis and Poole's
Concepts of Structural Features and Spirit," Journal of the Association for Information Systems (9:10/11)
2008, pp 609-632.
Mathieu, J. E., and Chen, G. "The Etiology of the Multilevel Paradigm in Management Research," Journal of
Management (37:2) 2011, pp 610-641.
Morgeson, F. P., and Hofmann, D. A. "The Structure and Function of Collective Constructs: Implications for
Multilevel Research and Theory Development," Academy of Management Review (24:2), April 1999, pp
249-265.
Mouzelis, N. "The Interaction Order and the Micro-Macro Distinction," Sociological Theory (10:1) 1992, pp 122128.
Nan, N. "Capturing Bottom-Up IT Use Processes: A Complex Adaptive Systems Model," MIS Quarterly (35:2)
2011, pp 505-532.
Norman, D. A. "Cognition in the Head and in the World: An Introduction to the Special Issue on Situated Action,"
Cognitive Science (17) 1993, pp 1-6.
Oborn, E., Barrett, M., and Davidson, E. "Unity in Diversity: Electronic Patient Record Use in Multidisciplinary
Practice," Information Systems Research (22:3), Sept 2011, pp 547-564.
45
Orlikowski, W. J., and Scott, S. V. "Sociomateriality: Challenging the Separation of Technology, Work and
Organization," The Academy of Management Annals (2:1) 2008, pp 433-474.
Parsons, J., and Wand, Y. "Using Cognitive Principles to Guide Classification in Information Systems Modeling,"
MIS Quarterly (32:4), Dec 2008, pp 839-868.
Pavlou, P., Dimoka, A., and Housel, T. J. "Effective Use of Collaborative IT Tools: Nature, Antecedents, and
Consequences," Proceedings of the 41st Hawaii International Conference on System Sciences (HICSS),
IEEE, Hawaii, Jan 7-10, 2008.
Pavlou, P., and El-Sawy, O. O. "From IT Leveraging Competence to Competitive Advantage in Turbulent
Environments: The Case of New Product Development," Information Systems Research (17:3) 2006, pp
198-227.
Riskin, D. "The Next Revolution in Healthcare," in: Forbes, viewed online (Oct 22, 2012):
http://www.forbes.com/sites/singularity/2012/10/01/the-next-revolution-in-healthcare/, 10/01/2012, 2012.
Sandberg, J. "Understanding Human Competence at Work: An Interpretive Approach," Academy of Management
Journal (43) 2000, pp 9-25.
Sandberg, J., and Pinnington, P. "Professional Competence as Ways of Being: An Existential Ontological
Perspective," Journal of Management Studies (46) 2009, pp 1138-1170.
Sandberg, J., and Tsoukas, H. "Grasping the Logic of Practice: Theorizing Through Practical Rationality," Academy
of Management Review (36:2), April 2011, pp 338-360.
Schon, D. A. The Reflective Practitioner: How Professionals Think in Action Basic Books, New York, 1983.
Smith, K. G., and Hitt, M. A. (eds.) Great Minds in Management: The Proces of Theory Development. Oxford
University Press, New York, 2005.
Strauss, A., and Corbin, J. Basics of Qualitative Research Sage Publications, Thousand Oaks, CA, 1998.
Strong, D. M., Lee, Y. W., and Wang, R. Y. "Data Quality in Context," Communications of the ACM (40:5), May
1997, pp 103-110.
Strong, D. M., Volkoff, O., Johnson, S. A., Pelletier, L. R., Bar-On, I., Tulu, B., Kashyap, N., Trudel, J., and Garber,
L. "A Theory of Clinic-EHR Affordance Actualization," Journal of the Association for Information
Systems) forthcoming.
Sundaram, S., Schwarz, A., Jones, E., and Chin, W. W. "Technology Use on the Front Line: How Technology
Enhances Individual Performance," Journal of the Academy of Marketing Science (35:1) 2007, pp 101-112.
Thompson, J. D. Organizations in Action McGraw-Hill, New York, 1967.
Thompson, N., and Pascal, J. "Developing Critically Reflective Practice," Reflective Practice (13:2), Apr 2012, pp
311-325.
Thorpe, K. "Reflective Learning Journals: From Concept to Practice," Reflective Practice (5:3) 2004, pp 327-343.
Trudel, M.-C., Pare, G., and Laflamme, J. "Health Information Technology Success and the Art of Being Mindful:
Preliminary Insights from a Comparative Case Study Analysis," Health Care Management Review (37:1),
Jan-Mar 2012, pp 31-42.
Urquhart, C., Lehmann, H., and Myers, M. D. "Putting the "Theory" Back into Grounded Theory: Guidelines for
Grounded Theory Studies in Information Systems," Information Systems Journal (20) 2010, pp 357–381.
Venkatesh, V., Morris, M. G., Davis, G. B., and Davis, F. D. "User Acceptance of Information Technology: Toward
a Unified View," MIS Quarterly (27:3), September 2003, pp 425-478.
Volkoff, O., and Strong, D. M. "Critical Realism and Affordances: Theorizing IT-Associated Organizational Change
Processes," MIS Quarterly (37:3) 2013, pp 819-834.
Wand, Y., and Weber, R. "On the Deep Structure of Information Systems," Information Systems Journal (5) 1995,
pp 203-223.
Weber, R. "Editor's Comments: Still Desperately Seeking the IT Artifact," MIS Quarterly (27:3) 2003, pp iii-xi.
Weick, K. E., and Roberts, K. H. "Collective Mind in Organizations: Heedful Interrelating on Flight Decks,"
Administrative Science Quarterly (38), Sept 1993, pp 357-381.
Weick, K. E., Sutcliff, K. M., and Obstfeld, D. "Organizing for High Reliability: Processes of Collective
Mindfulness," Research in Organizational Behavior (21) 1999, pp 81-123.
Wilkin, K., and Slevin, E. "The Meaning of Caring to Nurses: An Investigation into the Nature of Caring Work in an
Intensive Care Unit," Journal of Clinical Nursing (13:1), Jan 2004, pp 50-59.
Wilson, J. P. "Reflecting-on-the-Future: A Chronological Consideration of Reflective Practice," Reflective Practice
(9:2), May 2008, pp 177-184.
46