i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
journal homepage: www.ijmijournal.com
Transitioning from a computerized provider order entry and
paper documentation system to an electronic health record:
Expectations and experiences of hospital staff
Eric S. Kirkendall a,b,c,∗ , Linda M. Goldenhar c , Jodi L. Simon c ,
Derek S. Wheeler d , S. Andrew Spooner a,b
a
Division of Hospital Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA
Division of Biomedical Informatics, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA
c James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA
d Division of Critical Care Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, USA
b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Objectives: To examine healthcare worker’s perceptions, expectations, and experiences
Received 6 September 2012
regarding how work processes, patient-related safety, and care were affected when a qua-
Received in revised form
ternary care center transitioned from one computerized provider order entry (CPOE) system
21 July 2013
to a full electronic health record (EHR).
Accepted 7 August 2013
Methods: The I-SEE survey was administered prior to and 1-year after transition in systems. The construct validity and reliability of the survey was assessed within the current
Keywords:
population and also compared to previously published results. Pre- and 1-year post-
Electronic health records
implementation scale means were compared within and across time periods.
Medical informatics
Results: The majority of respondents were nurses and personnel working in the acute care
Quality of healthcare
setting. Because a confirmatory factor analysis indicated a lack of fit of our data to the I-SEE
Patient safety
survey’s 5-factor structure, we conducted an exploratory factor analysis that resulted in a
User satisfaction
7-factor structure which showed better reliability and validity. Mean scores for each factor
indicated that attitudes and expectations were mostly positive and score trends over time
were positive or neutral. Nurses generally had less positive attitudes about the transition
than non-nursing respondents, although the difference diminished after implementation.
Conclusions: Findings demonstrate that the majority of responding staff were generally positive about transitioning from CPOE system to a full electronic health record (EHR) and
understood the goals of doing so, with overall improved ratings over time. In addition, the
I-SEE survey, when modified based on our population, was useful for assessing patient care
and safety related expectations and experiences during the transition from one CPOE system
to an EHR.
© 2013 Elsevier Ireland Ltd. All rights reserved.
∗
Corresponding author at: Division of Hospital Medicine, Cincinnati Children’s Hospital Medical Center, 3333 Burnet Avenue ML-9009,
Cincinnati, OH 45229, USA. Tel.: +1 513 636 1260; fax: +1 513 636 7247.
E-mail addresses: [email protected], [email protected] (E.S. Kirkendall).
1386-5056/$ – see front matter © 2013 Elsevier Ireland Ltd. All rights reserved.
http://dx.doi.org/10.1016/j.ijmedinf.2013.08.005
1038
1.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
Introduction
Over the past 20 years, research findings have accelerated our
knowledge of how healthcare providers think about adopting
and using information technology in healthcare [1–5]. Numerous studies have described nurse and physician attitudes,
perceptions, expectations, and experiences around implementing new clinical information systems. While most early
studies focused on physicians, nurses have also reported
favorable attitudes in the last 20 years [1,6]. Studies conducted as early as the 1970s showed that both groups express
positive attitudes and expectations related to health information technology (HIT) [7,8]. In more recent studies, most
care providers said they believed technology could improve
healthcare and healthcare delivery, including patient safety
[1,9,10]. A recent study found 93% of physicians agreed or
strongly agreed that using computers in clinical care helps
improve healthcare quality [10]. Overall, factors shown to
positively influence successful EHR implementation include
training and support, mitigating unintended consequences,
minimizing adverse effects on time and efficiency, and managing or limiting the gap between expectations and perception
of outcomes [11–15].
While surveys have shown that healthcare providers are
overall optimistic toward EHRs, they are still concerned with
privacy and security, workflow changes, distraction from
direct patient care, and other unintended consequences of
using an EHR system [9,16–23], which has been shown can
result in lower stakeholder “buy-in” leading to potential
rejection of the system [12,15,24]. Indeed, “buy-in” and user
attitudes may prove to be a more critical variable for successful implementation and adoption than budget, technology,
or sophistication of the vendor [25]. While general attitudes
toward EHRs remain positive, attitudes about routine use are
often negative [26]. A recent study has indicated that attitudes
and the perceived usefulness of computer technology have
shifted over time [6]. Recent changes in the healthcare and
technology landscape including adoption driven by the Meaningful Use incentives may cause user attitudes and acceptance
of IT implementation projects to vary from past reports [27].
Most current studies examining the transition from one
HIT system to another have targeted the ambulatory setting
[28–30]. Only one example was found that examined transition from one electronic order entry system to a full EHR in
the inpatient setting which showed wide variation in expectations and experiences for both physicians and nurses [31]. The
overall aim of our study was to administer a previously validated nursing survey to a wide population of health providers
in the inpatient setting in order to better understand their perceptions of how changing from one CPOE system to a full EHR
would affect them personally and their ability to safely care
for patients.
2.
Methods
2.1.
Setting
This study was conducted at Cincinnati Children’s Hospital
Medical Center (CCHMC), a 523-bed tertiary care academic
medical center. CCHMC is a level 1 trauma center which, in
2010, had 1,078,798 patient encounters, 1498 active medical
staff, and 936 faculty members. In late 2008, implementation of the new EHR system (Epic SystemsTM ; Verona, WI)
began in several ambulatory pilot groups. Additional outpatient divisions were systematically brought on-line until all
units were live by January 2012. The inpatient implementation
go-live date was January 10th, 2010. Prior to implementation, inpatient care providers were using electronic order
entry through a proprietary vendor product with a highly customized user interface. Most patient care documentation was
done on paper, although some aspects of nursing care were
documented electronically to facilitate research studies and
quality improvement activities. Physicians did not document
electronically prior to implementation. All order entry and
documentation has been performed electronically since the
new EHR was implemented.
2.2.
Human subjects protection
This study was deemed exempt by the IRB since no patient
data were used and survey responses were anonymous.
2.3.
Data collection
We administered the Information Systems Expectations and
Experiences (I-SEE) survey developed by Wakefield and colleagues to evaluate the hospital staff’s expectations prior to
implementation and the change in perceptions after the EHR
transition [32]. The I-SEE was selected because of its strong
psychometric properties and direct relevance to our project
in terms of measuring perceived changes in work-process
and patient care/quality resulting from EHR implementation.
Using the I-SEE in a larger and more diverse audience allowed
us to further explore and establish the instrument’s reliability
and external validity. Original survey materials are available
on the Agency for Healthcare Research and Quality’s Health
IT Toolkit website (http://healthit.ahrq.gov); the CCHMC version is available as an online supplement. The I-SEE contains
35 questions/items distributed across 7 scales:
•
•
•
•
•
•
•
Provider–patient communication (3 items)
Inter-provider communication (3 items)
Inter-organizational communication (2 items)
Work life changes (4 items)
Improved care (7 items)
Support and resources (8 items)
Patient care processes (8 items)
The first five scales use a 7-point response scale ranging
from much worse (−3), to no change (0), to much improved
(+3) and measure perceptions of how the new clinical information system would (or did) impact various work processes
in the hospital. The last two scales use a non-neutral 6point agree/disagree Likert response scale (from Strongly
Disagree (+1) to Strongly Agree (+6)) to measure perceptions of
the information system’s implementation strategy and quality. We maintained the same response scales as the I-SEE
and except for modifying the question tense, items were
identically phrased for both pre and post implementation
1039
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
administrations (time period 1, referred to as T1 ; and time
period 2, referred to as T2 ). For instance, in the preimplementation survey questions were phrased in the future
tense, e.g., “To what extent do you think the following will be
either worsened, stay the same, or improved?” In the postimplementation survey, the question was phrased in the past
tense; “To what extent do you think the following has been
either worsened, stayed the same, or improved?” T1 responses
were considered staff’s perceived expectations regarding EHR
implementation, while T2 responses were understood to be
the perceived experiences of the actual implementation. We
also collected demographic data (work unit, staff role, tenure
as a health care provider, experience with healthcare technology, etc.) and provided a comments box for respondents to
share additional feedback. We analyzed all available data and
did not discard incomplete data sets (those with unanswered
questions).
2.4.
Survey administration timeline and participants
The survey was administered online via Survey MonkeyTM
(Palo Alto, CA) at two time periods, 1 year apart:
T1 : Five days prior to inpatient implementation until the day
prior to the go-live date (January 5th–9th, 2010)
T2 : One year after implementation (January 10th,
2011–February 10th, 2011).
Because our email system would not allow us to send the
email request to only in-patient staff, we had to send it to all
hospital employees at both T1 and T2 . A reminder email was
sent two weeks prior to the conclusion of time T2 . The email
contained a link to the survey as well as a disclaimer in both
the subject and the body of the email requesting that employees who did not work on inpatient units, had no direct patient
care responsibilities, or would not be using the EHR, delete the
message and not take the survey. Further filtering was done
by manually eliminating responses from non-inpatient staff
or those in non-clinical roles. We were able to do this because
respondents were asked to provide their primary work unit.
No monetary incentives were offered for participating in the
survey.
2.5.
Data analysis
Using SASTM statistical software (Cary, NC), we calculated
response rates by dividing the number of respondents (T1
and T2 separately) by the number of eligible employees (i.e.,
inpatient staff required to take the EHR training course), and
also descriptive statistics on participant groups for each time
period. Since our respondent pool was different from that in
Wakefield’s original study, we conducted a confirmatory factor
analysis for each target population and time period to determine if the original factor structure fit our data. Based on our
findings, we then conducted an exploratory factor analysis, as
well as calculations of Cronbach ˛ coefficients to determine
the construct validity and reliability of the resultant factors.
Those findings allowed us to proceed using the grouped items
that loaded onto individual factors as the survey scales to
measure staff’s expectations and experiences. Means for each
scale (scale scores, also called factor scores) were calculated
for nurses only and for non-nurse respondents. Student t-tests
were then conducted to examine if there were statistically significant changes in attitudes regarding the transition to the
EHR system from T1 , to T2 for nurses only, for the organization as a whole, and for nurses compared to everyone else in
the organization. A p-value of less than 0.05 was considered
significant.
3.
Results
3.1.
Response rates and demographics
Response rates were determined by dividing the number of
respondents (T1 and T2 ) by the number of eligible employees
(i.e., those required to take the EHR training course, n = 7213).
Response rates for T1 and T2 were 5.2% and 13.6% respectively.
In T1 , 97 sets of survey responses from outpatient providers
were removed prior to analysis (based on the staff’s primary
work unit response), while 731 sets of response data were
removed from T2 . Many of these sets of data were also incomplete, that is, providers did not answer all questions from the
survey.
The sample had similar distribution and representation
across all time periods (Table 1). Most respondents worked in
acute care medical units. The “other” category represents a
heterogeneous group of inpatient locations such as psychiatry or short stay units. Nurses and physicians represented
the largest homogeneous groups of staff positions across both
time periods. The “other” category included social workers,
nutritionists, unit clerks, and patient attendants. The majority
of respondents had more than 10 years of experience in health
Table 1 – Respondent demographics.
T1
T2
Inpatient work unit
Acute care
Critical care
Emergency
Perioperative
Other
n = 377
149 (40%)
64 (17%)
13 (3%)
46 (12%)
105 (28%)
n = 983
345 (35%)
127 (13%)
87 (9%)
117 (12%)
307 (31%)
Staff position
Prescriber (MDs, NPs)
Nurse
Other
n = 374
97 (26%)
146 (39%)
131 (35%)
n = 971
206 (21%)
358 (37%)
407 (42%)
Years in healthcare
0–3 years
4–10 years
>10 years
n = 375
48 (13%)
108 (29%)
219 (58%)
n = 981
188 (19%)
276 (28%)
517 (53%)
EHR experience (years)
<3 years
3–4 years
5–7 years
>7 years
n = 298
62 (21%)
60 (20%)
96 (32%)
80 (27%)
n = 667
199 (30%)
141 (21%)
181 (27%)
146 (22%)
Comparison of respondent demographics across survey time
periods.
T1 , time period 1 (pre-implementation); T2 , time period 2 (postimplementation); EHR, electronic health record.
1040
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
Table 2 – Item and scale analysis: factor loadings and reliability assessment.
Communication* (coefficient ˛ = 0.90)
Factor loading
How often families are asked the same question
Able to share important information with patients and families
Able to involve patients and families in the care planning process
Communication between departments
Communication at end of shift handoffs
Communication when patients are transferred to different units within the hospital
Communication when patients are transferred to other facilities
Communication when patients are readmitted or receive follow-up outpatient care
0.45
0.52
0.46
0.72
0.64
0.77
0.58
0.73
Job Satisfaction* (coefficient ˛ = 0.88)
Factor loading
The amount of time I spend preparing discharge documents
The amount of professional satisfaction I get out of my job
The effect on the hospital to recruit and retain high quality staff
How much I enjoy my job
0.45
0.84
0.63
0.83
Quality of Patient Data* (coefficient ˛ = 0.91)
Factor loading
The consistency with which patient care data are recorded
The accuracy and validity of the patient care data being recorded
0.73
0.65
Quality and Safety of Patient Care* (coefficient ˛ = 0.89)
Factor loading
The overall safety of patient care
The timeliness with which patient care services are provided
The appropriateness of patient care orders
Legibility and clarity of patient care orders
0.53
0.54
0.66
0.51
Employee Understanding and Support of Implementation** (coefficient ˛ = 0.86)
Factor loading
I support the planned change in current clinical information systems
My coworkers support the planned change in clinical information systems
My supervisor supports the planned change in clinical information systems
I will have no difficulty in adapting to information systems changes
I understand the decision to change clinical information systems
0.92
0.69
0.58
0.48
0.76
Organizational Support for Implementation** (coefficient ˛ = 0.77)
Factor loading
I know who the super users are on my work unit
Sufficient resources have been provided for me to learn to use the new system
Sufficient technical IT support will be/was available to operate the new system
0.42
0.85
0.83
The “Rights” of Patient Care*** (coefficient ˛ = 0.97)
Factor loading
The right treatment
To the right patient
At the right time
In the right amount, dose or intensity
In the right way
By the right person
With the right information
In the right location
0.87
0.89
0.88
0.90
0.94
0.92
0.91
0.93
Question items from the survey were loaded onto scales with other questions that respondents answered similarly. Factor loading scores
greater than 0.40 are considered significant. Cronbach ˛ values demonstrating the cohesiveness of the grouped question items are shown in
parentheses next to each of the 7 scales.
Question stem and scoring:
* (T1 ) To what extent do you think the following will be worse (−3, −2, −1), stay the same (0), or improved (1, 2, 3) as a result of implementation?
* (T2 ) To what extent do you think the following has been either worsened (−3, −2, −1), stayed the same (0), or improved (1, 2, 3) as a result of
implementation?
** To what extent do you agree/disagree with the following? (1–6 Disagree/Agree Likert scale)
*** (T1 ) The Epic clinical information system will improve our ability to give care. . . (1–6 Disagree/Agree Likert scale)
*** (T2 ) The Epic clinical information system has improved our ability to give care. . . (1–6 Disagree/Agree Likert scale)
1041
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
care and was fairly evenly distributed in terms of experience
using EHRs.
3.2.
Factor analysis
The initial confirmatory factor analysis yielded fit index values
that were outside the range for acceptable fit, indicating a large
enough departure from the original author’s model to warrant further examination by conducting an exploratory factor
analysis.
The exploratory factor analysis indicated that a 7-factor
rather than the original 5-factor structure provided the best
fit to our data. The chi-square fit statistics were statistically
significant, indicating that there was at least one common factor, but that more factors were needed. Since the large sample
sizes make the chi-square test extremely sensitive, we based
our choice of seven factors on the scree test and the interpretability of factors.
All eight communications-related items loaded onto one
‘Communication’ factor (Table 2) compared to Wakefield’s
three items (coefficient ˛ = 0.90 versus Wakefield ˛ = 0.82, 0.86,
0.83). Our Job Satisfaction factor and Wakefield’s Work-Life
Changes scale contained the same four items and had the
same reliability value (˛ = 0.88). We renamed it ‘Job Satisfaction’ because we believe it better reflected what the items
were intended to measure. The seven items in Wakefield’s
Improved Care Expectations factor (coefficient ˛ = 0.90) loaded
onto two separate factors: Quality of Patient Data (coefficient
˛ = 0.91) and Quality and Safety of Patient Care (coefficient
˛ = 0.89) in our study. One item on the original survey – access
to information improved my ability to make good patient
care decisions loaded onto both Job Satisfaction and Quality and Safety of Patient Care and was therefore eliminated.
Items in Wakefield’s Support and Resources scale (coefficient
˛ = 0.88), comprised of 8 items, loaded onto two 2 factors in
our study: Employee Understanding and Support of Implementation (5 items, coefficient ˛ = 0.86) and Organizational
Support for Implementation (3 items, coefficient ˛ = 0.77). The
final factor in both studies contained items measuring the
nursing tenets of Patients’ “Rights” (coefficient ˛ = 0.97; Wakefield = 0.99). SAS analytical output from the factor analysis is
available as online supplementary material.
3.3.
Trends over time and group comparisons
Nurse scores significantly improved from T1 to T2 , except for
Communication, which did not change significantly (Table 3).
All respondents (including nurses) answered more positively
(i.e., had fewer concerns) at T2 in terms of Job Satisfaction,
Quality and Safety of Patient Care, Organizational Support
for the Transition, and the “Rights” of Patient Care. No group
demonstrated a statistically significant decrease in mean
scores over time on any of the measurement scales.
To investigate changes in nursing perceptions compared to
those of all other care providers combined, we examined mean
scale score differences between the 2 subgroups at T1 and
T2 . At T1 , nurses reflected a less positive perception of the
transition in terms of its potential impact on 5 scales: Communication, Job Satisfaction, Quality of Patient Data, Quality
and Safety of Patient Care, and Employee Understanding and
Table 3 – Scale rating trends, times T1 to mean score trends from T1 to T2 for each survey scale for nurses only (a) and for
the entire organization (b). The p values and trends for each scale are depicted in the last 2 columns.
(a) Nurse respondents
Scale name
Communicationa
Job Satisfactiona
Quality of Patient Dataa
Quality and Safety of Patient Carea
Employee Understanding and Supportb
Organizational Supportb
The “Rights” of Patient Careb
T1 score
Mean (SD)
n = 146
T2 score
Mean (SD)
n = 358
Mean
difference 95%
CI
p-Value
Trend from
T1 to T2
0.82 (1.01)
0.03 (0.99)
0.80 (1.25)
0.56 (1.09)
4.46 (0.92)
4.58 (1.12)
4.40 (0.90)
0.88 (0.84)
0.56 (1.03)
1.04 (1.20)
1.04 (1.14)
4.64 (0.84)
4.80 (0.96)
4.78 (0.91)
(−0.113, 0.232)
(0.330, 0.724)
(0.007, 0.477)
(0.265, 0.699)
(0.009, 0.347)
(0.027, 0.421)
(0.202, 0.556)
0.499
<0.001
0.044
<0.001
0.039
0.026
<0.001
–
↑
↑
↑
↑
↑
↑
(b) All eligible respondents (organization, including nurses)
Scale name
T1 score
Mean (SD)
n = 377
T2 score
Mean (SD)
n = 983
Mean
difference 95%
CI
p-Value
Trend from
T1 to T2
Communicationa
Job Satisfactiona
Quality of Patient Dataa
Quality and Safety of Patient Carea
Employee Understanding and Supportb
Organizational Supportb
The “Rights” of Patient Careb
0.99 (0.94)
0.23 (1.01)
1.01 (1.17)
0.85 (1.09)
4.67 (0.92)
4.54 (1.16)
4.45 (0.91)
0.95 (0.88)
0.60 (1.09)
1.11 (1.27)
1.13 (1.11)
4.69 (0.89)
4.74 (1.03)
4.69 (0.95)
(−0.143, 0.071)
(0.242, 0.497)
(−0.055, 0.242)
(0.154, 0.418)
(−0.090, 0.126)
(0.064, 0.321)
(0.125, 0.352)
0.506
<0.001
0.215
<0.001
0.746
0.003
<0.001
–
↑
∼
↑
∼
↑
↑
SD, standard deviation; CI, confidence interval; T1 , time period 1 (pre-implementation); T2 , time period 2 (post-implementation).
Scale scoring: (−3, −2, −1, 0, +1, +2, +3) [worse ↔ improved].
b
Scale scoring: (+1, +2, +3, +4, +5, +6) [disagree ↔ agree].
a
1042
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
Table 4 – Comparing nurse responses to other care providers.
(a) Nurses vs. all other care providers (T1 )
Scale name
Nurses
T1 score
Mean (SD)
n = 146
Others
T1 score
Mean (SD)
n = 231
Mean
difference 95%
CI
p-Value
Highest rating
group
Communicationa
Job Satisfactiona
Quality of Patient Dataa
Quality and Safety of Patient Carea
Employee Understanding and Supportb
Organizational Supportb
The “Rights” of Patient Careb
0.82 (1.01)
0.03 (0.99)
0.80 (1.25)
0.56 (1.09)
4.46 (0.92)
4.58 (1.12)
4.40 (0.90)
1.10 (0.87)
0.34 (1.01)
1.16 (1.10)
1.03 (1.05)
4.81 (0.89)
4.52 (1.19)
4.49 (0.92)
(0.094, 0.482)
(0.103, 0.522)
(0.120, 0.606)
(0.251, 0.697)
(0.164, 0.543)
(−0.305, 0.182)
(−0.102, 0.282)
0.004
0.004
0.004
<0.001
<0.001
0.619
0.355
Others
Others
Others
Others
Others
–
–
(b) Nurses vs. all other care providers (T2 )
Scale name
Nurses
T2 score
Mean (SD)
n = 358
Others
T2 score
Mean (SD)
n = 625
Mean
difference 95%
CI
p-Value
Highest rating
group
Communicationa
Job Satisfactiona
Quality of Patient Dataa
Quality and Safety of Patient Carea
Employee Understanding and Supportb
Organizational Supportb
“Rights” of Patient Careb
0.88 (0.84)
0.56 (1.03)
1.04 (1.20)
1.04 (1.14)
4.64 (0.84)
4.80 (0.96)
4.78 (0.91)
1.00 (0.89)
0.61 (1.12)
1.13 (1.31)
1.18 (1.10)
4.72 (0.91)
4.69 (1.07)
4.62 (0.97)
(0.013, 0.242)
(−0.087, 0.200)
(−0.066, 0.268)
(−0.005, 0.288)
(−0.031, 0.207)
(−0.246, 0.030)
(−0.283, −0.027)
0.029
0.438
0.236
0.058
0.147
0.124
0.018
Others
–
–
–
–
–
Nurses
Nurse respondent scores compared to all other care providers in the organization at times T1 (a) and T2 (b). The p values and highest rating
group for each scale are depicted in the last 2 columns.
SD, standard deviation; CI, confidence interval; T1 , time period 1 (pre-implementation); T2 , time period 2 (post-implementation).
a
Scale scoring: (−3, −2, −1, 0, +1, +2, +3) [worse ↔ improved].
b
Scale scoring: (+1, +2, +3, +4, +5, +6) [disagree ↔ agree].
Support (Table 4a). At T2 , nurse respondents rated the “Rights”
of Patient Care higher than other care providers while the
other care providers rated Communication higher than nurses
(Table 4b). All other scale ratings showed no statistically significant differences across groups at T2 .
4.
Discussion
Our study is the first known to the authors that documents
inpatient staff expectations and experiences related to transitioning from a CPOE system to a comprehensive EHR. Previous
studies largely included only nurses or physicians and have
examined the introduction of novel health informatics technology to clinicians. Our study helped address the knowledge
gap by asking all staff involved in patient care about their
expectations (pre) and experiences (post) related to the Quality
and Safety of Patient Care while transitioning from a semielectronic system to a full EHR.
The survey respondents were heterogeneous, representing many staff roles including patient care assistants, child
life specialists, nutrition specialists, and others (Table 1). Most
had more than 10 years of medical experience, demonstrating
that the sample was not biased toward younger, and perhaps
more computer-facile staff. The majority had at least 5 years
of experience with one form of EHR or another, likely reflecting the fact that CCHMC was already utilizing a more limited
CPOE system prior to implementing an entirely new full EHR.
The I-SEE survey was originally developed and validated
exclusively in a nursing population that had no significant
experience with EMRs. Therefore we believed it was important to examine the factor structure to ensure it was a good
fit for our study population. Indeed, the confirmatory factor
analysis revealed a lack of fit to the original factor structure
and the ensuing exploratory analysis revealed the reasons
why. Most notable was that in our study, staff answered all
communication-related items similarly (loading onto one 8item factor compared to Wakefield’s 3 factors). They also made
a distinction between employee versus Organizational Support for the Transition, leading to two factors rather than one
as in the Wakefield study (Table 2). One possible explanation
for the difference is that our respondents were already experienced with EHRs and thus had a more nuanced understanding
of the potential impact of the changes. The inclusion of care
provider roles other than nurses may have also resulted in the
factor structure difference. Regardless, it was critical to use a
factor structure that best fit the data at hand to ensure more
valid results from additional analyses. Our work demonstrates
that the I-SEE survey can be used as a starting point to evaluate the expectations and experiences of inpatient staff in an
organization transitioning to from a hybrid electronic/paper
HIT care delivery system to a full-fledged EHR, but that additional factor analyses should be conducted to ensure the fit of
the factor structure to the population being examined.
The mean scores for each scale showed a neutral to positive trend (i.e., improved perceptions) across time periods,
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
indicating that expectations of staff were that the imminent implementation would not worsen any of the factors
measured. Nursing responses (Table 3a) indicated that their
experiences with the transition were more favorable than their
expectations, more so than was expressed from the organization as a whole (Table 3b) as six out of seven scale scores
statistically improved over time. Communication scores did
not improve with time for the nurses or for all respondents
(nurses plus others). This reflects the perception that implementing the EHR did not, in their opinion, influence the
various aspects of communication (e.g., with patients, with
colleagues, etc.). Future HIT optimization and implementation
strategies at our institution should consider these findings and
proactively demonstrate how EHRs can enhance communication.
The fact that the scores trended, for the most part, in the
positive direction from T1 to T2 indicates that the transition
to a full EHR with a new CPOE system was less disruptive
than the staff had anticipated. We were actually surprised
that the means at T1 were as high as they were given that
the survey was administered only 5 days before implementation, and many users were stressed (based on numerous
comments offered in the open-ended question) as staff were
beginning to understand the challenges of the HIT transition. We feel that the high regard from the staff was due to
our organizational communication and transparency strategy effectively managing expectations. An a priori governance
and communication framework consisting of a multidisciplinary membership (including clinical leaders, frontline
care providers, executives, and vendor representatives) facilitated a real-time mechanism for bidirectional communication
(before, during, and after implementation) as well as acting
as a conduit for the escalation of safety concerns from any
staff member. As a result, there were positive remarks on our
survey that reflected staff understanding the reasons behind
the decision to implement a new EHR and, overall, did not
anticipate long-term negative consequences. This contrast is
important to note as implementation leaders are often selectively exposed to negative commentary without the benefit
of seeing positive data such as the survey results we present
here.
In comparing the survey results from nurses against all
other providers, the nurses tended to be more concerned (less
positive ratings) about the impact of the transition when compared to all other groups combined (Table 4). At survey time
T2 , (Table 4b) they reported similar experiences to the other
staff members, demonstrating that the gap had narrowed
with time and that their experiences with the implementation were very similar to that of the other care providers.
One reason that the disparity in expectations may exist is
that nurses are typically heavy users of EHRs and may have
heightened fears that the implementation would affect them
more than other care providers perceived themselves being
affected, especially in the few days preceding implementation
[33,34]. In addition, all of the survey questions can be seen as
having direct relevance to a typical nursing role, whereas some
items may have had less practical meaning to some of the
other care providers that were included in the survey (such as
Job Satisfaction question 1 regarding time spent on discharge
documentation).
1043
A direct comparison of our results to previous literature
is difficult given the variety of variables to consider; provider
role, sophistication of the technology implemented, inpatient
versus outpatient care, transition from paper-based systems
to EHRs versus transition between EHR systems, etc. Nonetheless, there were both similarities and differences noted. Much
of what is known about nursing provider experiences is
based on the Stronge–Brodt questionnaire, which contains
fewer (and different) factors than the I-SEE survey [3,32]. Prior
research has demonstrated mixed results regarding expectations and attitudes toward EHRs by nursing staff, with Sultana
finding “unfavourable” responses and Simpson et al. shared
more positive results [4,6]. The latter study also highlighted
that nursing responses may be changing over time and underscores the importance of studying the effects of transitioning
between EHR systems. Additionally, Kossman et al. found that
nurses thought that patient safety had increased because of an
EHR implementation, but at the cost of quality of care [33,34].
In our study, nurses responded similarly to both quality and
patient safety questions, which resulted in the combination
of questions into one factor that was reliably rated in a positive manner. Previous literature on physician attitudes and
experiences was more limited than that available for nursing staff, and often evaluated computer skills and knowledge
[5,10]. Most physicians reported optimism when asked about
the effects of computerized records, which is congruent with
the findings presented in this survey. Finally, even less data is
available about the perceptions of healthcare providers during
EHR transition periods [28–30]. However, our results align well
with previous findings in the ambulatory setting that medical
staff are, on the whole, satisfied with the migration from one
EHR to another.
On the whole, our survey results demonstrated that the
organizational expectations for transitioning from a hybrid
paper/electronic HIT system to a fully functional EHR were
positive and that the organization met and even staff
exceeded expectations (for the items surveyed) 1 year postimplementation. These findings may be indicative of both
the implementation strategy and our organizational culture.
Specifically, the positive findings may be due to staff having
had experience with the CPOE system and with HIT in general,
therefore mitigating potential disconfirmed expectations and
negative rating trends, as seen in earlier studies [11]. Also, we
adhered to known informatics best practices for implementation by involving users in the design process, offering strong
support services (including just-in-time, at-the-elbow support
and a responsive call center), contingency planning, close
oversight and monitoring, and robust, accessible channels of
communication from the microsystem to organizational level
[13,15,12].
4.1.
Limitations
There are several limitations to this study. The T1 survey
administration was only 5 days prior to the new EHR rollout. As such, those responding may have been particularly
motivated to use the survey opportunity to share their concerns. However, even during a time of great stress, ratings
were positive. Unfortunately our response rates were fairly
low; however the distribution across roles and the variation in
1044
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
responses indicates that the data represent a broad spectrum
of opinions. Regardless, the findings and conclusions should
be interpreted with these low response rates in mind, as it
is possible that our findings would not have been as positive
had more staff participated. Future surveys, at our institution and others, should consider opportunities to incentivize
potential respondents, to encourage more robust participation. Additionally, another limitation is the possibility that a
small portion of responders had some exposure to the new
EHR in the outpatient environments where it had already been
implemented. This contamination was unavoidable, and likely
of minimal impact given the implementation schedule.
4.2.
Future studies
Future surveys about healthcare provider’s attitudes, expectations, and perceptions toward HIT will continue to be
informative as technology advances over time and people
become more computer-savvy. As institutions transition from
one HIT system to another, further data on user expectations and experience will offer insight into if, and how, best
practices differ from transitioning from paper to an electronic
system. Finally, studies that examine the period immediately
after implementation could shed light on how user opinions
shift during this time and will serve to inform and refine best
practices after a HIT go-live.
5.
Summary points
What was already known on the topic
• Healthcare providers generally have positive attitudes
and perceptions in regards to the potential of electronic health records to improve the Quality and Safety
of Patient Care.
• Despite this, many providers still have concerns about
the unintended consequences of implementation of
healthcare information technology.
• Little is known about the perceptions and experiences
of inpatient care providers in healthcare organizations
transitioning from a computerized/hybrid system to a
fully electronic health record.
What this study added to our knowledge
• Inpatient care providers at a large pediatric institution
reported positive perceptions prior to and 1 year after
a transition from a hybrid electronic health record to
a fully electronic system, with improvements in most
categories surveyed.
• The I-SEE survey, with some modifications, demonstrated utility in surveying all care providers about the
Quality and Safety of Patient Care during the transition
period.
Conclusions
Surveying healthcare staff expectations and experiences
while transitioning across healthcare delivery information
systems is informative in understanding the organizational
milieu during this time period and in targeting optimization
strategies. This study demonstrates the utility and validity of the I-SEE survey in measuring the expectations and
experiences of both nursing and non-nursing personnel in a
pediatric tertiary care institution. The resulting factor structure of the survey was similar to the original factor structure,
but did exhibit some differences, which made it critical for
us to use the new scales with our population. As such, other
institutions applying the instrument should strongly consider
repeating the factor analysis. Baseline expectations at our
institution were positive for all groups and experience scores
indicated that, for the most part, they improved at 1 year postimplementation.
Author’s contributions
All authors of this manuscript contributed to the (1) conception and design of the study, or acquisition of data or analysis
and interpretation of data, (2) drafting of the article or revising it critically for important intellectual content, and (3) final
approval of the version to be submitted.
organizations that could inappropriately influence (bias) this
work, including but not limited to the following: employment,
consultancies, stock ownership, honoraria, paid expert testimony, or patent applications/registrations.
Funding source
This project required no funding, either internal or external to
the organization/site of study.
Acknowledgements
The authors would like to acknowledge Mary Baggett for her
efforts in the statistical analysis of the data and for assistance
in editing the data analysis section of the manuscript.
Appendix A. Supplementary data
Supplementary data associated with this article can be
found, in the online version, at http://dx.doi.org/10.1016/
j.ijmedinf.2013.08.005.
references
Conflict of interest statement
No authors or contributors to this manuscript have any
financial or personal relationships with other people or
[1] R. Scarpa, S.C. Smeltzer, B. Jasion, Attitudes of nurses toward
computerization: a replication, Comput. Nurs. 10 (2) (1992)
72–80.
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 8 2 ( 2 0 1 3 ) 1037–1045
[2] A.H. Stockton, M.P. Verhey, A psychometric examination of
the Stronge–Brodt Nurses’ Attitudes Toward Computers
Questionnaire, Comput. Nurs. 13 (3) (1995) 109–113.
[3] J.H. Stronge, A. Brodt, Assessment of nurses’ attitudes
toward computerization, Comput. Nurs. 3 (4) (1985) 154–158.
[4] N. Sultana, Nurses’ attitudes towards computerization in
clinical practice, J. Adv. Nurs. 15 (6) (1990) 696–702.
[5] R.D. Cork, W.M. Detmer, C.P. Friedman, Development and
initial validation of an instrument to measure physicians’
use of, knowledge about, and attitudes toward computers, J.
Am. Med. Inform. Assoc. 5 (2) (1998) 164–176.
[6] G. Simpson, M. Kenrick, Nurses’ attitudes toward
computerization in clinical practice in a British general
hospital, Comput. Nurs. 15 (1) (1997) 37–42.
[7] J.M. Melhorn, W.K. Legler, G.M. Clark, Current attitudes of
medical personnel toward computers, Comput. Biomed. Res.
12 (4) (1979) 327–334.
[8] T.S. Startsman, R.E. Robinson, The attitudes of medical and
paramedical personnel toward computers, Comput. Biomed.
Res. 5 (3) (1972) 218–227.
[9] J.G. Anderson, S.J. Jay, H.M. Schweer, M.M. Anderson, Why
doctors don’t use computers: some empirical findings, J. R.
Soc. Med. 79 (3) (1986) 142–144.
[10] D. Thomas, A. Kushniruk, J. Kannry, Housestaff and
attending physician knowledge of and attitude towards an
EMR on the eve of implementation, in: AMIA Annu. Symp.
Proc., 2007, p. 1133.
[11] R.D. Henderson, F.P. Deane, User expectations and
perceptions of a patient management information system,
Comput. Nurs. 14 (3) (1996) 188–193.
[12] N.M. Lorenzi, R.T. Riley, Managing change: an overview, J.
Am. Med. Inform. Assoc. 7 (2) (2000) 116–124.
[13] K.G. Adler, How to successfully navigate your EHR
implementation, Fam. Pract. Manag. 14 (2) (2007) 33–39.
[14] Top 10 factors for successful EHR implementation, http://
healthcareitnews.com/news/top-10-factors-successful-ehrimplementation
[15] N.M. Lorenzi, A. Kouroubali, D.E. Detmer, M. Bloomrosen,
How to successfully select and implement electronic health
records (EHR) in small ambulatory practice settings, BMC
Med. Inform. Decis. Mak. 9 (2009) 15.
[16] P.A. Beiter, J. Sorscher, C.J. Henderson, M. Talen, Do
electronic medical record (EMR) demonstrations change
attitudes, knowledge, skills or needs? Inform. Prim. Care 16
(3) (2008) 221–227.
[17] E.M. Campbell, D.F. Sittig, J.S. Ash, K.P. Guappone, R.H.
Dykstra, Types of unintended consequences related to
computerized provider order entry, J. Am. Med. Inform.
Assoc. 13 (5) (2006) 547–556.
[18] N.M. Lorenzi, R.T. Riley, N.A. Dewan, Barriers and resistance
to informatics in behavioral health, Stud. Health Technol.
Inform. 84 (Pt 2) (2001) 1301–1304.
[19] S. McLane, Designing an EMR planning process based on
staff attitudes toward and opinions about computers in
healthcare, Comput. Inform. Nurs. 23 (2) (2005) 85–92.
1045
[20] D.B. Hier, A. Rothschild, A. LeMaistre, J. Keeler, Differing
faculty and housestaff acceptance of an electronic health
record, Int. J. Med. Inform. 74 (7–8) (2005)
657–662.
[21] W. Ventres, S. Kooienga, N. Vuckovic, R. Marlin, P. Nygren, V.
Stewart, Physicians, patients, and the electronic health
record: an ethnographic analysis, Ann. Fam. Med. 4 (2) (2006)
124–131.
[22] R. Frankel, A. Altschuler, S. George, J. Kinsman, H. Jimison,
N.R. Robertson, J. Hsu, Effects of exam-room computing on
clinician–patient communication: a longitudinal qualitative
study, J. Gen. Intern. Med. 20 (8) (2005) 677–682.
[23] E. Toll, A piece of my mind. The cost of technology, JAMA 307
(23) (2012) 2497–2498.
[24] G. Pare, C. Sicotte, H. Jacques, The effects of creating
psychological ownership on physicians’ acceptance of
clinical information systems, J. Am. Med. Inform. Assoc. 13
(2) (2006) 197–205.
[25] V. Castillo, A. Martínez-García, J. Pulido, A knowledge-based
taxonomy of critical factors for adopting electronic health
record systems by physicians: a systematic literature review,
BMC Med. Inform. Decis. Mak. 10 (2010) 60.
[26] M.V. Bloom, M.K. Huntington, Faculty, resident, and clinic
staff’s evaluation of the effects of EHR implementation, Fam.
Med. 42 (8) (2010) 562–566.
[27] Health information technology: initial set of standards,
implementation specifications, and certification criteria for
electronic health record technology. Final rule, Fed. Reg. 75
(144) (2010) 44589–44654.
[28] E.L. Abramson, S. Malhotra, K. Fischer, A. Edwards, E.R. Pfoh,
S.N. Osorio, A. Cheriff, R. Kaushal, Transitioning between
electronic health records: effects on ambulatory prescribing
safety, J. Gen. Intern. Med. 26 (8) (2011) 868–874.
[29] E.R. Pfoh, E. Abramson, S. Zandieh, A. Edwards, R. Kaushal,
Satisfaction after the transition between electronic health
record systems at six ambulatory practices, J. Eval. Clin.
Pract. 18 (2011) 1133–1139.
[30] S.O. Zandieh, E.L. Abramson, E.R. Pfoh, K. Yoon-Flannery, A.
Edwards, R. Kaushal, Transitioning between ambulatory
EHRs: a study of practitioners’ perspectives, J. Am. Med.
Inform. Assoc. 19 (2011) 401–406.
[31] C. Sicotte, G. Pare, M.P. Moreault, A. Lemay, L. Valiquette, J.
Barkun, Replacing an inpatient electronic medical record.
Lessons learned from user satisfaction with the former
system, Methods Inf. Med. 48 (1) (2009) 92–100.
[32] D.S. Wakefield, J.R. Halbesleben, M.M. Ward, Q. Qiu, J. Brokel,
D. Crandall, Development of a measure of clinical
information systems expectations and experiences, Med.
Care 45 (9) (2007) 884–890.
[33] S.P. Kossman, S.L. Scheidenhelm, Nurses’ perceptions of the
impact of electronic health records on work and patient
outcomes, Comput. Inform. Nurs. 26 (2) (2008) 69–77.
[34] S.P. Kossman, Perceptions of impact of electronic health
records on nurses’ work, Stud. Health Technol. Inform. 122
(2006) 337–341.