Case Study
Community Area Network Metro Cluster Technology
Utilizing a Community Area Network (CAN) to store and back up patient data efficiently, two hospitals
implemented compelling, reliable and cost-effective healthcare IT solutions to enhance patient care
Summary: This November 2011 case study examines how a Storage Area Network for each of two hospitals
connected by a redundant Community Area Network provided compelling, reliable and cost-effective storage and
backup of critical patient data with consistent real-time accessibility.
Challenges:
Electronic storage of patient data is the backbone for the full operation of any hospital facility. This
includes both Electronic Medical Records (EMR) and Electronic Health Records (EHR) which are both
critical to tracking patient care. EMR track and store patient information during a hospital admission.
EHR include historical patient data stored following a hospital discharge. This data supports HSHS Care
Integration and provides caregivers a point of reference for diagnosis, treatment and ongoing care. In
order for EMR and EHR to function, real-time 24/7 access to the data is critical. Until the beginning of
2005, the storage and backup of this data was a challenge for both Sacred Heart and St. Joseph’s
Hospitals because their system had no advanced broadband network.
Before the Chippewa Valley Inter-networking Consortium (CINC), the Chippewa Valley’s Community
Area Network (CAN), both hospitals handled their own storage on-site with no replication. Tape storage
was utilized for backup and stored off-site. A magnetic data storage tape, also known as Linear Tape
Open – [version] 4 tape or LTO-4, would store approximately 2 terabytes of data. At the time, Sacred
Heart would store via magnetic tape approximately 1 Terabyte of data per day. St. Joseph’s would store
approximately 400 Gigabytes of data per day as well.
If a disaster happened to the hospital’s network, the backup tapes would need to physically be taken down
to Springfield, Illinois, to the Hospital Sisters Health System headquarters, an approximately seven-hour
drive (430 miles) from central Western Wisconsin. In Springfield, the backup tapes would be recovered
to the servers and allow Sacred Heart or St. Joseph’s access to their data files over a 300 mbps
connection. In order for the hospital or hospitals to be fully functional, this recovery process would take
24 hours, with substantial interruption to patient care and other mission-critical support functions.
The cost of this data backup system was substantial because a Storage Area Network (SAN) was needed
for each hospital (including hardware, servers, and additional support contracts with outside vendors).
There were few alternatives to this backup system because there was no advanced broadband connectivity
between the hospitals. Broadband at 1 to 10 Gb/sec was unavailable at any price for most of the regional
area. Lesser speeds were available from commercial providers, but were prohibitively expensive to
implement in a widely-deployed high speed broadband imaging network and were not redundant and
dependable 24/7. Costs for non-redundant advanced broadband were in the thousands of dollars per
month.
Metro Cluster Solution
A new metro cluster model was a solution that substantially reduced cost and increased reliability. A
metro cluster mirrors or replicates data synchronously.i This allows patient data to be stored on a
database server physically located at one hospital with a synchronous back up copy written in real-time to
another database server within milliseconds at the other hospital. This was made possible by the CINC
network that redundantly connects Sacred Heart and St. Joseph’s hospitals. Redundancy maintains a
simultaneous connection in a community area network (CAN) even if a backhoe or beaver1 cut through
the fiber optic cable in one location. With a break in transmission, data transfer traffic is immediately
rerouted through the redundant connection. This allows for 100% uptime for patient records to be
accessed. In addition, other hospital processes are sustained without losses in network connectivity and in
turn no loss in productivity.
This new model required the redundant advanced broadband infrastructure of a community area network
(CAN). When both hospitals joined the Chippewa Valley Inter-Networking Consortium (CINC), a local
CAN, they worked with other stakeholders to expand the network between Eau Claire and Chippewa
Falls. Today, thanks to CINC and WiscNet, Sacred Heart and St. Joseph’s hospitals enjoy speeds of up to
10 Gbps which made a metro cluster solution possible.
Extensive planning and preparation over a period of 36 months by Sacred Heart Information Systems (IS)
staff and a CDW consultant made this transition possible. It took a month to set up and test the hardware.
Implementation was a complete overhaul from the old backup tape to a metro cluster of mirrored data
storage and backup. However, magnetic tape backup is still done in conjunction with backup to disc to
ensure no loss of critical patient data.
The hardware technology brands being utilized for this cluster are NetApp (Storage Area Network, where
the actual patient data resides), Cisco switches that connect the servers to the data, Hewlett Packard
servers for workstation access to the data and the brocade switches that operate as the primary and
secondary transfer of data for storage between Sacred Heart and St. Joseph’s.
The data being transferred over this connection to both the primary data storage site at the hospital and
then to the backup storage site at the other hospital and visa versa is approximately about one megabyte
per second, six megabytes per minute and one gigabyte every two hours. This averages out to a daily use
of about twelve gigabytes per 24 hour day. These processes being transferred include email, file storage,
EMR storage and other database storage usage, but do not including the Picture Archiving and
Communication System (PACS). The PACS files are not backed up utilizing this system, but are backed
up on their own dedicated magnetic tapes and disks and their saving and retrieval run on dedicated
servers. This system of data backup currently helps both hospitals run more efficiently and save money.
The cost savings include a reduction of overhead costs by 30%. This process eliminates taking magnetic
tape backups of the data offsite and avoids a costly and timely recovery, if ever needed. Over time, the
cost of maintenance and upgrades are reduced since less equipment is needed with a centralized storage
area network (SAN), and less contracts for ongoing support. An integrated resource of IT professionals
from both hospitals are today one department now, supported by a network that runs applications and
backups that are more efficient, more reliable and more cost effective.
Metro Cluster Solution Advantages
The advantages of a metro cluster solution data retrieval and backup include zero downtime and it has
improved overall performance in accessing data. The data servers run redundantly in case of a server
failure, the backup server picks up right where the primary left off. The transfer of server services in
about ten seconds is seamless. Most importantly, this system of data access and backup with server
redundancy is taking place at both hospitals.
1
In the last several years, hospitals in Superior, WI and Duluth, MN, were affected by an outage in a non-redundant fiber optic
cable that ran along a highway which was near a wetland and compromised by beavers.
In the event of a disaster at one of the hospitals, the other hospital would be able to accept and assume the
lead on all mission critical processes including EMR data access. A node, a device on the network which
could be a server, workstation or a computer peripheral, it would recognize a disaster has taken place and
would automatically reroute data traffic on a working route. In an event of a fiber optic connection
failure, IS staff has tested its data transfer rerouting process five times to ensure no loss in data
accessibility. This includes any interruption over the CINC fiber optic network between hospitals and has
been tested five times since its conception.
The SAN (data servers for EMR, EHR) and data storage and backup are all monitored by Information
Systems Network staff on a regular basis. They monitor data storage to make sure it is not being maxed
out in capacity. They receive email alerts to any links in the network they may have been cut, any hard
drive failures, and continually observe daily network usage logs. If a hard drive on a server does fail,
there are backup drives in place to take over in these situations. This ongoing monitoring sustains 100%
up time for data retrieval for normal EMR and EHR processes for any medical and support staff at either
hospital.
Medical staff needs to be able to access patient data for both EMR and EHR all the time. It is imperative
to providing the best possible patient care to every patient every time. Efficient and accurate EMR
records need to be accessible in real-time and be backed up synchronously for accurate data
documentation. When uptime is not 100%, accuracy in patient data is lost since paperwork is more time
consuming and tends not to be recorded when its fresh to a medical staff member. A lack of accurate data
because of loss efficiency can lead to misdiagnosis do to unreliable medical history. Advanced
broadband is an essential tool to providing the best possible care for each and every patient that walks
through the hospital doors.
i
http://it.toolbox.com/wiki/index.php/MetroCluster