Software Change Management at Mayo Clinic
Steve Pleschourt
Soft Senior System Manager
Mayo Clinic
Policy
Mayo Clinic Information Technology
changes will be managed and
communicated using change
management processes to ensure
systems integrity and acceptable risks
while supporting Mayo’s business and
practice needs.
4 “Take Aways”
• Determine what things require change
management
• Mechanism to receive and process requests
– Operational
• Routine requests (tests changes, printers, workstations,
etc.)
• Troubleshooting
– Project
• Coordinating delivery to non-PROD
• Implementation in PROD
The Mayo landscape picture...
• What change management practices need to be in
place to ensure stability in a system that supports:
–
–
–
–
–
500K tests performed per week
35 million tests per year
>1200 concurrent users and >3000 total users
90+ labs
4 primary sites in 3 time zones
• MCR/NEL/MCA/MCF
– Multiple sites in MCHS
– 11+ environments
– Multiple projects always in progress
Painting the picture...
“TWO WORLDS OF MAYO”
• Operational
– File Def
• Test Def
• Comments
– MDM
– Error/problem resolution
– Disposition meetings with
Soft
– Hosparam and config
changes
– Scheduled and unscheduled
impairments
– System maintenance
– SCRs
• Projects
– Multiple projects in flight
at the same time
– Adding modules
– Adding/modifying
interfaces
– Version upgrades and
maintenance releases
– Adding functionality to
existing modules
– Bringing up new
sites/clients
Painting the picture...
Soft Support Team (operational)
• 16 System Managers
• ~10 SQA Testers
– Another 25 that also assist
•
•
•
•
•
•
~6 Tech Specialists and DBA support
~16 System Analysts (Test builders)
~8 System Analysts (HelpDesk)
~20 Business Analysts
~10 upper management staff
TOTAL: 80-100 people supporting Soft modules
– Projects interfaces, and other system interactions add another 50100 staff that are touching or affecting Soft
Support Team
• System Managers
– 2 SoftLab
– 5 Gene
• BC/FL/Cyto/Mol/PDx
–
–
–
–
–
–
–
1 RnV/CSM
1 SoftID
1 BB
1 Mic
1 TQC
1 SEC/AR
1 Senior System Manager
Take Aways
• Determine what things require change
management
• Mechanism to receive and process requests
– Operational
• Routine requests (tests changes, printers, workstations, etc.)
• Troubleshooting
– Project
• Coordinating delivery to non-PROD
• Implementation in PROD
What things require change management?
Two types of changes
• Things the client can do
• Things Soft must do
What things require change management?
(Things the client can do)
• Everyone needs a “Who Does What” SOP
PURPOSE
A controlled list that identifies who is responsible (by roles)
for a specific definition and the process to request a change
What things require change management?
(Things the client can do)
Suggestions:
• Important to set up security to ensure the correct
people are making the changes
– Who has authority to make the change in non-PROD and
who can make the change in PROD?
– Master Data Management (MDM) vs Routine
• Each SOP should outline the approval process that is
to be followed to implement the change
What things require change management?
(Things the client can do)
What things require change management?
(Things the client can do)
What things require change management?
(Things the client can do)
What things require change management?
(Things the client can do)
What things require change management?
(Things the client can do)
What things require change management?
(Things the client can do)
What things require change management
(Things Soft must do)
• Hosparams
– High level client specific options that alter functionality of how
the system works (ie: interfaces, some layout, system level)
• Settings & Definitions (Configuration)
– Client specific options that alter functionality of how the system
or modules work (ie: columns, buttons appear or disappear,
templates, module specific)
– Some of these clients can change and some Soft must change
• Code installs (HF, version upgrade, scripts)
– Scheduled
– Emergency
• Environment maintenance (Hardware upgrades)
Take Aways
• Determine what things require change management
• Mechanism to receive and process requests
– Operational
• Routine requests (tests changes, printers, workstations, etc.)
• Support/Troubleshooting
– Project
• Coordinating delivery to non-PROD
• Implementation in PROD
Mechanism to receive and process requests
(Operational)
Routine requests (the “Who Does What” stuff)
Preferred intake mechanism
• Electronic submission process to request new tests, or
changes to existing tests/messages/printers/workstations/
etc
• System Analysts follow a well defined test
implementation and support process to validate changes
with the laboratory before promoting to PROD
• The changes are documented and
implemented with very little higher
level review/approval
Mechanism to receive and process requests
(Operational)
Support/Troubleshooting
Preferred intake mechanisms:
• Call HelpDesk (ticket created immediately)
• Web submission request to HelpDesk (ticket created
immediately)
– Ticket provides a common repository for recording
• Details/examples
• Latest correspondence
• Status
Mechanism to receive and process requests
(Operational)
Support/Troubleshooting
Preferred intake mechanisms:
• Electronic web submission
• System Analyst/System Manager
investigates the issue
• Determination is made that it requires some type of
investigation and/or delivery from Soft to resolve
• Ticket synched to Soft TMS
Mechanism to receive and process requests
(Operational)
System level operational changes requests
Not Preferred avenues to submit requests:
• Call or email your favorite person
– Emails often have many trails
– Lose traceability
– Attachments lost
– No common repository readily available by many
– Does not allow management reports to be
compiled
• Often create confusion and lost time
Mechanism to receive and process requests
(Project)
• Projects: >160 hrs of effort
• Before starting a project, it must be submitted to a
review group for
– Approval/rejection
– Prioritization
– Resource assignment
• After projects start, they basically need to follow
similar rules as the operational support requests to
make changes
Mechanism to receive and process requests
(Operational & Project)
So now you have a ticket in your system.
Good for you… That’s a start…
Mechanism to receive and process requests
(Operational & Project)
So now you have a ticket in your system.
Good for you… That’s a start…
Now what do you do with it?
Mechanism to receive and process requests
(Operational & Project)
So now you have a ticket in your system.
Good for you… That’s a start…
Now what do you do with it?
1. Cry
Mechanism to receive and process requests
(Operational & Project)
So now you have a ticket in your system.
Good for you… That’s a start…
Now what do you do with it?
1. Cry
2. Ignore it
Mechanism to receive and process requests
(Operational & Project)
So now you have a ticket in your system.
Good for you… That’s a start…
Now what do you do with it?
1. Cry
2. Ignore it
3. Analyze
Mechanism to receive and process requests
(Operational)
Processing (analyzing) tickets
1) Correspondence with Soft through TMS to get the issue
resolved
PROS
– Easy issues can be resolved fairly quickly
– Very effective in some situations
– Provides excellent documentation and traceability of
all interactions between client and Soft
CONS
-Can be slow
-Not always easy to interpret written communications
-Not good for complex issues
Mechanism to receive and process requests
(Operational)
Processing (analyzing) tickets
2) Disposition meetings (when TMS is not enough)
– Recurring conference calls between Soft and Mayo
SMs and SQA
– More complex issues discussed
– Frequency is dependent on module
– IMPORTANT: Get an agenda sent
ahead of time!!
• Soft needs time to prepare
• Don’t want to waste their time
Take Aways
• Determine what things require change management
• Mechanism to receive and process requests
– Operational
• Routine requests (tests changes, printers, workstations, etc.)
• Troubleshooting
– Project
• Coordinating delivery to non-PROD
• Implementation in PROD
Coordinating delivery to non-PROD
So…
1. You’ve done the analysis and submitted your
requirements
2. Soft has done the coding and wants to deliver
the fix to your world
You think you have the hard part done...
Coordinating delivery to non-PROD
• What happens next can be a real eye-opener!!
Coordinating delivery to non-PROD
•
•
•
•
•
•
•
Which of your environments should it be loaded to?
When should it be loaded?
Is the load silent or will it require an outage?
How long of an outage is needed?
Who will be impacted by the downtime?
Who do I need to communicate the delivery to?
Does Soft have ability to load on the
day/time we want?
Coordinating delivery to non-PROD
•
•
•
•
•
•
Will all modules be down or just some?
Will interfaces or system need to be cycled?
Who is going to do smoke testing?
When will we get release notes?
Are there any environment dependencies for the load?
What test definition and interfaces are needed in the
environment to perform testing?
Coordinating delivery to non-PROD
Configuration Management meetings
• Recurring meeting with Soft and Mayo resources to
review and plan routine or emergency changes to the
system
• Things that come to CM:
–
–
–
–
Hosparam changes
Settings and Definition changes
Requests to run scripts
Requests to change trace files
Coordinating delivery to non-PROD
Configuration Management meetings
• Documentation of discussions at this meeting are
entered in the ticket and synched to Soft TMS:
Approval granted to install hosparam
XYZ in the R3Q45 environment on
July 3, 2015.
NOTE: Critical to include approval,
when, and where to load in the ticket.
(System records who/when note entered.)
Coordinating delivery to non-PROD
So now you have granted approval and gotten it
scheduled and also communicated to your users.
What do you do to control access to the system by the
vendor and keep your users out of the system during the
install and smoke testing?
Coordinating delivery to non-PROD
Mayo does not want to allow vendors to have unlimited
access to their systems and environments.
Coordinating delivery to non-PROD
Mayo IT and IM created UNIX based functionality to
control access to environments. The process is referred
to as Break The Glass (BTG).
Coordinating delivery to non-PROD
BTG: Break The Glass
• BTG functionality allows Mayo IM staff to open or
close access to outsiders
• Not all environments have BTG
– PROD and highest level Non-PROD have BTG
– Lower level Non-PROD environments do not
• Requires entry of ticket # to refer to why BTG is
being performed
• Manually Broken and Repaired
Coordinating delivery to non-PROD
When Soft returns the system to the client, how do you
keep impatient users out while you perform smoke
testing?
Coordinating delivery to non-PROD
Mayo also uses environment password protection
functionality that Soft provides.
Coordinating delivery to non-PROD
Password protection
• Client personnel who have access to the UNIX
operating system can set a password for any
environment
• Password can be different for each environment
• The password locks out all users, including Soft
• If you use password protection, you need to provide
the password to both Soft and client staff who will be
doing the load and smoke testing
Coordinating delivery to non-PROD
Password protection
• When Soft is done making their changes, leave the
password protection on while IM staff do smoke
testing
• Remove password protection when you notify users
the outage is complete
Coordinating delivery to non-PROD
CM covers the daily things that usually do not involve
outages, but does not cover the higher level
environment planning needed for our 11 environments.
There were >120 scheduled installs
done at Mayo in the last 12 months.
We had to come up with something
else to help coordinate our
environment needs.
Coordinating delivery to non-PROD
Soft>Mayo Install meeting
• A weekly meeting with Soft to review install schedule
• Mayo Project Managers and management staff attend
• Review what happened last week, and look ahead to
what is happening the next few weeks
• Try to stay at least 1-2 months ahead
• Send communication to all people working in the
environments
– Many people depend on this communication to
schedule training and verification/validation activities
Coordinating delivery to non-PROD
Coordinating delivery to non-PROD
• Once the item is delivered
it needs to be tested
• SMs and SQA review tickets and RNs to determine
how to test
Coordinating delivery to non-PROD
• If issue can not be tested or reproduced, regression
testing is usually performed to ensure associated
functionality was not impacted
• Pass/Fail is entered in the ticket
• Passed tickets are ready for
promotion to PROD
•
Failed tickets go back for
further analysis
Take Aways
• Determine what things require change management
• Mechanism to receive and process requests
– Operational
• Routine requests (tests changes, printers, workstations, etc.)
• Troubleshooting
– Project
• Coordinating delivery to non-PROD
• Implementation in PROD
Implementation to PROD
Implementation to PROD
1. Testing in non-PROD is complete.
– Everything passed, or all issues are considered to be
“acceptable risk”
2. Install date/time to PROD is agreed upon
between Soft/client
3. Pre-activation meeting is held to review
implementation plan
4. All approvals to proceed have been obtained
– Documentation entered in ticket and synched to TMS
5. Notification is sent to users based on timeline in
Service Level Agreement
Implementation to PROD
6. BTG has been scheduled
7. Passwords have been shared with Soft and
smoke testers
8. Soft installs code and notifies client their
smoke testing is complete
9. Client performs smoke testing
10.Password is lifted and notification is sent to
users that the system is available
Implementation to PROD
Implementation to PROD