Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Training presentation for:
AHCA Florida 2016 Annual Conference
Finding & Managing
Power System
Vulnerabilities
David Stymiest, CHFM, CHSP, FASHE, (PE in LA MS MA)
[email protected]
Copyright © 2016, Smith Seckman Reid, Inc.; All rights reserved
Major topics
•
•
•
•
Finding common mode failure potential
Improving communication
Planning for different types of failures
Assessing reliability, availability and
dependability
• Importance of inspection, maintenance
& testing
• Lessons learned from natural disasters
and other events
Disclaimer
• Content Disclaimer: These slides are only
meant to be cue points, which were
expounded upon verbally by the original
presenter and are not meant to be
comprehensive statements of requirements or
best practices, nor represent all the content of
the presentation. Thus, care should be
exercised in interpreting content based solely
on the content of these slides.
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
1
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
NFPA Disclaimer
• Although the speaker is on the NFPA Technical
Committee on Emergency Power Supplies,
which is responsible for NFPA 110 and 111, the
views and opinions expressed in this
presentation are purely those of the speaker
and shall not be considered the official position
of NFPA or any of its Technical Committees
and shall not be considered to be, nor be relied
upon as, a Formal Interpretation. Readers are
encouraged to refer to the entire texts of all
referenced documents.
• NFPA members can obtain staff interpretations
of NFPA standards at www.nfpa.org.
A new paradigm
• Reliability
– Probability that system operates and gives the same
result on successive trials
• Availability
– Probability that system will function at any instant
required, including the next instant, and for as long
as required from that point
• Dependability
– Measures availability, reliability & maintenance
support
Lessons from real failures
Lessons from real failures
•
•
•
•
•
•
•
•
Things break
Ask critical questions
Pay attention to the details
Commonalities & history
Common-mode failures
Analyze impact of “what if” scenarios
Importance of testing & maintenance
Comprehensive vulnerability analyses
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
2
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
MORE lessons from real failures
Lessons from real failures
•
•
•
•
•
•
•
•
•
Generators run out of fuel
No cell service to call for assistance
Poor or impossible travel conditions
Cascading system failures
Staff training
FEMA realities during an emergency
Ongoing ITM is very important
You can’t control what you can’t control
The details will get you
Simplified power system
Potential vulnerabilities to consider
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
3
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Managing vulnerabilities
•
•
•
•
•
•
Finding
Prioritizing
Assessing
Reporting
Mitigating
Verifying
Common-mode failures
• Failures of two or more
components or systems due
to a single event or cause
• Identified failure mode can
usually be mitigated by
adding extra or redundant
equipment
• Cannot correct unidentified
failure modes
• Uncorrected common mode
failure removes advantage of
other redundancies
Common-mode failures
• Normal and emergency power equipment on
same level
• Fuel oil transfer pump set subject to flooding
• Common fuel oil transfer pumps, controls,
power circuits
• Feeders for elevated equipment located in
flooded levels
• Contaminated fuel oil system
• Co-location of equipment & systems
• One sump pump or multiple sump pumps on
same panel
• Transfer switch failure
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
4
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Common location / subsystems
Paralleled generator sets
can mitigate the impact
of one generator failure.
Can also be subject to
common- mode failures:
Shared location
Shared fuel system
Shared cooling system
Common distribution
• Paralleling switchgear
• Control power failure
• Internal short circuit (fault)
• Low probability but very high impact
• May become apparent when EPSS is
energized next.
Mitigating vulnerabilities
• All power systems and equipment
•
•
•
•
•
•
Understand
Plan
Inspect
Test
Maintain
Remove
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
5
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Explaining Normal vs. Emergency Power
Red
(Emerg.)
Outlet
Generator is usually off
Emergency
Generator
Explaining Normal vs. Emergency Power
Simplified Emergency Power Supply System
Red
(Emerg.)
Outlet
Generator on
Emergency
Generator
Types of failures
• Normal down with emergency power
working
• 1 emergency power branch down, normal
working, other branches working
• Total electrical failure
– Simultaneously
– Cascading events
• Email speaker to request “teaching slides”
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
6
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Most common generator failures
•
•
•
•
•
•
•
Starting system problems
Fuel oil system problems
Cooling system problems
Installation error / lack of acceptance testing
Inadequate maintenance
Overloads – generators, breakers, fuses
Load shed malfunctions – multiple
generators fail
Other causes of generator failures
• Lightning power surge damages
generator controls
• Generator auxiliaries on normal power
(fans, fuel transfer pumps)
• Failures during routine testing
(thrown engine rod, fuel hose rupture)
• Other mechanical or electrical failures
• Generator breaker trips – lack of
protective coordination
Planning for Internal Failures
•
•
•
•
Contingency planning: details important
Doesn’t need to be long
Just needs to be correct
Email for templates
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
7
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Planning for Internal Failures
• Consider different failure points
• Different responses
• Don’t wait until failure occurs
Approach to vulnerabilities
1.
2.
3.
4.
5.
Consider each component
“What if?” scenarios, including feeders
Common-mode failure potential
All possible causes of those scenarios
Address all resulting vulnerabilities
Preparedness for power failures
• Things break
• Sweat small stuff
• Small issues can
take out systems
• Different failures
• When to plan?
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
8
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Vulnerability Analysis: Infrastructure
• Infrastructure design, features, components,
condition, locations, operating flexibility,
spares, maintenance histories, vulnerabilities
– Electrical service and NP distribution
– EPSS and its auxiliary subsystems (FO +)
– ATSs, feeders, branches
– Power system documentation,
labeling, failure procedures,
test results, training
Vulnerability Analysis: Power Sources
• NP system: major distribution
• EP system, Gen, ATS, ATS sources
• Sort infrastructure systems, facility areas and
facility services by each power train
– By each main switchboard, generator, ATS
– Equipment or wiring failures take out redundant
systems / areas / functions?
– Common mode failure vulnerability analysis
Vulnerability Analysis: Areas
• For important functional areas, look at:
– Higher vulnerability from infrastructure
analysis
•
•
•
•
Less dependable equipment
Poorer documentation
Inadequate power failure procedures
Inadequate training
– Higher vulnerability from common mode
failure vulnerability analysis
– All other vulnerability assessment tools
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
9
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
System vulnerabilities: examples
• Inadequate acceptance testing
• Common-mode failure potential
• ATSs not maintained regularly
– Because not bypass-isolation type
• ATSs not transferred every month
• Lack of switchboard / breaker
maintenance
• Lack of EES maintenance
Vulnerability analysis results
• Preparedness
Activities
– Additional capacity
– Emergency equipment
– Identify additional
resources
– More training / testing
– Contingency planning
• Mitigation Activities
– Policies & procedures
– Change in process
– Maintenance program
(Plan electrical
shutdowns)
– Infrastructure
repair/upgrade
– Tighten rooms
– Leak detection
Gap analysis for VULNERABILITIES
• Gap Analysis can also address results of
Vulnerability Analysis
• How vulnerable is EP System to failures?
• How vulnerable is NP System to failures?
• Where are the common-mode failure
vulnerabilities?
• How to mitigate these vulnerabilities
– Short term
– Long term
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
10
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Gap analysis for SUPPLIED SERVICES
• Examples: fuel oil supplier, generator or ATS
service company, spare parts supplier
• Service provider vulnerability or
over-commitment
• Systematically identify gaps between where
the supplier’s crisis
management capabilities
end and your contingency
plans begin.
• Fix them.
PdM Example: Infrared Thermography
• Predictive maintenance (PdM)
vs. calendar-based maintenance (PM)
• Many facilities already scan NP equipment
• Include generator panel, paralleling
switchgear, and transfer switches in IR
scanning scope of work.
• Scan paralleling switchgear
when it is energized.
Some other PdM examples
• Diesel generator fluid testing
–Fuel oil testing is required annually
–BEST PRACTICE – also testing
lubricating oil & cooling water
• Rotating equipment vibration
analysis
• Ultrasonic analysis
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
11
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Electrical room maintenance
• Learn from required weekly EPSS
inspections
• Cleaning rooms: minimize contaminants
finding their way inside the electrical
equipment
• Change the filters
• Inspect for evidence of water
Leak detection in electrical rooms
• Warns of water-based
vulnerabilities when
relocation is not
practical
• Mech/Elect co-locations
• Elevation issues
• External water
• Internal piping leaks
• Broken sumps
Risk assessment considerations
•
•
•
•
Transfer switches maintenance history
Normal power operational history
Normal power maintenance history
User and facility management action plans
•
•
EP system maintenance shutdowns
Thermographic scanning results
•
Infrastructure conditions
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
12
Power System Vulnerabilities : AHCA 2016
Copyright (c) 2016 SSR, Inc. All rights reserved
Competency training for maintainers
• Responses to various internal failures
• Responses to simultaneous multiple
utility failures
• Operation of different equipment, not
just the same equipment every month
• Understand and look for second order
consequences
Thank You!
David Stymiest, P.E., CHFM, CHSP, FASHE
(P.E. in LA, MS, MA)
Senior Consultant
Smith Seckman Reid, Inc.
[email protected]
www.ssr-inc.com
Compliance News archives and sign-up are available at:
http://www.ssr-inc.com/blog/
See Content Disclaimer.
[email protected]
David Stymiest, PE CHFM CHSP FASHE.
13