SACHE Workshop
September, 2005
Effective Incident
Investigation
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
•
On February 1, 2003 the Space Shuttle Columbia was
returning from space.
•
Listening to the radio in my car and heard that NASA
Houston had lost contact with the Columbia.
•
Needless to say I got a very queasy feeling in my stomach.
•
Being a technical person I knew what that meant.
•
Still kept hoping for a miracle.
•
Finally NASA Houston declared a Contingency
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
•
What is a Contingency?
•
Start of an investigation
•
Impact on the Flight Controllers in Houston
» Ensure their logs are up to date
» Keep hands off all switches, knobs and controller
» Computer data was impounded
•
The same thing occurred at the Kennedy Space Center
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
•
Later declared an Expanded Contingency
» All data at the suppliers to NASA impounded.
•
Shuttle parts recovery
 Problem was public picking up the pieces
 NASA quickly made the point that this was illegal and
mobilized the personnel needed for the task.
 Got public participation in locating the debris but leaving it
in place.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Process Plant Incident Investigation.
• Why?
» Incident Investigation to determine the Root Cause
» Litigation Support
» Lack of Precise Information
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
•
Focus will be on a large incident
•
However scale this back to meet the event you have to deal
with
•
Goal is to always have the information to conclusively
determine the route cause.
•
If there were fatalities; litigation will follow.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• The Scene
» Plant damaged due to explosion and fire
» Control Room damaged
» Fragments are inside and outside the facility boundaries.
» Unfortunately have a fatality and multiple injuries.
» Emergency response actions have been completed
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• The plant has to take the lead until the
Incident Investigation Team can arrive on the
site and get organized.
• ACTIONS REQUIRED
» Rope off the area of the incident including the debris
field.
» Get short statement from the operating and Emergency
Response personnel involved.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
ACTIONS REQUIRED
» Protect records from damage by the weather.
» Prepare an area for the corporate investigation team to
work.
» Nominate one person from plant management and one
operator minimum to participate in the investigation.
» Prepare areas for OSHA and the CSB to work but away
from the investigation team
» Arrange to provide the investigation team legal counsel.
» The balance of the team has to arrive on site in < 24 hrs.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
•
Investigating Team Arrives on Site
•
Plant Responsibilities
•
Immediate Investigation Team Goals
» Organize
» Take Control
» Start work
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Plant Responsibilities to Investigation Team
Members
•
Plant Orientation – processes, hazards, etc.
•
Safety Equipment
•
ID Cards and Plant Access
•
Must be done quickly
•
Accounting
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Team Requirements at Incident Site
• Large meeting room with tables and chairs
• Computers/printers/copy machine(s)
• Access to plant network
• Rooms for interviews
• Recording Blackboards
• Refreshments – coffee, sodas, snacks etc.
• Lunch Service
• Unrestricted telephone access on several lines.
• Stationery supplies
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Process Information
• Mechanical
• Sampling
• Interviews
• Special Assignments
• Secretarial Support
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Who should be on the team?
•
Not someone who would have a conflict of interest
• But:
» Process Engineers
» Mechanical/Maintenance Engineers
» Operations Personnel – Supervisors, Foreman, Operators
» Chemists – QC, Analytical, R & D
» Control/Instrumentation Engineers
» Outside Consultants
» Legal Counsel
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
•
Process Information
» Operating data from the Control Room
» QC data from Control Room and Lab
» Maintenance information from the Control Room
» Engineering records
» Process shutdown check
» Construct the incident time line and analyze
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Mechanical
» Failed Equipment
 Recover the pieces and then reconstruct
» Process equipment
 Protocol on how equipment/instruments will be
disassembled
 Storage location
 Equipment/instruments to be disassembled and
inspected or tested.
 Coordination with the Sampling & Process Team
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
Sampling
Sampling Protocol including chain of custody
Hazards/PPE required
Storage for samples.
Sample material left in process lines or equipment
Coordinate with mechanical
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization, Sampling
cont.
• Sample Size
Take one large sample then divide to provide all the needed
samples.
• Sample analysis
On site, Research lab, Contract lab?
• Sample shipping
Hazardous materials, need expertise
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Interviewing
» Identify people to be interviewed
» Read the initial statements
» Develop a list of questions
» Develop an interview schedule.
» Will take longer than expected.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Special Assignments
» Coordinating with the Regulators
» Keeping upper management informed
» Keeping other stake holders informed
 Others operating the process
 Customers
» Coordinating plant issues
 Release of the area
 Cause of the accident
 Meeting the teams needs
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Team Organization
• Daily Team Meetings:
• Morning: Groups tell the team what their
plans are for the day.
• Afternoon: Groups report to the team what
they have accomplished and new information.
• Distribute/circulate completed interviews to
the team.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Basic Process Control System (BPCS)
» Questions:
 Can the information be retrieved?
 What were the control and alarm points?
 What were the emergency shutdown conditions?
 Actual software in the system?
 How is the operating data recorded?
 Alarm record?
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• BPCS Continued
» Answers needed about the BPCS
 Could a software error or corrupted software have
caused the incident?
 Was the software in the system what was supposed to
be there?
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Interview Process
• Initial statement: KEEP IT SHORT!
» Where were you when the incident happened?
» Who were you with?
» What you saw, heard, smelled or felt?
» Time you think the incident occurred?
» What you think might have caused it?
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Interview Scheduling
» Allow time for the interview
» Allow time for the write-up after an interview completed.
» Keep people to be interviewed from sitting in a waiting
room with others.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Interviewing Techniques
» Neutral and comfortable location
» Two interviewers – one who is familiar to the person
» During interview one asks the questions and the other
records the answers.
» Sit in circle
» Explain the interview process and confidentiality
» Final Question: “Do you have anything to add or tell us?”
» Review notes to ensure no misunderstanding
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Witness Questioning
» Start with personal background information
 Questions about the individual
 Questions about family
 Employment History
» Standard prepared questions
» Witness tailored questions
» Final Question: “Do you have anything to add or tell us?”
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• After the Interview
» Write-up the interview
» Allow the witness to review, correct, or add more
» Share the interview with the entire team
» Some contradictions are to be expected
» Arrange follow-up interview if necessary
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Dealing with Regulators - Interviews
» May want to interview witnesses before you.
» Attend
» Take notes
» Use the notes as input for the team interview .
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Dealing with Regulators – Evidence
» Their interest is to keep the site intact for their
investigation.
» Will need an attorney present when you deal with them.
» Will have to get them to accept your protocols for
evidence collection.
» What information do they want?
» What information might they have that you want?
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Special Challenges
• Photography
•
Using film
•
Digital technology
•
Photographic log book
» Roll number – memory card number
» Frame number – date and time
» Item photographed
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Get the data correctly and completely
• Resist the temptation to jump to a quick
conclusion or root cause.
• Stay open minded and only rule out a
possible cause that the data does not
support.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Determination of the Root Cause
• Investigation team brainstorming sessions
• Develop a Fault Tree (Qualitative)
• Top event is the accident
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• All possible causes that could lead to the top
event are listed.
• Then work on these events, always going
deeper, keep asking how or why? Identify
other conditions that have to occur to make
the cause feasible.
• Remember: “The only dumb question is the
one not asked.”
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Why do this?
•
May identify other information needs that have been
overlooked.
•
Identify additional questions for interviews
•
Calculation that may be needed or experiments that should
be made.
•
Can include management systems and human factor issues
in the Fault Tree.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
Fault Tree Events
• AND
» These are two events that would have to occur
simultaneously to make the above event occur.
• OR
» If either of two or more events or conditions occur
the above event could occur.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• Use of the Fault Tree
» Hang the Fault Tree on a large piece of paper in
the room
» As evidence comes in mark it on the tree to
indicate what is true and not true.
» This will allow determination of the Root Cause
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.
SACHE Workshop
September, 2005
• In summary an Incident Investigation is like a
project:
• It requires organization and planning
• To get it right you have to pay attention to the
details.
• Have to always question what is found or
learned.
Proprietary and Confidential. © 2004 ioMosaic Corporation. All rights reserved.