Harris Plant - Heater Drain System Transients
INPO ICES #305654
HNP Root Cause Evaluation 592995
Presentation Outline
 System Overview
 System History
 Atypical System Characteristics
 Recent Problems
 Cause(s) of Problems
 Generic Lessons Learned
2
Heater Drain System Schematic – ‘A’ Train
3
Heater Drain System Schematic – ‘A’ Train (Cont’d)
4
System History
 Initial operation experienced significant system instability issues.
 Unstable HD and FRV control valve operation due to excessive pressure drops across the
valves.
 10 plant trips during first five months of power ascension testing and commercial operation,
most of which were caused by BOP problems.
 A Study recommended seven modifications to the BOP. These included:
 Removing an impeller stage from the Heater Drain Pumps.
 Replacing the pneumatic controllers on the #4 FWHs, #5 FWHs, and MSR Drain Tanks
with electronic level controllers.
 Trimming the Main Feedwater Pump impellers.
 The result was significantly more stable and reliable plant operation.
 The modifications continued to work effectively for the next 25 years of operation.
5
Atypical System Characteristics
 No Heater Drain Tank.
 Small level transients often result in a Heater Drain Pump trip.
 Variable speed Condensate Booster Pumps.
 Heater Drain Pump trips have a small impact on overall plant output.
 Make and model of electronic controllers are unique to the Heater Drain System.
 Knowledge of controllers eventually began to fade.
 Knowledgeable personnel (with few exceptions) moved on.
 First 20 years - System largely operated and maintained by tribal knowledge and asneeded corrective maintenance.
6
Key:
Black Font – NCRs that were used for event investigation.
Red Font – Reactivity Management NCRs
Green Font – Quick Hit Self-Assessments
3/31/13
3/17/13
3/3/13
2/17/13
2/3/13
1/20/13
1/6/13
12/23/12
12/9/12
11/25/12
11/11/12
10/28/12
10/14/12
9/30/12
9/16/12
9/2/12
8/19/12
8/5/12
7/22/12
7/8/12
6/24/12
543175
479198
479893
593843
586269
588010 588462
589179 589563
589576
590483
589752 591567
592995
592336 592296
575996
571727 572026
572832 573783 573720 572876
573113
573471573430
573457 573473
567582
565214
562208
554685
551771
545818
542373
533198
518603
511503
503429
503201
491930
479193
479892
440623
572835
December 1, 2010 – March 5, 2013
6/10/12
469571
474208
469850
467287
454157
453181
444537
438110 437986
439729 439828
439824
439774
440479
441720
441734
Overall Timeline of NCRs
5/27/12
5/13/12
4/29/12
4/15/12
4/1/12
3/18/12
3/4/12
2/19/12
2/5/12
1/22/12
1/8/12
12/25/11
12/11/11
11/27/11
11/13/11
10/30/11
10/16/11
10/2/11
9/18/11
9/4/11
8/21/11
8/7/11
7/24/11
7/10/11
6/26/11
6/12/11
5/29/11
5/15/11
5/1/11
4/17/11
4/3/11
3/20/11
3/6/11
2/20/11
2/6/11
1/23/11
1/9/11
12/26/10
12/12/10
11/28/10
11/14/10
Attachment 3
Overall Timeline of NCRs
December 1, 2010 – March 5, 2013
Note: Three Clusters of Events
Note distribution of NCRs. Numbers and dates are expected to be illegible.
7
2010 - 2011 Timeline
Actions taken:
- Functional check of 1HD-17
- Visual instrument inspections
Actions taken:
- Instrument functional checks
- Tightened tubing on LC-1251A
CAUSE NOT IDENTIFIED
CAUSE NOT IDENTIFIED
Actions taken:
- Replaced failed LC-1250A
with controller found in TTF
- Created PM to replace all
electronic LTs and LCs in R17
1/4/11
4A FWH level
transient &
'A' HDP trip
12/25/10
4A FWH level
transient &
'A' HDP trip
12/14/10
4A FWH level
transient
1/2/11
4A FWH level
transient &
'A' HDP trip
12/22/10
4A FWH level
transient
Degraded
controller
installed
Summary of 2010/2011 events:
- 5 transients, 3 HDP trips
- Initiated by equipment failure
- Cause not identified 3 times
- New failure introduced
resulting in 1/4/11 event
Actions taken:
- LC-1251A setpoint adjusted
- Monitor LC-1251A output
CAUSE NOT IDENTIFIED
Replaced
LT-1250A
(cause)
Replaced
LC-1250A
(not refuted)
8
2012 - 2013 Timeline
From 2010/2011 events:
PMR processed to replace electronic
level controllers and transmitters in R17.
LT/LC-1250A was screen out from initial
scope due to replacement on 1/4/11.
Replaced
LS-1251B
Unsuccessful restoration of
4BFWH to 1HD-323 control
(clearance not lifted)
Replaced
LT-1250B
(cause)
Revised
OP-136
Replaced
LC-1250B
(not refuted)
Replaced
LC-1250A
11/15/12
4A FWH level
transient &
'A' HDP trip
11/16/12
Unsuccessful
'A' HDP start
RFO 17
Initial PM due for
replacement of
level controllers
and transmitters
Old controller
LC-1250A not
replaced in R17
OP-136
inadequacy
identified
Defective
controller
installed
Defective
transmitter
installed
2/15/13
4B FWH level
transient
2/27/13
4B FWH level
transient
Replaced
LC-1250B
Replaced
LT-1250B
OP-136 not
revised
2/11/13
4B FWH level
transient
3/2/13
Unsuccessful
4BFWH
restoration
resulting in ES
isolation and
4BFWH level
transient
Obsolete, aged level
transmitters installed
of unknown shelf life
Replace aged controllers and obsolete
transmitters in R18 per WO 2217334
Revised
OP-136
9
Event Type
Not Applicable
2 CRs
8%
Resulting
12 CRs
50%
Initiating
10 CRs
42%
10
11
Plant
Preventative
Maintenance
38%
Design
13%
Part Quality
37%
Setpoint Control
12%
12
People
Risk
Perception
39%
HU Tool
Usage
46%
Troubleshooting
15%
13
Process
Procedure
Quality
36%
Procurement &
Obsolescence
55%
Communication &
Task Preview
9%
14
Root Cause
• Previous actions taken to address Heater Drain System equipment failures, while
appropriate, have not been thorough and intrusive enough to prevent additional failures
and associated reactivity management events.
• Until the most recent series of failures, insufficient attention was given to the source, quality,
and age of replacement parts.
• In addition, considering the number of Heater Drain System components that can cause a
reactivity management event and the goal of minimizing the number of reactivity management
events, the establishment of preventative maintenance tasks has not been broad and
comprehensive enough.
15
Contributing Causes
 Untimely resolution of obsolescence issues and inadequate shelf life controls resulted in
defective components being installed.
 Components impacting Reactivity Management were not considered in the initial PM basis
development work for the Heater Drain System. The resulting lack of PMs resulted in several
initial transients and reactivity management events.
 Troubleshooting of Heater Drain System events were not managed effectively. 3 of 10
initiating events resulted in a total of 12 additional events before the System was
successfully returned to service.
 The portion of OP-136 used to set the alternate level controller setpoints for the 4A and 4B
FWHs was found to have deficiencies that contributed to the recurrence and severity of Heater
Drain System transients and reactivity management events.
16
Generic Lessons Learned
 Systematic troubleshooting is vital.
 If you don’t have enough data to determine cause, seriously consider restarting equipment to
collect more data, rather than replacing all the parts that potentially could have caused the
problem.
 Clearly define the roles and responsibilities of personnel on your troubleshooting team and
insist management respect and follow the organization established.
 Divide data collection and other field work from troubleshooting to allow the troubleshooting
team to stay focused and function well as a multi-discipline team.
 Don’t be so focused on producing a Support/Refute Matrix or Operational Decision
Recommendation, that you short change the data collection and analysis.
17
18