Hydro One Major Incident Report
Operational response and incident review of flooding at Manby and Richview
Transformer Stations on July 8, 2013
1
Toronto Flood, July 8, 2013
On July 8, 2013, the Greater Toronto Area was hit by record rain fall with Pearson Airport
reporting 90 mm of rain between 4:231 p.m. and 6:00 p.m. and 123 mm over a 24 hour period.
The rainfall broke the previous daily record of 121 millimeters of rain set in 1954 by Hurricane
Hazel. The rain came down with no advanced warning of the severity from Environment Canada
weather services and caused major damage to the infrastructure serving the West GTA.
The flash flooding that came with the rain overwhelmed rail, subway, sewers and drainage across
the affected area as well as major electricity infrastructure serving the western portion of the
Greater Toronto Area.
Hydro One Networks Inc. experienced severe flooding at its Richview and Manby Transformer
Stations (TS) causing interruptions of twenty-six 230,000 volt and eight 115,000 volt circuits
supported by these stations and damaging communications equipment within these stations.
Water damage continued to wreak havoc on operations as a building housing internal computer
systems and telecommunications infrastructure filled with water from cable conduit penetrations,
insufficient weeper sump pump capacity and loss of AC power to the weeper sump pumps.
The flooding was significant, and reports from the response team indicate that the water removed
from in and around Manby TS would have filled 15 Olympic-sized swimming pools.
Hydro One declared a Level 2 Transmission Emergency after 3,400 MW of load was interrupted,
causing widespread power outages to more than half a million households and businesses in the
GTA.
Six local distribution companies who receive their supply from Hydro One were directly affected
by this event: Toronto Hydro, Hydro One Brampton, Enersource, Oakville Hydro, PowerStream,
and Veridian. Four transmission connected customers were also directly affected: Ontario Power
Generation Kipling Complex, Ford Oakville, IBM and Kinetrics HV Lab.
During this time, Hydro One’s Ontario Grid Control Centre (OGCC) was inundated with more
than 126,500 alarms and 9600 phone calls.
The unprecedented rush of water flooded into buildings at Richview TS (specifically into the
OGCC Transmission and Distribution back-up control facilities) damaged and/or rendered critical
equipment unavailable. Compounding this situation, the ITMC control room had to evacuated to
their back up control room. Additionally, corporate IT systems were impacted, including mobile
email access on non-Blackberry devices and Call Centre 1-800 numbers.
1
All times are in Eastern Daylight Time (EDT)
2
Restoration began immediately and over the following five days the GTA power system was
reconfigured and stabilized while station repairs were made and flood damage was assessed.
Chronology of Restoration
Hydro One crews were immediately dispatched, however with transportation systems across the
city impacted by severe flooding, the speed of the response was hampered. Throughout the
evening and into the night, crews made their way to these stations, at times receiving escorts from
Toronto Police Services.
The restoration focus was to transfer the affected 115 kV Toronto West load to Leaside TS and
restore 230 kV circuits from remote stations outside of Richview TS and Manby TS. Hydro One
worked closely with Local Distribution Companies (LDCs) and the Independent Electricity System
Operator (IESO) to coordinate the system reconfiguration.
By 4:00 a.m. on July 9, about 10 hours after the intense rain front hit the GTA, the electricity
system was 91% restored with only 300 MW2 directly supplied load out of Richview TS and
Manby TS remaining
unrestored, roughly representing10,000 homes and businesses served
mostly by Toronto Hydro.
On site work continued around the clock as crews removed water from flooded buildings, dried
and inspected damaged equipment. Additionally, the OGCC dealt with the predetermined 115
kV corridor thermal limit issues that would knowingly ensue as the normal load profile demand
would increase the following day after the storm.
To alleviate the loading issues on these
corridors voltage reductions and load shedding was implemented at the associated stations
throughout the GTA.
By 3:24 p.m. on July 10, all affected load had been restored, however the GTA power system
required significant work to bring it back to normal configuration. The Transmission Emergency
Level 2 remained in effect until 5:34 p.m. on July 12th as Hydro One worked to reinforce the
system with restored transmission connections between Richview TS and its remote terminal
stations: Trafalgar TS, Cherrywood TS, Parkway TS and Claireville TS. This provided redundant
supplies and vastly improved network security.
Further remediation continued at both Richview and Manby over the following few days and at
2:44 p.m. on July 15th the system was officially back to normal status.
2
Some of the 300 MW was restored through internal load transfers within the LDC systems.
3
Communication with the public during the event was vital and Hydro One depended on the news
media to help tell the story. Hydro One’s media relations team conducted more than 200 media
interviews, issued 6 media releases and tweeted 36 times.
Hydro One’s President and CEO Carmine Marcello participated in an on-site press conference
with the Mayor of Toronto and the President and CEO of Toronto Hydro. Status updates and
conference calls were held to provide updates to government officials and the City of Toronto’s
Emergency Operations Centre.
Summary of Root Causes
The OGCC and Asset Management conducted investigations into the cause and response to this
event. The OGCC assessed the response and actions taken from an operational perspective
including IT, Corporate Communications and Security, identifying valuable lessons learned. Asset
Management identified the causes for the flooding and recommended corrective actions to
prevent a recurrence.
This summary report provides high level information taken from their
detailed internal analyses.
Weather
There was no prior warning of serious rainfall or potential for flooding issued by any of the
weather forecasting providers used by the OGCC. Water from the flash flooding primarily
entered impacted buildings via cable penetrations in building basements. AC powered sump
pumps at Manby TS and Richview TS Building O were quickly rendered inoperable due to loss of
AC station service power at both stations.
Employees were significantly delayed in reaching Richview TS and Manby TS to aid in the
restoration efforts during the onset of this event due to extreme weather and road closures. Local
news updates on the road conditions were indicating road closures, sink holes, nonfunctioning
traffic signals and high traffic volume. On site Construction crews at Manby TS proactively called
for vacuum trucks to help with the flood relief. Even though some on-call employees had been
able to respond within 15-20 minutes from dispatch, due to amount of rainfall received, they
could not prevent the resultant water damage.
4
Hot weather during the days following the initial flooding led to high energy usage across the
GTA. As a result transformers and transmission lines were heavily loaded as restoration efforts
progressed. The reconfigured circuits had additional loading, which led to heavy thermal loading
on several corridors.
Communication challenges
Incident Command Centre-Transmission was not readily initiated and could have been initiated
earlier; however the functional requirements of ICCT were already being implemented and
managed by the OGCC frontline support and the first responders without an official corporate
wide declaration. The initiating conditions were satisfied with the Level 2 SENS but declaration of
establishment corporately was not immediately conveyed.
OGCC and IESO complied with proper communication and policy protocols throughout the
event; however review of these protocols will take place to explore opportunities of increased
operating automomy during localized events which would further decrease restoration times.
Response to station alarms was affected due to the high number in a short period. As a result,
communications to customers did not occur immediately as would under normal operating
procedures. Customers that contacted the OGCC for status updates were kept apprised.
Constant requirement for information between involved agencies led to delays and discrepancies
(for example, forecasted restoration times). The Hydro One media line was inundated with calls
and triage was conducted with the largest outlets with the greatest reach receiving priority.
Equipment Condition and Design
Both site drainage systems at Manby TS and Richview TS were designed for a 100 year one-hour
storm event. Immediately, prior to the July 8, 2013, storm, there was no evidence to suggest that
either site drainage system was not operating as designed. At Manby TS, the site drainage
system was recently inspected, flushed and upgraded ($4.7M spent, completed fall 2011).
Similar investments were not required at Richview TS, as this site had no history of significant site
or building flooding.
At Manby TS, although the drainage capacity of the site drainage system and the City of Toronto
stormwater sewer system was likely exceeded during the storm peak, this was not the primary
cause of building flooding. The primary cause was 100+ year rainfall plus aging/plugged cable
trench underdrainage systems.
As a result, stormwater from flooded cable trenches in the
switchyards entered buildings via unsealed/poorly sealed cable penetrations in basement walls.
5
Water rendered inoperable critical power system equipment located in these basements (terminal
racks, battery rectifiers, etc).
At Richview TS, as a result of a 100+ year rainfall, stormwater once again flooded the yard cable
trenches and entered both the East and West Relay Buildings via cable penetrations in the
basement walls. Additional stormwater entered the West Relay Building via a poorly sealed
window well. Flooding of terminal racks in these Relay Basement buildings caused circuits to be
removed from service and the loss of AC power at Richview TS and Manby TS.
Stormwater also entered the Richview TS Telecom Building via cable penetrations, sump pit
overflow/backflow and the loss of AC power to the sump pumps.
Follow-up Actions
Following the event a rigorous review was undertaken to document the full scope of damage and
the resulting system repairs completed in the days following the initial damage. To conduct this
review information was collected from Operating and Field staff, various lines of business,
affected customers, the Network Management System information and storage retrieval system,
and the review of 9,600 phone calls relating to the event from the OGCC control room audio log
system.
The detailed report provides a breakdown of all affected stations, line reconfigurations, system
impacts related to Corporate IT infrastructure and relationship management with external
agencies and customers. Follow up actions related to the below Key Learning’s are documented
and will be tracked on an ongoing basis.
Follow-up actions with regards to the events surrounding the GTA flood will support Hydro One’s
compliance commitments and service level agreements and goal of operating a safe and reliable
system for all customers.
Key Learnings and Recommendations
Findings, relevant comments and concerns, lessons learned and recommendations arising from
this review of the GTA blackout that occurred on July 8th have been documented and continue to
be implemented into regular operating procedures and future emergency response planning.

Operational recommendations Review and evaluate back up options for key
computer and telecommunication services, and back-up systems integral to OGCC’s
6
operability of the power system. Business Information Technology will assess opportunities
to build further redundancies in the system thus avoiding future vulnerability to the
Corporate IT services.

Response time for Hydro One staff can be reduced by engaging local police services to
explore options for faster transfer of people and equipment to required sites. Operations
staff may be directed to drive personal vehicles to a non-impacted common location to
rendezvous with a police escort in order to get to the site of emergency quicker.

A review of the benefit of creating a standard system emergency message that could be
issued at the onset of extreme events to notify customers and internal staff that Hydro One
is not in a normal operating state. This could assist in reducing low priority calls and
would ensure that the customers have been notified in a prompt manor.

Clarification is required on declaration of ICCT dependent on the classification of the level
of emergency. As an example, when an ICCT is initiated non-emergency transmission calls
could be re-routed to other sectors not affected by the event. Other required logistical
processes are put in place under ICCT that assist with dissemination of communications
and coordination efforts.

Coordinate joint messaging with Toronto Hydro and emergency response organizations to
ensure consistent messaging is delivered during other large-scale events to avoid public
confusion.

Explore opportunities for IESO pre-approved load restoration plans during localized
emergencies.
Asset Management recommendations
To prevent the entry of stormwater during severe rainfall events into building basements
containing critical power system/IT equipment, Hydro One is undertaking the following actions:

Seal watertight all cable penetrations and window wells at Richview TS and Manby
TS.

Re-design the AC supply system to the Richview TS Telecom Building sump pumps to
ensure automatic back-up supply via the diesel generators.
7

Review consultant’s recommendations and compare against “as built” 2010-2013 site
drainage system/sump pump upgrades at Manby TS.

Identify all transmission station buildings with critical power system equipment located
in basements and, where appropriate, develop a mitigation plan for incorporation into
the 5 year investment plan. This plan may include the following: 1) sealing and
maintaining cable penetrations; (2) relocating new/replacement equipment and/or
cable entrances to above grade locations; 3) inspecting/replacing/maintaining
basement check valves (or backflow prevention valves); 4) upgrading cable trench
under drainage systems and/or 5) upgrading building sump pits/pumps.

Determine if changes to the Hydro One design criteria for transmission station site
drainage systems are required (ie, design for a 200 year one-hour storm event).

As part of planned infrastructure investment projects, replace the Richview TS 230 kV
air blast circuit breakers and relocate protection and control equipment to new relay
buildings.
8