2007 FRCC AGENT
OPERATIONAL AUDIT
REPORT
Audit Date: November 8th, 2007
FLORIDA RELIABILITY COORDINATING COUNCIL
December 5th, 2007
1408 N. Westshore Blvd., Suite 1002, Tampa, Florida 33607-4512
Phone 813.289-5644 * Fax 813.289-5646
www.frcc.com
Table of Contents
2
•
1.0 Executive Summary
Page 1
•
2.0 Introduction
Page 2
•
3.0 Purpose and Scope
Page 2
•
4.0 Agent Audit Task Force Membership
Page 3
•
5.0 Operational Audit Process and Procedure
Page 4
•
6.0 Observations and Discussions
Page 6
•
7.0 Recommendations
Page 8
•
Appendix A – Final Agenda – Agent Audit Task Force Audit
Page 10
•
Appendix B – On-site Audit Participants
Page 12
1.0
Executive Summary
The Agent Audit Task Force (AATF), a group of FRCC member, sector
representatives was established and charged with performing an operational audit of
the FRCC Agents implementing the functional requirements defined within the FRCC
Security Process for the FRCC Bulk-Power Electric System. During this audit
process, the 2007 AATF found no issues or concerns related to the FRCC Agents
performance of their Reliability Coordination functional roles and fulfillment of
responsibilities as the Real-Time Operating Function, the Operations Planning
Function and the State Capacity Emergency Coordination Function for the FRCC
Region. This conclusion is based on the following:
¾ The AATF did not receive or investigate any complaints or inquiries regarding
Agent performance.
¾ The AATF reviewed previous Agent Audit Reports as well as the most current
NERC Readiness Audit Report (Reliability Coordinator) and found that
recommendations and opportunities for improvement identified in the past have
been appropriately addressed.
¾ The AATF performed an on-site audit and review of the Agent facilities and
personnel.
¾ The previous experiences between AATF members and the Agents.
It was evident to the task force that both Florida Power and Light (FPL) and Progress
Energy – Florida (PEF) have performed their FRCC Agent responsibilities in an
effective and commendable manner. This was illustrated in the latest NERC Reliability
Coordinator Readiness Audit Report which cited 2 “Potential Examples of Excellence”
and 13, “Positive Observations” along with the Agents responsiveness to
recommendations received from both the NERC Readiness Program and the 2005
Agent Audit Task Force report.
The recommendations resulting from this 2007 Agent Audit are primarily directed to
the FRCC organizational documents supporting the Reliability Coordinator functions
as well as the FRCC’s role as an Eastern Interconnection Reliability Coordinator,
functioning in the new era of mandatory and enforceable NERC Reliability Standards.
The task force has included 11 recommendations that the FRCC should consider in
order to improve and enhance its Regional processes. While there is no priority
implied in the order of the recommendations, the task force believes that several of the
recommendations are more important in terms of documenting the existing Reliability
Coordinator function hierarchy that exists within the FRCC Region, namely, the update
of the Security Process, update of the agent agreements and execution of an OPC
Agreement. Other recommendations are included as enhancements of documentation
and clarifications of processes that exist today. The task force believes that the FRCC
should evaluate all of the recommendations, and prioritize them for implementation
based on available resources and reliability requirements.
-1-
2.0
Introduction
The Security Process for the FRCC Bulk-Power Electric System (Security
Process) was created to ensure the reliability of the FRCC Region and defines the
roles and responsibilities of the FRCC Security Coordinator (SC) along with the
Operating Entities within the FRCC. The FRCC organization is the Security
Coordinator for the FRCC Region and utilizes agents (Agents) to carry out the
functional roles and responsibilities of the Security Coordinator as defined in the
Security Process. Over the past several years as NERC (North American Electric
Reliability Corporation) has evolved its original policies into a set of Reliability
Standards, the terminology describing the functional role of the “Security Coordinator”
has been revised to “Reliability Coordinator”. For the purposes of this report, the
terms “Security Coordinator” and “Reliability Coordinator” will be used interchangeably.
In the past, in order to ensure that the Security Process was applied and implemented
in a manner consistent with the FRCC’s goals of no undue discrimination, the FRCC
developed the FRCC Monitoring Procedures for Agents (Monitoring Procedure)
which provides the mechanism for a requested review of Agent performance as well
as the periodic establishment of an Agent Audit Task Force (AATF) to audit Agent
performance. The procedure which was adopted in April of 1998 by the FRCC Board
of Directors and last revised in December of 2003, requires an operational audit of
each FRCC Security Process Agent, every two years. The last Agents operational
audit was performed on March 11th, 2005. The 2007 operational audit was scheduled
for November 8th.
3.0
Purpose and Scope
The purpose of the FRCC Monitoring Procedures for Agents is to ensure that all
parties operating in the FRCC Region, including non-FRCC members, have
confidence that the FRCC Agents have acted in a manner consistent with the FRCC’s
goals of no undue discrimination in the application of the Region’s Security Process
functions as defined in the Security Process for the FRCC Bulk-Power Electric
System.
The primary scope of the operational audit included the following activities:
•
•
•
•
•
•
Review of the FRCC Security Process for required revisions
Request for comments on performance of FRCC Agents from entities operating
in the FRCC Region
Pre-audit Agent questionnaire developed by the task force
Input from the adjacent Reliability Coordinator
Pre-audit documentation review
An on-site audit to review Agent’s facilities and internal procedures supporting
its Agent functions
The AATF did not perform an on-site visit of the Progress Energy Florida (PEF)
facilities during this audit. Since PEF only performs the Operations Planning function,
-2-
and will be transitioning that responsibility and function over to Florida Power and Light
(FPL) starting in January 2008, the audit team determined it would be more efficient to
have the PEF personnel travel to the FPL facilities for interview purposes. The task
force determined that there was significantly more benefit to the audit process to
preview the Operations Planning Function transition plan, PEF to FPL, on-site with
FPL personnel assuming the function.
The FRCC Security Process identifies three primary functions of the Security
Coordinator. These functions are the Real Time Operating Function, the Operations
Planning Function, and the State Capacity Emergency Coordination Function.
Currently the FRCC Security Process Agents are listed below:
Real-Time Operating Function
Florida Power and Light Company
Operations Planning Function
Progress Energy Florida * / Florida Power and Light Company
State Capacity Emergency Coordination Function
Florida Power and Light Company
* Through the end of 2007
4.0
Agent Audit Task Force Membership
The Monitoring Procedures for FRCC Agents requires that each FRCC industry
sector utilizing the functions performed by the FRCC Agent(s) shall be appropriately
represented on the task force. The Task Force was initially made up of
representatives from all five (5) of the FRCC member sectors. Due to unforeseen
circumstances, the Investor Owned Utility (IOU) Sector was not represented at the onsite audit of the Agents. Based on the lack of any complaints, review requests or
expressed concerns with agent performance since the last operational audit, the AATF
members present, determined that the participating members were adequately
representative to perform the on-site agent audit and that additional steps would be
taken to ensure IOUs would have the opportunity for input to the process.
The 2007 AATF members are as follows:
-3-
Eric Senkowicz,
FRCC Staff - Chair
Joe McKinney
Florida Municipal Power Agency
Non-Investor Owned Utility Wholesale Sector
Herman Dyal
Clay Electric Cooperative
Load Serving Entity Sector
Richard Gilbert
Lakeland Electric
Generating Load Serving Entity Sector
Doug Jensen
Northern Star Generation Services
Suppliers Sector
Ron Donahey *
Tampa Electric Company
Investor Owned Utility Sector
Donna Howard
FRCC (Observer)
* Mr Donahey was unable to participate in the on-site audit but as the FRCC Operating
Reliability Subcommittee (ORS) Chairman and for the purposes of this report, is being
included as a subject matter expert on agent performance and also as a proxy
representative for the Investor Owned Utility Sector which was unable to provide a member
representative to the Task Force.
5.0
Operational Audit Process and Procedure
Each member of the task force was required to execute a combined FRCC
Confidentiality Agreement for Security Coordinator Audit with the FRCC and the
Agent.
Notification was made to all FRCC members of the pending audit in order to solicit
comments and concerns regarding the performance of each FRCC Security Process
Agent. A letter was sent to the members of the FRCC Board of Directors along with
the Standing Committees on October 11, 2007 requesting comments from FRCC
members as well as any other market participants regarding the performance, conduct
and non-discriminatory activities of each FRCC Security Process Agent.
The previously completed FRCC Agent Operational Audit Report, dated,
May 15, 2005 as well as the Monitoring Procedures for FRCC Agents are publicly
posted on the FRCC website and provide access and direction to all market
participants, regardless of FRCC membership status, on addressing Agent
performance issues within the FRCC Region.
As a result of the notification, the AATF did not receive any comments, concerns or
requests for review from any FRCC member, non-member or market participant
regarding Agents performance issues. In light of this, the AATF team began a
document review which included the following:
¾
¾
¾
¾
Security Process for the FRCC Bulk-Power Electric System
NERC Reliability Coordinator Readiness Audit Report, dated March 8-10, 2005
FRCC Agent Operational Audit Report, May 15, 2005
FRCC Operating Reliability Subcommittee Recommendations / Comments to the
FRCC Agent Operational Audit Report, dated April 18, 2007
¾ RC Agent updates on previous NERC Readiness Audit recommendations
-4-
Subsequently, the AATF developed a pre-audit questionnaire requesting specific
information which illustrated how the Agent performs certain aspects of the Real-Time
Operating Function as well as the State Capacity Emergency Coordination
Function. The pre-audit correspondence also included a request to PEF, to have
personnel available during the on-site audit at the FPL facility to address the
Operations Planning Function as well as a request to FPL to provide a transition
plan detailing how FPL will assume responsibility for the Operations Planning
Function starting in 2008.
The AATF also solicited input regarding Agents performance from the only adjacent
Reliability Coordinator to the FRCC, the Southeastern RC. It was noted by the
adjacent RC that the two RCs maintain sufficient monitoring capabilities to ensure
potential reliability violations are identified within their general interface areas and that
the RCs have adequate procedures in place to share information regarding potential,
expected, or actual critical operating conditions that could negatively impact each
other. No specific issues or concerns were noted by the adjacent RC.
The AATF met for a coordination conference call meeting prior to the on-site meeting.
During this call the team developed additional questions and requests for clarifications
based on the pre-audit questionnaire responses and document reviews. The AATF
provided a final data request so that the Agent could be prepared to provide
clarifications during the on-site audit. Finally, the AATF met as a team, prior to the onsite audit to review and finalize the audit agenda and logistics and review an audit
guide that details some of primary aspects of the Security Process. The guide was
developed to provide the AATF a quick reference sheet and as an aid to the auditing
process.
The on-site audit took place on November 8th, 2007, at the FPL control center in
Miami, Florida. As noted earlier, the PEF Operations Planning function representative
was present at the on-site audit location.
6.0
Observations and Discussions
The AATF Chair appreciates the efforts of all involved. Both the AATF members as
well as the Agents were very accommodating and essential in helping the AATF
achieve their purpose. In spite of a compressed and aggressive schedule, and the
limited number of task force members, the AATF was able to perform the requisite
audit, on-schedule, while minimizing impacts to those involved with the process.
The AATF also recognizes and expresses its appreciation to the FRCC agents for their
openness, honesty, and responsiveness in answering some of the AATF questions.
This served as both, a useful review of the Security Process for the Agents, as well as
a good training and information sharing opportunity for AATF members. In light of
some of the discussions regarding mandatory Reliability Standards and the potential
obsolescence of the Agent Audit process, it was noted at the conclusion of the on-site
audit, that the overall Agent audit process had, in fact been a beneficial process to the
-5-
overall reliability of the FRCC and should be continued in the future, regardless of the
developing Regional Compliance Programs.
The on-site audit began with an orientation presentation by FPL. FPL summarized the
roles of the various personnel in the control center as well as how those personnel
interact with the rest of the FRCC operating community. Next, a question and answer
session with key Reliability Coordination infrastructure support personnel reviewed
and clarified Agent questionnaire responses as well as addressed emerging AATF
questions and concerns that dealt with several areas, including facilities and training.
The AATF was then provided a tour of all the facilities and infrastructure that support
the FRCC Reliability Coordinator functions. The tour included the control room,
various critical infrastructure protection systems as well as the computer and
communication infrastructure for not only FPL but the FRCC RC personnel, as well.
One of the concerns from previous audits was the fact that FPL provides its own Backup capability for the Reliability Coordinator function. This concern was expressed in
several recommendations from past RC audits and reviews. As a result of this, the
AATF opted to focus in on this area by providing the Agent with a specific scenario
that would require initiation of Back-up RC function, and review the current plans and
capabilities in place. The AATF offered up a scenario and the Agent responded
accordingly. The AATF witnessed the transition and within minutes the FRCC RC
applications could be monitored in a completely remote location from the control
center while personnel would have been dispatched to the permanent back-up
facilities.
Next, the AATF met independently to discuss any other areas of concern which may
have been discovered during the morning’s audit sessions and to begin developing
any recommendations for the Agents.
The afternoon session began with the PEF Operations Planning overview and was
followed by a presentation of the Operations Planning transition plan from PEF to FPL.
The plan included introduction of FPL personnel, transition efforts to date, planned
parallel OPC functions during December with FPL assuming full responsibility of the
Operations Planning function starting in January.
The AATF initiated a second question and answer session and requested clarification
on a few responses and RC processes. Finally, the AATF met independently to
discuss conclusions of the task force as well as any recommendations that the task
force concluded were appropriate to be provide to the Agents. During this meeting,
the task force developed an exit briefing which was provided to the Agents prior to
completion of the audit. The presentation reviewed the AATF audit process and scope
used for this particular audit, a preliminary list of recommendations that may be
included in the AATF final report and the next steps of the Agent audit report
development and review cycle at FRCC. A tentative report development schedule was
provided to the AATF and Agents and the on-site portion of the audit was concluded.
As a result of all pre-audit document review and activities as well as the conclusions of
the on-site audit, the AATF found no problems or concerns related to the FRCC
Agent’s performance of their functions and fulfillment of their responsibilities. It was
apparent to the task force that both FPL and PEF take their FRCC Agent responsibility
-6-
seriously and work to perform the functions in the most efficient and effective manner
possible.
As part of the audit scope and preparation was the review of the previous Agent Audit
report as well as the previous NERC RC Readiness Audit report, the task force was
pleased to note that all of the issues and recommendations identified in the past
reports have been, or are being addressed by both the Agents and the FRCC
Operating Reliability subcommittee.
-7-
7.0
Recommendations
The audit reviewed both; the agents performance of the Security Coordination
functions, and the FRCC Security Process. Procedurally a review of the Security
Process is required to determine if any changes to the Security Process are warranted
in support of FRCC’s reliability goals. With the evolution of mandatory and
enforceable Reliability Standards and FRCC’s role and obligations as an Eastern
Interconnection Reliability Coordinator, review of both the Security Process and the
Monitoring Procedure are timely and critical elements of the recommendations of this
agent operational audit.
Overall, the task force found no deficiencies in the agent’s performance; however, the
task force presents the following recommendations to the FRCC members in order to
enhance and improve the quality of the reliability processes of the Region.
1.
Update and revise, as appropriate, the FRCC Security Process for the FRCC
Bulk-Power Electric System document, to ensure it adequately addresses the
NERC Reliability Standards requirements for Reliability Coordinators. The revision
should also specifically address the following:
a.
b.
2.
Retain the Agent Audit process and update the FRCC Monitoring Procedures for
Agents document to reflect the revisions in item 1 above. Revisions should also
consider using the monitoring procedure as the first step to developing an internal
FRCC Reliability Coordinator compliance program that addresses mandatory RC
Reliability Standards as well as a more formal, internal RC, continuous
improvement program for the Region.
3.
Review and revise Reliability Coordinator training criteria as required.
a.
Review should document how appropriate Standards of Conduct training
requirements are achieved by the Agents.
4.
ORS should update and revise as necessary, the “Security Coordinator
Qualifications” handbook document. The revisions should address any specific
training requirements for FRCC Reliability Coordinators as well as training
requirements on the “next-day” Reliability Coordinator.
5.
Update Agent Agreements to reflect current Reliability Standards terminology.
a.
-8-
Replace “Security” with “Reliability”
Replace or re-define “Operating Entity”
Including specific training criteria within the agreements should be considered.
6.
Execute an Operations Planning Function Agreement with FPL to reflect the
transition of the function over to FPL.
7.
Real-time Reliability Coordinator function should fully implement dynamic line
rating transfers. All Transmission Owners line ratings should be provided to the
RC applications in a consistent and uniform manner that does not rely on manual
updates by RC EMS support personnel.
-9-
8.
Review RC communication requirements to registered entities and how to
incorporate contact information into the RC and host BA / TOP processes (ie.
IPPs).
9.
Review the IPP data (outage and capacity) information and how data is
incorporated into, or impacts the OPC process.
10.
Although the task force was unable to view the progress on the FPL Document
Management system being developed (planned completion, spring 2008), the task
force recommends that ORS track implementation of the planned system to ensure
that any impacts on RC documents and personnel are appropriately addressed
(ref. NERC Readiness Audit Report, March 8-10, 2005, Recommendation 1).
11.
ORS should update, and revise as necessary, the FRCC IROL list. The revisions
should take into account the current developments occurring at the NERC level
regarding the definition of IROLs as well as IROL methodologies.
Appendix A
Final Agenda – Agent Audit Task Force Audit
(on-site portion of audit process)
- 10 -
Final Agenda
Agent Audit Task Force Audit
Wednesday, November 7, 2007
3:00 p.m.
AATF pre-audit meeting to finalize agenda and formulate audit plan
Review with AATF major points of Agent Audit
Review AATF audit guide / checksheet
Select back-up scenario
List any specific AATF issues to be included in audit
Thursday, November 8, 2007
8:00 a.m.
Introduction of AATF / FPL
Review logistics
Additional Information requests
8:30 a.m.
Overview and walk-through tour of facilities
Position descriptions / roles of personnel
Relationships of positions
RC Communications to FPL
RC real-time function
Review Statewide overview
RC next-day planning function
Review of communications infrastructure
Review of EMS infrastructure
Review of RC Modeling functions
10:30 a.m.
RC Back-up Control Center Drill scenario
12:00 p.m.
Lunch
1:00 p.m.
Review of OPC functions
PEF OPC Representative
OPC Transition plan
FPL OPC plan
2:30 p.m.
AATF discussion on issues / open items / additional information
3:00 p.m.
FPL additional question session
3:30 p.m.
AATF conclusion summary session
4:00 p.m.
Exit briefing with FPL and PEF
- 11 -
Appendix B
2007 FRCC Agent Audit Participants
Herb Evelyn
Manager, FPL Computer Operations
Jeff Gooding
Manager, FPL Training & Certification
Don McInnis
Manager, FPL Operations Engineering
Pedro Modia
Director, FPL Power Supply
Andy Pankratz
Manager, FPL System Operations Load Dispatch
Gilbert Perez
Supervisor FPL
Art Pique
System Operator FPL
Frank Prieto
Manager, FPL System Operations Transmission
Juan Quintana
2008 SCEC Agent (FPL)
Ritha Ramos
Supervisor FPL
Bill Slater
2007 OPC Agent (PEF)
Kaveh Tarighy
Supervisor FPL
Dick Wielandt
System Operator FPL
Miguel Yanes
2008 OPC Agent (FPL)
FRCC AATF members (present on-site)
Eric Senkowicz
FRCC Staff - Chair
Herman Dyal
Clay Electric Cooperative
Load Serving Entity Sector
Joe McKinney
Florida Municipal Power Agency
Non-Investor Owned Utility Wholesale Sector
Richard Gilbert
Lakeland Electric
Generating Load Serving Entity Sector
Doug Jensen
Northern Star Generation Services
Suppliers Sector
Donna Howard
FRCC (Observer)
- 12 -