WORLD TRADE CENTER
And Related Recoveries
Len Guddemi
Vice President
Northeast Operations
Comdisco Confidential
SunGard Worldwide Disaster Support
Experience
100
80
60
40
20
0
'81 '82 '83 '84 '85 '86 '87 '88 '89 '90 '91 '92 '93 '94 '95 '96 '97 '98 '99 '00 '01
Recoveries per Year
SunGard Worldwide Disaster Support
Experience
Environmental 8/2%
Software 10/2%
Telecomm 14/3%
Terrorist Attack 100/19%
Bomb 16/3%
Civil Unrest 2/0%
Earthquake 19/4%
Miscellaneous
20/4%
Fire/Explosion
27/5%
Data Center
Hardware 97/18%
7/1%
Flood 57/11%
Hurricane 84/16%
Power Outage
76/14%
Disruption Cause: Number/Percentage
SunGard Worldwide Disaster Support
Experience
DEC 43/8%
HP 22/4%
Miscellaneous
26/5%
AS/400
56/10%
Advanced Recovery
13/2%RS/6000 10/2%
Unisys
2/0%
Shell 7/1%
Workarea Recovery 130/24%
Tandem 3/1%
Systems On-site Mobile
39/7%
Sun 16/3%
Mainframe 138/26%
Trading Floor 19/4%
Item Processing
Stratus
5/1% 1/0%
Mobile Cluster 3/1%
NT Server
4/1%
Recovery Platforms Used: Quantity/Percentage
SunGard Worldwide Disaster Support
Experience Civil Unrest 3/0%
Terrorist Attack 1154/20%
Telecomm 30/1%
Earthquake
136/2%
Data Center Move 269/5%
Miscellaneous
858/15%
Hurricane 318/6%
Fire/Explosion
359/6%
Flood
691/12%
Hardware Problem
666/12%
Power Outage
370/7%
Bomb
Environmental 573/10% 146/3%
Days of Occupancy: Days/Percentage
Declaration Information

Number of Events
 Overall
- 93 Schedules, 47 Customers
 Simultaneous
High Point - 76 Declarations, 16 Facilities, 14
Platforms
 Reason

Facilities Unavailable

Loss of Power, telecommunications, denied access to buildings,
concerns over building structural safety
Comdisco Confidential
SunGard Sites &
Platforms


Atlanta: Work Area Seats, Remote Sun,
 North Bergen: Voice/Work Area,
Main-Frame, ARS & Networking
AS400, RS6000
 Phoenix: Work Area
Carlstadt: Voice/Work Area, Business
 Queens: Work Area, Trading Floor,
Recovery/Quick Ship, Trading Floor,
Dedicated Seats, SUN, AS400,
DEC/VAX, RS6000, HP9000, Intel
Based Servers; Dedicated HP9000,
Dedicated, Trading PIT
 Rosemont: Work Area
 San Ramon: Work Area, Trading Floor
 Wood Dale: DEC/VAX, DEC/Alpha,
VAX3100, VAX7620, Filenet, AS400,
RS6000, NT Servers
Tandem

Boston: SUN, Work Area
 Wood Cliff Lake: Intel Servers, Work Area

Cypress: Work Area, Remote Tandem
 Jersey City, workgroup, AS/400

Mobile Recovery: 13 Units/5,000sq.ft., 10
 Philadelphia: Work area, RS6000,
Units/3,000sq.ft. for Work Area

Minnetonka: Work Area
AS/400,Mainframe, Voice
Herndon: Work Area
Warminster: Work Area,Unisys
Comdisco Confidential
Declaration Information - Continued
Items of Note
 Implement Your Plan & Associated Vendor Services with Urgency
 14 Declarations in 30 minutes; 24 within 1 hour; 19 in 2nd hour
 You will be in competition with other business for services & manufacturers
Quickly establish damage assessment & need for services, and implement

Under-subscription or additional needs assessment remains an issue

Many businesses requested additional needs ATOD

Responded to over 500 special requests for service.



Built out with best efforts
This puts recovery & business continuity at risk due to extended recovery
timeline.
Personnel & Intellectual Property Keys to Recovery

Methodologies, ideas and documented plans utilized.

Assigned backup roles and succession planning implemented.

Intellectual capital and personnel in alternate locations.

Use of professional organizations to augment staff.
Comdisco Confidential
Declaration Information - Continued
Items of Note - Continued
 Reluctance to Travel
 End Users generally won’t travel far from home.
Ensure adequate end user facilities available nearby (about 50 miles)


Air Travel Unavailable
 Delayed delivery of offsite stored media to recovery location
 Identify alternate means of transportation
 Assess need for electronic journaling of data to alternate location
 Expert staff unable to travel
 Identify options for local access by staff to remote alternate systems (and
possibly home systems)
 Assess value of ‘turnkey’ type services with vendors
Offsite Storage Critical
 Evaluate for frequency, completeness
 Paper records and transactions may be totally lost
 Evaluate if your offsite storage is far enough away
 Evaluate need/value of electronic journaling
Comdisco Confidential
Declaration Information - Continued
Items of Note - Continued
 High Availability Solutions Extremely Successful
 Electronic Journaling, Data Base Mirroring, SRDF, etc.
 45 Terabytes ready very quickly
 Data requiring physical transport delayed


Active Testers
 More successful at meeting RTO/RPO
 More realistic understanding of application performance
 Fewer problems
Very High Demand for Voice Recovery
 Realistically evaluate needs: line trunking, extensions, call recording, voice
mail, call forwarding
 Avoid Fast Busy
Comdisco Confidential
SunGard Challenges & Tactics




Disaster Hotline & Notification
 Increased Staff handling incoming Disaster calls
Security
 Immediate increase in security staff on all shifts.
 Contacted local authorities to assist in access control
Additional Staff
 Request for additional support staff
 Mobil Recovery Team
Additional Resource
 Contacted third party vendors
 Arranged additional parking with neighbors
SunGard Challenges & Tactics
 SunGard Facilities and Staff not directly affected

Implemented Swat Support Teams

Established 24/7 Command Center at Corporate HQ with Open Bridge

SunGard Canadian employees, Illinois, Minneapolis, and Texas employees
drove to Carlstadt & North Bergen to help

SunGard’s Professional Services Consultants, Sales, Sales Support,
Administration and SunGard’s Operations staff worked side by side.

Dispatched relief support from other SunGard facilities once air travel restored

Activated facilities, equipment procurement, shipping and solutions teams to
respond to special requests

Received offers of help from customers


Gratefully declined
Network Alternate Routing Critical

Significant damage due to the event

Fiber cut after event by construction/demolition crews

SunGard alternate routings worked without issue

Vendors terrific with T1s, DS3 installations in days
Comdisco Confidential
Customer Challenges & Tactics
 Inaccurate Needs Assessment
Special requests for PCs, & Servers
 IP addressing
 Storage
 Tape Drives
 Electricity
 Processing power
 LAN bridging
 Televisions & Printers
 Network capability, Voice services &
 Faxes
 Software and more
Routers
Comdisco Confidential
Customer Challenges & Tactics

Logistics
 Travel Restriction an Impact - used ground travel for many shipments
 Shipped over 4,200 PCs along with much other equipment
 Customer parking, bussing, and catering
 Getting Restoration Media to Alternate Site



Transportation challenge
Complete set available and labeled
Missing or unavailable documentation
Start-up scripts
Run instructions
Comdisco Confidential
Learned / Re-Affirmed
 Value in Pre-Planning Staff Placement within Rooms


End User priorities & usage
End User equipment requirements – avoids setting up more than once

Important to Have a Team Leader for Each Shift

Integration of recovery requirements into change control







Number of PCs, Positions
Network Capabilities
Processors, disk, tape capabilities
Printers, fax machines
PBX features & functionality
Platform Technology upgrades, changes, maintenance
Importance of Quick Assessment & Implementation of Plan when
appropriate
Comdisco Confidential
Learned / Re-Affirmed - Continued
Value of Solid Backup Records Program
 Tape, electronic, & paper records foundation of recovery
 Evaluate right balance for your company
 Documented & labeled so vendor(s) can assist
 Have scripts shipped with restore media or stored at alternate location
 Frequency of backups
 Offsite location & transportation dependencies
 Examine benefit of electronic transmission of offsite storage data
 Can significantly improve recovery time and reduce data loss
 Evaluate Strategy of Shared versus Dedicated Resources
 Guarantees resource access for some number of most critical
requirements
Complement with shared resources

 PC Image Restoration Tool Effective

Ghost, or similar
Comdisco Confidential
Learned / Re-Affirmed - Continued

Basics are Fundamental

Establish personnel roles & assign backup

Document methodologies, ideas, business plans & recovery plans

Review, update & follow established procedures

Test the way you would recover, recover the way you test

Worthwhile to review how home procedures apply in alternate site production
mode

Offsite Storage, Disk mirroring, Miscellaneous other
 Revisit your Plan & Recovery Requirements Often – Technology & Business
 In Times of Need, Help Comes From Many Sources
Comdisco Confidential
Customer Comments
“Although operating in our primary location is preferred, the
equipment, professional and courteous staff, and outstanding overall
accommodations provided for an effective and successful business
recovery. I am glad that we have contracted with your organization!”
“I have in the past worked with a number of outstanding individuals,
but never before have I worked with such a highly functional,
customer service oriented team. Individually each team member I
have had the pleasure to deal with was exceptional; as a team there is
none better.”
Comdisco Confidential
Customer Comments
“I would like you to know of the "more than wonderful" service we are
receiving at the SunGard site. ... have gone out of their way to assist us
with some special needs. ... have answered numerous questions and
assisted --she is great. ... worked with ENS last night, assisted me with
set up so I can stay on line (due to the situation in New York and the
weather closing down our site) and has been a great help to the IT
staff.”
“I would like you to know it is people like this that make SunGard a
vendor we choose to depend on in the event of an emergency. Please
pass this message along to their managers, it would be greatly
appreciated.”
“Thanks for all the support and assistance. We’re glad you are here.”
Comdisco Confidential
Questions and Answer
Comdisco Confidential