Achieving
Operational
Excellence
Asia Pacific Technical Services
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
1
What is Operational Excellence?
Process
People
Technology
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
2
Is this Rocket Science?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
3
What are the risks?
Network Costs = 20% Purchase, 80% Operations
Cost Drivers
Source: The Meta Group, 2005
Cost of Complex
Integration
• Purchase price is ONE
element of the total cost
of a system
Opportunity Costs
• Hidden or less obvious
cost/value drivers often
outweigh the purchase price
Training Costs
• A network foundation with
integrated advanced services
leads to lower TCO
Network Purchase Price
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
4
Architectures
ITIL or ETOM
Information
Technology
Infrastructure
Library
http://www.itilofficialsite.com/home/home.
asp
Enhanced
Telecom
Operations Map
http://www.tmforum.org/
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
5
A4
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
6
Act
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
7
Lets manage the network
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
8
First Problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
9
What information do we have?
Assess
1. Voice quality on a phone has problems
2. Quality make voice stream unrecognisable?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
10
What information do we have?
Acquire- Environmental
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
11
What information do we have?
Acquire - Lifecycle
?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
12
What information do we have?
Act
1. Check the devices between phone and CCM for
drops and resource issues
2. Connect a network analyser at the Phone
3. Observe phone during poor voice quality
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
13
IOS command line
AirServicesNat#show interfaces | inc drop
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Redirect IOS command line output to a URL example
AirServicesNat#show interfaces | tee tftp://10.5.1.2/command-out
AirServicesNat#show interfaces | append ftp://10.5.1.2/command-out
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
14
IOS command line
AirServicesNat#show buffers input-interface fast 2/0
Header DataArea
Pool Rcnt
Size Link
Enc
Flags
Input
Output
64030310
E0010C4 Small
1
60
7
1
280
Fa2/0
None
64031BAC
E001984 Small
1
60
7
1
280
Fa2/0
None
646A4BD8
E221FE4 Small
1
60
7
1
280
Fa2/0
None
646A5D6C
E222624 Small
1
60
7
1
280
Fa2/0
None
646A7608
E2234E4 Middl
1
292
7
1
280
Fa2/0
None
646A7D10
E223B64 Middl
1
292
7
1
280
Fa2/0
None
Header DataArea
Presentation_ID
Pool
© 2006 Cisco Systems, Inc. All rights reserved.
Rcnt
Cisco Confidential
Size
Original
Flags
caller_pc
15
XML PI and NETCONF
Programatic interface that uses CLI or NETCONF
RFC4741, RFC4742
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
16
XML PI and NETCONF
<get-config>
<source><running/></source>
<filter type="cli"><config-format-xml
options".."></config-format-xml></filter>
</get-config>
Supported in 12.4T
http://www.cisco.com/en/US/docs/ios/12_2sr/12_2sra/feature/guide/srnetcon.
html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
17
Wireshark – Initial Voice Capture
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
18
Wireshark – Analyse Voice Streams
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
19
Audacity– Playback Audio
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
20
SPAN and VLAN Capture
Supported on most switches
Used for examining traffic on a local switch
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
21
IP Traffic Export / IP Traffic Capture
Exports matched traffic out a specific interface or VLAN
on the router
Ideal for Network analyser or probe
Capture to memory device (flash, tftp, usbflash)
Only supported on software switching ISR’s 12.4(11)T
http://www.cisco.com/en/US/docs/ios/12_4t/12_4t11/ht_rawip.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
22
Phone during problem
Press ‘i’ twice to
obtain call
stastistics
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
23
What information do we have?
Acquire - Lifecycle
1. The packets are originating from the Media Server
(IVR) traversing the local switch/router infrastructure,
terminating at the phone.
2. Problem is seen in packets originating from server.
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
24
More Video
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
25
What information do we have?
Analyse
Possible Causes:
1. Server is “busy” causing a corruption of the audio
file as it is played on the LAN
2. The audio file itself is corrupt, incorrectly recorded
or using the wrong codec
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
26
What information do we have?
Act
1. Monitor server performance
2. Examine audio file that is on the server for quality
problems.
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
27
Monitor Server Performance
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
28
Examine audio file
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
29
What’s next?
Act
Next Problem!
What else can we do that will proactively
examine counters and alert us to
potential problems?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
30
Synchronised Clocks
Network Time Protocol (NTP)
ntp server (host) [version n]
ntp peer (host) [version n]
http://www.cisco.com/en/US/tech/tk648/tk362/tk461/tsd_technology_support_subprotocol_home.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
31
Embedded Event Manager (EEM)
Think of a policy
as an action
registered to an
event
ED notifies EEM
Server; which
triggers interested
policies
Tcl-based policies
Programmed in Tcl
As complex as you want
Applet-based policies
Defined via CLI
Simpler
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
32
Periodic MIB Data Collection
Device polls specific MIB counters
Stores this locally (memory)
Periodically transfers data to server (tftp, rcp, and ftp)
Introduced 12.3(2)T
http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_mib_
collect_trans.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
33
Second Problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
34
Second Problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
35
What information do we have?
Assess
1. Internet access is down
2. Users complaining
3. Large Business Impact
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
36
What information do we have?
Acquire- Environmental
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
37
What information do we have?
Acquire - Lifecycle
?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
38
Network Topology Diagram
R1
R2
R4
R8
R5
Internet
R3
PC A
PC B
R6
R9
R7
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
39
What information do we have?
Act
1. Gather Source and Destination addresses
2. Verify problem existence in the network
3. Identify which device(s) is causing the issue
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
40
Gather Source and Destination addresses
Translate the customer complaint to useful information
Helps identify what are possible causes
Narrow problem down to a specific lifecycle through the
network
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
41
Verify problem existence in the network
Gain access to the devices experiencing the issue
Confirm the customer’s symptoms
Find alternate device exhibiting same issue
Recreate in a lab environment
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
42
Traceroute and Debug ip icmp
Traceroute from source to destination
Traceroute in the opposite direction
Enable “debug ip icmp” on either side of suspected device
Turn off console and terminal logging
no logging console
no logging monitor
logging buffered
Traceroute again to observe debugs
http://www.cisco.com/warp/public/63/ping_traceroute.html#traceroute
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
43
Network Topology Diagram
R1
R2
R4
R8
R5
Internet
R3
PC A
PC B
R6
R9
R7
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
44
Using access-lists to find packet flows
Configure access lists ingress and egress
access-list 100 deny ip host a.b.c.d host w.x.y.z
access-list 100 permit ip any any
Verify access-list matches
show access-list 100
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
45
Netflow and IP accounting
Configure netflow ingress on the interface
ip route-cache flow OR
ip flow ingress
Verify access-list matches
show ip cache flow
IP Accounting on the egress
ip accounting [access-violations] [output-packets]
show ip accounting
http://www.cisco.com/en/US/docs/ios/12_4/netflow/configuration/guide/onf_bcf.html#wp1047360
http://www.cisco.com/en/US/docs/ios/12_3/ipaddr/command/reference/ip1_i1g.html#wp1091971
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
46
What now?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
47
What information do we have?
Acquire - Lifecycle
1. Problem narrowed down to a single device R3
2. Traffic direction is broken toward the internet
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
48
What information do we have?
Analyse
Possible Causes:
1. Access-list blocking traffic
2. Physical interface problems
3. Forwarding problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
49
What information do we have?
Act
1. Implement a workaround to minimise the impact
2. Gain console access to the device in question
3. Step through possible causes to identify the culprit
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
50
Access-list
Check ingress and egress interface configuration
show run interface [Interface name]
Verify access-list configuration and deny matches
show access-list [number}
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
51
Check Ingress and Egress Interfaces
Display interface counters
show interfaces [Interface name]
Check the command output for the following
Duplex Setting
Input and output drops
CRC’s
Overruns
underruns
Ignores
Throttles
http://www.cisco.com/en/US/products/hw/routers/ps133/products_tech_note09186a0080094791.shtml
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
52
Forwarding verification steps
Start by checking the routing table entry
Check arp entry for next hop
Verify switching method
show interfaces [Interface name] stats
show cef interface [Interface name]
show cef not-cef-switched
Check cef entries
show ip cef w.x.y.z detail
(w.x.y.z destination ip)
show adjacency a.b.c.d
(a.b.c.d next hop ip)
http://www.cisco.com/en/US/tech/tk827/tk831/tk102/tsd_technology_support_sub-protocol_home.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
53
Forwarding architectures
Centralised software
Centralised hardware
Distributed software
Distributed hardware
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
54
Root Cause
Drop adjacency in hardware cef on GSR linecard
show ip hardware-cef exact-route a.b.c.d w.x.y.z
FIB: 0x45AE0980, FIB->hwleaf: 0x7004FD60
PSA node: 0x86000FE0
PSA leaf from PSA node: 0x78003F80
Leaf FCR 1, psa_node 0x78003F80 found 1 deep
Prefix w.x.y.z
Leaf FCR 1, psa_node 0x78003F80 found 1 deep
default psa ip loadbalance
(hw rpfmask 0x80000000)
16 paths (hw maxpath 0)
Hash 1,3,5,7,9,11,13,15: psa adjacency: 0x700C8D40 (hw_adj
0x700C7D80)
[0-7] loq ABAB mtu 4 (Drop) oq BABA ai 0 oi 00000000 oacl
FFFF (encaps size 0)
punt gather 210 (bufhdr size 32 Punt profile 16)
counters 15458387815 bytes, 134893531 pkts; reported 0 bytes,
0 pkts.
Drop Adjacency
a.b.c.d
-> w.x.y.z
hash: 5
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
55
What Else?
Act
1. Scripts and Syslog
2. Test fix in lab
3. Interface usage graphs
4. Apply change management
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
56
Scripts and Syslog information
Very useful when time is limited
Large amount of Data
Across many devices
Check for any configuration changes made
Track any physical events in the network
Helps in isolating trigger conditions
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
57
Lab Setup and Usage Graphs
Root cause not found in production
Problem cleared itself
Large business impact
Monitored for network status
Produce possible causes
Pin point fault location in lifecycle
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
58
Enhanced Object Tracking and IPSLA to
avoid “black holes”
Cable
Fa0/0
Internet
LAN
Fa0/1
DSL
ip sla 99
icmp-echo <dstip> source-interface Fa0/0
timeout 1000
threshold 1000
frequency 2
ip sla schedule 1 life forever start-time now
!
access-list 101 permit icmp any host <dstip>
!
route-map track-primary-if permit 10
match ip address 101
set interface Fa0/0
set default interface Fa0/0
ip local policy route-map track-primary-if
!
track 1 rtr 99 reachability
delay down 10 up 10
!
ip route 0.0.0.0 0.0.0.0 Fa0/0 track 1
ip route 0.0.0.0 0.0.0.0 Fa0/1 200
http://www.cisco.com/en/US/docs/ios/12_4t/ip_appl/configuration/guide/taipbtrk.html#wp1054537
http://www.cisco.com/web/about/ac123/ac114/downloads/packet/packet/apr04/pdfs/apr04.pdf
(pages 9-12,88)
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
59
Change Management
Develop change plans
Prepare your team
Notify Stakeholders
Implement change plans and document
Assess gaps
Implement corrective actions
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
60
Third Problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
61
Third Problem
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
62
What information do we have?
Assess
Concerns:
1. Router’s User Interface cannot be accessed.
2. At risk of missing the provisioning deadline as a result
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
63
What information do we have?
Acquire- Environmental
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
64
What information do we have?
Act
1. Need to “see” the symptom that the user
experiences.
2. Need to understand the network topology
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
65
WebEx – Information sharing
http://www.cisco.com/web/products/webex/index.html
http://www.webex.com/index.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
66
Screen Shots – Information sharing
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
67
Document the network – logical/physical
R3
871
Lab
SW2
3750G
Internet
R2
3845
Home VPN User
PC 1
R1
ASA
2851
5520
SW1
4507R
Corporate
AD
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
ACS
R1
68
What information do we have?
Assess
Concerns:
1. Router “SDM” interface cannot be accessed.
“The page cannot be displayed”
2. At risk of missing the provisioning deadline as a result
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
69
What information do we have?
Acquire- Environmental
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
70
What information do we have?
Acquire - Lifecycle
Relevant Lifecycles:
1. Packet flow along the network lifecycle
2. TCP setup
3. HTTP Get/Reply lifecycle
4. Network device packet processing
Facts:
?
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
71
What information do we have?
Act
1. Get access to the network devices to run show
commands, debugs etc.
2. Need to look at the traffic at other points in the
network.
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
72
Network Topology Diagram
R3
871
Lab
SW2
3750G
Here
Here
Internet
R2
3845
Home VPN User
PC 1
R1
ASA
2851
5520
SW1
4507R
Corporate
AD
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
ACS
R1
73
Console access to devices
Methods:
Access server (reverse telnet)
Cross connecting the AUX port from one device to the
neighbouring CON port for reverse telnet
Send someone onsite to physically connect to the
console
Once on the device, you might try doing a
“debug ip packet <acl>” on an acl that matches the
users traffic.
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
74
ASA Capture
Steps to capture:
1. Create an ACL to match traffic you are interested in.
access-list
access-list
access-list
access-list
tac_acl_in permit ip host <clientrealip> host <serverip>
tac_acl_in permit ip host <serverip> host <clientrealip>
tac_acl_out permit ip host <clientglobalip> host <serverip>
tac_acl_out permit ip host <serverip> host <clientglobalip>
2. Capture the ACL traffic.
capture tac_cap_out access-list tac_acl_out packet-length 1522 interface outside
capture tac_cap_in access-list tac_acl_in packet-length 1522 interface inside
3. Capture traffic dropped by the ASP drop feature
capture <cap_name_3> type asp-drop all
4. Capture ARP traffic
capture <cap_name_4> ethernet-type arp interface inside
capture <cap_name_5> ethernet-type arp interface outside
“capture” command supported in 6.2(1) onwards
http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c1.html#wp2090739
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
75
ASA Capture cont…
Viewing the capture information:
a) ASA CLI:
⇒
show capture <capture name>
b) Packet Decoder:
1. Download the capture file via FTP/TFTP:
copy /pcap capture:<src_cap_name> tftp://10.1.1.10/<dst_capt_name>
2. Download the capture file via HTTP/HTTPS:
https://<asa-ip-address>/capture/<src_cap_name>/pcap
Note: You may need to allow https access to download the files:
http server enable
http <ip address_and_mask_of_workstation> <interface_name>
http <webclientip> 255.255.255.255 inside
3. Now just open the capture up in Ethereal/Wireshark!
“copy capture” command supported in 7.0(1) onwards
http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/c4.html#wp2104187
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
76
ASA Packet tracer
a) Enable Packet Tracing via the CLI:
syntax: packet-tracer input <interface> <protocol> <src_ip> <src_port> <dst_ip>
<dst_port> [detail]
example: packet-tracer input outside tcp 10.66.64.254 1 10.66.76.45 http
“packet-tracer” command supported in 7.2(1) onwards
http://www.cisco.com/en/US/docs/security/asa/asa80/command/reference/p.html
#wp1830068
Result:
input-interface: outside
input-status: up
input-line-status: up
output-interface: outside
output-status: up
output-line-status: up
Action: drop
Drop-reason: (acl-drop) Flow is denied by
configured rule
Phase: 1
Type: CAPTURE
Subtype:
Result: ALLOW
Config:
Additional Information:
Phase: 2
Type: ACCESS-LIST
Subtype:
Result: DROP
Config: Implicit Rule
Additional Information:
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
77
ASA Packet tracer cont…
b) Enable Packet Tracking via ASDM (GUI):
Step 1 In the main ASDM application window, choose Tools > Packet Tracer.
Step 2 Specify the following
- source interface, protocol type, source address, source port, destination IP address destination port
Step 3 Click Start to trace the packet.
=> The Information Display Area
shows detailed messages about the
packet trace.
The packet-tracer feature was
added to the ASDM in software
version 5.2(1)
http://www.cisco.com/en/US/docs/
security/asa/asa80/asdm60/user/g
uide/tools.html#wp1536158
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
78
What information do we have?
Acquire - Lifecycle
Facts:
1. Packets pass through firewall and reach router
2. No reply packets from router
3. TCP SYN sent by client, no SYN,ACK returned.
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
79
More Video
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
80
What information do we have?
Analyse
Possible Causes:
1. Router is blocking access
2. HTTP server is not configured correctly
3. Router is not responding due to load
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
81
What information do we have?
Act
1. Check router configuration
2. Monitor router performance
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
82
Monitor Router Performance
Troubleshooting CPU/Memory Utilization on Routers
show proc cpu sorted
show proc mem sorted
show interfaces [<interface name>]
http://www.cisco.com/warp/customer/63/highcpu.html
http://www.cisco.com/warp/customer/63/highcpu_interrupts.html
Memory Leak Detector
show memory debug leaks
http://www.cisco.com/en/US/docs/ios/12_3t/12_3t7/feature/guide/gtmleakd.html
http://www.cisco.com/en/US/docs/ios/12_4/cfg_fund/configuration/guide/hmleakd.html
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
83
Root Cause
1. TCP SYN (DoS) attack
2. High CPU due to IP Input
R3#sh proc cpu | ex 0.00
CPU utilization for five seconds: 71%/14%; one minute: 67%; five minutes:64%
PID Runtime(ms)
Invoked
uSecs
5Sec
1Min
5Min TTY Process
3
12816372 161039479
79 0.47% 0.40% 0.24%
0 OSPF Hello
155
218176261 976798810
223 45.68% 45.44% 43.90%
0 IP Input
206
41532046 155427986
267 1.91% 1.70% 0.87%
0 OSPF Router 1
230
24436795
536684
45533 4.15% 0.50% 0.34%
0 BGP Scanner
R3# debug ip packet <acl>
*Mar 3 03:54:40.436: IP: s=192.168.40.53 (Ethernet0/1), d=144.254.2.204
(Ethernet0/0), g=10.200.40.1, len 44, forward
*Mar 3 03:54:40.440: TCP src=11004, dst=53,
seq=280872555, ack=0, win=4128 SYN
Troubleshooting High CPU Utilization in IP Input Process
http://www.cisco.com/en/US/products/hw/routers/ps359/products_tech_note09186a00801c2a
f3.shtml
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
84
What’s next?
Act
1. Action to Fix: Limit connections on ASA, rate-limit
communications (apply a config change)
Sample Configuration:
http://www.ciscotaccc.com/kaidara-advisor/security/showcase?case=K18407732
2. Other preventative actions:
- ERM/EEM to monitor CPU and send email
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
85
Config Management
Keep regular archives of config
When changing config(s)
- test in a lab environment and/or off peak
- deploy config in a staggered form
Out of band access in case of connectivity loss
IOS Configuration Archive/Replace/Rollback
Store, organise and manage archives
No need to power cycle when rolling back
Config locking to prevent conflicting multiple accesses
12.3T and onwards
http://www.cisco.com/en/US/docs/ios/12_3t/12_3t7/feature/guide/gtrollbk.html#wp106
6709
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
86
Embedded Resource Manager (ERM) and
Embedded Event Manager (EEM) to
Notify of CPU threshold events
1. Configure ERM CPU event thresholds:
(config t)
resource policy
policy system-global-cpu global
system
cpu total
critical rising 90 interval 12 falling 20 interval 10
major rising 70 interval 12 falling 15 interval 10
minor rising 60 interval 12 falling 10 interval 10
2. Configure EEM to react to the ERM events:
event manager applet erm_cpu
event resource policy system-global-cpu
action 1.0 syslog msg “CPU $_resource_level alarm:
$_resource_current_value percent“
action 1.1 mail from [email protected] to [email protected] subject “CPU
$_resource_level alarm: $_resource_current_value percent” body “”
server email.xyz.com
http://www.cisco.com/en/US/docs/ios/12_4/netmgmt/configuration/guide/nm_erm.html
http://www.cisco.com/en/US/docs/ios/12_4t/12_4t2/ht_eem.html#wp1052497
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
87
Smart Call Home?
Interactive Technical Services
Customer Notification
Device and Message Reports
Exceptions/Fault Analysis
Customer
Internet
TAC
3
Secure Transport
Messages Received:
Call Home
Diagnostics
Environmental
Syslog
Inventory and
Configuration
© 2006 Cisco Systems, Inc. All rights reserved.
1
Service Request
Tracking System
2
Call
Home DB
IOS 12.2(33)SXH
Presentation_ID
Automated
Diagnosis
Capability
Cisco Confidential
88
Summary
People, Process and Technology
Small steps through best practices
Take action now!
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
89
End
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
90
Presentation_ID
© 2006 Cisco Systems, Inc. All rights reserved.
Cisco Confidential
91