CCIE RS – Written – Network Principles – Use IOS Troubleshooting Tools

Use IOS Troubleshooting Tools

Multiple troubleshooting tools are built into IOS.

  • show – monitor normal behavior and isolate problems
    • version – system hardware, software version, uptime, boot image
    • running-config – current configuration
    • startup-config – config stored in NVRAM
    • interface – interface statistics, bandwidth, errors
  • debug – assist in isolating a protocol and configuration problem
  • ping – determine connectivity
  • trace – show the path packets are taking

debug, conditional debug

Debugs must be turned on using the debug command. To show running debugs – show debug

Debugs are sent to console by default (no logging console – to turn off). User terminal monitor if you are remotely connected into the device. 

Turn off debug – R1#undebug all

Conditional debug – add parameters around what debugs you want displayed to the console.

Stacking multiple debug conditions will generate output if at least 1 condition is met.

R1#debug condition ?
called called number
callingcalling
cplCisco Provisioning Language debugging
glbp interface group
interfaceinterface
ip IP address
mac-addressMAC address
match-list apply the match-list
profileMedia Services Profile
standbyinterface group
username username
vcid VC ID
vrfVirtual Routing and Forwarding
xconnect Xconnect conditional debugging on segment pair

R1#debug condition ip 1.1.1.1 
Condition 1 set
R1#debug condition interface gi0/0
Condition 2 set
R1#
R1#
R1#sho debug

Condition 1: ip 1.1.1.1 (0 flags triggered)
Condition 2: interface Gi0/0 (1 flags triggered)
Flags: Gi0/0

R1#

Above condition will generate debugs messages for anything containing the ip 1.1.1.1 or interface gi0/0

ping, traceroute with extended options

Ping – common method for troubleshooting accessibility to a device

  • Uses ICMP echo
    • Tells if host is active / inactive
    • RTD to host
    • Packet Loss

Issues if cannot ping

  • Routing issue
  • Interface down
  • ACL
  • ARP issue
  • Delay
  • Source Address
  • High Input Queue drops

Traceroute – Discover the routers a packet takes to a destination

  • Sequence of UDP datagrams on an invalid port
  • 3 datagrams sent with TTL of 1
  • TTL of 1 causes datagram to timeout and first hop responds with ICMP “Time Exceeded Message (TEM)
  • Process continues increasing TTL by 1 each step until packets reach the destination
  • Destination responds with ICMP Port Unreachable message, indicates traceroute is finished
Traceroute Text Characters

Traceroute Text Characters

Embedded packet capture

  • Onboard packet capture facility
  • Consumes CPU and memory resources during its operation
  • Export captures via TFTP, FTP and local disk
  • Define a buffer size and type (circular or linear) and max number of bytes of each packet capture
  • Capture can be throttled using admin controls
    • Filter packets with ACL
    • Specify max packet capture rate or specify sampling interval
  • Benefits
    • Ability to capture IPv4 and IPv6 in CEF path
    • Flexible method to specify capture buffer parameters
    • Filter captured packets
    • Method to decode data packets
    • Facilty to export capture (PCAP)
    • Extensible infrastructure for enabling capture points

Performance monitor

Enables to be able to monitor the flow of traffic in the network. Similar to netflow.

Pre-req for configuration

  • IPv4
    • routing and CEF must be configured / enabled
  • IPv6
    • ipv6 cef must be enabled

Can monitor a long list of traffic – https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/media_monitoring/configuration/15-mt/mm-15-mt-book/mm-pasv-mon.html

Configuration Components

  1. Interface – attach performance monitor to interface – service-policy type performance-monitor
  2. Policy – Associate with flow monitor – policy-map type performance-monitor
  3. Class – filtering criteria – class-map
  4. Flow Monitor – Associated with flow record and optional flow monitor – flow monitor performance-monitor
  5. Flow Record – Specify match and collect – flow record type performance-monitor
  6. Flow Exporter – Specify the destination for exporting traffic

show performance monitor status

 

Router#show policy-map type performance-monitor 
Policy Map type performance-monitor PM_FLOW_MONITOR
Class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
react status: inactive
Router#
Router#sh run | s flow
flow record type performance-monitor FLOW_RECORD
 match ipv4 destination address
 match transport destination-port
 collect application media event
 collect counter bytes long
 collect ipv4 dscp
 collect monitor event
 collect routing forwarding-status
 collect timestamp interval
 collect transport packets expected counter
 collect flow direction
flow exporter FLOW_EXPORT
 description *** Export Flows ***
 destination 2.2.2.2
 source GigabitEthernet0/0
 output-features
 dscp 46
 transport udp 650
flow monitor type performance-monitor FLOW_MONITOR
 description *** FLOW MONITOR ***
 exporter FLOW_EXPORT
flow monitor FLOW_MONITOR
Router#
Router#sh run | s class
class-map match-all CM_FLOW_MONITOR
 match any 
 class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
Router#
Router#
Router#sh run | s policy
policy-map type performance-monitor PM_FLOW_MONITOR
 class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
Router#

Apply troubleshooting methodologies

Diagnose the root cause of networking issue (analyze symptoms, identify and describe root cause)

Design and implement valid solutions according to constraints

Verify and monitor resolution


Interpret packet capture

Using Wireshark trace analyzer

https://www.wireshark.org/docs/wsug_html/

Personal writing from experience – you need to understand traffic flows. Knowing protocol basics and using different filters in wireshark to remove the noise in the capture. Would love feedback for this section on your experience with wireshark.

Using IOS embedded packet capture

Router#monitor capture buffer CAPTURE size 256 max-size 100 circular 
Router#monitor capture point ip cef CAPTURE_POINT gi0/0 ?
bothcapture ingress and egress
incapture on ingress
out capture on egress
removeremove capture point
Router#monitor capture point ip cef CAPTURE_POINT gi0/0 both
Router#monitor capture point associate CAPTURE_POINT CAPTURE
*Aug 18 04:12:11.060: %BUFCAP-6-CREATE: Capture Point CAPTURE_POINT created.
Router#monitor capture point associate CAPTURE_POINT CAPTURE
Router#
Router#monitor capture point start CAPTURE_POINT
Router#
Router#
Router#
Router#
*Aug 18 04:12:29.789: %BUFCAP-6-ENABLE: Capture Point CAPTURE_POINT enabled.
Router#show monitor capture buffer CAPTURE dump

Router#monitor capture point stop all 
Router#
*Aug 18 04:15:23.960: %BUFCAP-6-DISABLE: Capture Point CAPTURE_POINT disabled.
Router#

Packet capture can be exported to TFTP server to look at the capture in wireshark

Router#monitor capture buffer CAPTURE export ?
  flash0:  Location to dump buffer
  flash1:  Location to dump buffer
  flash2:  Location to dump buffer
  flash3:  Location to dump buffer
  flash:   Location to dump buffer
  ftp:     Location to dump buffer
  http:    Location to dump buffer
  https:   Location to dump buffer
  pram:    Location to dump buffer
  rcp:     Location to dump buffer
  scp:     Location to dump buffer
  snmp:    Location to dump buffer
  tftp:    Location to dump buffer

Router#monitor capture buffer CAPTURE export 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.