CCIE RS – Written – Network Principles – Use IOS Troubleshooting Tools

Use IOS Troubleshooting Tools

Multiple troubleshooting tools are built into IOS.

  • show – monitor normal behavior and isolate problems
    • version – system hardware, software version, uptime, boot image
    • running-config – current configuration
    • startup-config – config stored in NVRAM
    • interface – interface statistics, bandwidth, errors
  • debug – assist in isolating a protocol and configuration problem
  • ping – determine connectivity
  • trace – show the path packets are taking

debug, conditional debug

Debugs must be turned on using the debug command. To show running debugs – show debug

Debugs are sent to console by default (no logging console – to turn off). User terminal monitor if you are remotely connected into the device. 

Turn off debug – R1#undebug all

Conditional debug – add parameters around what debugs you want displayed to the console.

Stacking multiple debug conditions will generate output if at least 1 condition is met.

R1#debug condition ?
called called number
callingcalling
cplCisco Provisioning Language debugging
glbp interface group
interfaceinterface
ip IP address
mac-addressMAC address
match-list apply the match-list
profileMedia Services Profile
standbyinterface group
username username
vcid VC ID
vrfVirtual Routing and Forwarding
xconnect Xconnect conditional debugging on segment pair

R1#debug condition ip 1.1.1.1 
Condition 1 set
R1#debug condition interface gi0/0
Condition 2 set
R1#
R1#
R1#sho debug

Condition 1: ip 1.1.1.1 (0 flags triggered)
Condition 2: interface Gi0/0 (1 flags triggered)
Flags: Gi0/0

R1#

Above condition will generate debugs messages for anything containing the ip 1.1.1.1 or interface gi0/0

ping, traceroute with extended options

Ping – common method for troubleshooting accessibility to a device

  • Uses ICMP echo
    • Tells if host is active / inactive
    • RTD to host
    • Packet Loss

Issues if cannot ping

  • Routing issue
  • Interface down
  • ACL
  • ARP issue
  • Delay
  • Source Address
  • High Input Queue drops

Traceroute – Discover the routers a packet takes to a destination

  • Sequence of UDP datagrams on an invalid port
  • 3 datagrams sent with TTL of 1
  • TTL of 1 causes datagram to timeout and first hop responds with ICMP “Time Exceeded Message (TEM)
  • Process continues increasing TTL by 1 each step until packets reach the destination
  • Destination responds with ICMP Port Unreachable message, indicates traceroute is finished
Traceroute Text Characters

Traceroute Text Characters

Embedded packet capture

  • Onboard packet capture facility
  • Consumes CPU and memory resources during its operation
  • Export captures via TFTP, FTP and local disk
  • Define a buffer size and type (circular or linear) and max number of bytes of each packet capture
  • Capture can be throttled using admin controls
    • Filter packets with ACL
    • Specify max packet capture rate or specify sampling interval
  • Benefits
    • Ability to capture IPv4 and IPv6 in CEF path
    • Flexible method to specify capture buffer parameters
    • Filter captured packets
    • Method to decode data packets
    • Facilty to export capture (PCAP)
    • Extensible infrastructure for enabling capture points

Performance monitor

Enables to be able to monitor the flow of traffic in the network. Similar to netflow.

Pre-req for configuration

  • IPv4
    • routing and CEF must be configured / enabled
  • IPv6
    • ipv6 cef must be enabled

Can monitor a long list of traffic – https://www.cisco.com/c/en/us/td/docs/ios-xml/ios/media_monitoring/configuration/15-mt/mm-15-mt-book/mm-pasv-mon.html

Configuration Components

  1. Interface – attach performance monitor to interface – service-policy type performance-monitor
  2. Policy – Associate with flow monitor – policy-map type performance-monitor
  3. Class – filtering criteria – class-map
  4. Flow Monitor – Associated with flow record and optional flow monitor – flow monitor performance-monitor
  5. Flow Record – Specify match and collect – flow record type performance-monitor
  6. Flow Exporter – Specify the destination for exporting traffic

show performance monitor status

 

Router#show policy-map type performance-monitor 
Policy Map type performance-monitor PM_FLOW_MONITOR
Class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
react status: inactive
Router#
Router#sh run | s flow
flow record type performance-monitor FLOW_RECORD
 match ipv4 destination address
 match transport destination-port
 collect application media event
 collect counter bytes long
 collect ipv4 dscp
 collect monitor event
 collect routing forwarding-status
 collect timestamp interval
 collect transport packets expected counter
 collect flow direction
flow exporter FLOW_EXPORT
 description *** Export Flows ***
 destination 2.2.2.2
 source GigabitEthernet0/0
 output-features
 dscp 46
 transport udp 650
flow monitor type performance-monitor FLOW_MONITOR
 description *** FLOW MONITOR ***
 exporter FLOW_EXPORT
flow monitor FLOW_MONITOR
Router#
Router#sh run | s class
class-map match-all CM_FLOW_MONITOR
 match any 
 class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
Router#
Router#
Router#sh run | s policy
policy-map type performance-monitor PM_FLOW_MONITOR
 class CM_FLOW_MONITOR
flow monitor FLOW_MONITOR
Router#

Apply troubleshooting methodologies

Diagnose the root cause of networking issue (analyze symptoms, identify and describe root cause)

Design and implement valid solutions according to constraints

Verify and monitor resolution


Interpret packet capture

Using Wireshark trace analyzer

https://www.wireshark.org/docs/wsug_html/

Personal writing from experience – you need to understand traffic flows. Knowing protocol basics and using different filters in wireshark to remove the noise in the capture. Would love feedback for this section on your experience with wireshark.

Using IOS embedded packet capture

Router#monitor capture buffer CAPTURE size 256 max-size 100 circular 
Router#monitor capture point ip cef CAPTURE_POINT gi0/0 ?
bothcapture ingress and egress
incapture on ingress
out capture on egress
removeremove capture point
Router#monitor capture point ip cef CAPTURE_POINT gi0/0 both
Router#monitor capture point associate CAPTURE_POINT CAPTURE
*Aug 18 04:12:11.060: %BUFCAP-6-CREATE: Capture Point CAPTURE_POINT created.
Router#monitor capture point associate CAPTURE_POINT CAPTURE
Router#
Router#monitor capture point start CAPTURE_POINT
Router#
Router#
Router#
Router#
*Aug 18 04:12:29.789: %BUFCAP-6-ENABLE: Capture Point CAPTURE_POINT enabled.
Router#show monitor capture buffer CAPTURE dump

Router#monitor capture point stop all 
Router#
*Aug 18 04:15:23.960: %BUFCAP-6-DISABLE: Capture Point CAPTURE_POINT disabled.
Router#

Packet capture can be exported to TFTP server to look at the capture in wireshark

Router#monitor capture buffer CAPTURE export ?
  flash0:  Location to dump buffer
  flash1:  Location to dump buffer
  flash2:  Location to dump buffer
  flash3:  Location to dump buffer
  flash:   Location to dump buffer
  ftp:     Location to dump buffer
  http:    Location to dump buffer
  https:   Location to dump buffer
  pram:    Location to dump buffer
  rcp:     Location to dump buffer
  scp:     Location to dump buffer
  snmp:    Location to dump buffer
  tftp:    Location to dump buffer

Router#monitor capture buffer CAPTURE export 

CCIE RS – Written – L2 – Implement and Troubleshoot Layer 2 Protocols

Implement and Troubleshoot Layer 2 Protocols


CDP – Cisco Discovery Protocol

  • Propriety, runs on all Cisco equipment
  • Used to obtain protocol address of neighboring devices and discover the platform of those devices
  • Media and protocol-independent
  • Sends periodic messages – advertisements, every 60 seconds
  • Multicast Address: 01-00-0C-CC-CC-CC
  • CDPv2 is latest version

TLV – Type Length Value: Embedded in CDP advertisements

  • Device-ID TLV: Identifies the device name in the form of a character string.
  • Address TLV: Contains a list of network addresses of both receiving and sending devices.
  • Port-ID TLV: Identifies the port on which the CDP packet is sent.
  • Capabilities TLV: Describes the functional capability for the device in the form of a device type, for example, a switch.
  • Version TLV: Contains information about the software release version on which the device is running.
  • Platform TLV: Describes the hardware platform name of the device, for example, Cisco 4500.
  • IP Network Prefix TLV: Contains a list of network prefixes to which the sending device can forward IP packets. This information is in the form of the interface protocol and port number, for example, Eth 1/0.
  • VTP Management Domain TLV: Advertises the system’s configured VTP management domain name-string. Used by network operators to verify VTP domain configuration in adjacent network nodes.
  • Native VLAN TLV: Indicates, per interface, the assumed VLAN for untagged packets on the interface. CDP learns the native VLAN for an interface. This feature is implemented only for interfaces that support the IEEE 802.1Q protocol.
  • Full/Half Duplex TLV: Indicates status (duplex configuration) of CDP broadcast interface. Used by network operators to diagnose connectivity problems between adjacent network elements.

http://www.cisco.com/c/en/us/td/docs/ios/12_2/configfun/configuration/guide/ffun_c/fcf015.html


LLDP – Link Layer Discovery Protocol

  • Open Standard – IEEE 802.1AB
  • Switch supports basic management TLV’s
    • Port description TLV
    • System name TLV
    • System description TLV
    • System capabilities TLV
    •  Management address TLV
    • These organizationally specific LLDP TLVs are also advertised to support LLDP-MED.
      • Port VLAN ID TLV ((IEEE 802.1 organizationally specific TLVs)
      • MAC/PHY configuration/status TLV(IEEE 802.3 organizationally specific TLVs)

Link Layer Discovery Protocol – Media Endpoint Discovery (LLDP-MED)

  • Extension of LLDP that operates between endpoint devices (IP Phones)
  • TLVs
    • Capabilities
    • Network Policy
    • Power Management
    • Inventory
    • Location

UDLD – UniDirectional Link Detection

  • Cisco Proprietary
  • Allows devices connected through fiber or copper to monitor the physical configuration of the cables and detect when a unidirectional link exists
  • Layer 2 protocol that works with layer 1 protocol to determine physical status of a link
  • RFC 5171

Aggressive Mode

  • Disabled by default
  • Configure only on p2p links
  • Bidirectional lnik send UDLD messages, if stops receiving, UDLD tries to reestablish connection. After 8 retriesthe port is disabled
    • Error Disables

Normal Mode

  • Default
  • Does not disable the port when unidirectional link is detected

Configuration Defaults

  • UDLD global enable state — Globally disabled
  • UDLD aggressive mode — Disabled
  • UDLD per-port enable state for fiber-optic media — Enabled on all Ethernet fiber-optic LAN ports
  • UDLD per-port enable state for twisted-pair (copper) media — Disabled on all Ethernet 10/100 and 1000BASE-TX LAN ports

CCIE RS – Written – Network Principles – Evaluate Proposed Changes to a network

Evaluate Proposed Changes to a network

Before making changes to the network you’ll want to know what the current state is. Capture appropriate show output depending on the type of change being made. Ensure network diagrams are up to date (physical and logical). Backup configurations. 

If possible test the changes in a lab environment that directly mimics the production network. Write out a plan for the flow of changes and validations after the change is made to ensure the expected result occurred.


Changes to routing protocol parameters

Each routing protocol has a set of parameters can be changed depending on a technical or business requirement. The parameters for each routing protocol will be gone into in depth in future blog posts.

Before making the changes, check the routing table (show route), and check specifics about the routing protocol your about to change.


Migrate parts of a network to IPv6

IPv4 and IPv6 can exist in the network at the same time (as long as the device supports IPv6). IPv6 has it’s own routing table and if a device has both IPv4 and IPv6 addressing, the IPv6 address will be preferred.

  • Dual Stack
    • Requires infrastructure to support IPv4 and IPv6
    • Applications choose between IPv4 and IPv6 based on response to DNS requests
  • Tunneling
    • Encapsulates IPv6 traffic within an IPv4 packet
    • Used between IPv6 sites over an IPv4 backbone
    • Tunneling Techniques
      • ISATAP
      • Teredo
      • 6PE
      • 6VPE
      • mGRE v6 over v4
    • Manual or automatically configured
  • Translation
    • NAT between v4 to v6, v6 to v4

Details about configurations will be in future blog posts 


Routing protocol migration

Moving between routing protocols can be done in a couple ways

  1. Modify administrative distance
  2. Redistribution 

Multiple routing protocols can be running at the same time, but only 1 route to the destination will be put into the routing table (disregarding multiple paths). Administrative distance is one of the criteria that says which routing protocol is more trustworthy than another.

Redistribution will be explained in future blog posts


Adding multicast support

To add IPv4 multicast support to a router you’ll need to configure multicast routing and enable PIM on interfaces that will be participating in the multicast routing path. 

 

Configuration of multicast will be covered in future blog posts


Migrate spanning tree protocol

Specifics of the different STP types will be explained in future blog posts.

PVST+ to Rapid-PVST+

  • Rapid uses the same BPDU format as PVST+
  • BackboneFast and UplinkFast get disabled when Rapid-PVST+ is configured
  • Start at the access switches and work your way up to the core
  • Clean up configuration
  • Do these changes during a maintenance window because a disruption will occur
  • Verify changes and make sure STP is behaving the way you intended

Rapid-PVST+ to MST

  • Identify p2p and edge ports (portfast)
  • Map vlans to instances
  • Place as many switches as possible into a single region
  • Start at core and work your way to the access

Evaluate impact of new traffic on existing QoS design

Before adding more traffic to an existing QOS queue you’ll want to know the behavior of the traffic and how it needs to be treated. Is the traffic UDP or TCP based? Is the existing queue policed or shaped? Is bandwidth over utilized? 

CCIE RS – Written – Network Principles – Explain General Network Challenges

Explain General Network Challenges


Unicast Flooding

  • Occurs when destination MAC address of the packet is not in the L2 forwarding table of the switch
  • Packet is forwarded out all forwarding ports in that VLAN except the originating port packet was received on
  • Reasons MAC address many not be known
    • Asymmetric routing
    • STP Topology Changes
    • Forwarding Table Overflow

How to Detect

  • No special commands used to detect flooding
  • Unicast Flood Protection – allows switch to monitor amount of unicast flooding per VLAN and take a specified action if flooding exceeds specified amount
  • Actions
    • Syslog
      • %UNICAST_FLOOD-4-DETECTED: Host 0000.0000.2100 on vlan 1 is flooding
        to an unknown unicast destination at a rate greater than/equal to 1 Kfps
    • Limit or shutdown VLAN
  • Show mac-address-table unicast-flood
  • Captures during time of slowdown / outage
    • See packets that are not destined for the workstation

Asymmetric Routing

  • Packets follow different paths depending on the traffic direction
  • Approaches to limit flooding
    • Bring routers ARP timeout and switches forwarding table aging time close to each other
      • Scenario – L3 switch with HSRP configured to LB different vlans
  • Traffic does not stop flooding even after the destination replies

How to detect

  • Follow the routing path
  • Traceroutes

STP Topology changes

TCN – Topology Change Notification

  • Designed to correct the forwarding table after forwarding table changes
  • Used to avoid connectivity outages. If a port goes down and the destination is available over a different port.
  • TCN operates by shortening the forwarding table aging time and if the MAC address is not relearned flooding will occur
  • TCNs are triggered by a port transitioning to or from the forwarding state
  • Issues arise when TCNs are occurring repeatedly with short intervals
    • Switch is constantly fast-aging forwarding table so flooding will be nearly constant
  • Limit TCN’s with the use of portfast

Forwarding Table Overflow

  • New MAC addresses cannot be learned and packets destined to such MAC are flooded until space becomes available
  • Can be caused by an attack on the network where a host starts generating frames sourced with different MAC addresses
  • Detected by examining the switches forwarding table
    • MAC addresses will point to a single port
  • Prevent by limiting the number of MAC addresses that can be learned on an untrusted port
    • See Switchport Security
  • High amount of packets or normal packets with high number of different source MAC addresses

Out of Order Packets

Using per-packet load balancing to share the traffic load across available paths to a given destination can lead to out-of-order packets for a given data flow.


Impact of Microbursts

  • Patterns or spikes of traffic causing interfaces to ne temporally oversubscribed and drop traffic
  • Typically buffers can handle bursts of traffic. This is in excess the buffers can handle
  • Detect by looking at Total Output Drops under show interface
    • Drops increment but utilization stays the same
  • Burst of traffic occur in microseconds
  • Show up as ignores and/or overruns
    • Input errors on show interface
  • Detect by capturing traffic over a long period of time