CCIE RS – Written – Network Principles – Explain TCP Operations

Explain TCP Operations

Protocol that ensures reliability in a transmission with minimum loss of packets

  • Duties
    • Assure packets maintain the right order
    • Error Checking
    • Delay is kept at an acceptable level
    • Prevent possibility of packet duplication
  • Ensure data received is consistent, in order, complete and smooth
  • OSI – Transport Layer (4)
    • Works before IP
    • Data gets bundled inside of TCP packets before sending to IP which encapsulates into IP packets

TCP Segment Structure

  • PDU – Protocol Data Unit
  • Consists of Header and Data section
    • Header – 10 mandatory fields
      • Source port – 16 bits – ID’s sending port
      • Destination Port – 16 bits – ID’s receiving port
      • Sequence Number – 32 bits – used for 3 way handshake, SYN flag
      • Acknowledgement Number – 32 bits
      • Data offset – 4 bits –
      • Reserved – 3 bits
      • Flags – 9 bits (Control bits)
      • Window Size – 16 bits – size of the receive window. Specifies number of window size units (bytes) sender on the segment is willing to receive
      • Checksum (16 bits) – error checking of header and data
      • Urgent point – 16 bits
      • Options – Variable 0-320 bit divisible by 32
      • Padding – Ensure TCP header ends and data begins on a 32 bit boundart. All zeros

Connection Establishment

  • 3 way handshake – Attempt connection between client and server before sending data
    • SYN – Active open is performed by the client to the server
    • SYN-ACK – Responseto SYN from server to client
    • ACK – Final response from client to server that SYN-ACK was received
  • Full duplex communication is established


Determine the size of the network path between 2 IP hosts

  • Goal is to avoid fragmentation
  • Intended for routers. All modern OS’s use it on endpoints


  • Works by setting DF bit in outgoing IP headers
  • Any device along path with smaller MTU sends ICMP – Fragmentation Needed (Type 3, Code 4)
  • Process is continued until smallest MTU is found to send packets without fragmentation


Determine the size of the network path between 2 IP hosts

  • Explicitly delegated to endpoints
  • Routers do not support fragmentation
  • ICMPv6 – Packet Too Big (Type 2)

MSS – Maximum Segment Size

  • Largest amount of data (in bytes) a host can receive in a single TCP segment
    • Does not count TCP or IP header
  • Defined to be the relevant IP datagram minus 40
  • Min MTU – 40 = MSS
  • IPv4 hosts required to handle MSS of 536 (=576 – 20 – 20)
  • IPv6 hosts required to handle 1220 (=1280 – 40 – 20)
  • MSS specified as TCP option
    • Send In SYN packet during TCP handshake
    • MSS cannot be changed after connection is established


Time interval between point A and point B. Product of delay from physical distance packets traverse a medium

  • Measured
    • One-way – Time from source sending a packet to destination receiving the packet
    • Round-trip Delay – Time from Source to Destination and Destination back to Source
  • Ping can provide latency, but isn’t fully accurate as ICMP can be treated differently if traffic shaping is applied
  • Accurate measurements can come from specific software


TCP Windowing

  • Amount of unacknowledged data that can be in transit at any given time
  • Referred to as Window Size – 16 bit field in TCP header

Window Scaling

Bandwidth Delay Product


Global Synchronization

Can occur during periods of congestion because each sender will reduce their transmission rate at the same time when packet loss occurs

Simplest queuing technique

  • Tail Drop – Allow queue to fil to max size and then discard any packets until there is space again
  • Problem occurs when there is bursty traffic and the queue is full
    • Full queue results in high latency
    • Introduction of sudden burst of traffic may cause large number of established streams to lose packets simultaneously
  • Recovery Mechanism
    • TCP recovers from dropped packets which is interprets as congestion
    • Senders reduce sending rate for period of time
    • Known as slow start algorithm
  • Tail drop is leading cause of the problem
    • RED (Random Early Detection) and WRED (Weighted RED) reduce likeliness of global synchronization


3 Types of TCP options

  1. Option-Kind – 1 byte
  2. Option-Length – 1 byte
  3. Option-Data – variable

Sent in the SYN packet

CCIE RS – Written – Network Principles – Explain IP Operations

Explain IP Operations


  • Standardizes the way machines talk over the Internet or any IP network forward or route packets based on IP
  • IP protocols must implement own reliability procedures if required

ICMP – Internet Control Management Protocol

  • Protocol number 1
  • Designed to report small set of error conditions
  • Report wide variety of error conditions, feedback and testing capabilities
  • Each message uses a common format
  • Enables IP to perform addressing, datagram packing and routing by allowing encapsulated messages to be sent/received between IP devices
  • ICMP does not make IP reliable or ensure delivery
  • Summary of Message Types
    • 0Echo Reply
    • 3Destination Unreachable
    • 4Source Quench
    • 5Redirect
    • 8Echo
    • 11Time Exceeded
    • 12Parameter Problem
    • 13Timestamp
    • 14Timestamp Reply
    • 15Information Request
    • 16Information Reply

ICMP Unreachable

Generated by host or its inbound gateway to inform client that the destination is unreachable for some reason

  • Reasons for the message
    • Protocol or port is not active
    • Physical connection to host does not exist
    • Data must be fragmented and DF flag is on

Type 3 Destination Unreachable Error Messages

  • 6 codes contained in ICMP header describing unreachable condition
    • 0 – Network unreachable
    • 1 – Host unreachable
    • 2 – Protocol unreachable
    • 3 – Port unreachable
    • 4 – Fragmentation needed and Don’t Fragment (DF) bit set
    • 5 – Source route failed
  • ICMP Unreachable on IOS
    • Default to no more than once every half second

RFC 792

Config – Prevents router from sending unreachable messages

No icmp unreachable

ICMP Redirect

Requests data packets to be send on an alternative route

  • Mechanism for routers to convey routing information to a host
    • Inform host to update its routing table
  • Redirects are only sent by gateways
  • Redirect message – type 5

RFC 1122 –

A host SHOULD NOT send an ICMP Redirect message; Redirects are to be sent only by gateways. A host receiving a Redirect message MUST update its routing information accordingly.  Every host MUST be prepared to accept both Host and Network Redirects and to process them as described in Section below.

A Redirect message SHOULD be silently discarded if the new gateway address it specifies is not on the same connected (sub-) net through which the Redirect arrived [INTRO:2, Appendix A], or if the source of the Redirect is not the current first-hop gateway for the specified destination (see

Config – Disable on Interface

No icmp redirect

IPv4 Options

  • Convey additional information on the packet or on the way it should be processed
  • Processing header options leads to forwarding performance hit

Addressing specifics are defined in a later section in the blueprint. I’ll cover more on IPv4 there

IPv6 Extension Headers

2 Header Types

  • Main/Regular – Equivalent to IPv4
    • 40 Bytes
  • Extension Header
    • This is where options are set
    • RFC 2460
      • RFC2460 also recommends the order in which they should be chained in an IPv6 packet:
  1. IPv6 main header
  2. Hop-by-Hop Options header (if present, it MUST be the first one following the main/regular header)
  3. Destination Options header
  4. Routing header
  5. Fragment header
  6. Authentication header
  7. Encapsulating Security Payload header
  8. Destination Options header
  9. Upper-layer header

The only MUST requirement is that the Hop-by-Hop EH has to be the first one.

Commonly used Extension Headers

  • Hop-by-Hop EH is used for the support of Jumbo-grams or, with the Router Alert option, it is an integral part in the operation of MLD. Router Alert [3] is an integral part in the operations of IPv6 Multicast through Multicast Listener Discovery (MLD) and RSVP for IPv6.
    • Only EH that must be fully processed by at network devices
  • Destination EH is used in IPv6 Mobility as well as support of certain applications.
  • Routing EH is used in IPv6 Mobility and in Source Routing. It may be necessary to disable “IPv6 source routing” on routers to protect against DDoS.
  • Fragmentation EH is critical in support of communication using fragmented packets (in IPv6, the traffic source must do fragmentation-routers do not perform fragmentation of the packets they forward)
  • Mobility EH is used in support of Mobile IPv6 service
  • Authentication EH is similar in format and use to the IPv4 authentication header defined in RFC2402.
  • Encapsulating Security Payload EH is similar in format and use to the IPv4 ESP header defined in RFC2406. All information following the Encapsulating Security Header (ESH) is encrypted and for that reason, it is inaccessible to intermediary network devices. The ESH can be followed by an additional Destination Options EH and the upper layer datagram.

IPv4 Fragmentation

Breaking datagrams into smaller pieces so packets can pass through a link with a smaller MTU (Maximum Transmission Unit)

  • RFC 791 – Procedure for IP fragmentation, transmission and reassembly
  • Router has 2 options when it receives PDU larger than next hops MTU
    • Drop PDU and send ICMP message “Packet to Big”
    • Fragment the packet
  • Reassembly is intended to happen by receiving host but in practice is done by intermediate router
    • Ex. NAT may need to re-assemble fragments
  • Fragments can cause excessive retransmissions if there is packet loss
    • TCP must retransmit all fragments in order to recover the loss of a single fragment
  • Routers perform the fragmentation
  • Reassembled size up to 576 bytes

IPv6 Fragmentation

IPv6 hosts are required to determine Path MTU before sending packets

  • Guaranteed IPv6 packet smaller than or equal to 1280 bytes must be deliverable without fragmentation
  • Routers do NOT fragment the packets and drop packets larger than the MTU
  • Reassembled size to 1280 bytes
  • Larger than 1500 bytes are silently dropped

TTL – Time to Live

Time to live in seconds; as this field is decremented at each device the packet is processed by. This limits the amount of time a packet can live in the network. TTL is meant to help a packet from being sent for an infinite time through the network.

8 Bits in length 

If the TTL reaches 0, the device will drop the packet. TTL can also be known as hop limit.

If the TTL is exceeded the device responds back to the source with a ICMP type 11 – Time Exceeded

In IPv6, TTL has been renamed to Hop Limit



  • Sets maximum MTU size for IP packets on an interface
  • Minimum is 128 bytes – Depending on interface medium

Interface MTU

  • Max packet size supported by interface
  • Generally defaults to largest size
    • Serial Interface size differs – Cannot be smaller than 64 bytes

CCIE RS – Written – Network Principles – Identify Cisco Express Forwarding Concepts

Identify Cisco Express Forwarding Concepts

CEF – Cisco Express Forwarding

Overcomes disadvantages of fast switching.

CEF builds its own structure that mirrors the entire routing and MAC table – CEF Table and Adjacency Table

  • Optimized L3 forwarding path through a router or multilayer switch
  • Optimizes routing table lookup
    • Creates special easily searched structure based on IP routing table
  • CEF is only used for unlabeled packets

CEF Table

  • Stripped down version of the routing table
  • show ip cef summary
    • 2 Components build the CEF table
      • FIB and Adjacency table

Distributed CEF (dCEF)

  • FIB that runs on each line card

Order of Operations

  1. Compression and Decompression
  2. Encyption
  3. Inbound Access List 
  4. Unicast Reverse Path Checking
  5. Input rate limiting
  6. Physical broadcast handling (ip helper address)
  7. Decrement TTL
  8. Inspection subsystem (firewall, ZBFW)
  9. Outside to Inside NAT
  10. Handle router alert flags in IP header
  11. Search outbound interface in routing table
  12. Policy routing
  13. WCCP
  14. Inside to Outside NAT
  15. Encryption
  16. Output Access List
  17. Inspection (ZBFW)
  18. TCP Intercept processing

Process Switching

First switching method implemented in IOS. Contains the least amount of performance optimization and consumes large amounts of CPU. 

Platform independant, universal across all Cisco IOS based products. Provides some load sharing capabilities – per packet. Packets are automatically distributed across multiple paths based on the routing metrics. The routing metrics determine which path to take

Disadvantages – Lack of speed. Requires a routing table lookup for every packet. As the routing table grows the time to perform a lookup increases. Longer lookup times increases CPU utilization

Fast Switching

Store frequently used subset of the larger data and set it in local storage for faster access. Fast switching uses caching. IOS uses concept of fast cache – copy of Reachability/Interface/MAC-header learned from process switching packets

Does not support per-packet load sharing. This is due to separation of routing and forwarding due to fast switching (lack of deterministic load-balancing). CEF addresses this issue


  • Lack of overlapping cache entries
  • Any routing or ARP change invalidated large sections of the route cache
  • First packet must be process switched to a destination to build route cache
  • Inefficient load balancing 

RIB – Routing Information Base

Each routing protocol creates its own routing database that get tied together to build the routing table

Routing Table

  • Routing table is built in the Control Plane (along with ARP table)
  • Routing table is passed down to the Data Plane to build the FIB (along with Adjacency Table)
  • FIB + Adj Table = CEF

FIB – Forwarding Information Base

  • Similar to routing table
  • Generated by multiple routing protocols, maintaining only the next hop for a particular route

LFIB – Label Forwarding Information Base

  • Used for labeled packets (MPLS)
  • Prefix to label mapping maintained in the LIB
    • LIB – Label Information Base: A database used by an LSR to store labels learned from other LSRs, as well as labels assigned by the local LSR.
  • LIB + Routing Table build the LFIB

Adjacency Table

  • Contains MAC header information needed to switch the packet
  • Packet header for directly connected next hops
  • Populated with data from the ARP table

Adjacency Entire Types

  • Cache adjacency: This type of entry contains the correct outbound interface and the correct MAC address for its FIB entry. The MAC address is the IP address’s MAC address if the destination’s subnet is directly connected to the router, or is the MAC address of the router that the packet needs to be sent to if the destination’s subnet is not directly connected to the router currently processing the packet.
  • Receive adjacency: This type of entry handles packets whose final destinations include the router itself. This includes packets whose IP addresses are assigned to the router itself, broadcast packets, and multicasts that have set up the router itself as one of the destinations.
  • Null adjacency: Handles packets destined to a NULL interface. Packets with FIB entries pointing to NULL adjacencies will normally be dropped.
  • Punt adjacency: Deals with packets that require special handling or can not be switched by CEF. Such packets are forwarded to the next switching layer (generally fast switching) where they can be forwarded correctly.
  • Glean adjacency: This adjacency is created when the router knows that either the destination IP’s subnet is directly connected to the router itself and it does not know that destination device’s MAC address, or the router knows the IP address of the router to forward a packet to for a destination, but it does not know that router’s MAC address. Packets that trigger this entry will generate an ARP request.
  • Discard adjacency: FIB entries pointing to this type of adjacency will be discarded.
  • Drop adjacency: Packets pointing to this entry are dropped, but the prefix will be checked.

Load Balancing Hash

  • Distribute packets across multiple links based on L3 routing information
  • CEF can use multiple parallel links without additional hardware multiplexers
  • Number of paths is limited by number of entries in routing protocols into the routing table
    • Default is 4 entries on IOS
  • BGP is exception with 1
    • Max of 6 paths
  • 2 Modes of Load Balancing
    • Per-Destination (Default)
  • All packets to a given destination are forwarded on same path
  • Route-cache built for every destination address
  • Software driven table
  • Memory and processor intensive
  • Does not guarantee even load balancing if a bulk of the traffic is destined to a single destination
    • Per-Packet
  • Guarantees full load balancing
  • Packets may arrive out of order
    • Problem for voice/video traffic
    • Causes delay
  • Process determines outgoing interface for each packet using the routing table and least used interface
    • Processor intensive
    • Not suited for higher speed links


Ip route-cache
Explicitly enable CEF on an interface
Ip load-sharing [per-packet | per-destination]
Change load balancing type
Show ip cef

Load Balancing Algorithm

  • Original algorithm -The original Cisco Express Forwarding load-balancing algorithm produces distortions in load sharing across multiple routers because the same algorithm was used on every router. Depending on your network environment, you should select either the universal algorithm (default) or the tunnel algorithm instead.
  • Universal algorithm -The universal load-balancing algorithm allows each router on the network to make a different load sharing decision for each source-destination address pair, which resolves load-sharing imbalances. The router is set to perform universal load sharing by default.
  • Tunnel algorithm -The tunnel algorithm is designed to balance the per-packet load when only a few source and destination pairs are involved.
  • Include-ports algorithm -The include-ports algorithm allows you to use the Layer 4 source and destination ports as part of the load-balancing decision. This method benefits traffic streams running over equal cost paths that are not load shared because the majority of the traffic is between peer addresses that use different port numbers, such as Real-Time Protocol (RTP) streams. The include-ports algorithm is available in Cisco IOS Release 12.4(11)T and later releases.

ECMP – Equal Cost Multi-path

Polarization concept and avoidance


  • An effect when the hash algorithm chooses a particular path and the redundant path remains unused
  • CEF performs load balancing once the RIB is calculated


  • Alternate between default (SIP and DIP) and full (SIP+DIP+L4 Ports) hashing inputs configurations at each layer of the network
    • Hashing Algorithms
      • Default – Use the source and destination IP address, with unequal weights given to each link in order to prevent polarization.
      • Simple – Use the source and destination IP address, with equal weight given to each link.
      • Full – Use the source and destination IP address and Layer 4 port number, with unequal weights.
      • Full Simple – Use the source and destination IP address and Layer 4 port number, with equal weights given to each link.
  • Alternate between even and odd number of ECMP links at each layer of the network
    • The hash algorithm load-balances this way by default:
      • 1: 1
      • 2: 7-8
      • 3: 1-1-1
      • 4: 1-1-1-2
      • 5: 1-1-1-1-1
      • 6: 1-2-2-2-2-2
      • 7: 1-1-1-1-1-1-1
      • 8: 1-1-1-2-2-2-2-2
    • Disable CEF polarization
      • Anti-polarization weight
      • Mls ip cef load-sharing full simple
  • Concept of unique-ID/ universal-ID to avoid cef polarization
    • Default: universal
      • Adds 32 bit router specific value to the hash function
      • Randomly generated by the router during boot process
    • Unique-ID does not work on even number of equal cost paths due to hardware limitations
      • IOS adds one link into hardware adjacency table when even number exisits to make system think there is an odd number
    • Ip cef load-sharing algorithm universal [id]


Describe basic software architecture differences between IOS and IOS-XE


  • Monolithic kernal
  • System processes and core functionality are tightly integration
  • Uses priority “run to completion” scheduler
    • Each process is a single thread
  • All memory is mapped into a single flat address space
  • IOS does not implement memory protection between processes or memory pools
    • Advantage: improves system performance and minimizes OS overhead
    • Disadvantage: Complex system, one process can cause software to crash
  • IOS images are unique for each platform
  • Feature sets determine what CLI and features are available

IOS allows for configuration archiving with the archive command. Configs can be stored on local or remote server (FTP, TFTP, etc.). The max number of copies to save (default 10). Set the time period (minutes) for saving the config.

Archive configuration commands:
default Set a command to its defaults
exitExit from archive configuration mode
log Logging commands
maximum maximum number of backup copies
noNegate a command or set its defaults
pathpath for backups
rollbackRollback parameters
time-period Period of time in minutes to automatically archive the running-config
write-memoryEnable automatic backup generation during write memory

IOS-XE – Based on a modular architecture

  • Linux based OS that employs a single daemon
    • Allows for multiple layers of abstraction
    • Individual functions have been isolated from the primary operations kernel into separate processes
    • Linux kernel and drivers are only component of IOS-XE that can access hardware directly
  • Has all IOS capabilities with enhanced operations and functionality
    • Similar CLI to IOS
  • Leverages symmetrical multiprocessing
    • Allows processes to execute over multiple CPU’s
      • Benefit of load balancing across multiple core CPUs
      • Binds process to different cores
  • Individual threads for each underlying process
  • Separates control plane from forwarding plane
  • APIs allow for development of drivers for the new data plane ASICs
    • Creates control plane and data plane separation
  • Logical and physical separation of control plane and data plane
    • Dedicated hardware resources
  • Separation archived through:
    • FFM – Forwarding and Feature Manager
  • Provides APIs to manage the control plane process
  • FFM programs the data plane through the FED and maintains all forwarding states for the system
    • FED – Forwarding Engine Driver
  • Allows the drivers to affect the data plane
  • Routing protocols run in the IOSd process


IOS-XE releases using consolidated packages and optional subpackages. Each consolidated package contains a collection of subpackages. 

Subpackage is an individual software file that controls different elements of the device. These can be upgraded individually

  • RPBase – Provides the operating system software for the Route Processor.
  • RPControl – Controls the control plane processes that interface between the IOS process and the rest of the platform.
  • RPAccess – Exports processing of restricted components, such as Secure Socket Layer (SSL), Secure Shell (SSH), and other security features.
  • RPIOS – Provides the Cisco IOS kernel, which is where IOS features are stored and run. Each consolidated package has a different RPIOS.
  • ESPBase – Provides the ESP operating system and control processes, and the ESP software.
  • SIPBase – Controls the SIP operating system and control processes.
  • SIPSPA – Provides the SPA driver and Field Programmable Device (FPD) images.

Upgrade Procedure

Best practice – Backup configuration – copy run flash: or to FTP server, or old school, copy and paste to text document

Upload new code to device – TFTP or RCP

  • copy tftp: flash:


Install subpackage

request platform software package expand file URL-to-consolidated-package to URL-to-directory-name 

Individual Processes

  • Chassis Manager – Responsible for all chassis management functions, including management of the HA state, environmental monitoring, and FRU state control.
    • RPControl,SIPBase,ESPBase
  • Host Manager – Provides an interface between the IOS process and many of the information-gathering functions of the underlying platform kernel and operating system.
    • RPControl,SIPBase,ESPBase
  • Logger – Provides IOS facing logging services to processes running on each FRU.
    • RPControl,SIPBase,ESPBase
  • Interface Manager – Provides an interface between the IOS process and the per-SPA interface processes on the SIP.
    • RPControl,SIPBase
  • IOS – The IOS process implements all forwarding and routing features for the router.
    • RPIOS
  • Forwarding Manager – Manages the downloading of configuration to each of the ESPs and the communication of forwarding plane information, such as statistics, to the IOS process.
    • RPControl
    • ESPBase
  • Pluggable Services – The integration point between platform policy application, such as authentication and the IOS process.
    • RPControl
  • Shell Manager – Provides all user interface features and handling related to features in the nonIOS image of the consolidated package, which are also the features available in diagnostic mode when the IOS process fails.
    • RPControl
  • SPA driver process – Provides an isolated process driver for a specific SPA.
    • SIPSPA
  • CPP driver process – Manages the CPP hardware forwarding engine on the ESP.
    • ESPBase
  • CPP HA process – Manages HA state for the CPP hardware forwarding engine.
    • ESPBase
  • CPP SP process – Performs high-latency tasks for the CPP-facing functionality in the ESP instance of the Forwarding Manager process.
    • ESPBase

Control Plane

The control plane of a router learns what the router will do with a packet. Its primary goal is learning about routes, static or dynamic. The routing table is contains a list of destination networks and outgoing interfaces. The control plane can define if a packet is discarded or given preferential treatment (QOS). There is a separate table, Forwarding Information Base, that is built by the control plane, but utilized by the Forwarding/Data Plane.

Forwarding Plane

The forwarding plane (aka: Data Plane) is responsible for moving packets based on what was learned in the control plane.

Impact to Troubleshooting and Performance

  • IOS XE (IOS 15.0) runs as a single daemon within a Linux operating system 
  • Additional system functions now run as additional, separate processes in the host OS environment
  • IOSd within the IOS XE environment supports multiple threads and multi-core CPUs
  • Wireshark and Mediatrace included, runs separately from IOS

Exclude Specific Platform Architecture

  • Non-IOS applications can either be tightly integrated with IOS or they could run side-by-side with IOS with very little or no interactions
  • If an application does require services from IOS, it integrates with IOS through a set of client libraries called “service points”