I have a lot to learn from talented people like Kevin.

Many people may be confused as to why I would dedicate an entire post to network monitoring tools and their configuration.  The reason is because these topics are tested relatively heavily on the actual CCNP SWITCH Exam.  Whether you agree or disagree about the weight given to these topics given the number of others is irrelevant.  It’s covered on the exam – so take the time to understand the topics (especially IP SLA).

Here we go…

Syslog

Syslog is a network management protocol that is not unique to Cisco devices, but integrates well within IOS.  Syslog allows a network-attached device to report and log error and notification messages either locally or to a remote Syslog server.

Syslog messages are plain text sent using UDP port 514.

Every syslog message contains two parts, a severity level and a facility.  The severity level goes from 0 to 7 with 0 being the most severe to 7 being simply informational. Facilities are service identifiers that categorize events and messages for easier reporting.

The most common facilities on IOS devices include:

• IP
• OSPF
• SYS (operating system)
• IP Security (IP Sec)
• Route Switch Processor (RSP)
• Interface (IF)

Messages are presented in the following format:
%FACILITY-SUBFACILITY-SEVERITY-MNEMONIC:Message-text

Configuring Syslog

To configure Syslog to export events to an external Syslog server, use the following commands:

Switch(config)# logging <ip address of server>
Switch(config)# logging trap <severity level>

To configure the local switch to store syslog messages, use the logging buffered command.

Switch(config)# loggingbuffered ?
<0-7>             Logging severity level

Use the show logging command to show the contents of the local log files.

SNMP network management application periodically uses UDP to poll the agent residing on a managed device for useful, predetermined information.  The problem is it polls the device on a set schedule, so there will be a lag between when an event occurs and when the application learns of it.

SNMP traps, are not so passive. When certain criteria are met, the agent sends the application a notification instantly, so it no longer has to wait around to find out.  This can introduce bandwidth savings.  Think of it like push notification in the cellular world.

The data that the agent collects is stored in its MIB.  Community strings are used to provide a level of authorization for the MIB contents (read or write) - kind of like a weak SNMP passwords.  They are transmitted in clear text across the network, so be careful.

SNMP Versions

• SNMP v1 (insecure)
• SNMP v2 – introduced the read/write community strings (insecure)
• SNMP v3 – provides encryption and authentication (recommended whenever possible)

SNMP Configuration

1. Configure SNMP access lists (optional, but recommended)
2. Configure community strings
3. Configure SNMP trap destination
4. Configure SNMP v3 user (optional, but recommended)

Example Configuration
Switch(config)# access-list 100 permit ip 10.1.1.0 0.0.0.255 any
Switch(config)# snmp-server community aaron RO 100
Switch(config)# snmp-server community thisismoresecure RW 100
Switch(config)# snmp-server trap 192.168.1.52

IP Service Level Agreement

Service level agreements or SLAs are contractual agreements usually between a customer and service provider that spell out the minimum acceptable levels of service.  SLAs are often attached to WAN and MPLS links because any downtime can significantly affect business performance/profits.

In terms of the exam, Cisco’s SLA attempts to measure latency, jitter, and packet loss for a given link.  Cisco does this by enabling IOS to send synthetic traffic  to a specific host computer or router that is configured to respond.  The router can then use determine one way jitter, delay, an packet loss.

Router <——–> Router
OR
Router <———> PC

Common IP SLA Functions

• Edge-to-edge network availability monitoring
• Network performance monitoring
• VoIP, video, and VPN monitoring
• IP heath assessment
• MPLS monitoring
• Troubleshooting

IP SLA can measure the following statistics:

• Network latency and response time
• Packet loss
• Jitter and voice quality scoring
• end-to-end network connectivity

IP SLA Operations

Multiple IP SLA operations (measurements) can run in a network at the same time.  The reporting tools use SNMP to fetch the data so they can report on it.

The source router needs to be configured with a target device, protocol, and UDP/TCP port number for each IP SLA operation.  The source router uses the IP SLA control protocol to confirm communication with the responding host before the source sends the test messages.

To increase security, the responder can use an MDF hash to authenticate the message from the source, securing the exchange.

When the operation is complete, the results are stored in the IP SLA MIB on the source and can be retrieved via SNMP (or by traps which can be conditionally set to send alerts if thresholds are exceeded).

Almost all of the configuration occurs on the source router.  The source sends the probe packets that test whatever protocols the administrator chooses.  Although any IP device can be a respoder, another IP SLA router is prefered because the measurement accuracy will be improved.

IP SLA Operation Breakdown

1. Source sends a IP SLA control message with the configured operation to the responder using UDP port 1967.  The control message carries the protocol, port, and duration defined when the operation was configured on the source router.
   *  If MD5 is enabled, the checksum is sent with the control message.
   *  I authentication is enabled, the responder verifies it.  If authentication fails, the responder returns an authentication failure message.
   * If a response is not received from the responder, it will attempt to retransmit until it eventually times out.
2. The responder sends a confirmation message back to the source router and listens on the specified port.
3. If the response from the control message is OK, it begins sending probe packets.
4. The responder responds to the incoming probe packets for the predetermined time.

 The diagram above outlines the timestamp process IP SLA uses to calculate round trip time (RTT) accurately.

1. The source sends a packet at time T1
2. The responder record both the receipt time (T2) and the transmitted time (T3).  Because there can delay between when the router receives t packet and when a confirmation is sent back out the interface, it tracks the difference in time(in sub-milliseconds).  The source later subtracts this difference from the total RTT because it was not time in transit, but rather router software processing time.

An additional benefit of so many timestamps is the ability to track one-way delay, jitter, and packet loss.  Remember that traffic behavior can be asynchronous.  Also, make sure that both devices are using the same source for clock information.  The same NTP server is a requirement for many of these functions.

Configuring IP SLA

1. Configure the source router
2. Activate the IP SLA on the source
3. Configure that tracking object on the source
4. Configure responder

Example Source Configuration
Switch(config)# ip sla 10 (number indicates the IP SLA test identifier)
Switch(config-sla)# type echo prot ipIcmpEcho 192.168.1.10 source-int fa0/1
Switch(config-sla)# frequency 20 (number of times the operation repeats)
Switch(config)# exit
Switch(config)# ip sla schedule 10 life forever start-time now
Switch(config)# track 1 ip sla 10 reachability

Responder Configuration:
Switch2(config)# ip sla monitor responder

Verifying IP SLA

Switch# show ip sla statistics
Switch# show ip sla configuration {operationID}
 

Some technology overviews to look forward to:

•  Network Monitoring
  • Syslog
  • SNMP
  • IP SLA
• Redundant Supervisor Cards
  • Route processor redundancy
  • SSO
  • NSF
• First Hop Redundancy Protocols
  • HSRP
  • VRRP
  • GLBP

Anything I’m missing for high-availability?  Have a great week!

Aaron

There are 3 inter-VLAN routing device options:

• layer 3 multilayer Catalyst switch
• external router that allows trunking (router-on-a-stick)
• external router with enough interfaces for every VLAN (this doesn’t scale and is very expensive)

All Catalyst multilayer switches support the following types of layer 3 interfaces:

• Routed port – a pure layer 3 port similar to that on a router
• Switch virtual interface (SVI) – virtual routed VLAN interface for inter-VLAN routing
• Bridge virtual interface (BVI) – a layer 3 bridging interface

Inter-VLAN Routing Types // Advantages and Disadvantages

External Router (router -on-a-stick)

Advantages

• Works with almost all switches because the switches do not have to support layer 3, just VLANs and trunking
• Simple configuration (one switch port, one router interface)

Disadvantages

• Router is a single point of failure
• If the trunk becomes congested, it can affect every VLAN
• Slightly higher latency because traffic must leave and renter the switch and the router makes the traffic decisions in software (which is slower than hardware)

Switch Virtual Interfaces

Remember that Cisco recommends using layer 2 between access and distribution layers and layer 3 routing between distribution and core layers.

SVIs are virtual VLAN interfaces on multilayer switches that are configured in the same way a trunk interface on a router would.  One SVI is created for each VLAN to be routed and it performs the process for all the packets associated with that VLAN.

The only SVI created by default is the SVI for VLAN 1.  The rest must be created manually, although the switch will created an SVI the first time a VLAN interface mode is entered for a particular VLAN.
For example, when thew command # interface vlan 10 is entered SVI 10 is automatically created on the switch.

Whenever an SVI is created, make sure the VLAN is present in the VLAN table, otherwise the interface will remain down.

Advantages

• Fast because all performed in hardware
• No need for external links for routing
• Low latency (doesn’t need to leave the switch)

Disadvantages

• May require a more expensive switch

Routed Ports

Routed ports are physical ports on the switch that act much like a router interface would with an IP address configured.   Routed ports are not associated with an particular VLAN and do not run layer 2 protocols like  STP and VTP.

Note:  Routed interfaces do not support subinterfaces.

Routed ports are point-to-point links that usually connect core switches to other core switches or distribution layer switches (if the distribution layer is running layer 3).

Make sure when configuring a routed port that you use the no switchport command to make sure the interface is configured to operate at layer 3.  Also make sure to assign IP addresses and any other layer 3 information required.  Check the routing protocols are configured.

Advantages

• A multilayer switch can have both SVIs and routed ports configured
• Multilayer switches forward all layer 2 and 3 traffic in hardware, so it is very fast

Configuring Inter-VLAN Routing with an External Router

Implimentation Planning

• Need to know how many VLANS require routing, the VLAN IDs, and what ports connect to the router
• Every router subinterface must be configured with the same type of frame encapsulation (usually 802.1q) as well as the switch side of the link
• Make sure the native VLAN is the same on both ends.  Now a subinterface on the router can be created for the native VLAN, also if it is a subinterface – make sure to define its encapsulation type with the encapsulation dot1q ID vlan command.
• It is best practice to match the subinterface ID to the VLAN ID

Configuring Router-on-a-stick

1. Enable trunking on the switch port
2. Enable the router interface with the no shut command
3. Crate the subinterfaces on the router for each VLAN
   
4. Configure IPs and encapsulation on each subinterface as they relate to their VLANs
   Switch (conf-subif)# encapsulation [dot1q | isl] vlan-id {native}
   Switch (conf-subif)# ip address x.x.x.x  x.x.x.x

Example router interface configuration
Router(config)# interface FastEthernet0/0
Router(config-if)#no shutdown
Router(config)# interface FastEthernet 0/0.1
Router(config-subif) description VLAN 1
Router(config-subif)# encapsulation dot1Q 1 native
Router(config-subif)# ip address 10.1.1.1 255.255.255.0
Router(config-subif)# exit
Router(config)# interface FastEthernet 0/0.2
Router(config-subif)# description VLAN 2
Router(config-subif)# encapsulation dot1Q 2
Router(config-subif)# ip address 10.2.2.1 255.255.255.0
Router(config-subif)# exit
Router(config)# end

Configuring Inter-VLAN Routing with SVIs

Implementation Planning

• Identify which VLANs require layer 3 gateways as you may not want all VLANs to be routable within the organization
• Make sure VLANs are create don the switch, then make SVIs
• Find out what IPs need to be configured on each SVI interface, then use the no shutdown command to enable them
• Configure any routing protocols that are required
• Determine if any switchports should be excluded from contributing to the SVI line-state up-and-down calculation

Configuring SVIs

1. Select an SVI
2. Assign an IP address to the interface
3. Enable the interface
4. Optional – Enable IP routing on the router
5. Optional – Enable an IP routing protocol

Note: Routing protocols are only required to allow different devices to communicate across different VLANs or networks.  They are not required to route between SVIs on the same switch because the switch sees the SVIs as connected interfaces.

Example Configuration

Switch# configure terminal
Enter configuration commands, one per line.  End with CNTL/Z.
Switch(config)# ip routing
Switch(config)# router rip
Switch(config-router)# network 10.0.0.0
Switch(config)# interface vlan 10
Switch(config-if)# ip address 10.10.1.1 255.0.0.0
Switch(config-if)# no shutdown
Switch(config-if)# interface vlan 20
Switch(config-if)# ip address 10.20.1.1 255.255.255.0
Switch(config-if)# no shutdown

SVI Autostate

An SVI is auto automatically created when the following conditions are met:

• The VLAN is active and exists in the VLN database
• The VLAN interface exists and is not administratively shut down
• At least a single port on the switch has a port in the VLAN, is in the up state, and is in the spanning-tree forwarding state.

This automatic SVI creation is called SVI Autostate.  If there are multiple ports on the switch in the same VLAN, the default action is to take down the SVI interface if all of the ports in that VLAN are shut down.

The command switchport autostate exclude, when applied to port, will allow the VLAN to go down if all of the other ports in the VLAN go down except the one autostate exclude was applied to.  This is often desirable when traffic analyzers are attached to a host.  They will stay up, but are just passive monitors, so if all other devices in the VLAN go down – this port would prevent the VLAN from going down, so autostate exclude is applied to allow the VLAN to still go down.

Configuring Inter-VLAN Routing with Routed Ports

• show ip interface interface_type_port| svi_number
• show interface interface_type_port| svi_number
• show running interfacetype_port| svi_number
• ping
• show vlan
• show interface trunk

Troubleshooting Inter-VLAN Problems

Here is a list to run through when identifying an issue related to inter-VLAN routing:

• Correct VLANs on switches and trunks
• Correct routes
• Correct primary and secondary root bridges
• Correct IP addresses and masks

The table below outlines common issues that may come up and some potential causes.

Problem	Potential Cause
Missing VLAN	Make sure: VLAN defined across switches VLAN enabled on trunk ports Ports in correct VLANs
Layer 3 port misconfiguration	SVI wrong IP and/or mask SVI not up SVI number does not match VLAN number Routing is not enabled
Routing protocol misconfiguration	Make sure all necessary interfaces and networks are in routing protocol. Only required if need to connect to other routers.
Host Misconfiguration	Check Host IP and mask Check host's default gateway. It should be the SVI or layer 3 interface IP address.

How create Layer 3 EtherChannels

1. Create a virtual layer 2 interface
   Switch(config)# interface port-channel 1
2. Change the port to layer 3
   Switch(config-if)# no switchport
3. Assign an IP address
   Switch(config-if)# ip address ip_address subnet_mask
4. Select the interface
   Switch(config)# interface range interface_id portnumber_range
5. Set the interface to layer 3.  If all of the physical interfaces are not acting in the same layer, the EtherChannel will not form.
   Switch(config-if-range)# no switchport
6. Assign all physical interfaces to the EtherChannel group.
   Switch(config-if-range) # channel-group channel-group-number mode {auto [non-silent] | desirable [non-silent] | on} | {active | passive}

Implementing DHCP in a Multilayer Switch Environment

By default, Catalyst multilayer switches include DHCP relay agent software.

Distribution multilayer switches often act as layer 3 gateways for clients connecting to the access switches.  Because of this DHCP can be provided within the same switches to serve the hosts with IP addresses and other necessary network parameters.

The other option is to consolidate the DHCP services to one or more dedicated servers.  In that case, the distribution layer must redirect incoming client DHCP requests to the external DHCP server.

Configuring DHCP service on the multilayer switch

1. By default the switch assumes the whole network range for the DHCP scope.  To exclude certain addresses or ranges, in global config mode, use the ip dhcp excluded-address command.  Follow it with a range of addresses to exclude from your scope.  For discontinuous ranges, use more than one ip dhcp excluded-address commands.
2. Configure the network value, which indicates the subnet to offer addresses from.
3. Configure any other network parameters you would like the switch to server in its DHCP offers (ex. default-gateway, lease duration, subnetmask, DNS server address).

Note:  Remember that a switch cannot offer DHCP addresses for a subnet it is not a member of.

Configuration Example
Switch(config)# ip dhcp excluded-address 10.1.10.1 10.1.10.20 (range beginning to end)
Switch(config)# ip dhcp pool example10
Switch(config-dhcp)# network 10.1.10.0 255.255.255.0
Switch(config-dhcp)# default-router 10.1.10.1
Switch(config-dhcp)# option 150 10.1.1.50 (Option 15- specifies a TFTP server IP – often for IP phones to reach Call Managers)
Switch(config-dhcp)# lease 0 8 0 (0 days 8 hours 0 minutes)

Switch(config)# interface vlan10
Switch(config-if)# ip address 10.1.10.1 255.255.255.0

Configuring DHCP Relay

If an enterprise is using external DHCP servers, then the ip helper-address command must be entered on the layer 3 interface.  Because hosts use broadcast messages to try to find the DHCP server, if it is in a different subnet, it will be dropped at the default gateway because broadcasts are not forwarded across VLAN boundaries.

The DHCP relay agent allows the DHCP request to be forwarded on as a unicast message to a single IP address.  It not only forwards DHCP services, but also TFTP, DNS, Time, NetBIOS, names server, and BOOTP packets by default.

The ip helper-address command must be applied to the layer 3 interface itself.

Configuration Example
switch(config)# interface vlan10
switch(config-if)# ip address 10.1.10.1 255.255.255.0
switch(config-if)# ip helper-address 10.1.100.1

Note:  You can apply to to an SVI or a routed interface.

Verifying DHCP Settings

Use these two commands to check its operation:

• show ip dhcp binding – displays client DHCP bindings including IP address and MAC
• debug ip dhcp server packet – shows in real-time the DHCP discover, offer, reply, and ack packets

BPDU Guard

Prevents problems related to switches accidentally being connected to PortFast-enabled ports.  Bridging loops would normally instantly occur.

It places the port in err-disable state if it receives a BPDU - disabling the interface.

To enable BPDU Guard globally on the switch:
Switch(config)# spanning-tree portfast edge bpduguard default

To enable BPDU Guard at the interface level:
Switch(config)# spanning-tree bpduguard enable

BPDU Filtering

Prevents BPDUs from being transmitted from PortFast-enabled interfaces.

When enabled globally on the switch:

• Configures all PortFast ports for BPDU filtering
• If BPDUs are seen, the port looses its PortFast status, BPDU filtering is disabled, and STP resumes default operation on the port
• When the port comes up, it sends 10 BPDUs, if it hears any BPDUs during that time PortFast and BPDU filtering are disabled

When applied to an individual port:

• It ignores all BPDUs it receives
• It does not transmit BPDUs

Note:  If you enable BPDU Guard and BPDU filtering on the same interface, BPDU Guard has no effect because BPDU filtering has precedence over BPDU Guard.

To enable BPDU filtering globally on the switch:
Switch(config)# spanning-tree portfast bpdufilter default

To enable BPDU filtering at the interface level:
Switch(config)# spanning-tree bpdufilter enable

To verify:
Switch# show spanning-tree summary
    OR
Switch# show spanning-tree interface fa 0/3 detail

Root Guard

Root guard was developed to control where root bridges can be located within the network.  Switches learn about and elect root bridges based on BPDUs they receive, so if a new switch is added to the environment with a lower bridge priority than the current root bridge, the new switch will become root – and in turn disrupting your carefully planned traffic patterns.  To prevent this from occurring, root guard can be applied to interface where a root bridge should never been seen.

When root guard is applied to an interface, it forces the port to essentially always remain a designated interface, never allowing it to transition to a root port.  If a root guard-enabled  port received a superior BPDU, it immediately moves the port to a root-inconsistent STP state (essentially the same as the listening state) and does not forward any traffic out that port.

When the root guard protected port stops receiving superior BPDUs, it automatically unblocks the port and proceeds through its normal listening, learning, and eventually forwarding states.  No intervention is required.

 To enable root guard on an interface:
Switch(config)# int fa 4/4
Switch(config-if)# spanning-tree guard root

Loop Guard

Most bridging loops that occur when STP is active happen when a port in blocking state stops receiving BPDUs on the interface and therefore transition the port to forwarding state – creating an all-ports-forwarding loop.  It blocks ports on a per-VLAN basis, so on trunks it will only block that VLAN – not the whole trunk.

Loop guard should be applied to all non-desgnated ports (ex. root, alternate). 

To enable loop guard on an interface:
Switch(config)# int fa 4/4
Switch(config-if)# spanning-tree guard loop

To enable loop guard globally on the switch:
Switch(config)# spanning-tree loopguard default

To verify:
Switch# show spanning-tree interface fa 0/3 detail

UDLD

UDLD is another loop-prevention mechanism for STP.  It tries to discover unidirectional links before they grow into bridging loops.  This situation is much more common in fiber optic networks where there is a physical Rx/Tx pair and a situation can arrise where one is not functioning correctly.

STP relies on constant and consistant reception of BPDU messages.  If a switch stops receiving BPDUs on a desgnated (upstream) port, STP ages out the information for the port and transistiones it into forwarding state.  This will lead to a loop.

UDLD sends UDLD protocol packets to it neighbor switch – 15 seconds is the default.  The neighbor is then expected to echo packet the packets before a timer expires.  If the switch does not hear a reply it waits, before finally shutting down the port.

There are two UDLD modes:

• Normal- UDLD simply places the port into an undetermined state if it stops hearing responses from its directly-connected neighbor
• Aggressive(Preferred) – Tries to re-establish the connection up to 8 times, then puts the port in err-disable state (essentially shutting down the port)

Note:  UDLD is enabled by default on all Ethernet fiber-optic interfaces.

To enable UDLD on an interface:
Switch(config)# int fa 4/4
Switch(config-if)# udld port {aggressive}

To enable UDLD globally on all fiber ports:
Switch(config)# udld {enable | aggressive}

Note:  While both loop guard and aggressive UDLD have many overlapping functions, enabling both provides the best protection.

Flex Links

Flex links is a layer 2 solution that allows you to disable STP at the access layer, while maintaining link redundancy and loop avoidance.  Flex links allow a convergence time of less than 50 milliseconds (super fast). 

Flex links work by combining a pair of links on a common access switch.  One link is active, while the other acts as a backup if the other link fails for any reason.  The interfaces can be physical ports or EtherChannel bundles.

Some more details:

• One a single backup port is allowed per active port and they must both be on the same switch or stack.
• The  active and backup links can be dissimilar types and work just fine (ex. fasteth/gig, gig/etherchannel, etc..)
• STP is always disabled on Flex links

The configuration is done exclusively on the “active” interface with the following command:
Switch(config)# int fa 4/4
Switch(config-if)# switchport backup interface fa 5/17

To verify:
Switch# sh int switchport backup

Spanning Tree Best Practices

• Something to consider with spanning tree is the lack of multipathing options.  STP eliminates loops by creating a tree structure where a single link is created to each switch.  This means that even with all the redundant links you put in place, STP will always only allow one – reducing much of your available bandwidth.Because of this and other limitations, it is recommended to use layer 3 at both the distribution and core layers.  Using layer 3 between the distribution and core allows you to use multipathing (up to 16 paths) using Equal-Cost Multipathing (ECMP) with the dependency of STP.  Also, the new Nexus 7ks allow layer 2 multipathing with two links using virtual port channels.
• Because a 50 second network convergence delay is usually not acceptable in modern networks, RSTP is prefered. 
• STP should absolutely be used on the network edge to prevent user/wiring errors from propagating throughout the network
• A root bridge should be manually assigned in every STP topology
• If using PVST+ or RPVST+, assign a root bridge for each VL:AN using the command:
  #spanning-tree vlan ID root
• Use the STP Enhancements (sometimes referred to as the STP toolkit) to optimize the topology
• Loop guard – Implement on layer 2 uplink ports between access and distribution layer
• Root guard – Implement on distribution switch ports facing the access ports
• UplinkFast- Implement on uplink ports from access to distribution switches
• BPDU guard or root guard- Implement on access ports connected to end devices, as is PortFast
• UDLD -Sometimes implemented on fiber ports between switches

Troubleshooting Spanning Tree

 Duplex Mismatch

If one side of a link is set to half duplex and the other is set to full , then the potential exists that the full duplex side will begin sending lots of traffic to the half duplex interface.  If that happens, the half duplex interface will experience collisions when it attempts to transmit STP BPDUs.  The full duplex interface will therefore never receive them, and assume other interfaces on the switch in blocking state can transfer to a forwarding state – creating a loop.

Unidirectional link failure

This occurs when a hardware failure causes a normally two-way link to become a one-way link.  The potential loop problem is the same as with the duplex mismatch issue, with one side moving from blocking to forwarding because they stop receiving superior BPDUs on the interface.

Aggressive UDLD can prevent loops from forming when this occurs by putting the offending port into err-disable state.  Cisco recommends using agressive UDLD on all point-point links in a switched environment.

Frame Corruption

This is a very uncommon cause of STP loops, but it exists when errors on an interface do not allow BPDU frames from being received.  Again, a pot moves from blocking to forwarding because they stop receiving superior BPDUs on the interface. This could be caused by a duplex mismatch, bad cable, or incorrect cable length.

Resource Errors

If for any reason the CPU of a switch is over-utilized, there exists the possibility that it will be unable to send out BPDUs.  STP is generally not very resource intensive, but be careful when running PVST+.

PortFast-related Errors

PortFast interfaces move directly into forwarding state, so if a hub or switch gets connected to an edge port configured with PortFast, a loop will form.  BPDO Guard can prevent this condition.
General STP Troubleshooting Methodology

1. Develop a plan.
2. Isolate the cause and correct an STP problem.
3. Document findings.

Develop a plan
In order to make a plan, you must know the following parts of the network:
* The switched topology
* The location of the root bridge
* The location of blocking ports

Correct the problem
1. Identify a bridging loop
The best way to determine a loop is to capture packets on a saturated link and look for duplicate packets.  Another option is to look for abnormally high interface utilization values.

Some common symptoms include HSRP may complain of duplicate IP addresses, consistent flapping of MAC values because MAC addresses do not flap.

Restore connectivity
Most of the time administrators do not have he luxury of time to identify the root cause of a loop, instead they must stop it as quickly as possible.  Here are some options:

• Disable every port that is providing redundancy, starting with areas of the network more affected.  Try to disable ports you know should be in blocking state if possible.
• If it is difficult to pin down, increase the level of STP logging on the switches.  The loops form when a port moes into forwarding state, so it can latter be identified.

Try this:
Switch# debug spanning-tree events

To log the events:
Switch(config)# logging buffered

Check Port Statuses
Start with blocking ports first – here are some more guidelines:

• Make sure switch both root and blocking ports are receiving BPDUs
• Switch# sh spanning-tree vlan ID detail   (enter multiple times to see if the number is increasing)
• Look for duplex mismatch errors using the show interface command
• Check port utilization with the show interface command.  Look at the load, input/output values for abnormally high rates.
• Look for an increase of input error fields using the show interface command.
• Check for resource errors

Resource Errors
Use the show process cpu command to check whether the CPU utilization is nearing 100%.

Disable Unnecessary Features
Sometimes it becomes easier to identify a solution when the network is simplified.  Try disabling unnecessary features to reduce complexity.  Save the configuration before making the changes so it can be restored after the issue is resolved.

Document Findings
It is important to document both your findings and any changes to the network after the dust clears.  Current and detailed documentation also reduces troubleshooting time in the future.

MST expands upon the IEEE 802.1w RST algorithm in an attempt to reduce the number of STP instances, thus reducing the required CPU cycles on a switch.  MST enables you to group VLANs and associate them with spanning tree instances.  Each instance’s topology can be independent of the rest, optimizing load balancing and fault tolerance measures.  MST is also backwards compatible with all older STP variations.

Switches participating in MST that have the same MST configuration information are referred to as a region.  Switches with different MST configurations or that are running legacy 802.1D are considered separate MST regions.

MST is usually not implemented in campus environments because if you follow the local VLAN model (recommended by Cisco), there should not be that many VLANs on any given switch because they should only extend to the switch block boundary.  That makes RPVST+ a better choice because of it’s simpler configuration.

Multiple Spanning Tree Regions

Each switch that runs MST in the network has a single MST configuration consisting of the following 3 items:

• Configuration name (alphanumeric)
• Configuration revision number
• A 4096-element table that associates each VLAN to a given instance

MST Configuration

Each switch that runs MST:

Enable MST globally:
Switch(config)# spanning-tree mode mst

Enter MST Submode:
Switch(config)# spanning-tree mst configuration
Switch(config-mst)# sh current

Define a configuration name:
Switch(config-mst)# name XYZ

Set the MST revision number:
Switch(config-mst)# revision 1

Map the VLANs to an MST instance:
Switch(config-mst)# instance 1 vlan 3, 5, 7
Switch(config-mst)# instance 2 vlan 2, 4, 6

Display configuration to be applied:
Switch(config-mst)# show pending

Display current running MST configuration:
Switch(config-mst)# show current

Apply the configuration:
Switch(config-mst)# end

Cancel the configuration:
Switch(config-mst)# abort

Assign an MST root bridge
Switch(config)# spanning-tree mst 2 root primary

Verification Commands

• Switch# show spanning-tree mst
• Switch# show spanning-tree mst 1  (to view MST info for a single instance)
• Switch# show spanning-tree mst 1 detail
• Switch# show spanning-tree mst interface fa 0/3

Our family was out of town for the weekend visiting family, so needless to say, my studying momentum has slowed.  I’m going to try and wrap up all the spanning tree related notes this week so I can move onto more interesting topics (like high-availability).  Hang with me while we push through spanning tree – it’s a big part of layer 2 in most enterprise deployments and is really important to understand more than just the fundamentals.

As always, if you have comments or questions, leave a comment.  Those command-line tips are always appreciated.

RSTP can revert to 802.1D (common spanning-tree) to inter-operate with legacy bridges on a per-port basis.

A rapid version of PVST+, RPVST+ is a per-VLAN implementation of rapid spanning-tree.

RSTP Port States

• Discarding
  • Merges the former disabled, blocking, and listening states
  • Prevents the forwarding of frames
  • Seen in both stable/active and synchronization/changes
• Learning
  • Receives frame to populate the MAC table
  • Seen in both stable/active and synchronization/changes
• Forwarding
  • Forwarding ports determine the active topology
  • An agreement process between switches occurs before frames can be forwarded
  • Only seen in stable/active topologies

Note:  In every RSTP port state, BPDU frames are accepted and processed.

Operational Status	STP Port State	RSTP Port State	Port Included in Active Topology
Enabled	Blocking	Discarding	No
Enabled	Listening	Discarding	No
Enabled	Learning	Learning	Yes
Enabled	Forwarding	Forwarding	Yes
Disabled	Discarding	Discarding	No

RSTP Port Roles

• Root port (active)
  • On non-root bridges only
  • Best port towards the root bridge
  • Only one per switch
  • Is always in forwarding state in an active/stable topology
• Designated port (active)
  • On root and non-root bridges
  • All ports on root bridge are designated ports
  • Receives and forwards frames towards the root bridge as needed
  • Only one per segment
• Alternate (inactive)
  • Offers an alternate path towards the root bridge, but is in discarding state in an active topology
  • Present on nondesignated switches and becomes designated if path fails
• Backup (inactive)
  • An additional switch port on a redundant (and designated) link
  • It has a higher port ID than it’s redundant peer port, so it assumes the discarding state
• Disabled port
  • No role in spanning tree

RSTP Link Type

In common spanning tree, it took 50 seconds before port could be placed in forwarding state after a network change.  RSTP’s biggest advantage is its ability to rapidly transition alternate ports to a forwarding state.  To do this, the protocol relies on two variables, link type and edge port.

• Link type
  • Point-to-point or shared
  • Determined by duplex mode of port
    • Full Duplex – assumed to be point-to-point
    • Half Duplex – assumed to be shared
  • Point-to-point links are considered candidates for rapid transition to forwarding state
  • Link type can be manually configured if desired

The link types cannot be determined until the port role is first established.

• Roots ports
  • Don’t use the link type parameter
  • Make rapid transition to forwarding state as soon as it recieves a BPDU from the root bridge and puts nondesignated ports in blocking state (called sync)
• Alternative and backup ports
  • Do not use link type in most cases
  • Simply go through RSTP operation process
• Designated ports
  • Most common use of link type parameter
  • Only allows rapid transition to forwarding if point-to-point

RSTP Edge Ports

Edge ports are assumed to connected to an and host and bever another switch.  Edge ports immediately transition to rapid forwarding sate when enabled.

Edge port

• the RSTP equivalent of PortFast
• allowed to transition directly into forwarding state
• designated through manual configuration
• Does not generate a topology when link transitions to enabled or disabled status
• If edge port receives a BPDU, looses edge port status and become a normal STP port and generates a topology change notification (TCN)

RSTP Topology Changes

In 802.1D spanning tree, when a switch detects a topology change, it first notifies the root bridge.  The root bridge then sets the TC (topology change) flag on the BPDUs it sends out, which gets relayed throughout the switched network.  When a switch receives the notification, it reduces its bridging-table aging time equal to the forward delay.  That allows the outdated topology information to be flushed from the switches.

This modle works well, but the problem is that it takes a minimum of twice the forwarding delay for bridges to transition back to forwarding state.  RSTP solves this.

In RSTP, only non-edge ports that are transitioning to forwarding state cause a topology change notification to be sent out.  Unlike with 802.1D, ports moving to blocking state do not cause a TC BPDU to be sent.

Synchronization

Synchronization is term used to describe the RSTP network convergence process.

Nonedge ports begin in the discarding state.  It then performs a handshake to determine the state of each end of the link.  Each switch assumes that its port should become the designated port for the link, and so it sends a proposal message (a configuration BPDU) to its neighbor switch.  When a switch recieves a proposal message, the following events occur:

1. If the sender has a superior BPDU, the local switch realizes that the sender should be the designated switch (thus have the designated port) and its own port should then become a new root port.
2. Before the switch agrees to anything, it must sncronize itself with the topology.
3. All nonedge ports are moved to discarding stae to prevent loops from forming.
4. An agreement message is sent back to the sender, affirming the new designated port choice.  This also lets the sender switch know that it is in the process of syncronizing itself.
5. The root port is moved into forwarding state.  The sender’s port can begin forwarding.
6. For each nonedge port in discarding state, a proposal message is sent to the respective neighbor.
7. An agreement message is expected and recieved.
8. The nonedge port is moved to forwarding state.

Because the recipient of a sync prosal isolate itself from the rest of the network (all other nonedge ports are temporarily in blocking state), the nearest neighbors must also syncronize themselves.  This creates a rippling wave of sayncronizing switches throughout the network which occurs very quickly.  Because timers are not used, changes occur at the speed of BPDU transmissions.

Bridge IDs

In 802.1D, each switch was required to have a unique bridge ID, consisting of a priority value + MAC address.  PVST+ and PVRST+ also require the BID, but they must also include VLAN information within the BID because a unique instance must run for each VLAN on each switch.  Tp accomplish this, a portion of the priority field is used to carry the VID.

Old bridge priority:
Priority value (default 32,768 – increments of one)
 +
 MAC address

New bridge priority:
Priority value (default 32,768 – increments of 4,096)
 +
Extended system ID (12 bit field carrying the VID)
 +
MAC Address
Remember that if the priority value is not manually configured, the root bridge for each VLAN will be determined by lowest MAC address.  Also, keep in mind that the priority value you configure is only half of the actual priority value used by the switch because the VLAN ID is also attached.  Here’s an example:

Default priority field for VLAN 11:
32768 + 11 = 32779

Higher priority for VLAN 11:
28672 + 11 = 28683

RSTP Compatibilty with 802.1D

802.1w is backwards compatible with common spanning tree, but it looses it’s fast convergence benefit for that particular segment.  If a switch recieves BPDUs that do not reflect its current operating mode, for two times the hello time, it switches STP modes.

Spanning Tree Defaults on Catalyst Switches

The default STP mode on current cisco switches is PVST+.  That means that all VLANS will elect the same root bridge and a topology change will impact all VLANs the same.  ALl redundant links would also be blocked in exactly the same manner.

PortFast

Spanning Tree Portfast causes layer 2 switch interfaces to enter forwarding state immediately, bypassing the listening and learning states.  It should be used on ports connected directly to end hosts like servers or workstations. 

Note:  If it isn’t enabled, DHCP timeouts can occur while STP converges, causing more problems.

To configure PortFast
Switch# conf t
Switch (config)# int fa 3/1
Switch (config-if)# [no] spanning-tree portfast

To verify PortFast on an interface:
Switch# sh spanning-tree int fa 3/1 portfast

PortFast can be configured globally on an access switch for all interfaces to save configurations.  Also, it only applies to access interfaces, not trunks.  Use the spanning-tree portfast trunk command if it is required on a trunk.  If you do so, make sure to disable it explicitly on uplink interfaces.

To configure PortFast globally:
Switch# spanning-tree portfast default

Switchport Mode Host

To configure PortFast and disable both channeling and trunking negotiation on an interface:
Switch (config-if)# switchport host

RPVST+ Configuration

1. Enable RPVST+ globally on all switches
   Switch(config)#spanning-tree mode rapid-pvst
2. Designate and configire a primary root brigde
   Switch(config)#spanning-tree vlan 2 root primary
3. Designate and configire a secondary root brigde
   Switch(config)#spanning-tree vlan 2 root secondary
4. Verify the configuration
   Switch#show spanning-tree vlan 2

Spanning Tree Protocol (STP) is designed to prevent problems related to bridging loops.  STP solves the problem by blocking redundant paths and allowing only a single active path.     

Severl different versions of Spanning Tree have been introduced over the years.  Here are a few:     

• Common Spanning Tree (CST)- IEEE 802.1D, One instance of spanning tree runs for the entire switched network resulting in low CPU requirements, but suboptimal traffic paths when multiple VLANs are used.  It is also slow to converge.
• Per VLAN Spanning Tree Plus (PVST+)- One instance of STP per VLAN, more resources required, slow convergence still, includes portfast, BPDU guard, BPDU filter, Root Guard, and loop guard.
• Rapid STP (RSTP)- IEEE 802.1w, One instance of STP, but very fast convergence time.  till suboptimal traffic flows because only a single instance for the entire switched network.
• Multiple Spanning Tree (MST) - An IEEE standard that allows you to map multiple VLANS with similar traffic flow requirements into the same spanning-tree instance.  MST also supports RSTP for fast convergence.  Each instance supports Portfast, BPDU guard, BPDU filter, Root Guard, and loop guard.
• PVRST+- A Cisco enhancement to RSTP that behaves similar to PVST+.  It supports a separate instance of RSTP for each VLAN and each instance supports Portfast, BPDU guard, BPDU filter, Root Guard, and loop guard.  This option has the largest CPU and memory requirements.

Note: MST and PVRST+ have become the dominate spanning-tree protocols of choice and in Cisco switches, PVST+ is the default flavor of STP that is enabled when a VLAN is created.     

STP Convergence

1. Root bridge election
   Each VLAN elects one root bridge.  All ports on the root bridge act as designated ports, which send and receive traffic as well as BPDUs.  The bridge with the lowest priority becomes root.
2. Root ports are determined on all non-root bridges
   Each non-root bridge is assigned a single root port that send and receives traffic.  The root port is chosen based on the port with the lowest-cost path between the non-root bridge and the root bridge.  If two paths are equal cost, the port with the lowest port ID (priority + port number) will win.
3. Designated port selection
   Each segment has a single designated port.  Designated ports are chosen from on non-root ports that have the lowest path cost to the root bridge.  In the event of a tie, the bridge ID acts as a tiebreaker (lowest wins).  All ports on a root bridge are designated ports.
   

STP Port Roles

• Root port
  • On non-root bridges only
  • Forwards traffic towards the root bridge
  • Only one per switch
  • Can populate the MAC table
• Designated port
  • On root and non-root bridges
  • All ports on root bridge are designated ports
  • Receives and forwards frames towards the root bridge as needed
  • Only one per segment
  • Can populate the MAC table
• Nondesignated port
  • Does not forward packets (blocking)
  • Does not populate the MAC table
• Disabled port
  • A port that is shut down

 STP Port States

• Blocking
  • In nondesignated status and does not forward frames
  • Receives BPDUs to determine root switch
  • Defualt 20 seconds in this state (max age)
• Listening
  • Receives and sends BPDUs
  • 15 seconds (forward delay)
• Learning
  • Populates the CAM table
  • 15 seconds (forward delay)
• Forwarding
  • Part of the active topology
  • forwards frames
  • sends and receives BPDUs
• Disabled
  • Does not participate in STP
  • Does not forward frames

 STP Path Cost

Spanning-tree uses a link cost calculation to determine the the root ports on non-root switches.  It is calculated by adding the costs of all links between the root bridge and the local switch.   

• 10 Gbps > Cost 1
• 1 Gbps > Cost 4
• 100 Mbps > Cost 19
• 10 Mbps > Cost 100

The last episode I listened to was particularly good.  Titled, Episode 11: If You Can’t Be Replaced, You Can’t Be Promoted they talk about what it takes to get to become “that guy”, the go-to guy, the true expert in their company.  They also discuss a very interesting topic – Cisco IOS educational licensing.  The idea is that Cisco could create a structure where IOS images where made available to those learning and preparing for Cisco certification.  Apparently Cisco has been kicking around the idea for some time, but there has been no action on it.  I’m fortunate enough to work at a company that gets free access to all the C IOS images for free, but many aren’t in my position and try to get by with just books, which is not the same as actually performing the commands in a CLI.

The PacketPushers even invited Aaron Conaway on from Aaron’s Worthless Words.  Aaron blogs in a very candid manner about his professional development through the Cisco Certification process (both his successes and failures).  He recently went through the SWITCH, ROUTE, and TSHOOT exams for the new curriculum and openly shares his experiences with the content.
Check out both Aaron’s blog as well as the PacketPushers podcast – you won’t be disappointed!