INSTRUCTOR VERSION
OVERVIEW
Debug mode is a feature of the Cisco IOS software to locate router configuration errors and software bugs. Log messages are similar to debug messages and are generally alerts to problems. You can think of log messages as debug messages that cannot be turned off. Problems are diagnosed by reviewing descriptive messages generated by the router. There are hundreds of different debug options that can be individually turned on and off depending on what part of the system is under examination. It is possible to turn on all debug modes simultaneously, however, this is rarely appropriate as the volume of information would be too voluminous. Debug mode should generally not be used on a production network as it is easy to generate hundreds of error messages per second and cause a router to crash and reboot. We will also explore some of the “show” commands used for debugging problems. This lab assignment assumes you have the base router configuration from the “Cisco Router Basics” loaded with the RIP routing protocol. The following is a sample of some debug and log messages. I have removed the timestamps to fit the messages on the page.
(Sample of
debug and log messages)
r1#term
monitor
r1#debug
all
This may
severely impact network performance. Continue? [confirm]y
All
possible debugging has been turned on
%LINEPROTO-5-UPDOWN:
Line protocol on Interface Serial1/2, changed state to down
%LINK-3-UPDOWN:
Interface Serial1/2, changed state to up
%SYS-5-CONFIG_I:
Configured from memory by console
%SYS-5-RESTART:
System restarted --
Cisco
Internetwork Operating System Software
IOS (tm)
GS Software (GS7-J-M), Version 11.1(24), RELEASE SOFTWARE (fc1)
%ENVM-2-SUPPLY:
Upper Power Supply is Non-Operational
%LINK-4-FDDISTAT:
Interface Fddi0/0, FDDI state c_wrap_b detected?
IP:
s=192.168.16.6 (Serial1/6), d=224.0.0.10, len 64, dispose 31
SMT I:
Fddi0/0, FC=SMT, DA=0000.309c.fb2d, SA=0000.309c.9e3f,
IP:
s=192.168.16.6 (Serial1/6), d=255.255.255.255, len 176, rcvd 2
UDP: rcvd
src=192.168.16.6(520), dst=255.255.255.255(520), length=152
RIP:
received v1 update from 192.168.16.6 on Serial1/6
0.0.0.0 in 5 hops
192.168.13.0 in 16 hops (inaccessible)
192.168.66.0 in 1 hops
Serial1/2:
HDLC myseq 8, mineseen 8*, yourseen 11, line up
RIP:
sending v1 update to 255.255.255.255 via Serial1/2 (192.168.12.1)
default, metric 6
network 192.168.66.0, metric 2
RIP:
Update contains 21 routes
RIP:
Update queued
RIP:
Update sent via Serial1/2
CDP-PA:
Packet received from cat1 on interface Ethernet2/0
r1#undebug all
PART 1 – SHOW COMMANDS:
Although not technically debug commands, there are several “show” commands that are helpful with debugging and worth mentioning. Read about the following “show” commands using either the hardcopy Cisco manuals or online manuals at www.cisco.com and try them out on your router. Include a brief description what each of these commands does for your assignment:
Using information gathered on your router using the above “show” commands, answer the following questions:
1. What IOS software is your router running? What is the filename of the IOS image? How much RAM? FLASH? What is the value of the configuration register? What model CPU does your router have?
2. For each of your router’s serial WAN interfaces, what kind of cable is attached (DTE, DCE, or none)?
3. Which adjacent routers are sending CDP messages to your router? What IOS software version is running on the adjacent CDP routers?
4. What is the MAC address of your router’s FDDI interface?
5. For each of your router’s active interfaces, is IP Split-Horizon enabled?
6. For the RIP protocol running on your router, what are the values of the RIP protocol update, invalid, holddown, and flush timers?
7. How much TOTAL, USED, and FREE RAM is in your router?
8. What is the average CPU utilization for the last 5 minutes?
9. On your 7000 router, what card is physically located in slot 0? What is its hardware revision and serial number?
Here is info on the above commands and some captures from
the commands:
show version
This displays the Cisco IOS software version information, amount of DRAM memory, amount of FLASH memory, etc.
show controller cbus (7000 only)
This will show many internal register values but on 7000 serial ports also will display whether a cable is present and whether the cable is DTE or DCE.
show controller serial (2511 only)
This shows many registers but also indicates whether a serial cable is present, and if so, whether the cable is DCE or DTE.
show cdp neighbors [detail]
If the Cisco Discovery Protocol is enabled, this will display the neighboring devices, the device types, the interface the device was heard from, and the remote device’s interface adjacent to your router. The detail keyword shows additional info such as the neighbor’s IOS software version.
show interface
Display detailed information about an interface.
show ip interface [brief]
Shows IP related information for an interface such as whether split-horizon is enabled. With the brief keyword, the display is condensed to one line per interface indicating the up/down status and IP address.
show ip protocol
Shows which routing protocols are running including verbose details on the protocol timers, neighbors, etc.
show memory
Show information on the current memory utilization.
show processes cpu
Show CPU usage information on the current router processes.
show diagbus (7000
only)
Show physical card information for the card slots in a 7000 chassis such as serial numbers, hardware revision numbers, etc.
show tech-support
This displays a very long list of information by concatenating the output of many “show” commands. When working with Cisco Technical support, they will often ask you to capture the output of this command and send it to them to help debug a problem. This command generates hundreds of lines of output. It also includes many of the previous SHOW commands.
What IOS software is your router running? What is the filename of the IOS image? How much RAM? FLASH? What is the value
of the configuration register? What
model CPU does your router have?
r1#show version
IOS (tm) GS Software (GS7-J-M), Version 11.1(24), RELEASE SOFTWARE (fc1)
System image file is "gs7-j-mz.111-24.bin", booted via flash
cisco RP1 (68040) processor (revision A0) with 65536K bytes of memory.
4096K bytes of flash memory sized on embedded flash.
Configuration register is 0x2102
For each of your router’s serial WAN interfaces, what kind
of cable is attached (DTE, DCE, or none)?
r3#show controller cbus
Interface 8 - Serial1/0, electrical interface is V.35 DTE
Interface 9 - Serial1/1, electrical interface is V.35
DCE
Interface 10 - Serial1/2, electrical interface is V.35 DCE
Interface 11 - Serial1/3, electrical interface is V.35
DCE
Interface 12 - Serial1/4, electrical interface is V.35 DTE
Interface 13 - Serial1/5, electrical interface is Universal (cable unattached)
Interface 14 - Serial1/6, electrical interface is V.35
DTE
Interface 15 - Serial1/7, electrical interface is Universal (cable unattached)
Which adjacent routers are sending CDP messages to your
router? What IOS software version is
running on the adjacent CDP routers?
r3#show cdp neighbor
Device ID Local Intrfce Holdtme Capability Platform Port ID
r2 Fddi0/0 150 R RP1 Fddi0/0
r2 Ser 1/2 150 R RP1 Ser 1/3
r3 Ser 1/0 146 R RP1 Ser 1/3
r3 Ser 1/3 146 R RP1 Ser 1/0
r1 Fddi0/0 170 R RP1 Fddi0/0
r1 Ser 1/1 170 R RP1 Ser 1/3
r4 Ser 1/4 137 R RP1 Ser 1/3
r4 Fddi0/0 137 R RP1 Fddi0/0
r5 Fddi0/0 162 R 4500 Fddi0
fw/r6 Ser 1/6 125 R 2511 Ser 1
r3#show cdp neighbor Serial 1/6 detail
IOS (tm) 2500 Software (C2500-D-L), Version 12.0(13), RELEASE SOFTWARE (fc1)
What is the MAC address of your router’s FDDI interface?
r3#show int fddi 0/0
Fddi0/0 is up, line protocol is up
Hardware is cxBus FDDI, address is 0000.0c30.34ad (bia 0000.0c30.34ad)
Internet address is 192.168.1.3/24
For each of your router’s active interfaces, is IP
Split-Horizon enabled?
R3# show ip interface
Fddi0/0 is up, line protocol is up
Split horizon is enabled
Serial1/1 is up, line protocol is up
Split horizon is enabled
Serial1/2 is up, line protocol is up
Split horizon is enabled
Serial1/4 is up, line protocol is up
Split horizon is enabled
Serial1/6 is up, line protocol is up
Split horizon is enabled
Loopback0 is up, line protocol is up
Split horizon is enabled
For the RIP protocol running on your router, what are the
values of the RIP protocol update, invalid, holddown, and flush timers?
r3#show ip protocol
Routing Protocol is "rip"
Sending updates every 30 seconds, next due in 18 seconds
Invalid after 180
seconds, hold down 180, flushed after 240
How much TOTAL, USED, and FREE RAM is in your router?
r3#show memory
Head Total(b) Used(b) Free(b) Lowest(b) Largest(b)
Processor 8F5E94 57713004 2139924 55573080 55470908 55470972
What is the average CPU utilization for the last 5 minutes?
r3#show processes cpu
CPU utilization for five seconds: 4%/3%; one minute: 6%; five minutes: 6%
PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
1 145932 69087 2112 0.00% 0.09% 0.08% 0 RIP Router
2 2540092 26008 97665 0.00% 2.51% 2.05% 0 Check heaps
3 0 1 0 0.00% 0.00% 0.00% 0 Pool Manager
On your 7000 router, what card is physically located in slot
0? What is its hardware revision and
serial number?
r3#show diagbus
Slot 0:
Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
FIP controller, HW rev 2.9, board revision B0
Serial number: 01240876 Part number: 73-0892-04
PART
2 – SET THE CLOCK:
Debug messages are often examined on multiple router devices to study the sequence of events. It is often very useful to configure the debug messages to include a timestamp in order to correlate events in different log files. Setting the router clock is important to make the correlation possible. The current system clock can be displayed with the “show clock” command and set with the “clock set” command. Like UNIX, the Cisco router internally maintains the time as a long integer indicating the number of seconds that have elapsed since January 1st, 1970 GMT (Greenwich Mean Time). Sometimes GMT is called UTC (Universal Time Coordinated). By setting the appropriate time zone, number of hours offset from UTC, and daylight savings time information, the router can display the correct local time. Configure your router’s time zone and daylight savings time information. Configure so that your router will display the local time appropriately and adjust automatically between standard time and daylight savings time. Manually set your router’s clock.
r1#show clock
*20:06:59.713 UTC Sat Dec 9 2000 ß (‘*’ indicates clock is unsynced)
r1#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
r1(config)#clock timezone ?
WORD name of time zone
r1(config)#clock timezone EST ?
<-23 - 23> Hours offset from UTC
r1(config)#clock timezone EST -5
r1(config)#clock summer-time EDT recurring
r1(config)#^Z
r1#show clock
*15:08:08.361 EST Sat Dec 9 2000
r1#clock set ?
hh:mm:ss Current Time
r1#clock set 15:05:00 ?
<1-31> Day of the month
MONTH Month of the year
r1#clock set 15:05:00 09 Dec 2000
r1#show clock
15:05:03.647 EST Sat Dec 9 2000 ß (No ‘*’ indicates clock is synced)
(Note the hours use a military-style 24-hour clock and
indicate
hours:minutes:seconds.milliseconds)
PART
3 – NETWORK TIME PROTOCOL:
In the previous section, we saw how to manually set the router clock and timezone information. Sometimes it is helpful to automatically keep the clocks in sync or synchronize them more accurately than can be done manually. Cisco routers include software that implements the NTP (Network Time Protocol) version 3. NTP can typically maintain the clock accuracy within a few milliseconds. NTP devices maintain relationships with other NTP devices such as “master”, “client”, and “peer”. Each NTP device has a stratum number which indicates the clock’s accuracy and believability. We will configure routers R1, R2, R3, R4, and R5 as NTP clients of router R6, a stratum 4 NTP server. Configure your router to be an NTP client of NTP server R6. Verify that your clock is synchronized using the “show ntp status”, “show ntp associations”, and “show ntp associations detail” commands. A full discussion of NTP is beyond the scope of this document, however, additional information can be found at http://www.eecis.udel.edu/~ntp/.
The configuration command “ntp server w.x.y.z” configures a
router to be an NTP client of server w.x.y.z.
The commands “show ntp status”, “show ntp associations”, and “show ntp
associations detail” display information relating to NTP.
fw/r6#show
ntp status
Clock is synchronized,
stratum 4, reference is 128.186.121.10
nominal freq
is 250.0000 Hz, actual freq is 249.9996 Hz, precision is 2**19
reference time
is BDDD0C96.D66965F4 (14:55:02.837 EST Sat Dec 9 2000)
clock offset
is -0.4068 msec, root delay is 35.08 msec
root
dispersion is 31.43 msec, peer dispersion is 0.64 msec
fw/r6#show
ntp associations
address ref clock
st when poll reach
delay offset disp
*~128.186.121.10 128.186.121.41 3 179 1024
377 6.2 -0.41
0.6
* master (synced), # master (unsynced), +
selected, - candidate, ~ configured
r1#show ntp
status
Clock is
synchronized, stratum 5, reference is 192.168.66.6
nominal freq
is 250.0000 Hz, actual freq is 249.9989 Hz, precision is 2**19
reference time
is BDDE84E4.9553F4CC (22:40:36.583 UTC Sun Dec 10 2000)
clock offset
is 4.60 msec, root delay is 38.01 msec
root
dispersion is 38.35 msec, peer dispersion is 1.85 msec
r1#show ntp
associations
address ref clock
st when poll reach
delay offset disp
*~192.168.66.6 128.186.121.10 4 34
64 377 6.7 4.60 1.8
* master (synced), # master (unsynced), +
selected, - candidate, ~ configured
r1#show ntp
associations detail
192.168.66.6
configured, our_master, sane, valid, stratum 4
ref ID
128.186.121.10, time BDDE849A.F6831ECE (22:39:22.962 UTC Sun Dec 10 2000)
our mode
client, peer mode server, our poll intvl 64, peer poll intvl 64
root delay
31.31 msec, root disp 31.91, reach 377, sync dist 52.750
delay 6.70
msec, offset 4.596 msec, dispersion 1.85
precision
2**19, version 3
org time
BDDE84E4.95A5E3E9 (22:40:36.584 UTC Sun Dec 10 2000)
rcv time
BDDE84E4.9553F4CC (22:40:36.583 UTC Sun Dec 10 2000)
xmt time
BDDE84E4.9370F437 (22:40:36.575 UTC Sun Dec 10 2000)
filtdelay
= 7.20 6.70 7.14 6.96
7.31 6.77 6.65 182.46
filtoffset
= 4.86 4.60 4.75 4.72
5.06 4.88 4.78
92.65
filterror
= 0.00 0.98 1.95 2.93
3.91 4.88 5.86
6.84
PART
4 – TIMESTAMPS:
Timestamps can be prepended to debug or log messages. A timestamp can be either an indication of the uptime (how much time has elapsed since the router was booted) or the current date and time. The date and time can be in UTC or the local timezone. Optionally, the timezone and/or the number of milliseconds can be included. Configure your router so that timestamps for both DEBUG and LOG messages will display the local time including the timezone and millisecond information. Verify that it is working.
fw/r6#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
fw/r6(config)#service timestamp ?
debug Timestamp debug messages
log Timestamp log messages
<cr>
fw/r6(config)#service timestamp debug ?
datetime Timestamp with date and time
uptime Timestamp with system uptime
<cr>
fw/r6(config)#service timestamp debug datetime ?
localtime Use local time zone for timestamps
msec Include milliseconds in timestamp
show-timezone Add time zone information to timestamp
<cr>
fw/r6(config)#service timestamp debug datetime localtime ?
msec Include milliseconds in timestamp
show-timezone Add time zone information to timestamp
<cr>
fw/r6(config)#service timestamp debug datetime localtime
show-timezone
fw/r6(config)#service timestamp log datetime localtime show-timezone
fw/r6(config)#^Z
fw/r6#
PART
5 -- OUTPUT OPTIONS:
Debug and log messages generated have three different modes of output: (1) console screen, (2) internal circular buffer, or (3) syslog server.
Using the console screen is probably the simplest way to view messages as they are generated. The command “term monitor” enables the display of messages while “term no monitor” inhibits the messages.
Part of a router’s RAM memory can be allocated to be a circular logging buffer using the configuration command “logging buffer XXXX” where XXXX indicates the size of the buffer. The contents of the buffer can be displayed with the “show logging” command.
A syslog server is a TCP/IP service that accepts log messages and appends them to log files. Both UNIX and NT server systems can be configured as syslog servers. Syslog servers can be used to centralize the collection of messages from many systems to ease system administration. Syslog uses the concepts of facility and severity level. Facility classifies the messages by subsystem to allow the server to append the proper log file. The severity level provides an indication of the importance of an error message where the system manager can set a severity level threshold on both the router and syslog server. On the router, messages with lower priority than the threshold are never sent to the syslog server. A threshold set on the syslog server indicates the minimal importance necessary for a message to be logged to a file which is otherwise discarded. By default, Cisco routers use the syslog facility “local7” and severity “informational”, but these parameters are adjustable. Severity “informational” will send more messages except those with severity “debug”. In this part, we will use severity level “debug” so that all messages are important enough to be forwarded from the router to the syslog server and all will be logged by the syslog server.
Configure your router so that debug messages will be logged to three different locations (1) to the console screen, (2) to the internal circular buffer, and (3) to your Linux system using facility “local7” and “severity debug”. Your Linux server should append the messages to file /var/log/cisco.log. We will work on generating messages in the next part.
r1#configure terminal
r1(config)#logging ?
WORD IP address of the logging host
buffered Copy logging messages to an internal buffer
console Set console logging level
facility Facility parameter for syslog messages
monitor Set terminal line (monitor) logging level
on Enable logging to all supported destinations
trap Set syslog server logging level
r1(config)#logging buffer ?
<4096-2147483647> buffer size
r1(config)#logging buffer 5000
r1(config)#logging trap debug
r1(config)#logging facility local7
r1(config)#logging 192.168.10.2
r1(config)#^Z
r1#term monitor
The syslog server example below is a Linux system which is
configured to log messages received on facility local7 with all severity levels
to file /var/log/cisco.log:
[curci@s1 /]$ cat
/etc/rc.d/init.d/syslog
#!/bin/sh
# syslog Starts syslogd/klogd.
start)
echo -n "Starting system logger:
"
daemon syslogd -m 0 –r
ß (Use option –r to accept syslog
from network)
...
[curci@s1 /]$ grep
local7 /etc/syslog.conf
# Log local7
at debug level to /var/log/cisco.log
local7.debug /var/log/cisco.log
[curci@s1 /]$ tail
-5 /var/log/cisco.log
Dec 9 16:43:08 192.168.10.1 174: Dec 9 15:36:02 EST: RIP: Update contains 24
routes
Dec 9 16:43:09 192.168.10.1 177: Dec 9 15:36:02 EST: RIP: sending v1 update to
255.255.255.255 via Loopback0 (192.168.11.1)
Dec 9 16:43:09 192.168.10.1 178: Dec 9 15:36:02 EST: RIP: Update contains 24 routes
Dec 9 16:43:09 192.168.10.1 179: Dec 9 15:36:02 EST: RIP: Update queued
Dec 9 16:43:09 192.168.10.1 180: Dec 9 15:36:02 EST: RIP: Update sent via
Loopback0
(A
timestamp prepended by the Linux server is recorded, followed by the IP address
or name of the router sending the message, followed by the Cisco router
timestamp, followed by the actual syslog message.)
PART 5 – DEBUG MODE:
The command “debug” is used to enable the various debug modes. You can see the options with “debug ?”. Each debug mode can be individually enabled or disabled using “debug xxxxx” to turn on a mode or “no debug xxxxx” to turn one off. The command “show debug” displays which debug modes are currently enabled. You can use the command “debug all” to turn on all debug modes, but it is generally not useful as it can generate hundreds of messages per second. You can turn off all debug modes with the command “no debug all” or “undebug all”. Turn on icmp debugging “debug ip icmp” and ping one of your router’s interfaces. Turn off debugging. Review the messages on your console screen, in your circular buffer, and on your syslog server’s /var/log/cisco.log file. Are the entries identical? If not, explain what is different
r1#debug ip icmp
ICMP packet debugging is on
r1#show debug
Generic IP:
ICMP packet debugging is on
r1#ping
Protocol [ip]:
Target IP address: 192.168.30.1
Repeat count [5]: 1
Sending 1, 100-byte ICMP Echoes to 192.168.30.1, timeout is 2 seconds:
!
Success rate is 100 percent (1/1), round-trip min/avg/max = 4/4/4 ms
r
r1# (console screen display)
Dec 10 01:54:17 EST: ICMP: echo reply sent, src
192.168.30.1, dst 192.168.30.1
Dec 10 01:54:17 EST: ICMP: echo reply rcvd, src 192.168.30.1, dst 192.168.30.1
r1#undebug all
All possible debugging has been turned off
r1#show logging
Syslog logging: enabled (0 messages dropped, 0 flushes, 0 overruns)
Console logging: level debugging, 819 messages logged
Monitor logging: level debugging, 0 messages logged
Trap logging: level debugging, 806 message lines logged
Logging to 192.168.10.2, 753 message lines logged
Buffer logging: level debugging, 20 messages logged
Log Buffer (5000 bytes):
Dec 10 01:54:06 EST: %SYS-5-CONFIG_I: Configured from console by console
Dec 10 01:54:17 EST: ICMP: echo reply sent, src
192.168.30.1, dst 192.168.30.1
Dec 10 01:54:17 EST: ICMP: echo reply rcvd, src 192.168.30.1, dst 192.168.30.1
r1#
(This is from the Linux server looking at the tail end of
the log file)
[curci@s1 log]$ tail -2 /var/log/cisco.log
Dec 10 02:59:38 192.168.10.1 805: Dec 10 01:54:17 EST: ICMP:
echo reply sent, src 192.168.30.1, dst 192.168.30.1
Dec 10 02:59:38 192.168.10.1 806: Dec 10 01:54:17 EST: ICMP:
echo reply rcvd, src 192.168.30.1, dst 192.168.30.1
The console and circular buffer messages are identical. The syslog messages on the Linux system are prefixed with a Linux timestamp and the IP address or name of the router that sent the message. The two timestamps in the Linux log are about 5 minutes apart because the Linux system and router clocks are about 5 minutes different. If both systems used NTP, it would help, however, normally the Linux timestamp will be a few seconds later than the other timestamp.