Ch 4 - Debugging Techniques
($Revision: 1.1 $)
Summary
- there are several tools
- the tools all have some overhead, to set up and learn
- all of them have inherent limitations
- they are intrusive, may change the kernel behavior
- a serious kernel fault will kill the tool
- you may ultimately need to fall back on the simpler techniques
Simple Techniques
- output: printk, controllable level of detail
- focussed tests/experiments
- built-in assertions (self-checks)
-
serial console device
Getting Debugging Output: printk
- messages go into a bounded circular buffer
- loglevel string: lower number means more likely to print to console
- if console is not locked, and priority is right, they are printed out immediately
- klogd - periodically checks console log buffer and prints messages
- klogd uses syslogd to log output
- /etc/syslog.con determines what is logged, and where
- for full logging to file, need syslogd and klogd
Places to Look for Log Messages
- console
- /proc/kmsg
- /var/log/messages, or alternate log location
Controlling Logging
- control logging level cut-off
- kernel compilation option - DEFAULT_CONSOLE_LOGLEVEL
- sys_syslog system call - misc-progs/setlevel.c
- write new value to /proc/sys/kernel/printk
- limit messages to one particular console
- ioctl(TIOCLINUX) - misc-progs/setconsole.c
- specifying log file
- /etc/syslog.conf
- -f option to klogd
- cat /proc/kmsg
- Use flags or macros in your own code to make
calls to printk conditional
Debugging Options in Kernel Configuration
- check for stack overflows
- debug high memory support
- debug memory allocations
- memory mapped I/O debugging (for ISA)
- magic sysrq key (see more detail below)
- spinlock debugging (catch missing initialization, combine with watchdog)
- compile the kernel with frame pointers
- -g debug flag
- use System.map and vmlinux files
Remember to copy System.map from the kernel source directory
to /boot, with appropriate renaming. Several kernel debugging
tools depend on it.
Remote Serial Console Device
- Connect serial ports of two computers, A and B.
- Computer A uses the serial port to log its console messages
- Computer B runs a terminal emulator program
(e.g., Linux miniterm or Windows hyperterm)
- Computer B can log messages that would scroll of the local console display of A
- To set this up, you need to
- run a "null modem" cable between the serial ports
(test this cable first, using terminal emulators on both ends)
- compile kernel for A with serial console support
- set the serial console option boot parameter for A
- enable logins on the serial console in the inittab file for A
- be sure to clear out old /etc/ioctl.save
See
serial console device Howto for more details.
Textbook Debugging Techniques
- printing (see printk and logs above)
- querying (see "pulling" below)
- watching
- oops messages
- watchdogs
- debuggers: gdb, kdb, IKD, kgdb
- crash dump analyzers (LKCD)
- user-mode Linux
- Linux trace toolkit (LTT)
- dyamic probes (DProbes)
"Pulling" Information
- printk pushes info to the log, all the time
slows system down, can mask certain bugs
- try pulling info from /proc - you write methods to pull
diagnostic info you need
- try pulling info from ioctl calls - you design in the calls
you need to put out diagnostic info
Defining a /proc Read Function
int (*read_proc) (char *page, char **start, off_t offset,
int count, int *eof, void *data);
int scull_read_procmem(char *buf, char **start, off_t offset,
int count, int *eof, void *data)
{
int i, j, len = 0;
int limit = count - 80; /* Don't print more than this */
for (i = 0; i < scull_nr_devs && len <= limit; i++) {
Scull_Dev *d = &scull_devices[i];
if (down_interruptible(&d->sem))
return -ERESTARTSYS;
len += sprintf(buf+len,"\nDevice %i: qset %i, q %i, sz %li\n",
i, d->qset, d->quantum, d->size);
for (; d && len <= limit; d = d->next) { /* scan the list */
len += sprintf(buf+len, " item at %p, qset at %p\n", d, d->data);
if (d->data && !d->next) /* dump only the last item - save space */
for (j = 0; j < d->qset; j++) {
if (d->data[j])
len += sprintf(buf+len," % 4i: %8p\n",j,d->data[j]);
}
}
up(&scull_devices[i].sem);
}
*eof = 1;
return len;
}
Connecting to a Directory in /proc
#ifdef SCULL_DEBUG /* use proc only if debugging */
int scull_read_procmem(...) { ...see above...}
#ifdef USE_PROC_REGISTER
static int scull_get_info(char *buf, char **start, off_t offset,
int len, int unused)
{
int eof = 0;
return scull_read_procmem (buf, start, offset, len, &eof, NULL);
}
struct proc_dir_entry scull_proc_entry = {
namelen: 8,
name: "scullmem",
mode: S_IFREG | S_IRUGO,
nlink: 1,
get_info: scull_get_info,
};
static void scull_create_proc()
{
proc_register_dynamic(&proc_root, &scull_proc_entry);
}
static void scull_remove_proc()
{
proc_unregister(&proc_root, scull_proc_entry.low_ino);
}
#else /* no USE_PROC_REGISTER - modern world */
static void scull_create_proc()
{
create_proc_read_entry("scullmem", 0 /* default mode */,
NULL /* parent dir */, scull_read_procmem,
NULL /* client data */);
}
static void scull_remove_proc()
{
/* no problem if it was not registered */
remove_proc_entry("scullmem", NULL /* parent dir */);
}
#endif /* USE_PROC_REGISTER */
#endif /* SCULL_DEBUG */
This code is included only if the debugging preprocessor flag is set.
There are two versions, for backward compatibility with older kernels.
Debugging by Watching
- strace gives trace of system calls and arguments
- easy to use: does not require any special preparation
- one can infer much from parameter values and return values
Debugging System Faults
- "oops" messages tell where fault occurred
- more can be extracted, especially with tools
Built-in Kernel Debugging Features (as of 2.6.21.1)
Built-in self-checking and tracing features are more useful than interactive
debugging tools for
large and complex software systems like the Linux kernel.
The following list of built-in kernel debugging options
has been growing over time.
- Show timing information on printks
- Enable __must_check logic
- Magic SysRq key
- Enable unused/obsolete exported symbols
- Debug Filesystem
- Run 'make headers_check'when building vmlinux
- Kernel debugging:
- Debug shared IRQ handlers
- Kernel log buffer size (16 => 64KB, 17 => 128KB)
- Detect Soft Lockups
- Collect scheduler statistics
- Collect kernel timers statistics
- Debug slab memory allocations
- RT Mutex debugging, deadlock detection
- Built-in scriptable tester for rt-mutexes
- Lock debugging: prove locking correctness
- Spinlock and rw-lock debugging: basic checks
- Mutex debugging: basic checks
- Lock debugging: detect incorrect freeing of live locks
- Lock debugging: prove locking correctness
apparently incompatible with the three preceding options
- Lock dependency engine debugging
- Spinlock debugging: sleep-inside-spinlock checking
- Locking API boot-time self-tests
- kobject debugging
- Highmem debugging
- Compile the kernel with debug info
- Debug VM
- Debug linked list manipulation
- Compile the kernel with frame pointers
- Force gcc to inline functions marked
- torture tests for RCU (read, copy, update)
- Fault-injection framework
- Check for stack overflows
- Stack utilization instrumentation
- Page alloc debug (is incompatible with Software Suspend on i386)
- Write protect kernel read-only data structures
- Use 4Kb for kernel stacks instead of 8Kb
If you insist on trying a debugger: kdb
- Maintained by Silicon Graphics, Inc. (http://oss.sgi.com/projects/kdb/
- A set of kernel patches
- Requires a different patch-set version for each minor kernel version and each architecture.
If your architecure is not one of those SGI supports, tough luck.
- Documentation is located in the kdb subdirectory of the Documentation
directory in the patched source tree.
How to install
Adapted from
http://www-128.ibm.com/developerworks/linux/library/l-kdbug/
|
© 2003
T. P. Baker.
No part of this publication may be reproduced, stored in a retrieval system, or
transmitted in any form or by any means without written permission.
(Last updated by $Author: baker $ on $Date: 2010/06/07 14:29:15 $.)
|