ch12.html

Linux Kernel & Device Driver Programming

Ch 12 - PCI Drivers

PCI - Peripheral Component Interconnect

specifies many details of how peripherals connect to computer
supports autodetection of interface boards
devices are generally jumperless, automatically configured at boot time
subsumes PC-Card bus
specification includes both electrical and programming interfaces
we concentrate on the programming aspect

PCI Bus in Context

Bus Architecture of a PC Using a VIA Chipset for the Pentium IV, reproduced from a VIA Technologies web page, at http://www.via.com.

PCI Addressing

address = (bus_number, device_number, function_number)
up to 256 buses
up to 32 devices per bus (connected by "bridges")
up to 8 functions per device
Linux type struct pci_dev used to specify address

Output of lspci -tv

Output is for www.cs.fsu.edu, the Web server, as of summer 2007. Top-level numbers are parsed as domain:bus.

-[0000:00]-+-00.0  Intel Corp. E7501 Memory Controller Hub
           +-00.1  Intel Corp. E7000 Series Host RASUM Controller
           +-02.0-[0000:01-03]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
           |                    +-1d.0-[0000:02]--
           |                    +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
           |                    \-1f.0-[0000:03]--+-02.0  Intel Corp. 82546EB Gigabit Ethernet Controller (Copper)
           |                                      \-02.1  Intel Corp. 82546EB Gigabit Ethernet Controller (Copper)
           +-04.0-[0000:04-06]--+-1c.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
           |                    +-1d.0-[0000:05]--
           |                    +-1e.0  Intel Corp. 82870P2 P64H2 I/OxAPIC
           |                    \-1f.0-[0000:06]--
           +-1d.0  Intel Corp. 82801CA/CAM USB (Hub #1)
           +-1d.1  Intel Corp. 82801CA/CAM USB (Hub #2)
           +-1d.2  Intel Corp. 82801CA/CAM USB (Hub #3)
           +-1e.0-[0000:07]----01.0  ATI Technologies Inc Rage XL
           +-1f.0  Intel Corp. 82801CA LPC Interface Controller
           +-1f.1  Intel Corp. 82801CA Ultra ATA Storage Controller
           \-1f.3  Intel Corp. 82801CA/CAM SMBus Controller

The following is an alternate, more verbose, format of the lspci output for the above system.

00:00.0 Host bridge: Intel Corp. E7501 Memory Controller Hub (rev 01)
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- 

00:00.1 Class ff00: Intel Corp. E7000 Series Host RASUM Controller (rev 01)
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O- Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- TAbort- SERR- Reset- FastB2B-

00:04.0 PCI bridge: Intel Corp. E7000 Series Hub Interface D PCI-to-PCI Bridge (rev 01) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz+ UDF- FastB2B+ ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-

00:1d.0 USB Controller: Intel Corp. 82801CA/CAM USB (Hub #1) (rev 02) (prog-if 00 [UHCI])
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O+ Mem- BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- TAbort- SERR- TAbort- SERR- Reset- FastB2B-

00:1f.0 ISA bridge: Intel Corp. 82801CA LPC Interface Controller (rev 02)
	Control: I/O+ Mem+ BusMaster+ SpecCycle+ MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- 
	Region 1: I/O ports at 
	Region 2: I/O ports at 
	Region 3: I/O ports at 
	Region 4: I/O ports at 2060 [size=16]
	Region 5: Memory at 40000000 (32-bit, non-prefetchable) [size=1K]

00:1f.3 SMBus: Intel Corp. 82801CA/CAM SMBus Controller (rev 02)
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O+ Mem- BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B-
	Status: Cap- 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR- TAbort- SERR- 

01:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-
	Capabilities: 

01:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 

01:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-
	Capabilities: 

03:02.0 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
	Subsystem: Intel Corp. PRO/1000 MT Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR- 

03:02.1 Ethernet controller: Intel Corp. 82546EB Gigabit Ethernet Controller (Copper) (rev 01)
	Subsystem: Intel Corp. PRO/1000 MT Dual Port Server Adapter
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV+ VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=medium >TAbort- SERR- 

04:1c.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 

04:1d.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-
	Capabilities: 

04:1e.0 PIC: Intel Corp. 82870P2 P64H2 I/OxAPIC (rev 04) (prog-if 20 [IO(X)-APIC])
	Subsystem: Super Micro Computer Inc: Unknown device 3880
	Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- 

04:1f.0 PCI bridge: Intel Corp. 82870P2 P64H2 Hub PCI Bridge (rev 04) (prog-if 00 [Normal decode])
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR+ FastB2B-
	Status: Cap+ 66Mhz+ UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- SERR- Reset- FastB2B-
	Capabilities: 

07:01.0 VGA compatible controller: ATI Technologies Inc Rage XL (rev 27) (prog-if 00 [VGA])
	Subsystem: ATI Technologies Inc Rage XL
	Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping+ SERR- FastB2B-
	Status: Cap+ 66Mhz- UDF- FastB2B+ ParErr- DEVSEL=medium >TAbort- SERR-

Numbers above are parsed as bus:device.function.

Other options of lspci produce other forms of output.

One may also get information about devices on the PCI bus by looking at /proc/pci and /proc/bus/pci/devices.

Partial output of /proc/bus/pci/devices

0000    8086254c        0               00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000
0001    80862541        0               00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000        
0010    80862543        0               00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000        
0020    80862547        0               00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000
        00000000        00000000        00000000        00000000        
00e8
        80862482        a9              00000000        00000000
        00000000        00000000        00002001        00000000
        00000000        00000000        00000000        00000000
        00000000        00000020        00000000        00000000
        uhci_hcd
00e9    80862484        b1              00000000        00000000
        00000000        00000000        00002021        00000000
        00000000        00000000        00000000        00000000
        00000000        00000020        00000000        00000000
        uhci_hcd
...
00f9    8086248b        b9              00000000        00000000
        00000000        00000000        00002061        40000000
        00000000        00000000        00000000        00000000
        00000000        00000010        00000400        00000000
        PIIX_IDE
...

The above outputs are both from the the department's web server. Of course, the values will be different for each system.

The /sys/bus/pci pseudo-filesystem als provides information about devices.

Output of ls -l /sys/devices/pci0000:00/0000:00:10.2

total 0
-r--r--r--  1 root root 4096 May 30 08:36 class
-rw-r--r--  1 root root  256 May 30 08:36 config
-rw-r--r--  1 root root 4096 May 30 09:30 detach_state
-r--r--r--  1 root root 4096 May 30 08:36 device
-r--r--r--  1 root root 4096 May 30 08:36 irq (assigned IRQ)
drwxr-xr-x  2 root root    0 Feb 28 04:05 power
-r--r--r--  1 root root 4096 May 30 08:36 resource 
-r--r--r--  1 root root 4096 May 30 09:30 subsystem_device
-r--r--r--  1 root root 4096 May 30 09:30 subsystem_vendor
-r--r--r--  1 root root 4096 May 30 08:36 vendor

PCI Interrupts

four lines per bus
required to be shareable between devices

Addresses Spaces Seen/Implemented by a PCI Device

memory locations - shared by all devices on bus - 32-bit or 64-bit
I/O ports - shared by all devices on bus - 32-bit
configuration registers - addressed per-device ("geographical addressing")
256 bytes for each device function
first 64 bytes standardized
first four bytes hold unique vendor and device ID

PCI Configuration Space

Boot Time

firmware (BIOS, NVRAM/PROM) - performs PCI configuration
allocates non-overlapping memory and I/O address spaces
so, device driver does not need to worry about this

BIOS resolves PCI address conflicts and sets up configuration table at boot time, or when device is plugged in. Device drivers do not need to probe. They can just read the configuration data from the existing table.

PCI Configuration Registers

vendorID (16 bits)
- identifies hardware manufacturer
- e.g., 0x8086 for Intel
- global registrry maintained by PCI Special Interest Group
deviceID (16 bits)
- decided by manufacturer
- signature = <vendorID, deviceID>
class (16 bits)
- to 8 bits identify the base class (group)
- e.g., network group contains Ethernet and token ring classes
subsystem vendorID
subsystem deviceID

PCI register values are always little endian. For portability, you should use the functions and macros in asm/byteorder.h.

Recall that "little endian" means the low-order ("little") byte of the number is stored at the lowest address (and the high-order byte is stored a the highest address). This is in contrast to "big endian". The Intel processor architecture is little-endian. Since PCI is used with more than one processor architecture, a pci I/O device may be used with more than one processor architecture, and so it is desirable that the device driver not depend on any assumptions about the PCI devices and the processor having the same endian-ness.

Look at function calls to access PCI configuration data in

Compare code in Windows driver for Pixelsmart frame grabber:

   // Scan for grayscale digis
   pLDI->pciDigis += getPCIBaseAddresses(PCI_PIXELSMART_VENDORID, PCI_PIXELSMART_GRAYDID, &pciData, &BAR0[0], &busNumList[0], &slotNumList[0]);
   // Fill in digiType[]
   setDigiType(&BAR0[0], &dt[0], PCI_DIGI_GRAYSCALE);
   // Scan for color digis
   pLDI->pciDigis += getPCIBaseAddresses(PCI_PIXELSMART_VENDORID, PCI_PIXELSMART_COLORDID, &pciData, &BAR0[0], &busNumList[0], &slotNumList[0]);
   // Fill in digiType[]
   setDigiType(&BAR0[0], &dt[0], PCI_DIGI_COLOR);

Generic Kernel Support for PCI Devices

pci_register_driver - the way to register a PCI driver
pci_register_driver
pci_unregister_driver - the way to unregister a driver
pci_get_device and pci_dev_put - a lower-level way of getting information about a PCI device
pci_enable_device - wakes up the device and may assign IRQ and I/O regions
various functions for accessing PCI configuration data, including:
- pci_read_config_byte, pci_write_config_byte, etc.
- pci_bus_read_config_byte, pci_bus_write_config_byte, etc.
- pci_resource_start - first address of specified I/O memory address or port number range
- pci_resource_end - address of last byte of specified I/O range
- pci_resource_flags - returns flags associated with specfified resource
  - IORESOURCE_IO or IORESOURCE_MEM - whether a range of ports or memory addresses
  - IORESOURCE_PREFETCH - whether prefetchable
  - IORESOURCE_READONLY - whether write protected (never set for PCI resources)

We will look at the newer, more abstract, PCI support functions. If you read the code of older drivers, you will find cases where they drop down to older, lower-level, PCI support functions. It is preferable to use the higher-level interfaces in new drivers.

Kernel pci_device_id Structure

Used to tell kernel about devices a driver supports.

struct pci_device_id {
        __u32 vendor, device;           /* Vendor and device ID or PCI_ANY_ID*/
        __u32 subvendor, subdevice;     /* Subsystem ID's or PCI_ANY_ID */
        __u32 class, class_mask;        /* (class,subclass,prog-if) triplet */
        kernel_ulong_t driver_data;     /* Data private to the driver */
};

Example of a Real PCI Driver

The bttv driver, is the driver for a family of "Brooktree" video input cards.

The bttv module initializer when calls pci_register_driver.

ret = pci_register_driver(&bttv_pci_driver);

The parameter to that call is a reference to the struct pci_driver object bttv_pci_driver:

static struct pci_driver bttv_pci_driver = {
    .name     = "bttv",
    .id_table = bttv_pci_tbl,
    .probe    = bttv_probe,
    .remove   = __devexit_p(bttv_remove),
    .suspend  = bttv_suspend,
    .resume   = bttv_resume,
};

The components "suspend" and "resume" are optional. The others are required.

The id_table component refers to the list of struct pci_device_ide values given in bttv_pci_tbl

static struct pci_device_id bttv_pci_tbl[] = {
    {PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT848,   
     PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
    {PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT849,
     PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
       {PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT878,
        PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
       {PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT879,
        PCI_ANY_ID, PCI_ANY_ID, 0, 0, 0},
       {0,}
};

According to the text, it is better to use the helper macros to build such tables, e.g.

static struct pci_device_id bttv_pci_tbl[] = {
   {PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT848),
    PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT849),
    PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT848),
    PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT849),
    PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT878),
    PCI_DEVICE(PCI_VENDOR_ID_BROOKTREE, PCI_DEVICE_ID_BT879),
       {0,}
};

The device id is given by constants like PCI_DEVICE_ID_BT848

#define PCI_DEVICE_ID_BT848     0x350

The vendor id is given by constants like PCI_VENDOR_ID_BROOKTREE

#define PCI_VENDOR_ID_BROOKTREE         0x109e

The function bttv_probe will be called by the PCI subsystem to probe for all devices whose IDs match an entry in the table bttv_pci_tbl. It will be called when the driver registers, and will be called later if a new card is plugged in. The function returns 0 (success) if the device corresponding to the dev and pci_id parameters is one of the devices that the driver can drive. Otherwise, it returns a negative error code. If the probe succeeds, it also does whatever per-device initialization is required, including creation of a driver-specific structure btv, and hangs a pointer this driver-specific data on the "hook" of the struct pci_dev object corresponding to dev via a call to pci_set_drvdata.

The function bttv_remove is called by the PCI subsystem whenever a PCI device is removed. It gives the driver a chance to undo whatever it did in the probe function, including release of any resources allocated for driving that device and removing all references to that device.

static void __devexit bttv_remove(struct pci_dev *pci_dev)
{
...
}

The function bttv_suspend is called by the PCI subsystem whenever the device is suspended

static int bttv_suspend(struct pci_dev *pci_dev, u32 state)
{
....
}

The function bttv_resume is called by the PCI subsystem whenever the device is resumed

static int bttv_resume(struct pci_dev *pci_dev)
{
....
}

The driver module cleanup function should call pci_unregister_driver to undo the driver registration. This will call the remove method specified in the driver registration for all devices that have been successfully probed and which have not previously been removed.

pci_unregister_driver(&bttv_pci_driver);