Linux Kernel & Device Driver Programming

Ch 14 - The Linux Device Model

This file uses the W3C HTML Slidy format. The "a" key toggles between one-slide-at-a-time and single-page mode, and the "c" key toggles on and off the table of contents. The ← and → keys can be used to page forward and backward. For more help on controls see the "help?" link at the bottom.

Linux Grand Unified Device Model

data structure, reflecting relationships between devices
made visible through sysfs (the /sys virtual filesystem)
supports:
- power management & system shutdown
- communications with user space
- hot-plug devices
- device classes
- object lifecycles (including reference count management)

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

The /sys filesystem is large. Executing "ls -R /sys" on the CS Dept. Web server resulted in 2414 lines of output.

USB Mouse Driver Device Model Entities (LDD3)

This has changed somewhat, as shown on the next slide.

Example: USB Mouse Driver Device Model Entities

and this is not complete

Kobjects, Ksets, and Subsystems

struct kobject is used for:

reference counting
sysfs representation
"data structure glue" - representing relationships between devices
hotplug event handling

Example: `struct cdev`

Objects of type struct kobject are embedded within other types of objects, to provide capabilities
For example, see type struct cdev

Finding the struct cdev object associated with a struct kobject pointed to by kp
```
struct cdev *device = container_of(kp, struct cdev, kobj);
```

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

Observe that container_of is not checked. It will not work correctly if kp is pointing to an object that is not contained in an object some type of object with the specified type and field name.

Kobject initialization

The kobj field of struct cdev is initialized in cdev_alloc:

  struct cdev *p = kzalloc(sizeof(struct cdev), GFP_KERNEL);
  if (p) {
     INIT_LIST_HEAD(&p->list);
     kobject_init(&p->kobj, &ktype_cdev_dynamic);
  }

The kobj field of struct cdev is named in register_chrdev:

kobject_set_name(&cdev->kobj, "%s", name);
   for (s = strchr(kobject_name(&cdev->kobj),'/'); s; s = strchr(s, '/')) *s = '!';
  err = cdev_add(cdev, MKDEV(cd->major, 0), 256);
  if (err) goto out:
  cd->cdev = cdev;
  return major ? 0 : cd->major;
out:
  kobject_put(&cdev->kobj);...

Reference Count Manipulation

The reference count is incremented by

struct kobject *kobject_get(struct kobject *kobj);

It is decremented by
```
void kobject_put(struct kobject *kobj);
```
through a call to kref_put, which cleans up by calling release(kref) if the reference count goes to zero
A release method must be defined for each type of kobject
```
void my_object_release(struct kobject *kobj);
```

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

Go back to the example above to see how this fits in.

The function cdev_alloc initializes the reference count to 1 by calling kobject_init, which calls kref_init.

Therefore, the error-recovery code inside register_chrdev must decrement the reference count before freeing the struct cdev object.

kobject_put calls kref_put and passes along the kobject_release function, which kref_put calls if the reference count goes to zero.

The kobject_release function calls the release method of the ktype of the specific kobject.

Go back and look at cdev_alloc to see where the ktype is set to ktype_cdev_dynamic.

`struct kobj_type`

Every kobject has a field type

This describes the type of a kobject, and provides members

        void (*release)(struct kobject *);
        struct sysfs_ops        * sysfs_ops;
        struct attribute        ** default_attrs;

For members of sets this is overridden by the type of the set
A release method must always be specified

Use the standard function to extract the type:

struct kobj_type *get_ktype(struct kobject *kobj);

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

In the case of struct cdev, only the release method is specified as

static struct kobj_type ktype_cdev_dynamic = {
        .release        = cdev_dynamic_release,
};

static void cdev_dynamic_release(struct kobject *kobj)
{
        struct cdev *p = container_of(kobj, struct cdev, kobj);
        cdev_purge(p);
        kfree(p);
};

Kobject Hierarchies

Each kobject has a parent
Each kobject has a kset (possibly null)
Usually, if kset if not null is same as parent, but not always

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

It is possible (though not shown in the diagram) that the parent and kset pointers of a kobject could point to different objects.

It is also seems possible (though not shown in the diagram) that an object belongs to more than one kset, in which case the kset pointer of the kobject could only point to one of the ksets.

Ksets

struct kset {
        struct list_head list;
        spinlock_t list_lock;
        struct kobject kobj;
        struct kset_uevent_ops * uevent_ops;
};

A kset represents a collection of kobjects of the same type
The ktype is the type of the elements in the set
Each fully "set up" kset corresponds to a sysfs directory
Each kset "is" an object, in the sense of containing a struct kobject field
To add kobject to the set specified by its kset field
```
int kobject_add(struct kobject *kobj);
```
To delete
```
void kobject_del(struct kobject *kobj);
```

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

kobject_register combines initialization and adding to a set, and kobject_unregister combines deletion from set and "put".

There have been many changes to this structure between kernels 2.6.16 and 2.6.25, and between 2.6.25 and 2.6.31. For example, there was once a type struct subsystem, and a field subsys in struct kset. Instead, there is now list.

Operations on Ksets

Operations are extensions of those on struct kobject:

kset_init
kset_add
kset_register
kset_unregister
kset_get
kset_put
kset_set_name

Kobjects and Sysfs

kobject_add creates a sysfs entry for the object
each object corresponds to a directory
the directory contains attributes of the object
names of objects must be unique within the containing (parent) directory
if object has null parent then kobject_add sets parent to kset
if both are null, object becomes child-member of top-level sys directory

Attributes

struct attribute {
         const char     *name;  /* as it appears in a sysfs directory */
         struct module  *owner; /* no longer used */
         mode_t          mode;  /* file protection bits, e.g., S_IRUGO */
};

each struct kobj_type has a list of default attributes

implementation is provided by kobj_type->sysfs_ops

struct sysfs_ops {
        ssize_t (*show)(struct kobject *, struct attribute *,char *);
        ssize_t (*store)(struct kobject *,struct attribute *,const char *, size_t);
};

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

The kobj_types for subsystem cdev, including ktype_cdev_dynamic, do not specify any default attributes or provide any sysfs_ops to show or store them.

We can (finally) find an example of default attributes via a struct driver_attribute in the declaration of driver_attribute_serio_driver_attrs. This defines "description" and "bind_mode" attributes, and an example of a show method in serio_driver_show_description.

Non-Default Attributes

To create a sysfs attribute entry for attr under kobj

int sysfs_create_file(struct kobject *kobj, struct attribute *attr);

To remove one

int sysfs_remove_file(struct kobject *kobj, struct attribute *attr);

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

For example, see fs_add_slot in linux/drivers/pci/hotplug/pci_hotplug_core.c.

Binary Attributes

struct bin_attribute {
        struct attribute        attr;
        size_t                  size;
        void                    *private;
        ssize_t (*read)(struct kobject *, struct bin_attribute *,
                        char *, loff_t, size_t);
        ssize_t (*write)(struct kobject *, struct bin_attribute *,
                         char *, loff_t, size_t);
        int (*mmap)(struct kobject *, struct bin_attribute *attr,
                    struct vm_area_struct *vma);
};
int sysfs_create_bin_file(struct kobject * kobj, struct bin_attribute * attr)
int sysfs_remove_bin_file(struct kobject * kobj, struct bin_attribute * attr)

special case, for large chunks of binary data
e.g., proprietary configuration data/firmware for a device (not allowed in Gnu sources)
scenario:
- device is detected
- user-space program is fired up, passes binary data to kernel via sysfs binary attribute
- kernel module uses binary data

Symbolic Links

int sysfs_create_link(struct kobject * kobj, struct kobject * target, char * name);
void sysfs_remove_link(struct kobject * kobj, char * name);

Symbolic links allow an object to exist at more than one place in the sysfs hierarchy
Watch out for dangling links, due disappearance of objects

Uevent Generation

struct kset_uevent_ops {
        int (*filter)(struct kset *kset, struct kobject *kobj);
        char *(*name)(struct kset *kset, struct kobject *kobj);
        int (*uevent)(struct kset *kset, struct kobject *kobj, char **envp,
                       struct kobj_uevent_env *env);
};

uevent() results in invocation (as user-space process) of /sbin/hotplug
each kset has a kset_uevent_ops member
filter method determines whether kobject_add and k_object_del generate a hotplug event
name provides the name of the subsystem generating the event, to be passed as the one-and-only argument of /sbin/hotplug
env parameter of uevent() provides environment variable values to be passed to /sbin/hotplug
hotplug events are usually handled at the bus driver level

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

It is interesting to read through the code of kobject_uevent_env to see how this all works.

We will abbreviate the treatment of the folowing topics, due to in-class time limits:

Buses
Devices
Device Drivers
Classes
Uevent processing
Standard device subsystems, classes, and attributes

Some of these will be encountered in examples, as we walk through bits of the Linux source tree on other topics.

Bus Kobjects

struct bus_type {
        const char              *name;
        struct bus_attribute    *bus_attrs;
        struct device_attribute *dev_attrs;
        struct driver_attribute *drv_attrs;
        int (*match)(struct device *dev, struct device_driver *drv);
        int (*uevent)(struct device *dev, struct kobj_uevent_env *env);
        int (*probe)(struct device *dev);
        int (*remove)(struct device *dev);
        void (*shutdown)(struct device *dev);
        int (*suspend)(struct device *dev, pm_message_t state);
        int (*resume)(struct device *dev);
        struct dev_pm_ops *pm;
        struct bus_type_private *p;
};

See example of use in pci-driver.c

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

struct bus_type pci_bus_type = {
        .name           = "pci",
        .match          = pci_bus_match,
        .uevent         = pci_uevent,
        .probe          = pci_device_probe,
        .remove         = pci_device_remove,
        .shutdown       = pci_device_shutdown,
        .dev_attrs      = pci_dev_attrs,
        .bus_attrs      = pci_bus_attrs,
        .pm             = PCI_PM_OPS_PTR,
};

Device, Bus, Class, etc.

struct device {
        struct device           *parent;
        struct device_private   *p;
        struct kobject kobj;
        const char              *init_name; /* initial name of the device */
        struct device_type      *type;
        struct semaphore        sem; /* semaphore to synchronize calls to its driver */
        struct bus_type *bus; /* type of bus device is on */
        struct device_driver *driver; /* which driver has allocated this */
        void            *driver_data;   /* data private to the driver */
        void            *platform_data; /* platform specific data, device
                                           core doesn't touch it */
        struct dev_pm_info      power;
#ifdef CONFIG_NUMA
        int             numa_node;      /* NUMA node this device is close to */
#endif
        u64             *dma_mask;      /* dma mask (if dma'able device) */
        u64             coherent_dma_mask;
        struct device_dma_parameters *dma_parms;
        struct list_head        dma_pools;      /* dma pools (if dma'ble) */
        struct dma_coherent_mem *dma_mem; /* internal for coherent mem override */
        /* arch specific additions */
        struct dev_archdata     archdata;
        dev_t                   devt;   /* dev_t, creates the sysfs "dev" */
        spinlock_t              devres_lock;
        struct list_head        devres_head;
        struct klist_node       knode_class;
        struct class            *class;
        struct attribute_group  **groups;       /* optional groups */
        void    (*release)(struct device *dev)
};

struct device_driver {
        const char              *name;   
        struct bus_type         *bus;
        struct module           *owner;
        const char              *mod_name;      /* used for built-in modules */
        int (*probe) (struct device *dev);
        int (*remove) (struct device *dev);
        void (*shutdown) (struct device *dev);
        int (*suspend) (struct device *dev, pm_message_t state);
        int (*resume) (struct device *dev);
        struct attribute_group **groups;
        struct dev_pm_ops *pm;
        struct driver_private *p
};

struct pci_driver {
        struct list_head node;
        char *name;
        const struct pci_device_id *id_table;   /* must be non-NULL for probe to be called */
        int  (*probe)  (struct pci_dev *dev, const struct pci_device_id *id);   /* New device inserted */
        void (*remove) (struct pci_dev *dev);   /* Device removed (NULL if not a hot-plug capable driver) */
        int  (*suspend) (struct pci_dev *dev, pm_message_t state);      /* Device suspended */
        int  (*suspend_late) (struct pci_dev *dev, pm_message_t state);
        int  (*resume_early) (struct pci_dev *dev);
        int  (*resume) (struct pci_dev *dev);                   /* Device woken up */
        void (*shutdown) (struct pci_dev *dev);
        struct pci_error_handlers *err_handler;
        struct device_driver    driver;
        struct pci_dynids dynids;
};

struct pci_dev {
        struct list_head global_list;   /* node in list of all PCI devices */
        struct pci_bus  *bus;  /* bus this device is on */
        struct pci_bus  *subordinate /* bus this device bridges to */
        void            *sysdata;       /* hook for sys-specific extension */
        struct proc_dir_entry *procent; /* device entry in /proc/bus/pci */
        unsigned int    devfn;          /* encoded device & function index */
        unsigned short  vendor;
        unsigned short  device;
        unsigned short  subsystem_vendor;
        unsigned short  subsystem_device;
        unsigned int    class;          /* 3 bytes: (base,sub,prog-if) */
        u8              revision;       /* PCI revision, low byte of class word */
        u8              hdr_type;       /* PCI header type (`multi' flag masked out) */
        u8              pcie_type;      /* PCI-E device/port type */
        u8              rom_base_reg;   /* which config register controls the ROM */
        u8              pin;            /* which interrupt pin this device uses */
        struct pci_driver *driver; /* which driver has allocated this device */
        u64             dma_mask;       /* mask of the bits of bus address this implements */
        struct device_dma_parameters dma_parms;
        ...
};

struct sysdev_class {
        const char *name;
        struct list_head        drivers;
        /* Default operations for these types of devices */
        int     (*shutdown)(struct sys_device *);
        int     (*suspend)(struct sys_device *, pm_message_t state);
        int     (*resume)(struct sys_device *);
        struct kset kset;
};

struct sys_device {
        u32 id;
        struct sysdev_class   * cls;
        struct kobject          kobj;
};

struct bus_type_private {
        struct kset subsys;
        struct kset *drivers_kset;
        struct kset *devices_kset;
        struct klist klist_devices;
        struct klist klist_drivers;
        struct blocking_notifier_head bus_notifier;
        unsigned int drivers_autoprobe:1;
        struct bus_type *bus;
};

struct driver_private {
        struct kobject kobj;
        struct klist klist_devices;
        struct klist_node knode_bus;
        struct module_kobject *mkobj;
        struct device_driver *driver;
};

Kobject Relationships as of summer 2007

Kobject Relationships as of summer 2008

Kobject Relationships as of summer 2010

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

This changed considerably from 2007 to 2008.

In the 2007 figure the black lines indicate containment of an instance of the lower structure within the upper structure. In the 2008 figure this relationsihp is shown only by nesting.

The dashed blue lines indicate fields of the source object that point to an object of the target class.

The dashed red lines in the 2007 figure indicate fields of the source object that are a kset of the target class.

The diagram is not complete. Left out are many other relationships, including fields that are sets of pointers to another class, fields that are lists of other classes of objects with pointers to another class, etc., etc.

This is an exceedingly (and, to me, probably unnecessarily) complex structure, with apparent redundancies and many apparent opportunities for inconsistencies and errors. It seems to be evolving, but it is unclear whether this evolution is converging and whether it is improving or just reflecting a change in personal styles of whoever currently has most influence over the code base. Specifically, from 2007 to 2008 we see the introduction of several new stylistic elements, including for example:

Splitting some structures into two parts, for example bus_type and bus_type_private
The introduction of klists, for example replacing ksets for the drivers and devices fields of struct bus_type

Are these trends? That is, will other structures be split into two parts, and will the remaining ksets be replaced by klists?

One should probably expect continued evolution in these details with future Linux kernel releases.

How It Plays Together: Adding a PCI Device

The PCI system declares a bus_type object pci_bus_type and calls bus_register from pci_driver_init, which creates /sys/bus/pci.
A specific PCI driver creates a struct pci_driver object, initializes it, and passes it to pci_register_driver
pci_register_driver calls driver_register(indirectly, through __pci_register_driver)
driver_register calls bus_add_driver
bus_add_driver creates a storage object of type driver_private, which includes a kobject, klist, etc. It intializes these structures. Along the way, it calls driver_attach.
driver_attach in turn calls __driver_attach, for each device in the list. (Since the drivers are likely to register after the bus is created, this first pass is not likely to turn up any devices. This operation will be peformed again later, when the device drivers register with the bus, but we might as well review the steps at this point.)
__driver_attach calls driver_probe_device, for the device.
driver_probe_device calls really_probe, which calls the device-specific probe.
really_probe is responsible for creating the sysfs object by calling driver_sysfs_add. Because the probe may do something that assumes this object exists, it creates the object and then destroys it if the probe fails.

See kobject_uevent for more of the story.

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

The above diagram is an attempt to bring the LDD3 diagram up to date. I hope it is accurate. In any case it is likely to change again. If nothing else, this is another example of how volatile the kernel internals are.

You may want to also take a look at pci_scan_bus, which appears to be called from architecture-dependent code, and eventually calls the mechanism described above.

Hotplug

Two views:

Kernel: hardware, kernel, and driver interact
Userspace: kernel and userspace interact, via /sbin/hotplug
called by kernel when it wants to notify user about hotplug event in kernel

When a PCI device is removed

Eventually, system calls the remove function of the driver
In the mean time, the bus returns 0xff (all bits set) values
Driver must allow for this
e.g., see comment /* card removed */ in function handshake in drivers/usb/host/ehci-hcd.c
For USB devices, any pending urbs on the device start failing with -ENODEV

/sbin/hotplug Utility

Usually a small bash script
Called whenever a kobject is created or destroyed
Allows, for example, loading/unloading of kernel modules
Single command-line argument is the name of the event
Other information passed via environment variables

Passes event on to a list of programs, e.g.,

DIR="/etc/hotplug.d"
for I in "${DIR}/$1/"*.hotplug "${DIR}/"default/*.hotplug ; do
  if [ -f $I ]; then
     test -x $I && $I $1 ;
  fi
done
exit 1

See default.hotplug and usb.agent, as well as kobject_uevent for more of the story.

/sbin/hotplug Evironment Variables

ACTION: add (or remove? ... dangerous)
DEVPATH: directory path relative to sysfs root, pointing to the modified kobject
SEQNUM: 64-bit number, incremented for each hotplug event
SUBSYSTEM: string the replicates the command-line argument
Others, specific to type of bus, are set through the uevent() callback for the bus (see slide on Bus Kobjects above)

This slide has additional "handout" notes. You need to use the 'a' key to toggle into handout mode, in order to see them.

Note that the earlier Linux "hotplug" utilities have been subsumed by a more general "userspace event" (see kobject_uevent) mechanism.

Note also that both kobject_uevent is called many places other than at kobject registration, and that call_usermodehelp() is called in more placeg, including for purposes besides uevents.

Hotplug Scripts

Can use /lib/module/KERNEL_VERSION/modules.*map to figure out what modules to load for a given device

udev

Is a daemon process that scans for devices and dynamically creates the corresponding nodes in /dev/
Replaces devfs

See Linux symposium paper for more details.

Loading Firmware (accessing user-space files)

Some devices require firmware to be loaded by OS
(e.g., to save cost of EEPROM on device)
Coding firmware into driver is bad
(Why?)
Should load the firmware from a file
Opening files directly from inside the kernel is bad (why?)
Special firmware interface solves this problem for drivers

Kernel Firmware Interface

#include <linux/firmware.h>
int request_firmware
   (const struct firmware **fw, char *name, struct device *device);

struct firmware {
   size_t size;
   u8 *data;
};

name is name of firmware file
Will block until the operation is done
Result is returned in firmware struct
When done, release the in-kernel firmware structure

void release_firmware(struct firmware *fw);

If you cannot afford to block, use callback version
When data is available, kernel calls handler cont with parameters context and the firmware

int request_firmware_nowait
   (struct module *module,
    char *name,
    struct device *device,
    void *context,
    void (*cont)(const struct firmware *fw, void *(context));

Ch 14 - The Linux Device Model

Linux Grand Unified Device Model

USB Mouse Driver Device Model Entities (LDD3)

Example: USB Mouse Driver Device Model Entities

Kobjects, Ksets, and Subsystems

Example: struct cdev

Kobject initialization

Reference Count Manipulation

struct kobj_type

Kobject Hierarchies

Ksets

Operations on Ksets

Kobjects and Sysfs

Attributes

Non-Default Attributes

Binary Attributes

Symbolic Links

Uevent Generation

Bus Kobjects

Device, Bus, Class, etc.

Kobject Relationships as of summer 2007

Kobject Relationships as of summer 2008

Kobject Relationships as of summer 2010

How It Plays Together: Adding a PCI Device

Hotplug

When a PCI device is removed

/sbin/hotplug Utility

/sbin/hotplug Evironment Variables

Hotplug Scripts

udev

Loading Firmware (accessing user-space files)

Kernel Firmware Interface

Example: `struct cdev`

`struct kobj_type`