Topics

• managing ioctl command numbers
• blocking/unblocking a process
• seeking on a device
• access control
• spinlocks

ioctl

• generic: nearly all details are device-specific
• application view:

  int ioctl (int fd, unsigned long cmd, ...);

• driver view:

  int (*ioctl) (struct inode *inode, struct file *filp,
        unsigned int cmd, unsigned long arg);

• cmd - an integer specifying what to do
• arg - an integer argument value or a pointer to an argument structure
• return value - if negative indicates an error
  

The text says POSIX specifies -NOTTTY for an invalid ioctl command, though returning -EINVAL is still common in practice. I believe the book is wrong on that. If you look at the on-line Single Unix Specification (which is a superset of the POSIX spec.) you will see that different errno values are specified for different errors, and EINVAL is specified for a request argument that is not valid for the device. I'm sorry to tsay that this has been pretty typical of the Linux development community, that they don't pay very careful attention to the Unix standards, though that may be improving.

Managing ioctl Command Numbers

• unique across drivers, for safety
• 32 bit value, viewed as four parts
  • type - chosen unique to the driver (_IOC_TYPEBITS wide)
    (see include/asm/ioctl.h and Documentation/ ioctl-number.txt)
  • number - ordinal assigned to functions by the driver (_IOC_NRBITS wide)
  • direction - _IOC_NONE, _IOC_READ, _IOC_WRITE, or _IOC_READ | _IOC_WRITE
  • size - size of the user data, in bytes (_IOC_SIZEBITS long)
• standard macros are provide to manipulate these fields (see examples below)

The book uses the term "magic number" for the first field of the ioctl command. Don't be confused later when you see that term used for other things, such a number at the beginning of a file.

Example of ioctl Command Set

/* Use 'k' as magic number */
#define SCULL_IOC_MAGIC  'k'
/* Please use a different 8-bit number in your code */
#define SCULL_IOCRESET    _IO(SCULL_IOC_MAGIC, 0)
/*
 * S means "Set" through a ptr,
 * T means "Tell" directly with the argument value
 * G means "Get": reply by setting through a pointer
 * Q means "Query": response is on the return value
 * X means "eXchange": switch G and S atomically
 * H means "sHift": switch T and Q atomically
 */
#define SCULL_IOCSQUANTUM _IOW(SCULL_IOC_MAGIC,  1, int)
#define SCULL_IOCSQSET    _IOW(SCULL_IOC_MAGIC,  2, int)
#define SCULL_IOCTQUANTUM _IO(SCULL_IOC_MAGIC,   3)
#define SCULL_IOCTQSET    _IO(SCULL_IOC_MAGIC,   4)
#define SCULL_IOCGQUANTUM _IOR(SCULL_IOC_MAGIC,  5, int)
#define SCULL_IOCGQSET    _IOR(SCULL_IOC_MAGIC,  6, int)
#define SCULL_IOCQQUANTUM _IO(SCULL_IOC_MAGIC,   7)
#define SCULL_IOCQQSET    _IO(SCULL_IOC_MAGIC,   8)
#define SCULL_IOCXQUANTUM _IOWR(SCULL_IOC_MAGIC, 9, int)
#define SCULL_IOCXQSET    _IOWR(SCULL_IOC_MAGIC,10, int)
#define SCULL_IOCHQUANTUM _IO(SCULL_IOC_MAGIC,  11)
#define SCULL_IOCHQSET    _IO(SCULL_IOC_MAGIC,  12)
/*
 * The other entities only have "Tell" and "Query", because they're
 * not printed in the book, and there's no need to have all six.
 * (The previous stuff was only there to show different ways to do it.
 */
#define SCULL_P_IOCTSIZE _IO(SCULL_IOC_MAGIC,   13)
#define SCULL_P_IOCQSIZE _IO(SCULL_IOC_MAGIC,   14)
/* ... more to come */

#define SCULL_IOC_MAXNR 14

The above are from scull.h.

See include/asm/ioctl.h for definitions of macros _IO(), _IOC_SIZE(), etc.

Avoid Using Predefined Commands in a Driver

• These are filtered out and handled by kernel, so your driver's ioctcl method won't be called
• FIOCLEX/FIONCLEX - close-on-exec/no-close-on-exec flags
• FIOASYNC - not used anymore
• FIOQSIZE - returns size of file or directory, but ENOTTY for a device
• FIONBO - set or clear the O_NONBLOCK flag

The function fcntl is much like ioctl in usage. Some functions overlap. Historically, ioctl was for devices and fcntl was for files, but this distinction has blurred over time. If you look at the Single Unix Spec reference above, you will see that ioctl is actually described as an operation on streams.

Checking Validity of User-Space Addresses

• We have already seen copy_from_user and copy_to_user
• put_user (from_datum, to_ptr) and get_user (to_local, from_ptr) are similar
  but optimized for transfer of one, two, four, or eight bytes
• access_ok(VERIFY_READ/VERIFY_WRITE, user_ptr, size) checks user pointer validity
• __put_user (from_datum, to_ptr) and __get_user (to_local, from_ptr) do not do the complete validity checking, and so are for use after a call to access_ok

Example of Command and Argument Validation

int scull_ioctl(struct inode *inode, struct file *filp,
                 unsigned int cmd, unsigned long arg)
{  int err = 0, tmp;
   int retval = 0;
    /*
    * extract the type and number bitfields, and don't decode
    * wrong cmds: return ENOTTY (inappropriate ioctl) before access_ok()
    */
   if (_IOC_TYPE(cmd) != SCULL_IOC_MAGIC) return -ENOTTY;
   if (_IOC_NR(cmd) > SCULL_IOC_MAXNR) return -ENOTTY;
   /*
    * the direction is a bitmask, and VERIFY_WRITE catches R/W
    * transfers. `Type' is user-oriented, while
    * access_ok is kernel-oriented, so the concept of "read" and
    * "write" is reversed
    */
   if (_IOC_DIR(cmd) & _IOC_READ)
      err = !access_ok(VERIFY_WRITE, (void __user *)arg, _IOC_SIZE(cmd));
   else if (_IOC_DIR(cmd) & _IOC_WRITE)
      err =  !access_ok(VERIFY_READ, (void __user *)arg, _IOC_SIZE(cmd));
   if (err) return -EFAULT;
   switch(cmd) { ...

The above is from file scull/main.c.

Checking Capabilities

• Used to manage permission to perform certain operations
  at a finer grain than just root/non-root

case SCULL_IOCSQUANTUM: /* Set: arg points to the value */
  if (! capable (CAP_SYS_ADMIN)) return -EPERM;
  ret = __get_user(scull_quantum, (int __user *)arg);
  break;

• e.g.
  • CAP_DAC_OVERRIDE - override directory and file access permissions
  • CAP_SYS_ADMIN - system administration operations
  • CAP_SYS_RAWIO - perform raw I/O operations
• full list in linux/capability.h

The above is from file scull/main.c.

You should already be familiar with the concept of "capability" in the general sense in which the term is used for operating systems. This is a special case, for the Linux kernel.

From my own reading of these capabilities, I'm skeptical about how far they go toward the general goal of capabilities as originally defined by OS security researchers. In particular, it seems to me that it would not be hard to use some of these capabilities to achieve (indirectly) the effect of others, and that several of them could pretty easily be used to escalate to a "superuser" shell, so long as the system has a superuser (one with all capabilities).

Implementation of ioctl Commands

 switch(cmd) {
#ifdef SCULL_DEBUG
   case SCULL_IOCRESET:
     scull_quantum = SCULL_QUANTUM;
     scull_qset = SCULL_QSET;
     break;
   case SCULL_IOCSQUANTUM: /* Set: arg points to the value */
     if (! capable (CAP_SYS_ADMIN))
 	return -EPERM;
     retval = __get_user(scull_quantum, (int __user *)arg);
     break;
   ...
}

What is the effect of resetting scull_quantum on an existing device? (Read the code to find out.)

Blocking I/O

• There are several kernel functions that can be used to block the current process, and wake up another process
• Some of these are safer and more reliable than others
• All are based on the wait_queue_head_t structure
• A process waits on a queue, and another process signals all (or a subset) of the processes waiting on the queue to wake up
• The processes that wake up must check to see whether they can now do what they were waiting for, or need to go back to sleep.

For an example of the right way to block and unblock a client process, look at the implementation of the scull pipe device:

• declaration of wait queue heads.
• initialization of wait queue heads in scull pipe initializer.
• use of wait_event_interruptible in scull pipe read method.
• use of wake_up_interruptible in scull pipe write method.

The example above uses wait_event. We will not talk much about the operations with the word "sleep" in their name, because they are very prone to misuse. However, you may see them used in older kernel code.

Warnings for Use of Sleep/Wakeup

• the sleep forms are very prone to missed wakeups
• being woken up does not mean the event you are waiting for has occurred
• generally, you need a loop around the wait call, to check the wakeup condition and put the process back to sleep if the event has not occurred
• try to avoid having multiple processes sleep on the same queue,
  or end up with a "thundering herd" of processes every time you issue a wakeup
• think hard before allowing a process to sleep/wait while it is holding a semaphore
  (never sleep/wait while holding a spinlock)

Beware that most situations in which a process goes to sleep involve a potential race condition between the wait and the wakeup. That is why should should probably always be using the event versions.

The correct style of using the event wait and wakeup operations is somewhat like the Pthread condition variable operations, in needing a loop around the wait call to check for the wakeup condition.

Take a look at the implementation of wait_even_interruptible to see that there is a loop to check the condition, but that the check is subject to races.

That is, while the check is made there is mutual exclusion, i.e., the wait operation does not unlock and relock any lock. If the check on the wakeup condition requires mutual exclusion (as usual), you need to do the semaphore operations explicitly in an outer loop, as in the scull/pipe.c example.

If you did not see this the first time, go back and look at the loop around the use of wait_event_interruptible in the scull pipe read method.

Wait Queue Implementation

Processes are linked through temporary pointers stored near the top of the stack of a blocked process. See the definition of the macro DEFINE_WAIT, and how it is used in __wait_event_interruptible. This relies on the invariant that a process cannot be on a wait queue while it is running.

Avoiding the Thundering Herd

1. one process per queue
2. do-it-yourself wait queue manipulations
3. use of add_wait_queue_exclusive

Reentrant Code

• always good
• absolutely necessary whenever you call (directly or indirectly) a potentially blocking function, including:
  • calls to schedule
  • functions like get_user() that access user-space data
• avoids keeping status information in (static) global variables
• use local variables for small items
• use a local pointer & kmalloc for larger items

"Using local variables for large items is not good practice, because the data may not fit the single page of memory allocated for the stack space"

Blocking vs. Nonblocking Option

• normal semantics of some operations (read, write) requires blocking under certain circumstances
  • read - block until at least some data is available
  • write - block until there is space for at least some data in the output buffer
  • open - block until device can be opened (e.g., until file is unlocked
• O_NONBLOCK in filp->f_flags specifies operations should fail in situations where they would otherwise block
• For example, see treatment of nonblocking option in the scull pipe read operation

Asynchronous I/O: poll and select

• these operations require a process to block on more than one file/device
• so we can't use a simple per-device wait queue

poll Data Structures

Example

static unsigned int scull_p_poll(struct file *filp, poll_table *wait)
{
	struct scull_pipe *dev = filp->private_data;
	unsigned int mask = 0;

	/*
	 * The buffer is circular; it is considered full
	 * if "wp" is right behind "rp" and empty if the
	 * two are equal.
	 */
	down(&dev->sem);
	poll_wait(filp, &dev->inq,  wait);
	poll_wait(filp, &dev->outq, wait);
	if (dev->rp != dev->wp)
		mask |= POLLIN | POLLRDNORM;	/* readable */
	if (spacefree(dev))
		mask |= POLLOUT | POLLWRNORM;	/* writable */
	up(&dev->sem);
	return mask;
}

The above is from scull/pipe.c.

Normally, the poll method should return POLLHUP if not more data is (or may become) available. The example above is an exception to this rule.

Poll Codes that the Driver May Return

• POLLIN - device can be read without blocking
• POLLRDNORM - normal data is available for reading
• POLLRDBAND - out-of-band data is available for reading (rarely used)
• POLLPRI - high-priority out-of-band data can be read without blocking
  (reported by select as an exception
• POLLHUP - a read will see end-of-file
• POLLERR - an error condition occurredon the device
• POLLOUT - device can be written to without blocking
• POLLWRNORM - same as POLLOUT
• POLLWRBAND - data with nonzero priority can be written to the device (only used for datagrams)

Asynchronous Notification (via signals)

• Application view of enabling asynchronous notification

  signal (SIGIO, &input_handler); /* enable generation of SIGIO */
  fcntl (STDIN_FILENO, F_SETOWN, getpid()); /* driver sets f_owner */
  oflags = fcntl (STDIN_FILENO, F_GETFL);
  fcntl (STDIN_FILENO, F_SETFL, oflags | FASYNC);  /* calls driver fasync method */

• Most of the work is done by functions provided by the kernel,
  some called by the driver.
• See usage examples in pipe.c:
  • fasync_struct pointer in struct for scull pipe
  • fasync method for scull pipe
  • call to kill_fasync to awake reader in write method on scull pipe
  • call to scull_p_fasync in release method

The signal API is simpler, but "deprecated". The sigaction API is more general and more portable.

See also file asynctest.c.

Seeking on a Device

• kernel does non-device-specific default implementation
  just modifies filp->f_pos
• driver must do device-specific seeks
  e.g. repositioning tape
• driver must also do seeks from end, if supported
• driver must override default seek, of seeking on the device does not make sense

See example in scull pipe implementation.

Spin Locks

• do not put the process to sleep
• loop until another processor releases the lock
• should only be used for short critical sections
• should always be released before the process goes to sleep
• must be initialized explicitly, before use