Processes and Daemons

• Fundamentally, kernels provide a few logical constructs that mediate access to either real or virtual resources. The two most important in Unix are processes and filesystems.
• You can view the characteristics of processes on a Unix machine with a variety of programs, including ps, top, lsof, and ls.

What Unix/Linux system administrators see — ps

$ cat /etc/lsb-release
DISTRIB_ID=LinuxMint
DISTRIB_RELEASE=17.1
DISTRIB_CODENAME=rebecca
DISTRIB_DESCRIPTION="Linux Mint 17.1 Rebecca"
$ ps -elf
F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root         1     0  0  80   0 -  8476 poll_s 10:12 ?        00:00:01 /sbin/init
1 S root         2     0  0  80   0 -     0 kthrea 10:12 ?        00:00:00 [kthreadd]
1 S langley   2559     1  0  69 -11 - 127788 poll_s 10:12 ?       00:01:06 /usr/bin/pulseaudio --start --log-target=syslog
4 S rtkit     2561     1  0  81   1 - 42228 poll_s 10:12 ?        00:00:00 /usr/lib/rtkit/rtkit-daemon
4 S root      2570     1  0  80   0 - 59838 poll_s 10:12 ?        00:00:00 /usr/lib/upower/upowerd

What Unix/Linux system administrators see -- top

[root@localhost root]# top -b -n1   # run in batch mode for one iteration
 08:17:41  up 1 day, 18:12,  2 users,  load average: 9.69, 9.14, 8.89
115 processes: 114 sleeping, 1 running, 0 zombie, 0 stopped
CPU states:  cpu    user    nice  system    irq  softirq  iowait    idle
           total    0.0%    0.0%    0.9%   0.0%     0.9%    0.0%   98.0%
Mem:   510344k av,  392504k used,  117840k free,       0k shrd,   17208k buff
                    240368k actv,   55488k in_d,    4760k in_c
Swap:  522104k av,   90392k used,  431712k free                   72852k cached

  PID USER     PRI  NI  SIZE  RSS SHARE STAT %CPU %MEM   TIME CPU COMMAND
 1090 root      20   0  1088 1088   832 R     0.9  0.2   0:00   0 top
    1 root      15   0   492  456   432 S     0.0  0.0   0:08   0 init
    3 root      15   0     0    0     0 SW    0.0  0.0   0:00   0 keventd

What Unix/Linux system administrators see - lsof

[root@localhost root]# lsof       # heavily redacted to fit on page
COMMAND    PID    USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
ssh       3075 langley    3u  IPv4  20919      0t0  TCP langley.cs.fsu.edu:48558->www.cs.fsu.edu:ssh (ESTABLISHED)
firefox   3169 langley   63u  IPv4  22194      0t0  TCP langley.cs.fsu.edu:41363->mia07s25-in-f14.1e100.net:https (ESTABLISHED)

Processes and Daemons : fork and clone

• One of the most important logical resources in Unix is that of processes.
• A new process is created by fork; or, alternatively, in Linux with clone since processes and threads are both just task_struct in Linux.
• With clone, memory, file descriptors and signal handlers are still shared between parent and child.
• With fork, these are copied, not shared.

Starting a Unix/Linux process

• exec instantiates a new executable:
• Usually, when doing an exec the named file is loaded into the current process's memory space
• If the executable is dynamically linked, then the dynamic loader maps in the necessary bits (not done if the binary is statically linked.)
• Then code in the initial .text section is then executed. (There are three main types of sections:
  • .text sections for executable code
  • .data sections (including read-only .rodata sections)
  • .bss sections (Blocks Started by Symbol) which contains ``uninitialized'' data)
• SCRIPTS: However if the first two characters of the file are #! and the following characters name a valid pathname to an executable file, in which case that executable is instead loaded into memory. This is done for "scripts", such as bash, Perl, and Python scripts.

Some Typical Assembly Code

	.file	"forty_two.c"
	.text
	.globl	main
	.align	16, 0x90
	.type	main,@function
main:                                   # @main
	.cfi_startproc
# BB#0:
	pushq	%rbp
.Ltmp2:
	.cfi_def_cfa_offset 16
.Ltmp3:
	.cfi_offset %rbp, -16
	movq	%rsp, %rbp
.Ltmp4:
	.cfi_def_cfa_register %rbp
	movl	$42, %eax
	movl	$0, -4(%rbp)
	popq	%rbp
	ret
.Ltmp5:
	.size	main, .Ltmp5-main
	.cfi_endproc


	.ident	"Ubuntu clang version 3.4-1ubuntu3 (tags/RELEASE_34/final) (based on LLVM 3.4)"
	.section	".note.GNU-stack","",@progbits 
        ; this means this code does not require an executable stack

Daemon processes

When we refer to a daemon process, we are referring to a process with these characteristics:

• Generally persistent (though it may spawn temporary helper processes like xinetd does)
• No controlling terminal (and the controlling tty process group tpgid) is shown as -1 in ps)
• Parent process is generally init (process 1)
• Generally has its own process group id and session id;

Daemon processes

Generally a daemon provides a service. So why not put such services in the kernel?

• Another level of modularity that is easy to control
• Let's keep from growing the already largish kernel
• Ease (and safety) of killing and restarting processes
• Logically, daemons generally share the characteristics one expects of ordinary user processes (except for the lack of controlling terminal.)

Kernel and user daemons: init

TRADITIONAL: init (pid 1) daemon: The first ``user'' process started by the kernel; its userid is 0. All other ``normal'' processes are descendants of init. Depending on the boot parameters init, you might see something along these lines:

• Spawn a single-user shell at the console.
• Begin the multi-user start-up scripts (which are, unfortunately, not standardized across UNIXes; see section pp. 32-41 and pp. 886-887 in LAH)
• Perhaps start up daemontools "svscan" (probably indicated something like SV:123456:respawn:/command/svscanboot in /etc/inittab
• Or start systemd, initng, upstart, or !?!?

There is a lot of flux in this area; Redhat and Debian have both moved to systemd; hopefully, whatever the engine, we can get better dependency resolution than we have had previously and faster boot times. Gentoo (which uses OpenRC) has a nice comparison and contrast page here.

While systemd can support old AT&T scripts, it is designed to instead to have any startup parameters actually processed by systemd rather than the execution of a standalone script.

Using systemd's "systemctl" to control services.

• systemctl start SERVICE → start service
• systemctl stop SERVICE → stop service
• systemctl restart SERVICE → restart a service
• systemctl enable SERVICE → add a service to the boot
• systemctl disable SERVICE → remove a service to the boot

See Redhat for more on systemd.

Treating a file system gently

• It's best not to just turn off a UNIX machine without flushing the buffer cache. It is better to halt the system using /etc/shutdown, /etc/halt, or poweroff; these commands attempt to put the system in a quiescent state (including calling sync()).
• I like to do something like sync ; sync ; poweroff or sync ; sync ; reboot just to make sure a few manual synchronizations are made. When I am removing a USB drive, I like to do something like sync ; umount /media/disk ; sync .
• The update daemon goes by many names (also see pdflush, bdflush(2), and bdflush(8) in Linux and fsflush in Solaris).

Comments in the code: what Linux kernel comments say about dirty buffers and pages

/*
 * The relationship between dirty buffers and dirty pages:
 *
 * Whenever a page has any dirty buffers, the page's dirty bit is set, and
 * the page is tagged dirty in its radix tree.
 *
 * At all times, the dirtiness of the buffers represents the dirtiness of
 * subsections of the page.  If the page has buffers, the page dirty bit is
 * merely a hint about the true dirty state.
 *
 * When a page is set dirty in its entirety, all its buffers are marked dirty
 * (if the page has buffers).
 *
 * When a buffer is marked dirty, its page is dirtied, but the page's other
 * buffers are not.
 *
 * Also.  When blockdev buffers are explicitly read with bread(), they
 * individually become uptodate.  But their backing page remains not
 * uptodate - even if all of its buffers are uptodate.  A subsequent
 * block_read_full_page() against that page will discover all the uptodate
 * buffers, will set the page uptodate and will perform no I/O.
 */

(from fs/buffer.c in kernel 3.9.4)

Kernel and user daemons: inetd and xinetd

• Even though well-written daemons consume little CPU time they do take up virtual memory and process table entries.
• Years ago, as people created new services, the idea of a super-daemon inetd was created to manage the class of network daemons.
• Many network servers were mediated by the inetd daemon at connect time, though some, such as sendmail, postfix, qmail, and sshd were not typically under inetd.
• The original inetd listened for requests for connections on behalf of the various network services and then started the appropriate daemon, handing off the network connection pointers to the daemon.
• Some examples are pserver, rlogin, telnet, ftp, talk, and finger.
• The configuration file that told inetd which servers to manage was /etc/inetd.conf.

Amusingly enough, this very same line of reasoning is being revived by systemd; see this blog posting by its author. (note that daemontools also has used a related idea since 2001, but more for monitoring purposes.)

Kernel and user daemons: inetd and xinetd

• The /etc/services file: This file maps TCP and UDP protocol server names to port numbers.
• The /etc/inetd.conf file This file has the following format (see page 887-893 in LAH and ``man inetd.conf''):
• The successor to inetd was xinetd, which combined standard inetd functions with other useful features, such as logging and access control.

Kernel and user daemons: inetd and xinetd

The configuration file structure for xinetd is /etc/xinetd.conf and also /etc/xinetd.d/*. These files are used to modify general behavior of the daemon and the directory /etc/xinetd.d contains separate files per service. Almost all distros now use xinetd instead of inetd.

Kernel and user daemons: inetd and xinetd

When installing new software packages you may have to modify /etc/inetd.conf, /etc/xinetd.d/ files, and/or /etc/services. A hangup signal (kill -HUP SOMEPID) will get the inetd/xinetd to re-read its config file. Or you might be able to use a startup script, such as ``/etc/init.d/inetd restart'') or ``service inetd restart''.

Kernel and user daemons: portmap and rpcbind

portmap/rpcbind : portmap (rpcbind on OpenSolaris and BSD) maps Sun Remote Procedure Call (RPC) services to ports (/etc/rpc). Typically, /etc/rpc looks something like:

[root@vm5 etc]# more /etc/rpc
#ident  ``@(#)rpc        1.11    95/07/14 SMI''   /* SVr4.0 
#
#       rpc
#
portmapper      100000  portmap sunrpc rpcbind
rstatd          100001  rstat rup perfmeter rstat_svc
rusersd         100002  rusers
nfs             100003  nfsprog
ypserv          100004  ypprog
mountd          100005  mount showmount
ypbind          100007
walld           100008  rwall shutdown
yppasswdd       100009  yppasswd

Kernel and user daemons: portmap/rpcbind

• Sun RPC is used by other services, such as NFS and NIS. Theoretically, RPC servers should register with this daemon and RPC clients would get the port number for a service from the daemon. If this is happening, you can find such information using rpcinfo, such as rpcinfo -p.
• Also, there were some subtle points that had oddly creeped in from the old tcpwrappers package that can affect the portmapper. See for example /etc/hosts.deny; however, most distros have moved away from this code.

Kernel and user daemons: syslogd

(r)syslogd : (r)syslogd is a daemon whose function is to handle logging requests from

• the kernel
• other user processes, primarily daemon processes
• processes on other machines, since syslogd can listen for logging requests across a network

Note that the older syslog has generally being replaced with rsyslog.

Kernel and user daemons: syslogd

A process can make a logging request to the syslogd by using the function syslog(3). syslogd determines what to do with logging requests according to the configuration file /etc/syslog.conf

/etc/syslog.conf generally looks something like:

*.info;mail.none;news.none;authpriv.none;cron.none  /var/log/messages
authpriv.*                                          /var/log/secure
mail.*                                              /var/log/maillog
cron.*                                              /var/log/cron
*.emerg                                             *
local7.*                                            /var/log/boot.log

Kernel and user daemons: syslogd

• For a standalone UNIX machine, the default /etc/[r]syslog.conf will suffice.
• If you are going to manage a number of UNIX machines, consider modifying /etc/[r]syslog.conf on the machines so all the syslog messages are routed to a single ``LOGHOST''.
• You can also use the "perf" kernel monitoring tools to look at system activity. For instance, "perf top" or "perf state CMD" can occasionally yield interesting data for a system administrator.