OS Security



The study of OS security provides a context for review of the OS. Throughout this course we have pointed out security issues as they arose in the context of various other aspects of the OS. Security issues we have discussed before include:

Computer and Network Security Requirements

The key factor here is authorization. We cannot lock up system in a sealed vault. Authorized users must still be able to use it.

Aspects of Security

Types of Threats

Threats to Computer System Assets

Communication Lines and Networks: Active Attacks

The data communicated might be a telephone conversion, an electronic mail message, or a transferred file. We want to prevent the release and prevent the undetected deletion or modification of the content.

OS Security Fundamentals

When one studies security, it is easy to lose perspective and adopt a "Maginot Line" attitude. One can pay a lot of attention to the design of security mechanisms and policies, but ignore other weaknesses. This is like devoting all one's attention to designing better locks, but ignoring other problems. People who buy expensive locks but forget to lock their doors, or live in houses that have easily broken windows, don't have any better security for the expensive locks. Similarly, one can pay attention to designing better alarm systems, but ignore other weaknesses, like people who turn off the alarm system, or ignore the alarms (maybe because there are too many false alarms). One can design an impregnable safe, lock up all one's valuables in it, and then forget the combination. One can buy security tools, but hire a system administrator who is untrustworthy. etc.

Security Issues for Operating Systems: What can the OS do to help?

Features are easier to provide than trustworthiness, but are useless if the system is not trustworthy.

Trustworthiness starts with reliability.

Buffer Overflow: Examples of Security Exploits

Please see the separate notes on buffer overflow exploits.

One moral of this story is that programming languages used for security-critical software should always prevent or detect array overflows. For example, the Ada language has strict runtime tests for array overflows and undefined/null pointers, which would prevent a buffer overflow from going undetected.

The deeper moral is that system security depends on software quality assurance. Very mundane, but hard to find, software defects can be exploited to break through security.

The Fundamental Problem with Security: Two Kinds of Properties

Which can be guaranteed by adding code to a system?

Which can be verified by just looking at part of the code in a system?

Which form would a security assertion take?

Which form would an insecurity assertion take?

The Problem with Making Real Operating Systems Secure

Why Making Big Improvements in OS Technology is Hard, in General

Protection versus Security

Design Principles for Protection Mechanisms

How do any of the above relate to the phrase ``security through obscurity''?

Degrees of Protection

Protection of Memory

User-Oriented Access Control

Data-Oriented Access Control

Access Matrix

This is useful as an academic abstraction, but nobody actually implements it. Why?

Access Control List

ACL Example

SubjectsAccess Rights
Chang read, write, execute
Yuread, write

ACL Pros & Cons

Shell & API Example: Access Control Lists in SunOS

#include <sys/acl.h>
int  acl(char *pathp, int cmd, int nentries, aclent_t *aclbufp);
int  facl(int fildes, int cmd, int nentries, aclent_t *aclbufp);

The SunOS man-pages say:

The acl() and facl() functions get or set the ACL of a file whose name is given by pathp or referenced by the open file descriptor fildes. The nentries argument specifies how many ACL entries fit into buffer aclbufp. The acl() function is used to manipulate ACL on file system objects.

The values for the parameter cmd are:

Shell commands getfacl, setfacl, aclcheck, and aclsort are also provided. See the Solaris man pages for more detail, if you are interested.

Capability Tickets versus Capability Lists

The textbook is not very clear on this subject. There is a big distinction between a "capability list" (conceptually, a row of an access matrix), and a "capability" (also known as a capability ticket or capablity token).

UNIX Open File Descriptions Viewed as Capability Tickets

How and why?

By now you should know enough about Unix to be able to recognize the ways in which an open file description behaves like a capability token.

  1. Can open file descriptions be propagated? If so how? Can this be prevented? If so, How?
  2. Can open file descriptions be reviewed? If so, how?
  3. Can open file descriptions be revoked? If so, how?
  4. Are open file descriptions garbage-collected? If so, how?

Summary of Unix (Solaris) Protection Mechanisms

We have covered these protection mechanisms in the course.

Windows 2000 Security

Access Token

Security Descriptor

Access Mask

Intruder Types

Intrusion Techniques

Passwords: Techniques for Learning a Password

ID Provides Security

Password Selection Strategies

Intrusion Detection

Profiles of Behavior of Intruders and Authorized Users

Classification of Malicious Programs

Taxonomy of Malicious Software


Logic Bomb

Trojan Horse


Zombie Program

This is not the same thing as a Unix zombie process.


Virus Stages

Types of Viruses

Macro Viruses

E-mail Virus

Antivirus Approaches

Generic Decryption

Trusted Systems

Reference Monitor Concepts

The reference monitor controls every access by subjects to objects. It enforces policies that are recorded in the security kernel database, and maintains an audit file, which records attempts at security violations, and may also be useful in analysis of security failures after the fact.

Trojan Horse Attack

User Bob has a private file, that only he should be able to access. User Alice creates Trojan horse program, which can access a "back pocket" file of Alice's. Alice persuades Bob to use her program.

Trojan Horse Attack

When Bob run's the Trojan horse, the program surreptitiously reads the content of Bob's file and copies it to Alices "back pocket" file.

Trojan Horse Defense

Now suppose Bob has a higher security level than Alice, and they are using a trusted sytem, with a reference monitor.

Trojan Horse Defense

When Alice's Trojan horse, executed by Bob, tries to write to Alice's "back pocket" file, it is detected and prevented, because it would be a "write down" in security level. (Note that this also would prevent Bob from intentionally passing information to Alice.)

© 2002 T. P. Baker & Florida State University. No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means without written permission. (Last updated by $Author: cop4610 $ on $Date: 2002/09/26 20:06:06 $.)