Copyright R.A. van Engelen, FSU Department of Computer Science, 2000


A Secure Linux OS

  • Goal: The monitoring of sensitive tasks in an OS to verify their normal behavior
  • Abnormal task behavior is immediately detected by the OS, upon which
    • The task can be terminated
    • Sandboxing: critical OS resources can be protected
    • Intrusion detection: find and prevent hacker attack
  • Development of prototype Linux implementation
  • Team members:
    • Damon Snyder, Intel Corp.
    • Wei Yu
    • Prof. van Engelen
    • Prof. Gallivan



  • We use Linux because it is open source
  • System deployment mode:
    • The Linux system calls made by a task are matched against expected normal behavior of the task
    • The matching is done by system-call wrapper routines added to Linux
    • The patterns of normal behavior are stored in a finite state machine in the Linux kernel
  • System training mode:
    • A task is run for some time in a secure environment and its system call traces are stored
    • A finite state machine is constructed from the traces that models normal task behavior



  • Suppose a sensitive task such as the sendmail deamon has the following trace of normal system calls
  • A B C D A D D C D ...
  • A sliding window of size four breaks this down into manageble chunks and we get
  • A B C D
    B C D A
    C D A D
    D A D D
    A D D C
    D D C D
    D C D ...
    C D ... ...
    D ... ... ...
  • These patterns form the finite state machine of normal behavior
    • E.g. after system call A, either BCD or DDC should follow as the next to be expected system calls
    • If anything else follows A, the task must be malbehaving



  • Prototype implementation finished
  • Detection and prevention of e.g. sendmail attacks
  • False alarms?
  • Best window size?
  • Finite state machine size?
  • Investigate resource management options (sandboxing)