COP4610: Operating Systems & Concurrent Programming up ↑

Buffer Overflows: Why are they a security problem?


These notes are intended to serve two different purposes:

  • Show what can happen when a programmer neglects checking for array bounds violations (buffer overflows). This is an important lesson in programming.
  • Show how a cracker may exploit a typical programmer oversight to subvert an application program or operating system component. This is an important lesson in computer security.
  • Example of Buffer Overflow

    buffer overflowing into variable

    A buffer is a region of memory (an array) that is used to hold a copy of some input or output data. A buffer overflow occurs when a program writes data beyond the bounds of the buffer. For example, if a program attempts to copy a string of length 14 into a buffer of length 10, the last four characters will overflow into adjacent memory.

    The C language is especially vulnerable to array bounds violations because it has many library functions, such as strcpy that rely on null characters or null pointers as "sentinels" to delimit the end of arrays.

    The problem with buffer overflows the contents of other variables, or of critical runtime data structures used by the system, are overwritten. For example, in the figure above, the buffer overflows onto an integer variable that happens to be stored in the adjacent memory.

    Subprogram Activation Records

    activation records

    The system stores the return address for each subprogram call in an activation record, on the runtime stack. Local variables of a subprogram re also stored in the activation record.

    Stack Crash (Overflowing Buffer Overwrites Return Address)

    buffer overflow

    If a buffer is declared as a local variable of a subprogram, a buffer overflow can overwrite the saved return address of the function call. When the instruction to return from the subprogram is executed, it loads the content of the buffer into the program counter, as if it were the return address. At best, this can cause the program to crash.

    If we are unlucky, the content of that part of the buffer might turn out to be the address of a valid instruction, an the program might execute for a while before it crashes. If the execution includes any output instructions, this might cause damage to data on permanent storage.

    Stack Crashing Exploit

    stack crashing exploit

    At worst, an unchecked buffer overflow can be exploited by an unscrupulous person to cause intentional harm, up to cracking the security of an operating system. This techniques, sometimes called "stack crashing", is the most frequently exploited trick for breaking system security, by causing transfer of control to code provided or selected by a cracker.

    Some programming languages, such as Ada, check the bounds of arrays at run time and raise an exception before a buffer overflow can cause adjacent data to be overwritten. However, many application programs and most system software components are written in languages (e.g., C and C++) that do not check array bounds.

    CERT Example

    The vast majority of serious operating system security holes, as reported by CERT, are due to unchecked buffer overflows.

    For example, look at the report of an unchecked buffer overflow in SSL (the Secure Socket Layer).

    The existence of unchecked buffer overflows in SSL shows how pernicious the problem is. SSL is a library upon which many security tools rely. One expects the programmers of SSL thought they were taking special care to write robust code. The fact that they missed one or more buffer overflow problems, and the consequences, should make everyone more careful. It should also make us question the wisdom of continuing to program in languages that make missing such overflows so easy.

    Two Examples of Security Exploits

    The above are two examples of how an unchecked buffer overflow in a program can be exploited to subvert the intended function of a program were pulled from web sites, including They are worthy of study, to understand how to defend against such attacks. Both of these attack the xterm application by taking advantage of an unchecked buffer overflow for the configuration variable xterm*inputMethod that is imported from file.Xdefaults. The value they give for this variable is a string that is designed to overflow the buffer and overlay the return address of the subprogram.

    A Graduated Set of Example Programs

    I have extracted the basic technique used by the first of these two examples, and broken it into a graduated series of simpler examples, which you may view in the directory examples/hacks.

    References for Further Reading

    If you are interested in knowing more about this subject, you can find many articles on buffer overflows, stack crashing, and security exploits on the Web. The first four links above are to tutorial articles on exploitation of buffer overflows that I found at just one site on the web.

    The other links are to some Web sites that provide additional information about computer security and techniques used to attack operating systems, including exploitation of buffer overflows.

    More information on this subject continues to be published. Just search.

    T. P. Baker. ($Id)