Buffer Overflows: Why are they a security problem?

These notes are intended to serve two different purposes:

Example of Buffer Overflow

A buffer is a region of memory (an array) that is used to hold a copy of some input or output data. A buffer overflow occurs when a program writes data beyond the bounds of the buffer. For example, if a program attempts to copy a string of length 14 into a buffer of length 10, the last four characters will overflow into adjacent memory.

The C language is especially vulnerable to array bounds violations because it has many library functions, such as strcpy that rely on null characters or null pointers as "sentinels" to delimit the end of arrays.

The problem with buffer overflows the contents of other variables, or of critical runtime data structures used by the system, are overwritten. For example, in the figure above, the buffer overflows onto an integer variable that happens to be stored in the adjacent memory.

Subprogram Activation Records

The system stores the return address for each subprogram call in an activation record, on the runtime stack. Local variables of a subprogram re also stored in the activation record.

Stack Crash (Overflowing Buffer Overwrites Return Address)

If a buffer is declared as a local variable of a subprogram, a buffer overflow can overwrite the saved return address of the function call. When the instruction to return from the subprogram is executed, it loads the content of the buffer into the program counter, as if it were the return address. At best, this can cause the program to crash.

If we are unlucky, the content of that part of the buffer might turn out to be the address of a valid instruction, an the program might execute for a while before it crashes. If the execution includes any output instructions, this might cause damage to data on permanent storage.

Stack Crashing Exploit

At worst, an unchecked buffer overflow can be exploited by an unscrupulous person to cause intentional harm, up to cracking the security of an operating system. This techniques, sometimes called "stack crashing", is the most frequently exploited trick for breaking system security, by causing transfer of control to code provided or selected by a cracker.

Some programming languages, such as Ada, check the bounds of arrays at run time and raise an exception before a buffer overflow can cause adjacent data to be overwritten. However, many application programs and most system software components are written in languages (e.g., C and C++) that do not check array bounds.

CERT Example

The vast majority of serious operating system security holes, as reported by CERT, are due to unchecked buffer overflows.

For example, look at the report of an unchecked buffer overflow in SSL (the Secure Socket Layer).

The existence of unchecked buffer overflows in SSL shows how pernicious the problem is. SSL is a library upon which many security tools rely. One expects the programmers of SSL thought they were taking special care to write robust code. The fact that they missed one or more buffer overflow problems, and the consequences, should make everyone more careful. It should also make us question the wisdom of continuing to program in languages that make missing such overflows so easy.

Two Examples of Security Exploits

• exploiting buffer overflow with an executable stack

• exploiting buffer overflow with system calls

The above are two examples of how an unchecked buffer overflow in a program can be exploited to subvert the intended function of a program were pulled from web sites, including www.insecure.org. They are worthy of study, to understand how to defend against such attacks. Both of these attack the xterm application by taking advantage of an unchecked buffer overflow for the configuration variable xterm*inputMethod that is imported from file.Xdefaults. The value they give for this variable is a string that is designed to overflow the buffer and overlay the return address of the subprogram.

• exploiting buffer overflow with an executable stack: This example causes the return to transfer control to some code that is embedded in the same string that overwrites the return address. A defense against this attack is to set the memory protection on the stack to prevent execution of code from the stack, i.e., to make the stack readable and writeable, but not executable. With such protection in place, an attempt to use the return address to transfer control to the stack will be trapped by the OS, and the process will be killed.
• exploiting buffer overflow with system calls: This example shows how to work around stack protection, by inducing the program to "return" to a different location in its own code, where it would normally execute an exec() system call. One can trick the program into doing the exec() on a different program, by overwriting the registers and stack locations that hold the system call parameters. This demonstrates that just protecting the stack is not enough. One must write code that never allows buffer overflows.

A Graduated Set of Example Programs

I have extracted the basic technique used by the first of these two examples, and broken it into a graduated series of simpler examples, which you may view in the directory examples/hacks.

References for Further Reading

• Writing buffer overflow exploits - a tutorial for beginners
• Smashing The Stack For Fun And Profit
• w00w00 on Heap Overflows
• Advanced buffer overflow exploit
• http://www.insecure.org
• http://www.security-focus.com
• www.phrack.org, the Phrack journal

If you are interested in knowing more about this subject, you can find many articles on buffer overflows, stack crashing, and security exploits on the Web. The first four links above are to tutorial articles on exploitation of buffer overflows that I found at just one site on the web.

The other links are to some Web sites that provide additional information about computer security and techniques used to attack operating systems, including exploitation of buffer overflows.

More information on this subject continues to be published. Just search.