1. Learn to write a simulation program, such as is useful for evaluating Operating System designs.
2. Deepen your understanding of file system implementations, and the overhead of system calls.
3. Improve your skill as a C programmer and get exposure to a typical "real world" problem definition, where some aspects of the design are left up to you.
4. (optionally) Get experience working in a group with other students. By choosing to work in a group you will be agreeing that all members of the group will receive the SAME GRADE for P5. GROUPS WILL BE EXPECTED TO DO BENCHMARKS ON A WIDER VARIETY OF MACHINES AND OPERATING SYSTEMS OR PROVIDE MORE EXTENSIVE METRICS, RUNS AND ANALYSES!! Groups can be anywhere from two to three people in size.
1. An electronic copy of your program, in the form of a single compilable file, fsbench.c, emailed to cop4610@cs.fsu.edu, by midnight on Friday December 5th, 1997. (the last day of classes).
2. A report, approximately 5 pages long (add 2 more pages per team member for teams with two or more people) and also turned in by the due date above, explaining:
a. how you resolved those design details that are left up to you by the assignment,
b. the data obtained from your experiments, complete with graphs and/or charts showing the relative performance of the different system calls.
3. You only have to turn in one program and one report per team (or individual).
This project will be graded on a scale of 0-100.
You are a systems analyst at a major government lab. Through some mistake of Congress the funding for your lab actually went up this year and you have the enviable task of spending $2 million on new equipment! Since your lab generates a lot of data you want to buy machines that perform I/O especially well. Unfortunately, you just don't believe all the marketing hype from the vendors and the available benchmark suites just seem to be lacking in good I/O tests.
Being a well-trained analyst and a firm believer in the "Not Invented Here" syndrome, you decide to create a system call execiser with the name fsbench, which stands for File System Benchmark. You will write fsbench to be portable enough to run on practically any UNIX environment so porting from one machine to another will be a snap.
You have been given the input specifications for fsbench (details follow). You have to design suitable output metrics that allow measurements to be taken and then generate a series of fsbench runs on different platforms. The final result will be a report that tells you which machines perform which I/O-based system calls the best.
The benchmark program will accept an input file of system call strings. Your program will read the line of input and perform the system call, then read the next line and perform that call until the input file has been exhausted. Only the system calls specified below should be benchmarked, as they are the most common file system and I/O system calls :
Syntax of syscall script file
Each line is preceded by a number, "x", that indicates how many times the call should be made.
x open filename Open/Create "filename" x read n Read *n* bytes from currently-opened file x write n Write *n* bytes into c.o. file x seek l Seek to location *l* in c.o. file x flush Flush pending writes to disk of c.o. file x close Close currently-opened file # comment x fopen filename Open/Create "filename" x fread n Read *n* bytes from c.o. file x fwrite n Write *n* bytes into c.o. file x fseek l Seek to location *l* in c.o. file x fflush Flush pending output to disk of c.o. file x fclose Close currently-opened file
A sample script file. This will open a file named "bigfile" if it already exists, or create it if it doesn't. Then, it writes 25 blocks into the file of random data, each block being 1024 bytes in size. Next, it seeks to position 0 in the file and reads 50 512-byte blocks from the file (notice that the blocks are just read and nothing is done with them). Finally, the file is closed.
1 open bigfile 25 write 1024 1 seek 0 50 read 512 1 close
The Q/A team in your company (Quality Assurance) demands that all research tools developed by your group follow a common calling convention, so that other research teams can just ``plug and play'' your benchmark program.
Thus, your fsbench should accept a number of command line arguments that control the parameterization of the benchmark run. You can use the getopt() library function to parse these arguments or just parse the command line yourself. The parameters are :
fsbench -n repeat_count
where :
-n repeat_count repeat_count = number of iterations per system call.
If not specified, assume 1.
An example fsbench call would be :
% fsbench -n 1 < script_file
The result of the fsbench run will be messages written to standard output with timing metric information (see below for more details on metrics).
The basic design of the benchmark program is as follows :
o Parse the command line for the optional parameter.
o Begin reading standard input for the script file
commands and examine each line :
- Read the first integer value, the count of
the number of times this call will be made
consecutively. Notice the actual number of
times each call will be done is this
value * "repeat_count" parameter from the command line.
- Read the next string, which is the system
call name. Check it against the list of
supported calls (listed above) and if it doesn't
exist, abort with an error message.
- Depending on the system call, read the next
part of the line for any parameter it may require.
- Initialize your benchmark metrics and
make the system call (see the "Metrics" section below).
- Get your metrics again and find the
difference between the initial call.
- Print out the differences between the metrics
in a format suitable for later analysis.
o Continue on until all of standard input has been read.
In order to evaluate the performance of each system call, you'll need to measure some system parameters before and after the call. The difference in the values can then be used for comparison of different machines and operating systems.
One necessary metric would be the amount of wall clock time elapsed while executing the system call. Use the gettimeofday() call, which has a microsecond resolution on SunOS.
Since some of the calls make take a very short amount of time, you should structure your timestamp code like this (example for the read() call) :
#include <sys/time.h>
struct timeval tp;
struct timezone tzp;
unsigned long start, elapsed;
int x; /* value read in from first field of line */
.
.
.
gettimeofday(&tp, &tzp);
start = tp.tv_usec;
for (i = 0; i < x*repeat_count; i++) {
read(fd, buffer, bufsize);
}
gettimeofday(&tp, &tzp);
elapsed = tp.tv_usec - start; /* # of microseconds it took */
Another metric is to see how many system resources were consumed by the system call. The getrusage() call contains a large variety of system parameters, such as user CPU and system CPU time consumed and can be structured in the same manner as the gettimeofday() call above to get a "delta" usage value.
You should calculate the average of the "repeat_count" runs as well as the maximum and the minimum and report all three values on standard output, for use later in the analysis phase.
In order to compare different machines, you'll need to write your code so it will compile and run on different platforms. Some suggested environments to try :
o SunOS 5.3 machines (Suns in the Majors lab; xi)
o Linux machines (like cs1)
o Local disk I/O versus Network File System (NFS) I/O (run fsbench in the local "/tmp" directory for a local file system and in your home directory for an NFS file system).
o Different models of Suns and PCs within the department.
o PCs, if the compilation environment supports enough of the UNIX-style system calls. You only have to support those system calls that are in the environment (such as the Borland IDE).
o Any other machines that you have access to and that support UNIX-style file I/O.
You should try to obtain specific information for each type of machine to make the comparison, such as :
o Type of machine (e.g., Sun 4/40, Pentium @ 90Mhz, etc.)
o Operating System (SunOS 5.3, Linux 2.0.30, Windows '95, Windows NT, etc.)
o Compiler (gcc, cc, Borland IDE version 5.02, etc.)
Create a series of script files that you can use to compare system calls on a range of machines and operating systems. Then, make a series of runs of these script files on whatever machines and operating systems you can find.
The number of runs and types of runs is totally up to you -- the important criterion is that you generate and analyze enough data to show the ``top brass'' your work is thorough and correct (otherwise, you'll be assigned to work as a human subject in the plasma field experiments!).
This phase emphasizes the importance of using the benchmark written in the first phase to create and manage output files for the next phase.
Try to make your runs on machines that are lightly-loaded or where you are the only user. Repeat your runs to make sure you are getting consistent results.
This final phase is where you put all the pieces together. Using the results of the benchmark runs, generate a series of graphs and/or charts that indicate the performance of each machine and/or operating system.
Depending on what machines, operating systems, and disk environments you chose for your testing ground, you should generate charts, tables, and/or graphs that demonstrate on a system call-by-system call basis the performance differences. The performance comparisons should at a minimum include the amount of wall clock time, user time, and system time.
GROUPS OF TWO OR MORE ARE EXPECTED TO COMPARE ADDITIONAL PERFORMANCE METRICS.
Also analyze the differences and attempt to explain why a particular machine and software combination performed better or worse than another. Be sure to include this analysis in your paper.
Your paper should, at a minimum, include these topics :
1. Design of your "fsbench" program, including the structure of your output and any other design decisions you had to make. 2. List of run variations, including the types of machines, operating systems, and any other parameters that you change (local disk *vs* network disk, for instance). 3. Charts/graphs/tables of your results. 4. Discussion of the results.
Some general questions you should try to answer in your discussion :
o Which system calls are faster, the "f" versions of open, read, write, flush, seek, and close or the standard ones? Why?
o Is a read faster or slower than a write? What about repeated reads and repeated writes? Why?
o Given the same machine and operating system environment, do local disk drives run faster or slower than network disk drives? Why?
o Do failed system calls run faster or slower than system calls that don't fail? Why? Think about executing this script :
1 open testfile 100 read 10000 100 write 10000 1 close
versus running this script :
100 read 10000 100 write 10000