Measuring Performance

A systems administrator needs to be able to judge the difference between a system that is operating at peak performance from one that is thrashing due to overcommittment of resources. This exercise will help you determine that boundary. Also you may be asked to fairly evaluate performance characteristics of different machines to figure out which ones deliver better performance or even better price/performance. This assignment will provide both experiences.

You may find it easier to do the first two parts on your Linux or Solaris machine.

Part I - Single machine baseline performance measurements

• CPU: Write (or obtain) a CPU-bound program (a C program that performs a few million square roots, for instance). On your own machine, while there is little to no other CPU activity, use the "time" command to obtain the amount of user, system, and wall clock time. Make sure you keep the runtime low (no greater than a few minutes, at the most). Record these numbers.
• MEMORY: Write (or obtain) a memory-bound program (a C program that continually "malloc()"s large blocks of memory, for instance). On your own machine, while there is little to no other CPU activity, use the "time" command to obtain the amount of user, system, and wall clock time. Make sure you keep the runtime low (no greater than a few minutes, at the most). Record these numbers. The amount of memory that your program consumes will be a function of how much real and virtual memory you have installed on your machine.
• DISK: Either write a disk-bound program or use an existing disk-bound set of programs/shell scripts. For example, an excellent disk-bound benchmark is the Linux kernel build. Use the "time" command to obtain the vital statistics. Record these numbers.

Part II - Single machine stress tests

Create a series of simple tests, using combinations of the above three baseline benchmarks. What you are trying to do is see how well your machine scales with an increased workload. Measure the time it takes to perform your stress tests and calculate the load distribution.

For example, a CPU-only stress test could involve running ten simultaneous copies of your CPU baseline benchmark. Assume it's in a binary called "cpu". Then, to obtain timings on all ten of them running concurrently, you could do this (assume all ten lines are in a single shell script):

		#!/bin/sh
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		time ./cpu &
		wait

If you place this in a file named "cpustress.sh" and set the execute bit ("chmod +x cpustress"), then you can perform the CPU stress time and get an overall timing via:

		time ./cpustress.sh

Your stress tests should include the following mixes:

• Multiple CPU only
• Multiple Memory only
• Multiple Disk only
• Combination of CPU, Memory, and Disk

After you are done, write up a report that shows if your machine reached a point where the overall time was much greater than the expected individual times. This indicates you have stressed your machine beyond the point of being able to equitably share the resources.

For example, you may be able to run one copy of the memory benchmark in one minute, two copies in two minutes, but you find it takes ten minutes to run three copies (due to excessive paging behavior).

Part III - Heterogenous Machine Benchmarks

Take your CPU benchmark and run it on the following machines. Notice that for your NT machine you will need access to a C compiler (if your benchmark is in C). You can use the Borland C++ compiler installed on the NT machines in the Majors lab or your personal computer to generate the ".EXE" binary that you then run on your '486 NT machine.

• Your team's Linux machine
• Your team's NT machine
• Your team's Solaris machine
• xi.cs.fsu.edu
• smitty.cs.fsu.edu (an IBM RS/6000 running AIX)
• hpucks.cs.fsu.edu (an HP running HP-UX)
• midas.cs.fsu.edu (an SGI O2 running Irix)
• Any other machines you have access to

Generate a table that lists the relative performances of each of the following machines (e.g., let your '486 Linux machine = 1, then the Sparc may be .75 of that, etc.). Research what you think the machine actually costs and then, based on the CPU relative performance, calculate the price/performance ratio. For example, a $250 '486 with a performance ratio of 1 would have a price/performance ratio of $250 ($250/1), while a $500 ancient Sparc with a performance ratio of .75 would have a price/performance ratio of $667 ($500/.75). Pricing of computer equipment can be somewhat difficult; try to discover a ball-park purchase price.

Try to answer the following questions in your journal:

• Which machine is the fastest?
• Which is the slowest?
• Based on the CPU price/performance, which machine would you recommend?
• Given the stress test results, which operating system handled the increased load better?
• Did you find a reliable way to measure execution times under NT?
• Given comparable hardware, how does Linux compare with NT?
• How fair is this benchmark (in terms of how well it measures the difference between the machines)?

Suggested columns for your table:

Machine	CPU	Operating Relative		
Type	Speed	System	  Timing of CPU benchmark System Price Price/Performance
-------	-----	--------- ----------------------- ------------ -----------------

Note that for a more robust comparison you would want to stress-test the CPU, memory and disk of your target machines. In this case, though, we will limit ourselves to comparing only the relative CPU performance.