Chapter Overview

This chapter serves as a nexus for threads from several other courses. The ideas of sequential and binary search have been introduced in the mathematics courses required in our curriculum. These processes become important algorithms for many computing applications, and they are made into generic algorithms later in this course. Binary search, in particular, is sufficiently complicated that correctness of an implementation is never completely obvious. A process of proof of correctness is necessary to be certain. (Mathematical proof was also introduced in prior math courses.) It is important to have a working knowledge of the efficiency of algorithms to make sensible software decisions. Efficiency is typically measured in terms of computational complexity, a topic we introduce and use somewhat informally in this chapter and which is a central thread of the course Complexity of Data Structures and Algorithms which follows this one. Here are the chapter topics listed:


Algorithms

There are four components required of every algorithm: Assumptions, Asserted Outcomes, Body, and Proof. All four are important. Whenever an algorithm is introduced, in this course or any other environment, you should be certain that these four parts are clearly identified and understood.

Assumptions
Every algorithm is designed to operate in an environment in which certain assumptions hold. Sometimes these assumptions are quite explicit, and sometimes they are implicit, but assumptions are always there and are essential to designing the algorithm body and to verifying that the algorithm body in fact accomplishes the algorithm goals. One of the most common problems with algorithms is attempting to apply them when the assumptions are not met.

Asserted Outcomes
Goals, desired outcomes, purported outcomes, claims, whatever description you care to use, this part of an algorithm should state as clearly and precisely as possible what the algorithm is supposed to accomplish.

Body
The body of an algorithm is the description of the action of the algorithm. Often, this is the only part of an algorithm that is explicitly given. In such cases, the algorithm assumptions, outcomes, and proof must be deduced.

Proof
This is an important part of any algorithm, serving to validate the whole package as consistent and correct. The proof does not have to be formal, but it has to be convincing. It should verify that the algorithm body does indeed result in the asserted algorithm outcomes given that the algorithm assumptions hold. Specifically, the algorithm proof is a proof of the following theorem:

There are also optional assertions about the performance of an algorithm, typically asserting that the body of the algorithm runs in a certain time complexity or space complexity (or both). (We will discuss complexity later in this chapter.) These assertions, when present, should also be verified as part of the algorithm proof.

The heapsort algorithm, for instance, has the following assumptions, outcomes, and performance goals:

Heapsort Assumptions

Heapsort Outcomes

Heapsort Performance

(The heapsort algorithm body will be introduced and studied in a later chapter.) Any performance claims that are made should also be verified as part of the algorithm proof.

Sequential Search

Sequential search is the most basic, and most often used, search algorithm. It is used in all aspects of life, not just computing. In essence, sequential search is the process of examining a collection of things one at a time, in the order in which you come upon them, until either you find the thing you are looking for (success) or run out of things in the collection (failure). This algorithm can be applied to collections of physical things, such as books on a shelf, and to symbolic things, such as elements of an array. Sequential search requires only the most basic of amenities of the collection:

It is hard to imagine a collection that doesn't have these amenities. For example, books on a shelf have a left-most book (a place to start) and a right book-end (a way to stop). You get the next book by moving to the book on the right (or, if the books have all been looked at, the right book-end). And, presumably, you know what you are looking for and can tell when you find it. Two possibilities to keep in mind are:

  1. You are searching for a certain book title
  2. You are searching for a book in which you have hidden a $100 bill

We will see that the search for a specific title can be sped up significantly using a more sophisticated algorithm, while there is little hope of speeding up the search for the $100 bill.

Sequential Search Algorithm

Here is a more formal description of sequential search:

The language in which the algorithm is written we will call pseudocode. For this course, pseudocode will be a C-like language with certain liberties taken, such as "begin(L)" and "item is in L", when the meaning is clear. The proof, required, is postponed to the discussion of loop invariants.

Binary Search

The binary search algorithm has roughly the same asserted outcome as sequential search, while the assumptions and body are quite different. To understand these differences, return to the example problem of finding a book on a shelf.

Suppose, first, that we are searching for a specific book title. If we arranged the books in alphabetical order by title, we could speed up the search by a test/divide process. We start the search for a certain book title in the middle of the shelf. If the title of the middle book is the one we want, the search is over successfully. If the title we want precedes the middle title, then we know the book we seek must be on the first (left) half of the shelf, otherwise, it must be on the right half (or it might not be there at all). By looking at the middle book, we narrow the search down to one half of the original range. Now we repeat the process, working with the half of the shelf to which the search was narrowed in the previous step. Looking at the book in the middle of the new range, we either find the book we seek, or we narrow the search range again by a factor of two, to 1/4 of the original. Testing again narrows the range to one-half of 1/4, or 1/8. And so on, until we either find the book or deduce that it is not on the shelf. Note that after k steps, the algorithm has reduced the range by a factor of 2k. As soon as 2k equals or excedes the number of books, the process terminates. Note that 2k >= n is equivalent to k >= log2 n.

The process just described is called binary search, probably because it systematically reduces the search range by a factor of 2 at each step. The essential features of the bookshelf we used are:

Note that the order of the books is by title, the same as the item of data for which we are searching. This is a critical point. For example, having the books set up as described will do nothing to speed up our search for the $100 bill. We can test the middle book for the $100, but not finding it will not give us any information about which side of the test book the $100 my reside on. To find the $100, we have to go back to sequential search. Neither being sorted by title nor having direct access will help.

Recapping:

Binary Search


Binary Search - the idea

We illustrate the idea behind binary search with a vector v of 32 elements. The element type of the vector is a 2-character string. We must decide whether the string t = "PB" is in the array. The vector is set up with the elements in alphabetical order, and it provides random access through the bracket operator, so we should be able to perform binary search. Here is the vector:

AB CB CF DA DD DF DG EE FE FG FJ FL HK LA LB LM LX LZ MZ PA PX QQ QZ ST SX SZ TA TX WX WZ YA YX
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Binary search begins with the middle element of v, element 16. Element 16 has value v[16] = "LX". Comparing with the search value t = "PB", we see that

which tells us that if t is in v, then it must be at an index higher than 16. This narrows the search to the range [17,32) = v[17]..v[31]. The slide shows this process repeating in this smaller range, testing the middle index, going left or right, for five tests. We conclude that t is not in v. The proccess proceeds as follows: 

Search value: t = PB
Search range: [0,32) = {0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31}
Compare middle element with t: v[16] ? t
                                  LX < PB => t can only be in [17,32) = {17...31}
New search range: [17,32) = {17 18 19 20 21 22 23 24 25 26 27 28 29 30 31}
Compare middle element with t: v[24] ? t
                                  SX > PB => t can only be in [17,24) = {17...23}
New search range: [17,24) = {17 18 19 20 21 22 23}
Compare middle element with t: v[20] ? t
                                  PX > PB => t can only be in [17,20) = {17,18,19}
New search range: [17,20) = {17,18,19}
Compare middle element with t: v[18] ? t
                                  MZ < PB => t can only be in [19,20) = {19}
New search range: [19,20) = {19}
Compare middle element with t: v[19] ? t
                                  PA < PB => t can only be in [20,20) = {}
Search range is empty, so element is not found.
Return false (not found)

A more detailed version of the process is captured in Stillmation 1. See a similar process for the found case in Stillmation 2.

Binary Search Algorithm

We present the binary search algorithm using a vector v whose elements are arranged in increasing order and a search value t that may or may not be an element of v. There are three versions of binary search that are important and will eventually become generic algorithms: binary_search, lower_bound, and upper_bound. These versions have very similar algorithm bodies and only subtly different asserted outcomes. They all have the same assumptions:

The version binary_search answers the fundamental question: is the search value in the search range? The answer is returned as a boolean, yes or no. The other two versions return a location in the search range with the following definitions:

These two values are, in the usual technical sense, the indices of the lower bound and upper bound of t in the range v. To gain understanding of their significance, it helps to consider the two cases "t is an element of v" and "t is not an element of v".

Suppose, first, that t is an element of v. Then lower_bound returns the index of the first occurrence of t in v, and upper_bound returns the index of the first element larger than t in v. In other words, if L = lower_bound and U = upper_bound, then all of the elements in v from index L up to (but not including) index U are equal to t, and all instances of t in v are included in this range. The elements of v that are equal to t are precisely those in the index range [L,U).

On the other hand, suppose that t is not an element of v. Then lower_bound and upper_bound return the same value: because the possibility t == v[i] does not exist, the inequalities defining the lower_bound and upper_bound are equivalent. The index defined (by either inequality) is the place where t would be if it were in v, either the index of an existing element (the first element of v larger than t) or the "past the end" index.

These descriptions are a bit intricate, but you should now be convinced that lower_bound and upper_bound are valuable pieces of information. In fact, the boolean-valued binary_search can be defined in terms of either of these. We have chosen to use lower_bound, as shown in the next slide.

If you want to experiment with a working test program for binary search, try the executable area51/fgss.x. The program asks for a sentinal (enter a character, say '.') and then character data. The character data will fill up an array, a vector, and a deque with the entered characters (stop by entering the sentinal). After sorting the three containers, the program asks for a character to search for. Try the experiment.

Binary Search Code

Here is working C++ code for the three versions of binary search. The body of the function comprises the body of the algorithm. Some assumptions are explicit in the function header. Other assumptions, such as the need for bracket operators, are implicit in the body itself. This code is somewhat optimized from the previous description: the possibility that we happen to find t early is very unlikely, which makes the test for equality more time consuming, on average, than simply following the test/divide process to its end. Thus, instead of dividing into three cases (one for the unlikely early discovery of t) these implementations divide into two cases, "less than" and "not less than". 

unsigned int lower_bound (T* v, unsigned int size, T t)
{
  unsigned int   low  = 0;
  unsigned int   mid;
  unsigned int   hih = size;
  while (low < hih)
  {
    mid =  (low + hih) / 2;  
    if (v[mid] < t)          
      low = mid + 1;         
    else                     
      hih = mid;             
  }  
  return low;
}

unsigned int upper_bound (T* v, unsigned int size, T t)
{
  unsigned long   low = 0;
  unsigned long   mid;
  unsigned long   hih = size;
  while (low < hih)
  {
    mid =  (low + hih) / 2;
    if (t < v[mid])
      hih = mid;                
    else                        
      low = mid + 1;            
  }  
  return low;
}

bool binary_search (T* v, unsigned int size, T t)
{
  unsigned int lb = lower_bound(v,size,t);
  if (lb < size)
  {
    if (t == v[lb])
    {
      return true;
    }
  }
  return false;
}

READ the lower_bound code carefully. Then take some specific cases and trace the code. This should get you to the point of believing that the code follows the general idea of binary search. But, do not be concerned if you are uncertain of a proof of this algorithm. Binary search is not very difficult to understand in principle, but specific algorithm bodies are notoriously difficult to get exactly right. We will trace the algorithm once here. We return to an actual proof later in the chapter. Here is a trace: 

  // input:
    v = [ab, cb, eb, fx, ga, gf, pb, pv, sx, uu, wx]
index =  0   1   2   3   4   5   6   7   8   9   10
  max = 11
    t = pv
  // setup:
  low = 0
  hih = 11
  // while loop
  { // begin while
    test: true = (0 < 11) = (low < hih)
    {
      mid = 5 = 11/2 = (0 + 11)/2 = (low + hih) / 2;
      test: true = ("gf" < "pv") = (v[mid] < t)
        low = 6 = 5 + 1 = mid + 1;
    }
    test: true = (6 < 11) = (low < hih)
    {
      mid = 8 = 17/2 = (6 + 11)/2 = (low + hih) / 2;
      test: false = ("sx" < "pv") = (v[mid] < t)
        hih = 8 = mid;
    }
    test: true = (6 < 8) = (low < hih)
    {
      mid = 7 = 14/2 = (6 + 8)/2 = (low + hih) / 2;
      test: false = ("pv" < "pv") = (v[mid] < t)
        hih = 7 = mid;
    }
    test: true = (6 < 7) = (low < hih)
    {
      mid = 6 = 13/2 = (6 + 7)/2 = (low + hih) / 2;
      test: true = ("pb" < "pv") = (v[mid] < t)
        low = 7 = 6 + 1 = mid + 1;
    }
    test: false = (7 < 7) = low < hih)
  } // end while
  return 7 = low;

Thus 7 is returned as the lower bound index.

Exercise: Trace the upper_bound algorithm with the same input.

Tracing Specific Binary Search Algorithms

Here are some traces of specific calls to the lower_bound and upper_bound algorithms

Enter search value ('.' to quit): h
  =================
  lower_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
     ----
       m
     dghhkksy
     --
      m
     dghhkksy
       ^lb
  =================
  upper_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
     ----
       m
     dghhkksy
        -
        m
     dghhkksy
         ^ub

Enter search value ('.' to quit): s
  =================
  lower_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
          ---
           m
     dghhkksy
          -
          m
     dghhkksy
           ^lb
  =================
  upper_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
          ---
           m
     dghhkksy
            -
            m
     dghhkksy
            ^ub

Enter search value ('.' to quit): p   
  =================
  lower_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
          ---
           m
     dghhkksy
          -
          m
     dghhkksy
           ^lb
  =================
  upper_bound trace
  =================
     dghhkksy
     --------
         m
     dghhkksy
          ---
           m
     dghhkksy
          -
          m
     dghhkksy
           ^ub

The executable binarySearchTrace.x is available for experimentation in area51 of the course library.

Issues of Proof

Often issues of proof are introduced in the setting of a trivial algorithm, which in my view obscures the whole point. In the setting of a complex algorithm, such as the lower_bound algorithm, it is much easier to understand the need for proof: most people, including me, are not completely sure the algorithm is a consistent package just by reading the code.

These issues are made even more important by our desire to make binary search algorithms reusable as generic algorithms. Generic algorithms make the actual code reusable across a variety of containers. Delivering a faulty generic algorithm would be the ultimate software disaster.

There are two kinds of assertions we need to prove for binary search: those having to do with correctness and those having to do with performance. The former are mandatory for any algorithm; the latter are optional but highly recommended. For lower_bound, in particular:

We consider these issues in the next few slides.

Correctness and Loop Invariants

For any loop there are issues of correctness summarized as follows:

For simple fixed-size loops, such as the following for loop, loop termination is not an issue, because it is more or less "obvious":

The body of this loop will execute exactly n times and program execution will jump to the line following the loop body. These conclusions are clear, rarely questioned, taken for granted. But, suppose someone asked you to sabotage this loop, make its termination less obvious? You could engage in some rather subversive activities (not that we would suggest actually doing such things) such as:

Oops. The function subvert() takes its parameter by reference and decrements it, effectively negating the effect of the future incrementation to occur at the end of the loop body. But wait - the parameter is an unsigned integer, and it's first value is zero, so after decrementation it has a very large value, probably at least as large as n. Maybe this loop terminates after one execution. Maybe it isn't so obvious that the simple for loop terminates! But if we read the actual loop body, and there isn't anything there that might be subversive (e.g., that might change the value of the loop counter i), then it should be straightforward to conclude that the loop terminates.

The structure of the loop in lower_bound presents a more complex proof-of-termination problem:

To begin the analysis, identify the loop continuation condition. This loop body executes iff the boolean expression (low < hih) returns true. The contrapositive is that the loop body does not execute if the expression returns false, i.e., the loop terminates if (low >= hih) returns true. The key question is then: are we guaranteed that (low >= hih) returns true after a finite number of iterations of the loop body?

A loop invariant is a statement that is provable true for each iteration of a loop. Loop invariants are analogous to the induction statements that are traditional components of proof by mathematical induction.

To show that the lower_bound loop above terminates, we will introduce loop invariants and use them to show that the quantity hih - low is non-negative and gets smaller with every iteration of the loop, a process that cannot proceed more than a finite number of times.

Mathematical induction is based on the fundamental fact that a non-negative integer quantity can get smaller only a finite number of times before becoming zero, which is in effect the loop termination condition. We refer to this argument as collision with zero.

Loop Invariants - Sequential Search

Here is the sequential search algorithm, with two loop invariants embedded as comments in the code: 

boolean sequential_search
{
  T item = first(L);
  // Before entering loop: t has not been found
  while (item is in L)
  {
    // loop invariant: current item has not been tested
    if (t == item)
      return true;
    // loop invariant: t is not current item
    item = next(L);
  }
  return false;
}

The first loop invariant is that the current item has not been tested. This is true upon entering the loop body. The second loop invariant is that the current item is not the one we are searching for. This is true as we make a standard exit of the loop body. For, if the current item is the search item, then we would have exited the loop and the algorithm at the previous step.

Using these loop invariants, we can argue that the loop is guaranteed to terminate and return a correct value. Either the loop terminates abnormally through an early, successful return of true, or it terminates normally without finding the search value, and then returns false.

Loop Invariants - Binary Search

Here is the lower_bound algorithm with some loop invariants embedded as comments: 

unsigned int lower_bound (T* v, unsigned int max, T t)
{
  unsigned int   low  = 0;
  unsigned int   mid;
  unsigned int   hih = max;
  while (low < hih)
  {
    // (1) low < hih
    // (2) v[low - 1] < t (if index is valid)
    // (3) t <= v[hih]     (if index is valid)
    mid =  (low + hih) / 2;  
    if (v[mid] < t)          
      low = mid + 1;         
    else                     
      hih = mid;             
    // (4) low <= hih
    // (5) hih - low has decreased
    // (6) v[low - 1] < t (if index is valid)
    // (7) t <= v[hih]     (if index is valid)
  }  
  return low;
}

It is a telling fact that the loop body has only four lines of code, yet we have seven loop invariants. That is a big hint that the proof is subtle. We have to show that the loop terminates, a job started a few paragraphs earlier. Then, we have to prove that the algorithm delivers the items promised in the deliverables statement. First, let's prove the invariants one at a time.

Assertion (1) low < hih

OK, so we're starting out easy: loop invariant (1) asserts that the loop entry condition is true as we enter the loop.

Assertion (2) v[low - 1] < t (if index is valid)
Assertion (3) t <= v[hih] (if index is valid)

If the current loop iteration is not the first iteration of the loop, then (2) and (3) just repeat for the record what was true as we exited the previous iteration (assertion (6) and (7)). We only need to verify (2) and (3) in the case of the first iteration.

In the first iteration, we know that low = 0 and hih = max. Thus, low - 1 = -1 and hih = max are each invalid index values. The assertions (2) and (3) are therefore true by default.

The remaining assertions are placed at the end of the loop body, so they must be proved under the assmption that (1), (2), and (3) were true when entering the loop body and then the body has been executed. There are two cases:

Case 1: v[mid] < t

Assertion (4) low <= hih

Proof: 

low = mid + 1 = (low + hih)/2 + 1
              < (hih + hih)/2 + 1 // by (1)
              = hih + 1
Because we are working with integers, (low < hih + 1) implies that (low <= hih).

Assertion (5) hih - low has decreased

Proof: 

hih - low =  hih - (mid + 1)
          =  hih - mid - 1
          <  hih - mid
          =  hih - (old_low + hih)/2
          =  hih - hih/2 - old_low/2
          =  old_hih - old_hih/2 - old_low/2    // because hih = old_hih
          <= old_hih - old_low/2 - old_low/2    // because old_low <= old_hih
          =  old_hih - old_low
Therefore, hih - low has decreased.

Assertion (6) v[low - 1] < t (if index is valid)

Proof: 

v[low - 1] = v[mid + 1 - 1] = v[mid] < t

Assertion (7) t <= v[hih] (if index is valid)

Proof:

hih has not changed values, so this is true in the current iteration because it was true in the previous iteration.

Case 2: v[mid] >= t

Assertion (4) low <= hih

Proof: 

hih = mid =  (low + hih)/2 
          >= (low + low)/2  // by (1)
          =  low

Assertion (5) hih - low has decreased

Proof: 

hih - low = mid - low 
          = (old_hih + old_low)/2 - low
          = (old_hih + old_low)/2 - old_low  // because low =  old_low
          = (old_hih + old_low)/2 - (old_low + old_low)/2
          = (old_hih - old_low)/2
          <  old_hih - old_low               // because old_hih - old_low > 0

Therefore, hih - low has decreased.

Assertion (6) v[low - 1] < t (if index is valid)

Proof:

low has not changed values, so this is true in the current iteration because it was true in the previous iteration.

Assertion (7) t <= v[hih] (if index is valid)

Proof: 

v[hih] = v[mid] >= t

We have now proved all of the loop invariants (1) through (7). It remains to prove that the algorithm terminates and that it delivers what it promises. The loop invariants are key to the proofs.

Assertion: The lower_bound algorithm terminates.

Proof: Loop invariant (5) says that the quantity hih - low decreases in each loop iteration. Invariant (4) says that this quantity is non-negative. Therefore, the loop terminates when hih - low collides with zero.

Assertion: The return value of lower_bound is the lower bound index for t in v, that is, the smallest index i such that t <= v[i].

Proof: When the algorithm terminates, low = hih. Assertion (6) says that no index smaller than low has element equal to t. Assertion (7) says that the element with index hih (= low) is no smaller than t. This makes low = hih the lower bound index.

Assertion: If t is in v, then lower_bound is the index of the first occurance of t in v.

Proof: (Left as an exercise.)

Assertion: If t is in v, then upper_bound is the index one past the last occurance of t in v.

Proof: (Left as an exercise.)

Assertion: If t is not in v, then lower_bound and upper_bound are each the index where t should be found in v.

Proof: (Left as an exercise.)

We now turn to the issues related to performance.

Computational Complexity

The notion of computational complexity was introduced in the prerequisite math courses, so you should be somewhat familiar with it. We will re-introduce the ideas in this chapter. (See reference texts for elaboration.)

Computational complexity provides a language for comparing the growth of functions that are defined on (all but finitely many) non-negative positive integers and which have non-negative real number values. The important class of examples that we will use are functions whose values are run time or run space of an algorithm, and whose input is a number representing the size of the data set on which the algorithm operates.

Complexity of a function is dependent only on its "eventual" behavior, that is, complexity is independent of any finite number of initial values. The practical effect of this is to ignore initialization phases of algorithms.

Complexity of a function is independent of any constant multiplier. The practical effect of this is to ignore differences in such things as processor speed when comparing performance of two algorithms.

The properties just stated come directly from the definition of computational complexity. A more subtle property, deducible from the definition, is that complexity is independent of so-called "lower order" effects. We will not attempt to make this last statement precise, but we will give some examples to illustrate the concept.

The key measures of computational complexity are known as "Big O" notation, "Big Omega" notation, and "Big Theta" notation. You should be somewhat familiar with at least some of these. The definitions appear in the next slide.

Asymptotic Relations and Notation

The Big O class of a function f consists of all functions that are "asymptotically bounded above by f". The official definition from [Cormen] follows:

Notice that f(n) is in O(f(n)), as are 100f(n) and 100 + f(n). Notice also that the condition defining O(f(n)) is an upper bound condition. Big Omega is defined using the corresponding lower bound condition:

and Big Theta is defined using both at once:

The following facts about computational complexity are not extremely difficult to prove. The proofs are left as exercises.

Theorem 1 (transitivity).
(1) If f(n) is in O(g(n)) and g(n) is in O(h(n)) then f(n) is in O(h(n))
(2) If f(n) is in Ω(g(n)) and g(n) is in Ω(h(n)) then f(n) is in Ω(h(n))
(3) If f(n) is in Θ(g(n)) and g(n) is in Θ(h(n)) then f(n) is in Θ(h(n))

Theorem 2 (anti-symmetry). f(n) is in O(g(n)) iff g(n) is in Ω(f(n)).

Theorem 3 (symmetry). f(n) is in Θ(g(n)) iff g(n) is in Θ(f(n)).

Theorem 4 (reflexivity). A function is asymptotically related to itself: f(n) is in O(f(n)), f(n) is in Ω(f(n)), and f(n) is in Θ(f(n)).

In particular, of the three, Θ defines an equivalence relation on functions while O and Ω define an anti-symmetric pair of relations on functions that is analogous to the pair of order relations (<=, >=) on numbers. Thus, Θ is analogous to '=' (equality), while O is analogous to '<=' (less than or equal to) and Ω is analogous to '>=' (greater than or equal to). There is even a form of the dichotomy property of order relations:

Theorem 5 (dichotomy). If f(n) <= O(g(n)) and g(n) <= O(f(n)) then f(n) = Θ(g(n)).

All of these theorems can be restated in an equivalent form using set notation. For example, Theorem 5 restates as follows:

Theorem 5a (dichotomy). If O(f(n)) subset_of O(g(n)) and O(g(n)) subset_of O(f(n)) then Θ(f(n)) = Θ(g(n))).

We will use the notation of equality and inequality, as shown in the slide, as a notational device that helps reinforce our perception of the nature of these three relations. (We recognize that this is an abuse of notation. See the discussion of notation "abuse" and notation "misuse" in [Cormen].)

Big O notation is commonly confused with Big Theta in informal settings (usually without harm). Another reason for our use of the inequality notation for Big O and Big Omega is as a device to prevent slipping into the habit of such confusion. These important concepts are studied in detail in the reference texts. We conclude this slide with some examples.


Algorithm Complexity

The complexity of an algorithm is stated in terms of Big O, Big Omega, or Big Theta. It is independent of specific implementation details including hardware platform, software platform (operating system), and programming language. We apply the notation and theory of computational complexity to algorithms using a four step process.

  1. Discover a measure of size of input to the algorithm:
    n = some measure of size of input to the algorithm
  2. Decide on a notion of atomic computational activity that captures the work of the algorithm
  3. Find the function f(n) = the number of atomic computations performed on input of size n.
  4. The complexity of the algorithm is the complexity of f(n).

We illustrate this process in several examples as we conclude this chapter, as well as in various places throughout the remainder of the course.

Algorithm Complexity - Loops

Despite the facetious remarks we made earlier about how "obvious" it is that a simple fixed-bound loop terminates, it actually is true that simple loops are straightforward to analyze. Usually it is clear when and why they terminate and what computational work is accomplished in each iteration of the loop body. If, for example we define an atomic computation as a call to a comparison operator, there might be three such calls in the loop body.

Consider for example the following loop: 

for (i = 0; i < n; ++i)
{
  // 3 atomics in loop body
}

The complexity of the loop is defined to be the complexity of the function
   f(n) = (no. of atomics in loop body) x (no. of iterations of loop)
= (3) x (n)
= 3n
Thus, the complexity of this loop is equal to
   Θ(f(n)) = Θ(3n)
= Θ(n)
The situation is often not quite this simple, however.

Algorithm Complexity - Loops with Break

The case of a loop with a conditional breakout is shown in the the following code: 

for (i = 0; i < n; ++i)
{
  // 3 atomics in loop body
  if (condition) break;
}

In the cases where the loop runs to normal termination, the run time of the loop is correctly modelled by the same function as for the simple loop above. But in other cases, the loop may terminate sooner. These cases are data dependent; that is, the runtime of the loop varies from an upper bound of 3n = Θ(n) to a lower bound of 3 = Θ(1), depending on the specific input to the loop. We cannot conclude that the algorithm has complexity Θ(n) because the lower bound condition >= Ω(n) does not hold. Therefore, the best we can conclude is that the loop has complexity <= O(n).

Algorithm Complexity - Loops in Sequence

The following code fragment has two loops one following the other in the source code. 

for (i = 0; i < n; ++i)
{
  // 3 atomics in loop body
}
for (i = 0; i < n; ++i)
{
  // 5 atomics in loop body
}

These are sometimes referred to as concatenated loops. Concatenated program blocks execute in sequence, one after another. Therefore the runtime of two concatenated blocks is the sum of the runtimes of the individual blocks. For the situation depicted on the slide, the runtime is bounded above by O(3n + 5n) <= O(n).

Algorithm Complexity - Loops Nested

The following fragment has two loops one inside the other. 

for (i = 0; i < n; ++i)
{
  // 2 atomics in outer loop body
  for (j = 0; j < n; ++j)
  {
    // 3 atomics in inner loop body
  }
}

These are sometimes referred to as composed or nested loops. The runtime of two composed blocks is the product of the runtimes of the individual blocks. Thus, the runtime of the composed loops depicted in the slide is bounded above by O((2 + 3n)n) <= O(2n + 3n2)) <= O(n2). It is also legitimate to conclude the similar result using Big Theta:

Complexity = Θ((2 + 3n)n) = Θ(2n + 3n2) = Θ(n2)

The lesser conclusion is nevertheless correct and often the only one stated explicitly.

Algorithm Complexity - Sequential Search (Simplified)

The following version of sequential search is simplified by removing the possibility of early breakout. As such, the algorithm is modelled precisely by the simple fixed-size loop. 

T    item  = first(L);
bool found = false;
while (item is in L)
{
  if (t == item)
    found = true;
  item = next(L);
}
return found;

The complexity of this algorithm is Θ(n).

Algorithm Complexity - Sequential Search

Here is the usual (more efficient) version of sequential search in which there is early breakout of the loop immediately upon successful search: 

T    item  = first(L);
while (item is in L)
{
  if (t == item)
    return true;
  item = next(L);
}
return false;

This is modelled by the simple loop with break, and hence has complexity <= O(n).

Algorithm Complexity - Binary Search

Binary search is captured in the lower bound algorithm: 

lower_bound
{
  unsigned int   low  = 0;
  unsigned int   mid;
  unsigned int   hih = size;
  while (low < hih)
  {
    mid =  (low + hih) / 2;  
    if (v[mid] < t)          
      low = mid + 1;         
    else                     
      hih = mid;             
  }  
  return low;
}

This algorithm consists of a single loop that does not have any abnormal breakout termination, so the complexity of the loop is essentially the number of loop iterations as a function of n = size of input. Some helpful observations (derived from our previous analysis of correctness) follow.

Observation 1. The loop terminates after the search range is reduced to size <= 1.

Observation 2. The search range is n = size for the first iteration of the loop.

Observation 3. The search range is reduced by 1/2 at each iteration.

Thus, the loop runs for as many times as we can divide n by 2 before reaching the value 1. The first k sizes in these iterations are n, n/2, n/4, n/8, ..., n/2k, and the loop termination condition is

    n/2k <= 1     equivalent to     n <= 2k     equivalent to     log2n <= k.

The last inequality shows that the loop terminates in log2n iterations (give or take one). The complexity of binary search is therefore <= O(log2n).

Exercises:

  1. Verify the three observations above.
  2. Verify that the complexity of binary search is >= Ω(log2n), and conclude that the complexity is exactly Θ(log2n).
  3. Discussion question: Why is it appropriate not to include the "early return" option for binary search, as we did for sequential search? (Hint: Think cost effectiveness.)