Greedy Algorithms

The "greedy" approach to algorithm design is essentially: In a sequential algorithm, look at all options for next step and choose the option that looks best at that time, given that we might not have all information. I.e., choose the option looks best "locally".

Example: Making Change (US Coinage)

Problem: Given a purchase with a fractional dollar price, determine the change due in as few coins as possible.

The greedy approach would be to add the largest coin that doesn't excede the amount remaining until the amount is reached.

Make change (US currency) with fewest possible coins:

initialize amount = amount_tendered - total_bill
[assume amount is less than one dollar]

while (amount > 0)
{
  select largest coin <= amount // <-- greedy step
  put(coin,hand)
  amount -= coin
}

Example runs: 87 cents = 50 + 25 + 10 + 1 + 1 [5 coins]
              45 cents = 25 + 10 + 10         [3 coins]

Exercise: Show that the greedy algorithm always determines correct change using as few coins as possible.

Note that the result is dependent on the design of the coinage denomination set. For example, if we added a new 13¢ coin, the greedy approach would not be optimal:

Greedy:  17 = 13 + 1 + 1 + 1 + 1  [5 coins]
Optimal: 17 = 10 + 5 + 1 + 1      [4 coins]

Exercise: What conditions on coin denominations guarantee greedy approach works?

1 Examples of Greedy Algorithms

We will encounter other "greedy" algorithms later in the course. These will be discussed in greater detail in context. However here are brief descriptions of some of them:

1.1 Kruskal's Minimum Spanning Tree Algorithm

A minimum spanning tree [MST] for a weighted graph G = (V,E,w) [undirected] is a subgraph (V,T) such that T is a tree with total weight as small as possible. (Note that the vertices for the tree are all of the vertices for G.) Obviously it is necessary for G to be connected in order to have a MST. It turns out that is also sufficient.

Kruskal: construct MST for G = (V,E,w)
assumption:  G is a connected weighted graph

initialize forest F = (V,S) with S = empty
L = list of edges sorted by edge weight
while (F is not connected)
{
  // greedy step:
  choose edge e = {x,y} with minimal weight such that x and y are in different components of F
  add e to S
}

Kruskal's approach is to build a forest F, starting with all the vertices, connecting two trees in F with a minimal weight edge (the greedy step) until they are all connected, whence F is a tree.

1.2 Prim's Minimum Spanning Tree Algorithm

In contrast to Kruskal, Prim's approach is to built a tree T, expanding it by adding a new edge of minimal weight and exactly one end in T (the greedy step) until there are no edges left with only one vertex in T. At that point, all vertices must be in T:

Prim: construct MST for G = (V,E,w)
assumption: G is a connected weighted graph

initialize tree T = (X,S) with X = {v} and S = empty
while (X != V)
{
  // greedy step:
  choose edge e = {x,y} with minimal weight such that x is in X and y is not in X
  add y to X
  add e to S
}

1.3 Dijkstra's Shortest Paths Algorithm

Performing breadth-first search from a vertex v in a connected graph (directed or undirected) results in shortest paths from v to every other vertex in G. Similarly, in an undirected weighted graph, The Prim and Kurskal algorithms provide minimum weight paths from the root of the MST.

These are often called "single source shortest paths" [SSSP] algorithms. BFS solves the SSSP problem in both undirected and directed graphs, and Prim and Kruskal solve the SSSP problem for undirected weighted hraphs. The last case, graphs that are both weighted and directed, is a more subtle problem. This last case is solved, with the additional assumption that weights are non-negative, by Edsger Dijkstra:

Dijkstra: construct shortest path from v to all other vertices
assumptions: G = (V,E,w) is connected weighted directed graph and all weights are >= 0

assign d(v)  = 0 and d(x) = infinity for all vertices x != v // d(x) = current known distance from v to x
assign p(x)  = NULL for all vertices x  // p(x) = predecessor of x along current shortest known path
initialize S = V
do
{
  // greedy step:
  x = a vertex in S with minimal d() // initially this must be v
  for each y adjacent to x (with edge e = (x,y))
  {
    if ( d(x) + w(e) < d(y) )
    {
      d(y) = d(x) + w(e) // update current known distance to y
      p(y) = x // update current known shortest path to y
    }
  }
  remove x from S
}
while (S is not empty)

The shortest paths are encoded in the predecessor array p.

Large issues remains for all of these graph algorithms: What is the structure of the set used to feed the loop with the optimal (greedy) choice for the next vertex? Proofs of correctness? Run times? These are taken up in the Graphs notes.

2 When Does the Greedy design approach work? (Optimal Substructure + Greedy Choice Property)

The greedy approach to algorithm design requires the same fundamental characteristic as dynamic programming [DP]:

Bellman Optimal Substructure Property. An optimization problem has optimal substructure iff any optimal solution contains within it optimal solutions to (some of) its subproblems.

In contrast to the "brute force" power of DP, where we essentially solve all sub-problems in order to use just a thin O(n) slice of those solutions, the greedy approach avoids solving the sub-problems altogether by making a "locally optimal" choice, that is, a choice that is best given our currently incomplete state of knowlege:

Greedy Choice Property. An optimization problem has the greedy choice property iff a globally optimal solution can be assembled making locally optimal choices. In other words, when we are considering which choice to make, we make the choice that is optimal in the current problem without considering results from subproblems.

For example, in Dijkstra's SSSP algorithm, we choose a vertex x based on minimal d(x) and extend paths from x to its neighbors y, updating d(y) = d(x) + w(x,y). But d(x) is only the length of our current best known shortest path from v to x. Conceivably there might be other paths as yet undiscovered that are shorter. The comparable step in a DP solution would be to calculate all lengths of paths from v to neighbors x of y and minimize d(x) + w(x,y). Yet, the Dijkstra algorithm does correctly produce a set of shortest paths from v to all other reachable vertices. (The proof is taken up in the Graph notes.)

One can gain intuition as to when a greedy approach will work, or is at least worth trying, and if sucessful the resulting algorithm is likely to be more efficient than a DP approach would be, due to the latter solving many (or all) subproblems and having to store those solutions. On the other hand, greedy does not always work, as we saw in the change-making example, and it may take considerable insight into the problem itself to figure out why. (Note that the change-making problem can always be solved with DP, for any coinage design.)

However: The question of which problems succumb to a greedy algorithm (compared to the same question for DP) is a subtle one for which there is still no straightforward answer. One large class of problems, modelled by Matroids, is guarenteed to be solvable by greed. (Matroids, introduced by Hassler Whitney in 1935 ["On the abstract properties of linear dependence", American Journal of Mathematics, 57(3):509-533.] and their greedy solution are discussed in [Cormen].) But Matroids do not cover all problems solvable by the greedy approach. A precise characterization of exactly which problems do is an open question.

3 Knapsack Problem (Revisited)

In the notes on Dynamic Programming we used the following notation for a Knapsack problem:

Knapsack variables and notation:

n = number of resources
W = maximum weight
{1,2, ... n} = R = set of resources
v₁, v₂, ..., v_n: resource values
w₁, w₂, ..., w_n: resource weights or costs
Thus item k has value v_k and weight w_k.

We assume that weights are non-negative integers and values are non-negative real numbers. The weight (and value) for a subset S⊆R are defined as the sum of all values (weights) of items in the subset:

v(S) = ∑_s∊Sv(s)
w(S) = ∑_s∊Sw(s)

Knapsack Goal: Find a subset S of R such that v(S) is maximized subject to the constraint w(S) ≤ W.

The fractional knapsack problem asks for a solution when fractional portions of items are allowed to be used. The 0-1 knapsack problem asks for a solution when items are indivisible so that an item may be chosen or not (1 or 0) but partial items are not allowed. In the fractional version, items might be, say, 3 pounds of rice, 1 pound of sugar, and 0.25 pounds of gold dust. In the 0-1 knapsack problem, items might be a car, a bicycle, and an oil painting (possibly by Rembrandt).

The fractional knapsack problem can be solved with a greedy algorithm. The greedy choice is to select the item with the highest value per unit weight and take as much of that item as possible (pretty much the definition of "greed"):

double KS_fractional (size_t n, size_t W, const fsu::Vector& v, const fsu::Vector& w)
V = vector of indices i, sorted by v[i]/w[i]
S = set of portions of items
w = 0 // current weight of selected items
while (w < W)
{
  b = V.Back(); // <-- greed applied here
  V.PopBack();
  if (w[b] < W)
  {
    add all of item b to S;
    w += w[b];
  }
  else
  {
    add (w[b] - W) of item b to S;
    w = W;
  }
  return total value of items in S
} 

The example of a coinage system for which a greedy change-making algorithm does not produce optimal change can be converted into a 0-1 knapsack problem that is not solved correctly by a greedy approach.

Exercise. Find the asymptotic runtime and runspace of the fractional knapsack algorithm and compare to those of the 0-1 knapsack algorithm.

4 Huffman Codes

(To be added)