Homework #4 – Linear
Discriminant Functions
CAP 5638, Pattern Recognition, Fall, 2005
Department of Computer Science,
¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾
Points: 100
Due: Thursday, October 27, 2005
Problem 1
(15 points) Problem 2 (Chapter 5 in the textbook). 
Problem 2 (15 points) Problem 4 (Chapter 5 in the textbook).
Problem 3 (20 points) Problem 5 (Chapter 5 in the textbook).
(a) Here we have three weight vectors. Note that a line joining the heads of two weight vectors defines the decision boundary between the two corresponding classes.
(b) In this case, when a constant vector c is added to each of the three weight vectors. Even though the three weight vectors change, the lines joining their heads remain unchanged and so the decision boundaries between the classes.
Problem 4 (20 points) Problem 6 (Chapter 5 in the textbook).
If the samples are totally linearly separable, by
definition for each class 
, there exists 
such that
if
and 
if 
(because any samples
in 
can be separated from
all others by a single hyperplane by definition). Now to show that the samples
are linearly separable, consider the following linear discriminant function for
each class ,
.
These discriminant functions form a linear machine that
classifies all of the samples correctly. To see this, suppose
, 
and 
, and thus 
. Thus 
is classified correctly by the constructed linear machine and
so all the samples are classified correctly.
To show the converse is not true, now suppose we add an
additional class 
, which consists of samples where 
, whose discriminant function is given by
.
Now that the problem is still linearly separable, if , we have
Now for a sample 
from , we have
But now the problem is not totally linearly separable
any more. Becauseis given by a set of linear discriminant functions, it forms
a convex region and thus any point inside that be written a linear combination
of vertices. Suppose that the samples from are totally linearly separable, which means that exists 
. Since all the vertices are on the decision boundaries, we have 
. Since any point inside is a linear combination, thus 
. It contradicts . Thus the samples from are not totally linearly separable.
Problem 5 (30 points) Computer Exercise 2 (Chapter 5 in the textbook).
Here is a Matlab implementation. Note that you need to argument and normalize the samples before you use Algorithm 3.
(a) It requires 24 iterations to converge. The following plot shows the data points as well as the obtained solution.
(b) In this case, it requires 17 iterations to converge.
(c) In the first case, many points are close to a potential decision boundary and thus more misclassified samples during the learning process. On the other hand, in the second case, only two samples from class 3 are close to a potential decision boundary and after 1 iteration, all except the two in class3 are classified correctly. Note that both problems are not “easy”.
Extra credit problems
Problem 6 (10 points) Problem 7 (Chapter 5 in the textbook).
(Based on a paper available at http://www.rpi.edu/~bennek/tr1127.ps).
In the given example, clearly it is pair wise linearly separable as we have 3 hyperplanes that separate the samples in a pairwise manner. However, this problem is not linearly separable.
To show that the problem is not linearly separable, we use Kesler’s construction to convert the original problem into a set of 18 inequalities. It can be that these inequalities can not be satisfied simultaneously and thus the problem is not linearly separable.
Problem 7 (10 points) Problem 9 (Chapter 5 in the textbook).
We have two sets
and
Suppose that the problem is linearly separable, then there exists a linear function g such that
and 
.
Then for any point x1
in the convex hull of the first set, we 
because
(since
)
Similarly, for any point x2
in the convex hull of the second set, we have 
. Therefore, the intersection of the convex hulls must be
empty. In other words, the problem must be either linearly separable or their
convex hulls intersect.