Programming Assignment -- Backpropagation Algorithm

CAP 5615, Artificial Neural Networks, Fall 2004

Department of Computer Science, Florida State University

¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾¾

Points: 100

Due: Week 10, Monday, October 25, 2004

 

Background: Computer simulations play a critical role in the study of neural networks and in applications of using neural networks to solve real-world problems. Due to the nonlinearity and complexity of neural networks, many aspects of neural networks can be understood only by computer simulations and experiments.

 

Purpose: In this programming assignment, you need to implement a set of programs 1) to train a multiple–layer perceptron neural network using the backpropagation algorithm, 2) to test the trained neural network on a test set.  Then you need to test programs on the parity problem (a generalization of XOR problem) and a function approximation problem and study and analyze how the performance is affected by neural network architectures and learning rule parameters.

 

Assignment: Implement a multiple-layer perceptron neural network with the backpropagation learning algorithm. Your implementation needs to be flexible enough to implement any reasonable sized multiple-layer perceptron. In other words, an implementation that assumes a fixed number of inputs, a fixed number of outputs, or fixed architecture is not sufficient. To be more specific, we assume that the network architecture will be read from a parameter file (i.e., how many layers, which transfer function for each layer, parameters for the backpropagation algorithm; you need to design your own parameter file format). Additionally, you will also need to read training examples and test examples that are given in files respectively. You can use any programming language for implementation.

 

 

After the programming is done, apply your programs on the following two problems.

1)     Parity problem:  Here the inputs consist of a fixed number of binary inputs and the desired output is 1 if the input pattern contains an odd number of 1’s and 0 otherwise. The learning procedure is stopped when the maximum error of 0.1 is reached for all the input patterns or a maximum number of iterations is reached. Here the training set consists of all the valid inputs with their desired output. You need to apply your program to parity problems with two inputs, four inputs, and eight inputs.

2)     Function approximation: Here use your program to approximate, where i takes on values of two, four, and eight. Here the training set consists of the function sampled at points p=-2, -1.6, -1.2, -0.8, -0.4, 0, 0.4, 0.8, 1.2, 1.6, and 2.0. After training, plot your approximation along with the original function.

 

For each problem, report two best but different neural networks (in terms of performance) you have obtained. Analyze and explain your findings. You may need to vary the learning rate, momentum and other learning parameters, and neural network architectures (see some examples in the book pp. 11-17 – 11-11-23).

 

Milestones: To make this assignment more manageable and help you avoid problems associated with the last minute approach, you need to achieve the following milestones at the specified dates.

 

  • Week 5, Sep. 20   - Choose programming language, design parameter file format, training and test example file format, neural network weight file format, and implement functions to read/write these files.
  • Week 7, Oct. 4 – Implement multiple layer perceptron architecture (dynamic allocation of neurons, layers, and necessary weights and biases) and feedforward calculation for a given network.
  • Week 8, Oct. 11 – Implement backpropagation algorithm.
  • Week 9, Oct. 18 – Apply your network on the two problems using different architectures and different parameters.
  • Week 10, Oct. 25 – Document your results, summarize your results, your findings and your analysis, and generate a report, and then submit your report along with source code and your results.

 

Submission:

 

  • Project report – You need to turn in a report, summarizing your results, your findings and your analysis based on your simulations.
    • In addition, you also need to submit a one-page brief summary along with your source code at each specified milestone.
  • Source code – You should attach all the source programs you developed for this lab.
  • Test cases – Typical running results of your implementation for the two problems that sufficiently demonstrate that your program works properly.

Hardcopy is required for submission.

 

Extra Credit (15 %): Please state clearly in your report if you have solved the following extra credit option.

 

Wine classification: Apply your network to the wine classification problem (ftp://ftp.ics.uci.edu/pub/machine-learning-databases/wine/, three classes: class 1, 59 data points; class 2, 71 data points; and class 3, 48 data points; each data point is a vector of 13 attributes) using the leave-one-out training/test method. That is, for each data point, train a neural network using the rest as training data and then classify the chosen data point using the trained network. The classification performance is measured by the percentage of data points that are classified correctly. You need to report your performance and compare yours with those given in the description of the dataset (100%, 99.4%, 98.9%, and 96.1% depending on the methods).

 

Grading

 

  • Report – 30 points
  • Correct implementation – 40 points
  • Results – 30 points
  • Wine classification extra credit option – 15 points

 

Additional Information

 

Please refer to Chapters 12 and 13 in the textbook for the backpropagation algorithm. You may find the paper by Wang (IEEE Experts, 1993) helpful.

 

The wine dataset is available at ftp://ftp.ics.uci.edu/pub/machine-learning-databases/wine/ as part of the UCI Machine Learning Repository (http://www.ics.uci.edu/~mlearn/MLSummary.html). You can also obtain the dataset from the class web page.