| Prev | Next | Start of Chapter | End of Chapter | Contents | Index | (3 out of 7)

Trainer


The Trainer block can train any neural network in NeurOn-Line. Attach one of its action links to a neural network, and attach the other action link to a data set. You cannot attach more than one neural network or data set to the block.

To configure the Trainer block, first, it must be connected to a neural network, then you can choose configure from the Trainer block's menu. The configuration panel contains different options depending on what type of neural network is connected to it. For more information, see "Configuring".

To evaluate the Trainer block, either pass it a control signal or choose evaluate from its menu. The Trainer trains the neural network with the data from the data set by adjusting the network's weights and other internal parameters. The Trainer Block does not modify the network's architecture or modify any data in the data set. You may evaluate either the attached neural network or data set during training. If you evaluate the neural network during training, the output is based on the weights before the training started.

To cancel training, choose Controls > Remote Process > Kill and then reset the block. The neural network's weights and other internal parameters are left unchanged and the Trainer outputs nothing.

When the Training block finishes evaluating, it passes a control signal on cp-out and a scalar value on dp-out. The scalar value tells you how well it trained. If you are training a Backpropagation, Autoassociative, or Radial Basis Function Network, it passes the square root of the mean of the squared errors for all the training data. If you are training a Rho Network, it passes the negative mean of the logarithms of the predicted probabilities for all training data. In both cases, lower numbers mean a closer fit of the training data.

Watching the Training Happen

Sometimes you may want to watch the Trainer's progress, especially for long training runs. When you select Enable RPC from the Remote Procedure menu in the Print Configuration Tools menu bar, the Trainer displays an output of its progress. If you launched the NeurOn-Line remote procedure yourself, the table appears in the window where you launched it. If you let NeurOn-Line launch the remote procedure automatically, the table appears in the window where you launched G2.

Here is a sample of the output for Backpropagation Network training:

The first column is how many passes, or gradient calculations, have happened so far. Each pass represents a change of direction as the Trainer searches for the best weights.

The second column is how many times the Trainer needed to evaluate the objective function in that pass. Each evaluation of the objective function represents one "step," and the total is bounded by the maximum iterations (in this case 40). Typically, the trainer takes 3 to 5 steps in a given direction before it calculates a new gradient.

The third column is the value of the least-squares objective function. The lower the value, the closer the Trainer is to a good fit. Typically, when the training begins, the objective decreases greatly with each pass (or gradient). As the training comes to an end, the object decreases much more slowly.

The last column is the training method used on this pass (or gradient). This is usually the method you specified in the configuration panel. However, the trainer may use steepest descent from time to time to accelerate the training.

Configuring

The configuration of the trainer depends on the type of the network being trained. To configure the trainer, it must first be connected to a specific network using an action link. The configure menu option will then bring up a configuration panel appropriate to the type of network to be trained.

This is the configuration panel you see when you are training a Backpropagation or Autoassociative Network.


The following headings describe how to configure the block.

Choosing the Maximum Number of Training Iterations

The Trainer improves the weights in a number of individual steps. To specify the maximum number of steps, enter a number in the Maximum Iterations attribute. Typically, values range from 50 to as much as 1000. However, you will usually want to use several short training runs so you can monitor the Trainer's progress. When you evaluate the Trainer again, it continues the training from where it stopped.

Choosing the Training Method

To choose how to train the network, select one of the options below Training Method. The options fall into two main categories:

In general, the Second Order options are more powerful and use significantly more memory, and the Conjugate Gradient options use less memory and take less time per step. The Conjugate Gradient options are generally better for larger networks (over 100 weights), and the Second Order options are better for smaller networks.

Once you choose which category of training methods to use, you may want to experiment with the different methods in each category to find out which is best for your network.

Note: Some neural network packages allow the user to choose fixed or scheduled "learning rates" and "momentum factors." NeurOn-Line automatically optimizes the training parameters and does not require the user to do this. The parameters that NeurOn-Line chooses will always be better than those chosen by hand.

Choosing Whether to Accelerate Training

 The Accelerate Training by Input Projection? option can speed up the training time. Set it to yes if your input data has columns that might be correlated.

To speed up training, the block projects your input data vector to a vector with fewer dimensions, trains with the smaller vector, and projects the smaller vector and its training results backwards to obtain results useful for the original vector. In general, this option is recommended if you have more than ten inputs.

Configuring the Trainer for a Radial Basis Function Network

This is the configuration panel you see when you are training a Radial Basis Function Network. When you have selected your option, click the Done button.


Choose an option for Clustering Method. The option you choose depends on what kind of problem you are trying to solve:

These methods differ in how they assign locations for unit centers. When Regular clustering assigns locations, it uses all the data in the data set simultaneously. When class-separate clustering assigns locations, it goes through all the members of one class before going through the members of another. This method prevents centers from being placed near the boundaries between classes, where they do not help to discriminate between the classes.

During training, NOL finds and outputs to the background window the cluster centers as shown in this matrix. Each row represents the coordinates of a cluster center. Up to five centers are shown. Each column represents the coordinates for a dimension. 

Matrix In cluster, now m =       size =  14 by 10

	1	2	3	4	5	6	7	8    

------------------------------------------------------------------------

1	|0.565	0.738	0.374	0.38	0.415	0.257	0.372	0.345

2	|0.536	0.532	0.536	0.83	0.509	0.482	0.275	0.356

3	|0.603	0.687	0.328	0.243	0.394	0.287	0.604

4	|0.573	0.446	0.337	0.363	0.749	0.656	0.437	0.777

5	|0.675	0.774	0.391	0.328	0.3	0.419	0.685	0.399
If the Unit Overlap attribute is automatic, NOL then determines the optimum unit widths by searching for the value of p that minimizes the training error, with the unit centers previously determined by k-means clustering during the first stage of the training, as shown in the following output: 

Optimizing overlap parameter

Trying p = 1.000000, objective = 19.973553

Trying p = 2.000000, objective = 18.852538

Trying p = 3.618034, objective = 17.939625

Trying p = 4.135448, objective = 17.873745

Trying p = 4.972642, objective = 18.574853

Trying p = 4.455228, objective = 17.979710

Trying p = 3.937814, objective = 17.871113

Trying p = 4.025800, objective = 17.867350

Trying p = 3.985542, objective = 17.868146

Trying p = 4.066058, objective = 17.868195

Optimized overlap with p = 4.025800

Final objective function value = 17.665130

Configuring the Trainer for a Rho Network

 This is the configuration panel you see when you are training a Rho Network. When you have selected your option, click the OK button


Which option you choose depends on what kind of problem you are trying to solve:

Example

 Below is a simple diagram for training a network. To train this network, configure the network architecture and the trainer, then choose evaluate from the Training block's menu.


See Also

For more information on how to use this block, see the pages below.

Click here for more information...
Basic Block Behavior
"Neural Network Blocks" Chapter
Data Set
Fit Tester
Train and Test
Five Fold CV

| Prev | Next | Start of Chapter | End of Chapter | Contents | Index | (3 out of 7)

Copyright © 1996, Gensym Corporation, Inc.