Back Propagation Neural Networks
Input 1
Input 2
Output 1
Output 2
Input 1
Input 2
W1,1 W2,1
W1,2 W2,2
Output 1
Output 2
W1,1 W1,2 W2,2
W2,1
Input neuron layer Input neuron layer
Output neuron layer Output
neuron layer
Figure 7.2: Two views of the same two-layer neural network; the view on the right shows the connection weights between the input and output layers as a
two-dimensional array.
am using the network in Figure 7.2 just to demonstrate layer to layer connections through a weights array.
To calculate the activation of the first output neuron O 1, we evaluate the sum of the
products of the input neurons times the appropriate weight values; this sum is input to a Sigmoid activation function see Figure 7.3 and the result is the new activation
value for O 1. Here is the formula for the simple network in Figure 7.2:
O 1 = SigmoidI1 ∗ W [1, 1] + I2 ∗ W [2, 1]
O 2 = SigmoidI2 ∗ W [1, 2] + I2 ∗ W [2, 2]
Figure 7.3 shows a plot of the Sigmoid function and the derivative of the sigmoid function SigmoidP . We will use the derivative of the Sigmoid function when
training a neural network with at least one hidden neuron layer with classified data examples.
A neural network like the one seen in Figure 7.2 is trained by using a set of training data. For back propagation networks, training data consists of matched sets of input
with matching desired output values. We want to train a network to not only produce the same outputs for training data inputs as appear in the training data, but also to
generalize its pattern matching ability based on the training data to be able to match test patterns that are similar to training input patterns. A key here is to balance
the size of the network against how much information it must hold. A common mistake when using back-prop networks is to use too large a network: a network
that contains too many neurons and connections will simply memorize the training
117
Figure 7.3: Sigmoid and derivative of the Sigmoid SigmoidP functions. This plot was produced by the file src-neural-networksGraph.java.
examples, including any noise in the training data. However, if we use a smaller number of neurons with a very large number of training data examples, then we
force the network to generalize, ignoring noise in the training data and learning to recognize important traits in input data while ignoring statistical noise.
How do we train a back propagation neural network given that we have a good training data set? The algorithm is quite easy; we will now walk through the simple
case of a two-layer network like the one in Figure 7.2, and later in Section 7.5 we will review the algorithm in more detail when we have either one or two hidden
neuron layers between the input and output layers.
In order to train the network in Figure 7.2, we repeat the following learning cycle several times:
1. Zero out temporary arrays for holding the error at each neuron. The error, starting at the output layer, is the difference between the output value for a
specific output layer neuron and the calculated value from setting the input layer neuron’s activation values to the input values in the current training ex-
ample, and letting activation spread through the network.
2. Update the weight W
i,j
where i is the index of an input neuron, and j is the index of an output neuron using the formula W
i,j
+ = learning rate ∗ output error
j
∗I
i
learning rate is a tunable parameter and output error
j
was calculated in step 1, and I
i
is the activation of input neuron at index i. This process is continued to either a maximum number of learning cycles or until the
calculated output errors get very small. We will see later that the algorithm is similar but slightly more complicated, when we have hidden neuron layers; the difference is
that we will “back propagate” output errors to the hidden layers in order to estimate errors for hidden neurons. We will cover more on this later. This type of neural
118
Figure 7.4: Capabilities of zero, one, and two hidden neuron layer neural networks. The grayed areas depict one of two possible output values based on two
input neuron activation values. Note that this is a two-dimensional case for visualization purposes; if a network had ten input neurons instead of
two, then these plots would have to be ten-dimensional instead of two- dimensional.
network is too simple to solve very many interesting problems, and in practical applications we almost always use either one additional hidden neuron layer or two
additional hidden neuron layers. Figure 7.4 shows the types of problems that can be solved by zero hidden layer, one hidden layer, and two hidden layer networks.