Hopfield Neural Networks Practical Artificial Intelligence Programming With Java

very different than back propagation networks covered later in Section 7.4 because the training data only contains input examples unlike back propagation networks that are trained to associate desired output patterns with input patterns. Internally, the operation of Hopfield neural networks is very different than back propagation networks. We use Hopfield neural networks to introduce the subject of neural nets because they are very easy to simulate with a program, and they can also be very useful in practical applications. The inputs to Hopfield networks can be any dimensionality. Hopfield networks are often shown as having a two-dimensional input field and are demonstrated recogniz- ing characters, pictures of faces, etc. However, we will lose no generality by imple- menting a Hopfield neural network toolkit with one-dimensional inputs because a two-dimensional image can be represented by an equivalent one-dimensional array. How do Hopfield networks work? A simple analogy will help. The trained connec- tion weights in a neural network represent a high dimensional space. This space is folded and convoluted with local minima representing areas around training input patterns. For a moment, visualize this very high dimensional space as just being the three dimensional space inside a room. The floor of this room is a convoluted and curved surface. If you pick up a basketball and bounce it around the room, it will settle at a low point in this curved and convoluted floor. Now, consider that the space of input values is a two-dimensional grid a foot above the floor. For any new input, that is equivalent to a point defined in horizontal coordinates; if we drop our basketball from a position above an input grid point, the basketball will tend to roll down hill into local gravitational minima. The shape of the curved and convoluted floor is a calculated function of a set of training input vectors. After the “floor has been trained” with a set of input vectors, then the operation of dropping the basket- ball from an input grid point is equivalent to mapping a new input into the training example that is closest to this new input using a neural network. A common technique in training and using neural networks is to add noise to training data and weights. In the basketball analogy, this is equivalent to “shaking the room” so that the basketball finds a good minima to settle into, and not a non-optimal local minima. We use this technique later when implementing back propagation networks. The weights of back propagation networks are also best visualized as defining a very high dimensional space with a manifold that is very convoluted near areas of local minima. These local minima are centered near the coordinates defined by each input vector.

7.2 Java Classes for Hopfield Neural Networks

The Hopfield neural network model is defined in the file Hopfield.java. Since this file only contains about 65 lines of code, we will look at the code and discuss the 111 algorithms for storing and recall of patterns at the same time. In a Hopfield neural network simulation, every neuron is connected to every other neuron. Consider a pair of neurons indexed by i and j. There is a weight W i,j between these neurons that corresponds in the code to the array element weight [i, j]. We can define energy between the associations of these two neurons as: energy [i, j] = −weight[i, j] ∗ activation[i] ∗ activation[j] In the Hopfield neural network simulator, we store activations i.e., the input values as floating point numbers that get clamped in value to -1 for off or +1 for on. In the energy equation, we consider an activation that is not clamped to a value of one to be zero. This energy is like “gravitational energy potential” using a basketball court analogy: think of a basketball court with an overlaid 2D grid, different grid cells on the floor are at different heights representing energy levels and as you throw a basketball on the court, the ball naturally bounces around and finally stops in a location near to the place you threw the ball, in a low grid cell in the floor – that is, it settles in a locally low energy level. Hopfield networks function in much the same way: when shown a pattern, the network attempts to settle in a local minimum energy point as defined by a previously seen training example. When training a network with a new input, we are looking for a low energy point near the new input vector. The total energy is a sum of the above equation over all i,j. The class constructor allocates storage for input values, temporary storage, and a two-dimensional array to store weights: public Hopfieldint numInputs { this.numInputs = numInputs; weights = new float[numInputs][numInputs]; inputCells = new float[numInputs]; tempStorage = new float[numInputs]; } Remember that this model is general purpose: multi-dimensional inputs can be con- verted to an equivalent one-dimensional array. The method addT rainingData is used to store an input data array for later training. All input values get clamped to an “off” or “on” value by the utility method adjustInput. The utility method truncate truncates floating-point values to an integer value. The utility method deltaEnergy has one argument: an index into the input vector. The class variable tempStorage is set during training to be the sum of a row of trained weights. So, the method deltaEnergy returns a measure of the energy difference between the input vector in the current input cells and the training input examples: private float deltaEnergyint index { 112