The Self-Organizing Map and variants

 

Kristof Van Laerhoven

 

 

The Kohonen Self-Organizing Map
 

The Kohonen Self-Organizing Map (KSOM or just SOM) (1982) is an unsupervised neural network: this means it does not need any feedback from a teacher (or the environment as a teacher), unlike many other neural networks. It  is based upon earlier work of Willshaw and von der Malsburg (1976).

The KSOM usually has an output layer of interconnected neurons that are fully connected to the input layer, so every neuron from the output layer is connected to every neuron from the input layer. This connection has a certain value, which is just like in the domain of the supervised networks called a weight. Every neuron from the output layer has consequently as many weights as the neural network has inputs. The output neurons are furthermore ordered in a particular way, usually a two-dimensional grid, where each neuron has ‘neighbours’ (to the left, the right, up and down in the case of a grid). The KSOM was inspired by the way in which various human sensory impressions are topographically mapped into the neurons of the brain.  
 

Structure of a Kohonen SOM:

Each time an input is presented to the SOM, this input vector is compared to the weight vector of every neuron from the output layer. The neuron with the most similar weight vector is selected and is permitted to update its weights towards the values of the input vector.

If a similar input is presented afterwards, this neuron will be more likely to win again because it could update its weights while the weights of the other neurons remained unchanged. The formula to find a winner is consequently given by:

for all weights wij of the neurons on the position (i,j)

To introduce topology in the output layer, this algorithm is modified in such a way that the neighbouring neurons of the winner are also allowed to update their weights, but not as much as the winner itself:

where a is the learning rate and h depends on the distance to the winner.

This algorithm is responsible for creating and adapting neurons that are trained to trigger for a specific input: the values from the high dimensional input space are mapped to a discrete output space (the grid of neurons).

 

1. Initialize the weights from N inputs to M outputs neurons to small random values.
2. Present a new N dimensional input vector xi(t) to the neural network
3. Calculate the distances between this input vector and the weights wi(t) of every neuron, e.g. using the Euclidean distance.
4. Find the minimal distance and select the corresponding neuron as the winner
5. Update the weights of the winner towards the input: 

wI(t+1) = wi(t) + a .(xi(t)-wI(t)) with a the learning rate between 0 and 1.
6. Update the weights of the neighbours, depending on the distance to the winner: 

 

wI(t+1) = wi(t) + a .h (t).(xi(t)-wi(t)) with h Î ]0,1[ the neighbourhood function which decreases when the neurons get further away from the winner. The neighbourhood function has usually a bell-shaped curve. A common choice for this function is:

with ri and rc the coordinate vectors of the current neuron and the winning neuron.

The function plot is bell-shaped around rc with a width depending on s .
7. Go to step 2 

The algorithm to produce a Kohonen SOM.

 

  

The output layer of the KSOM after one epoch of training.

The output layer of the KSOM after two epochs of training.
 

The output layer of the KSOM after three epochs of training.

The output layer of the KSOM after four epochs of training.
 

The output layer of the KSOM after 5 epochs of training.

 

 

References & pointers

 

    
 

Compiled by Kristof Van Laerhoven. Last Update: 02/01/2002 15:16 -0000