Neural Networks - Basics

What are neural networks?

Neural networks are algorithms that get input data and adapt parameters on an internal (unvisible) model so it works well on the known data and (hopefully) delivers good results on new data.
Neural networks are not new, the first approaches have been introduced already in 1943 by Warren McCulloch and Walter Pitts under the name of "Threshold Logic" models. In the '80s the effective backpropagation algorithm was introduced which spread the usage of neuronal networks. But still large scaled applications have not been used as they required huge datasets and storage. This is why they got popular in the last 10 years, when those parameters do not provide any difficulties anymore.

How do neural networks look like?

If you see pictures of neural networks, they will usually look like this:

The round forms are called nodes, they are ordened in different colums called layers.
- the first layer (on the left) is called the input layer. It determines the number of input parameters.
- the last layer (on the right) is called the output layer which holds the calculated results of the algorithm.
- the columns in the middle are called hidden layers (unvisible from outside).

For a neural network you have the input x (here x is a vector with two components) that are passed to the input layer, mapped through the network and create an output (here a vector with two components), that in the ideal case is a good approximation to the known output y. During the way through the network, every input component is multiplied with parameters and enriched with some constant bias, the composition is then passed to a node, in which the activation is applied.
From the error of the prediction with the choosen parameters the algorithm can calculate deltas and adapt each parameter, so that in the next run the prediction will get closer to the actual known result.
So the algorithm learns by minimizing the error.

An example for supervised learning is the recognition of handwriting: by feeding the system with lots of letter sets of different hand writing styles and providing the correct result the system can find substructures and compare them to a new dataset. Then it can calculate probabilities that the input data matches one of the known letters and propose a solution. This often cited example is used as introducting program in Google's Machine Learning Platform Tensorflow, which is a very powerful tool for the interested community.