Artificial Neural Networks (ANNs) are what makes up the fabric of high level Machine Learning systems, enabling ML and Artificial Intelligence systems to learn. They themselves are composed of Artificial Neurons that are grouped in layers and are linked to each other in such as way as to model the human brain with inputs, weighted outputs and variable influence on other neurons.
Suggested supplementary reading: What is Artificial Intelligence?
Suggested supplementary reading: What is Machine Learning?
Suggested supplementary reading: What is Deep Learning?
Artificial Neural Networks are the high level workhorses of Artificial Intelligence that facilitate Machine Learning in systems that aim to model how the human brain works. ANNs are composed of layers of mathematical representations/models of individual human brain neurons that are interconnected and interact, and in doing so learn how to understand the data supplied to them via the implementation of processes and algorithms known as forward and back propagation, activation and differentiation, which have the net result of changing the influence of each individual neuron in the ANN - making the ANN learn through adaptation.
Artificial Neural Networks are made up of layers:
The input layer of an ANN is where data is fed into the ANN. The input layer is one neuron or node thick (there is only ever one input layer per ANN), however can contain an infinite number of input nodes on the vertical plane (or within it). For example, if we tasked an ANN to learn from website user statistics we may have about 50 data points or fields to learn from, and each of these would make up an individual input. On the other hand we may have an ANN that is learning how to recognise facial expressions from images and each input might be a single pixel, or group of pixels and therefore can be many thousands in number. The input layer only ever talks to downstream layers, there is no cross talk within the input layer. In fact, there is no actual processing in the Input layer at all, it is simply an input interface to the rest of the ANN, but is key in dictating the input size and ultimately the structure of our Artificial Neural Network.
The hidden layer of an Artificial Neural Network is where the majority of the processing and learning happens. It is composed of layers of Artificial Neurons. Each layer can vary in the number of individual neurons, and all upstream hidden layer neurons connect with all neurons in the next hidden layer. Weighted (random to start with) numerical input data/values are received from the input layer and fed into each first hidden layer neuron where a process called activation takes place. Activation is a threshold calculation that decides if the combined inputs to a particular hidden layer neuron are sufficient to trigger an output. The activation threshold is the same in every neuron in the network. This process of summation and activation continues left to right through the hidden layer meaning that each neuron contains a different output value, which is important for learning.
Immediately after the last hidden layer is the output layer, which unsurprisingly is where the ANN outputs its final result(s). Like the input layer, the output layer is only one neuron/node thick but can contain multiple neuron outputs within that layer. For example if you had some known outputs of "Bad customer" or "Good customer" the output layer would comprise of 2 output neurons. Unlike the input layer, activation occurs within the output layer before churning out the final result.
Described above is the structure of an Artificial Neural Network and how the flow of data goes left to right, input to hidden to output layer using weighted summation and activation at each neuron, ultimately leading to an output value. Remember, the goal of the ANN is to achieve an output based on an input - it has to learn how to get the known output target value from the provided input(s). When the ANN goes left to right for the first time, weights are random, and the calculated output is understandably nowhere near the target output (know as loss). This first attempt or first pass is called Forward Propagation. What needs to happen now is the ANN has to adjust to get closer to the target output value. It does this by adjusting all the weights in the ANN (which started off randomly for this very reason). This is called Back Propagation.
Back Propagation is the process of an ANN systematically going backwards (output to hidden to input layer) over the ANN to adjust each weight within the system to achieve a single goal of being able for the next round of Forward Propagation to get closer to the target output. The maths behind this is fairly complex and out of the scope of this overview however it involves working out the derivative with respect to activation of all neurons or nodes in the ANN. This forward and back propagation happens many thousands of times (each cycle is called an epoch) until the ANN achieves the expected target output.
Activation occurs before the output of each hidden layer and output layer neuron and as stated above it works based on a threshold value. E.g. if a certain value is higher than X, then give an output of Y. The calculation or formula used for activation is consistent throughout the ANN however many different ones can be used. The type of activation formulae used is one of the key ways to optimise ANN learning efficiency.
Weights were mentioned a few times above. Weights are values that reside between the output from each neuron in the ANN and modify the input going into each hidden and output layer neuron by simply multiplying output by weight. It is the weights that change as the ANN learns and are fundamentally how AI and Machine Learning is achieved. When AI/ML training has finished, it is the weights that are stored and used. The weights are literally the manifestation of the learning process of an ANN and how the output is achieved from set of inputs.
Biases are also commonly used in ANN and if in place act as a positive reinforcer on all neurons in all layers - ensuring there is always an input value to start with. Biases remain constant in value however they have weights that change during the learning process. Biases have been shown to positively influence the learning process.
Artificial Neural Networks have a single job to do, which they do exceptionally - and that is to learn how to get from a set of inputs to a set of outputs. However, of course, the data they handle can vary massively in size and complexity, and this ultimately must influence the size and shape of our ANN.
The input layer of an ANN is directly related to the number of input fields, and the same is true of the output layer, so these are easy to define in terms of size. However, the hidden layer's composition - which can contain as many layers and as many neurons within each layer as we want - needs a bit of thought.
It is a safe bet to assume that the more complex the input data and/or learning task, the greater the requirement for a more robust hidden layer section e.g. if you are carrying out image processing with 10,000 inputs, clearly a hidden layer 2 thick with 3 neurons in each layer is not going to be sufficient.
The honest answer is, you simply do not know when you first start with a new ANN the optimal make-up of the hidden layer(s) is. You therefore have to go through a period of trial and error, optimisation and scaling up.
The total number of neuron to neuron connections has an impact on processing power requirements, and in an Artificial Neural Network it is calculated by multiplying the number of neurons in each layer by each other. E.g. if you have an input layer of 10 neurons, 3 hidden layers of 10 neurons each, and an output layer of 2, you would have 10 x 10 x 10 x 10 x 2 = 20,000 neuron connections. For image recognition systems you can easily end up with millions of artificial neuronal connections.
Fennaio AI and ML software and systems contain ANN automatic optimisation helpers as standard.
Fennaio designs, develops and provides Artificial Neural Network software than can be used for all Machine Learning and Artificial Intelligence systems.
We have available off-the-shelf turnkey Artificial Neural Network/Machine Learning/Deep Learning AI software and can create bespoke software and systems solutions where required.
Please get in touch with us to discuss your Artificial Neural Network requirements.
Thanks for reading about Artificial Neural Networks, there's more to learn below:
Whether you are starting out on your first AI project, just interested in the possibilities of AI or are wanting to expand your existing AI suite, we are here to help.
We will discuss with you where you are, where you want to be, and how we can achieve it with AI - whether by a bespoke solution or using one of our off-the-shelf products
We will work with you to gather, analyse and prepare all your relevant data sources for use in the AI system(s)
We will run and tune the AI throughout the AI learning process and enable the AI to produce a real time visual output to confirm the AI is producing beneficial results
When you are satisfied the AI is delivering the results you desire, we will integrate the AI with your new or existing systems
Fennaio has the expertise to get you up and running with AI, Machine Learning, Deep Learning and Data Science in your new or existing systems, software and operations.
Get Started