Neural Network Types.

Types of Neural Networks.

Hi guys!!!

Welcome all. In this article we are going to talk about 3 major types of Neural Networks found in Deep Learning.

But “What is Deep Learning?”, “What are Neural Networks?” …...Once you go through this article, you will get answers for these questions. Let’s begin.

Artificial Intelligence (AI) which is a most popular topic nowadays. AI is a computer science field which enables machines to imitate behavior of humans. Machine Learning (ML) is a subset of AI. ML consists of algorithms trained with data. Therefore, AI can be achieved through these ML algorithms. ML is all time training based. As an example we can build a robot who can differentiate fruits in a fruit basket. With respect to ML we have to tell the machine the features that can be used in this process of differentiating fruits. Now you may wonder about another way to do the same task without human intervention. At that stage Deep Learning (DL) comes to play. DL is again a subset of ML and based on Neural Networks (NNs). In DL, features that needed in differentiating the fruits are picked by the NN in the absence of human intervention.

Neural Network (NN) types…

A Neural network is very much similar to a biological human brain. Therefore, it can be stated that Deep Learning is inspired by human brain. Numerous neural network types are available in DL.

• Feed-forward NNs
• Radial Basis Function NNs
• Convolutional NNs
• Recurrent NNs
• Modular NNs

Here my intention is to describe you about 3 main types of NNs namely Feed-forward NNs, Convolutional NNs and Recurrent NNs.

Let me explain these NN types one by one.

Feed-forward Neural Networks.

Feed-forward Neural Network is made of neurons which are core processing units of the network. This NN is mainly composed of 3 main layers. They are,

• Input layer
• Hidden layers
• Output layer

Input layer is responsible for taking inputs. Between the input layer and the output layer the hidden layers are situated. There can be more than one hidden layers in a feed-forward neural network. When number of hidden layers increase, the accuracy of prediction is also getting higher. Most of the computation required by network happens in the hidden layer.

Let’s see what functions are taking place inside these layers during an image based classification process in detail.

• Image is made of pixels. When the image is given as the input to the network, input image is broken in to pixels and each pixel is taken as input to each neuron of the input layer.

• Neurons of one layer are connected to neurons of the next layer via channels. 
• Each of these channels is attached with a unique numerical value called 'weight'. 
• The inputs are multiplied to the corresponding weights and their sum is sent as input to the neurons in the hidden layer. 
• Each of these neurons is associated with a numerical value called bias. 
• Then the bias is added to the input sum. 
• After that the result obtained is passed through a threshold function named ‘Activation Function’. The output of the activation function decides whether the neurons of the next layer should get activated or not. 
• The activated neurons by the activated function propagate data through the network via connected channels. 
• The same process will occur over and over again until reaching the second last layer of the network.

This process can be termed as ‘Forward Propagation’.

• In the output layer the neuron with the highest value determines the output.

Convolutional Neural Network (CNN).

Moving to CNN it is a multi-layer neural network with a special architecture used for deep learning. A CNN architecture is mainly composed of 4 layers: convolutional layer, pooling layer, ReLU layer and fully-connected layer. CNN has been significantly used in image based classification as it removes the necessity for feature extraction by using image processing tools. 

Let’s talk about topology of CNN in detail by discussing the functionality of each layer of this network.

Input Layer.

Input images are fed to input layer.

Convolutional Layer.

Primary task of convolutional layer is to extract significant features from the image input. Here every image is considered as a matrix of pixel values.  

As shown in the above picture, ‘I’ is the pixel values of the input image as a matrix. ‘K’ is known as several names such as ‘Filter’, ‘Kernel’, ‘Feature detector’. The kernel slides over the pixel matrix of the image. Here the size of the kernel is 3x3 and corresponding 3x3 section is selected from I at a time by sliding. Then 2 corresponding values in same position of the 2 matrices are multiplied as shown above. Then those multiplied values are added together.

 [(1x1) +(0x0) +(0x1) +(1x0) +(1x1) +(0x0) +(1x1) +(1x0) +(1x1)]

The resultant value (=4) can be seen in the ‘feature map’. 
When number of filters increase, more image features can be extracted.

ReLU Layer.

This layer introduces non-linearity in CNN.

Pooling Layer.

Pooling layer acts further as a feature extractor. It reduces the dimensions of feature map and retain most significant features of the image needed for the image classification.

Advantages of pooling are, it makes the feature dimension smaller and more manageable. Therefore, this is also known as sub sampling.

There are 3 types of pooling known as,

•       max pooling - picks the maximum value from the selected region.
•       average pooling -  picks the average value from the selected region.
•       min pooling - picks the minimum value from the selected region.

Max pooling is the most commonly used pooling in most of the cases.

Fully Connected Layer (FCL).

Every neuron in the previous layer is connected to every neuron in the next layer. Therefore, it is known as fully connected layer. Output from convolutional and pooling layer is high level features of input image.  FCL uses these so far extracted high level features to classify the input images in to various classes.

Output Layer.

Output Layer is responsible for producing Output.

Recurrent Neural Network (RNN).

When we talk about structure of RNN, a looping constraint on the hidden layer of feed-forward neural network turns to RNN.

RNN also consists of 3 layers. Input layer receives input. In hidden layers activation functions are processed after sum of product of weight and features are calculated just like in feed-forward NN. Then output layer receives the output.

Let me confuse you little bit. “about networks know I to neural need more….”. Did you understand what I meant by this sentence. I don’t think so. Now try this one. “I need to know more about neural networks….”. Jumbling a sentence doesn’t make any sense to the reader. If the human brain fails in such a situation; can you imagine about the response of a neural network in a situation where sequence is disturbed. Recurrent Neural Networks come to play in these kind of situations. We can use recurrent neural networks to solve the problems related to sequence data. Therefore, RNNs works better in Natural Language Processing applications such as text-to-speech, language translation etc. In such occasions process of recurrent layers begin.

So a neuron in RNN functions in this manner because of it's capability to store the information of previous time-step and merges with present input thereby ensuring some connection between present input and the earlier input. Therefore, It can be stated that each node of RNN is a memory cell which performs both tasks; remembering as well as computing.

Other applications of RNNs. 

Sentiment classification.

Image captioning.

We are done with this article. Hope you got a basic idea about 3 major types of NNs found in deep learning namely Feed-forward NNs, Convolutional NNs, Recurrent NNs.

Till we come up with new article, stay safe and cool…….