What is transfer learning in neural networks:



Transfer learning is a concept in machine learning where, instead of training a new neural network from scratch, a new network is trained from a pre-trained network with a different task. This benefits model as it enables them to start from an existing state rather than having to learn all over again from scratch. Transfer learning can come in handy when one discovers inadequate training data for a particular task.

For a specific task in deep learning, workhorses, the neural networks often need a large-scale data set to generalize. For instance, an image classification model is trained using ImageNet, which has over one million images shared in 1000 categories. This is practically impossible in most applications; it would be almost impossible to gather enough data to train a model of that size from scratch. This is where transfer learning comes in handy: if one takes a model pre-trained on Imagenet, removing the final output classification layer and retraining on the small target dataset containing only the pertinent output categories.



There are generally two ways transfer learning is applied in neural networks:

There are generally two ways transfer learning is applied in neural networks:



1. Feature extraction:

Build upon the representations that a pre-trained model has acquired to create features on fresh samples. In general, the pre-trained neural network operates more or less as a feature extractor. These features are then used to train a new model for the target task, which is the goal of most methods in this line of work.

2. Fine-tuning:

Apart from extracting features, the pre-trained model is also fine-tuned on the new data during the process. This means that several upper layers of the model undergo their retraining in conjunction with the original output layers. The learning rates are small, so the weights in the original network change slowly, and the new layers added have significant changes frequently.

One significant advantage of transfer learning is learning the feature representation of the target task with a small amount of data, less computing power, and quicker model creation. In the case of computer vision, deep learning models, including VGG, ResNet, and Inception, are used for transfer learning since they are pre-trained for image classification on the ImageNet dataset. There are also some libraries, such as Keras, through which one can easily import or modify the pre-trained models. In general, using transfer learning is beneficial to deep learning since it tries to enhance the deep learning model by incorporating some existing knowledge embedded in the weights of a neural network.

Thanks for reading my article.