“Transfer Learning” is the process of using a model that was trained to complete one task (ex: Identify Cats), and training it further to complete a similar, but different task (ex: Identify Cancer).

This allows us to harness the foundational knowledge learned during “pre-training”, such as identifying lines and shapes in image recognition.

This is especially powerful when we have significantly more data available for pre-training than we have for the task we are interested in (Cat vs Cancer Images).

Additionally, complex models take a very long time to train.  Transfer Learning allows us to skip ahead and only use our time to “fine-tune” the model, making huge and complex models tractable for the everyday user.

In practice, we load the model and all of its parameters, but replace the “output layer” with a new one for our task.

We can also choose to “freeze” specific layers in the pre-trained model, meaning they will not be updated during training. 

Choosing which layers to unfreeze varies based on the model, the problem, and the amount of data available for fine-tuning.

Overall, Transfer Learning is an extremely powerful tool, particularly when faced with limited data for a specific task.