For most of the first decade of modern machine learning, building a useful AI model meant gathering a massive labeled dataset, renting a cluster of GPUs, and waiting days or weeks for training to converge -- then doing it again if performance was unsatisfactory. This made high-quality AI the exclusive domain of companies with research budgets in the tens of millions.
Transfer learning changed that equation. By allowing models to carry knowledge from one task into another, it reduced the data and compute required to reach state-of-the-art performance by orders of magnitude. It is the reason a startup with a few thousand labeled images can build a medical imaging classifier that rivals systems built by large institutions. It is also why large language models like GPT-4 can answer questions about legal contracts after being trained primarily on general web text.
This article explains what transfer learning is, how it works technically, what its most important applications are, and why it remains central to the current AI landscape.
The Core Idea: Knowledge Does Not Have to Be Learned Twice
What transfer learning means
Transfer learning is a machine learning technique in which a model trained to perform one task is reused -- in whole or in part -- as the starting point for training a model on a different task. The knowledge the model has already acquired, encoded in its weights, serves as a prior that guides learning on the new problem.
The analogy to human cognition is straightforward. A person who already speaks French will learn Spanish faster than someone who speaks only Mandarin, because French and Spanish share vocabulary, grammar structures, and phonological patterns. The prior knowledge transfers. AI models can exploit a similar dynamic.
More formally, transfer learning involves a source domain (where the model was originally trained), a source task (what it was trained to do), a target domain (where the model will be applied), and a target task (what it needs to do in deployment). The central question is: which knowledge encoded during training on the source task is useful for the target task, and how should it be transferred?
Why it works mathematically
Deep neural networks learn hierarchical representations. In image models, the early layers detect low-level features -- edges, corners, gradients -- while deeper layers detect increasingly abstract structures like textures, object parts, and whole objects. These early representations are domain-general: an edge detector useful for classifying cats is equally useful for classifying X-rays.
When you take a network trained on ImageNet -- the benchmark dataset of 1.2 million labeled images across 1,000 categories -- and apply it to a new image classification problem, the early and middle layers can often be reused almost verbatim. Only the final classification head needs to be replaced and retrained. Sometimes a few upper layers are also "unfrozen" and fine-tuned on the new data to adapt higher-level features to the new domain.
A landmark empirical demonstration came from Yosinski et al. (2014) in a paper titled "How Transferable Are Features in Deep Neural Networks?" published at NeurIPS. The researchers systematically measured how well features from different layers of a network trained on ImageNet transferred to a different image classification task. Their finding: the first few layers were highly transferable (essentially universal), middle layers were moderately transferable, and only the final layers were highly task-specific. This layer-by-layer transferability analysis provided the empirical foundation for the feature-extraction approach to transfer learning.
The ImageNet Moment
How ImageNet created the conditions for transfer learning
ImageNet is a dataset assembled by researchers at Princeton and Stanford, first published by Fei-Fei Li and colleagues in 2009, containing over 14 million hand-annotated images. The ImageNet Large Scale Visual Recognition Challenge (ILSVRC), run annually from 2010 to 2017, became the proving ground for deep learning methods.
In 2012, AlexNet -- a convolutional neural network submitted by Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton -- reduced the top-5 error rate from 26 percent to 15 percent. Nothing had produced a jump of that magnitude before. The result established convolutional neural networks as the dominant approach to image recognition and created a shared reference model that the entire research community could build on.
Because AlexNet's weights were published, and because subsequent winners like VGG, ResNet, and Inception were also open-sourced, the machine learning community gained a library of strong image feature extractors that could be downloaded and reused freely. Any researcher who wanted to classify a new kind of image -- satellite imagery, skin lesions, plant diseases -- no longer had to start from nothing.
ResNet and the stability of deep networks
The 2015 ResNet architecture, introduced by He et al. at Microsoft Research in a paper that won the ILSVRC that year with a top-5 error rate of 3.57%, brought a crucial innovation: residual connections (skip connections that let gradients flow directly through the network without passing through nonlinear transformations). This solved the vanishing gradient problem that had made networks deeper than about 20 layers difficult to train.
ResNet enabled training networks 100 or more layers deep without performance degradation. ResNet-50 and ResNet-152, both trained on ImageNet and openly published, became the default feature extractors for a generation of downstream tasks. Their learned representations were so general that fine-tuning them on a medical imaging dataset of only a few thousand examples routinely outperformed custom networks trained from scratch on the same data.
"The representations learned by deep networks for one visual domain are surprisingly transferable to other domains. This is not trivially expected -- it reflects something important about the shared structure of natural images." -- Fei-Fei Li, 2015 CVPR keynote
By 2016, the standard workflow for any new image classification problem had consolidated: download a ResNet pretrained on ImageNet, replace the final layer, fine-tune on your data. Projects that had previously required months of careful model design could be prototyped in days.
Transfer Learning in Natural Language Processing
The pre-transformer era: word embeddings
Before large language models, the primary form of transfer learning in NLP was word embeddings. Word2Vec (Mikolov et al., 2013, Google) and GloVe (Pennington et al., 2014, Stanford) produced dense vector representations of words by training on large text corpora. A model for sentiment analysis or named entity recognition could initialize its embedding layer with these pre-trained vectors rather than learning word meanings from scratch on limited labeled data.
Word embeddings transfer lexical knowledge -- which words are semantically similar -- but not contextual or syntactic structure. The same word "bank" had the same embedding regardless of whether it meant a riverbank or a financial institution. This static representation was a significant limitation.
Word2Vec's skip-gram model was trained on a corpus of approximately 100 billion words of Google News text. GloVe's largest model used a 840-billion-token common crawl corpus. The scale of pretraining data was already orders of magnitude larger than any downstream task would use, establishing the paradigm of large-scale unsupervised pretraining followed by small-scale supervised adaptation.
ELMo and contextual representations
The 2018 paper "Deep contextualized word representations" by Peters et al. at the Allen Institute for AI introduced ELMo (Embeddings from Language Models), which generated word representations conditioned on the entire sentence using a bidirectional LSTM. For the first time, pre-trained representations captured context: the word "bank" received a different embedding in "bank robbery" versus "river bank." ELMo significantly improved performance across multiple NLP benchmarks when used as a feature extractor in downstream models. Across 6 NLP tasks, ELMo produced an average relative improvement of 20% over previous state-of-the-art approaches.
BERT: bidirectional transformers and the fine-tuning paradigm
The same year, Google published BERT (Bidirectional Encoder Representations from Transformers) by Devlin et al. BERT was pre-trained on two tasks: masked language modeling (predicting randomly masked tokens) and next sentence prediction. Its architecture, the Transformer (introduced by Vaswani et al. in the landmark 2017 paper "Attention Is All You Need"), captured long-range dependencies in text far more effectively than recurrent networks.
BERT established the canonical NLP transfer learning pipeline:
- Pre-train a large transformer on an unsupervised language task using billions of tokens.
- Fine-tune the entire model (or add a task-specific head) on a small labeled dataset.
Models fine-tuned from BERT set new state-of-the-art results on eleven NLP benchmarks simultaneously, including question answering, sentence similarity, and language inference. On the SQuAD 2.0 question answering benchmark, BERT improved the F1 score by 5.1 points over the previous state-of-the-art. On GLUE, it improved 7.6 points. The community immediately produced dozens of BERT variants: RoBERTa (Liu et al., 2019, Facebook AI), DistilBERT (Sanh et al., 2019, Hugging Face), ALBERT (Lan et al., 2020, Google), DeBERTa (He et al., 2021, Microsoft).
GPT and generative transfer learning
OpenAI's GPT series extended the paradigm to generative language models. GPT-1 (Radford et al., 2018) demonstrated that a decoder-only transformer pre-trained on autoregressive language modeling could be fine-tuned with minimal task-specific data. GPT-2 (2019) scaled this up to 1.5 billion parameters; GPT-3 (Brown et al., 2020) scaled to 175 billion and introduced in-context learning: the model could perform tasks described in the prompt without any weight updates at all.
"GPT-3 shows that language models can be meta-learners -- they internalize so much knowledge during pre-training that task descriptions at inference time are sufficient to activate appropriate behavior." -- Paraphrase of the reasoning in Brown et al., 2020
This is often called few-shot prompting, and it represents a form of transfer learning that operates entirely at inference time. The model transfers its general knowledge to a new task specification without fine-tuning. GPT-3's few-shot performance on translation, arithmetic, and question answering tasks -- using only examples provided in the prompt and no gradient updates -- demonstrated that sufficient scale could eliminate the need for fine-tuning entirely for many tasks.
Types of Transfer Learning
| Type | How It Works | When to Use |
|---|---|---|
| Feature extraction | Freeze pre-trained layers; train only new head | Small dataset, similar domain |
| Fine-tuning | Unfreeze some or all layers; retrain at low learning rate | Medium dataset, related domain |
| Domain adaptation | Align feature distributions between source and target | Different data distribution |
| Multi-task learning | Train jointly on multiple tasks sharing a common backbone | Related tasks with complementary data |
| Zero-shot transfer | Use task descriptions alone; no target-domain training | Very few or no labeled examples |
| Few-shot learning | Train on a handful of labeled target examples | Limited labeled data available |
Feature extraction vs fine-tuning
The choice between these two strategies depends on the size of the target dataset and how different it is from the source domain.
Feature extraction treats the pre-trained network as a fixed function. You remove the final layer, pass your images through the network, collect the high-dimensional activations, and train a simple classifier (often logistic regression or a small fully connected network) on those activations. This is fast and avoids overfitting on small datasets.
Fine-tuning unfreezes some or all of the pre-trained weights and continues training with a small learning rate. This allows the model to adapt its representations to the new domain. The risk is overfitting when target data is scarce; the benefit is higher ceiling performance when sufficient data exists.
A practical rule of thumb: start with feature extraction; if you have more than a few thousand labeled examples and compute budget allows, try fine-tuning the upper layers. Howard and Ruder (2018), in the ULMFiT paper, introduced the technique of discriminative fine-tuning (using different learning rates for different layers) and gradual unfreezing (unfreezing one layer at a time from the top), which significantly improved fine-tuning stability and final performance.
Domain Adaptation
Domain adaptation is the branch of transfer learning that explicitly addresses the mismatch between training and deployment distributions. It matters any time you want to deploy a model in a context different from where it was trained.
Covariate shift
In covariate shift, the input distribution differs but the relationship between inputs and outputs remains the same. A model trained on daytime driving images performs poorly at night because the pixel distribution has changed, not because the semantics of roads and pedestrians have changed. Techniques like importance weighting and domain-randomized data augmentation address this. NVIDIA's synthetic data generation pipelines for autonomous driving training are a commercial-scale implementation of domain randomization: by training on images with randomly varied lighting, weather, and textures, the resulting models are more robust to the natural distribution shift between simulated and real driving conditions.
Adversarial domain adaptation
A more powerful approach uses adversarial training. A domain discriminator network tries to predict which domain an intermediate representation came from; the main network's encoder is trained simultaneously to fool the discriminator. This gradient-reversal trick forces the encoder to learn features that are invariant across domains while still being predictive of the output. Introduced by Ganin et al. (2016) in the DANN (Domain-Adversarial Neural Network) paper, this approach achieves state-of-the-art domain adaptation performance on multiple benchmarks and is widely used in computer vision, NLP, and speech applications.
Few-Shot and Zero-Shot Learning
Prototypical networks and metric learning
Few-shot learning methods in computer vision often use metric learning: the model learns an embedding space where examples of the same class cluster together. Snell et al.'s Prototypical Networks (2017) compute a prototype (mean embedding) for each class from its support examples, then classify queries by nearest-prototype distance. Given one or five labeled support examples for a new class, classification becomes a nearest-neighbor search in embedding space. These systems transfer the ability to compare examples rather than explicit category knowledge.
Model-Agnostic Meta-Learning (MAML), introduced by Finn, Abbeel, and Levine (2017) at Berkeley, takes a different approach: it finds model initializations that can be quickly fine-tuned to new tasks with very few gradient steps. MAML has been applied successfully to few-shot image recognition, few-shot language generation, and rapid adaptation of robotic policies.
Prompt engineering as zero-shot transfer
In large language models, zero-shot transfer is achieved by writing an informative prompt. Because GPT-4 was trained on text describing almost every domain of human knowledge, a well-crafted prompt can invoke the appropriate knowledge without any fine-tuning. Wei et al. (2022) demonstrated that chain-of-thought prompting -- asking the model to reason step-by-step before giving an answer -- dramatically improved performance on math word problems and logic tasks, achieving results competitive with fine-tuned smaller models. Zero-shot performance depends heavily on the scale and breadth of pre-training.
Why Transfer Learning Democratized AI
Before transfer learning became standard practice, building a production-quality image classifier required roughly one million labeled examples. Labeling costs run from $0.05 to $1.00 per image depending on complexity, meaning data acquisition alone could cost $50,000 to $1,000,000. Compute costs compounded this barrier.
Transfer learning shattered this threshold. In benchmark studies, fine-tuned ResNet models achieved competitive performance on domain-specific datasets as small as 500 to 1,000 images. Small companies, university research groups, non-profits, and individual developers could suddenly build state-of-the-art classifiers.
The effect on NLP was even more dramatic. Fine-tuning BERT on a new text classification task using 1,000 examples consistently outperformed models trained from scratch on tens of thousands. Projects that would have required months of data collection could be prototyped in a weekend.
A 2020 analysis by Bommasani et al. at Stanford found that across 12 common NLP tasks, models fine-tuned from large pre-trained transformers outperformed task-specific models trained from scratch by an average of 18 percentage points -- and this advantage was largest precisely when labeled data was scarcest, confirming that transfer learning's democratizing impact is greatest for resource-constrained organizations and low-resource languages.
Reducing carbon cost
Training a large model from scratch is computationally expensive and has a meaningful environmental cost. Strubell, Ganesh, and McCallum (2019) estimated that training a large NLP model from scratch can emit carbon dioxide equivalent to the lifetime emissions of five cars. Transfer learning, by reusing already-trained models, substantially reduces the marginal cost of each new application. An organization fine-tuning BERT for a specific language task may require only a few GPU-hours -- three to four orders of magnitude less compute than the initial BERT pre-training.
Lottick et al. (2019) proposed energy usage reports as a standard component of machine learning research publications to make this cost visible. The growing ecosystem of model repositories -- Hugging Face's model hub alone hosts over 400,000 pre-trained model checkpoints as of 2024 -- allows organizations to build on existing work rather than repeating expensive pretraining.
Limitations and Failure Modes
Negative transfer
Not all knowledge transfers positively. Negative transfer occurs when the source domain is sufficiently different from the target domain that pre-trained features hurt rather than help. Wang et al. (2019), reviewing transfer learning in NLP, documented cases where fine-tuning from BERT on semantically distant tasks produced worse results than training small task-specific models from scratch. Negative transfer is relatively rare when source and target domains share surface-level characteristics, but it becomes more likely when the tasks require fundamentally different types of reasoning or when the data distributions differ in ways that cause earlier representations to be misleading.
Catastrophic forgetting
When a model is fine-tuned aggressively on a small target dataset, it can catastrophically forget what it learned during pre-training, losing performance on the original task and generalizing poorly to new examples. Kirkpatrick et al. (2017) introduced Elastic Weight Consolidation (EWC) at DeepMind, which penalizes large changes to weights identified as important for prior tasks. This preserves the core knowledge from pretraining while allowing the model to adapt to the target task.
Catastrophic forgetting is particularly relevant for continual learning systems that must learn from a sequence of tasks without access to prior task data -- a common requirement in production settings where new categories or domains are introduced over time.
Bias amplification
Pre-trained models encode the biases present in their training data. A language model trained on internet text inherits gender and racial associations prevalent in that text. Zhao et al. (2019) demonstrated that fine-tuning on small datasets can amplify these biases: a model fine-tuned on a dataset with moderate gender bias for occupational terms showed substantially increased bias compared to the base model, because the fine-tuning data concentrated rather than distributed the model's probability mass. Any deployment of a fine-tuned model requires careful audit for unwanted biases.
Distribution shift at deployment
A model fine-tuned on data from one time period may degrade as the world changes. News classification models trained on articles from 2019 may perform poorly on 2022 articles because topics, terminology, and named entities have shifted. Continuous monitoring and periodic retraining are necessary for production systems. Rabanser et al. (2019) developed statistical methods for detecting distribution shift in production ML pipelines; similar monitoring tools are now integrated into major ML deployment platforms including AWS SageMaker Model Monitor and Google Vertex AI.
Practical Applications
Medical imaging
Radiology AI is among the most commercially successful applications of transfer learning. Companies like Aidoc, Enlitic, and PathAI have built diagnostic tools by fine-tuning ImageNet-pre-trained CNNs on thousands of annotated medical scans. The FDA had cleared over 500 AI-enabled medical devices as of early 2024, the majority of which use transfer learning to compensate for the inherent scarcity of labeled medical imaging data.
A 2019 Stanford study by Rajpurkar et al. fine-tuned a DenseNet-121 model (pretrained on ImageNet) on chest X-rays and achieved radiologist-level performance on detecting 14 pathological conditions -- including pneumonia and pleural effusion -- across a dataset of 112,120 chest X-rays. Without transfer learning, this result would have required vastly more labeled data than is practically available.
Document processing
Pre-trained transformer models fine-tuned on legal, financial, or scientific corpora extract structured information from unstructured documents at scale. Contract analysis, insurance claim processing, and academic literature review all rely on transfer learning. LayoutLM (Xu et al., 2020, Microsoft) extends the paradigm to documents with spatial structure, using pre-training on document images to understand the relationship between text and its position on the page. LayoutLM and its successors have substantially improved performance on form understanding, receipt analysis, and document classification benchmarks.
Speech and audio
Wav2Vec 2.0 (Baevski et al., 2020, Facebook AI) and Whisper (Radford et al., 2022, OpenAI) pre-train on large corpora of unlabeled or weakly labeled audio, then fine-tune on labeled speech for automatic speech recognition. Languages with limited transcribed speech data -- many indigenous languages, regional dialects -- have benefited enormously: Wav2Vec 2.0 fine-tuned on just 10 minutes of labeled Swahili speech outperformed models trained from scratch on 1,000 hours of data, a 6,000-fold reduction in labeled data requirement.
OpenAI's Whisper, pre-trained on 680,000 hours of multilingual audio, achieves near-human transcription accuracy in English and competitive performance in 96 other languages with no language-specific fine-tuning required.
The Future: Foundation Models and Transfer at Scale
The current frontier is foundation models: extremely large models (10 billion to 1 trillion parameters) trained on broad, multimodal data that serve as the base for thousands of downstream applications. GPT-4, PaLM 2, Gemini, and Claude are all examples in language. DALL-E, Stable Diffusion, and Midjourney use similar principles for image generation.
Bommasani et al. (2021) coined the term "foundation models" in a Stanford report documenting their properties and risks. A defining characteristic is emergence: capabilities that were not present at smaller scales and were not explicitly trained appear as model scale increases. Chain-of-thought reasoning, multi-step arithmetic, code generation, and in-context learning all emerged as capabilities as GPT-class models scaled beyond certain parameter counts.
The emergent capability of very large models -- the ability to solve novel tasks that were never explicitly trained for -- suggests that scale itself produces a qualitative shift in how knowledge is encoded and transferred. Wei et al. (2022) quantified this in a paper titled "Emergent Abilities of Large Language Models," documenting over 100 tasks on which performance jumped discontinuously as model scale crossed certain thresholds.
Parameter-efficient fine-tuning
Parameter-efficient fine-tuning (PEFT) methods address the compute cost of fine-tuning multi-billion-parameter models. Rather than updating all weights, these methods inject small trainable components into a frozen pre-trained model.
LoRA (Low-Rank Adaptation, Hu et al., 2022) injects small trainable low-rank matrices into selected layers, reducing the number of trainable parameters by a factor of 10,000 while preserving most of the performance gain. A 7-billion-parameter language model that would require weeks to fully fine-tune can be adapted via LoRA in hours on a single consumer GPU.
Prefix tuning (Li & Liang, 2021) and prompt tuning (Lester et al., 2021) prepend trainable "soft prompt" tokens to the input sequence, leaving the model weights entirely frozen. These methods are particularly attractive for organizations that cannot afford to store full model copies for each of many specialized applications -- a LoRA adapter might be 10 MB rather than 14 GB for a 7B parameter model.
Practical Evaluation Framework
When applying transfer learning to a new problem, practitioners benefit from a structured evaluation process rather than assuming the pre-trained model will transfer effectively by default.
Domain similarity assessment: How similar is your target domain to the pre-training domain? Visual similarity for image models (natural images vs. satellite vs. medical), textual similarity for language models (web text vs. legal documents vs. scientific literature). Greater similarity generally predicts better transfer. Kornblith et al. (2019), in a large-scale empirical study, found that ImageNet accuracy is a strong predictor of transfer learning performance -- but only for datasets similar to natural images. For highly specialized domains like dermoscopy or satellite imagery, domain-specific pretraining consistently outperformed ImageNet-pretrained models regardless of base model quality.
Dataset size assessment: How many labeled examples do you have for the target task? Under 500 examples: feature extraction only. 500-5,000: fine-tune upper layers. Over 5,000: consider full fine-tuning. These thresholds are rough guidelines, not hard rules.
Baseline comparison: Always compare the fine-tuned model against both a task-specific model trained from scratch and against the pre-trained model's zero-shot performance. This three-way comparison reveals the actual contribution of transfer learning in your specific case.
Evaluation on representative held-out data: Fine-tuning on small datasets can produce models that perform well on the fine-tuning distribution but poorly on the actual deployment distribution. Invest in collecting evaluation data that represents the real deployment environment.
Bias and fairness audit: Check whether the fine-tuned model has inherited or amplified biases from the pre-training data, particularly for applications in hiring, lending, healthcare, or any other consequential domain. Deploying a biased model exposes the organization to legal and reputational risk, and the social costs of biased AI in high-stakes domains are well documented.
Key Takeaways
Transfer learning is not a single technique but a broad paradigm: the recognition that knowledge extracted from one problem can accelerate learning on another. It has progressed from reusing word embeddings to fine-tuning billion-parameter language models, and each step has expanded who can build AI and what is economically feasible to attempt.
For practitioners, the default recommendation today is to start with the largest, most capable pre-trained model you can afford to fine-tune, provide clean labeled examples of your target task, and evaluate carefully before deploying. Building from scratch is the exception, not the rule.
For the field as a whole, transfer learning has been the single most important factor in the rapid commercialization of AI since 2012 -- more than any hardware advance, algorithmic innovation, or investment cycle. Understanding how it works is foundational to understanding the AI landscape.
References
- Yosinski, J., Clune, J., Bengio, Y., & Lipson, H. (2014). How Transferable Are Features in Deep Neural Networks? NeurIPS 2014. arXiv:1411.1792.
- He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. CVPR 2016. arXiv:1512.03385.
- Devlin, J., Chang, M., Lee, K., & Toutanova, K. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. NAACL 2019. arXiv:1810.04805.
- Brown, T. et al. (2020). Language Models Are Few-Shot Learners. NeurIPS 2020. arXiv:2005.14165.
- Bommasani, R. et al. (2021). On the Opportunities and Risks of Foundation Models. Stanford CRFM. arXiv:2108.07258.
- Hu, E. J. et al. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. arXiv:2106.09685.
- Wei, J. et al. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. arXiv:2206.07682.
- Strubell, E., Ganesh, A., & McCallum, A. (2019). Energy and Policy Considerations for Deep Learning in NLP. ACL 2019. arXiv:1906.02629.
- Howard, J. & Ruder, S. (2018). Universal Language Model Fine-tuning for Text Classification. ACL 2018. arXiv:1801.06146.
- Vaswani, A. et al. (2017). Attention Is All You Need. NeurIPS 2017. arXiv:1706.03762.
Frequently Asked Questions
What is transfer learning in simple terms?
Transfer learning is a technique where an AI model trained on one task is reused as the starting point for a model on a different but related task. Instead of training from scratch, the model brings prior knowledge — such as recognizing edges, shapes, or language patterns — and applies it to the new problem, saving time and data.
How does transfer learning differ from training a model from scratch?
Training from scratch requires enormous datasets and compute resources to learn even basic features. Transfer learning starts with a model that already understands low-level representations, so only the later layers or a small adapter need adjustment. This means competitive performance can be reached with far less labeled data — sometimes only a few hundred examples.
Is GPT a form of transfer learning?
Yes. GPT models are first pre-trained on massive corpora of internet text to learn general language representations, then fine-tuned on specific downstream tasks like summarization, question answering, or classification. This two-stage approach — pre-train then fine-tune — is the defining pattern of modern large language model transfer learning.
What is domain adaptation in transfer learning?
Domain adaptation addresses the situation where the source domain (where the model was trained) and the target domain (where it will be used) have different data distributions. Techniques include fine-tuning on target-domain data, adversarial domain adaptation, and feature alignment methods that minimize the statistical gap between the two distributions.
What is few-shot learning and how does it relate to transfer learning?
Few-shot learning is the ability to learn a new task from very few labeled examples, typically one to five. It relies heavily on transfer learning: a model with rich pre-trained representations can generalize from minimal data because it already understands the underlying structure of the input. Large language models like GPT-4 can perform few-shot learning directly in the prompt without any weight updates.