Transfer Learning
1. Overview
A. Definition
A machine-learning paradigm that transfers the knowledge and representations acquired from large-scale data on one task (the Source Task) and reuses them on a different task (the Target Task) where data is scarce or learning is hard, thereby achieving high performance with little data and cost.
The premise of transfer learning is the observation that "low-level representations are shared even across different tasks." Foundational features such as the edges and textures of images, or the grammar and word meanings of language, are reusable even when the task changes. So instead of learning from scratch every time, we take the general knowledge already captured by a trained model as a starting point and add only what the target task needs. This is likened to someone who knows English learning Spanish faster, and as "reuse of learning" it became a core driver of making deep learning practical.
This "representation-sharing" hypothesis is explained by the hierarchical structure of neural networks. The lower layers of a deep network learn universal, local patterns of the input, while the higher layers learn increasingly task-specific, abstract patterns. Lower layers are therefore reused almost as-is across tasks, and transfer is essentially the work of "inheriting the shareable lower representations and rebuilding only the task-specific upper representations." This structural understanding is the theoretical basis for the later strategy of "which layers to freeze versus retrain."
B. Background and Necessity
Traditional supervised learning assumes that "training and test data come from the same distribution, and each task has sufficient labels." Yet real medical, manufacturing, and financial domains have high labeling costs, scarce data, and frequently shifting distributions. Collecting millions of samples per task to train a huge model from scratch is unrealistic for most organizations. Transfer learning fills this gap. It treats a model pre-trained on abundant data such as ImageNet or a large corpus as a shared asset, adapting it to the target task with only a small amount of data.
This trend took off after 2012, when AlexNet confirmed that CNN features transfer well to other vision tasks, and became the de facto standard around 2018 when the pre-train-and-fine-tune paradigm of BERT, GPT, and others came to dominate natural language processing. Today's foundation models and LLMs are all products of large-scale transfer learning, and in practice "building a model" usually means something close to "transferring a public pre-trained model."
The necessity can be summarized in three points. First is economics: only a handful of organizations have the capital and data to train a huge model from scratch, so the rest have no choice but to rely on transfer. Second is speed: when time-to-market is a core competitive edge, the weeks to months of scratch training are hard to bear. Third is quality: to secure high accuracy with only small data, borrowing the power of the general knowledge captured by pre-training is the surest way. These three axes together elevated transfer learning from a "nice-to-have technique" to a "premise without which things do not hold."
C. Characteristics
Transfer learning offers three practical benefits: ① data efficiency (converging with few labels), ② faster training (starting from a good initialization), and ③ improved generalization (rich prior knowledge suppressing overfitting). These three benefits work in an interlocking way. Starting from a good initialization begins optimization in a more favorable region of the loss function, so it converges in fewer iterations (speed); in that process it finds a stable solution even with little data (efficiency); and the general knowledge captured by pre-training acts as a kind of implicit regularization that suppresses overfitting (generalization).
On the other hand, if the source and target tasks differ too much, it carries the risk of negative transfer, where performance actually drops. In addition, the biases and vulnerabilities embedded in an external pre-trained model can propagate directly into the target service, so transfer is "reuse of knowledge" and at the same time "reuse of risk." Transfer learning must therefore be treated not as an unconditional benefit but as a design-decision problem that includes source selection, transfer scope, and validation.
2. Structure and Classification of Transfer Learning
flowchart LR
subgraph S["Source domain (large-scale data)"]
SD["Source Data"] --> PT["Pre-trained model<br/>Pre-trained Model"]
end
PT -->|"Transfer of knowledge/weights"| TM["Target model"]
subgraph T["Target domain (small-scale data)"]
TD["Target Data"] --> TM
TM --> OUT["Target-task performance"]
end
style PT fill:#e8f0fe,stroke:#2f6fed,stroke-width:2px
style TM fill:#fef3e8,stroke:#ed8a2f,stroke-width:2px
Transfer learning is structured around three questions: "What to transfer, how, and when." Depending on what is transferred, it divides into instance, feature-representation, parameter, and relational-knowledge transfer; depending on how, it splits into feature-extraction and fine-tuning types. When one should "not" transfer is directly tied to the problem of avoiding negative transfer.
Classifying by whether the domain and task are identical gives the following. This classification matters because label availability and distribution difference determine which technique to apply. For example, if there are no labels at all, domain adaptation (unsupervised) is valid; if even a small amount of labels exists, fine-tuning is valid.
| Type | Domain | Task | Representative situation |
|---|---|---|---|
| Inductive transfer | Same/similar | Different | Target labels exist — fine-tune after pre-training |
| Transductive transfer | Different | Same | No target labels — Domain Adaptation |
| Unsupervised transfer | Different | Different | Labels scarce on both sides — representation-learning focused |
A. Feature Extraction approach
This approach freezes the pre-trained model and trains only the final classifier (Head) anew. It uses the model's intermediate-layer outputs (embeddings) as a fixed feature extractor; because very few parameters are trained, it suits cases when data is very scarce or overfitting is a concern. For example, in a medical-image classification task with only a few hundred images, simply extracting features from an ImageNet pre-trained CNN and feeding them into logistic regression or an SVM already yields substantial performance.
The reason this approach is strong with little data is that the parameters to be trained are confined to a single classifier layer, so the model's very degrees of freedom to memorize are small. Compute cost is also low, offering the practical advantage of caching embeddings extracted once and rapidly iterating many experiments. However, if the source and target distributions differ greatly, the fixed features do not fit the target and reach a limit; in that case a compromise is needed—switching to fine-tuning or selectively unfreezing intermediate layers.
B. Fine-tuning approach
This approach takes the pre-trained weights as an initialization and additionally trains some or all layers. Typically the lower layers close to the input, holding general features (edges, shapes), are frozen, while the upper layers close to the output, holding task-specific features, are retrained. The more data there is, the deeper the layers that are unfrozen for training, and the standard here is to keep the learning rate small (to prevent destroying prior knowledge). Fine-tuning has a higher performance ceiling than feature extraction but carries greater risk of overfitting and catastrophic forgetting.
C. Domain Adaptation
This handles the situation where the task is the same but the data distribution differs (covariate shift). The core idea is to align the feature distributions of the two domains to learn a domain-invariant representation. Representatively, DANN (Domain-Adversarial Neural Network) places a domain discriminator adversarially, inducing the feature extractor to produce representations where "it cannot tell whether it is source or target." For example, it is used to adapt an autonomous-driving perception model trained on daytime road footage to nighttime or rainy environments (the target).
Domain adaptation matters because real-world distributions are not fixed. When sensors are replaced, seasons change, or the user population shifts, the input distribution moves, and the original model's performance drops sharply. Adaptation techniques align representations using only the target's input distribution even with no target labels at all (unsupervised), so they are especially useful in operating environments where labeling is impossible. Besides DANN, there are diverse families such as minimizing a statistical distance (MMD) and generative approaches that convert source features into the target style, and all of them share one question: "how to measure and reduce the distribution difference."
D. Relationship with multi-task and meta-learning
Multi-task learning obtains a shared representation by learning multiple tasks simultaneously, corresponding to "simultaneous transfer" as opposed to the "sequential transfer" of transfer learning. Because multiple tasks reinforce each other's learning signals, each task generalizes better than when learned alone, as the structure shared across tasks pushes the model toward a more general representation. Meta-learning (Learning to Learn) learns from many tasks "the very method of adapting quickly," adapting to a new task with only a few shots (Few-shot) (e.g., MAML). Both fall within the broad category of transfer learning and are used complementarily when data is extremely scarce.
Summarizing the relationship of the three concepts: transfer learning is the higher-level concept of "moving knowledge from one task to another," multi-task learning is a variant that shares that knowledge "simultaneously," and meta-learning is a variant that elevates "the adaptation capability itself" to the object of transfer. In practice, the three techniques combine hierarchically: building a general representation through large-scale multi-task pre-training, transferring it to individual tasks, and layering meta-learning and Few-shot onto long-tail tasks with extremely little data.
3. Application Procedure and Core Mechanisms
flowchart TD
A["1.Select source model<br/>Assess task similarity"] --> B["2.Decide transfer strategy<br/>Freeze vs fine-tune"]
B --> C["3.Adjust model structure<br/>Replace head, select layers"]
C --> D["4.Train<br/>Small LR, early stopping"]
D --> E["5.Evaluate/validate<br/>Check negative transfer"]
E -->|"Underperforms"| B
E -->|"Meets target"| F["6.Deploy and monitor"]
style D fill:#e8f0fe,stroke:#2f6fed,stroke-width:2px
The success of transfer hinges on deciding "how much to transfer (How much) and where to unfreeze (Where)." This decision is judged along two axes: ① the task similarity of source to target, and ② the target data size. The four-quadrant strategy below is widely used as a standard practical guideline.
| Target data \ Similarity | High similarity | Low similarity |
|---|---|---|
| Little data | Feature extraction (freeze) only — avoid overfitting | Hard — use intermediate-layer features, combine with data augmentation |
| Much data | Fine-tune some upper layers | Full fine-tuning (effectively using only the initialization) |
The basis of this guideline is clear. When data is scarce, unfreezing many parameters overfits, so widen the frozen range; when data is abundant, unfreezing is safe, so widen the fine-tuning range. When similarity is low, upper-layer features do not fit, so deeper lower layers must be retrained. In fact, one Google study (Transfusion, 2019) reported that ImageNet pre-training yields little benefit on tasks with very different distributions, such as medical imaging, and that smaller custom models even perform comparably—supporting the principle that "when similarity is low, the transfer effect diminishes." This is a representative case showing that strategy must vary with domain characteristics rather than applying transfer mechanically.
The mechanism of the training stage also matters. During fine-tuning, a small learning rate and early stopping are used so that prior knowledge is not destroyed abruptly, and forgetting is mitigated with techniques such as discriminative fine-tuning (giving different learning rates per layer) or gradual unfreezing (slowly raising then lowering the learning rate early on). All of these techniques stem from the common principle of "carefully exploring around the good solution that pre-training has already found."
As a concrete example, consider classifying factory product defects at a scale of 1,000 images. Scratch training may plateau in the 70-percent accuracy range due to overfitting, but freezing the lower layers of an ImageNet pre-trained ResNet and fine-tuning only some upper layers often raises it above 90 percent on the same data. In a field where increasing data tenfold is hard, transfer learning becomes a realistic solution of "borrowing knowledge instead of increasing data." Conversely, if the source is unsuitable, this benefit disappears or reverses, so strategy selection and validation decide success or failure.
A. Negative Transfer and mitigation
Negative transfer is the phenomenon where source knowledge actually harms target performance. It occurs when source and target are unrelated, or when the source holds biases that conflict with the target. Mitigations include ① measuring transferability in advance (e.g., assessing source suitability with metrics like LEEP or H-score), ② selecting highly relevant sources, and ③ adjusting the strength of prior-knowledge preservation with a small learning rate and regularization. For example, bringing a model from a completely different domain into financial-fraud detection can perform worse than scratch training, so the transfer effect must always be confirmed on a validation set.
Negative transfer is dangerous because it is hard to notice. When a transferred model works to some degree, developers tend to believe "it is thanks to transfer anyway," when in reality scratch training might have been better. The criterion for judging the transfer effect must therefore always be not "the absolute performance of the transferred model" but "the relative gain over scratch training under identical conditions." Omitting this control comparison leaves transfer learning standing on an unverified assumption.
B. Transfer learning in the LLM era
LLMs are the pinnacle of transfer learning. After self-supervised pre-training on a web-scale corpus, they are adapted to downstream tasks by ① fine-tuning (SFT, PEFT/LoRA), ② prompting and in-context learning (inference-time transfer with no parameter updates), and ③ knowledge distillation (transferring capability from a large model to a small one). In particular, in-context learning draws attention as a new mode distinct from traditional transfer learning in that it transfers a task from examples alone without changing weights.
These three paths show clear trade-offs in cost and flexibility. Fine-tuning permanently imprints knowledge into the model with high consistency but carries training and deployment burdens per task; in-context learning is immediately applicable but requires inserting examples at every inference, incurring token cost and context limits. Knowledge distillation lowers operating cost but accepts the loss of the distillation process. In practice, a hybrid strategy of "fine-tune for frequent, fixed tasks; prompt for variable, exploratory tasks" is taking hold.
Adding external knowledge, RAG (Retrieval-Augmented Generation) is yet another mode of "transferring" knowledge without changing model weights, widely used as an alternative or complement to fine-tuning in enterprise environments where recency and source traceability matter. In the end, transfer in the LLM era can be seen as having expanded along two axes: "transfer that changes parameters" and "transfer that injects via context."
4. Comparison with Conventional Machine Learning
| Category | Traditional supervised learning | Transfer learning |
|---|---|---|
| Data assumption | Train/test same distribution, sufficient labels per task | Allows different distribution/task between source and target |
| Learning starting point | Random initialization (from scratch) | Pre-trained weights (warm start) |
| Data requirement | Large | Possible even with little |
| Main risk | Overfitting when data is scarce | Negative transfer |
The fundamental reason the difference between the two approaches arises is "whether knowledge is acquired anew each time or reused." The traditional approach is independent per task, so data and compute grow linearly, whereas transfer learning assetizes a representation learned once and lowers the marginal cost. This is the same principle by which software reuse boosts productivity, and from an MLOps perspective, pre-trained model hubs (such as Hugging Face) function as a "model-reuse ecosystem."
In concrete numbers the difference is clear. Scratch training requires tens of thousands to millions of labels and tens to hundreds of GPU-hours per task, whereas transfer learning often reaches near-equivalent performance on the same task within hundreds to thousands of labels and a few GPU-hours. In other words, the real value of transfer learning is cutting data and compute by an order of magnitude or more. However, since this saving holds only "when an appropriate source is chosen," the conclusion of the comparison is summarized as "transfer is not free but entails a new design responsibility of source selection."
5. In Depth — Industrial Applications and Recent Trends
Transfer learning has already penetrated deeply across industry. In healthcare, models pre-trained on ImageNet and vast chest X-rays are transferred to reading rare diseases, with reported cases approaching specialist-level accuracy from only a few hundred data points. In manufacturing, a defect-detection model from one production line is transferred to a new line or new product, shortening retraining from weeks to days. In finance, domain-adaptive pre-training of a general language model on financial documents, as in BloombergGPT and the KB-ALBERT family, is representative.
Beyond this, natural language processing as a whole has standardized on transferring BERT and GPT family pre-trained models to dozens of downstream tasks such as sentiment analysis, named-entity recognition, and question answering, and the formula of large-scale pre-training followed by domain adaptation repeats in speech, autonomous driving, and recommendation as well. The common thread is "returning the knowledge earned on data-rich general tasks to data-poor specialized tasks," and this feedback structure is the decisive factor that extended the reach of AI into data-scarce domains.
In sum, transfer learning is no longer an optional technique but has become the default way of working in modern AI development. Amid the trend of huge pre-trained models accumulating as public assets, PEFT lowering adaptation cost, and transferability estimation automating source selection, an organization's AI capability will likely be measured going forward less by "how much data it holds" and more by "how cleverly it transfers and operates shared model assets."
Recent trends: first, foundation-model-centric transfer has become universal, with "giant pre-training → lightweight adaptation" the default formula across vision, language, and multimodal. Second, with PEFT (LoRA, QLoRA, Adapter) training only around 1 percent of the total parameters, transfer has become possible even on a single GPU, sharply lowering the cost barrier of transfer. Third, transferability estimation research is active, proposing metrics (LEEP, LogME, etc.) that predict before training which of many pre-trained models to bring in. Fourth, federated learning and privacy-preserving transfer is rising, with the direction of transferring only model knowledge without sharing raw data drawing attention in regulated medical and financial environments.
In Hàn Quốc as well, transfer learning is establishing itself as a core foundation of public and industrial AI. For example, Korean-specialized pre-trained models (KoBERT, KoGPT, KB-ALBERT, etc.) are typical transfer cases of domain-adaptive pre-training on a Korean corpus to avoid the performance drop of using English-centric models as-is. In addition, transferring to small-label tasks using AI Hub and public data is recommended as a means of lowering the cost of AI adoption for SMEs and public institutions. In this way, transfer learning functions as a bridge to the democratization of AI connecting "the few who own huge models" and "the many with little data."
6. Considerations and Implications
From a professional engineer's perspective, the following must be comprehensively considered when adopting transfer learning.
- Pre-validating transferability (application strategy): Assess source-target similarity and transferability metrics before training, and be sure to confirm performance relative to scratch training on a validation set to rule out negative transfer. Treat the assumption that "a pre-trained model is always better" as forbidden.
- Data/compute trade-off: Decide the freeze/fine-tune scope according to target data size and similarity. With little data, take overfitting avoidance via freezing-centric, strong regularization; with much data, take the performance ceiling via expanded fine-tuning. Prioritize designs that minimize compute and storage cost with PEFT.
- Governance, licensing, security: External pre-trained models carry risks of training-data provenance, licensing, bias, and backdoors. Secure traceability at the level of a Model Card and SBOM, and from a supply-chain security perspective use only trustworthy model hubs. Run fairness assessments in parallel so transferred bias does not propagate into the target service.
- Continuous operation and distribution-shift response (outlook): Because the transfer effect can diminish from data-distribution drift after deployment, embed a cycle of monitoring and re-fine-tuning into the MLOps pipeline. In the long run, combined with foundation models, federated learning, and meta-learning, it is expected to evolve into a continually adaptive AI that adapts quickly even under small-data and privacy constraints.
- Talent and organizational capability (outlook): Since the real benefit of transfer learning hinges on the capability to judge "which pre-trained model to adapt and how," organizations must internalize know-how in model selection, transferability assessment, and tuning. The center of gravity of AI organizations is shifting toward a direction where the capability to handle model assets, rather than data itself, becomes the competitive edge.
- Related technologies: It is tightly linked with fine-tuning (PEFT/LoRA), foundation models, self-supervised learning, federated learning, knowledge distillation, and domain adaptation, and the key is to combine these to design an optimal transfer strategy that fits the organization's data, regulatory, and cost constraints.
References
- Pan & Yang, "A Survey on Transfer Learning", IEEE TKDE, 2010: https://ieeexplore.ieee.org/document/5288526
- Ganin et al., "Domain-Adversarial Training of Neural Networks(DANN)", JMLR, 2016: https://jmlr.org/papers/v17/15-239.html
- Raghu et al., "Transfusion: Understanding Transfer Learning for Medical Imaging", NeurIPS, 2019: https://arxiv.org/abs/1902.07208
- Hugging Face, "Transfer Learning / Fine-tuning": https://huggingface.co/docs/transformers/training
- Yosinski et al., "How transferable are features in deep neural networks?", NeurIPS, 2014: https://arxiv.org/abs/1411.1792
In one line: Transfer learning is a paradigm that reuses representations learned from large-scale data on a new task with little data, selecting feature-extraction, fine-tuning, and domain-adaptation strategies according to data size and task similarity, while always validating against negative transfer.