IM Engineer Study Notes
← Back to list
AI & Data
#자기지도학습#SSL#표현학습#대조학습#마스킹모델링
Last updated · 2026-10-17

Self-Supervised Learning

1. Overview

A. Definition

Self-Supervised Learning (SSL) is a machine learning paradigm that learns representations by generating its own supervisory signal from the structure of the data itself, without human-assigned labels. A model is trained to mask or transform part of the input and then predict the hidden portion, the original, or a relationship from the rest; the general-purpose representation obtained this way is later fine-tuned with a small amount of labels or transferred to downstream tasks as-is (frozen).

The fundamental backdrop for the emergence of SSL lies in the "labeling cost" limit of supervised learning. Deep learning performance improves in proportion to data scale, but data hand-labeled by humans — like ImageNet for image classification — takes enormous time, cost, and expertise to create. Attaching disease labels to medical images requires specialists, and the same holds for speech, text, and industrial sensor logs. By contrast, unlabeled raw data — web text, YouTube videos, process logs — exists in virtually unlimited quantities. SSL is a method of extracting "free labels" from this abundant unlabeled data, bypassing the labeling bottleneck to enable large-scale pretraining.

Another backdrop is the transferability of representation learning. Instead of training a model from scratch for each task, if the general structure of the data is learned deeply once, that representation can be reused across many downstream tasks. SSL is precisely the core engine that produces this "general-purpose representation learned deeply once," and today both large language models such as GPT and BERT and vision foundation models such as CLIP and DINO all stand on self-supervised pretraining.

Historically, too, the rise of SSL was a turning point in the deep learning paradigm. If the success of vision in the 2010s owed itself to large-scale "labeled" data such as ImageNet, the flow in natural language processing around 2018 running word2vec → ELMo → BERT·GPT proved that stronger representations could be obtained "from large unlabeled corpora alone." This success was soon re-imported into vision, leading to SimCLR, MAE, and DINO, and was ultimately established as a common strategy cutting across modalities. In short, SSL symbolizes less a specific technique than the shift of deep learning away from label dependence toward learning from data itself.

B. Necessity and Characteristics

The reasons SSL has established itself as a foundational technology of modern AI can be organized into three. First, data efficiency and scalability. Data scale can be grown to billions or trillions of tokens without labeling, making it the premise for large-model training where "scaling laws" operate. Second, transferability of general-purpose representations. A pretrained representation adapts to diverse downstream tasks — classification, retrieval, detection, generation — with only a small amount of labels, so it is especially valuable in data-scarce domains (medicine, manufacturing, finance). Third, similarity to human cognition. Humans learn rules by observing the world without an explicit answer key, and SSL implements this "learning by observation" in machines; Yann LeCun described it as the "dark matter of intelligence."

Looking a little more closely at its characteristics, the decisive point is that SSL belongs to unsupervised learning yet borrows the form of supervised learning. Whereas traditional unsupervised learning such as clustering and dimensionality reduction stops at "discovering structure without answers," SSL automatically generates (input, answer) pairs from the data and learns with exactly the same predict-loss-backpropagate framework as supervised learning. Hence it is called learning that "has no labels but does have supervision." Moreover, the pretext task is not an end in itself but a pretext for obtaining good representations, so the quality of task design governs the quality of the representation.

SSL also differs from ordinary learning in its evaluation philosophy in that its objective is the transfer performance of the representation it produces as a byproduct, rather than performance on the pretext task itself. The measure of success is not whether it fills in a blank word 95% of the time, but how well the resulting representation works in downstream classification and retrieval. For this reason, research in SSL is an exploration of task design asking "which task induces good representations," and the history of progress from early tasks such as colorization and rotation prediction to contrastive and masking methods was itself "a process of finding a better pretext."

2. Principles and Learning Structure of SSL

The principle of SSL lies in "predicting one part of the data from another part." It predicts the next word of a sentence from preceding words, or restores a hidden image patch from surrounding pixels. In the process of unraveling the internal structure of the data (contextual, spatial, and temporal correlations) in this way, the model acquires meaningful representations. The overall pipeline is composed of a two-stage continuum: "pretraining → representation freezing/transfer → downstream fine-tuning."

flowchart LR
  D["Large unlabeled data<br/>(text, images, logs)"] --> A["Automatic task generation<br/>(masking, augmentation, splitting)"]
  A --> PT["Pretraining<br/>(Pretext Task)"]
  PT --> E["Encoder<br/>(general representation learning)"]
  E --> R[("Pretrained model<br/>Representation")]
  R --> FT["Downstream adaptation<br/>(Fine-tuning·Linear Probe)"]
  L["Small labels<br/>(Downstream)"] --> FT
  FT --> T["Classification, detection, retrieval, generation"]
  style PT fill:#e8f0fe,stroke:#2f6fed,stroke-width:2px
  style R fill:#e6f4ea,stroke:#34a853

A point to note in this two-stage structure is that the data and the objective of pretraining and of the downstream task are separated. Pretraining learns "broadly" from large unlabeled data spanning the whole domain, while downstream adaptation specializes "narrowly" with a small amount of labels for a specific task. Thanks to this separation, an economy of scale arises in which one pretrained model is reused across many tasks and many organizations. The essence of the foundation model economy — a structure in which a few pretrain gigantic models and the many adapt and use them — originates precisely in this two-stage separation.

The core of learning lies in the design of the pretext task. The task must be one in which "the answer can be extracted automatically from the data, yet getting the answer right requires a deep understanding of the data's meaning." For example, filling in the blank word of a sentence requires mobilizing grammar, common sense, and context all at once, so a model that solves this task well acquires a rich representation of language. Conversely, if the task is too easy (e.g., a task solvable by color histogram alone), the model learns a "shortcut" and the representation becomes impoverished. Therefore task difficulty and shortcut prevention are the first principle of design.

Once pretraining is finished, the representation generated by the encoder is transferred to the downstream task. There are two transfer methods: linear probing, which fixes the encoder and evaluates with only a thin classifier placed on top, and fine-tuning, which retrains the encoder as well. Linear probing is often used as a yardstick measuring the quality of the representation itself, while fine-tuning is used to maximize downstream task performance. In this way, the value of SSL is measured by "how widely a once-learned representation can be reused with how few labels."

Viewing the learning signal formulaically, the loss function differs by task type. The generative/reconstructive type uses cross-entropy or mean squared error (MSE) to measure the difference between the original and the reconstructed value, while the contrastive type uses InfoNCE, which places positive pairs in the numerator and negative pairs in the denominator to maximize relative similarity. An important practical implication is that a low loss value is no guarantee that the representation is good. For example, in the non-contrastive type, making all inputs into the same constant vector drives the loss toward 0 but renders the representation entirely useless (collapse). Therefore, in SSL, the standard practice is to track diagnostic metrics such as the variance, uniformity, and alignment of the representation alongside the loss curve.

3. Types of SSL Methodologies

SSL methodologies are broadly divided, according to how the learning signal is created, into predictive (generative), contrastive, non-contrastive (self-distillation), and masked modeling. The detailed architecture diagram below shows the differences in "how each transforms, contrasts, or reconstructs a single input."

flowchart TB
  X["Original input x"] --> AUG["Data augmentation/transformation"]
  AUG --> V1["View 1 (view)"]
  AUG --> V2["View 2 (view)"]
  V1 --> F1["Encoder f"]
  V2 --> F2["Encoder f / Momentum f'"]
  F1 --> Z1["Representation z1"]
  F2 --> Z2["Representation z2"]
  Z1 --> OBJ{"Learning objective"}
  Z2 --> OBJ
  OBJ --> C["Contrastive: pull positives, push negatives<br/>(InfoNCE)"]
  OBJ --> NC["Non-contrastive: match two views<br/>(Stop-gradient·EMA)"]
  X --> MASK["Masking/corruption"]
  MASK --> DEC["Reconstruction decoder"]
  DEC --> REC["Original reconstruction<br/>(reconstruction loss)"]
  style OBJ fill:#fff4e5,stroke:#f9a825,stroke-width:2px

A. Predictive (Generative) Type — Autoregressive/Reconstruction-Based

The predictive type is the most classical and yet the most powerful family today. Its core is setting aside part of the data as the "answer" and generating/reconstructing it from the rest. The representative in the language domain is the autoregressive LM; the GPT family repeats the simple task of "predicting the next word from preceding words" over trillions of tokens and acquires grammar, knowledge, and reasoning ability in language. Because the answer (the next word) already exists in the text, this task needs no labels and, decisively, the data scale can be grown without limit.

In the vision domain, early tasks such as colorization, jigsaw puzzles, and rotation-angle prediction were proposed. For example, having a model restore color to a black-and-white image forces it to learn semantic knowledge such as "the sky is blue and grass is green." However, such handcrafted tasks are vulnerable to shortcuts and have clear performance limits, so leadership later passed to the contrastive and masking families. Even so, the idea of "inducing understanding through generation" is inherited by the later masked modeling.

The practical implication of the predictive type lies in scalability. GPT-3 (175 billion parameters) and its successor models made general-purpose abilities spanning translation, summarization, coding, and reasoning emerge from the single task of next-word prediction alone. This shows the core lesson of SSL: that even if the task is simple, rich representations emerge when the data and model scale are sufficiently large.

Meanwhile, even within the predictive type, the choice between unidirectional (autoregressive) and bidirectional (reconstruction) divides downstream-task suitability. Autoregression sees "only the left context," so it is natural for text generation, but bidirectional reconstruction is advantageous for classification and extraction tasks that require understanding the whole sentence. This is why the generation-strong GPT family and the understanding-strong BERT family diverged, and recently hybrid designs that unify the two objectives or switch between them by situation have appeared. Choosing the pretraining objective to match the nature of the task (generation or understanding) is a practical point of design.

B. Contrastive Learning

Contrastive learning learns representations with the distance geometry of "keep the same close and the different far." Applying different augmentations (crop, color jitter, blur) to one image to create two views makes them a positive pair since they came from the same original, while the other images in the batch are negative pairs. The model learns to pull the representations of positive pairs together and push negative pairs apart, and the loss used here is InfoNCE. Representative models are SimCLR (securing negative samples via large batches) and MoCo (maintaining a bank of negative samples with a momentum encoder and a queue).

The principled strength of contrastive learning is learning semantic invariance. Because it forces the same representation to come out even when the same object is transformed in various ways, the model obtains a representation that is insensitive to peripheral changes such as color, angle, and background and sensitive to the essence of the object. Its weaknesses, however, are also clear: it requires many negative samples, so large batches or a memory bank are needed, and the "false negative" problem — in which images of the same meaning get mixed into the negative samples — eats into performance. In practice, on the ImageNet linear-probing benchmark, SimCLR achieved accuracy (about 76%) close to supervised pretraining, proving that "supervised-grade representations are possible even without labels."

That the choice of data augmentation determines the character of the representation is also an important design insight in contrastive learning. This is because what augmentation "treats as the same" is precisely what defines "to what the representation will be made invariant." For example, including color jitter in the positive pairs produces a color-insensitive representation, which is advantageous for general object recognition but can be harmful for tasks where color is a key cue (e.g., flower or bird classification). Therefore the augmentation policy must be designed in consideration of the nature of the downstream task, and this shows that contrastive learning is a technique that "injects the prior knowledge (inductive bias) embedded in augmentation into learning."

C. Non-Contrastive (Self-Distillation) Type — Learning Without Negative Samples

The non-contrastive type learns representations with positive pairs alone, without negative samples. Without negative samples, merely saying "make the two views match" causes collapse, in which all inputs converge to the same vector, so the ingenious device that prevents this is key. BYOL places an online network and a target network updated by exponential moving average (EMA), and avoids collapse by putting a predictor and a stop-gradient on one side. SimSiam achieves the same effect more simply with stop-gradient alone. DINO combines teacher-student self-distillation with centering and sharpening.

The practical implication of the non-contrastive type is efficiency and stability. Since no negative samples are needed, it works even with small batches, and the false-negative problem disappears. In particular, DINO/DINOv2, combined with the Vision Transformer (ViT), drew attention by making attention maps that segment object boundaries on their own emerge even without labels. Meta's DINOv2, pretrained on 140 million curated images, is used as a vision foundation model that provides powerful general-purpose vision representations across many tasks — depth estimation, segmentation, classification — even without fine-tuning.

How the non-contrastive type "prevents collapse" looks trivial on the surface but is a theoretically interesting point. The mechanism by which BYOL avoids collapse without negative samples using only stop-gradient and a predictor was at first a mystery of "why it works," and analyses followed that the asymmetric structure implicitly maintains the diversity of representations. The lesson from a practical standpoint is that the success or failure of SSL is governed less by grand architecture than by fine learning devices such as collapse prevention and suppression of dimensional reduction, so it is important to follow these detailed settings exactly when reproducing.

D. Masked Modeling

Masked modeling is a family that hides a substantial portion of the input and then has it reconstructed, and it has now become the de facto standard in both language and vision. Language's BERT masks about 15% of sentence tokens with [MASK] and predicts them with bidirectional context (MLM, Masked Language Modeling), learning a representation that reads context from both sides. This contrasts with GPT's unidirectional autoregression and shows strength in sentence-understanding and classification tasks.

Vision's MAE (Masked Autoencoder) hides a high ratio of 75% of image patches and then reconstructs the whole from the remaining 25%. The higher the hidden ratio, the more shortcuts are blocked and the more meaningful the learned representation, and since the encoder processes only visible patches, computational efficiency is also high. The principled strength of masked modeling is that by combining the generative (reconstruction) and self-supervised, it is robust to shortcuts and highly scalable, and it is being broadly extended to multimodal (image + text), audio, and time series. As an industrial application case, a method of applying masked reconstruction to process-sensor time series to learn normal patterns without labels and then transferring to anomaly detection is utilized on the manufacturing floor.

That the masking ratio differs greatly between language and vision (BERT 15% vs MAE 75%) is explained by the difference in the modalities' information density. In language, each token carries high-density meaning, so even a little masking makes reconstruction hard; in images, adjacent pixels are redundant, so a little masking creates a shortcut of copying the surroundings verbatim to reconstruct. Therefore images must be masked heavily before the model finally "infers meaning." That even the hyperparameters of a self-supervised task must be tuned to match the statistical properties of the data shows once again that SSL is an engineering of task design.

4. Comparison with Supervised, Unsupervised, and Transfer Learning

To clarify the position of SSL, one must point out the differences from adjacent paradigms down to "the reason the differences arise." The table is a supplementary summary; the core is the source of labels and the character of the learning objective.

Category Label source Learning objective Representative techniques Data requirements
Supervised Hand-assigned by humans Input→answer mapping CNN classification, regression Many labels, expensive
Unsupervised None Discovering structure/distribution K-means, PCA, autoencoder No labels needed
Self-supervised Auto-generated from data General representation learning BERT, SimCLR, MAE Large unlabeled + small labels
Transfer learning (Reuse of pretrained model) Knowledge transfer Fine-tuning, feature extraction Pretraining + downstream labels

The essential difference from supervised learning is whether a human gives the labels or the data supplies them itself. Supervised learning uses exact answers, so the task is clear, but labeling cost determines the ceiling of scale. SSL auto-generates answers from the data, so the scale constraint is lifted, but if task design is done poorly, there is a risk of learning a representation unrelated to the downstream task. In short, the difference between the two resolves into a trade-off of "accuracy vs. scalability," and in practice the combination of self-supervised pretraining + small-label fine-tuning has become the dominant strategy that takes both advantages.

Concrete numbers reveal the practical gain of this trade-off. In many studies in the medical-imaging field, in low-resource settings where labels are only 1–10% of the total, a model fine-tuned after self-supervised pretraining shows accuracy higher than a model trained supervised from scratch — by several percentage points and sometimes by double-digit percentage points. When labels become abundant, the gap narrows, but the pretrained representation still gives an advantage in convergence speed and generalization. In short, the regularity that "the rarer the labels, the greater the relative benefit of SSL" is the core implication of the comparison.

The difference from unsupervised learning is subtle but important. SSL broadly belongs to unsupervised learning (no labels), but whereas traditional unsupervised learning aims at "discovering structure" itself, such as clustering and density estimation, SSL aims to borrow supervised learning's predict-loss framework as-is to make transferable representations. The relationship with transfer learning is close to an inclusion relation. Transfer learning is the higher concept of "moving pretrained knowledge downstream," and SSL is a concrete method of performing that pretraining stage without labels. Today the transfer learning of foundation models almost entirely presupposes self-supervised pretraining.

5. Deep Dive — Recent Trends and Industrial Diffusion

SSL is diffusing beyond a single-model technique into the common engine of the foundation model era. First, multimodal SSL surged. CLIP aligned 400 million image-text pairs with contrastive learning, obtaining a zero-shot ability to classify arbitrary categories even without labels. This principle of "aligning different modalities into the same representation space" later became the foundation of multimodal models spanning images, audio, and video.

Second, the generalization of masked modeling. Masked reconstruction, which first started in language (BERT), extended to vision (MAE), audio (wav2vec 2.0), graphs, and time series, establishing itself as "a self-supervised formula that works for all modalities." Meta's data2vec presented a unified self-supervised framework that handles speech, vision, and language with a single algorithm by applying the identical learning framework (a student predicts teacher representations) regardless of modality. Third, the world model and JEPA. I-JEPA/V-JEPA, proposed by Yann LeCun, performs prediction in representation space instead of pixel-level reconstruction, drawing attention as the latest direction that seeks to learn more meaning-centric representations while reducing the generation burden.

Behind this trend is the problem awareness that "the model capacity spent on reconstructing pixels is wasted on reproducing fine textures irrelevant to meaning." The JEPA family, after masking the input, predicts only the representation (embedding) of that part, inducing the model to focus on abstract information such as the structure and relationships of a scene rather than trivial textures. This shows the direction in which SSL evolves from "a generative task of guessing pixels" to "a task of predicting the abstract structure of the world," and it has great potential to connect with predictive world models for robotics and autonomous driving going forward.

Fourth, the combination with efficient adaptation techniques. Since fully fine-tuning a pretrained giant representation every time is expensive, efficient adaptation (PEFT) that updates only a few parameters — such as LoRA, adapters, and prompt tuning — pairs with self-supervised pretraining. In short, a division of labor of "pretrain broadly with self-supervision → adapt cheaply with lightweight techniques" is taking hold, and this has become the core operating model for practically running SSL in industry.

On the industrial-application side, SSL exerts particular value in specialized domains where labels are expensive. In medicine, data scarcity is overcome by pretraining on a large volume of unlabeled X-rays and CTs and then fine-tuning with a small amount of diagnostic labels; in finance, normal patterns are learned from unlabeled transaction logs to detect abnormal transactions and fraud. In manufacturing, masked-reconstruction self-supervision is applied to process-sensor and equipment logs to serve as the representation base for predictive maintenance. The commonality is that all are environments where "labels are few but raw data is abundant," and this is the reason SSL establishes itself as the basic strategy of applied AI.

6. Considerations and Implications

The strategic considerations from a professional engineer's perspective when applying SSL to organizations and systems are as follows.

First, the trade-off of pretraining cost versus transfer effect must be designed. Large-scale self-supervised pretraining demands enormous GPU, data, and energy, so not every organization needs to train from scratch. A staged strategy of reusing open foundation models (DINOv2, BERT, CLIP), doing additional self-supervised adaptation (continued pretraining) with one's own domain data, and then fine-tuning with a small amount of labels is cost-efficient. Pretraining from scratch is justified only when domain specificity is large and data is sufficient.

Second, preventing representation collapse and shortcut learning governs quality. Like the collapse of the non-contrastive type and the shortcuts of handcrafted tasks, in SSL a design failure can lead to "a representation whose surface loss is low but which is useless." Therefore an evaluation system that independently verifies representation quality through linear probing and downstream-task performance must always run in parallel, and hyperparameters such as data augmentation and masking ratio must be tuned carefully.

Third, one must guard against data bias and governance. Because SSL has no label-review process, the bias, toxicity, and copyright problems of large-scale data scraped from the web are imprinted directly into the representation. Data governance such as source management, filtering, personal-information de-identification, and license checks is the premise of model trustworthiness, and it must be managed in connection with AI trustworthiness and explainability requirements.

Fourth, the standardization of evaluation/transfer and MLOps integration. Since self-supervised representations are reused across many downstream tasks, representation version management, transfer-performance tracking, and drift monitoring must be integrated into the MLOps pipeline to secure reproducibility and operational stability. Only when combined with a model registry that registers and manages pretrained models as assets is the organization-wide reuse effect maximized.

Fifth, the verification of downstream-task suitability and guarding against overexpectation. Self-supervised representations do not always surpass supervised learning; when labels are sufficient and the task is narrow, a model trained supervised from scratch may be better. Also, if the pretraining data distribution and the downstream task distribution differ greatly (domain gap), the transfer effect plummets. Therefore one must guard against the overexpectation that "self-supervision is a panacea" and have a decision-making system that selects the appropriate strategy among supervised learning, self-supervised learning, and use of external foundation models based on label availability, domain proximity, and cost.

Sixth, the outlook for integration with related technologies. Since SSL is the common underlying technology of embedding, transfer learning, foundation models, RAG, and multimodal AI, it must be recognized and invested in not as an individual technique but as the representation layer of the AI platform. Going forward, the center of gravity is expected to shift from pixel-reconstruction type to representation-prediction type (JEPA), from single modality to unified multimodal, and from manual fine-tuning to prompt and in-context adaptation.

References

In one line: Self-supervised learning is a paradigm that obtains general-purpose representations by generating its own learning signal from the data itself without human-assigned labels, and through predictive, contrastive, non-contrastive, and masking methodologies it has become the pretraining engine of foundation models such as GPT, BERT, and DINO — a core strategy of modern AI that bypasses the "labeling bottleneck."