IM Engineer Study Notes
← Back to list
Infrastructure & Cloud
#중복제거#청킹#콘텐츠주소화#스토리지최적화#백업
Last updated · 2026-10-13

Data Deduplication

1. Overview

A. Definition

Data deduplication is a capacity-optimization technique that splits the data to be stored into units (chunks), compares each chunk's hash fingerprint, and stores only one physical copy of identical chunks while replacing the rest with reference pointers, thereby eliminating redundant storage.

The essence of deduplication is to implement the simple principle "never write the same content twice" through a mechanism called content addressing. Traditional storage identifies data by its location (address), whereas a deduplication system identifies data by its content (hash). An identical byte sequence has the same fingerprint no matter how often or where it appears, so the system looks the fingerprint up in an index to decide whether the chunk is already stored and records only new chunks. As a result, the original data is logically preserved intact, while physically only a set of unique chunks and the reference metadata that points to them remain.

This technique is easily confused with compression, but it operates at a different layer. Compression reduces statistical redundancy (entropy) within a single data block, whereas deduplication eliminates macroscopic redundancy across multiple blocks, files, and systems. The two are not mutually exclusive and are usually used together: deduplication first keeps only unique chunks, and those chunks are then compressed in a two-stage optimization that maximizes the overall reduction ratio.

Another way to understand deduplication is as a separation of the logical and physical views of data. To applications and users, every file appears to exist in full (the logical view), while the actual medium holds only unique chunks and reference metadata (the physical view). The file recipe bridges this gap, and the reliability of a deduplication system ultimately depends on how safely it maintains this mapping metadata. If the metadata is corrupted, the original cannot be reassembled even if the unique chunks are intact, so protecting metadata is as important as protecting the data.

B. Background and Necessity

The primary reason deduplication rose to prominence is that the growth rate of data has outpaced the price decline of storage media. In virtualized, cloud, and collaborative environments, nearly identical OS images, repeatedly transmitted mail attachments, and daily full backups that barely change are copied endlessly. For example, if 100 virtual machines (VMs) use the same OS, tens of GB of the OS region are stored 100 times; with deduplication, only one copy is physically stored and the remaining 99 are replaced by references.

Backup in particular is deduplication's most powerful use case. When a full backup is performed daily, the change delta is usually only 1-5%, yet the entire capacity is re-stored anyway. With deduplication, the 97-99% of chunks that did not change are eliminated, yielding a reduction ratio of 10:1 to as much as 30:1 or more depending on the backup environment. This translates directly into savings in disk purchases, data-center rack space, power and cooling costs, and network bandwidth for remote replication.

A new demand has recently been added in the form of ransomware defense. Combining immutability and retention lock on a deduplication-based backup appliance lets you keep many recovery points that cannot be changed for a set period, even if an attacker tries to encrypt or delete the backups. Thanks to deduplication, more recovery points can be kept longer within the same capacity, increasing the likelihood of recovering to a clean point in time.

Fundamentally, the need for deduplication is a matter of storage economics and data-movement cost. At the multi-PB scale, a 2x difference in reduction ratio means a difference of thousands of disks and the operating costs attached to them; and in remote replication for disaster recovery (DR), "deduplicate, then transmit" source deduplication becomes a decisive means of overcoming WAN bandwidth shortages. In other words, deduplication has established itself not merely as a capacity-saving technique but as a foundational technology for economically storing and cheaply moving exploding volumes of data.

C. Key Characteristics

The character of deduplication can be summarized in four points. First, content-based identification — data is identified by the hash of its content rather than its location, so identical content automatically converges into one. Second, dependence on reference integrity — because many references point to a single unique chunk, corruption of that one chunk propagates to the corruption of all data that references it. Third, read/write asymmetry — writes incur the extra operations of chunking, hashing, and index lookup, while restores must gather scattered chunks and thus suffer a fragmentation penalty. Fourth, metadata intensity — smaller chunks raise the reduction ratio but enlarge the fingerprint index and reference tables, increasing memory and compute burden. These characteristics form the basis for judging "when and in what manner to apply deduplication," discussed later.

The key metric quantifying the effect of deduplication is the deduplication ratio, usually defined as "logical capacity ÷ physical capacity" and expressed as 10:1, 20:1, and so on. However, the ratio is extremely sensitive to data characteristics, swinging from 2:1 to 50:1 on the same device depending on the workload, so rather than trusting a vendor's stated maximum, a prior assessment with one's own data is essential. Moreover, deduplication and compression are often reported together as a combined "data reduction ratio," so one must clearly distinguish whether the metric refers to pure deduplication alone or includes compression and thin provisioning before comparing.

2. Operating Principles and Architecture of Deduplication

A. Chunking and Fingerprinting

The first step of deduplication is to split the data stream into chunks, the units of comparison. The chunking method is the most important design decision governing deduplication's performance and reduction ratio. The simplest fixed-size chunking cuts data into fixed sizes such as 4KB or 8KB. It is simple and fast to implement, but has a fatal weakness: if even a single byte is inserted near the front of a file, all subsequent boundaries shift (boundary shift) and every chunk thereafter becomes different.

Content-Defined Chunking (CDC), i.e. variable-length chunking, solves this. CDC determines boundaries from the content itself. It slides a window while computing a rolling hash (e.g. Rabin fingerprint) and marks as a chunk boundary the point where that value satisfies a specific condition (e.g. the lowest n bits are zero). This way, even when data is inserted or deleted, the boundary stays anchored to the content, so only the changed neighboring chunks differ while the rest preserve their fingerprints, maintaining a high redundancy-detection rate. That is why CDC is effectively the standard for workloads such as backup and synchronization where small changes recur.

Each split chunk has its fingerprint computed with a cryptographic hash function (such as SHA-256). This fingerprint becomes the chunk's global identifier, and when an identical fingerprint is found in the index, the chunk is judged a duplicate. In theory a hash collision, in which different chunks have the same fingerprint, is possible, but SHA-256's collision probability is astronomically low (on the order of 2^-128), so in practice one assumes "if the fingerprints match, the contents match." That said, conservative designs sometimes add a byte-level verification after a fingerprint match to eliminate collision risk at the source.

The choice of hash function is also a trade-off between performance and safety. SHA-256 has strong collision resistance but is computationally costly, which can become a bottleneck in a high-speed data path. Some systems therefore use fast yet secure hashes such as BLAKE2/BLAKE3, or adopt two-stage verification that first screens candidates with a fast non-cryptographic hash and then confirms with a strong hash or byte comparison. Conversely, weak hashes such as MD5 and SHA-1 allow deliberate collision generation, so if an attacker disguises different content as the same fingerprint and overwrites one with the other, there is a risk of data poisoning (hash collision attack); they should be avoided in security-sensitive environments.

flowchart LR
  IN["Input data stream"] --> CHUNK["Chunking (CDC rolling hash sets boundaries)"]
  CHUNK --> HASH["Compute per-chunk fingerprint (SHA-256)"]
  HASH --> IDX{"Exists in fingerprint index?"}
  IDX -->|"Yes (duplicate)"| REF["Record reference pointer only + increment ref count"]
  IDX -->|"No (new)"| STORE["Store unique chunk + register in index"]
  REF --> META["Update file-to-chunk mapping (recipe)"]
  STORE --> META
  META --> OUT["Logical view stays identical to original"]

B. Index/Reference Management Architecture and the I/O Path

The heart of a deduplication system is the fingerprint index and the file recipe. The fingerprint index maps "where is the chunk with this fingerprint stored," and the file recipe is a reference list recording "which chunks, in what order, this file is concatenated from." On restore, chunks are gathered along the recipe to reassemble the original. Thanks to this structure, logical capacity does not change while only physical capacity shrinks.

The biggest practical challenge is index size. Handling 1PB with 8KB chunks yields billions of fingerprints, pushing the index into the hundreds of GB, so it cannot all be held in memory. To address this, a Bloom filter quickly screens out "definitely absent" to reduce disk lookups, and techniques that group similar fingerprints with locality to raise cache hit rates (e.g. Data Domain's SISL, Stream-Informed Segment Layout) are used. Mitigating this disk bottleneck is the core competitive edge in the performance of commercial deduplication appliances.

sequenceDiagram
  participant C as Client/backup server
  participant D as Deduplication engine
  participant I as Fingerprint index
  participant S as Chunk store
  C->>D: Data write request
  D->>D: Chunking + fingerprint computation
  D->>I: Fingerprint lookup (Bloom filter pre-check)
  I-->>D: Existence response
  alt Duplicate chunk
    D->>I: Increment reference count
  else New chunk
    D->>S: Store chunk (optional compression)
    D->>I: Register in fingerprint index
  end
  D-->>C: Write complete (recipe recorded)

Deletion is managed by reference counting. Because many files share a chunk, deleting one file cannot delete the actual chunk until its reference count reaches zero. The task of reclaiming chunks whose reference count has reached zero is garbage collection (GC), performed as a heavy background job based on mark-and-sweep because it must safely track the reference relationships of billions of chunks. The fact that space is not returned immediately upon deletion is an important operational characteristic of deduplication.

C. Scalability and the Limits of Global Deduplication

Single-node deduplication eliminates redundancy only within the data that node has seen. However, to implement global dedup across all nodes in a scale-out storage system of tens of nodes, the entire cluster must share which fingerprints exist on which node, so inter-node fingerprint-lookup traffic explodes. For this reason many systems use fingerprint sharding, routing data to a specific node by its fingerprint hash so the same fingerprint always goes to the same node, or compromise with local deduplication that eliminates redundancy only within a node. In other words, deduplication's reduction ratio and system scalability inherently conflict, and determining the balance point between them is the crux of distributed deduplication design.

Chunk size selection is also directly tied to this scalability problem. For example, managing 100TB of unique data with an average 8KB chunk yields about 13 billion fingerprints; the SHA-256 fingerprints alone (32 bytes) exceed 400GB. Adding reference and location metadata makes it hard to keep resident in memory, so the Bloom filter and locality cache described earlier become essential to minimize disk lookups. Conversely, enlarging chunks to 64KB or 128KB reduces metadata, but even a small change causes a whole large chunk to be stored anew, lowering the reduction ratio. Tuning this reduction-ratio / metadata / performance triangular trade-off to the workload's characteristics is the essence of practical tuning.

3. Classification

Deduplication is classified along three axes — processing time, where it is performed, and the unit of comparison — and each choice entails trade-offs in performance, reduction ratio, and resource usage. The table below is a supplementary aid for comparison; why each axis produces its differences is explained in prose after the table.

Axis Type Characteristics Suitable environment
Processing time Inline Eliminated in real time before storage, write latency incurred Flash, ample CPU
Post-process Stored first, eliminated asynchronously, needs staging space Write performance first
Location Source-side Eliminated at the client before transmission, saves bandwidth Remote backup, WAN
Target-side Eliminated at the storage device, no client burden LAN, dedicated appliance
Comparison unit File-level (SIS) Whole-file hash comparison, light but low reduction Mail attachments, documents
Block/byte-level Chunk-level comparison, high reduction but heavy compute Backup, VM images

The choice of processing time is a trade-off between write latency and space efficiency. The inline method eliminates duplicates before data touches disk, so it needs no extra temporary space and yields immediate reduction, but chunking, hashing, and index lookup intrude on the write path and can increase latency. The post-process method stores the original quickly first and eliminates duplicates during idle time, so write performance is good, but it requires a separate staging space (landing zone) to hold the original before elimination. On high-performance storage such as flash and NVMe with ample CPU headroom, inline has become the mainstream.

The choice of the two methods ultimately depends on the nature of the workload. In environments with a tight backup window where write throughput is the top priority, post-process is advantageous for advancing the backup completion time; in environments sensitive to rack and power cost or where securing staging space is difficult, inline is advantageous. Some commercial products combine both, providing an adaptive policy that behaves like post-process during peaks and switches to inline-level behavior during idle time. Thus processing time is not a fixed right answer but a design variable chosen and adjusted to operational goals (RTO, backup window, TCO).

The choice of location is a trade-off between network bandwidth and client load. Source deduplication eliminates duplicates at the client before transmission and sends only unique chunks over the WAN, dramatically saving bandwidth in remote backup. However, the client bears the chunking and hashing computation. Target deduplication processes at the storage device (appliance), so the client is light, but the full volume including duplicates traverses the network. In environments where bandwidth is scarce, such as branch-to-headquarters backup, the source method is common, while on an intra-data-center LAN the target method is common.

The choice of comparison unit is a trade-off between reduction ratio and compute volume. File-level deduplication (SIS, Single Instance Storage) eliminates only when entire files are identical, so it is light but cannot see a duplicate if even one byte within the file differs. Block/byte-level sharing applies even when only part of a file is identical, so the reduction ratio is far higher, but the chunks and metadata to manage explode. The smaller the chunk size, the higher the reduction and the greater the index burden — the typical reduction-ratio / metadata trade-off appears.

4. Comparison and Application Cases

A. Deduplication vs. Compression — Why Use Them Together

Deduplication and compression are not competitors but complements. Compression locally removes repeated patterns within a single block, so it is very effective on low-entropy data such as text and logs, but it cannot recognize the duplication of 100 identical files scattered apart. Conversely, deduplication eliminates wide-area identical chunks but does not touch the duplication within a chunk. That is why commercial storage applies them in two stages in the order deduplication → compression, first keeping only unique chunks and then compressing those chunks. In practice it is common to achieve, say, 20:1 total reduction by getting 10:1 with deduplication alone in a VM backup environment and then an additional 2:1 with compression.

There is a reason the order is deduplication first. If you compress first, even an identical original can produce slightly different compression results depending on block boundaries and dictionary state, making duplicate detection harder. Therefore the order "first group and eliminate chunks with identical content by fingerprint (deduplication), then compress the remaining unique chunks" is advantageous for the reduction ratio. That said, data already highly compressed and randomized such as JPEG, video, and ciphertext gains little from either deduplication or compression while only incurring compute cost, so it is reasonable to have a policy (dedup/compression bypass) that excludes such data types from reduction.

B. Concrete Cases and Figures

As a representative commercial case, Dell EMC's Data Domain line has combined inline, variable chunking, and SISL index locality to publish reduction ratios in the 10-55:1 range for backup workloads, and this is the core technology that opened the purpose-built backup appliance (PBBA) market. In open source, ZFS provides block-level inline deduplication, but its index memory cost is so high that the rule of thumb "about 5GB of RAM per 1TB of chunks" is common, and cases of performance degradation from indiscriminate enablement are frequently reported. The backup software Veeam combines source/target deduplication and compression to routinely achieve large reduction in incremental backups. Cloud backup services likewise use source deduplication to reduce transmission and storage volume, making it a core means of directly lowering network and storage costs under usage-based pricing.

The container ecosystem's Docker image layers can also be seen as a kind of content-addressing-based deduplication. When several images share an identical base layer (e.g. the same Ubuntu layer), that layer is stored once on disk and in the registry and shared by reference. Likewise, Git's object store is a classic example of content addressing, identifying blobs by content hash (SHA-1/256) and keeping identical content only once. In other words, the principle of deduplication is deeply embedded across version control and cloud-native software design, beyond dedicated storage devices.

C. Virtual Desktop (VDI) and Mail Archiving Cases

Virtual Desktop Infrastructure (VDI) is a representative case where deduplication's effect is dramatically evident. In VDI, where thousands of users use nearly identical OS and office images, applying block-level deduplication means the common OS region is effectively stored only once physically, yielding tens-to-one reduction, with the side effect of even improved performance as common blocks concentrate cache hits during a boot storm. In mail archiving as well, identical attachments that used to be replicated for as many recipients converge into one through file/block-level deduplication, greatly lowering the long-term retention cost for regulatory compliance. Thus the more a structure consists of "many using nearly the same data," the greater the return on investment (ROI) of deduplication.

5. Deep Dive — Latest Trends and Expanding Applications

Deduplication technology is evolving in four directions. First, the move to all-flash/NVMe storage. In the past, deduplication suffered performance losses on HDDs due to the fragmentation of chunks scattered across the disk on restore, but on flash with virtually no random-access latency this penalty has effectively disappeared. As a result, keeping inline deduplication always on has become standard even on primary (production) all-flash arrays, and Pure Storage, NetApp, and others provide data reduction bundling deduplication + compression + thin provisioning as a default feature. Furthermore, since flash has finite write endurance (wear), deduplication, which eliminates duplicate writes at the source, reduces actual written volume and thereby contributes even to extending SSD lifespan, a factor that supports always-on application in production as well.

Second, tension with security and privacy. In multi-tenant cloud environments, cross-user dedup saves a great deal of space but creates side-channel attacks in which whether a file already exists on the server is revealed by response speed, and a conflict with end-to-end encryption because server-side deduplication requires seeing data in plaintext. To mitigate this, convergent encryption, which makes identical plaintext produce identical ciphertext, and proof-of-ownership (PoW) techniques are being researched and applied. However, convergent encryption has the limitation of being vulnerable to known-plaintext (dictionary) attacks, so the trade-off between security and space efficiency must be evaluated carefully.

Third, expansion to AI and large-scale datasets. Large language model training data contains large amounts of duplicate and near-duplicate documents collected by web crawling, and recent research reports that deduplicating training data contributes directly to model quality improvement (reduced memorization, better generalization) and training efficiency, beyond saving storage cost. Also, near-dedup, which groups data that is similar though not entirely identical, and hybrid reduction combined with delta encoding are expanding in the backup and archive domains. Thus deduplication is broadening its scope from a storage-cost-optimization technique to a data-quality and governance technique.

Fourth, computing storage/offloading. Chunking, hashing, and compression are CPU-intensive operations that burden the response latency of primary storage. To mitigate this, approaches that offload deduplication computation to a DPU (Data Processing Unit) or computational storage drives (CSD) with built-in compute functions have emerged. This lets the host CPU focus on application processing while dedicated hardware handles data reduction, further lowering the performance cost of "always-on deduplication" in all-flash environments. This is a trend of deduplication shifting from a software function to a hardware-accelerated function on the data path, understandable in the same context as encryption and erasure-coding offloading.

6. Considerations and Implications

Deduplication looks like "free capacity savings," but in reality it is a technology that demands design judgments spanning performance, integrity, security, and governance.

The strategic items to consider when adopting and operating data deduplication from a Professional Engineer's perspective are as follows.

  • Selective application based on data characteristics (application strategy): Not all data is a deduplication candidate. Data with much redundancy such as backup, VM, and file shares has a large effect, while already-compressed/encrypted media or unique measurement data has a low reduction ratio and only incurs compute cost. Assess the reduction ratio per workload in advance to apply selectively, and design by distinguishing the method (inline/post-process) suited to the primary vs. backup tier.

  • Quantifying the performance-capacity trade-off (trade-off): Reducing chunk size raises the reduction ratio but increases index memory and CPU burden and restore fragmentation. For operations with a strict RTO (recovery time objective), a balance is needed that prioritizes restore performance over excessive reduction by enlarging chunks or choosing post-process. As the ZFS "5GB RAM per 1TB" case shows, neglecting resource sizing can instead collapse performance.

  • Data integrity and single-failure propagation risk (availability): Because many data share a few unique chunks, corruption of one chunk spreads to wide-ranging data loss. It must always be paired with underlying data protection such as end-to-end checksums, periodic scrubbing, RAID/EC/replication, and deduplication must be confined to "storage efficiency" rather than a "replacement for backup," combined with the 3-2-1 backup principle.

  • Security, regulation, and deletion assurance (governance): In a reference-sharing structure, deleting one file does not immediately lead to physical deletion, which can conflict with the requirement for complete destruction of personal data (right to erasure). The GC cycle and destruction-evidence framework must be made clear, and in multi-tenant environments a policy such as confining the scope within a tenant, after evaluating the side-channel and encryption conflicts of cross-tenant deduplication, is needed.

  • Outlook and related technologies (outlook): Deduplication is broadening its scope into a data-reduction platform bundled with compression, thin provisioning, and tiering, and into a data-quality management stage of the AI data pipeline. Surveying together the offloading to computing storage and DPUs and the combination with erasure coding and CDP (Continuous Data Protection) widens the field of view for technology selection.

References

In one line: Data deduplication is a content-addressing-based capacity-optimization technique that splits data into chunks, identifies them by hash fingerprint, and replaces identical chunks with references to reduce physical capacity; the design of chunking, indexing, and reference management must be considered together with the trade-offs of performance, integrity, and security.