Medallion Architecture

1. Overview

Definition: Medallion Architecture is a data design pattern that progressively refines and structures data inside a data lakehouse into Bronze, Silver, and Gold layers according to quality and degree of processing — a multi-hop refinement model in which each successive layer raises the data's integrity, business meaning, and usability.

Enterprise data architecture has long been torn between two extremes: the strict schema-on-write of the data warehouse (DW) and the flexible schema-on-read of the data lake. The DW delivered high quality and governance for structured data but struggled with semi-structured/unstructured data and large-scale, real-time processing, while the data lake stored all forms of data cheaply yet, lacking quality and consistency control, frequently degenerated into a data swamp. When untrustworthy data piles up in the lake, analysts must re-validate the source every time, and the organization loses a shared "Single Source of Truth," so no one agrees on the same numbers.

The lakehouse combines the strengths of both, layering DW-grade reliability features such as ACID transactions, schema enforcement, and version management on top of the lake's low-cost object storage. But having a lakehouse storage foundation does not make data organize itself. What is needed is a disciplined, standardized flow on which the organization agrees — "up to where is the data raw as sourced, from where has it been validated, and what is the final consumable." Medallion Architecture is precisely the design convention that provides this discipline, popularized by Databricks as the model pattern for the lakehouse.

The pattern's core value lies in progressive refinement. Named after Olympic medal grades (bronze–silver–gold), data enters the least-processed Bronze when ingested from the source, is promoted to Silver through cleansing and integration, and is finally completed as golden Gold through business aggregation and modeling. Making these stages explicit achieves, all at once, immutable preservation of source data (auditable and reprocessable), reuse of intermediate outputs, and performance optimization of the consumption layer, greatly improving the pipeline's reliability and maintainability.

2. Overall Structure and Design Principles

Medallion Architecture has a pipeline-shaped structure in which data ingested from source systems (OLTP DBs, logs, IoT, APIs, etc.) passes sequentially through three layers while value accumulates. Each layer is physically stored as a separate table (or storage path), and the upper layer takes the layer immediately below as input. The overall structure is as follows.

graph LR
  subgraph SRC["Source systems"]
    S1["OLTP DB"]
    S2["Logs/Events"]
    S3["IoT/API"]
  end
  subgraph LH["Lakehouse"]
    B["Bronze<br/>ingest as-is from source"]
    SV["Silver<br/>cleanse/validate/integrate"]
    G["Gold<br/>aggregate/business model"]
  end
  subgraph CON["Consumption layer"]
    C1["BI dashboards"]
    C2["ML train/inference"]
    C3["Ad-hoc analysis"]
  end
  S1 --> B
  S2 --> B
  S3 --> B
  B -->|"Cleansing"| SV
  SV -->|"Aggregation"| G
  G --> C1
  G --> C2
  SV --> C2
  G --> C3

The design principles that underpin this structure are summarized as follows.

• Immutability & replayability. Because Bronze preserves the source without processing, when a bug is found in an upper layer's transformation logic or requirements change, the entire chain can be rebuilt by re-reading Bronze. In other words, Bronze serves as the pipeline's "backup and replay log."
• Separation of concerns. By splitting the distinct responsibilities of ingestion, quality, and serving across layers, each stage can be developed, tested, and optimized independently.
• Progressive quality increase. The higher the layer, the higher the quality and cohesion. Lower layers are general and raw; higher layers are specialized to specific business purposes.
• Single source of truth. Because all Gold tables share Silver as a common cleansed foundation, inter-departmental numeric discrepancies are prevented and organization-wide consistent metrics are guaranteed.

Thanks to these principles, Medallion naturally satisfies the modern data-engineering demand to "store fast but consume carefully." The source is received into Bronze without delay to prevent data loss, and consumption is forced to occur only in validated upper layers, reconciling the conflicting demands of ingestion agility and usage reliability through layer separation.

Note that Medallion does not necessarily mean "exactly three layers." Both Microsoft's and Databricks' guidance state that the pattern is flexible and that the number and names of layers can be adjusted to the organization's needs. In practice, teams add a 'Raw/Landing' area to receive raw files before Bronze, or re-split Gold into purpose-specific (BI, ML) marts. The key is not the number of layers but explicit separation by level of processing and organizational agreement on it.

3. Role of Each Layer and Data Flow

Expressing in detailed process form what transformations the actual data undergoes in each layer gives the following.

flowchart TB
  R["Source data (structured/semi/unstructured)"] --> I["Incremental Ingestion"]
  I --> B["Bronze: append-only load<br/>+ attach load metadata (source, timestamp)"]
  B --> V{"Data quality validation"}
  V -->|"Pass"| SV["Silver: dedup/type-conform/join/integrate"]
  V -->|"Fail (quarantine)"| Q["Quarantine table"]
  SV --> A["Business aggregation/dimensional modeling (star schema)"]
  A --> G["Gold: aggregate metrics/features/marts"]
  G --> O["Consumption: BI/ML/reports"]

A. Bronze layer (source-preservation layer). Bronze is the landing point that loads data ingested from source systems as-is, as far as possible. This layer's design philosophy is "do not transform." It minimizes schema changes and cleansing, and instead attaches metadata such as load timestamp, source identifier, and file path as additional columns so that data lineage can be traced. The load method defaults to append-only, which does not overwrite existing records, in order to preserve even the source's change history and enable audit and time travel.

The reason for preserving Bronze in its original form is reprocessing flexibility. For example, if a bug in the Silver transformation logic miscalculated the last six months of metrics, recovery is possible by replaying only Bronze without accessing the source system again. Had cleansing already been done at the ingestion stage, the original information would be lost and such recovery would be impossible. Taking an e-commerce company as an example, storing the order API's JSON responses whole in Bronze without parsing or processing means that even when a new requirement such as "refund rate by payment method" arises later, it can be extracted immediately from past data.

B. Silver layer (cleansing/integration layer). Silver is the layer that validates, cleanses, and integrates Bronze's raw data to build a 'trustworthy enterprise view.' The core transformations performed here are (1) handling missing/outlier values and removing duplicates, (2) standardizing data types, units, and code values, and (3) joining multiple sources on common keys to integrate entities (customer, product, order). Records that fail quality validation are not discarded but separated into a quarantine table to analyze the cause — this is best practice. Doing so prevents bad data from propagating upward while still tracking it without loss.

Silver matters because this layer becomes the common foundation shared by most data consumers. If Gold is specialized to a particular department's particular metric, Silver is a general-purpose asset that is "cleansed but not yet biased toward a specific use." Machine-learning engineers often use Silver directly rather than Gold for free feature exploration, because Gold's aggregation loses the original's fine-grained information. In finance, integrating account/transaction data from multiple core-banking systems into a single standard transaction model in Silver means that diverse Gold outputs for risk, marketing, regulatory reporting, and so on are subsequently built consistently on this single foundation.

C. Gold layer (business-consumption layer). Gold is the top layer that holds 'business-ready' data that has been aggregated and modeled to fit a specific business purpose from Silver's cleansed data. Typically it is composed of pre-aggregated metrics (daily sales, KPIs) so BI tools can query quickly, analysis-friendly dimensional models (fact and dimension tables of a star schema), or feature tables that ML models can use directly for training. Because Gold's design goals are read performance and ease of use, the strategy is to pre-perform joins to denormalize and to materialize frequently used aggregations.

It is also important that having multiple Golds by purpose is common. The finance team's 'monthly P&L mart,' the marketing team's 'campaign performance mart,' and the data-science team's 'churn-prediction feature set' are all different Golds derived from the same Silver. This independently accommodates each department's needs while, because the underlying data shares the single foundation of Silver, numeric discrepancies are prevented. The table below compares the characteristics of the three layers.

Category	Bronze	Silver	Gold
Data state	raw	cleansed	curated
Main transform	load/metadata attach	dedup/standardize/join	aggregate/dim model/denormalize
Schema	source schema/loose	structured/enforced	business-oriented
Primary consumer	data engineers	ML engineers/analysts	BI/executives/business
Load method	append-only	upsert/merge	overwrite/aggregate
Quality level	low (preservation first)	medium (trust base)	high (finished)

4. Implementation Technologies and Operational Patterns

Because Medallion Architecture is a conceptual pattern it is not tied to a specific product, but open table formats play a core role in practical implementation. Formats like Delta Lake, Apache Iceberg, and Apache Hudi provide ACID transactions, schema evolution, time travel, and upsert (MERGE) on top of object storage, enabling each layer's tables to be updated reliably. For example, the MERGE INTO (upsert) operation that reflects only changed data in the Silver layer is hard to perform safely without Delta Lake's ACID guarantees. A typical implementation stack is composed of the following elements.

• Storage/format layer: object storage (S3, ADLS, GCS) + open table format (Delta, Iceberg, Hudi).
• Processing engine: distributed processing engines such as Spark, Flink, Trino and the SQL transformation tool dbt.
• Ingestion/incremental: CDC connectors, streaming ingestion (Auto Loader, Kafka), declarative pipelines (Delta Live Tables).
• Quality/validation: implement layer-transition gates with Great Expectations, DLT expectations, and the like.
• Governance/catalog: centrally manage lineage, permissions, and audit with Unity Catalog, OneLake, or an open catalog.

For efficient data flow, incremental processing is essential. Reprocessing all data every time explodes cost and latency, so CDC (Change Data Capture) that flows only the changes to lower layers is combined with streaming ingestion (e.g., Spark Structured Streaming's Auto Loader). Connecting each layer by streaming makes it possible to build a unified pipeline beyond Lambda that processes batch and real-time in a single codebase. To enforce quality discipline as code, data-quality framework expectations such as Delta Live Tables/dbt tests and Great Expectations are applied at layer-transition points to gate bad data from being promoted.

From an operational standpoint, each layer's physical optimization also matters. Bronze tends to generate many small files that degrade query performance, so periodic compaction that merges small files and partitioning/clustering (e.g., Z-Order, Liquid Clustering) matched to query patterns are applied. Also, while Bronze sets a retention period and cleans up (vacuum) old versions to optimize storage cost, Gold is queried frequently and so speeds up responses with caches and materialized views. The fact that each layer has opposing optimization goals (Bronze = preservation/cost, Gold = performance) is a core design point of Medallion operation.

Comparison: difference from traditional ETL/DW. Traditional DW was schema-on-write, fixing the schema before ingestion and cleansing via ETL before loading. Because this discards the original, reprocessing on requirement change is hard and it cannot hold unstructured data. Medallion, by contrast, follows the ELT (load first, transform later) philosophy, preserving the original (Bronze) first and deferring and distributing transformation in layers. As a result it has an edge in flexibility (reprocessing, accommodating new requirements) and scalability (low-cost storage). However, there is a trade-off: many layers cause storage duplication and latency.

Comparison: relationship with Data Mesh. If Medallion is a technical pattern that prescribes the vertical refinement flow 'inside' a single lakehouse, Data Mesh is an organizational/governance paradigm in which per-domain teams own and supply data as a 'data product.' The two are not competing but complementary. In practice they are often combined by having each domain team implement its own data product internally in a Medallion structure (Bronze → Silver → Gold).

Comparison: difference from Lambda architecture. Real-time analytics was once implemented with a Lambda architecture that ran a batch layer and a speed layer separately and merged them at the serving stage, which carried the burden of maintaining the same logic redundantly in two codebases, batch and streaming. Medallion, by leveraging the ACID of open table formats and streaming tables, unifies batch and incremental streaming into a single layered pipeline, removing code duplication and applying the same refinement rules consistently to both. This is close to the 'Kappa-oriented' single-pipeline philosophy.

Concrete (quantitative) case. Taking a global retailer's demand-forecasting pipeline as an example, POS transaction logs from hundreds of stores (hundreds of millions of events per day) and weather/promotion data are loaded as-is into Bronze. In Silver, cancellations and refunds are offset, product codes are mapped to a standard classification scheme, and a join with the store master builds 'normalized daily sales facts.' In Gold, these are aggregated at the SKU × store × week level into a feature table fed to the demand-forecasting model. Organizations that adopted this structure report that removing preprocessing-code duplication and using incremental processing cut daily pipeline run time and reprocessing cost by tens of percent, and that, thanks to the single foundation of Silver, finance and logistics departments came to share the same sales figures (the specific numbers vary by organization and environment, so they should be understood as a generalized tendency).

5. Deep Dive: Recent Trends and Anti-patterns

Recently Medallion Architecture has been evolving along three axes: governance, automation, and real-time. First, integrated catalog/governance layers such as Databricks' Unity Catalog and Microsoft Fabric's OneLake are combined so that inter-layer data lineage, access control, and audit are centrally managed. Whereas Bronze/Silver/Gold used to be mere folder conventions, now the catalog tracks each layer table's owner, quality, and usage as metadata. Second, the direction is toward declarative pipelines such as Delta Live Tables (DLT) that define inter-layer transformations by declaration rather than code, with the framework automatically managing quality expectations, dependencies, and incremental processing. Third, 'streaming Medallion,' which connects streaming tables at every layer to flow near real-time from ingestion to Gold, is becoming common.

Conversely, the anti-patterns that frequently occur in practice must also be clearly recognized. Below are representative failure types repeatedly observed in the field.

• Over-processing Bronze. If cleansing/transformation is done at the ingestion stage and the original is lost, reprocessing becomes impossible on logic change, forfeiting replay capability — Medallion's core benefit.
• Skipping Silver (Bronze → Gold directly). Skipping the cleansing/integration layer removes the quality gate, and each Gold interprets the raw data on its own, so numeric discrepancies recur.
• Layering in name only. If per-layer responsibilities, quality standards (SLA), and owners are not documented and enforced, it becomes just another data swamp divided into three folders.
• Dogmatic three layers ignoring scale. Mechanically applying three layers even to small/simple data only increases needless duplication, latency, and cost. Adjusting the number of layers to data scale and organizational maturity is recommended.
• Unauthorized discarding of quality-failed data. Discarding validation-failed records without quarantine makes cause-tracing and correction impossible, so they must always be separated and preserved in a quarantine table.

6. Considerations and Implications

Medallion Architecture is not a mere folder partition but a strategy that institutionalizes data quality and governance as a discipline of flow. When reviewing adoption from a professional engineer's perspective, the following should be judged comprehensively.

• Application strategy (flexibility of layer design). Rather than dogmatically enforcing three layers, the number and names of layers should be adjusted to data characteristics and usage purpose. Tailored design reflecting organizational needs — such as adding a Landing for raw-file landing or splitting Gold into BI and ML uses — is key. For small/simple data, the judgment to reduce to two layers is also needed.

• Trade-off (quality vs. cost/latency). The more layers, the higher the reprocessing flexibility and quality control, but the same data is duplicated across layers, increasing storage cost and end-to-end latency. Operational design must proceed in parallel: reducing latency with incremental processing/streaming and managing cost with Bronze's retention, compaction, and file optimization.

• Outlook (integration with governance). Going forward, Medallion will settle in not as a standalone pattern but as part of a governance system combined with data catalog, lineage, access control, and data-quality automation. Once an integrated catalog certifies each layer's trust level as metadata, consumers can use "which layer and which validation this table has passed" as grounds for trust.

• Related technologies (table formats, CDC, quality frameworks). Medallion's effectiveness is realized when ACID-guaranteeing open table formats (Delta, Iceberg, Hudi), CDC/streaming that enable incremental processing, and a data-quality framework as a layer-transition gate are all in place. Splitting only layers without these loses the pattern's core advantages.

• Operations/observability. A system is needed that continuously monitors the success/failure of inter-layer transitions, quality metrics (missing rate, duplication rate), end-to-end latency, and data freshness, and alerts on SLA violation. The load-volume trend of the quarantine table can serve as a signal to detect source quality degradation early.

• Organizational/governance perspective. Per-layer responsible owners and quality standards (SLA) should be codified, and by combining with Data Mesh's domain-ownership concept, the owner, quality, and usage rules of each layer's data product should be defined. A technical pattern alone cannot prevent a data swamp; organizational agreement and operational discipline determine success or failure.

References

• OneLake medallion lakehouse architecture (Microsoft Learn): https://learn.microsoft.com/fabric/onelake/onelake-medallion-lakehouse-architecture
• Describe medallion architecture (Microsoft Learn): https://learn.microsoft.com/training/modules/describe-medallion-architecture/2-describe-medallion-architecture
• Demystifying the Medallion: Practical Insights into the Layers (O'Reilly): https://www.oreilly.com/library/view/building-medallion-architectures/9781098178826/ch02.html

---

In one line: Medallion Architecture is a multi-hop data design pattern that progressively refines lakehouse data into Bronze (source preservation), Silver (cleansing/integration), and Gold (aggregation/consumption) layers, securing both the reprocessing flexibility of the original and the reliability of consumption data.