Medallion Architecture

The Bronze layer is the initial landing zone for all incoming data. It preserves the original structure and fidelity of the source data in an append-only, immutable format. Key characteristics include:

• Stores data in its native format (JSON, CSV, Parquet) without transformation
• Acts as a full historical archive, enabling reprocessing from the source at any time
• Provides a single source of truth for data lineage and audit trails
• Minimizes the load on source systems by decoupling ingestion from transformation
• Commonly implemented using Change Data Capture (CDC) for streaming sources

The Bronze layer prioritizes data retention and fidelity over query performance, making it essential for compliance and data reprocessing scenarios.

The Silver layer contains cleansed, deduplicated, and validated data suitable for exploratory analysis. This is where data transitions from an operational record to an analytical asset. Core processes include:

• Schema enforcement and data type standardization across sources
• Deduplication and handling of slowly changing dimensions (SCDs)
• Application of data quality rules and null-value handling
• Joining and conforming related entities from disparate source systems
• Preserving column-level lineage to track every transformation applied

The Silver layer eliminates the chaos of raw data, providing data scientists and analysts with a query-ready, trusted foundation without the noise of ingestion artifacts.

The Gold layer contains business-level aggregates, feature stores, and denormalized views optimized for specific consumption patterns. This is where data becomes a product. Key attributes:

• Pre-computed key performance indicators (KPIs) and metrics for dashboards
• Feature stores for machine learning models with point-in-time correctness
• Denormalized star schemas for business intelligence tools
• Aggregations that reduce query latency for real-time applications
• Data marts tailored to departmental or application-specific needs

The Gold layer is designed for read-optimized performance, serving as the interface between the data platform and its consumers, from C-suite dashboards to production ML models.

The Medallion Architecture enforces a unidirectional flow of data quality improvement. Each layer adds structure and trust:

• Bronze: Guarantees data completeness—nothing is lost or altered from the source
• Silver: Guarantees data correctness—values are validated, types are consistent, and duplicates are resolved
• Gold: Guarantees data fitness—aggregations and features are accurate for their intended business use case

This progression enables idempotent reprocessing: if a bug is found in Silver logic, the pipeline can re-read from the immutable Bronze layer and recompute all downstream tables without data loss. The architecture naturally supports time travel queries when built on table formats like Delta Lake or Apache Iceberg.

A foundational principle of the Medallion Architecture is the physical or logical separation of compute and storage. This design enables:

• Independent scaling: Storage grows with data volume; compute scales with query demand
• Multi-engine access: Bronze, Silver, and Gold tables can be queried by Spark, Trino, or Python using the same underlying files
• Cost optimization: Cheap object storage (S3, ADLS) holds all layers while ephemeral compute clusters process transformations
• Concurrent workloads: ETL jobs, ad-hoc queries, and ML training run simultaneously without resource contention

This separation is typically implemented using open table formats like Apache Iceberg or Delta Lake on cloud object storage, avoiding vendor lock-in and enabling a true data lakehouse architecture.

The Medallion Architecture differs from traditional Extract-Transform-Load (ETL) and Extract-Load-Transform (ELT) patterns in its multi-stage design:

• Traditional ETL: Transforms data in-flight before landing, risking data loss if logic changes
• Traditional ELT: Lands raw data and transforms in-place, but often lacks the disciplined staging layers
• Medallion: Provides explicit, immutable staging at each quality tier, enabling:
  • Reprocessing from any layer without re-ingesting from source
  • Graduated access control: Analysts query Silver; executives query Gold
  • Clear ownership boundaries between data engineering (Bronze→Silver) and analytics engineering (Silver→Gold)

This pattern is particularly well-suited for data mesh implementations where domain teams own their Gold products while a central platform team manages Bronze ingestion.