Medallion Architecture Isn't One Pattern — It's Five

“We use medallion architecture” is the data engineering equivalent of “we use microservices.” It tells you almost nothing about the actual implementation. Bronze, silver, gold — three words that launch a thousand architecture diagrams, each looking completely different from the last.

After building medallion architectures across banking, manufacturing (four separate business domains), FMCG retail analytics, and regulated financial services, we’ve stopped treating medallion as a pattern and started treating it as a vocabulary. The words are the same. The implementations are not.

Here are five real variations we’ve built, what drove each design, and what we’d standardize if we were starting from scratch.

Variation 1: Config-Driven DQ with Quarantine Tables

Context: A banking institution ingesting CDC streams from core SQL Server systems through Kafka into a Delta Lake lakehouse.

The defining characteristic of this implementation is the silver layer’s data quality engine. Validation rules are defined in configuration files — JSON documents specifying null checks, range validations, regex patterns, referential integrity checks, and custom business rules per source system.

Records that pass validation flow to silver. Records that fail are routed to quarantine tables. Each quarantine record preserves the original data alongside failure metadata: which rule triggered, the severity level, and a timestamp. This dual-stream pattern means the pipeline never drops data — it classifies it.

The quarantine tables aren’t just an error dump. They feed a reconciliation workflow where data stewards review failures, correct source issues, and resubmit. In a regulated banking environment, this is not optional — you must account for every record.

What made this different: The DQ engine is entirely config-driven. Adding a new validation rule means editing a configuration file, not writing code. Non-engineers manage the rules. The pipeline doesn’t need redeployment for new rules — they take effect on the next run.

Silver layer decision: Schema enforcement with explicit quarantine. Every record is classified as clean or quarantined. No records are silently dropped or coerced.

Variation 2: Schema-on-Read with VARIANT Columns

Context: A regulated financial services platform ingesting deeply nested JSON from multiple upstream systems with frequently changing schemas.

The challenge here was forward compatibility. Upstream systems changed their JSON payloads regularly — new fields, restructured nested objects, deprecated attributes. A rigid silver schema broke every time. But a completely schemaless approach made gold-layer queries unreliable.

We landed on a hybrid: bronze stores raw JSON in Delta tables. Silver preserves the full payload in a VARIANT column (Databricks’ semi-structured type) while promoting frequently queried fields to typed columns. When upstream schemas change, the VARIANT column absorbs the new structure. Typed columns are added when a new field becomes analytically important — a conscious promotion decision, not an automatic one.

What made this different: The silver layer doesn’t attempt to fully normalize the data. It provides a stable query surface for known fields while preserving everything else for ad-hoc exploration. This is the right trade-off when you can’t control upstream schemas and breaking changes are a monthly occurrence.

Silver layer decision: Selective typing with full payload preservation. The team explicitly decides which fields get promoted to typed columns based on query patterns.

Variation 3: ParserFactory for Heterogeneous Sources

Context: A German manufacturing conglomerate with four distinct business domains, each with wildly different data sources: Excel reports from SharePoint, time-series data from Siemens Historian, crystallography files from lab instruments, and transactional data from SAP via OData.

You can’t apply a single silver transform to Excel files and industrial sensor data. They share nothing — not schema, not semantics, not cadence.

We built a ParserFactory framework: a base parser class with domain-specific implementations. Each parser is registered in a YAML configuration that maps source systems to parser classes. Azure Data Factory pipelines are parameterized — the same pipeline template handles ingestion across all four domains, selecting the appropriate parser based on configuration.

Bronze is genuinely raw: files landed as-is with ingestion metadata. Silver is where the ParserFactory applies domain-specific transforms — parsing Historian timestamps, normalizing SAP field names, extracting structured data from Excel templates with known layouts.

What made this different: Four domains, one architecture template, four completely different silver transforms. The ParserFactory is the abstraction that makes this manageable. Adding a new source type means implementing a parser class and adding a YAML config entry.

Silver layer decision: Domain-specific transforms behind a unified interface. The transform logic is completely different per domain, but the orchestration and monitoring are shared.

Variation 4: Multi-Model ML Consumption Layer

Context: A global FMCG company running 12+ ML models daily across 10K+ retail outlets and 100K+ SKUs.

This medallion architecture is defined by its gold layer, not its silver layer. Gold isn’t just “business-ready aggregations” — it’s the feature store and scoring input for a fleet of ML models. Demand forecasting, customer segmentation, compliance detection, pricing optimization, and KPI prioritization models all read from gold tables, run their scoring pipelines, and write results back to gold.

The circular dependency (gold feeds models, models write to gold) required careful orchestration. We built dependency DAGs that enforce execution order: data refresh first, then independent models in parallel, then dependent models (segmentation before prioritization), then composite outputs.

Model configurations live in OmegaConf YAML files — features, hyperparameters, training windows, and scoring schedules are all config-driven. Deployment goes through Databricks Asset Bundles.

What made this different: The gold layer is both a consumption layer and a production layer. Models are first-class consumers and producers. The medallion architecture extends beyond traditional analytics into ML operations.

Silver layer decision: Aggressive deduplication and entity resolution. With 40+ data sources providing overlapping information about the same outlets and SKUs, the silver layer’s primary job is reconciling conflicting data into canonical entities.

Variation 5: Auto Loader with DLT Expectations

Context: A financial services platform with streaming ingestion requirements and strict data quality SLAs.

This is the most “modern Databricks” implementation. Bronze uses Auto Loader for incremental file ingestion with schema evolution — new files are detected automatically, and schema changes are handled via mergeSchema. Silver and gold are implemented as Delta Live Tables (DLT) pipelines with @dlt.expect decorators defining data quality expectations.

DLT expectations handle what the config-driven DQ engine handles in Variation 1, but with a different trade-off: expectations are defined in code (Python decorators), not configuration files. This is simpler for engineering-led teams but harder for business-led rule management.

Streaming semantics run end-to-end: Auto Loader detects new files, triggers the DLT pipeline, and updates gold tables with exactly-once guarantees via checkpoint state. Latency from file landing to gold availability is minutes, not hours.

What made this different: Fully streaming, fully declarative. The pipeline definition is the documentation. DLT handles retries, checkpointing, and data quality metrics natively. Less custom code than any other variation.

Silver layer decision: Declarative expectations in code. Data quality is defined alongside transformations, not in a separate configuration layer.

The Silver Layer Is Where Architectures Diverge

If you take one thing from these five implementations: the silver layer is the architectural fulcrum. Bronze is always “land the raw data with metadata.” Gold is always “serve the business use case.” But silver — that’s where you decide:

• Schema enforcement: Strict typing vs. selective promotion vs. schema-on-read
• Error handling: Quarantine tables vs. DLT expectations vs. silent coercion
• Deduplication: Entity resolution vs. idempotent upserts vs. append-only with versioning
• Transform ownership: Config-driven vs. code-driven vs. declarative

These decisions define the platform more than the choice of Databricks vs. Snowflake or batch vs. streaming.

What We’d Standardize

If we were starting a new platform today with no existing constraints:

1. A config schema for DQ rules. The quarantine pattern from Variation 1 is universally useful. Every platform benefits from separating “what to validate” from “how to validate.”

2. Standard metadata columns on every table. Ingestion timestamp, source system identifier, batch/job ID, and a record hash for deduplication. These cost nothing and save hours of debugging.

3. Quarantine as a first-class pattern. Not an error table — a structured quarantine layer with failure metadata, reconciliation workflows, and reprocessing capabilities.

4. Silver as explicitly typed contracts. Whether you use Pydantic models, Delta table schemas, or DLT expectations, the silver layer should declare its schema as a contract that downstream consumers can depend on.

The vocabulary is always bronze, silver, gold. The architecture is always specific to the problem.