AI Glossary / Data engineering and architecture / Advanced
Medallion architecture
Definition
Medallion architecture is a data-design pattern that organizes processing into layers commonly called bronze, silver and gold. Bronze retains ingested source data, silver provides cleaned and validated detail, and gold provides data shaped for particular business uses. The layers describe intended responsibilities and increasing preparation. Their names do not, by themselves, guarantee accuracy, freshness, security or suitability for a machine-learning task.
What it means
Different users need different representations of the same incoming information. An engineer investigating a malformed event may need its original form, while a manager needs a consistently defined weekly count. The pattern keeps those purposes explicit. Databricks presents medallion architecture as a recommended, optional design pattern for progressively refining data. In a fictional learning service, raw activity messages, validated activity records and a course-usage summary can occupy different layers. The summary is useful because of its definition and processing rules, rather than because gold is inherently a certification of truth.
How it works
Data moves through transformations whose responsibilities are defined for each layer. Databricks describes bronze ingestion, silver cleaning and validation, and gold modeling or aggregation for downstream needs. A practical implementation must specify identifiers, rejected-record handling, transformation versions and refresh behavior. In an original pipeline design, a malformed record could remain traceable in the ingestion layer while being withheld from an accepted silver dataset. A later correction could then be processed deliberately. The naming convention alone does not decide how to manage such cases, and transaction guarantees depend on the underlying platform and implementation.
Three practical examples
- A learning platform retains incoming activity messages, validates their identifiers and timestamps, and produces a weekly course-activity summary.
- An equipment service separates received device readings from standardized measurements and from operational aggregates tailored to a particular dashboard.
- An AI data team uses detailed validated records for a modeling task while using a separately defined business summary for management reporting.
Think of it this way
Imagine a document-processing desk with incoming envelopes, checked individual forms and a prepared weekly report. Each stage serves a different purpose. The report is easier to read, but it cannot replace the forms when someone investigates an unusual entry. The analogy explains preparation and traceability; it does not imply that every record must be copied unchanged into every stage.
Worked example
Consider an original fictional batch of 100 activity messages retained in bronze. Validation identifies five messages with missing required identifiers. Among the remaining 95 messages, ten are surplus repeat deliveries of already represented events under an agreed event-ID rule. Assume these groups do not overlap. Silver therefore contains 100 minus 5 minus 10 = 85 accepted distinct events. A gold summary groups them by course: A has 40, B has 30 and C has 15, totaling 85. The three gold rows represent 85 events, not three events. The five invalid messages remain available for investigation under the stated retention policy. If two are later corrected, the pipeline must distinguish those corrected events from repeat deliveries and update the relevant summary. The example assumes the defined validation and identity rules; it does not establish a universal quality score from the acceptance count.
Illustrative case study
In a fictional AI learning dashboard, a team initially feeds raw activity messages directly into a weekly engagement summary. Repeated delivery inflates counts, and missing identifiers make some results difficult to explain. The team defines separate ingestion, accepted-event and reporting datasets, with an explicit reason recorded for each withheld message. Reviewers reconcile a small batch from incoming messages to the final course totals and test a corrected record arriving later. The assistant's explanation includes the summary period and its accepted-event definition. This scenario illustrates a traceable processing design. It does not claim that using three layers automatically detects every bad record or makes engagement a complete measure of learning.
Common misconceptions and mistakes
Gold does not mean perfect or universally appropriate. A summary can omit detail needed for a different analysis. Bronze also does not mean that access controls or retention decisions can be ignored. Another mistake is treating every repeated value as a duplicate: two genuine events can share a learner, timestamp or activity type unless the chosen identity rule distinguishes them. Layer names do not require three separate physical servers. Finally, a successful pipeline run does not prove that its business definitions are right. An incorrect aggregation can execute consistently and still produce a misleading answer.
Business use
For a proposed AI information service, decide what each layer promises and who uses it. In the fictional activity pipeline, specify whether the gold measure counts messages, distinct events or active learners; these are different quantities. Define how users can trace a result to accepted records and how engineers investigate excluded messages. Measure freshness and processing cost at the relevant stages. Keep transformation rules and ownership understandable as requirements change. The pattern can organize this work, but its value comes from implemented responsibilities and checks rather than adding bronze, silver and gold names to existing folders.
People also ask
Are three physically separate systems required?
No. The layers are logical responsibilities and can be implemented in different arrangements.
Is gold always the best source for machine learning?
No. A particular task may need validated detail or another suitable representation instead of a business aggregate.
Does the pattern eliminate data-quality problems?
No. The actual rules, transformations and monitoring determine what problems are detected and handled.
Check your understanding
How many accepted events reach silver in the toy batch?
Answer: Eighty-five.
Why are there only three gold rows?
Answer: Each row summarizes one course.
Does that mean only three events remain?
Answer: No; their counts total 85.
Why must the invalid and duplicate groups be disjoint for the subtraction shown?
Answer: Otherwise subtracting both counts could remove the same message twice.
Related terms
- Data Lakehouse
- Data pipeline
- Aggregation
- data validation
- deduplication
- data lineage
Learn next
Design a miniature three-layer pipeline for ten fictional records. Include one invalid record and one repeat delivery, and write the rule that identifies each. Reconcile the accepted detail with a final summary. Then introduce a correction and explain how it should affect each relevant dataset without silently double-counting the event.
More in Data engineering and architecture
- Master-checker - Master-checker is a redundant processor arrangement in which a primary processor performs a computation while a checker performs the corresponding...
- Master data - Master data describes the core entities that an organization repeatedly refers to, such as customers, products, suppliers and locations.
- Materialized view - A materialized view stores the result of a defining query so that subsequent reads can use the precomputed data.
- Mega-Merger - Mega-Merger is a distributed algorithm presented through the idea of groups of network nodes merging into larger groups until one connected structure remains.
- Memory coherence - Memory coherence is the property that accesses to the same shared memory location behave according to a consistent ordering, even when processors hold...
- Memory-level parallelism - Memory-level parallelism, or MLP, is the overlap of multiple memory operations, particularly outstanding cache misses, so that a processor does not have to...
Sources
Databricks documentation, What is the medallion lakehouse architecture?: https://docs.databricks.com/aws/en/lakehouse/medallion The activity-message arithmetic, document-desk analogy and AI-dashboard scenario are original educational illustrations. The source supports the layer convention and design-pattern context; the entry does not infer automatic quality or transaction guarantees from the layer names alone.
Previous: Mechanism design Next: Medial axis transform