Beyond Bronze, Silver, and Gold: Designing Medallion Layers Around Real Data Needs

When we talk about organizing data and its transformations in a lakehouse, Medallion Architecture is one of the most widely used approaches today. To understand how to adapt it to a project, let’s first review what Databricks proposes and the responsibility of each layer.

Conceptual diagram of the main responsibilities.

Bronze: preserve source data

Bronze keeps data with minimal transformations and preserves what was received from the source. It provides a basis for auditing and reprocessing, depending on the history retained. It can also include ingestion metadata, such as the source file name.

Silver: validate and refine

Silver handles cleaning, deduplication, data type standardization, and quality checks. It can also combine sources through joins and introduce data models, creating useful datasets for analysis and further processing.

Gold: prepare data for business use

Gold organizes data around specific consumption needs. It can include dimensional models and aggregations for reporting, dashboards, and applications, with definitions that follow business requirements.

Databricks describes these responsibilities in its official Medallion documentation.

The challenge comes when we need to decide how many stages each responsibility requires and which stages need their own output.

In practice, layer names can change

In projects, we will not always find the labels Bronze, Silver, and Gold. In my experience, I have worked with architectures that followed a similar structure but used different names for their layers.

We might find Raw or Landing for source data, Curated or Refined for prepared data, and Consumption, Serving, or Processed for business outputs. These are not fixed equivalents: what one team calls Curated may have different responsibilities in another project.

That is why, when I review an architecture, I first try to understand what each layer receives, what it transforms, and how its output is used. This helps me recognize the Medallion approach beyond the names and assess whether the structure makes sense for the project.

When does it make sense to split a layer?

Beyond naming, projects can also differ in the number of layers and the work done in each one. Medallion provides Bronze, Silver, and Gold as a reference, but each layer can include several stages. Databricks supports, for example, building Silver tables from other Silver tables.

Depending on the use case, it may make sense to divide these responsibilities into sublayers. This brings us to the main question: when does that separation earn its place?

The following questions help us decide whether a responsibility needs a separate output. They are a practical checklist, not Databricks requirements.

No single answer automatically requires another layer. A join can be one step within a transformation. A change in grain may justify another table without requiring a new architectural layer.

Separation makes more sense when the output serves a clear purpose of its own.

Separating, storing, and publishing

Once we identify a reason to separate a responsibility, we need to decide how to implement it: do we need to store its result, and who should be able to use it?

The table below compares implementation choices. “Published” describes visibility, not a separate dataset type: materialized views and streaming tables can also be published in Unity Catalog. The last row refers to a regular Delta table maintained by our own processing code.

PRIVATE persists for the pipeline's lifetime, not just one run. It is different from publishing an object and restricting its permissions.

A materialized view does not guarantee incremental refresh. Refresh support and policy affect whether Databricks updates part of the result or recomputes it.

The layer name does not determine the processing method. In Lakeflow Spark Declarative Pipelines, both materialized views and streaming tables support triggered and continuous execution. A streaming table does not require a continuously running pipeline.

An intermediate Silver stage with PRIVATE

This small example uses order headers with two source columns: order_id and status. We normalize the status and calculate a shared quality flag. For this example only, accepted statuses are PAID, SHIPPED, and CANCELLED.

The code belongs in a SQL source file in one Unity Catalog-enabled Lakeflow Spark Declarative Pipeline. We configure its output catalog and schema, and use an existing source table, demo.bronze.orders.

CREATE OR REFRESH PRIVATE MATERIALIZED VIEW silver_source_orders_prepared
AS
SELECT
  order_id,
  status AS source_status,
  UPPER(TRIM(status)) AS order_status,
  COALESCE(
    order_id IS NOT NULL
    AND UPPER(TRIM(status)) IN ('PAID', 'SHIPPED', 'CANCELLED'),
    FALSE
  ) AS is_valid
FROM demo.bronze.orders;

Both accepted and rejected outputs can reuse this preparation. COALESCE makes a null validation result false, so a missing status cannot fall between the two branches.

This is a small syntax example, not a reason to materialize every normalization step. If analysts need to query the intermediate output directly, PRIVATE would not meet that need.

Good practices when defining layers and sublayers

• Preserve source data during ingestion. I try to avoid adding business logic too early. In Bronze, my goal is to keep a faithful representation of what we received, with minimal transformations and metadata about its source.

  This matters when the team maintaining the pipelines does not have direct access to every source system. Keeping the received data helps us investigate whether an issue was already present at ingestion or appeared during later transformations.

• Make responsibilities clear, then simplify. I usually start by separating the main steps and then review which ones I can combine. Understanding what each stage does helps me remove divisions that do not add enough value.

  This is a personal preference. It does not mean storing every intermediate result: I can separate responsibilities in the design or code before deciding which ones need storage.

  In one project, we kept an intermediate result after complex transformations, including pivots on large tables. That detail helped us investigate incorrect metrics without repeating the full preparation each time. The reason for keeping the stage was the work it saved during investigation, rather than the name of the layer.

• Keep documentation simple, clear, and current. A solution may run for years while team members change. That is why I consider it essential to document each layer’s purpose, transformations, inputs, outputs, and scripts or pipelines.

  The documentation should also explain what one row represents, which checks apply, who maintains the output, and how it is updated.

  Unity Catalog automatically captures lineage for supported operations, helping us understand dependencies between datasets. However, it does not explain why we created a layer or what its consumers should expect. Coverage also has limitations, including pipelines that use private tables. Explicit documentation is still needed.

• Apply data quality checks. Beyond cleaning and standardizing data in Silver, we should define the conditions it must meet before other processes use it. For example, an order should have an identifier, and its status should belong to a set agreed with the business.

  With Lakeflow Spark Declarative Pipelines, we can implement these checks through expectations. Depending on the rule, we can keep records and measure violations, exclude them from the output, or fail the update of the affected flow. Different problems may need different responses.

  When invalid records need investigation or correction, a quarantine output lets us separate them from accepted data. We must implement this separation explicitly: an expectation does not send records to quarantine automatically. We should keep the reason and source information, and define who reviews the records and how they are reprocessed. Quarantine can be an operational output without becoming another business layer.

  The next two definitions build on the private dataset above. The first publishes accepted records:

CREATE OR REFRESH MATERIALIZED VIEW silver_source_orders
(
  CONSTRAINT valid_order EXPECT (is_valid) ON VIOLATION DROP ROW
)
AS SELECT * FROM silver_source_orders_prepared;

DROP ROW excludes a failing record from this output. It does not delete the Bronze record or create a quarantine destination.

We define a second output with the opposite condition to keep rejected records:

CREATE OR REFRESH MATERIALIZED VIEW silver_source_orders_quarantine
(
  CONSTRAINT invalid_order EXPECT (NOT is_valid) ON VIOLATION DROP ROW
)
AS
SELECT *, 'Missing order ID or unsupported status' AS rejection_reason
FROM silver_source_orders_prepared;

Both outputs read the same prepared dataset. Reading quarantine data from the accepted output would be too late: the rejected rows would already be gone. This adapts the documented pattern of separate paths for valid and invalid records.

The quarantine keeps the original status and order identifier with a rejection reason. It is a refreshed result, not a permanent record of every past rejection. A separate incident history would need its own design.

Materialized views with expectations use full refresh, so both published outputs in this example are recomputed during updates. This processing cost should be part of the decision to store them.

• Separate shared logic from specific logic. When several models need the same preparation or definition, we can consider a shared dataset. More specific rules can build on it, without assuming that every business area interprets the data in the same way.

• Keep the detail that is needed. Before aggregating or filtering data, we should identify what information other analyses or investigations may require. Databricks recommends that Silver include a validated, non-aggregated representation of the records.

Common mistakes to avoid

• Creating a layer for every transformation. A pivot, join, or calculation can belong to the same responsibility. Better code organization does not require storing every step.

• Assuming that storing data guarantees reproducibility. An intermediate table lets us query its contents, but rebuilding a previous result also depends on the available data, logic versions, and history. In Delta Lake, review both delta.logRetentionDuration and delta.deletedFileRetentionDuration when defining the recovery window. Time travel requires the relevant logs and data files to remain available. Increasing retention settings does not recover files that have already been deleted.

• Assuming that more separation guarantees better performance. Stored intermediate results can reduce repeated processing, but they also use storage and compute to stay updated. Compare those costs with the processing they avoid, considering data volume, refresh frequency, and how often the output is reused.

A simple example: prepare source data and integrate business concepts

In this hypothetical example, we receive orders from an online store and physical stores. The business needs to analyze both channels and create reports at different levels of detail.

Hypothetical example. The subdivisions represent responsibilities and do not necessarily require separate schemas.

Bronze keeps what each system sends, along with metadata that identifies its source.

In Silver Source, we convert data types, standardize formats, and check identifiers and required fields. The tables still represent sales from each source separately.

In Silver Business, we integrate those representations. For this example, we identify each row using the source system, order ID, and order line ID. This distinguishes orders or lines that share identifiers across systems.

That key identifies records from both systems, but it does not tell us whether two product codes represent the same product or whether their statuses have the same meaning. To build a common representation, we also need to agree on those mappings and decide what to do when we cannot resolve them.

Finally, Gold prepares the representations that consumers need. In this example, these could be daily sales by channel or monthly results by product. If consumers need to analyze individual order lines, Gold could also provide a model at that level of detail.

Why split Silver in this case?

Silver Source handles the details of each source system. Silver Business creates a common representation for several reports and detailed queries. Integrating sources within Silver is covered in the Databricks data modeling documentation.

If a source format changes, we can review its preparation while trying to maintain the output expected by the integration stage. Separation helps us locate the change, although its effects may still reach later stages.

If the preparation were simple and its results did not need to be queried or reused separately, we could keep these steps within a single process.

Representing layers in Unity Catalog

Once we define the responsibilities, we need to decide how to organize them into catalogs, schemas, and tables. The Databricks governance guide recommends schemas for Bronze, Silver, and Gold. This does not mean creating one for every intermediate stage.

In our example, we could keep both Silver sublayers in the same schema if they share administration and permissions. If we need to manage datasets with different access requirements or owners, separate silver_source and silver_business schemas may help.

This is a decision that depends on the project. Separate schemas also do not automatically restrict access: we need to review privileges inherited from the catalog.

Processing stages and schemas do not need to match one to one.

Define the responsibility first

For me, a layer earns its place when we can explain its responsibility and who needs its output. The name and location come after that.

Sometimes, the three familiar layers are enough. Other times, it helps to separate source preparation, shared integration, or a reusable result. It is also worth reviewing these divisions and simplifying them when they no longer add value.

Before adding a sublayer, I would try to answer one question:

What specific need are we addressing by keeping this output separate?

That answer should guide the decision.

In a future post, I’ll explore how to organize catalogs and schemas in Unity Catalog around an organization’s needs, including business areas, environments, ownership, and access requirements.

TL;DR

Bronze, Silver, and Gold describe responsibilities, but each layer can involve more than one processing stage. The important question is what a separate stage adds to the architecture.

• Give each stage a clear purpose. Reuse, quality checks, business definitions, consumer needs, or troubleshooting can justify a separate output. A complex transformation alone does not automatically require another layer.

• Separate three decisions: organizing the logic, storing the result, and making it available to consumers. A logical boundary does not always need another stored dataset.

• Make the output clear. Define what one row represents, which checks have passed, who maintains the data, and how it is updated.

• Review the cost of separation. Intermediate results can save repeated processing, but they also add storage, refresh work, and maintenance. Sublayers do not necessarily need separate Unity Catalog schemas.

A layer earns its place when we can explain its responsibility and who needs its output. Its name and location come after that