Bronze Layer: Raw and Un-validated Data

Purpose: A repository for storing data in its original, un-validated state. It houses un-validated data (without requiring predefined schemas). Data arrives through full loads or delta loads.

Key characteristics include:

o    Maintains the raw state of the data source in the structure “as-is”.
o    Data is immutable (read-only).
o    Managed using interval partitioned tables, for example, using a YYYYMMDD or date-time folder structure.
o    Retains the full (unprocessed) history of each dataset in an efficient storage format, for example, Parquet or Delta.
o    For transactional data: This can be appended incrementally and grow over time.
o    Provides the ability to recreate any state of a given data system.
o    Can be any combination of streaming and batch transactions.
o    May include extra metadata, such as schema information, source file names, or recording the time data was processed.

A common query I encounter is, “Which file format is superior? Should I opt for Delta or Parquet?” While Delta offers faster speed, its benefits aren’t as pronounced if your data is already versioned or historised using a folder structure. As such, maintaining a transaction log or applying versioning isn’t crucial. Bronze data is typically new or appended data. Therefore, selecting Parquet is perfectly acceptable. However, you could also choose Delta to remain consistent with other layers.

For transactional data: This can be appended incrementally and grow over time.

   o    Provides the ability to recreate any state of a given data system.

   o    Can be a combination of streaming and batch transactions.

  o  May include additional metadata, such as schema information, source file names, or data processing timestamps.

Format Selection: While Delta offers faster speeds, its benefits are less pronounced if your data is already versioned or historiated using a folder structure. In such cases, maintaining a transaction log or applying versioning isn't essential. Parquet is perfectly acceptable for Bronze data. However, Delta can be chosen for consistency across layers.

Business User Access: It's uncommon for raw data in the Bronze layer to be directly used for business user queries or ad-hoc analyses due to its complexity and lack of inherent structure. Securing this layer with numerous small tables can also be challenging. Therefore, Bronze serves primarily as a staging layer and a source for other layers, accessed mainly by technical accounts.

Beyond Parquet and Delta: Exploring Apache Iceberg

Parquet and Delta Lake are popular data storage options for Medallion Architecture. While these are excellent choices, Apache Iceberg is emerging as a strong contender.

Here's why you might consider it:

o    Scalability: Iceberg excels at handling massive datasets efficiently, making it ideal for large-scale data lakes.
o   ACID Compliance: It offers ACID (Atomicity, Consistency, Isolation, Durability) guarantees, ensuring data integrity in concurrent write operations.
o   Schema Evolution: Iceberg gracefully handles schema changes, allowing you to evolve your data model over time without compromising data integrity.

Silver Layer: Refined and Enriched Data

Purpose: Provides a refined structure over ingested data. It represents a validated, enriched version of the data that can be trusted for downstream operational and analytical workloads.

Key characteristics include:

o    It utilizes data quality rules for data validation and processing.
o    It typically contains only functional data, so technical or irrelevant data from Bronze is filtered out.
o  Historization is usually applied by merging all data. Data is processed using slowly changing dimensions (SCD), either type 2 or type 4. This involves adding additional columns, such as start, end, and current columns.
o    Data is stored in an efficient storage format; preferably Delta, alternatively Parquet.
o    Uses versioning for rolling back processing errors.
o    Handles missing data and standardizes clean or empty fields.
o    Data is usually enriched with reference and/or master data.
o    Data is often clustered around specific subject areas.
o   Data may still be source-system aligned and organized, meaning it has not been integrated with data from other domains yet.

Considerations for Silver as a Transient Layer:

   Some propose Silver as a transient storage layer, enabling the deletion of outdated data or the spontaneous creation of storage accounts. The decision depends on your needs. If you don't plan to use data in its original context for operational reporting or analytics, then Silver can be temporary.

However, if you require historical data for operational reporting and analytics, establish Silver as a permanent layer.

Silver Layer Design Optimization:

Denormalized Data Model: When querying Silver data, consider using a denormalized data model. This approach eliminates the need for extensive joins and aligns better with the distributed columnar storage architecture. However, there's no need to abandon established data modeling practices like the 3rd normal form or Data Vault if they suit your needs.

Regarding historicization, the Delta file format already versions Parquet files, and you have a historical record in your Bronze layer for reloading data if necessary. For automation and adaptability, a metadata-driven strategy is recommended for managing tables and notebooks. Data redundancy in a lake environment is less expensive than the computational processing and data joining required to eliminate it. Unless you're dealing with significant daily schema changes, the complexity of a data vault might not be necessary.

Silver Layer Integration Considerations:

Data Integration: The question of whether to integrate data between applications and source systems in the Silver layer arises frequently. For easier management and isolation of concerns, it's generally recommended to keep things separate if possible. This suggests that if you're using Silver for operational reporting or analytics, avoid prematurely merging and integrating data from source systems. Doing so could lead to unnecessary connection points between applications. Users interested only in data from a single source might be linked to other systems if the data is first combined in a harmonized layer before being provided.

Consequently, these data users are more likely to experience potential impacts from other systems. If you're striving for an isolated design, the integration or combination of data from different sources should be moved up to the Gold layer. The same reasoning applies to aligning your Lakehouses with the source-system side of your architecture. If you plan to build data products and are particular about data ownership, then advise against engineers prematurely cross-joining data from applications in other domains.

Data Enrichment: When considering enrichments like calculations in the Silver layer, your goal plays a key role. If operational reporting requires enrichments, you can start enriching data in Silver. This might lead to some additional calibration when merging data later in the Gold stage. While this may require additional effort, the flexibility it offers can be worthwhile.

Gold Layer: Project-Specific, Integrated Data

Purpose: The Gold layer houses data structured in "project-specific" databases, readily available for consumption. This integration of data from various sources may result in a shift in data ownership. Key characteristics of the Gold layer include:

o    Gold tables represent data transformed for consumption or specific use cases.
o    Data is stored in an efficient storage format, preferably Delta.
o    Gold uses versioning for rolling back processing errors.
o  Historization is applied only for the relevant use cases or consumers. So, Gold can be a selection or aggregation of data found in Silver.
o    Complex business rules are applied in Gold. It utilizes post-processing activities, calculations, enrichments, and use-case-specific optimizations.
o    Data is highly governed and well-documented.