{"id":1305,"date":"2024-05-13T07:58:13","date_gmt":"2024-05-13T07:58:13","guid":{"rendered":"https:\/\/dotlabs.ai\/blogs\/?p=1305"},"modified":"2024-05-13T07:58:40","modified_gmt":"2024-05-13T07:58:40","slug":"medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels","status":"publish","type":"post","link":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/","title":{"rendered":"Medallion Architecture: Best Practices for Managing Bronze, Silver, and Gold Levels"},"content":{"rendered":"\n\n\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Many organizations\nleverage Medallion Architecture to organize data logically within a data\nlakehouse. This approach involves processing incoming data through distinct stages,\noften called &#8220;layers.&#8221; The most common structure utilizes Bronze,\nSilver, and Gold tiers, which is why &#8220;Medallion Architecture&#8221; is\nused.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">While the 3-layer design\nremains popular, discussions regarding the scope, purpose, and best practices\nfor each layer are ongoing. Additionally, a gap often exists between\ntheoretical concepts and real-world implementation. Here&#8217;s a breakdown of these\ndata architecture layers with practical considerations:<br><br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Data\nPlatform Strategy: A Foundation for Layering<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">The initial and crucial\nconsideration for your data architecture&#8217;s layering is your data platform&#8217;s\nusage. A centralized and shared platform will likely have a different structure\nwhen compared to a federated, multi-platform setup used by various domains.\n<br><br>Layering can also vary depending on platform alignment source-system alignment or consuming-side alignment. Source-system-aligned platforms are generally easier\nto standardize in terms of layering and structure due to the more predictable\ndata usage on the source side.<br><br><br><o:p><\/o:p><\/span><\/p>\n<h1 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Exploring\nEach Layer: Theory and Practice<o:p><\/o:p><\/span><\/h1>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Let&#8217;s delve into each\nlayer, outlining its goals and practical observations from real-world\napplications.<br><br><o:p><\/o:p><\/span><\/p>\n<h3 class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading2Char\"><span style=\"font-size:13.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Landing\nZone (Optional):<br><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:\n&quot;Helvetica&quot;,sans-serif\">Purpose:<\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\"> A temporary storage\nlocation for data gathered from various sources before it&#8217;s transferred to the\nBronze layer. This layer becomes particularly relevant when extracting data\nfrom challenging source systems, such as those of external clients or SaaS\nvendors.<br><\/span><span style=\"font-size: 12pt; line-height: 107%;\"><br>Implementation Considerations:<\/span><span style=\"text-indent: -0.25in; color: var(--body-text-default-color); font-family: Helvetica, sans-serif; font-size: 12pt; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); line-height: 107%;\">\nThe design of a landing zone can vary significantly. It&#8217;s often a simple Blob\nstorage account, but in some cases, it might be integrated directly with data\nlake services like a container, bucket, or designated folder for data\ningestion. The data housed here can be quite diverse, encompassing file formats\nlike CSV, JSON, XML, Parquet, Delta, and more.<\/span><\/h3>\n<p>Lorem Ipsum has been the industry&#8217;s standard dummy text ever since the 1500s.<\/p>\n\n\n\n\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Bronze\nLayer: Raw and Un-validated Data<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Purpose:<\/span><\/span><span style=\"line-height: 107%;\"><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\"> A\nrepository for storing data in its original, un-validated state. It houses\nun-validated data (without requiring predefined schemas). Data arrives through\nfull loads or delta loads.<\/span><\/font><br><br><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">Key characteristics include:<\/span><\/font><br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; color: rgb(36, 36, 36); font-weight: var(--single-content-weight); letter-spacing: -0.05pt; text-indent: -0.25in;\">Maintains\nthe raw state of the data source in the structure \u201cas-is\u201d.<br><\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); line-height: 115%; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span>Data\nis immutable (read-only).<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; color: rgb(36, 36, 36); letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">Managed\nusing interval partitioned tables, for example, using a YYYYMMDD or date-time\nfolder structure.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\"><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">Retains\nthe full (unprocessed) history of each dataset in an efficient storage format,\nfor example, Parquet or Delta.<\/span><span style=\"font-size: 16px;\"><br><\/span><\/font><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); line-height: 115%; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">For\ntransactional data: This can be appended incrementally and grow over time.<\/span><\/font><font face=\"Times New Roman\" size=\"1\"><br><\/font><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span>Provides\nthe ability to recreate any state of a given data system.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); font-family: Helvetica, sans-serif; font-size: 12pt; letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">Can\nbe any combination of streaming and batch transactions.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); font-family: Helvetica, sans-serif; font-size: 12pt; letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">May\ninclude extra metadata, such as schema information, source file names, or\nrecording the time data was processed.<\/span><\/p><p style=\"text-align: justify;\"><p><\/p><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"margin-left:.75in;mso-add-space:auto;\ntext-align:justify;text-indent:-.25in;line-height:115%;mso-list:l0 level1 lfo4\"><span style=\"font-size:12.0pt;line-height:115%;\nfont-family:&quot;Helvetica&quot;,sans-serif;mso-fareast-font-family:&quot;Times New Roman&quot;;\ncolor:#242424;letter-spacing:-.05pt;mso-font-kerning:0pt;mso-ligatures:none\"><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify;line-height:115%\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif;\nmso-fareast-font-family:&quot;Times New Roman&quot;;color:#242424;letter-spacing:-.05pt;\nmso-font-kerning:0pt;mso-ligatures:none\">A common query I encounter is, \u201cWhich\nfile format is superior? Should I opt for Delta or Parquet?\u201d While Delta offers\nfaster speed, its benefits aren\u2019t as pronounced if your data is already\nversioned or historised using a folder structure. As such, maintaining a\ntransaction log or applying versioning isn\u2019t crucial. Bronze data is typically\nnew or appended data. Therefore, selecting Parquet is perfectly acceptable.\nHowever, you could also choose Delta to remain consistent with other layers.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">For\ntransactional data:<\/span><\/span><span style=\"font-size:12.0pt;line-height:\n107%;font-family:&quot;Helvetica&quot;,sans-serif\"> This can be appended incrementally\nand grow over time. <o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpFirst\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; &nbsp;o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;&nbsp; <\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Provides\nthe ability to recreate any state of a given data system.<o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpMiddle\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; &nbsp;o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;&nbsp; <\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Can\nbe a combination of streaming and batch transactions.<o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;<\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">May\ninclude additional metadata, such as schema information, source file names, or\ndata processing timestamps.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Format\nSelection:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> While Delta offers faster speeds, its\nbenefits are less pronounced if your data is already versioned or historiated\nusing a folder structure. In such cases, maintaining a transaction log or\napplying versioning isn&#8217;t essential. Parquet is perfectly acceptable for Bronze\ndata. However, Delta can be chosen for consistency across layers.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Business\nUser Access<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"><strong>:<\/strong> It&#8217;s uncommon for raw data in the Bronze\nlayer to be directly used for business user queries or ad-hoc analyses due to\nits complexity and lack of inherent structure. Securing this layer with\nnumerous small tables can also be challenging. Therefore, Bronze serves\nprimarily as a staging layer and a source for other layers, accessed mainly by\ntechnical accounts.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Beyond Parquet and Delta:\nExploring Apache Iceberg<br><o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"margin-right:.2in;text-align:justify;line-height:\n115%\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif\">Parquet\nand Delta Lake are popular data storage options for Medallion Architecture.\nWhile these are excellent choices, Apache Iceberg is emerging as a strong\ncontender.<br><br>Here&#8217;s why you might consider it:<br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span class=\"Heading3Char\" style=\"font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); text-indent: -0.25in;\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp; &nbsp;&nbsp;<\/span><strong>Scalabilit<\/strong><strong>y:<\/strong><\/span><\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); text-indent: -0.25in; font-size: 12pt; line-height: 115%;\">&nbsp;Iceberg\nexcels at handling massive datasets efficiently, making it ideal for\nlarge-scale data lakes.<br><\/span><span class=\"Heading3Char\" style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\"><span style=\"font-weight: var(--single-content-weight);\">&nbsp;<\/span><strong>&nbsp;&nbsp;<\/strong><\/span><strong>ACID Compliance:<\/strong><\/span><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 115%;\">&nbsp;It\noffers ACID (Atomicity, Consistency, Isolation, Durability) guarantees,\nensuring data integrity in concurrent write operations.<br><\/span><span class=\"Heading3Char\" style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp; &nbsp;<\/span><strong>Schema Evolution:<\/strong><\/span><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 115%;\">&nbsp;Iceberg\ngracefully handles schema changes, allowing you to evolve your data model over\ntime without compromising data integrity.<\/span><\/p><p style=\"text-align: justify;\"><p><\/p><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"margin-right:.2in;mso-add-space:auto;\ntext-align:justify;text-indent:-.25in;line-height:115%;mso-list:l4 level1 lfo5\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer: Refined and Enriched Data<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Purpose:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">\nProvides a refined structure over ingested data. It represents a validated,\nenriched version of the data that can be trusted for downstream operational and\nanalytical workloads.<br><br>Key characteristics include:<br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"color: var(--body-text-default-color); font-size: 12pt; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); text-indent: -0.25in;\">It utilizes data quality rules for data\nvalidation and processing.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">It typically contains only functional data,\nso technical or irrelevant data from Bronze is filtered out.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Historization is usually applied by merging\nall data. Data is processed using slowly changing dimensions (SCD), either type\n2 or type 4. This involves adding additional columns, such as start, end, and\ncurrent columns.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis stored in an efficient storage format; preferably Delta, alternatively\nParquet.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Uses\nversioning for rolling back processing errors.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Handles\nmissing data and standardizes clean or empty fields.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis usually enriched with reference and\/or master data.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis often clustered around specific subject areas.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nmay still be source-system aligned and organized, meaning it has not been\nintegrated with data from other domains yet.<\/span><\/p><p style=\"text-align: justify;\"><p><\/p><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l2 level1 lfo2;tab-stops:list .5in\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Considerations\nfor Silver as a Transient Layer:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoListParagraphCxSpFirst\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l5 level1 lfo3;tab-stops:list .5in\"><span style=\"line-height: 107%; font-size: 13.3333px;\"><font face=\"Courier New\">&nbsp; &nbsp;<\/font><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Some\npropose Silver as a transient storage layer, enabling the deletion of outdated\ndata or the spontaneous creation of storage accounts. The decision depends on\nyour needs. If you don&#8217;t plan to use data in its original context for\noperational reporting or analytics, then Silver can be temporary. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">However,\nif you require historical data for operational reporting and analytics,\nestablish Silver as a permanent layer.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer Design Optimization:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Denormalized\nData Model:<\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> When querying Silver data, consider using\na denormalized data model. This approach eliminates the need for extensive\njoins and aligns better with the distributed columnar storage architecture.\nHowever, there&#8217;s no need to abandon established data modeling practices like\nthe 3rd normal form or Data Vault if they suit your needs. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Regarding <strong>historicization<\/strong>, the Delta file format\nalready versions Parquet files, and you have a historical record in your Bronze\nlayer for reloading data if necessary. For automation and adaptability, a\nmetadata-driven strategy is recommended for managing tables and notebooks. Data\nredundancy in a lake environment is less expensive than the computational\nprocessing and data joining required to eliminate it. Unless you&#8217;re dealing\nwith significant daily schema changes, the complexity of a data vault might not\nbe necessary.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer Integration Considerations:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Data\nIntegration:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> The question of whether to integrate data\nbetween applications and source systems in the Silver layer arises frequently.\nFor easier management and isolation of concerns, it&#8217;s generally recommended to\nkeep things separate if possible. This suggests that if you&#8217;re using Silver for\noperational reporting or analytics, avoid prematurely merging and integrating\ndata from source systems. Doing so could lead to unnecessary connection points\nbetween applications. Users interested only in data from a single source might\nbe linked to other systems if the data is first combined in a harmonized layer\nbefore being provided. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Consequently, these data\nusers are more likely to experience potential impacts from other systems. If\nyou&#8217;re striving for an isolated design, the integration or combination of data\nfrom different sources should be moved up to the Gold layer. The same reasoning\napplies to aligning your Lakehouses with the source-system side of your\narchitecture. If you plan to build data products and are particular about data\nownership, then advise against engineers prematurely cross-joining data from\napplications in other domains.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Data\nEnrichment:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"><strong> <\/strong>When considering enrichments like calculations\nin the Silver layer, your goal plays a key role. If operational reporting\nrequires enrichments, you can start enriching data in Silver. This might lead\nto some additional calibration when merging data later in the Gold stage. While\nthis may require additional effort, the flexibility it offers can be\nworthwhile.<br><br><\/span><\/p><h1 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Gold\nLayer: Project-Specific, Integrated Data<br><o:p><\/o:p><\/span><\/h1><p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Purpose:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>\n<\/strong>The Gold layer houses data structured in &#8220;project-specific&#8221;\ndatabases, readily available for consumption. This integration of data from various\nsources may result in a shift in data ownership. Key characteristics of the\nGold layer include:<br><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Gold tables represent data transformed for\nconsumption or specific use cases.<br><\/span><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data is stored in an efficient storage\nformat, preferably Delta.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Gold uses versioning for rolling back\nprocessing errors.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><span style=\"color: var(--body-text-default-color); font-family: Helvetica, sans-serif; font-size: 12pt; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); text-indent: -0.25in;\">Historization is applied only for the\nrelevant use cases or consumers. So, Gold can be a selection or aggregation of\ndata found in Silver.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Complex business rules are applied in Gold.\nIt utilizes post-processing activities, calculations, enrichments, and\nuse-case-specific optimizations.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data is highly governed and\nwell-documented.<\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Bronze\nLayer: Raw and Un-validated Data<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Purpose:<\/strong><\/span><\/span><span style=\"line-height: 107%;\"><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\"> A\nrepository for storing data in its original, un-validated state. It houses\nun-validated data (without requiring predefined schemas). Data arrives through\nfull loads or delta loads.<\/span><\/font><br><br><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">Key characteristics include:<\/span><\/font><br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; color: rgb(36, 36, 36); font-weight: var(--single-content-weight); letter-spacing: -0.05pt; text-indent: -0.25in;\">Maintains\nthe raw state of the data source in the structure \u201cas-is\u201d.<br><\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); line-height: 115%; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span>Data\nis immutable (read-only).<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; color: rgb(36, 36, 36); letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">Managed\nusing interval partitioned tables, for example, using a YYYYMMDD or date-time\nfolder structure.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\"><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">Retains\nthe full (unprocessed) history of each dataset in an efficient storage format,\nfor example, Parquet or Delta.<\/span><span style=\"font-size: 16px;\"><br><\/span><\/font><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); line-height: 115%; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><font face=\"Helvetica, sans-serif\"><span style=\"font-size: 12pt;\">For\ntransactional data: This can be appended incrementally and grow over time.<\/span><\/font><font face=\"Times New Roman\" size=\"1\"><br><\/font><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif; color: rgb(36, 36, 36); letter-spacing: -0.05pt;\"><span style=\"color: rgb(64, 64, 64); font-family: &quot;Courier New&quot;; font-size: 13.3333px; letter-spacing: normal;\">o<\/span><span style=\"color: rgb(64, 64, 64); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; letter-spacing: normal;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span>Provides\nthe ability to recreate any state of a given data system.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); font-family: Helvetica, sans-serif; font-size: 12pt; letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">Can\nbe any combination of streaming and batch transactions.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"color: rgb(36, 36, 36); font-family: Helvetica, sans-serif; font-size: 12pt; letter-spacing: -0.05pt; text-indent: -0.25in; font-weight: var(--single-content-weight);\">May\ninclude extra metadata, such as schema information, source file names, or\nrecording the time data was processed.<\/span><\/p><p style=\"text-align: justify;\"><p><\/p><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"margin-left:.75in;mso-add-space:auto;\ntext-align:justify;text-indent:-.25in;line-height:115%;mso-list:l0 level1 lfo4\"><span style=\"font-size:12.0pt;line-height:115%;\nfont-family:&quot;Helvetica&quot;,sans-serif;mso-fareast-font-family:&quot;Times New Roman&quot;;\ncolor:#242424;letter-spacing:-.05pt;mso-font-kerning:0pt;mso-ligatures:none\"><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify;line-height:115%\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif;\nmso-fareast-font-family:&quot;Times New Roman&quot;;color:#242424;letter-spacing:-.05pt;\nmso-font-kerning:0pt;mso-ligatures:none\">A common query I encounter is, \u201cWhich\nfile format is superior? Should I opt for Delta or Parquet?\u201d While Delta offers\nfaster speed, its benefits aren\u2019t as pronounced if your data is already\nversioned or historised using a folder structure. As such, maintaining a\ntransaction log or applying versioning isn\u2019t crucial. Bronze data is typically\nnew or appended data. Therefore, selecting Parquet is perfectly acceptable.\nHowever, you could also choose Delta to remain consistent with other layers.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">For\ntransactional data:<\/span><\/span><span style=\"font-size:12.0pt;line-height:\n107%;font-family:&quot;Helvetica&quot;,sans-serif\"> This can be appended incrementally\nand grow over time. <o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpFirst\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; &nbsp;o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;&nbsp; <\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Provides\nthe ability to recreate any state of a given data system.<o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpMiddle\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; &nbsp;o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;&nbsp; <\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Can\nbe a combination of streaming and batch transactions.<o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo1;tab-stops:list .5in\"><!--[if !supportLists]--><span style=\"font-size:10.0pt;mso-bidi-font-size:12.0pt;line-height:107%;font-family:\n&quot;Courier New&quot;;mso-fareast-font-family:&quot;Courier New&quot;\">&nbsp; o<span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp;&nbsp;<\/span><\/span><!--[endif]--><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">May\ninclude additional metadata, such as schema information, source file names, or\ndata processing timestamps.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Format\nSelection:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> While Delta offers faster speeds, its\nbenefits are less pronounced if your data is already versioned or historiated\nusing a folder structure. In such cases, maintaining a transaction log or\napplying versioning isn&#8217;t essential. Parquet is perfectly acceptable for Bronze\ndata. However, Delta can be chosen for consistency across layers.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Business\nUser Access<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"><strong>:<\/strong> It&#8217;s uncommon for raw data in the Bronze\nlayer to be directly used for business user queries or ad-hoc analyses due to\nits complexity and lack of inherent structure. Securing this layer with\nnumerous small tables can also be challenging. Therefore, Bronze serves\nprimarily as a staging layer and a source for other layers, accessed mainly by\ntechnical accounts.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Beyond Parquet and Delta:\nExploring Apache Iceberg<br><o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"margin-right:.2in;text-align:justify;line-height:\n115%\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif\">Parquet\nand Delta Lake are popular data storage options for Medallion Architecture.\nWhile these are excellent choices, Apache Iceberg is emerging as a strong\ncontender.<br><br>Here&#8217;s why you might consider it:<br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span class=\"Heading3Char\" style=\"font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); text-indent: -0.25in;\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp; &nbsp;&nbsp;<\/span><strong>Scalabilit<\/strong><strong>y:<\/strong><\/span><\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); text-indent: -0.25in; font-size: 12pt; line-height: 115%;\">&nbsp;Iceberg\nexcels at handling massive datasets efficiently, making it ideal for\nlarge-scale data lakes.<br><\/span><span class=\"Heading3Char\" style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\"><span style=\"font-weight: var(--single-content-weight);\">&nbsp;<\/span><strong>&nbsp;&nbsp;<\/strong><\/span><strong>ACID Compliance:<\/strong><\/span><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 115%;\">&nbsp;It\noffers ACID (Atomicity, Consistency, Isolation, Durability) guarantees,\nensuring data integrity in concurrent write operations.<br><\/span><span class=\"Heading3Char\" style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size: 12pt; line-height: 115%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight); font-family: &quot;Courier New&quot;; font-size: 13.3333px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;;\">&nbsp; &nbsp;<\/span><strong>Schema Evolution:<\/strong><\/span><\/span><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 115%;\">&nbsp;Iceberg\ngracefully handles schema changes, allowing you to evolve your data model over\ntime without compromising data integrity.<\/span><\/p><p style=\"text-align: justify;\"><p><\/p><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"margin-right:.2in;mso-add-space:auto;\ntext-align:justify;text-indent:-.25in;line-height:115%;mso-list:l4 level1 lfo5\"><span style=\"font-size:12.0pt;line-height:115%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer: Refined and Enriched Data<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Purpose:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">\nProvides a refined structure over ingested data. It represents a validated,\nenriched version of the data that can be trusted for downstream operational and\nanalytical workloads.<br><br>Key characteristics include:<br><\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"color: var(--body-text-default-color); font-size: 12pt; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); text-indent: -0.25in;\">It utilizes data quality rules for data\nvalidation and processing.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">It typically contains only functional data,\nso technical or irrelevant data from Bronze is filtered out.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Historization is usually applied by merging\nall data. Data is processed using slowly changing dimensions (SCD), either type\n2 or type 4. This involves adding additional columns, such as start, end, and\ncurrent columns.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis stored in an efficient storage format; preferably Delta, alternatively\nParquet.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Uses\nversioning for rolling back processing errors.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Handles\nmissing data and standardizes clean or empty fields.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;&nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis usually enriched with reference and\/or master data.<br><\/span><span style=\"text-align: justify; font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-align: justify; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data\nis often clustered around specific subject areas.<\/span><\/p><p style=\"text-indent: 0px; text-align: justify;\"><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); background-color: ; color: var(--body-text-default-color); text-align: justify; font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); background-color: ; color: var(--body-text-default-color); font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-align: justify; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"color: var(--body-text-default-color); font-family: Helvetica, sans-serif; font-size: 12pt; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); text-align: justify; text-indent: -0.25in;\">Data\nmay still be source-system aligned and organized, meaning it has not been\nintegrated with data from other domains yet.<\/span><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l2 level1 lfo2;tab-stops:list .5in\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Considerations\nfor Silver as a Transient Layer:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoListParagraphCxSpFirst\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l5 level1 lfo3;tab-stops:list .5in\"><span style=\"line-height: 107%; font-size: 13.3333px;\"><font face=\"Courier New\">&nbsp; &nbsp;<\/font><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Some\npropose Silver as a transient storage layer, enabling the deletion of outdated\ndata or the spontaneous creation of storage accounts. The decision depends on\nyour needs. If you don&#8217;t plan to use data in its original context for\noperational reporting or analytics, then Silver can be temporary. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">However,\nif you require historical data for operational reporting and analytics,\nestablish Silver as a permanent layer.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer Design Optimization:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>De-normalized\nData Model:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> When querying Silver data, consider using\na de-normalized data model. This approach eliminates the need for extensive\njoins and aligns better with the distributed columnar storage architecture.\nHowever, there&#8217;s no need to abandon established data modeling practices like\nthe 3rd normal form or Data Vault if they suit your needs. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Regarding <strong>historicization<\/strong>, the Delta file format\nalready versions Parquet files, and you have a historical record in your Bronze\nlayer for reloading data if necessary. For automation and adaptability, a\nmetadata-driven strategy is recommended for managing tables and notebooks. Data\nredundancy in a lake environment is less expensive than the computational\nprocessing and data joining required to eliminate it. Unless you&#8217;re dealing\nwith significant daily schema changes, the complexity of a data vault might not\nbe necessary.<br><br><o:p><\/o:p><\/span><\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Silver\nLayer Integration Considerations:<o:p><\/o:p><\/span><\/h2>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Data\nIntegration:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> The question of whether to integrate data\nbetween applications and source systems in the Silver layer arises frequently.\nFor easier management and isolation of concerns, it&#8217;s generally recommended to\nkeep things separate if possible. This suggests that if you&#8217;re using Silver for\noperational reporting or analytics, avoid prematurely merging and integrating\ndata from source systems. Doing so could lead to unnecessary connection points\nbetween applications. Users interested only in data from a single source might\nbe linked to other systems if the data is first combined in a harmonized layer\nbefore being provided. <br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Consequently, these data\nusers are more likely to experience potential impacts from other systems. If\nyou&#8217;re striving for an isolated design, the integration or combination of data\nfrom different sources should be moved up to the Gold layer. The same reasoning\napplies to aligning your Lakehouses with the source-system side of your\narchitecture. If you plan to build data products and are particular about data\nownership, then advise against engineers prematurely cross-joining data from\napplications in other domains.<br><br><o:p><\/o:p><\/span><\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Data\nEnrichment:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> When considering enrichments like calculations\nin the Silver layer, your goal plays a key role. If operational reporting\nrequires enrichments, you can start enriching data in Silver. This might lead\nto some additional calibration when merging data later in the Gold stage. While\nthis may require additional effort, the flexibility it offers can be\nworthwhile.<br><br><\/span><\/p><h1 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Gold\nLayer: Project-Specific, Integrated Data<br><o:p><\/o:p><\/span><\/h1><p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Purpose:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">\nThe Gold layer houses data structured in &#8220;project-specific&#8221;\ndatabases, readily available for consumption. This integration of data from various\nsources may result in a shift in data ownership. Key characteristics of the\nGold layer include:<br><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Gold tables represent data transformed for\nconsumption or specific use cases.<br><\/span><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data is stored in an efficient storage\nformat, preferably Delta.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Gold uses versioning for rolling back\nprocessing errors.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Historization is applied only for the\nrelevant use cases or consumers. So, Gold can be a selection or aggregation of\ndata found in Silver.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Complex business rules are applied in Gold.\nIt utilizes post-processing activities, calculations, enrichments, and\nuse-case-specific optimizations.<br><\/span><span style=\"font-family: &quot;Courier New&quot;; font-size: 13.3333px; text-indent: -24px;\">o<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp; &nbsp;&nbsp;<\/span><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\">Data is highly governed and\nwell-documented.<\/span><\/p>\n<p>Lorem Ipsum has been the industry&#8217;s standard dummy text ever since the 1500s.<\/p>\n<p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Gold\nLayer Complexity:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"> The Gold layer is often the most intricate\nlayer due to its design being dependent on the overall architecture&#8217;s scope. In\na basic setup where your Lakehouses are solely source-system aligned, data in\nthe Gold layer represents &#8220;data product&#8221; data, making it general and\nuser-friendly for distribution to numerous domains. Following this\ndistribution, the data is expected to be housed in another platform, possibly\nanother Lakehouse.<br><br><o:p><\/o:p><\/span><\/p><h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Gold\nLayer Variations Based on Architecture Scope<o:p><\/o:p><\/span><\/h2><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Depending on the breadth\nof your architecture, your Gold layer could be a combination of data product\ndata and use-case-specific data. In such scenarios, the Gold Layer in these\nplatforms caters to the exact needs of analytic consumers. The data is modeled\nwith a shape and structure tailored specifically for the use case in progress.\nThis method fosters principles where you only allow domains to function on\ninternal data that isn&#8217;t directly distributed to other domains. Therefore, some\ntables may be flexible and adjustable, while other tables with a formal status\nare consumed by other entities.<br><br><o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">For broader Lakehouse\narchitectures encompassing both provider and consumer ends, anticipate\nadditional layers, often referred to as workspace or presentation layers. In\nthis configuration, the data in Gold is more universal, integrated, and ready\nfor various use cases. These workspace or presentation layers then contain\nsubsets of data, resembling traditional data modeling within data warehousing.\nEssentially, Gold serves as a universal integration layer from which data marts\nor subsets can be filled.<br><br><o:p><\/o:p><\/span><\/p><h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Gold\nLayer for Data Dissemination and Governance:<o:p><\/o:p><\/span><\/h2><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Certain organizations\nutilize workspace or presentation layers to disseminate data to other platforms\nor teams. They achieve this by selecting and\/or pre-filtering data for specific\nscenarios. Some organizations implement tokenization, which involves substituting\nsensitive data with randomized strings. Others employ additional services for\ndata anonymization. This data can be seen as behaving similarly to &#8220;data\nproduct&#8221; data.<br><br><o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">The Gold layer often\nstands out for its adherence to enterprise-wide standards. While enterprise\ndata modeling can be complex and time-consuming, many organizations still\nutilize it to ensure data reusability and standardization. Here are some\napproaches to consider for achieving standardization in the Gold layer:<strong><br><br><o:p><\/o:p><\/strong><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Enterprise\nData Model:<\/strong><\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\"><strong> <\/strong>The enterprise data model provides an\nabstract framework for organizing data within teams&#8217; Lakehouses. The enterprise\nmodel typically outlines specific data elements and requires teams to follow\nspecific reference values or include master identifiers when sharing datasets\nwith other teams.<o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">&nbsp;<\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><strong>Centralized\nData Harmonization:<\/strong> <\/span><\/span><span style=\"font-size:12.0pt;line-height:\n107%;font-family:&quot;Helvetica&quot;,sans-serif\">An alternative approach involves\ngathering, harmonizing, and distributing data in a separate Lakehouse\narchitecture. From there, data can be consumed downstream by other domains,\nsimilar to a master data management approach.<o:p><\/o:p><\/span><\/p><p style=\"text-align: justify;\">\n<\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span class=\"Heading3Char\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Domain-Specific\nOwnership: <\/span><\/span><span style=\"font-size:12.0pt;line-height:107%;\nfont-family:&quot;Helvetica&quot;,sans-serif\">Another option is to mandate teams to take\nresponsibility for specific harmonized entities. Other teams must then conform\nby consistently using these entities.<br><br><o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">In most cases,\nstandardization and harmonization primarily occur within the Gold layer.<o:p><\/o:p><\/span><\/p>\n<p>Lorem Ipsum has been the industry&#8217;s standard dummy text ever since the 1500s.<\/p>\n<h2 style=\"text-align:justify\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Technology\nConsiderations for the Gold Layer<o:p><\/o:p><\/span><\/h2><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Choosing a database or\nservice for the Gold layer is a complex decision. It requires careful\nconsideration of various factors and involves making compromises. Beyond data\nstructure, you&#8217;ll need to assess your requirements in terms of:<br><strong><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><\/span><\/strong><span style=\"font-family: Inter, sans-serif; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><\/span><strong><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Consistency:<\/span><\/strong><span style=\"font-family: Helvetica, sans-serif; font-size: 12pt; letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); text-indent: -0.25in; line-height: 107%;\"><span style=\"font-weight: var(--single-content-weight);\">\nHow strictly does your data need to be consistent across reads?<\/span><br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Availability:<\/span><\/strong><span style=\"text-indent: -0.25in; letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight);\">\nHow crucial is it for your data to be constantly accessible?<\/span><br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Caching:<\/span><\/strong><span style=\"text-indent: -0.25in; letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight);\">\nCan caching strategies improve query performance?<\/span><br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Timeliness:<\/span><\/strong><span style=\"text-indent: -0.25in; letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight);\">\nHow quickly do you need data updates to reflect in the Gold layer?<\/span><br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Indexing:<\/span><\/strong><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\">\nHow can indexing strategies optimize specific queries?<\/span><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l1 level1 lfo2\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"text-align:justify\"><span style=\"font-size:12.0pt;\nline-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">There is no single service\nthat excels in all of these aspects simultaneously. A typical Lakehouse\narchitecture often comprises a blend of diverse technology services, such as:<br><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><\/span><\/strong><span style=\"font-family: Inter, sans-serif; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Server-less<\/span><\/strong><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%;\">\nSQL services for ad-hoc queries.<br><\/span><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Columnar<\/span><\/strong><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\">\nstores for fast reporting.<br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Relational<\/span><\/strong><span style=\"text-indent: -0.25in; letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\"><span style=\"font-weight: var(--single-content-weight);\">\ndatabases for more complex queries.<\/span><br><\/span><span style=\"font-size: 16px; text-indent: -24px;\">&#8211;<\/span><span style=\"font-variant-numeric: normal; font-variant-east-asian: normal; font-variant-alternates: normal; font-kerning: auto; font-optical-sizing: auto; font-feature-settings: normal; font-variation-settings: normal; font-variant-position: normal; font-stretch: normal; font-size: 7pt; line-height: normal; font-family: &quot;Times New Roman&quot;; text-indent: -24px;\">&nbsp;&nbsp;<\/span><strong style=\"text-indent: -0.25in; font-family: var(--single-content-family); font-size: var(--single-content-size); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color);\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\">Time-series<\/span><\/strong><span style=\"text-indent: -0.25in; font-weight: var(--single-content-weight); letter-spacing: var(--single-content-letterspacing); color: var(--body-text-default-color); font-size: 12pt; line-height: 107%; font-family: Helvetica, sans-serif;\">\nstores for IoT and stream analysis.<\/span><\/p><p class=\"MsoListParagraphCxSpLast\" style=\"text-align:justify;text-indent:-.25in;\nmso-list:l0 level1 lfo1\"><span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif\"><br><o:p><\/o:p><\/span><\/p><p style=\"text-align: justify;\">\n<span style=\"font-size:12.0pt;line-height:107%;font-family:&quot;Helvetica&quot;,sans-serif;\nmso-fareast-font-family:Calibri;mso-fareast-theme-font:minor-latin;mso-ansi-language:\nEN-US;mso-fareast-language:EN-US;mso-bidi-language:AR-SA\">By understanding the\ntheoretical foundations and practical considerations of each layer, you can\neffectively implement Medallion Architecture for your data lakehouse. Remember,\nthe &#8220;best practices&#8221; can vary depending on your specific data\nplatform strategy and the overall scope of your Lakehouse architecture. Choose\nthe approaches and technologies that best align with your unique needs and\ngoals for optimal data management.<br><\/span><br><\/p>\n\n<p>Lorem Ipsum has been the industry&#8217;s standard dummy text ever since the 1500s.<\/p>\n<p class=\"MsoNormal\" style=\"margin-right:.2in;text-align:justify;line-height:\n150%\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\"><\/span><\/p><p style=\"text-align: justify;\"><span style=\"line-height:1.5;\"><\/span><\/p><p class=\"MsoNormal\" style=\"margin-right: 0.2in; text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\"><\/span><\/p><p style=\"text-align: justify;\"><span style=\"line-height:1.5;\"><p class=\"MsoNormal\" style=\"margin-right: 0.2in; text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Dot Labs is an IT\noutsourcing firm that offers a range of services, including software\ndevelopment, quality assurance, and data analytics. With a team of skilled\nprofessionals, Dot Labs offers nearshoring services to companies in North\nAmerica, providing cost savings while ensuring effective communication and\ncollaboration.<br><br><o:p><\/o:p><\/span><\/p><p class=\"MsoNormal\" style=\"margin-right: 0.2in; text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">Visit our website: <\/span><a href=\"http:\/\/www.dotlabs.ai\/\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">www.dotlabs.ai<\/span><\/a><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">, for more information on how Dot\nLabs can help your business with its IT outsourcing needs.<br><br><o:p><\/o:p><\/span><\/p><p>\n<\/p><p class=\"MsoNormal\" style=\"margin-right: 0.2in; text-align: justify;\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">For more informative\nBlogs on the latest technologies and trends <\/span><a href=\"https:\/\/dotlabs.ai\/blogs\/\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\">click\nhere<\/span><\/a><\/p><\/span><\/p><p class=\"MsoNormal\" style=\"margin-right: 0.2in; text-align: justify;\"><a href=\"https:\/\/dotlabs.ai\/blogs\/\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\"><\/span><\/a><\/p><p><\/p><p class=\"MsoNormal\" style=\"margin-right:.2in;text-align:justify;line-height:\n150%\"><a href=\"https:\/\/dotlabs.ai\/blogs\/\"><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\"><\/span><\/a><span style=\"font-family:&quot;Helvetica&quot;,sans-serif\"><o:p><\/o:p><\/span><\/p>\n\n\n\n\n\n\n","protected":false},"excerpt":{"rendered":"<p>Explore the intricacies of Medallion Architecture&#8217;s Bronze, Silver, and Gold layers in data lake-houses, uncovering theoretical concepts, practical applications, and technology considerations for optimal data management and integration.<\/p>\n","protected":false},"author":2,"featured_media":1306,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"pagelayer_contact_templates":[],"_pagelayer_content":"","footnotes":""},"categories":[2,5],"tags":[],"class_list":["post-1305","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-emergingtech","category-trends"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>Medallion Architecture: Practices for Managing Levels | Dot Labs<\/title>\n<meta name=\"description\" content=\"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Medallion Architecture: Practices for Managing Levels | Dot Labs\" \/>\n<meta property=\"og:description\" content=\"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/\" \/>\n<meta property=\"og:site_name\" content=\"Dot Blogs\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/dotlabsai\" \/>\n<meta property=\"article:published_time\" content=\"2024-05-13T07:58:13+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-05-13T07:58:40+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png\" \/>\n\t<meta property=\"og:image:width\" content=\"975\" \/>\n\t<meta property=\"og:image:height\" content=\"415\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"author\" content=\"Basim Khan\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Basim Khan\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"16 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/\"},\"author\":{\"name\":\"Basim Khan\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#\\\/schema\\\/person\\\/78401fb87235f953b1737839e409b455\"},\"headline\":\"Medallion Architecture: Best Practices for Managing Bronze, Silver, and Gold Levels\",\"datePublished\":\"2024-05-13T07:58:13+00:00\",\"dateModified\":\"2024-05-13T07:58:40+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/\"},\"wordCount\":3790,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2024\\\/03\\\/image-8.png\",\"articleSection\":[\"Emerging Technologies\",\"Market Trends\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/\",\"name\":\"Medallion Architecture: Practices for Managing Levels | Dot Labs\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2024\\\/03\\\/image-8.png\",\"datePublished\":\"2024-05-13T07:58:13+00:00\",\"dateModified\":\"2024-05-13T07:58:40+00:00\",\"description\":\"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#primaryimage\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2024\\\/03\\\/image-8.png\",\"contentUrl\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2024\\\/03\\\/image-8.png\",\"width\":975,\"height\":415,\"caption\":\"Centralized Layer Model\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/2024\\\/05\\\/13\\\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Medallion Architecture: Best Practices for Managing Bronze, Silver, and Gold Levels\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#website\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/\",\"name\":\"Dot Blogs\",\"description\":\"A Technology Company\",\"publisher\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#organization\",\"name\":\"Dot Labs\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-BlogsLogo_Gray_TransparentBG_Width320.png.png\",\"contentUrl\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/uploads\\\/2023\\\/04\\\/cropped-BlogsLogo_Gray_TransparentBG_Width320.png.png\",\"width\":320,\"height\":68,\"caption\":\"Dot Labs\"},\"image\":{\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/dotlabsai\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/dotlabs-ai\"]},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/#\\\/schema\\\/person\\\/78401fb87235f953b1737839e409b455\",\"name\":\"Basim Khan\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/litespeed\\\/avatar\\\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138\",\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/litespeed\\\/avatar\\\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138\",\"contentUrl\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/wp-content\\\/litespeed\\\/avatar\\\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138\",\"caption\":\"Basim Khan\"},\"sameAs\":[\"http:\\\/\\\/www.dotlabs.ai\"],\"url\":\"https:\\\/\\\/dotlabs.ai\\\/blogs\\\/author\\\/basim-khan\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Medallion Architecture: Practices for Managing Levels | Dot Labs","description":"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/","og_locale":"en_US","og_type":"article","og_title":"Medallion Architecture: Practices for Managing Levels | Dot Labs","og_description":"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.","og_url":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/","og_site_name":"Dot Blogs","article_publisher":"https:\/\/www.facebook.com\/dotlabsai","article_published_time":"2024-05-13T07:58:13+00:00","article_modified_time":"2024-05-13T07:58:40+00:00","og_image":[{"width":975,"height":415,"url":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png","type":"image\/png"}],"author":"Basim Khan","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Basim Khan","Est. reading time":"16 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#article","isPartOf":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/"},"author":{"name":"Basim Khan","@id":"https:\/\/dotlabs.ai\/blogs\/#\/schema\/person\/78401fb87235f953b1737839e409b455"},"headline":"Medallion Architecture: Best Practices for Managing Bronze, Silver, and Gold Levels","datePublished":"2024-05-13T07:58:13+00:00","dateModified":"2024-05-13T07:58:40+00:00","mainEntityOfPage":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/"},"wordCount":3790,"commentCount":0,"publisher":{"@id":"https:\/\/dotlabs.ai\/blogs\/#organization"},"image":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#primaryimage"},"thumbnailUrl":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png","articleSection":["Emerging Technologies","Market Trends"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/","url":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/","name":"Medallion Architecture: Practices for Managing Levels | Dot Labs","isPartOf":{"@id":"https:\/\/dotlabs.ai\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#primaryimage"},"image":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#primaryimage"},"thumbnailUrl":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png","datePublished":"2024-05-13T07:58:13+00:00","dateModified":"2024-05-13T07:58:40+00:00","description":"Optimize organization and efficiency in complex design projects with effective strategies for managing architectural levels.","breadcrumb":{"@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#primaryimage","url":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png","contentUrl":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2024\/03\/image-8.png","width":975,"height":415,"caption":"Centralized Layer Model"},{"@type":"BreadcrumbList","@id":"https:\/\/dotlabs.ai\/blogs\/2024\/05\/13\/medallion-architecture-best-practices-for-managing-bronze-silver-and-gold-levels\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/dotlabs.ai\/blogs\/"},{"@type":"ListItem","position":2,"name":"Medallion Architecture: Best Practices for Managing Bronze, Silver, and Gold Levels"}]},{"@type":"WebSite","@id":"https:\/\/dotlabs.ai\/blogs\/#website","url":"https:\/\/dotlabs.ai\/blogs\/","name":"Dot Blogs","description":"A Technology Company","publisher":{"@id":"https:\/\/dotlabs.ai\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/dotlabs.ai\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/dotlabs.ai\/blogs\/#organization","name":"Dot Labs","url":"https:\/\/dotlabs.ai\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dotlabs.ai\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2023\/04\/cropped-BlogsLogo_Gray_TransparentBG_Width320.png.png","contentUrl":"https:\/\/dotlabs.ai\/blogs\/wp-content\/uploads\/2023\/04\/cropped-BlogsLogo_Gray_TransparentBG_Width320.png.png","width":320,"height":68,"caption":"Dot Labs"},"image":{"@id":"https:\/\/dotlabs.ai\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/dotlabsai","https:\/\/www.linkedin.com\/company\/dotlabs-ai"]},{"@type":"Person","@id":"https:\/\/dotlabs.ai\/blogs\/#\/schema\/person\/78401fb87235f953b1737839e409b455","name":"Basim Khan","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/dotlabs.ai\/blogs\/wp-content\/litespeed\/avatar\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138","url":"https:\/\/dotlabs.ai\/blogs\/wp-content\/litespeed\/avatar\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138","contentUrl":"https:\/\/dotlabs.ai\/blogs\/wp-content\/litespeed\/avatar\/fe70d225f8e3da97115062685a8b183f.jpg?ver=1776875138","caption":"Basim Khan"},"sameAs":["http:\/\/www.dotlabs.ai"],"url":"https:\/\/dotlabs.ai\/blogs\/author\/basim-khan\/"}]}},"_links":{"self":[{"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/posts\/1305","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/comments?post=1305"}],"version-history":[{"count":11,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/posts\/1305\/revisions"}],"predecessor-version":[{"id":1422,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/posts\/1305\/revisions\/1422"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/media\/1306"}],"wp:attachment":[{"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/media?parent=1305"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/categories?post=1305"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dotlabs.ai\/blogs\/wp-json\/wp\/v2\/tags?post=1305"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}