Getting Databricks right the first time

In my last post I wrote about why context and use cases must come before the platform. This is what that looks like in practice on Databricks. This is the second in a series on modern data architecture. The first post, Before You Build the Platform, Build the Purpose, covers the foundational mindset behind purpose-built data warehouses – By Vrajesh Patel

I have been in rooms where a leadership team decides to adopt Databricks, and the energy is high, the investment is real, and everyone is aligned. Six months later I was in those same rooms when the questions started. Why is the data still wrong? Why does it take two weeks to answer a simple business question? Why does the AI tool keep returning results nobody trusts?

The platform did not fail them. The approach did.

Databricks is one of the most capable platforms in enterprise technology right now. Agentic workflows, natural language querying, real-time ML inference, self-serve analytics built in minutes. But none of that works if the foundation underneath it is not solid. And in organization after organization, I have seen the same foundational mistake: data gets moved into the lake before anyone asks what it is supposed to do there.

The Swamp Forms Fast

The pattern usually starts with good intentions. A leadership team sees a competitor moving on AI and makes the call to modernize. A data lake gets stood up and the operating logic becomes get everything in first, figure out what to do with it later. Storage is cheap. Ingestion is straightforward. It feels like progress.

The problems compound quietly. Teams build pipelines to move data into the lake regardless of whether that data serves an active business need. Those pipelines consume engineering time and create maintenance overhead. Business users assume the data is current, but most implementations are still running hourly or nightly batches. Nobody communicated that clearly, so when users find out, trust takes a hit.

Then the real problem surfaces. Without governance built in from the start, tables multiply. Column names are inconsistent across teams. There is no lineage. PII (Personally Identifiable Information) is scattered with no clear ownership. New team members cannot find anything. The lake has become a liability.

The platform has not failed. The strategy has.

I have seen this play out in organizations that were running Hadoop before Databricks and in organizations that went straight to cloud native. The technology changes but the anti-pattern does not. You cannot fix a swamp by adding more data to it.

Medallion Architecture Is a Business Contract, Not a Tech Pattern

Most practitioners have heard of the Databricks Medallion Architecture. Bronze, Silver, Gold. What is less well understood is that it only delivers value when it is treated as a contract with the business, not just a way to organize folders in a data lake.

Here is how I explain each layer to the teams I work with.

Bronze is your raw ingestion layer. An exact, immutable copy of what came from the source system. It is preserved for audit, reprocessing, and lineage. Bronze is not for business consumption. Think of it as your insurance policy.

Silver is where data becomes trustworthy. Cleansed, conformed, and governed. Business rules apply. Bad records are quarantined. Column names are standardized. Schema is enforced. This is the layer where you stop tolerating ambiguity.

Gold is where data becomes valuable. Gold tables are purpose-built for specific business domains and specific consumers. Do not build Gold for everyone. Build it for specific people answering specific questions.

The sequence matters more than most teams realize. The most common mistake is building pipelines and then asking what business problems they solve. The right direction is the reverse: start with the business question, work backward to what the gold table needs to look like, then backward again through Silver to Bronze. Any pipeline that does not trace to a clear business question is a candidate for elimination.

Unity Catalog: This Is Not Optional

If I could go back and give one piece of advice to every team, I have worked with adopting Databricks, it would be this: instrument Unity Catalog from day one. Not as an afterthought. Not after the pipelines are built. From the start.

I want to tell you about a client that learned this the hard way, because I think it illustrates the cost better than any architectural diagram can.

When I walked in, they were using Databricks as a storage layer, a data lake, running on the Hive Metastore. Unity Catalog was not even in the picture. Data was coming in from multiple sources, pipelines were running, tables were being created. On the surface, it looked like an activity. Underneath, there was no documentation on who was using which tables, no data ownership assigned, no distinction between raw data and data that had been validated. The operating principle was: if it solves a problem, put it in the data lake.

Leadership has changed. The new team recognized the problem and made the decision to do things properly. That created two tracks. One side built an entirely new catalog from scratch, purpose-built, with Bronze, Silver, and Gold clearly defined and use cases driving every design decision. The other side took the existing Hive Metastore environment and migrated it to Unity Catalog, but the underlying problem did not get fixed in the migration. Everything landed in a single schema. No medallion structure. No data ownership. No documentation. Just the same swamp, now inside Unity Catalog.

Here is what costs them. A business user needs near real-time data for a reconciliation report. The data lake could not deliver that. The pipelines were batched, the data was already stale by the time it arrived, and there was no path to get what that user needed.

So the user solved their own problem. They went to the source application, downloaded the data to Excel, and combined it manually with the reference data from the lake to build the report themselves.

That is the moment I use to explain what a data lake built without purpose becomes. The user found a workaround because the system could not be trusted to do the job. And once users start working around the platform, that behavior spreads. Trust does not come back easily.

The data lake was built as a filing cabinet. And nobody builds self-service analytics on a filing cabinet.

Most organizations treat governance as something you layer on after the platform is running. Unity Catalog inverts that. It is the governance foundation that the entire Databricks platform sits on, and it is the single most important architectural decision you will make.

The way to think about it: there is no path from a workload to your data that bypasses Unity Catalog. Every query, every pipeline runs, every model inference flows through it before it can touch a single Delta table. It is not a feature. It is a mandatory pass-through.

What Unity Catalog gives you in practice:

• Fine-grained access control down to the row and column level
• Automatic data lineage across every pipeline and query
• Full audit logs of every access event
• A discovery layer so your teams can actually find data
• Delta Sharing for governed cross-organizational data exchange

The Genie natural language query tool, for example, is only as accurate as the metadata registered in Unity Catalog. An organization that invested in Unity Catalog gets answers it can trust. An organization that skipped governance gets answers that are wrong in ways that are hard to detect. That is a worse outcome than no answer at all, because at least a missing answer tells you something is broken.

What the Platform Unlocks When the Foundation Is Right

Once the medallion structure is in place and Unity Catalog is governing the environment, Databricks stops being a pipeline tool and becomes something closer to an intelligence platform. Here is the sequence of capabilities that open up.

One architecture, all workloads

Traditional stacks forced a choice between a data warehouse (performance and governance, but rigid and expensive) and a data lake (scale and flexibility, but unreliable). Most organizations ended up building both, with pipelines copying data between them and two versions of truth to reconcile. Databricks collapse this. Delta Lake brings ACID (Atomicity, Consistency, Isolation, Durability) transactions, schema enforcement, time travel, and query performance to cloud object storage. One copy of your data. All workloads are running against it.

Delta Live Tables

Delta Live Tables make pipelines declarative. You define what your data should look like at each medallion layer. The platform handles execution order, dependency resolution, retries, and quality enforcement. Data quality expectations are written directly into the pipeline definition. When a record fails, it is dropped, quarantined, or flagged automatically. Your Gold layer stays clean regardless of what the source system sends.

Structured Streaming

Databricks Structured Streaming processes data continuously from sources like Kafka, Event Hubs, or Kinesis, writing to Delta Lake tables with full ACID guarantees. You can make the batch-versus-streaming decision per use case, mix both pipeline types on the same platform, and serve both from the same governed Delta tables. This is what closes the freshness gap that batch-only architectures leave open. The reconciliation report I described earlier is a good example of a use case that would have driven a streaming decision if the question had been asked before the pipeline was built.

Databricks SQL and Photon

Photon is Databricks’ vectorized query engine. It delivers performance that matches or exceeds legacy warehouses on open Delta formats, at cloud storage economics. Your Gold tables serve both SQL analysts and ML engineers from the same source. No sync jobs, no data drift, no explaining to the business why two systems give different answers.

Genie and AI/BI

Genie is the natural language query interface. A business user types a question in plain English, Genie generates the SQL, runs it, and returns a result. The reason it works is because it is grounded in your Unity Catalog metadata. Tables that have been properly described, tagged, and certified return accurate answers. Tables that have not returned noise.

This is the self-service analytics outcome that BI teams have been chasing for years. The CoE (Center of Excellence) stops being the bottleneck. They build the foundation and business users serve themselves. But it only works if the governance work was done first.

Mosaic AI

Because your data lives in the Lakehouse, governed and versioned with full lineage, it is already in the right shape for AI. MLflow tracks experiments and manages model registry. Feature Engineering builds feature tables from silver and gold delta tables, versioned and reused across models. The Mosaic AI Agent Framework enables multi-step agentic workflows governed by Unity Catalog guardrails.

In regulated industries, the auditability matters as much as the capability. When a model makes a prediction, you can trace it back to the exact training data and the exact table row that influenced it. That traceability is the difference between deploying AI and being blocked by compliance.

The Playbook: Where to Start

Whether you are building from scratch or trying to recover from a swamp, the starting moves are the same.

Audit before you build.  Map every existing pipeline to an active business use case. Any pipeline that cannot be traced to a specific business question with a named stakeholder is a candidate for decommission. This exercise is uncomfortable and necessary.

Instrument Unity Catalog immediately.  Register existing datasets. Write table descriptions. Tag columns. Define certified metrics. Do not wait until the platform matures. Do it now, because every AI capability the platform offers depends on this metadata being in place.

Define your Gold consumers before you define your gold tables.  Identify the five to ten highest-value business questions your organization needs answered. Work backward from those to the gold schema, then build Silver and Bronze to support it. Not the other way around.

Set explicit data freshness commitments.  Document and communicate what freshness business users can expect at Bronze, Silver, and Gold. The reconciliation report problem I described was partly a communication failure. When users know what to expect, they stop working around the platform.

Use what the platform already provides.  Before building a custom quality framework, a custom lineage tool, or a custom discovery interface, check whether Databricks already offers it. Custom tooling has maintenance costs. Platform capabilities are Databricks’ problem to maintain, not yours.

AI Readiness Starts with Data Readiness

There is a version of this story that ends well and a version that does not. The difference is not the quality of the AI tools. It is the quality of the data foundation underneath them.

Organizations that did the foundational work, adopted Unity Catalog from day one, built medallion with use cases driving the design, and invested in metadata before it felt urgent, are now accelerating. Organizations that skipped the foundation are in triage, trying to govern a swamp while explaining to business users why the platform they were promised is not delivering.

Databricks is moving fast. The capabilities that seem advanced today will be standard within two years. The organizations building the right foundation now will be ready for what comes next. The ones that did not will be play catch-up on both the data and the AI side at the same time.

You cannot bolt AI onto a data swamp. The sequence is the strategy.

Vrajesh Patel

Author

Follow us on