Before you build the platform, build the purpose

Lessons from three decades in data architecture that nobody puts in the sales deck – By Vrajesh Patel

It was 2am and my pager was going off. Again!

We ran out of tape space on the mainframe. Somewhere in a data center, a batch job had stalled, a file had stopped moving, and a healthcare client’s morning reports were quietly in jeopardy. My job was to get on the MF bridge line, walk an engineer through the JCL over the phone, help them free up memory or tape space, and hope we caught it before anyone showed up to work.

That was my introduction to enterprise data. Painful as it was, it gave me a perspective I carry into every engagement today: the platform is never the point. What the data needs to do, and for whom, always is.

Where We Started

When I joined that healthcare organization, they were mid-way through converting a Mainframe and DB2-based data warehouse over to Informatica ETL pipelines. Files would arrive on the mainframe, COBOL/JCL jobs would parse and stage them, and a Linux server would hand them off to Informatica to load into DB2. Every link in that chain was a potential failure point.

The 2am pages were not anomalies. Tape space. Copybook errors. File formatting issues. Everyone on the operations team had their turn.

Phase 3 brought a migration from DB2 to Teradata and it felt like a real breakthrough at the time. We rebuilt every pipeline in Informatica, introduced what today you would call a medallion-style framework we named Stage, Work, and Base, and the nightly jobs ran faster. The pages stopped coming. The team finally got some sleep.

But here is what did not change. The data still ran in nightly batches. Every morning, business users were opening reports that reflected yesterday. We had upgraded the engine without ever questioning what the engine was actually built for.

Better Tools, Same Old Problem

The industry kept evolving. Hadoop was supposed to solve scale. Hive and Spark were supposed to solve speed. Then Databricks and Snowflake came along, cloud-native and AI-ready, and suddenly organizations had access to tools more powerful than most of them knew what to do with.

I have been through migrations on both sides of this shift, including full conversions from Hadoop ecosystems into Databricks, rebuilding pipelines, porting notebooks, moving entire warehouses to modern architecture. Every time, the technology improved. And every time, the same question came back up.

What exactly are we building this for?

That question is not a technical question. It is a business question. And skipping it is where most platform projects go sideways.

Context Before Platform

Here is what I have seen happen more than once. An organization decides to modernize. They select the platform, stand up the infrastructure, and start moving data in. Technically, everything looks right. Then six months later they realize they have built a data swamp. Data nobody fully trusts, reports that conflict, numbers that vary depending on who ran them.

I have personally seen a situation where three different teams produced three different year-end P&L numbers, all for the same fiscal year. Three teams, three answers. That is not a platform problem. That is a context problem.

Before a single pipeline is built, you need two things: a clear understanding of the context of your data (where it comes from, what it actually represents, who owns it) and a defined set of use cases it needs to serve. Without both, you are not building a data warehouse. You are building an expensive storage problem.

Purpose-Built Means the Business Goes First

Back in the Teradata days, one of the requirements we received went like this: identify beneficiaries living more than 30 miles from their assigned provider and trigger outreach letters to those individuals.

That requirement made us stop and think before we built anything. We had to map the full data flow, define business rules, integrate geo-mapping software to get latitude and longitude, calculate drive time and distance, and wire it all back to the member records. It took time. But the work had a purpose before it had a single line of code.

That is the shift in mindset I am talking about. Use case first. Architecture second. Not the other way around.

What AI Changes, and What It Does Not

That same beneficiary workflow today could be built in Databricks in a fraction of the time. Here is roughly what it would looks like now if we leverage the modern cloud stack: 

• A Databricks MCP (Model Context Protocol) server to orchestrate the workflow
• A geocoding API (Google Maps or OpenStreetMap) to resolve addresses to coordinates
• A routing API to calculate real-time drive time and distance
• Python notebooks to apply the business rules and generate the output

What used to take weeks of requirements gathering and months of development is now days. Business users can type what they need in plain English and the platform generates the query, runs it, and returns results.

But here is the thing. AI accelerates whatever is underneath it. Well-governed, purposeful data makes it enormously useful. A data swamp with AI on top is still a swamp.

Governance Is Not Optional

A purpose-built data warehouse needs more than good architecture. It needs accountable ownership.

Every dataset needs a data owner. Someone responsible for quality, someone who understands the lineage, someone who controls access and is the authoritative voice when two reports disagree. Without that structure, you do not have a single source of truth. You have competing versions of it, and trust erodes a little bit every time the numbers do not line up.

Governance is not a technology problem. It is a people and process problem. The tooling in Databricks and Snowflake can support governance well, but only when someone is actually accountable for acting on what those tools surface. Set that up before you start building pipelines.

The Sequence That Works

Whether you are coming off a mainframe, migrating out of Hadoop, or standing up your first cloud platform, the order of operations matters:

• Understand your data context.  Where does it come from? What does it represent? Who created it and why?
• Define your use cases.  What decisions does this data need to support, and who is making them?
• Assign ownership.  Get the data owner in the room before the architects are. Not after.
• Then pick the platform.  Let the use cases drive the architecture, not the vendor pitch.

Databricks and Snowflake are good tools. The AI capabilities being layered on top are changing what is possible. But the organizations getting real value out of them are not necessarily the ones who adopted fastest. They are the ones who knew what they needed before they started building.

The platform is never the point. Purpose is.

Vrajesh Patel

Author

Follow us on