From legacy ETL to modern data engineering: My journey through the stack

I started my career as a data engineer, or as we called it back then, an ETL developer working entirely with structured data and traditional data warehouses. Back then, I learned the foundations: the differences between Inmon and Kimball methodologies, how to write solid DDLs, how to design tables that wouldn’t fall apart under real-world use, and how to actually get good at SQL. Most of my hands-on time went into working with business analysts, translating business rules into ETL logic, primarily in Informatica PowerCenter 5.x. (Yes, it’s been that long.)

Before I wrote a single ETL mapping, I’d spend real time understanding the data model first. That discipline taught me lessons that, many platform migrations later, still hold up:

• Understand the data model. Know the structure before you build anything on top of it.
• Know the business context. Understand why a table exists in the way it does, not just what columns it has.
• Design with purpose. When you’re creating something new, tie it back to a real use case.
• Map out your transformation strategy. This is what surfaces gaps and lets you apply quality checks early, instead of finding out in production.
• Know your platform’s limits. Every database has its quirks. Knowing them upfront saves you weeks later.
• Code for the next person, not just yourself. Clean, documented pipelines are a gift to whoever inherits them — including future you.

These six habits are what let me build ETL workflows that actually held over time, and they turned out to be the foundation for everything that came after.

What a Decade in Production Support Actually Teaches You

Later in my career, I spent close to a decade in production support for a healthcare client, keeping a 24x7x365 ETL environment running. It’s not the part of the job people put on a highlight reel, but it might be the most valuable stretch of my career.

Production support forces you past the data pipeline and into the infrastructure underneath it. I was in Linux environments, writing shell scripts, installing and patching Informatica versions, learning how all the pieces of a legacy platform fit together. That full-stack understanding — not just the ETL logic, but the systems it ran on and gave me a mindset that made my eventual move to the cloud far smoother than it would have been otherwise.

If there’s one thing that stretch taught me, it’s this: the tools you use will keep changing. The stack you’re an expert in today will look dated in five years. Stay grounded in fundamentals and stay open to learning, and you’ll be fine no matter what comes next which is exactly what happened when I moved to AWS.

Learning to See AWS Through an ETL Lens

When I moved to cloud platforms especially AWS my ETL background ramped up faster than I expected. One of the first things I ran into was Amazon S3, which practically everyone was using for storage. To get comfortable with it, I did what I always do with new tech: I compared it to what I already knew.

S3 vs. file systems. Unlike the hierarchical file paths, I was used to, S3 is flat object storage  keys that look like folder paths but don’t behave like them.

The ETL workflow analogy. In PowerCenter, we read from and wrote to file systems directly. In AWS, that same job happens through connectors, ODBC/JDBC drivers, or APIs talking to S3.

Permissions. Traditional ACLs gave way to IAM roles and much more granular policies more powerful, but also more to get wrong.

Operational differences. “Moving” a file in S3 isn’t really a move it’s a copy followed by a delete. Slower, and behaviorally different from what I was used to.

Those comparisons kept adding up, and eventually I found myself drawing a bigger analogy between the legacy stack and the modern one.

The Assembly Line vs. the Smart Factory

Here’s how I’ve come to picture the shift.

Informatica PowerCenter (the legacy stack) is an assembly line in a traditional factory. You manually design every conveyor belt — every ETL mapping — and physically wire up the machines, i.e., the data sources. The environment underneath it all — hardware, OS, databases — needs constant babysitting: patches, upgrades, capacity planning. Everything is tightly coupled, which limits how much you can flex. And scaling means buying and provisioning more servers, which is expensive and slow.

AWS and Databricks (the modern stack) are that same factory, rebuilt as a smart, cloud-native operation. The conveyor belts become programmable robot’s jobs, notebooks, pipelines — that run on demand instead of running because a schedule says so. Raw materials, your data, live in scalable storage like S3 or Delta Lake, and tools like Databricks or Glue can process it in place, without shuffling it into a traditional database first. Infrastructure scales elastically: need to compute, scale your Spark cluster out instantly. The whole architecture becomes modular, automated, and a lot more cost-efficient. And collaboration stops being a series of handoffs — engineers, analysts, and data scientists actually work side by side in shared spaces like Databricks notebooks.

Different Tools, Same Fundamentals

As the stack evolved, so did I working across Oracle, SQL Server, Teradata, Vertica, and more recently Snowflake, Redshift, Postgres, and Databricks.

Here’s the good news: SQL still reigns. If you understand data structures and can write good SQL, you can work on pretty much any platform, legacy or modern. Whether that means stored procedures, orchestration tools, or Python pipelines, the underlying fundamentals don’t change.

The shift wasn’t really about the tools. It was a mindset change. But the core principles I picked up early in my career understand the model, know the business context, design with purpose are exactly as relevant now as they were on day one.

What Actually Carries Over

Tools will keep changing on you. That’s just the job. But if you:

• understand the business problem,
• actually, know your data,
• design clean, scalable pipelines, and
• stay curious,

And you’ll stay relevant no matter what the stack looks like years from now. I’ve watched the tools underneath me change completely, more than once. The fundamentals never did.

Vrajesh Patel

Author

Follow us on