When batch is not enough: The business case for real-time data integration

Overnight batch is the default heartbeat of enterprise data, and for much of the estate it is exactly right. But in a handful of processes (pricing, fraud, compliance, operations), latency carries a cost. The business case for real-time integration starts with those processes, not with the technology.

For most enterprises, data moves on a nightly rhythm. Extracts run after close of business, warehouses refresh in the early hours, and dashboards greet the morning with yesterday’s world. For decades that was not a limitation; it was the design. Most decisions were daily or slower, and batch was the cheapest, most reliable way to serve them.

What has changed is not fashion but decision speed. Prices move intraday, fraud completes in minutes, regulators increasingly expect controls that operate in the moment, and operations teams are judged on how quickly they catch exceptions, not how thoroughly they report them the next day. Real-time data integration, meanwhile, has matured from an exotic capability into an accessible one. But accessible is not the same as necessary, and the weakest possible business case is the one that starts with the technology. The strongest starts where latency is already costing money, and is honest about where it is not.

Where Batch Latency Creates Measurable Cost

Latency cost rarely announces itself. It hides inside ordinary processes as margin quietly conceded, losses quietly absorbed, and labour quietly spent. Four domains account for most of it.

Pricing and revenue. Wherever demand moves intraday and prices move nightly, the gap between the two is a cost. Airlines, hotels, and cruise lines reprice perishable inventory continuously; retailers adjust promotions against competitor moves; distributors quote against volatile input costs. A pricing engine fed by last night’s data is optimizing for a market that no longer exists — selling too cheaply into surging demand and holding price into fading demand. The measurable version is revenue per available unit, quote win rates, and markdown depth against competitors who reprice hourly.

Fraud and risk. Fraud detected in tomorrow’s batch is fraud funded today. The interval between a transaction and its scoring is precisely the window in which losses accumulate: card fraud rings that complete their run in minutes, promotion abuse that scales overnight, account takeovers that monetize before the morning refresh. A model scored in batch can be perfectly accurate and still perfectly late. Here the cost is directly countable:  chargebacks, write-offs, and recovery effort per hour of detection delay.

Compliance. A growing share of regulatory obligations are clock-bound: sanctions screening at the moment of transaction, transaction monitoring that must flag patterns as they form, breach notifications with statutory deadlines, operational-resilience reporting measured in hours. A control that runs after the fact is not a control; it is a report. The cost shows up as findings, remediation programs, and, in the worst cases, penalties whose size dwarfs the integration investment that would have prevented them.

Operations. An overnight failure discovered at 8 a.m. buys a full day of downstream rework. Stock-outs surface after the sale is lost, logistics exceptions cascade for hours before anyone intervenes, and service teams work from a view of the customer that is stale by a shift. Latency converts small exceptions into large interventions, and the invoice arrives as expedited freight, SLA credits, and the reconciliation headcount every batch estate quietly employs.

The discipline that turns these observations into a business case is quantifying the hour: for each candidate process, what does one hour of staleness cost in margin conceded, losses absorbed, penalties risked, or labour spent? Processes with a defensible number belong in the case.

Real Time Is a Spectrum, Not a Switch

Between nightly batch and millisecond streaming lies a wide spectrum: hourly micro-batch, change data capture replicating in minutes, event streams delivering in seconds. Each step up the freshness curve

costs more to build, operate, and govern, always-on pipelines demand always-on monitoring, and out-of-order events demand engineering that daily loads never needed. So the right question is never “how fast can we go?” but “what is the cheapest latency that removes the cost we just quantified?” A fraud decision may genuinely need seconds. A pricing desk may need fifteen minutes. A finance close will never need either (and month-end reporting, model training, and historical analytics remain squarely, correctly, batch).

This matters because over-provisioned freshness is one of the quieter anti-patterns in modern data estates: A streaming platform faithfully serving a decision that is made weekly. Right-sizing latency per process is where the economics of real-time are won or lost.

Three Readiness Questions Before You Invest

Once a process clears the value test, three questions determine whether the investment will actually pay.

1. What actually changes, and how fast? Real-time economics are driven by change rates, not table sizes. A billion-row table that changes a thousand rows an hour is a modest CDC workload; a modest table rewritten every minute is not. Understand peak versus average velocity, late-arriving and out-of-order data, and how volumes will grow, because streaming costs scale with events, and surprises here surface as invoices.
2. Can your sources speak in events? Real-time integration is only as fresh as its slowest source. Modern databases expose change logs; many SaaS platforms do not, hiding behind API rate limits that cap effective freshness regardless of what sits downstream. Legacy systems often cannot be polled harder without endangering the workloads they exist to run. If the system of record cannot emit changes, “real-time” quietly degrades into an aggressive batch schedule in disguise, and the assessment should say so before the procurement does.
3. Can anyone act at that speed? Freshness only creates value at the moment of a decision, an alert, or an automated action. A dashboard reviewed weekly gains nothing from second-level latency. The consumption test is concrete: name the decision that will be made differently, the alert that will fire sooner, or the process that will trigger automatically because the data arrived earlier. If no one can name it, the pipeline delivers cost without value.

There is also a fourth prerequisite: Quality and Governance at speed. Real-time integration amplifies whatever foundation it runs on:  Trusted data arrives faster, and so does bad data, now with less time to catch it. Estates still reconciling nightly discrepancies should repair the governed foundation first; we have written elsewhere about what that end state looks like. Streaming a number no one trusts simply delivers the argument sooner.

Start with an Assessment, Not a Procurement

The most common failure mode in real-time programs is starting with a platform decision and reverse-engineering the justification. We deliberately front-load the opposite. The first two phases of our delivery model exist for exactly this moment. In Discovery & Readiness, we inventory the candidate processes, baseline today’s end-to-end freshness from source system to decision, quantify the cost of the hour for each, and answer the three readiness questions against the real source systems, not the vendor datasheet. 

In Architecture & Proof of Concept, platform selection is earned rather than assumed: the shortlisted pattern (change data capture, micro-batch, or streaming) is proven against the estate’s hardest source and heaviest workload, and evaluated on run cost and operability rather than demo performance.

That sequence matters because the right answer differs by estate, and it is not always the exotic one. In one engagement with a leading global cruise operator, the freshness problem in Salesforce-bound data was not batch itself but architecture: customer and reservation data crossed three separate legacy middleware technologies before reaching the CRM. Collapsing that chain into a centralized Snowflake-based integration framework, with managed ingestion and version-controlled transformation, 

lifted data-delivery SLA adherence from 85% to 95% and cut integration maintenance costs by roughly 30%, delivering the timeliness the business needed without a streaming platform to operate. Sometimes the latency problem is the plumbing, which is precisely why assessment precedes procurement. 

Our heritage runs deepest in the Snowflake ecosystem, where the real-time toolkit has matured rapidly, but the assessment is platform-neutral by design : The pattern should follow the workload, not the partnership.

And because always-on pipelines are operationally demanding, we build them the way we build everything: co-designed with the client’s engineers and delivered by staying embedded through early operation. A streaming estate is not something to hand over in a deck.

Frame the Outcome in Business Terms

No executive funds change data capture. They fund revenue captured, losses prevented, penalties avoided, and hours returned, and a real-time program should be proposed, governed, and reported in exactly that currency. 

The discipline is to define the business metric before the first pipeline is built, and to state results as before-and-after sentences a CFO can audit: Fraud now scored at authorization instead of overnight, shrinking the loss window from hours to seconds; Inventory repriced every fifteen minutes instead of nightly, measured in revenue per available unit; Operational exceptions surfaced in minutes, measured in expedited-freight spend and SLA credits avoided.

The same ledger must carry the other side: the run-rate cost of achieving that freshness: Compute, Licensing, and the Engineering attention an always-on pipeline consumes. If the annual cost of the pipeline exceeds the value of the latency it removes, the business case fails its own test. The strongest real-time programs we see publish both numbers and let the spread make the argument.

Batch Is Not Broken : It Is Just Not Universal

The case for real-time data integration is not that batch failed; it is that a specific, identifiable set of business processes now outruns it. Find those processes and quantify the hour. Right-size the latency to the decision rather than the demo. Test volumes, sources, and consumption honestly before committing. 

Prove the platform on your hardest source, and report every outcome in business terms. If your organization suspects it has crossed that line, or wants to know before a competitor answers the question for it, a short, structured assessment is the right first investment.

Varun Chabbra

Author

Follow us on