Definition
Real-time data processing is a system that responds to events and produces results quickly enough to influence decisions or actions as they happen. Real-time is relative. For financial trading, real-time means microseconds. For fraud detection, it means milliseconds to seconds. For inventory tracking, it might mean 30 seconds. The definition depends on the cost of delay. The longer you wait to act, the more value you lose.
The architecture starts with event sources: application code, user interactions, sensors, or database changes. Events flow into a message broker that buffers them. A stream processor reads the broker, applies logic, and outputs results. Results might update a database, trigger an alert, or update a model that influences downstream decisions. The entire path must complete within your latency budget. A delay anywhere breaks the contract.
Real-time differs from batch processing, which waits to accumulate data before processing. It also differs from near-real-time, which accepts a few seconds of latency. True real-time systems are more complex and expensive because every millisecond counts. Most organizations use near-real-time or batch for most workloads, reserving true real-time for cases where the cost of delay justifies the infrastructure burden.
The distinction between real-time and near-real-time matters for architecture decisions. True real-time often requires specialized tools like Apache Flink and careful infrastructure design. Near-real-time can use simpler systems. Understanding actual requirements prevents overbuilding.
Key Takeaways
- Real-time is defined by business requirements, not a fixed number. Subsecond latency is necessary for fraud detection. Thirty seconds is real-time for inventory tracking.
- The typical architecture is event source, message broker, stream processor, state store, and action. Latency is incurred in every stage. Optimization requires finding the slowest component.
- Apache Flink processes events individually for low latency. Spark Structured Streaming micro-batches events for simpler programming at the cost of higher latency.
- State stores track information across events. A fraud detector remembers user history. A personalization system remembers preferences. State must be fast and durable.
- Most organizations implement near-real-time, accepting 5-60 seconds of latency, because true real-time infrastructure is complex and expensive for limited business cases.
- Observability in real-time systems focuses on latency percentiles (P99), throughput, backlog, and end-to-end delay. Monitoring individual component speed misses where the system is actually slow.
Defining Real-Time Requirements
The first question in any real-time project is what latency is actually required. This is often a surprising conversation because teams frequently overestimate requirements. They assume real-time means milliseconds. Deeper questioning reveals that minutes would work fine. A payment system needs to decline fraud in 100ms. A customer waiting to complete a transaction can't wait. A daily analytics report can wait 24 hours. The business impact of delay decreases exponentially with each second. Optimizing to sub-100ms latency when 10 seconds is acceptable wastes resources.
Real-time has degrees. Subsecond latency is expensive. Tens of milliseconds requires specialist infrastructure. Hundreds of milliseconds is achievable on standard platforms. Multiple seconds is achievable on simpler systems. Most teams benefit from defining their latency SLA explicitly. P99 latency of 500ms means 99 percent of transactions complete within 500ms. Communicating this clearly drives architecture decisions. A 500ms SLA allows Spark Structured Streaming with micro-batches. A 50ms SLA requires Flink with per-event processing.
Cost is always the constraint. True real-time infrastructure is 3-5x more expensive than near-real-time equivalents. Higher availability requirements increase cost further. If your use case doesn't justify it, near-real-time or batch is smarter. The difference between waiting 5 seconds and 50ms for a nightly report is invisible to users. Infrastructure engineers know. Users don't. This reality drives most organizations toward near-real-time rather than true real-time.
Event Source to Action: The Data Path
A customer swipes a credit card at a store. The payment terminal sends the transaction to a processor. The processor streams it to a real-time fraud detection system. Within 100ms, the system checks the transaction against known patterns. It approves normal transactions instantly. It declines suspicious transactions. The decision flows back to the terminal. The customer sees approval or decline. This entire sequence is real-time processing.
Each stage adds latency. The network takes 5ms. The broker takes 10ms. Processing takes 50ms. Returning the result takes 5ms. Total 70ms. This leaves only 30ms margin to an SLA of 100ms. If any component gets slower, the SLA breaks. Optimization requires measuring each stage. You might discover the processor is the bottleneck, needing parallelization. Or the broker is the issue, needing faster hardware. Or the network is slow, needing co-location. Without measuring, optimization is guessing.
State lookups are another latency factor. A fraud detector needs to look up user history in a state store. An in-memory cache returns in microseconds. A database query takes milliseconds. A network call to another service takes tens of milliseconds. The choice of state store architecture directly impacts latency. For subsecond requirements, everything must be in-memory. For 10-second latency, you have more flexibility.
Apache Flink for True Real-Time Processing
Flink processes events individually as they arrive, with no batching. An event enters the processor. Logic runs immediately. Results exit immediately. This individual event processing means latency is primarily determined by how long logic takes, not how long you wait for a batch to fill. Flink's state backend stores state durably on disk. Periodically, state is checkpointed. On failure, the processor rewinds to the last checkpoint and replays events. This ensures exactly-once semantics. Each event affects the result once, even during failures.
Flink supports complex operations. Windowing groups events by time. You can compute the average transaction amount per hour. Joins combine multiple event streams. You can correlate user actions with product catalogs. Flink handles all this while maintaining low latency and exactly-once semantics. The programming model is intuitive. You write stateful logic similar to business code. Flink handles distribution and state management transparently. This abstraction made Flink popular among data engineers.
Flink scales to high throughput. A single Flink cluster can process hundreds of millions of events per day. Scaling is achieved by partitioning data. Each partition processes a subset of events independently. Partitioning by user ID means user 1-1000 are processed by instance 1, user 1001-2000 by instance 2. This scale-out approach allows linear scaling with more instances. The trade-off is coordination overhead. Adding instances to a running cluster requires rebalancing partitions, which causes temporary latency spikes.
Spark Structured Streaming: Near-Real-Time Simplicity
Spark Structured Streaming groups events into micro-batches. Typically 100-500ms windows. All events arriving during a 100ms window are processed together. This batching improves throughput but adds 100ms+ latency from waiting for the batch to fill. For use cases tolerating sub-second latency, this is acceptable. For microsecond requirements, Flink is better. The benefit of batching is simpler programming. State management is straightforward. Joins are easier. Aggregations are clearer. Engineers with SQL experience can use Spark SQL. This accessibility made Spark popular in data teams.
Spark Structured Streaming unifies batch and streaming. The same code often works for both. Process historical data with batch Spark. Stream new data with Spark Structured Streaming. Results are identical. This unification simplifies operations. One framework to learn. One set of tools. Most organizations have more Spark expertise than Flink. Choosing Spark Structured Streaming leverages existing knowledge. The latency trade-off is often acceptable for operational dashboards, near-real-time personalization, and most analytics.
Structured Streaming runs on top of Spark. This means it scales to the same clusters. You can add nodes to the Spark cluster to process more data. You pay for consistent infrastructure whether running batch or streaming jobs. Organizations with existing Spark infrastructure find Structured Streaming a natural fit. You avoid learning a new framework. You avoid maintaining another system.
State Management and Durability
Real-time systems are stateful. A system detecting fraud remembers user history. A personalization system remembers user preferences. Without state, you can't perform meaningful operations. State storage is fast but volatile. In-memory stores return in microseconds but lose data if the process dies. Durability requires checkpointing state to persistent storage. Periodically, the processor writes state to disk or a remote service. On failure, the processor restores state from the last checkpoint.
State size grows with the number of users or sessions tracked. If you track state for every user in the world, state becomes enormous. Sharding spreads state across multiple processor instances. User 1-1000 state lives on instance 1. User 1001-2000 on instance 2. This allows state to scale. Queries to state hit the correct instance directly. Queries are fast because they're local. The trade-off is rebalancing complexity. If an instance fails, its state must move to other instances. If you add instances, state must redistribute.
Different processors handle state differently. Flink includes an embedded state store. Processing state is always managed by the processor. Other processors rely on external stores like Redis or DynamoDB. External stores are more flexible but add network latency. Queries to Redis over the network take milliseconds. Embedded stores take microseconds. For subsecond latency, embedded stores are preferred. For 10-second latency, external stores work fine and offer more operational flexibility.
Real-Time vs Near-Real-Time Trade-Offs
The difference is latency and cost. True real-time systems achieve latency in the single-digit milliseconds. They require specialized infrastructure, careful optimization, and expert operations. Infrastructure costs are high. Near-real-time systems tolerate seconds of latency. Latency might be 5-60 seconds depending on design. Infrastructure is simpler. Costs are lower. Most organizations use near-real-time for most workloads because the business impact of 5-second delay is minimal compared to the operational burden.
Real-time is justified when decisions must happen faster than you can be late. Fraud detection needs to act in 100ms. Stock trading happens in microseconds. Autonomous vehicles respond to sensors in milliseconds. Surgical robots respond to surgeon inputs in microseconds. Each has a fundamental business requirement for low latency. Most other workloads don't. Analytics can wait hours. Personalization can wait seconds. Inventory tracking can wait minutes. When latency has no business impact, save money with near-real-time or batch.
The line between real-time and near-real-time is blurry but important operationally. Near-real-time allows you to use proven systems. Spark, batch, and orchestration tools are mature. Real-time requires more specialized knowledge. Flink expertise is less common. Operations are harder. If near-real-time meets your needs, choose it. If true real-time is required, invest in the right infrastructure and expertise. Making this decision explicitly, rather than defaulting to real-time for everything, saves significant costs.
Challenges in Operating Real-Time Systems
Debugging real-time systems is harder than batch because data flows continuously. A batch job has clear inputs and outputs. Reproducing issues is tractable. Real-time data moves constantly. State lives on a distributed cluster. Reproducing failures might require capturing production traffic and replaying it. Logs are voluminous. Searching them for relevant information is tedious. Silent failures are worse than crashes. If processing gets stuck but doesn't error, no alert fires. Hours later, someone notices results are stale. The lack of clear job boundaries makes debugging and monitoring more challenging.
Scaling real-time systems introduces complexity. Adding instances to Spark usually scales linearly. Adding Flink instances requires rebalancing state. During rebalancing, latency spikes. Backlog of unprocessed events accumulates. Eventually the system catches up. But end-users might see degraded experience during rebalancing. Avoiding rebalancing requires overprovisioning capacity. You keep enough headroom to handle spikes without scaling. This costs more than optimal sizing. Capacity planning for real-time systems is harder than batch. Batch needs enough capacity for peak load once a day. Real-time needs enough capacity for sustained peak load 24/7.
Exactly-once semantics requires coordination between the processor and the sink. This coordination is expensive and slow. Many real-time systems settle for at-least-once delivery. Results might be duplicated if failures occur. Downstream systems must handle duplication. Deduplication is possible if you have unique identifiers. But it adds complexity. For many real-time use cases, duplicates are acceptable because they're infrequent and impact is minor. For financial transactions, exactly-once is critical. The cost-benefit must be evaluated per use case.
Monitoring real-time systems is complex because the surface area is larger. Batch jobs have one execution per schedule. Real-time systems run continuously. You must monitor for subtle degradation. Latency creeping from 50ms to 100ms. Throughput declining. State size growing. Backlog accumulating. A minor issue in batch goes unnoticed until the next run. In real-time, it impacts users immediately. Alerting must be precise and actionable. Too many false positives cause alert fatigue. Too few and you miss real problems.
Best Practices
- Define latency requirements explicitly as a percentile, such as P99 latency under 100ms. This drives architecture and tool selection more clearly than vague talk of real-time.
- Measure end-to-end latency, not individual component latency. The overall system latency is what matters. Optimize the slowest stage first before micro-optimizing fast stages.
- Avoid overengineering for latency you don't need. If 30 seconds is acceptable, don't build 30-millisecond infrastructure. The cost difference is substantial.
- Implement comprehensive monitoring of latency percentiles, throughput, backlog, and state size. Real-time systems require continuous visibility to catch subtle degradation.
- Test real-time systems under failure conditions before deploying. Kill instances, degrade networks, delay state stores. Ensure recovery happens automatically within acceptable latency bounds.
Common Misconceptions
- Real-time means instantaneous. In practice, real-time means responsive to business requirements. Fraud detection real-time is 100ms. Inventory tracking real-time is 30 seconds. Both are real-time.
- Real-time systems are always faster than batch. A real-time system can introduce latency waiting for state lookups or processing. Well-designed batch systems can produce results faster for specific queries.
- Flink is always better than Spark for real-time. Flink provides lower latency but is harder to program and operate. Spark Structured Streaming is simpler and sufficient for many real-time use cases.
- Real-time systems are more cost-effective than batch. The opposite is true. Real-time systems cost 3-5x more for the same throughput because infrastructure must run 24/7.
- You need real-time data processing for all use cases. Most analytics and reporting work fine with batch or near-real-time. Defaulting to real-time everywhere wastes resources unnecessarily.
