Batch processing is the scheduled execution of data processing jobs on accumulated data in fixed intervals. Instead of processing data as it arrives, batch systems collect events, transactions, or logs over a time period. Midnight arrives. A job kicks off that reads all the data collected during the day, applies transformations, and writes results to a destination. The job runs to completion and stops.
This contrasts sharply with streaming, which is continuous and handles each event as it arrives. Batch processing works on finite datasets with clear boundaries. You process all data from 12:00am to 11:59pm. Streaming has no natural boundaries. It runs indefinitely, always waiting for the next event.
Batch processing dominated data infrastructure for decades because it's simpler to build and operate than streaming. A batch job either completes successfully or fails. There's no ambiguity about whether processing is still happening or stuck. Results arrive in well-defined windows, predictable and easy to monitor. Most data warehousing still runs on batch because it's cost-effective at scale and aligns with how analytics teams work.
The modern data stack often combines both. Streaming handles real-time operational needs. Batch handles historical analysis and reporting at lower cost. Understanding when each is appropriate prevents overbuilding complexity into your infrastructure.
Batch processing predates computers as we know them. Banks batch-processed checks overnight in the 1960s. The term evolved into data processing. Early batch systems read data from tape, processed it on mainframes, and wrote results to another tape. Scheduling was manual. Batch windows were fixed. If a job overran its window, it delayed downstream jobs or jobs had to wait until the next day.
Hadoop and MapReduce changed everything around 2005. MapReduce solved the problem of processing petabyte-scale datasets on commodity hardware. You wrote a map function to process data chunks in parallel, a shuffle to aggregate results, and a reduce function to combine them. Hadoop clusters made this accessible. Companies could rent commodity servers and process massive data volumes without specialized hardware. The trade-off was complexity. MapReduce required understanding distributed systems concepts.
Apache Spark emerged around 2010 and addressed MapReduce's main limitation: writing intermediate results to disk between stages slowed everything down. Spark kept data in memory, making iterative algorithms and complex pipelines dramatically faster. Spark's SQL interface made it accessible to analysts without deep programming knowledge. By 2015, Spark had become the standard batch processing framework, used by organizations from startups to tech giants. The evolution was driven by the need to process more data, faster, more cheaply, and more accessibly.
A typical batch pipeline has distinct stages. First, data ingestion. Raw data is extracted from source systems. A database dump from your operational system might produce millions of rows written to a file. Application logs are shipped to cloud storage. Sensor data is collected and staged. All raw data lands in a staging area, usually cloud object storage like S3 or GCS.
Next, transformation. A batch job reads staging data, applies business logic, and produces transformed data. Duplicates are removed. Data types are standardized. Revenue figures are converted to a common currency. Personally identifiable information might be anonymized. This transformation stage is where data is cleaned and shaped to be useful for analysis.
Finally, the load. Transformed data is written to a data warehouse like Snowflake or BigQuery, to a data lake for further processing, or to an operational database for application use. Indexes are updated. Aggregations are precomputed and cached. A successful batch run usually updates multiple downstream systems, triggering cascading updates. ETL pipelines often run dozens of jobs in sequence, each depending on the previous, each finishing before the next starts.
The cost advantage of batch is significant. A streaming pipeline requires always-on infrastructure. A Kafka cluster with brokers running 24/7. A Flink cluster processing events continuously. Network bandwidth is consumed whether traffic is high or low. A batch system uses resources only during the batch window. You spin up a Spark cluster at 10pm, process a day's worth of data in 2 hours, and shut down. You've used compute for 2 hours. The streaming system runs for 24 hours.
This cost difference multiplies at scale. A company ingesting a billion events a day might spend 100,000 dollars a month on a streaming pipeline with Kafka, Flink, and operational overhead. A batch system ingesting the same billion events, processing them into a data warehouse nightly, might cost 20,000 to 30,000 dollars. The gap widens with data volume. At petabyte scale, batch becomes economically dominant. Most organizations use batch for reporting and streaming for operational decisions because the math is clear.
Batch also amortizes operational overhead. Scaling a batch system means adjusting resources for the batch window. Scaling a streaming system means provisioning always-on capacity for peak load. If your peak event rate is 10 times your average, you provision 10x the resources even though you use 1x most of the time. Batch lets you match resources to the actual workload more precisely.
Without orchestration, batch jobs are scheduled with cron, the Unix scheduling utility. A cron entry might run a job at 2am daily. Cron doesn't understand dependencies. If Job A usually takes 1 hour but takes 2 hours due to data volume, Job B scheduled for 3:30am still starts at 3:30am. Job B reads incomplete data from Job A. Results are wrong. Days later, someone notices the discrepancy.
Orchestration systems solve this by making dependencies explicit. Apache Airflow, Prefect, and Dagster all manage directed acyclic graphs (DAGs) where each job is a node. Edges represent dependencies. The system ensures Job B doesn't start until Job A succeeds. If Job A fails, Job B is skipped automatically. If Job A runs late, Job B waits. Orchestration also handles retries. If a job fails transiently, it can retry automatically. Retries reduce operational noise from intermittent failures.
Orchestration is now table stakes for batch systems at any scale. Managing dependencies manually is error-prone and tedious. Teams spend nights debugging failures caused by out-of-order execution. Adopting orchestration eliminates that. Airflow became popular because it solved a real operational problem that every data team faced. It lets non-engineers define complex pipelines as Python code or YAML, making pipelines versionable and reviewable like any other code.
Data warehouses like Snowflake and BigQuery are built for batch workloads. Queries run to completion. Results are returned. There's no streaming ingestion model. Data arrives in batches. Files are staged, then loaded in bulk. This design choice has major implications. Warehouses optimize for high throughput and massive scans. They assume you're processing gigabytes or terabytes at once. Warehouses are terrible for single-row lookups but excellent for aggregate queries across billions of rows.
Batch ETL tools integrate tightly with warehouses. dbt (data build tool) lets analysts write SQL transformations executed by the warehouse. Transformations run in the warehouse itself, removing the need to move data. A dbt model might aggregate sales by region. It runs SQL against the warehouse, producing a new table. Other models depend on this output. dbt orchestrates the execution order, handling failures and retries. This approach is simpler than traditional ETL tools that moved data around.
Most data teams today use a cloud warehouse with dbt or similar transformation tools for their primary analytics workload. This is batch processing, optimized for simplicity and cost. It dominates because it works. Analysts write SQL. Data flows nightly. Reports are fresh by morning. The total cost of ownership is low. This is why batch is not dying. It's evolving, becoming simpler and more integrated with modern cloud platforms.
The choice between batch and streaming is ultimately about latency tolerance and cost. Batch is right when you can wait. Most analytics can wait until morning. Historical trend analysis doesn't need real-time results. Batch is right when data volumes are massive. Processing a petabyte of data nightly is feasible. Processing it continuously would be extremely expensive. Batch is right for data warehouse loading because warehouses are optimized for bulk operations.
Streaming is right when minutes of latency is unacceptable. Fraud detection must respond in seconds. Stock trading happens in microseconds. Real-time personalization updates user models as they interact with your product. Operational monitoring detects system failures in real-time. Streaming is right when data arrives in low volumes continuously. IoT sensors streaming readings. User interactions streaming. Processing millions of events per day is cheap in streaming but expensive in batch infrastructure.
Most organizations use both because they solve different problems. Streaming handles the real-time operational layer. Batch handles the analytics and reporting layer. Events stream into Kafka for operational decisions. They're also batched and loaded into a warehouse nightly for historical analysis. This hybrid approach gives you the best of both. Real-time operational capability. Cost-effective analytics. Neither is universally better. Context determines the right choice.
Batch processing introduces a scheduling hazard that doesn't exist in streaming. Batch jobs have fixed windows. A 6-hour job must complete by 4am for morning dashboards to be fresh. If data volume grows or processing slows, you miss the window. Dashboards show stale data. Alerts fire. Recovery is often manual and stressful because rerunning a 6-hour job during business hours disrupts the entire pipeline schedule. Streaming doesn't have this problem because there's no window. Results arrive continuously as processing completes.
Debugging batch failures is harder at scale because you can't easily reproduce production conditions locally. If a job fails processing 10TB of data, you can't download that data to your laptop. You have to debug in production or find a subset that reproduces the issue. Logs are your only window into what happened. If logging is sparse, troubleshooting takes hours. State management in batch is less intuitive than streaming. If a job fails partway through, did it process some data already? Rerunning it might duplicate results or miss data. Idempotency, ensuring a job can safely rerun, requires careful design and is often overlooked.
Data quality issues hit downstream systems harder in batch because feedback is delayed. A bad data file might corrupt the entire warehouse load. You don't discover it until the next morning when reports are wrong. Streaming systems catch data quality issues faster because problems are visible within minutes. Recovery from batch failures is also more involved. A single failed transformation job can cascade, invalidating all downstream dependencies. If Job B depends on Job A and Job A failed, do you skip Job B or mark it failed? Different orchestration approaches handle this differently.
Scaling batch processing introduces new complexity. As data volume grows, a job that completed in 1 hour now takes 4 hours. You need to either optimize the job or provision more compute. Adding nodes to a Spark cluster should parallelize work, but not all transformations parallelize equally. Some transformations require shuffling data across the network, creating bottlenecks. Debugging performance issues requires understanding how your framework distributes work. Most batch engineers eventually spend weeks tuning jobs, learning parallelism settings and optimization techniques through painful experience.
Batch processing collects data over a time period, then processes it all at once in a single scheduled job. You accumulate transactions, logs, or events during the day. At midnight, a batch job reads all of them, applies transformations, and writes results to a data warehouse. The job runs once, completes, and stops. Batch processing works on finite datasets with clear start and end points. Streaming, by contrast, is continuous and never stops. Batch systems are simpler to reason about because they have boundaries. A batch job either succeeds or fails clearly. Monitoring is straightforward.
Batch processing is cost-effective and scales to enormous data volumes efficiently. Processing 100GB of data in a batch job costs less than streaming that same 100GB. Batch systems amortize overhead like resource allocation and state management across the entire dataset. You pay for a fixed number of compute resources for a fixed duration. Streaming requires always-on infrastructure that processes continuously, even during low-traffic periods. For analytics workloads where you can wait a few hours for results, batch is often the right choice economically. Most data warehousing still relies on batch ETL because it's proven, well-understood, and cost-effective.
MapReduce is a programming model and framework for processing large datasets distributed across a cluster. It divides data into chunks, maps a function across those chunks in parallel, then reduces the results into a final answer. A MapReduce job processing website logs might map over each log file, extracting page views, then reduce across all map outputs to count total views per page. MapReduce made distributed batch processing accessible to engineers without deep systems knowledge. Hadoop implemented MapReduce, democratizing large-scale data processing. Most systems have moved beyond pure MapReduce to Spark, but the fundamental concept of map-reduce-shuffle persists in modern batch frameworks.
Spark introduced in-memory computation while MapReduce wrote intermediate results to disk after each stage, causing slow performance. Spark keeps data in memory between stages, making iterative algorithms and complex transformations much faster. Spark's SQL interface lets analysts write queries in SQL instead of Java, widening access beyond engineers. Spark also unified batch and streaming in one framework, though the batch aspects are more mature. A Spark batch job reading a terabyte from S3, joining multiple datasets, and writing results might run in minutes where the equivalent MapReduce job took hours. Cost per query dropped dramatically, making it feasible to run more exploratory analysis.
Daily batch jobs are the most common. Data accumulates throughout the day, and a job runs at 2am to process everything collected since midnight. Results are available by morning. Hourly batches run every hour, common for large-scale websites tracking traffic or commerce. Weekly batches run Sunday nights for less time-sensitive analyses like customer cohort studies. Some organizations use a hybrid approach: hourly for operational data, daily for reporting, weekly for archival. The schedule depends on how fresh your data needs to be and how long processing takes. If your daily job takes 4 hours, you need to start by 8pm to have results by morning. If adding more data or complexity makes it take 6 hours, your SLA starts breaking.
ETL stands for Extract, Transform, Load. It's a specific pattern within batch processing. Extract reads data from source systems. Transform applies business logic, cleaning and reshaping data. Load writes results to a target like a data warehouse. Not all batch processing is ETL. A batch job that runs statistical analysis on existing data is batch processing but not ETL. Similarly, not all ETL is batch. Some ETL pipelines stream data continuously. But the most common ETL is batch. You extract all transactions from the source database, transform them into a standard format, and load them into the warehouse nightly. ETL is the pattern. Batch is the execution mode.
Batch systems use compute efficiently because they process large chunks of data in one job. A streaming system might maintain a Kafka cluster with multiple brokers running 24/7, a Flink cluster processing events continuously, and associated infrastructure. A batch system spins up a Spark cluster at a specific time, processes data, writes results, and shuts down. For the same workload, batch might cost 30-50 percent of streaming. If you collect a billion events a day and have until morning for processing, batch is far cheaper. The equation changes if you need results within minutes. Then streaming's always-on cost is justified by the business value of speed. Most organizations use both. Streaming for operational alerts and real-time personalization. Batch for reporting and analytics.
Batch jobs have clear dependencies. Job B waits for Job A to finish. Job C waits for both A and B. Without orchestration, you schedule jobs manually or with cron. Cron has limitations. If Job A runs late, Job B starts before A finishes, causing failures. Orchestration systems like Apache Airflow or Prefect solve this. You define a DAG (directed acyclic graph) where each job is a node and edges represent dependencies. The system ensures jobs start only when their dependencies complete. Retries happen automatically if a job fails. If a job runs late, downstream jobs shift automatically. Airflow became popular because it made this easy for data teams. Before Airflow, companies built custom scheduling systems or suffered from fragile cron-based pipelines that broke regularly.
Batch jobs are scheduled windows of risk. If a batch takes 6 hours and runs at 10pm, it should finish by 4am before morning dashboards need fresh data. If processing slows unexpectedly or data volume grows, you miss the window. Dashboards show stale data. Alerts fire. Recovery is manual and stressful. Debugging is harder when data volumes are large. You can't easily run the same job locally with production data. Reproducing failures takes time. State management can be tricky if you need to restart failed jobs. Did you already process half the data? Idempotency, ensuring a job can safely rerun without duplicating work, requires careful design. Data quality issues also compound in batch. A bad data file in the source can break the entire downstream pipeline because there's no real-time feedback until the batch completes hours later.
Partitioning data is the first lever. Instead of reading one massive file, split it by date or geography. A job processing a year of data processes 365 files in parallel, each smaller and faster. Caching intermediate results avoids recomputing expensive steps. If a transformation produces a dataset used by multiple downstream jobs, cache it once. Sampling is useful in development. Test your logic on 1 percent of data before running the full batch. Pushing filters early saves I/O. Read only the columns you need and filter rows before costly transformations. Tuning parallelism matters. Too few parallel tasks underutilizes the cluster. Too many causes contention. Most frameworks have defaults that are reasonable but not optimal. Profiling shows which steps are slow, guiding optimization. Some batches are bottlenecked on I/O, others on CPU. Different bottlenecks need different solutions.
Use batch when latency tolerance is measured in hours. Most analytics fit this category. You can wait until tomorrow morning for yesterday's report. Batch is right for data warehouse loading. You extract from operational systems nightly and load the warehouse. Batch works for machine learning feature engineering. Training data is stable and doesn't change during training. Batch is cost-effective for large data volumes. Petabyte-scale analytics is nearly always batch. Conversely, use streaming when minutes of latency is unacceptable. Fraud detection, real-time personalization, and operational monitoring are streaming workloads. When you need a mix, use both. Stream real-time events for operational decisions. Batch them later for reporting and historical analysis. That's how most modern data platforms work.
Apache Spark dominates for open-source batch processing. Presto and Trino are fast SQL engines for querying data warehouses. dbt (data build tool) simplifies batch ETL by letting analysts write transformations in SQL rather than code. Cloud providers offer managed services like AWS Glue, Google Cloud Dataflow, and Azure Data Factory that handle infrastructure. For specific domains, specialized tools exist. Pandas in Python handles small-to-medium batches. Polars is a newer library with better performance. SQL remains the most common interface for batch transformations, powering most data warehouse tools. The landscape is fragmented. Different teams use different tools depending on their cloud platform, programming language preference, and existing investments. Spark remains the bridge, working across clouds and use cases.
