Micro-Batch

Micro-batch processing introduces a configurable latency by processing data in small, fixed-duration intervals (e.g., 5 seconds, 30 seconds). This creates a predictable upper bound on data freshness, unlike pure streaming's variable latency. The system trades off some of the lowest possible latency for significant gains in throughput efficiency and fault tolerance, as the batch framework can optimize resource usage and provide simpler recovery semantics than continuous operator-based streaming.

The micro-batch interval itself acts as a natural tumbling window. This provides deterministic, non-overlapping windows aligned to processing time, simplifying the logic for time-based aggregations. For example, in a 10-second micro-batch system, a SUM operation will naturally aggregate all events received within each contiguous 10-second period. This contrasts with event-time windowing in pure streaming, which must handle late-arriving data and manage complex watermark mechanisms.

State is managed at the granularity of a batch. The system can persist the intermediate state of a computation (like a count or aggregate) after processing each micro-batch to durable storage. This enables efficient recovery via lineage-based recomputation (e.g., Spark's RDD model) or checkpointing. If a node fails, the system can re-compute the lost micro-batch from the source data or the last checkpoint, providing strong fault-tolerance guarantees that are often simpler to implement than the distributed, continuous state snapshots required in pure streaming.

By grouping small amounts of work, micro-batch engines can achieve high resource utilization. They can schedule tasks efficiently on a cluster, packing multiple operations within a batch. This model also naturally handles backpressure: if a downstream sink is slow, the processing engine can simply delay the start of the next micro-batch cycle, allowing the system to absorb temporary slowdowns without complex signaling protocols. The batch size or interval can be tuned to match the sustainable throughput of the slowest component in the pipeline.

The fundamental trade-off is that micro-batch processing is not real-time. There is always a minimum latency equal to the batch interval. This makes it unsuitable for use cases requiring sub-second or millisecond-level responses, such as high-frequency trading fraud detection or real-time bidding. It is ideal for near-real-time analytics, dashboard updates, and ETL where data freshness on the order of seconds to a minute is acceptable and operational simplicity is valued.

Apache Spark Structured Streaming is the canonical example of a micro-batch engine, where developers write batch-like queries that are executed repeatedly on small batches of data. Google Cloud Dataflow (and Apache Beam) runners can execute pipelines in micro-batch mode. These frameworks abstract the batch cycling, allowing developers to focus on transformation logic. The choice between a framework's micro-batch and pure streaming execution mode is often a key configuration decision based on latency SLOs.

Micro-batching is the engine behind operational dashboards that require near-real-time updates. By processing small batches of user interaction or transaction data every few seconds, it enables dashboards to display metrics like active users, revenue per minute, or system error rates with minimal delay. This provides teams with a timely view of business health without the overhead of sub-second streaming.

• Example: An e-commerce platform updates a live dashboard showing conversion rates and cart abandonment every 5 seconds using micro-batched web event data.

Micro-batching is a standard pattern for replicating database changes to data warehouses, caches, or search indexes. Change Data Capture (CDC) tools capture inserts, updates, and deletes from transaction logs (like MySQL's binlog or PostgreSQL's WAL) and ship them in micro-batches to downstream systems.

• Key Benefit: It provides low-latency data synchronization (often within seconds) while maintaining transactional consistency within each batch, which is critical for maintaining data integrity across systems.
• Tools: Debezium, AWS DMS, and Striim often operate in this mode.

In Internet of Things (IoT) deployments, thousands of devices generate continuous telemetry data (temperature, pressure, GPS coordinates). Micro-batching aggregates this high-volume, low-latency data on edge gateways or in the cloud for efficient processing.

• Use Case: A fleet management system collects GPS pings from vehicles every 10 seconds, batches them for 30 seconds, and then processes the batch to update real-time location maps and calculate estimated arrival times.
• Advantage: It dramatically reduces the number of writes to cloud databases compared to per-event streaming, optimizing cost and throughput.

Financial institutions use micro-batching to score transactions for fraud in near-real-time. Instead of analyzing each transaction in isolation (pure streaming), a micro-batch system aggregates the last 2-5 seconds of transactions. This allows the model to consider cross-transaction patterns and velocity checks (e.g., multiple transactions from the same card in different locations) within the batch window.

• Process: Transactions are ingested, batched for a short interval, and then scored by a machine learning model before a decision (approve/flag/decline) is returned.
• Result: Enables low-latency risk mitigation (1-5 second decisioning) with more contextual analysis than per-event processing allows.

Micro-batching is the foundational model for many modern streaming Extract, Transform, Load (ETL) pipelines that populate cloud data lakes. Systems like Apache Spark Structured Streaming and Delta Live Tables use micro-batches to read from streaming sources (Kafka, Kinesis), apply transformations, and write results as new files (e.g., Parquet) to object storage every 10-30 seconds.

• Advantage over Batch: Provides fresher data for analytics compared to hourly/daily batch jobs.
• Advantage over Pure Streaming: Offers efficient, file-based writes to storage layers like Amazon S3 or Azure Data Lake Storage, which perform poorly with a massive number of tiny, individual writes.

To keep recommendation engines (e.g., "users who bought this also bought") relevant, the underlying user behavior features must be updated frequently. Micro-batching processes recent clicks, views, and purchases to update user and product embedding vectors or feature stores at regular, short intervals (e.g., every 60 seconds).

• Impact: This ensures recommendations reflect very recent user intent, improving click-through rates and engagement compared to models updated only daily.
• Architecture: A common pattern involves streaming events to a queue, using a micro-batch processor (like Spark) to compute updated features, and publishing them to a low-latency serving store (like Redis).

A critical processing guarantee for stateful micro-batch jobs. It ensures each event in a data stream is processed effectively once, and its resulting state updates are applied precisely one time, despite potential node failures, retries, or network duplicates. This is achieved through a combination of:

• Idempotent operations that produce the same result if executed multiple times.
• Transactional writes to output sinks.
• Distributed checkpointing of state. Without exactly-once semantics, micro-batch jobs risk double-counting or data loss, corrupting downstream aggregates and business metrics.

The foundational mechanism for fault tolerance in stateful micro-batch processing. It involves periodically persisting a job's intermediate state (like in-memory aggregates or Kafka offsets) to a durable storage system like HDFS or S3. Key aspects include:

• Recovery Point: On failure, the job restarts from the last successful checkpoint, not from the beginning.
• Performance Trade-off: Frequent checkpointing reduces recovery time but adds I/O overhead, impacting latency.
• Consistency: In frameworks like Apache Spark, checkpoints are often coordinated with barriers in the data flow to ensure a globally consistent snapshot. This is essential for maintaining data integrity across long-running micro-batch pipelines.

A flow control mechanism that becomes vital when a micro-batch job's processing speed cannot keep up with its ingestion rate. Instead of overwhelming and crashing the system, backpressure signals the data source to slow down. Implementation varies:

• Reactive Streams: Uses a pull-based model where the consumer requests data based on its capacity.
• Apache Kafka: Consumers control pace via fetch requests; Spark Structured Streaming can dynamically adjust read limits.
• Buffering: Temporary queues absorb bursts, but uncontrolled buffering leads to increased latency and memory exhaustion. Properly managed backpressure allows micro-batch systems to gracefully handle traffic spikes.

A timestamp-based progress tracker for event time in windowed micro-batch operations. It represents the system's belief that no events with an earlier timestamp will arrive. For example, a watermark of 10:05 means the system assumes all events with event time < 10:05 have been seen.

• Triggering: Windows are evaluated and their results emitted once the watermark passes the window's end time.
• Handling Late Data: Events arriving after the watermark are considered late data. Systems can drop them or route them to a side output for special handling.
• Heuristic: Watermarks are often inferred from observed event timestamps, with a delay threshold to account for network disorder. This mechanism is key to balancing result completeness with low-latency output.

While micro-batch aims for predictable latency, performance is measured by its worst cases. Tail latency refers to the high-percentile latencies (e.g., P95, P99, P99.9) in the distribution of batch processing times.

• Impact: A single slow batch (high tail latency) can stall downstream consumers and break freshness SLOs.
• Causes: Often caused by data skew (uneven work distribution), garbage collection pauses, or noisy neighbor problems in shared clusters.
• Monitoring: Essential to track P99 latency alongside average latency to ensure consistent performance. A growing gap between average and P99 indicates instability. Optimizing for tail latency is crucial for production-grade micro-batch systems.

A safety mechanism for handling records that a micro-batch job repeatedly fails to process. Instead of blocking the entire pipeline or silently dropping data, problematic records are diverted to a dedicated queue or storage location.

• Common Causes: Malformed data, schema evolution mismatches, or transient external API failures.
• Process: After a configurable number of retries, the record is written to the DLQ with error context.
• Operational Benefit: Allows the main pipeline to continue processing healthy data while engineers can later inspect, repair, and potentially replay the quarantined records. This pattern is fundamental to building robust, observable data pipelines.