Amazon Kinesis

A massively scalable, durable real-time data streaming service that continuously captures gigabytes of data per second from hundreds of thousands of sources. Data is ingested into shards—the base throughput unit, each providing 1 MB/sec input and 2 MB/sec output. The service guarantees strict ordering per partition key within a shard and retains data for a default 24 hours, configurable up to 365 days for long-term replayability. This is the foundational low-level primitive for building custom streaming applications.

• Use case: Ingesting clickstream events from a web application for real-time sessionization.
• Key concept: A shard is a sequence of data records with a fixed ingestion rate.

A fully managed service for loading streaming data into destinations. It handles all infrastructure, scaling, and batching automatically. Firehose ingests records, optionally transforms them using an AWS Lambda function for tasks like format conversion or data masking, and delivers the batched output to targets like Amazon S3, Amazon Redshift, Amazon OpenSearch Service, or third-party endpoints like Splunk. It is a zero-administration solution for data delivery, not for real-time custom processing.

• Use case: Persisting raw IoT telemetry to S3 in Apache Parquet format for cost-effective batch analytics.
• Key concept: Near real-time delivery with minimum buffer intervals of 60 seconds.

A service for processing and analyzing streaming data in real time using standard SQL or Apache Flink. With the SQL engine, you can write queries against streaming sources to create materialized views over sliding or tumbling windows. The Flink runtime provides a fully managed, stateful stream processing environment for complex event-driven applications, exactly-once semantics, and custom operators. The output can be sent to a Kinesis Data Stream or Firehose delivery stream.

• Use case: Computing per-minute anomaly scores on a sensor stream using a tumbling window aggregation.
• Key concept: A stagger window reduces late-arriving data issues by opening windows at the arrival of the first matching event.

A purpose-built service to ingest, durably store, encrypt, and index time-encoded video and audio streams. It automatically provisions and elastically scales infrastructure. The platform provides APIs for frame-level time-indexed retrieval and integrates with machine learning services like Amazon Rekognition Video for object detection and face recognition directly on the stream. It supports WebRTC for real-time, low-latency peer-to-peer media streaming for interactive applications like video chat.

• Use case: Building a smart home security camera that streams video for real-time person detection.
• Key concept: A fragment is a self-contained sequence of frames with a single codec configuration.

A Java library that simplifies consuming and processing data from a Kinesis Data Stream. The KCL manages complex distributed computing tasks: it handles load balancing across multiple workers, responds to instance failures, and checkpoints processed records. It uses a lease table in Amazon DynamoDB to coordinate shard assignments among workers. Each worker instantiates a record processor that contains the custom business logic for handling data records.

• Use case: A multi-threaded application consuming a stream with 50 shards, automatically rebalancing when an EC2 instance fails.
• Key concept: The KCL abstracts shard management, allowing developers to focus solely on the record processing logic.

A feature of Kinesis Data Streams that provides a dedicated throughput pipe per registered consumer, bypassing the standard 2 MB/sec per shard shared output limit. Instead of using HTTP/1.1 polling, it pushes records over a persistent HTTP/2 connection with a dedicated throughput of 2 MB/sec per shard per consumer. This eliminates the latency of polling and contention between multiple consumer applications, making it essential for low-latency, high-throughput, multi-tenant streaming architectures.

• Use case: A fraud detection service requiring sub-200ms latency while a separate data lake ingestion pipeline reads the same stream.
• Key concept: Each registered consumer gets its own 2 MB/sec pipe, decoupled from all other consumers.

An architectural pattern that persists the state of a business entity as an immutable sequence of state-changing events. Unlike traditional CRUD systems that store only the current state, event sourcing captures every fact as an append-only log.

• Kinesis Integration: Kinesis Data Streams serves as the durable event log, storing raw events for replay and audit.
• Key Benefit: Enables temporal queries, full audit trails, and the ability to rebuild any past state by replaying the event sequence.
• Example: A retail cart service stores ItemAdded, ItemRemoved, and CheckoutStarted events rather than just the current cart contents.

Command Query Responsibility Segregation separates read and write operations into distinct models. Commands mutate state and are processed asynchronously, while queries are optimized for fast, denormalized reads.

• Kinesis Integration: Kinesis Data Streams carries commands to write-side processors, while Kinesis Data Analytics populates read-optimized materialized views in DynamoDB or ElastiCache.
• Why It Matters: Allows independent scaling of high-write ingestion paths and low-latency query paths.
• Common Pairing: Frequently implemented alongside Event Sourcing to project event streams into queryable views.

An open-source, unified stream-processing framework that provides exactly-once state consistency and sub-second latency for stateful computations over unbounded data streams.

• Kinesis Integration: Available as a managed runtime via Amazon Kinesis Data Analytics for Apache Flink, consuming directly from Kinesis Data Streams.
• Core Capabilities: Sophisticated event-time windowing, complex multi-stream joins, and checkpointing to durable storage.
• Use Case: Detecting fraud patterns by joining a live transaction stream with a slowly-changing customer profile stream over a 10-minute tumbling window.

A design pattern that identifies and captures row-level changes (INSERT, UPDATE, DELETE) in a source database and streams them as real-time events to downstream systems.

• Kinesis Integration: Services like AWS Database Migration Service (DMS) can stream CDC logs from RDS or Aurora directly into Kinesis Data Streams.
• Advantage: Eliminates batch ETL latency by making database changes instantly available to stream processors, search indexes, and caches.
• Example: Streaming inventory count changes from an RDS table to Kinesis, where a Lambda function invalidates the product cache in real-time.

A storage destination for events that a consumer cannot process successfully after all retry attempts are exhausted. DLQs prevent data loss and enable asynchronous failure diagnosis.

• Kinesis Mechanism: Kinesis Data Streams supports an on-failure destination for Lambda consumers, routing poisoned-pill records to an SQS queue or S3 bucket.
• Operational Value: Isolates malformed records without blocking the processing of valid data in the shard.
• Pattern: A monitoring alarm on the DLQ triggers a Data Engineer to inspect the failed payloads, fix the schema mismatch, and replay them.

A centralized service that stores and enforces the contracts for data serialization formats like Apache Avro or JSON Schema. It ensures producers and consumers agree on the structure of data in a stream.

• Kinesis Integration: The AWS Glue Schema Registry integrates natively with Kinesis Data Streams, allowing producers to serialize records with a schema ID rather than embedding the full schema.
• Governance: Enforces compatibility checks (BACKWARD, FORWARD, FULL) to prevent breaking changes from crashing downstream consumers.
• Efficiency: Reduces payload size by replacing verbose schema definitions with a compact fingerprint, critical for high-throughput streams.