A partition is an ordered, immutable, append-only sequence of records within a topic. While a topic is a logical category, a partition is the physical unit of storage and parallelism. Each record within a partition is assigned a sequential ID called an offset, which uniquely identifies it. This strict ordering guarantee within a single partition is a critical property for stateful stream processing and event sourcing, ensuring that consumers read events in the exact sequence they were produced.
Glossary
Partition
A partition is a unit of parallelism and scalability in a distributed log, where a topic is divided into multiple ordered, immutable sequences of records distributed across different brokers.
SCALABILITY UNIT
What is a Partition?
A partition is the fundamental unit of parallelism and data distribution in a distributed streaming platform, enabling a single topic to be split across multiple brokers for horizontal scalability.
Partitions are distributed across different brokers in a cluster, allowing a topic's total throughput to scale horizontally beyond the limits of a single machine. Producers can write to different partitions in parallel, and multiple consumers within a consumer group can each read from an exclusive set of partitions, enabling massive concurrent processing. The number of partitions defines the maximum parallelism for a topic's consumers, making it a crucial configuration parameter for performance and scalability.
SCALABILITY UNIT
Core Characteristics of Partitions
Partitions are the fundamental building blocks that transform a logical topic into a physically scalable, parallel processing unit within a distributed streaming platform.
01
Ordered, Immutable Sequence
A partition is a totally ordered, append-only log of records. Each record is assigned a sequential offset, a unique identifier that strictly increases within the partition. This immutability guarantees that once a record is written, it cannot be modified, providing a durable source of truth. Cross-partition ordering is not guaranteed; only records within the same partition maintain strict temporal sequence.
02
Unit of Parallelism
Partition count directly determines the maximum consumer parallelism for a topic. A topic with N partitions can be consumed by at most N consumer instances within a single consumer group. Each partition is assigned exclusively to one consumer in the group, enabling linear horizontal scaling. Adding more partitions is the primary mechanism for increasing throughput beyond the capacity of a single broker.
03
Data Distribution via Key Hashing
Producers assign records to partitions using a partitioning strategy. The default strategy computes a hash of the record's key modulo the partition count: hash(key) % num_partitions. Records with null keys are distributed in a round-robin fashion. Records sharing the same key are guaranteed to land in the same partition, preserving order for that logical entity (e.g., all events for a specific user_id).
04
Physical Distribution Across Brokers
Partitions of a single topic are physically distributed across multiple brokers in a cluster. Each partition has one leader replica that handles all reads and writes, and zero or more follower replicas for fault tolerance. This distribution spreads the I/O load and storage requirements, preventing any single node from becoming a bottleneck. A topic's total throughput is the aggregate capacity of all its partition leaders.
05
Retention and Compaction Policies
Partitions enforce data retention through two mechanisms:
- Time-based retention: Records are discarded after a configurable age (e.g., 7 days), regardless of consumption.
- Size-based retention: Old segments are deleted when the partition exceeds a byte threshold. Alternatively, log compaction retains only the latest record for each unique key, transforming the partition into a snapshot of current state, ideal for changelogs and restoring state stores.
06
Consumer Offset Management
Consumers track their position in each partition using the offset of the last processed record. This offset is periodically committed to a special internal topic (__consumer_offsets). On restart or failure, a consumer resumes from its last committed offset, enabling at-least-once delivery semantics. Managing offsets manually allows precise control for exactly-once patterns by committing offsets transactionally alongside output results.
DISTRIBUTED SYSTEMS FUNDAMENTALS
Frequently Asked Questions About Partitions
A partition is the fundamental unit of parallelism and scalability in a distributed log. Understanding how partitions work is essential for any engineer building high-throughput streaming data pipelines. These answers address the most common architectural and operational questions about partitioning strategies.
A partition is an ordered, immutable, and append-only sequence of records within a topic that is distributed across multiple brokers in a cluster. While a topic is a logical category for messages, a partition is the physical unit of storage and parallelism. Each record within a partition is assigned a unique, sequential identifier called an offset, which consumers use to track their reading position. Partitions enable horizontal scaling because multiple consumers within a consumer group can read from different partitions concurrently, dramatically increasing throughput. The total throughput of a topic is bounded by the number of partitions, as each partition can only be consumed by one consumer instance within a group at a time. Partitions also provide fault tolerance through replication: a leader partition handles all reads and writes, while follower partitions on other brokers replicate the data, ensuring durability if a broker fails.
Enabling Efficiency, Speed & Accuracy
Intelligent Analysis, Decision & Execution
We build AI systems for teams that need search across company data, workflow automation across tools, or AI features inside products and internal software.
Talk to Us
Search across company data
Give teams answers from docs, tickets, runbooks, and product data with sources and permissions.
Useful when people spend too long searching or get different answers from different systems.
Enterprise searchRAGPermissions
Read more
Automate internal workflows
Use AI to route work, draft outputs, trigger actions, and keep approvals and logs in place.
Useful when repetitive work moves across multiple tools and teams.
AI agentsWorkflow automationGovernance
Read more
Add AI to products and internal tools
Build assistants, guided actions, or decision support into the software your team or customers already use.
Useful when AI needs to be part of the product, not a separate tool.
AI integrationDecision supportModel routing
Read moreDISTRIBUTED DATA PRIMITIVES
Partition vs. Shard vs. Queue
A comparison of three fundamental abstractions for distributing data across nodes in streaming and storage systems, clarifying their distinct ordering, consumption, and scalability semantics.
| Feature | Partition | Shard | Queue |
|---|---|---|---|
Primary Purpose | Ordered, immutable log for parallel consumption | Horizontal data distribution by key range | Decoupled, asynchronous message delivery |
Ordering Guarantee | Strictly ordered within partition | Unordered within shard | Typically FIFO within queue |
Consumption Model | Publish-subscribe; multiple consumer groups | Direct read by key; single owner | Point-to-point or competing consumers |
Message Retention | Durable, time/size-based retention | Persistent until deleted | Removed after acknowledgment |
Typical System | Apache Kafka, Apache Pulsar | MongoDB, Elasticsearch | RabbitMQ, Amazon SQS |
Parallelism Unit | One consumer per partition in group | Shard key determines node | Multiple consumers compete for messages |
Replayability | |||
Use Case | Event sourcing, stream processing | Database scaling, load distribution | Task distribution, job workers |
About the author
Prasad Kumkar
CEO & MD, Inference Systems
Prasad Kumkar is the CEO & MD of Inference Systems and writes about AI systems architecture, LLM infrastructure, model serving, evaluation, and production deployment. Over 5+ years, he has worked across computer vision models, L5 autonomous vehicle systems, and LLM research, with a focus on taking complex AI ideas into real-world engineering systems.
His work and writing cover AI systems, large language models, AI agents, multimodal systems, autonomous systems, inference optimization, RAG, evaluation, and production AI engineering.
LinkedIn
Limited slotsGet a Free AI Consultation
Partnered with leading AI, data, and software stack.
How We Work
Custom AI workflows for your Business
One-fit-all AI don't work for modern businesses. At Inferensys, we aim to understand your business & custom requirements; which we use to define most efficient agentic workflows, the data, and the tools for your business.
01
Review the use case
We understand the task, the users, and where AI can actually help.
Read more02
Pick the right approach
We define what needs search, automation, or product integration.
Read more03
Build the first useful version
We implement the part that proves the value first.
Read more04
Improve from there
We add the checks and visibility needed to keep it useful.
Read moreThe first call is a practical review of your use case and the right next step.
Talk to Us