Consumer Group

When multiple consumer instances join the same consumer group, the topic's partitions are automatically distributed among them. Each partition is assigned to exactly one consumer within the group, ensuring that every message is processed only once by the group. This enables horizontal scaling—adding more consumers increases throughput up to the number of partitions.

• Example: A topic with 12 partitions and a consumer group with 3 instances results in each instance handling 4 partitions.
• Rebalancing: When a consumer joins or fails, the group coordinator triggers a rebalance, reassigning partitions to the remaining active members.

Within a single consumer group, messaging follows a point-to-point (queuing) model. A message published to a topic is delivered to one and only one consumer instance in the group. This guarantees ordered processing per partition and prevents duplicate work across the group.

• Contrast: Without consumer groups, a publish-subscribe model delivers every message to every subscriber.
• Use Case: Processing an e-commerce order event exactly once across a pool of worker instances.

Each consumer group maintains its own offset per partition, tracking the position of the last successfully processed message. This allows multiple consumer groups to read from the same topic independently, at different speeds, without interfering with one another.

• Commit Strategies: Offsets can be committed automatically at intervals or manually after processing, enabling at-least-once or at-most-once delivery semantics.
• Reset Capability: A group can reset its offset to earliest to reprocess historical data or latest to skip to real-time.