Event Sourcing

The system of record is an append-only, immutable sequence of domain events. Each event represents a fact about a state change that occurred in the system. This log serves as the single source of truth, providing a complete audit trail and enabling deterministic replay of the system's entire history. Events are never updated or deleted, only new events are appended.

• Example: In an order system, events are OrderPlaced, ItemAdded, PaymentProcessed, OrderShipped.
• Key Benefit: Provides a complete, verifiable history for compliance, debugging, and analytics.

The current state of an entity (e.g., an order, a user account) is not stored directly. Instead, it is derived by sequentially applying all relevant past events from the log to an initial state (often empty). This process is known as replaying or hydrating the aggregate. In orchestration, this allows a workflow engine to reconstruct the exact state of a process instance by replaying its event history.

• Mechanism: Current State = Initial State + Σ(Event₁, Event₂, ... Eventₙ)
• Use in Orchestration: Enables fault tolerance; a failed workflow can be restarted and its state perfectly reconstructed from the event log.

Because every state change is captured, event sourcing enables temporal queries. You can query the state of the system as it existed at any point in the past, not just its current state. This "time travel" capability is crucial for debugging complex, long-running workflows and for generating historical reports.

• Example: "What was the status of shipment #12345 at 3:00 PM last Tuesday?"
• Implementation: Replay events only up to the timestamp in question to reconstruct the past state.

Event sourcing is frequently paired with CQRS. This pattern separates the model for updating information (Command) from the model for reading information (Query). Commands generate events that are written to the log. Separate, optimized read models (or projections) are asynchronously built from the event stream to serve queries efficiently.

• Command Side: Handles PlaceOrder, UpdateInventory. Writes events.
• Query Side: Handles GetOrderStatus, ListRecentOrders. Reads from denormalized projections.
• Benefit: Allows independent scaling of read and write workloads and optimization of each for its specific purpose.

As business logic evolves, the structure of events (their schema) may need to change. Event sourcing requires strategies for schema evolution to maintain the ability to replay old events. Common patterns include:

• Upcasting: Transforming an old event version to a new version during replay.
• Adding Optional Fields: New fields can be added as optional to avoid breaking existing replay logic.
• Event Adapters: Code that translates between different event versions.
• Critical Rule: Changes must not break the ability to replay the historical log to reconstruct past states.

To prevent conflicts when multiple commands try to modify the same entity, event sourcing typically uses optimistic concurrency control. This is implemented by storing a version number (e.g., the sequence number of the last event) with the aggregate. A command must specify the expected version it is modifying. If the current version has changed since the command was issued, the write fails, and the command must be retried with the new state.

• Mechanism: Command: "Add item to Order #456, expecting version 7." If the order is already at version 8, the command is rejected.
• Benefit: Enables high-throughput systems by avoiding pessimistic locks, aligning well with distributed, multi-agent orchestration.

CQRS is an architectural pattern that separates the model for updating information (commands) from the model for reading information (queries). It is frequently paired with event sourcing.

• Commands are intent to change state and generate events.
• Queries read from optimized, denormalized projections built from the event stream.
• This separation allows independent scaling of read and write workloads and enables the creation of multiple, purpose-built views of the same event history.

The Saga pattern is a failure management pattern for coordinating a sequence of local transactions across multiple services in a distributed system. Each local transaction updates its own database and publishes an event to trigger the next step.

• In a Choreography-based saga, services listen for events and react autonomously.
• In an Orchestration-based saga, a central coordinator (often a workflow engine) sends commands and listens for outcome events.
• Event sourcing provides a perfect audit trail for saga execution, making recovery and debugging deterministic.

A projection is a derived, read-optimized data model created by processing an event stream. It is the primary mechanism for building queryable state in an event-sourced system.

• Projections subscribe to events and update their state accordingly (e.g., incrementing a counter, updating a row in a SQL table).
• They can be rebuildable from the full event history, ensuring consistency between the source of truth (events) and the query model.
• Common examples include materialized views, document stores for APIs, and in-memory caches.

An event store is the specialized persistence layer for an event-sourced system. It is an append-only database optimized for storing and retrieving immutable events.

• Core operations are append(event) and read_stream(stream_id, from_version).
• Events are typically stored with a sequence number (version) per aggregate/stream to ensure consistency.
• Provides built-in functionality for event subscriptions, allowing projections and other services to react to new events.
• Examples include dedicated databases like EventStoreDB, or message brokers with persistence like Apache Kafka (when used with log compaction).

Domain-Driven Design is a software design approach that focuses on modeling core business logic. Event sourcing aligns powerfully with several DDD tactical patterns.

• Aggregates are consistency boundaries; their state changes are published as domain events.
• The event stream becomes a historical record of all aggregate state transitions.
• Bounded Contexts can communicate asynchronously via events, promoting loose coupling. Event sourcing makes the ubiquitous language of the domain explicitly visible in the event schema.

Deterministic replay is the capability to reconstruct an application's state by processing the exact same sequence of events in the same order. This is the foundational guarantee of event sourcing.

• Enables debugging by recreating the exact conditions that led to a bug.
• Facilitates migration and fixing of projection logic by replaying history into a new version.
• Supports auditing and compliance by providing a complete, verifiable history of all state changes.
• In orchestration, it allows a failed workflow instance to be resumed from the last persisted event.