Event Sourcing for Feedback

The foundational principle where every piece of feedback is stored as an immutable event in an append-only log. Each event is a self-contained record of a state change, such as UserRatedOutput, CorrectionSubmitted, or PreferenceLogged. This log serves as the system of record, enabling:

• Exact reconstruction of the feedback dataset's state at any historical point in time.
• Full traceability from a model's prediction to the feedback it received.
• Resilience against data corruption, as events cannot be altered or deleted, only new corrective events can be appended.

A strictly defined structure for each feedback event, ensuring consistency and enabling reliable processing. A typical schema includes:

• Event ID & Timestamp: A unique identifier and precise time of occurrence.
• Event Type: e.g., explicit_rating, implicit_engagement.
• Inference Context: The request_id, model_version, and input features that produced the related prediction.
• Feedback Signal: The core data (e.g., rating: 1, selected_output: B, dwell_time_ms: 4500).
• Metadata: User session ID, source application, and any enrichment data. This schema is the contract between the feedback ingestion API and downstream consumers.

The process of deriving a current or historical read model (or projection) from the raw event stream. This is how the system answers queries like "What was the average reward for Model v2.1 last Tuesday?"

• Projection Functions: Code that consumes the event stream and builds optimized queryable views (e.g., a rolling accuracy table, a dataset of preference pairs).
• Multiple Views: Different projections can be built for different purposes—one for real-time dashboards, another for training dataset compilation.
• Rebuildability: Any projection can be deleted and recreated from the source event log, ensuring data integrity.

The ability to query the feedback system not just by what happened, but when it happened. This is critical for:

• Diagnosing Model Regression: Identifying exactly when a performance metric began to degrade by analyzing feedback trends over time.
• Causal Analysis: Correlating a drop in feedback quality with a specific model deployment or external event.
• Compliance & Auditing: Providing immutable evidence of model behavior and human feedback for regulatory reviews. This capability turns the event log into a time-series database of model interaction.

A key characteristic where components that generate feedback events are decoupled from those that process them. The event log acts as a durable message bus.

• Producers: Applications, APIs, or user interfaces that append events. They have no knowledge of downstream consumers.
• Consumers: Independent services that subscribe to the event stream, such as:
  • Real-time aggregation services.
  • Feedback-to-dataset compilation pipelines.
  • Drift detection monitors.
• This architecture enables scalability, as new consumers (e.g., a new reward model trainer) can be added without modifying producers.

Operational strategies to manage the unbounded growth of the immutable event log.

• Event Compaction: Periodically replacing a series of fine-grained events with a single coarse-grained summary event (e.g., replacing millions of individual clicks with a daily aggregate). The original sequence remains recoverable for a defined period.
• Snapshots: Periodically saving the full materialized state of a projection (e.g., the current feedback dataset). To rebuild a projection, the system loads the latest snapshot and then applies only the events that occurred after it. This dramatically reduces recovery time for long-running projections.

Event sourcing provides a complete, immutable audit trail of all feedback signals, enabling precise attribution of model changes to specific user interactions. This is critical for debugging performance regressions, complying with regulatory standards like the EU AI Act, and understanding the provenance of training data.

• Example: Reconstructing the exact sequence of user corrections that led a fraud detection model to change its decision boundary.
• Mechanism: Each feedback event (e.g., UserCorrectionApplied) is appended to a log, linked to the original inference request ID and model version.

The event log allows the system to rebuild the exact state of the feedback dataset at any historical point in time. This enables reproducible training runs, A/B testing of different dataset versions, and recovery from corrupted data states by replaying events from a known-good checkpoint.

• Key Benefit: Eliminates "dataset drift" in experiments by guaranteeing the training data is identical to a previous run.
• Process: A training job specifies a log sequence number; the system replays all events up to that point to materialize the dataset.

The immutable event stream serves as the source of truth for real-time feedback aggregation and alerting. Stream processing engines like Apache Flink can consume this log to compute rolling performance metrics (e.g., 5-minute accuracy) or trigger immediate model interventions when feedback patterns indicate rapid concept drift.

• Use Case: A content recommendation system detects a spike in "thumbs down" events for a new topic and automatically reduces the weight of that topic's features within seconds.
• Architecture: Events are published to a durable log (e.g., Apache Kafka), which is then subscribed to by real-time aggregators.

Event sourcing cleanly integrates human review into automated loops. Uncertain predictions or contentious feedback can be routed as events to a HITL gateway. The human's judgment is then appended as a new, higher-fidelity event, enriching the log without disrupting the system's flow.

• Workflow: ModelPrediction → LowConfidenceFlagged → HumanReviewRequested → HumanLabelApplied.
• Advantage: Maintains a complete lineage from automated inference to human-corrected ground truth, which is invaluable for training reward models.

By treating feedback as a queryable event history, teams can perform retrospective analysis to detect systemic biases. Analysts can query the log to see if feedback signals are disproportionately coming from certain user segments or if model corrections exhibit unwanted patterns.

• Example: Querying all ExplicitCorrection events to check if the model is being corrected more frequently for queries from non-native speakers, indicating a potential bias in language understanding.
• Tooling: The event log can be ingested into analytical databases (e.g., ClickHouse) for complex temporal and cohort-based queries.

The event log acts as a natural experience replay buffer for continual learning algorithms. New events can be sampled directly for incremental learning jobs, while older events can be replayed to mitigate catastrophic forgetting. This provides a unified data source for both online and batch retraining strategies.

• Reinforcement Learning Context: Each (state, action, reward, next_state) tuple is stored as an event, enabling efficient sampling for offline RL training.
• Sampling Strategy: Advanced feedback sampling strategies (e.g., prioritized experience replay) can be implemented by processing and indexing the event stream.

The systematic capture of a model's inputs, outputs, and internal states (like logits or embeddings) during live prediction requests. This creates an immutable, traceable record that is the prerequisite for event sourcing.

• Primary Use: Provides the factual context (the "state change") that feedback events will later annotate.
• Critical Data: Must include a unique request ID, model version, timestamp, and full input/output payloads.
• Architecture: Typically implemented as a sidecar process or within the model serving framework itself, writing to a durable log like Apache Kafka or a cloud-based object store.

A predefined, versioned data structure that standardizes the format of all feedback events in the system. It defines the contract between applications generating feedback and the event-sourced log consuming it.

• Core Fields: Must include the inference request ID (for attribution), the feedback signal (e.g., {"rating": 5, "corrected_answer": "..."}), a timestamp, and source metadata.
• Importance: Enforces data quality, enables schema evolution, and allows for automated validation and parsing downstream.
• Example: A JSON Schema or Protobuf definition that is shared as a library across all services that produce or consume feedback.

The downstream pipeline process that transforms the raw event log into a curated training dataset. It is the consumer of the event-sourced feedback stream.

• Process: Involves joining feedback events with their corresponding inference-time logs (via the request ID), applying validation rules, sampling strategies, and formatting the data for model consumption.
• Output: Produces versioned datasets (e.g., incremental datasets) ready for continuous training pipelines or incremental learning jobs.
• Key Challenge: Managing the feedback attribution correctly to ensure each training example is constructed from the precise model state that generated the output being evaluated.

The total time delay between a user interaction with a model's output and the subsequent integration of the resulting feedback into an updated model serving live traffic. Event sourcing is a foundational pattern for managing and measuring this latency.

• Components: Sum of feedback ingestion delay, event processing time, training job duration, and new model deployment time.
• Spectrum: Ranges from near-real-time (seconds-minutes for online learning) to batch-oriented (hours-days for full retraining).
• Trade-off: Lower latency enables faster adaptation to drift or new patterns but increases system complexity and risk. The event log provides a buffer and audit trail to manage this trade-off.

A rule-based or learned policy that automatically initiates a model retraining or update job. It consumes aggregated metrics from the feedback event stream and other monitors.

• Common Triggers: Based on volume of new feedback, statistical drift detection, degradation in a streaming performance metric, or a scheduled cadence.
• Event-Driven: In an event-sourced architecture, the trigger is often itself an event (e.g., "concept_drift_detected_v1") published to a stream, which then kicks off a continuous training pipeline.
• Importance: Automates the decision to learn, moving the system from a passive logger to an active, self-improving application.

The process of correctly and immutably linking a piece of feedback to the exact model version, parameters, and input data that produced the output being judged. This is the core integrity guarantee provided by event sourcing.

• Mechanism: Achieved by storing a unique identifier (like a request_id) with both the inference log event and the subsequent feedback event. The event log's append-only nature preserves this link.
• Consequence of Failure: Without perfect attribution, model updates are trained on feedback meant for a different model state, leading to ineffective or harmful learning—a form of feedback poisoning.
• Audit Value: Enables precise debugging of model regressions by replaying the exact sequence of inferences and feedback that led to a problematic state.