Feedback Payload Schema

This mandatory block provides the forensic link between feedback and the original model prediction. It prevents feedback leakage and enables precise model version rollbacks.

• request_id: A unique identifier (UUID) for the original inference request.
• model_version: The exact model artifact hash or tag (e.g., gpt-4-0125-preview, resnet-v3.2.1).
• timestamp: The precise time of the original inference, often in ISO 8601 format.
• session_id: A user or interaction session identifier for grouping related events.

Without this, feedback cannot be correctly joined with the logged inputs and outputs for training.

This is the core user or system-provided evaluation of the model's output. It defines the signal type and its metadata.

• signal_type: Categorical label (e.g., explicit_correction, implicit_click, preference_pair, reward_score).
• signal_value: The payload of the signal. For a correction, this is the corrected text. For a rating, it's a scalar (e.g., 1 to 5). For a preference pair, it's the IDs of the chosen and rejected outputs.
• confidence (optional): The user's or system's confidence in the provided feedback.
• feedback_timestamp: When the feedback was given, which may differ from the inference timestamp.

This component captures the origination context of the feedback, crucial for bias detection, segmentation, and feedback enrichment.

• source_application: The client app or service ID (e.g., mobile-app-v2.1, customer-chatbot).
• user_id / actor_id: An anonymized identifier for the source of the feedback.
• geolocation / locale: Context like country code or language setting.
• device_context: Information such as device type, OS, or connection quality.

This data allows engineers to answer questions like, "Is the negative feedback concentrated from a specific app version or region?"

Optional fields reserved for application-specific data and post-processing. These are not used for direct model training but for pipeline logic.

• business_rule_version: Indicates which logic generated a piece of synthetic or derived feedback.
• enrichment_flags: Placeholders for data added later by a Feedback Enrichment Service, such as:
  • Feature attribution scores from the original inference.
  • Session history summaries.
  • Results from a Reward Model Scoring pass.
• pipeline_metadata: Internal tags for routing, priority, or sampling (e.g., { "sampling_cohort": "A", "priority": "high" }).

A critical operational field that ensures forward and backward compatibility as the schema evolves.

• schema_version: A immutable version string (e.g., v1.2.0). Every change to required fields or semantics necessitates a version bump.
• validation: The payload must be validated server-side by a Feedback Validation Service against a formal schema definition (e.g., JSON Schema, Protobuf, Avro). This rejects malformed payloads that could corrupt training datasets.
• Example Validation Rules:
  • request_id is a valid UUID.
  • signal_value matches the expected type for the given signal_type.
  • All required fields are present and non-null.

A concrete, annotated example of a feedback payload for a text generation model.

jsonCopy
{
  "schema_version": "v1.1.0",
  "inference_context": {
    "request_id": "550e8400-e29b-41d4-a716-446655440000",
    "model_version": "llm-chat-assistant-2024-04-15",
    "inference_timestamp": "2024-04-15T10:30:00Z"
  },
  "feedback_signal": {
    "signal_type": "explicit_correction",
    "signal_value": "The capital of France is Paris.",
    "original_output": "The capital of France is Lyon.",
    "feedback_timestamp": "2024-04-15T10:31:05Z"
  },
  "source_context": {
    "application_id": "web-helpdesk-v3",
    "user_id": "usr_7f2c1a",
    "locale": "en-US"
  }
}

This structured event can be directly processed by a Feedback-to-Dataset Compilation pipeline.

The dedicated application programming interface (API) endpoint that receives, authenticates, and initially processes structured feedback payloads from client applications. It acts as the secure gateway into the feedback loop, performing basic validation before passing events to downstream services like stream processors or validation services.

• Primary Function: Accept POST requests containing feedback payloads.
• Key Responsibilities: Rate limiting, authentication, and initial schema compliance checks.
• Output: Validated events are typically published to a message queue or event stream for further processing.

The systematic capture of a model's inputs, outputs, and contextual metadata at the moment a prediction is served. This creates an immutable, traceable record that is essential for later feedback attribution.

• Logged Data Includes: Request ID, timestamp, model version, input features, raw output (logits/embeddings), and final prediction.
• Purpose: Provides the necessary context to correctly join a user's feedback with the exact model state that generated the evaluated output. Without this, feedback cannot be accurately used for training.

The process of correctly linking a feedback event to the specific model inference that generated the associated output. It relies on a shared unique identifier (like a request_id) between the inference log and the feedback payload.

• Critical for: Ensuring model updates are trained on correct input-output-feedback triplets. Misattribution leads to noisy or harmful training data.
• Implementation: Typically involves a join operation in a data pipeline, merging the feedback payload with the stored inference log using the request ID and model version.

The data pipeline that transforms raw, logged feedback events into a curated, formatted dataset ready for model training. This process applies the schema to structure the data.

• Key Steps:
  • Joining feedback payloads with full inference context from logs.
  • Applying feedback validation rules.
  • Executing feedback sampling strategies.
  • Formatting data into the specific structure (e.g., TFRecord, Parquet) required by the training framework.
• Output: A versioned incremental dataset used for retraining or online learning.

A dedicated service that applies integrity checks and business logic to incoming feedback events. It ensures only high-fidelity, legitimate signals enter the training pipeline.

• Checks Performed:
  • Schema compliance (required fields, data types).
  • Logical validity (e.g., a rating within 1-5).
  • Anti-spam and anomaly detection.
  • Contextual plausibility (e.g., feedback timestamp is after inference timestamp).
• Outcome: Invalid payloads are rejected or routed to a dead-letter queue for investigation, protecting the model from corrupt training data.

An architectural pattern where all feedback events are stored as an immutable, append-only sequence. The current state of the feedback dataset is derived by replaying these events.

• Core Principle: The event log is the source of truth, not a derived database table.
• Benefits for Feedback Loops:
  • Provides a complete audit trail for compliance and debugging.
  • Enables easy reconstruction of past dataset states for experiment reproducibility.
  • Naturally supports stream processing and complex event-time analytics.