Embedding Model Drift and Grounding Quality Prompt

Use this prompt when your team is migrating to a new embedding model and needs to verify that retrieval quality and grounding accuracy have not regressed. This playbook is designed for MLOps engineers, search team leads, and RAG developers who must compare production traces from the old and new embedding models side by side. The prompt analyzes paired retrieval results for the same queries, identifies where the new embeddings missed critical evidence, and correlates retrieval gaps with hallucination or grounding failures in the generated output.

The required input is a set of paired production traces—each trace must contain the same user query executed against both the old and new embedding models, the retrieved context from each, and the final generated output from each. Without paired traces, you cannot isolate the embedding model's contribution to any observed quality difference. The prompt expects structured trace data including query text, a list of retrieved document IDs and relevance scores per model version, and the corresponding generated answers. If your observability pipeline does not capture retrieval scores alongside generation traces, you must instrument that collection before this analysis is meaningful.

Do not use this prompt for single-model evaluation, pre-release unit tests without production traces, or when you lack paired trace data from both embedding versions. It is not designed to evaluate the embedding model in isolation—it specifically diagnoses the downstream impact on a RAG system's grounding quality. If your goal is to benchmark embedding models on a static retrieval dataset, use a dedicated retrieval evaluation framework instead. This prompt assumes you are comparing two production systems where the only intentional change is the embedding model, and it will flag any confounding variables it detects in the traces.

This prompt is designed to be wired into an automated trace analysis pipeline. It expects structured trace data from two time windows—one before and one after an embedding model change—and instructs the model to perform a comparative analysis. The goal is not to generate prose, but to produce a structured, machine-readable report that flags specific queries where the new embeddings failed to retrieve critical evidence, leading to a hallucinated or unsupported response. The prompt uses square-bracket placeholders for all dynamic inputs, ensuring it can be parameterized by your observability platform or a script.

textCopy
You are an AI observability analyst specializing in RAG system quality. Your task is to compare two sets of production traces to detect grounding quality regressions caused by an embedding model change.

## INPUT DATA

### Embedding Model Change Details
- **Previous Embedding Model:** [PREVIOUS_EMBEDDING_MODEL_ID]
- **New Embedding Model:** [NEW_EMBEDDING_MODEL_ID]
- **Change Date:** [CHANGE_DATE]

### Trace Batches
You will receive two sets of traces for the same or statistically similar queries. Each trace contains the user query, the list of retrieved documents with relevance scores, and the final generated answer.

**Baseline Traces (Pre-Change):**
[BASELINE_TRACES_JSON]

**Comparison Traces (Post-Change):**
[COMPARISON_TRACES_JSON]

## OUTPUT SCHEMA

Return a single JSON object conforming to this structure. Do not include any text outside the JSON.

{
  "analysis_summary": {
    "total_queries_compared": <integer>,
    "overall_grounding_regression_detected": <boolean>,
    "average_relevance_score_baseline": <float>,
    "average_relevance_score_comparison": <float>,
    "hallucination_rate_baseline": <float>,
    "hallucination_rate_comparison": <float>
  },
  "regressed_queries": [
    {
      "query_id": "<string>",
      "user_query": "<string>",
      "severity": "critical|major|minor",
      "baseline_top_relevance_score": <float>,
      "comparison_top_relevance_score": <float>,
      "evidence_lost": "<string describing the critical evidence the new model missed>",
      "hallucinated_claim": "<string or null>",
      "root_cause_assessment": "retrieval_failure|context_truncation|ranking_degradation|source_conflict"
    }
  ],
  "improved_queries": [
    {
      "query_id": "<string>",
      "user_query": "<string>",
      "baseline_top_relevance_score": <float>,
      "comparison_top_relevance_score": <float>,
      "evidence_gained": "<string>"
    }
  ]
}

## CONSTRAINTS
- Only flag a query as regressed if the new model's retrieved context is missing evidence that was present in the baseline, AND this missing evidence correlates with an unsupported or incorrect claim in the output.
- If a query has a lower relevance score but the output is still fully grounded, classify it under `improved_queries` or omit it.
- For `severity`, use `critical` if the hallucination poses a safety, legal, or financial risk; `major` if it is a clear factual error; `minor` if it is imprecise but not misleading.
- Base hallucination rates on a strict claim-by-claim comparison against the retrieved context for each trace.
- If the input traces do not contain enough information to make a determination, set `overall_grounding_regression_detected` to `false` and leave the arrays empty.

To adapt this template, replace each bracketed placeholder with data from your trace store. The [BASELINE_TRACES_JSON] and [COMPARISON_TRACES_JSON] fields should be populated with arrays of trace objects, each containing at minimum a query_id, user_query, retrieved_documents (with scores), and generated_answer. If your traces are large, pre-filter to a representative sample of high-traffic or high-risk queries before sending them to the model. The output JSON is designed to be ingested directly by a monitoring dashboard or incident ticket. Always run this analysis in a controlled evaluation pipeline—never use raw model output to trigger automated rollbacks without human review of critical severity findings.

This prompt is designed to operate as a gated analysis step within a model migration or continuous monitoring pipeline, not as a one-off manual review. The primary integration point is immediately after an embedding model change (upgrade, vendor switch, or dimension change) where you replay a holdout set of queries through the new retrieval pipeline and compare the results against the previous model's traces. The prompt expects paired trace data—old and new—for the same set of queries, along with retrieval scores and hallucination annotations from your existing eval framework. Wire it as a batch processing step that runs after your re-indexing and re-retrieval job completes, before you promote the new embedding model to production traffic.

The implementation harness should enforce a strict input contract. Each evaluation record must include: a query ID, the original user query, the top-k retrieved document IDs and relevance scores from both the old and new embedding models, the generated answer from each retrieval set, and any hallucination flags or faithfulness scores from your LLM judge. Package these as a structured JSONL file or a database view that the prompt can consume row by row. Add a pre-processing validator that rejects records missing paired retrieval data or hallucination annotations—partial traces produce unreliable drift signals. For high-stakes migrations (healthcare, legal, finance), insert a human review checkpoint that samples flagged regressions before the pipeline proceeds. The prompt's output—a structured JSON report with per-query drift flags, severity classifications, and recommended actions—should be written to your observability store (e.g., a monitoring database or dashboard) with trace IDs linking back to the source data for auditability.

Model choice matters here. Use a model with strong instruction-following and structured output capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent) because the prompt requires precise comparison across multiple evidence dimensions and must not hallucinate drift signals. Set temperature to 0 or near-zero to maximize consistency across batch runs. Implement retry logic with exponential backoff for malformed JSON outputs, and add a post-processing validator that checks the output schema before writing results. Log every run with the prompt version, embedding model versions compared, holdout set size, and summary statistics (queries flagged, severity distribution) so you can track drift trends over successive migrations. Avoid wiring this prompt into real-time serving paths—it is a batch analysis tool, and the latency and cost profile are unsuitable for per-request evaluation.