Embedding Drift Detection Prompt for Semantic Search Outputs

This prompt is for search infrastructure teams and prompt engineers who need to quantify how a prompt change alters the ranking and relevance of semantic search results. Use it when you have modified a prompt that generates, rewrites, or filters search queries, or when you have changed the prompt that synthesizes final ranked results from retrieved documents. The core job-to-be-done is detecting unintended semantic ordering shifts that simple string matching or overlap metrics would miss. The ideal user has already collected a golden set of queries and run both the baseline and candidate prompt versions against the same retrieval pipeline to produce two sets of ranked result lists.

Do not use this prompt to generate search results or to evaluate the absolute quality of a single ranking. It is a meta-evaluation prompt that compares two ranked lists produced by different prompt versions. It assumes the input data is already collected and formatted as paired ranked lists per query. The prompt requires you to supply the query text, the baseline ranked list, and the candidate ranked list for each test case. It works best when the result items have consistent identifiers and textual content that can be embedded for semantic comparison. Avoid using this prompt when the result lists are empty, when the two prompt versions produce structurally incompatible outputs, or when you lack a stable embedding model to power the similarity harness.

Before running this prompt, ensure your golden query set covers the diversity of user intents your search system handles, including edge cases, ambiguous queries, and queries with known relevant documents. The prompt produces a structured drift report with Kendall's tau, top-K overlap, and semantic ordering drift metrics. Use this report as a gate in your CI/CD pipeline: set thresholds for acceptable drift and block promotion if the candidate prompt causes statistically significant ranking degradation. Pair this prompt with a human review step for high-traffic or revenue-critical search surfaces where ranking changes directly impact user experience.

The following prompt template is designed to be pasted directly into your evaluation harness. It compares two ranked lists of search results—a baseline list produced by the current prompt and a candidate list produced by the new prompt—and generates a structured drift report. The report quantifies ranking changes using Kendall's tau, top-K overlap, and pairwise embedding similarity. Use this template as the core instruction for an LLM judge or as the specification for a programmatic evaluation script.

textCopy
You are an expert evaluator for semantic search systems. Your task is to compare two ranked lists of search results for the same query and detect meaningful ranking drift.

## INPUT
- Query: [QUERY]
- Baseline Results (ordered list, each with an ID and text snippet): [BASELINE_RESULTS]
- Candidate Results (ordered list, each with an ID and text snippet): [CANDIDATE_RESULTS]
- Embedding Model: [EMBEDDING_MODEL_NAME]
- Embedding Similarity Threshold: [SIMILARITY_THRESHOLD]

## TASK
1. Compute the Kendall tau rank correlation coefficient between the two lists based on shared result IDs.
2. Calculate the top-K overlap for K=[TOP_K_VALUES], reporting the Jaccard similarity at each level.
3. For each pair of results at the same rank position, compute the cosine similarity between their text embeddings. Flag any pair below the similarity threshold as a semantic drift point.
4. Identify any results that appear in one list but not the other, noting their rank position and text.

## OUTPUT_SCHEMA
Return a JSON object with this exact structure:
{
  "query": "string",
  "kendall_tau": number,
  "top_k_overlap": {
    "[k_value]": number
  },
  "positional_similarity": [
    {
      "rank": number,
      "baseline_id": "string",
      "candidate_id": "string",
      "cosine_similarity": number,
      "drift_detected": boolean
    }
  ],
  "missing_from_candidate": [
    {
      "baseline_rank": number,
      "id": "string",
      "text": "string"
    }
  ],
  "missing_from_baseline": [
    {
      "candidate_rank": number,
      "id": "string",
      "text": "string"
    }
  ],
  "overall_drift_severity": "none" | "low" | "medium" | "high",
  "summary": "string"
}

## CONSTRAINTS
- Only compare results that share an ID across both lists for positional similarity.
- If a result ID appears in both lists but at different ranks, include it in positional_similarity at the rank where it appears in the baseline list, noting the candidate rank.
- Set drift_detected to true if cosine_similarity is below the threshold OR if the result is missing from one list.
- overall_drift_severity must be "high" if kendall_tau < 0.7 or top-K overlap at the largest K is below 0.5.
- Do not fabricate embedding similarity values. If you cannot compute embeddings, state this clearly and set cosine_similarity to null.

To adapt this template, replace each square-bracket placeholder with concrete values from your test harness. The [BASELINE_RESULTS] and [CANDIDATE_RESULTS] placeholders expect an ordered list of objects, each containing at minimum an id and a text field. If your search system returns additional metadata, include only the fields needed for drift detection to keep the prompt focused. The [SIMILARITY_THRESHOLD] should be tuned to your application's tolerance for semantic change; a starting value of 0.85 is common for high-precision use cases. For programmatic evaluation pipelines, consider replacing the LLM judge's embedding computation step with a direct call to your embeddings API and passing the pre-computed cosine similarities as part of the input to reduce model hallucination risk.

This prompt is designed to run as a batch comparison step, not a real-time user-facing call. It compares two ranked lists—a baseline from the current prompt and a candidate from a new prompt version—for the same set of queries. The harness should iterate over a golden query set, collect both ranked lists, and feed each pair into this prompt. The output is a structured drift report per query, which the harness aggregates into a release-gate decision. Do not use this prompt for single-query ad-hoc checks; its value comes from statistical aggregation across a representative query sample.

The implementation loop should: (1) load the golden query set and any per-query context documents; (2) for each query, run the baseline prompt and the candidate prompt to produce ranked result lists; (3) call this drift detection prompt with both lists, the query, and the context; (4) parse the JSON output and extract the drift_severity score and kendall_tau value; (5) aggregate scores across all queries to compute a mean drift severity and the percentage of queries exceeding your threshold. Set a CI/CD gate: if mean drift severity exceeds 0.6 or more than 15% of queries show high drift, block the release. Log every per-query report for post-hoc review. Use a model with strong JSON adherence (e.g., gpt-4o or claude-3.5-sonnet) and set temperature=0 to minimize measurement noise.

Validation must happen at two levels. First, validate the JSON schema of every prompt response—check that kendall_tau is a float between -1 and 1, top_k_overlap is an integer between 0 and K, and drift_severity is one of the allowed enum values. Reject and retry any response that fails schema validation (max 2 retries). Second, validate the harness itself by running a no-change baseline test: compare the same prompt version against itself. The resulting mean drift severity should be below 0.1 and mean kendall_tau above 0.95. If not, your measurement system is too noisy to detect real drift. Fix your golden query set or model parameters before relying on this gate.

For production monitoring, schedule this harness to run weekly against a sample of live queries from your search logs. Store the per-query drift reports in your observability platform alongside the prompt version tags. If you detect a statistically significant increase in drift severity week-over-week without a prompt change, investigate retrieval pipeline degradation, embedding model drift, or data source changes. The prompt itself is stable; the harness should treat unexpected drift as a signal that something upstream broke. Wire alerts to your on-call channel when the weekly drift severity exceeds your release-gate threshold for two consecutive runs.