Cross-Encoder Reranking Trace Review Prompt

This prompt is for search engineers and RAG developers who need to audit whether a cross-encoder reranker improved retrieval quality or introduced ranking errors in a live production request. The job-to-be-done is a structured before/after relevance analysis: given the initial retrieval ranking and the final reranked order from a production trace, the prompt produces a systematic comparison that identifies which passages moved up or down, whether those moves were justified by relevance to the user query, and where the reranker may have overridden correct initial rankings or failed to correct poor ones. This is not a prompt for designing reranking models, tuning cross-encoder thresholds, or evaluating retrieval pipelines in aggregate across many queries—it is a single-trace diagnostic tool for understanding one reranking decision at a time.

The ideal user is a search engineer or AI reliability engineer who has access to production trace data including the user query, the initial retriever's ranked list with scores, and the cross-encoder's reranked output with scores. Required context includes the raw query string, the full list of retrieved passages in their original order with retriever scores, and the reranked list with cross-encoder scores. Without all three inputs, the prompt cannot produce a meaningful before/after comparison. The prompt is most valuable when investigating specific user-reported relevance failures, calibrating reranker behavior after a model update, or spot-checking whether the cross-encoder is consistently improving ranking quality rather than introducing subtle regressions that aggregate metrics might mask.

Do not use this prompt when you need aggregate reranking quality metrics across thousands of traces, when you lack the initial retriever ranking to compare against, or when the reranker is a learned fusion model rather than a cross-encoder with per-pair scoring. This prompt also should not substitute for offline reranker evaluation using labeled relevance judgments—it is a production trace review tool, not a benchmark replacement. If the trace shows that the reranker made no changes to the ranking order, the prompt will still produce a useful analysis confirming that the initial ranking was preserved, but the value is lower. For cases where the user query is ambiguous or underspecified, expect the prompt to flag uncertainty rather than confidently asserting relevance judgments. Always pair this analysis with human review for high-stakes ranking decisions, especially in regulated domains where ranking errors could cause harm or bias.

This prompt template is designed to be copied directly into your observability or evaluation harness. It instructs the model to act as a search quality auditor, comparing the initial retrieval ranking against the final reranked order from a production trace. The goal is to produce a structured before/after relevance analysis, not just a summary. Use the square-bracket placeholders to inject trace-specific data before execution.

textCopy
You are a search quality auditor reviewing a production retrieval trace. Your task is to compare the initial retrieval ranking against the final cross-encoder reranked order and assess whether the reranking improved relevance.

## INPUT DATA

### User Query
[USER_QUERY]

### Initial Retrieval Results (Ranked List)
[INITIAL_RANKING]

### Final Reranked Results (Cross-Encoder Order)
[RERANKED_ORDER]

### Optional: Relevance Judgments or Click Data
[RELEVANCE_LABELS]

## OUTPUT SCHEMA

Produce a JSON object with the following structure:
{
  "analysis_summary": "A one-sentence summary of whether reranking helped, hurt, or had no effect.",
  "ranking_comparison": [
    {
      "document_id": "string",
      "initial_rank": integer,
      "final_rank": integer,
      "rank_change": integer,
      "relevance_assessment": "improved" | "degraded" | "unchanged",
      "justification": "Brief explanation of the change, referencing query-document fit."
    }
  ],
  "error_flags": [
    {
      "document_id": "string",
      "error_type": "highly_relevant_demoted" | "irrelevant_promoted" | "unexpected_shift",
      "description": "What went wrong and why it matters."
    }
  ],
  "overall_effectiveness_score": integer (1-5, where 5 is a perfect improvement),
  "recommendations": ["Actionable suggestions for tuning the reranker or retrieval pipeline."]
}

## CONSTRAINTS
- Compare every document that appears in either list.
- If a document is only in one list, note it as a new insertion or complete removal.
- Base your relevance assessment strictly on the semantic fit between the query and the document content provided.
- If [RELEVANCE_LABELS] are provided, use them as ground truth to validate your assessments.
- Flag any case where a clearly relevant document was demoted or an irrelevant one was promoted.
- If the reranked order is identical to the initial order, state that explicitly and set the effectiveness score to 3.

To adapt this template, replace the placeholders with data extracted from your production trace system. The [INITIAL_RANKING] and [RERANKED_ORDER] fields should include document IDs, snippets, and original rank positions. If you have human relevance judgments or click-through data, inject them into [RELEVANCE_LABELS] to ground the analysis. For high-stakes search applications where ranking errors directly impact user trust, always route the output to a human reviewer before accepting tuning recommendations. Validate the JSON output against the schema before storing results in your observability dashboard.

This prompt is designed to be called programmatically as part of an automated trace review pipeline, not as a one-off chat interaction. The primary integration point is a post-retrieval audit step that fires after a user session completes or when a trace is flagged for quality review. The application should extract the initial retrieval ranking and the final reranked order from the trace store, format them into the [INITIAL_RANKING] and [RERANKED_ORDER] placeholders, and include the original [USER_QUERY] for context. The [RELEVANCE_CRITERIA] placeholder should be populated from a configuration store that defines what relevance means for your domain—this might include factors like topical match, factual accuracy, authority, recency, or task-specific utility. Do not hardcode these criteria into the prompt; keep them configurable so that different search surfaces can apply different relevance standards without forking the prompt.

Validation and retry logic must be built around the expected JSON output schema. The prompt requests a structured JSON object with before_ranking, after_ranking, improvements, regressions, and overall_assessment fields. Your harness should parse this output and validate that: (1) all document IDs in the before and after arrays match the input IDs exactly, (2) the improvements and regressions arrays contain only valid document IDs from the input set, (3) each entry includes a non-empty reason string, and (4) the overall_assessment field contains one of the expected enum values. If validation fails, retry with the same inputs and append the validation error to the prompt as additional context: "The previous output failed validation: [VALIDATION_ERROR]. Please correct the output and ensure all document IDs match the input set." Set a maximum of two retries before flagging the trace for human review. Log every validation failure and retry attempt with the trace ID, timestamp, and error details for later analysis.

Model choice matters for this workflow. Cross-encoder reranking analysis requires the model to compare document relevance before and after reranking, which demands strong comparative reasoning. Use a model with demonstrated strength on pairwise comparison tasks—frontier models like Claude 3.5 Sonnet or GPT-4o are appropriate here. Avoid smaller or older models that may hallucinate document IDs or produce inconsistent relevance judgments. If cost is a concern, consider sampling traces rather than running this prompt on every session: target traces where the reranker changed the top-3 document order, where user feedback was negative, or where the retrieval pipeline was recently updated. For high-stakes search surfaces where ranking errors have compliance or safety implications, always route outputs to a human review queue before accepting the analysis as ground truth. The prompt's [RISK_LEVEL] placeholder should be set to high for regulated domains, which will cause the prompt to include stronger caveats and recommend human verification.

Tool integration is straightforward but requires careful trace extraction. Your harness needs access to the trace store to pull the initial retrieval results (typically from a vector database or hybrid search endpoint) and the final reranked order (from the cross-encoder service). If your infrastructure logs these as separate trace spans, join them on the session ID and query timestamp. The [INITIAL_RANKING] should include the document ID, rank position, and a content snippet or summary for each document—do not pass full document text unless the trace already contains it, as this will blow out the context window. The [RERANKED_ORDER] should mirror this structure with the post-reranking positions. If your cross-encoder also produces relevance scores, include those as well; the prompt can use them to detect cases where the reranker was uncertain but still reordered documents. Wire the output of this prompt into your observability dashboard as a structured log event, and set up alerts for patterns like "regression count exceeds improvement count for three consecutive traces" or "overall assessment is 'worse' for more than 10% of reviewed traces."

Avoid running this prompt on traces where the initial retrieval returned fewer than three documents or where the reranker made no changes to the ranking order—these cases produce noisy or trivial analyses that waste inference budget. Also avoid using this prompt as a real-time guard during user requests; the analysis is too slow and expensive for synchronous use. Instead, treat it as an asynchronous audit step that feeds into your retrieval quality monitoring and prompt improvement cycle. When the analysis identifies regressions, route the findings to the search engineering team with the specific document IDs and reasons, not just the aggregate assessment. This turns the prompt from a passive diagnostic into an actionable feedback loop for reranker tuning.