Vector Similarity Threshold Calibration Prompt

This prompt is designed for infrastructure and search engineers who need to make a data-driven decision about their vector search similarity threshold. It is not a prompt for live retrieval or real-time filtering. Instead, it operates on a batch of sampled production traces where retrieved results fell near the current threshold boundary. The prompt analyzes each trace's query, retrieved document snippets, similarity scores, and downstream relevance signals—such as click data, user feedback, or LLM-as-judge scores—to produce a recommendation to raise, lower, or keep the current threshold. The core job-to-be-done is turning ambiguous boundary cases into a clear, evidence-backed tuning action before you modify index configuration, reranking weights, or fallback behavior.

Use this prompt when you have access to a representative sample of production traces that include both the similarity score and a reliable relevance outcome for each retrieved item. A reliable outcome is critical: without it, the prompt cannot distinguish between a threshold that is too permissive and one that is correctly calibrated. Ideal inputs include traces where a human labeled relevance, a user clicked the result, or an LLM judge provided a calibrated score. Do not use this prompt if your only signal is the raw similarity score itself, as this creates a circular analysis. The prompt is also inappropriate for real-time decision-making, for tuning brand-new indices with no production traffic, or for cases where the retrieval pipeline has changed so significantly that historical traces are no longer representative.

After running this prompt, you will receive a structured recommendation with supporting evidence, including precision-recall trade-offs at alternative threshold values. The next step is to review the flagged boundary cases manually to confirm the analysis aligns with your quality bar before applying the change. Avoid the temptation to automate threshold changes directly from this prompt's output without human review, especially in high-stakes or regulated domains where a drop in recall could mean missing critical evidence. Pair this analysis with a separate prompt for evaluating the impact of the change on context window utilization and answer faithfulness to ensure the tuning decision improves the overall system rather than optimizing one metric in isolation.

This prompt template is designed to be dropped directly into your trace analysis harness. It takes a production trace containing vector search results, their similarity scores, and the downstream relevance outcome, and produces a structured recommendation for adjusting your similarity threshold. The prompt forces the model to reason about the precision-recall tradeoff explicitly, rather than suggesting a threshold in isolation. Replace every square-bracket placeholder with real data from your observability platform before execution.

textCopy
You are a retrieval quality engineer reviewing a production trace from a vector search pipeline. Your task is to recommend whether the current similarity threshold should be raised, lowered, or kept unchanged, based on the trace evidence.

## INPUTS

### Current Threshold Configuration
- Current similarity threshold: [CURRENT_THRESHOLD]
- Similarity metric: [METRIC_TYPE, e.g., cosine, dot_product, euclidean]
- Threshold operator: [OPERATOR, e.g., greater_than_or_equal, less_than]

### Trace Data
For each retrieved document in this trace, the following fields are provided:

[TRACE_RESULTS]
<!-- Format as a JSON array of objects with keys: doc_id, similarity_score, was_used_in_answer (boolean), was_relevant_to_query (boolean), retrieval_rank -->

### Query Context
- User query: [USER_QUERY]
- Retrieval index or collection name: [INDEX_NAME]
- Total documents retrieved before threshold filter: [PRE_FILTER_COUNT]
- Total documents after threshold filter: [POST_FILTER_COUNT]

## OUTPUT SCHEMA

Return a JSON object with the following structure:
{
  "recommendation": "raise" | "lower" | "maintain",
  "recommended_threshold": number,
  "confidence": "high" | "medium" | "low",
  "rationale": "string explaining the tradeoff",
  "precision_recall_analysis": {
    "current_estimated_precision": number,
    "current_estimated_recall": number,
    "projected_precision_at_recommended_threshold": number,
    "projected_recall_at_recommended_threshold": number
  },
  "flagged_documents": [
    {
      "doc_id": "string",
      "similarity_score": number,
      "issue": "below_threshold_but_relevant" | "above_threshold_but_irrelevant" | "borderline"
    }
  ],
  "risks": ["string describing a risk of this recommendation"]
}

## CONSTRAINTS
- Do not recommend a threshold change unless the trace evidence clearly supports it. If the sample is too small or noisy, set confidence to "low" and recommend "maintain."
- When calculating projected precision and recall, explain your estimation method in the rationale.
- Flag every document where the similarity score and relevance label disagree with the current threshold decision.
- If the trace contains fewer than 10 retrieved documents, note this as a sample-size limitation in the risks array.
- Do not invent document IDs, scores, or outcomes. Use only the data provided in [TRACE_RESULTS].

After pasting this template, the most critical adaptation step is populating the [TRACE_RESULTS] array with production data that includes both the similarity score and a ground-truth relevance label. The relevance label can come from downstream signals such as whether the chunk was cited in the final answer, whether a user clicked the source, or an explicit human annotation. Without a reliable relevance signal, the prompt cannot produce a meaningful threshold recommendation. If your observability pipeline does not yet capture relevance outcomes, start by instrumenting citation usage or user feedback before running this calibration prompt.

This prompt is designed to be executed as a batch analysis step, not a real-time user-facing call. It should be wired into an offline trace analysis pipeline that periodically samples production traces where vector search was invoked. The pipeline should filter for traces where the maximum similarity score of retrieved chunks falls within a configurable band (e.g., 0.65–0.85) or where a downstream quality signal (like user feedback or a low answer-faithfulness score) indicates a potential retrieval failure. The prompt's output—a structured threshold recommendation with precision-recall tradeoffs—should be written to an observability database or a calibration report, not returned directly to an end user.

The implementation harness must enforce strict input validation before the prompt is called. The [TRACE_DATA] placeholder expects a JSON object containing the user query, a list of retrieved chunks with their vector similarity scores, and the final generated answer. Validate that similarity scores are floats between 0 and 1, that chunk IDs are unique, and that the trace includes a model-generated answer to compare against. If the trace is missing any of these fields, reject it before the LLM call and log an INVALID_TRACE_INPUT event. For model choice, use a model with strong reasoning capabilities (such as GPT-4o or Claude 3.5 Sonnet) because the task requires comparing nuanced relevance judgments across multiple chunks. Set temperature=0 for deterministic, repeatable analysis. Implement a retry wrapper with a maximum of 2 retries on 500-level errors or timeout exceptions, using exponential backoff starting at 1 second.

After receiving the prompt's output, parse the structured JSON response and validate that the recommended_threshold is a float between 0 and 1, that the precision_estimate and recall_estimate fields are present and within the same range, and that the tradeoff_rationale string is non-empty. If validation fails, log a THRESHOLD_OUTPUT_VALIDATION_ERROR and do not automatically apply the recommendation. For high-stakes retrieval pipelines (e.g., medical, legal, or financial RAG), route the recommendation to a human reviewer through a calibration review queue before any threshold change is deployed. For lower-risk pipelines, you can automate the threshold update if the recommendation appears in N consecutive calibration runs with consistent direction and magnitude. Always log the full prompt input, output, and validation result to your trace observability platform for auditability and future drift analysis.