API Documentation Evidence Ranking Prompt

This prompt is built for developer-platform and DX engineering teams who need to rank API documentation passages by their actual utility for a developer trying to solve a specific task. The job-to-be-done is not generic relevance ranking—it is evidence selection that accounts for endpoint specificity, version accuracy, example completeness, and deprecation status. Use this prompt when your retrieval system returns multiple candidate passages from API docs, SDK references, changelogs, or migration guides, and you need a scored, ordered set of references before generating an answer or showing results to a user. The ideal user is an AI engineer integrating this into a RAG pipeline for a developer-facing copilot, documentation search, or support bot.

Do not use this prompt when the input is a single, obviously correct passage, or when the query is a general conceptual question that doesn't require endpoint-level precision. This prompt is also inappropriate for ranking non-technical content, marketing pages, or community forum threads where authority and versioning rules differ. The prompt assumes your retrieval step has already produced candidate passages with metadata including source URL, last-updated timestamp, and deprecation status. If you lack this metadata, the prompt's ranking logic will degrade because it relies on version freshness and deprecation signals as primary ranking factors. You must also provide the user's SDK language or framework context, because an example in the wrong language is worse than no example at all.

Before wiring this into production, define your failure boundaries. The prompt will rank passages, but it cannot verify that the code examples actually compile or that the documented behavior matches the live API. Your harness must include a post-ranking validation step: execute or lint code examples where feasible, check for broken links, and flag passages where the authentication method described doesn't match the user's configured auth context. The prompt outputs a structured ranking with reasoning, which makes it auditable, but the reasoning itself can contain errors—plan for spot-checking ranked outputs against human developer judgment during eval. If your use case involves safety-critical API calls (billing, provisioning, data deletion), add a human-review gate before any ranked documentation is used to generate executable code or configuration.

This prompt template is designed to be copied directly into your application or evaluation harness. It accepts a set of retrieved API documentation passages, a developer query, and optional context about the user's environment, then produces a ranked evidence set with explicit scores and reasoning. Every placeholder is enclosed in square brackets so you can substitute values programmatically before sending the request to the model.

textCopy
You are an API documentation evidence ranker. Your job is to score and rank retrieved documentation passages by how well they answer a developer's query, using strict criteria for endpoint specificity, version accuracy, example completeness, and deprecation status.

## INPUT
Developer query: [QUERY]
User's SDK version: [SDK_VERSION]
User's authentication method: [AUTH_METHOD]
Current date: [CURRENT_DATE]

Retrieved passages:
[RETRIEVED_PASSAGES]

## RANKING CRITERIA
For each passage, assign a score from 0 to 10 on each of the following dimensions, then compute a weighted total:

1. **Endpoint specificity (weight 0.30):** Does the passage reference a specific endpoint, method, or resource path that matches the query? Generic overviews score lower than endpoint-specific documentation.
2. **Version accuracy (weight 0.25):** Does the passage match the user's SDK version or explicitly state which versions it applies to? Passages for deprecated versions or incompatible major versions score lower. If version is unstated, score 5.
3. **Example completeness (weight 0.25):** Does the passage include a complete, runnable code example with proper authentication, request shape, and response handling? Partial examples score lower. Missing authentication context in the example reduces the score by at least 3 points.
4. **Deprecation status (weight 0.20):** Is the endpoint or feature described as deprecated, sunset, or replaced? Deprecated content scores 0 on this dimension. Active endpoints score 10. Passages with deprecation warnings but no migration path score 3.

## OUTPUT FORMAT
Return a JSON object with this exact schema:
{
  "ranked_passages": [
    {
      "passage_id": "string",
      "weighted_score": number,
      "dimension_scores": {
        "endpoint_specificity": number,
        "version_accuracy": number,
        "example_completeness": number,
        "deprecation_status": number
      },
      "reasoning": "string explaining the scores in 1-2 sentences",
      "flags": ["string"]
    }
  ],
  "query_coverage_assessment": "string describing whether the retrieved set adequately covers the query",
  "missing_information": ["string"]
}

## FLAGS
Use these flags where applicable: "deprecated_endpoint", "version_mismatch", "missing_auth_context", "broken_example", "generic_overview", "no_code_example", "incomplete_response_handling", "superseded_by_newer_endpoint"

## CONSTRAINTS
- Do not invent endpoints, parameters, or examples not present in the retrieved passages.
- If no passage scores above 6.0 weighted, set query_coverage_assessment to "insufficient" and list specific gaps in missing_information.
- If a passage contains a code example that would fail due to missing authentication, flag it as "missing_auth_context" and score example_completeness no higher than 4.
- Rank passages in descending order of weighted_score.
- If two passages describe the same endpoint but different versions, prefer the one matching [SDK_VERSION].

To adapt this template, replace each square-bracket placeholder with values from your retrieval pipeline and user context. The [RETRIEVED_PASSAGES] placeholder should contain pre-formatted passages with unique IDs, source URLs, and full text. If your retrieval system returns passages without version metadata, adjust the version accuracy scoring logic or set a default score. For production use, add a [RISK_LEVEL] parameter that triggers human review when the top-ranked passage scores below a threshold or carries deprecation flags. Always validate the output JSON against the schema before surfacing results to users, and log any passages that receive conflicting scores across dimensions for later eval review.

The API Documentation Evidence Ranking Prompt is designed to sit between your retrieval pipeline and the final answer generation or developer-facing display layer. It accepts a set of retrieved API doc passages and a developer query, then returns a scored, ranked, and annotated evidence set. The harness must enforce strict input validation before the prompt is assembled: each passage must include a source identifier (URL, file path, or doc section anchor), a retrieval timestamp, and the raw text. If your retriever cannot provide these fields, add a pre-processing step that attaches them before calling the ranking prompt. Missing source identifiers are the most common cause of downstream citation failures.

The output contract is a JSON array of ranked passages, each with a score (0.0 to 1.0), a rank integer, a rationale string explaining the score, and a set of boolean flags: endpoint_specific, version_accurate, example_complete, deprecation_aware, and auth_context_present. Your harness must validate this schema on every response. If the model returns malformed JSON, missing required fields, or scores outside the expected range, implement a retry loop with a maximum of three attempts. On the second and third retries, append the validation error message to the prompt as a [PREVIOUS_ERROR] context block. After three failures, log the raw response, flag the passage set for human review, and fall back to a simpler recency-based ranking heuristic.

Model choice matters for this workflow. Use a model with strong structured output capabilities and a context window large enough to hold your maximum passage set plus the prompt instructions. For most API doc corpora, 8K to 32K tokens is sufficient. If you are ranking more than 20 passages, consider batching them into groups of 10-15 and running a merge-rank step afterward. Enable structured output mode (JSON mode or function calling with a strict schema) rather than relying on free-text parsing. This reduces repair-loop frequency by an order of magnitude in production.

The most dangerous failure mode is the model assigning high scores to passages that contain broken code examples or missing authentication context. Your eval harness must include a set of seeded negative examples: passages with syntax errors in code samples, passages that omit required auth headers, passages from deprecated API versions, and passages that describe endpoints removed three versions ago. Run these through the prompt weekly and assert that no negative example scores above 0.3. If any does, your prompt needs hardening—add stronger negative constraints or few-shot examples that demonstrate low-scoring these patterns.

Logging and observability are critical because evidence ranking directly shapes the answers your developer users see. Log every ranking call with: the query, the number of input passages, the top three ranked passage IDs and scores, the model version, the latency, and whether validation passed on the first attempt. Set an alert if the first-attempt validation failure rate exceeds 10% over a rolling hour. For high-stakes API documentation surfaces—such as billing endpoints, authentication flows, or data deletion operations—route ranked outputs through a human review queue before they influence generated answers or code suggestions. The prompt is a ranking tool, not a correctness guarantee.