Technical Documentation Authority Assessment Prompt

This prompt is designed for internal platform and DevOps teams that need to automatically assess the trustworthiness of retrieved technical documentation. It scores each document by source type (official, community, vendor), version specificity, and maintenance status, producing a trust-weighted reference list. Use this prompt when your RAG pipeline or internal knowledge base pulls from heterogeneous sources such as official docs, community wikis, vendor knowledge bases, and GitHub READMEs, and you need to prioritize which passages to use for answering technical questions.

This prompt belongs in the evidence ranking stage of a RAG pipeline, after retrieval but before answer generation. It assumes you have already retrieved a set of candidate documents and need to rank them by authority before synthesis. The ideal user is an AI engineer or platform developer who controls the retrieval and ranking logic and needs a repeatable, auditable way to separate authoritative references from outdated or unverified community content. The prompt works best when each retrieved document includes metadata about its source, version, and last-updated date. Without that metadata, the model cannot apply the scoring criteria reliably, and you should instead use a simpler relevance-only ranking prompt.

Do not use this prompt for real-time user-facing chat where latency is critical and the retrieval set is small. It is also inappropriate for domains where authority is not the primary ranking signal, such as creative writing or general knowledge Q&A. If your pipeline already enforces source filtering at ingestion, this prompt may add unnecessary latency. Before deploying, validate that your retriever passes source metadata alongside document text, and test the prompt against a golden set of known authoritative and non-authoritative documents to confirm it ranks them correctly. The next section provides the copy-ready prompt template you can adapt for your own pipeline.

This prompt template is designed to be pasted directly into your system prompt or user message template. It instructs the model to act as a technical documentation authority assessor, scoring each retrieved document against criteria that matter in production engineering environments: official source precedence, version match with the user's environment, and maintenance freshness. The output is a trust-weighted reference list that your application can use to prioritize which documents to synthesize into a final answer or surface to the user.

textCopy
You are a technical documentation authority assessor for an internal platform team. Your job is to score each retrieved document by source type, version specificity, and maintenance status. Produce a trust-weighted reference list.

## INPUT
- User Environment: [USER_ENVIRONMENT]
- Retrieved Documents: [RETRIEVED_DOCUMENTS]

## SCORING CRITERIA
For each document, assign a score from 0 to 10 based on:
1. **Source Type (weight 40%)**: Official docs (10) > Vendor docs (8) > Maintained community docs (6) > Unmaintained community docs (3) > Personal blog/forum (1).
2. **Version Specificity (weight 35%)**: Exact version match to [USER_ENVIRONMENT] (10) > Minor version match (7) > Major version match (4) > No version info (1) > Conflicting version (0).
3. **Maintenance Status (weight 25%)**: Actively maintained with recent commits (10) > Maintained but stale >6 months (5) > Deprecated or archived (0).

## OUTPUT FORMAT
Return a JSON object with this exact schema:
{
  "documents": [
    {
      "id": "string",
      "title": "string",
      "source_type": "official" | "vendor" | "maintained_community" | "unmaintained_community" | "personal",
      "version_match": "exact" | "minor" | "major" | "none" | "conflict",
      "maintenance_status": "active" | "stale" | "deprecated",
      "trust_score": number,
      "deprecation_warning": boolean,
      "version_mismatch_detail": "string | null",
      "reasoning": "string"
    }
  ],
  "ranked_ids": ["string"],
  "warnings": ["string"]
}

## CONSTRAINTS
- Flag any document with a deprecation warning as `deprecation_warning: true`.
- If a document's version conflicts with [USER_ENVIRONMENT], set `version_mismatch_detail` to explain the conflict.
- Include a `warnings` array for any global issues like "no documents matched the user's major version" or "all community sources lack maintenance data."
- Do not invent version numbers or maintenance status. If the information is absent, mark it as such and score accordingly.

To adapt this template, replace [USER_ENVIRONMENT] with a structured description of the user's stack—such as python:3.11, kubernetes:1.29, postgres:16—and populate [RETRIEVED_DOCUMENTS] with the raw text or structured metadata from your retrieval pipeline. If your retrieval system already provides metadata fields like source_type or last_updated, pass them alongside the document content to reduce model inference burden. For high-stakes production use, add a validation step after the model output that cross-checks deprecation warnings against a known registry of end-of-life dates and version compatibility matrices. This prompt is a scoring layer, not a final answer generator; the ranked list should feed into a downstream synthesis or display step that respects the trust scores.

This prompt is designed to be called after document retrieval but before evidence synthesis. In a typical RAG pipeline, a user query triggers a search across your documentation corpus (e.g., internal wikis, vendor docs, community forums, GitHub repositories). The retrieved passages are then passed to this prompt for authority scoring. The output is a trust-weighted reference list that downstream answer-generation prompts can use to prioritize sources, cite confidently, or refuse to answer when only low-authority sources are available. The harness must supply the user's current environment context—such as software versions, operating system, and deployment type—so the prompt can detect version mismatches and deprecation risks.

Input Assembly: The harness must populate the [RETRIEVED_DOCUMENTS] placeholder with a structured list of passages, each containing a unique doc_id, source_type (e.g., official_docs, vendor_kb, community_forum, github_issue, internal_runbook), title, url, last_updated timestamp, version_info (if available), and the raw content snippet. The [USER_ENVIRONMENT] placeholder should be a machine-readable object with fields like product_name, product_version, os, deployment_type, and dependency_versions. The [OUTPUT_SCHEMA] placeholder should be replaced with a strict JSON schema definition that the model must follow, including fields for doc_id, authority_score (0.0–1.0), trust_tier (e.g., preferred, acceptable, use_with_caution, deprecated, version_mismatch), rationale, and warnings (list of strings for deprecation notices, version conflicts, or maintenance concerns).

Validation and Retry Logic: After the model returns its scored list, the harness must run a validation layer before the results are used downstream. First, validate that every doc_id in the output matches an input document—hallucinated IDs are a common failure mode. Second, check that any document flagged with version_mismatch or deprecated in the warnings array has a correspondingly low authority_score and a trust_tier no higher than use_with_caution. Third, verify that documents with source_type: official_docs and recent last_updated timestamps are not incorrectly downgraded unless a specific version conflict is detected. If validation fails, retry the prompt once with the validation errors appended to the [CONSTRAINTS] field as explicit correction instructions. If the retry also fails, log the failure, flag the retrieval set for human review, and fall back to a conservative ranking that treats only official_docs with matching versions as preferred.

Model Choice and Tool Use: This prompt works well with capable instruction-following models (e.g., Claude 3.5 Sonnet, GPT-4o, Gemini 1.5 Pro). Avoid smaller or older models that struggle with structured output and multi-attribute scoring. If your platform supports tool use or structured output APIs (e.g., OpenAI's response_format with a JSON schema, or Anthropic's tool use with a defined input_schema), prefer those over relying solely on the prompt's output format instructions—this dramatically reduces parsing errors. For high-throughput pipelines, consider caching authority scores for documents that haven't changed since the last scoring run, keyed by doc_id and last_updated. Do not cache scores across different [USER_ENVIRONMENT] inputs, as version-specific assessments will differ.

Logging and Observability: Log every invocation with the user_environment hash, the count of documents scored, the distribution of trust_tier values, and any validation failures. This data is critical for detecting drift: if the proportion of preferred sources drops over time, your documentation corpus may be accumulating stale content. Set up an alert if more than a configurable threshold (e.g., 30%) of retrieved documents receive a version_mismatch or deprecated warning, as this indicates a systemic freshness problem that retrieval alone won't fix.

Human Review Gates: For most internal documentation use cases, fully automated authority scoring is acceptable. However, if the scored documents will inform customer-facing answers, safety-critical procedures, or compliance-related decisions, insert a human review step when the highest-scoring document has an authority_score below 0.7 or when conflicting version information is detected across official_docs sources. The review interface should display the original documents, the model's scores and rationale, and the user's environment context, allowing a human to override scores or escalate to the documentation owner.