Stale Context Confidence Adjustment Prompt

This prompt is for AI systems that attach a confidence score to their outputs and need to calibrate that score when the underlying context is suspected to be stale. The core job is to take an original output with its initial confidence, a set of staleness signals, and produce an adjusted confidence value that honestly reflects the risk of acting on outdated information. The ideal user is an AI engineer or product developer building a RAG-augmented assistant, a real-time copilot, or any system where context freshness directly impacts output reliability. You need a structured staleness assessment as input—this prompt does not detect staleness itself; it adjusts confidence based on already-identified staleness factors.

Use this prompt when your system already has a staleness detection mechanism in place and you need to translate those signals into a calibrated confidence adjustment. The prompt requires the original output, the original confidence score, a list of staleness factors with severity ratings, and the domain's tolerance for stale information. It is not a replacement for staleness detection, nor is it appropriate for systems that do not expose confidence scores to users or downstream logic. Avoid using this prompt when the cost of a wrong answer is catastrophic and no confidence adjustment can safely mitigate the risk—those cases require blocking the output entirely and triggering a full context refresh or human review.

Before wiring this into production, define your staleness factor taxonomy and severity scale. The prompt works best when staleness factors are concrete and machine-readable—for example, 'retrieved_document_age_exceeds_threshold' or 'user_correction_contradicts_prior_fact'—rather than vague natural-language descriptions. Pair this prompt with an evaluation harness that measures whether adjusted confidence correlates with actual output correctness on a labeled dataset of stale and fresh context examples. The most common failure mode is over-adjustment, where the system becomes too hesitant and erodes user trust by flagging every output as uncertain. Calibrate the adjustment logic against your specific domain's tolerance for false positives versus false negatives.

The following prompt template is designed to be dropped into a production pipeline where an assistant has already generated a draft output and attached an initial confidence score. Its job is to re-evaluate that confidence in light of potential context staleness—outdated retrieval results, session history that may have been invalidated, or assumptions that no longer hold. The template uses square-bracket placeholders so you can wire in your own context, output schema, and risk parameters without modifying the core logic.

textCopy
You are a confidence calibration module in a production AI system. Your task is to adjust the confidence score of an assistant output when the underlying context may be stale.

## INPUTS
- [ORIGINAL_OUTPUT]: The assistant's draft response.
- [INITIAL_CONFIDENCE]: The original confidence score (0.0 to 1.0).
- [CONTEXT_SNAPSHOT]: The context used to generate the output, including retrieval results, session history, and any assumptions.
- [STALENESS_SIGNALS]: Structured signals indicating potential staleness, such as document age, user corrections, topic shifts, or API response age.
- [CURRENT_TIME]: The current timestamp or turn index for recency comparison.

## CONSTRAINTS
- Do not change the assistant's output text. Only adjust the confidence score and add a staleness caveat.
- If no staleness signals are present, return the original confidence unchanged.
- If staleness is severe, confidence must drop below [LOW_CONFIDENCE_THRESHOLD] and the caveat must recommend verification.
- Never invent staleness signals. Only use the provided [STALENESS_SIGNALS].

## OUTPUT SCHEMA
Return valid JSON matching this schema exactly:
{
  "adjusted_confidence": number,
  "staleness_factors": [
    {
      "factor": string,
      "impact": "high" | "medium" | "low",
      "reasoning": string
    }
  ],
  "user_facing_caveat": string | null,
  "recommendation": "use_as_is" | "use_with_caveat" | "suppress_output"
}

## STALENESS ADJUSTMENT RULES
1. For each staleness signal, determine whether it affects a claim in [ORIGINAL_OUTPUT].
2. If a signal contradicts a claim, flag it as high impact and reduce confidence by at least 0.3.
3. If a signal suggests a claim may be outdated but does not directly contradict it, flag it as medium impact and reduce confidence by 0.1 to 0.2.
4. If a signal is present but unlikely to affect the output, flag it as low impact and do not reduce confidence.
5. The adjusted confidence must never exceed the initial confidence.
6. If multiple high-impact factors apply, confidence must drop below [LOW_CONFIDENCE_THRESHOLD].

## USER-FACING CAVEAT
- If adjusted confidence is below [LOW_CONFIDENCE_THRESHOLD], generate a concise, non-alarming caveat that tells the user which information may be outdated and what they should verify independently.
- If adjusted confidence is above the threshold but reduced, generate a brief note that the information may need confirmation.
- If confidence is unchanged, return null for the caveat.

## EXAMPLES
[EXAMPLES]

## RISK LEVEL
[RISK_LEVEL]

To adapt this template, start by defining your staleness signals. These should come from upstream detectors—document age checks, user correction extraction, API response TTL monitors, or session summary contradiction detectors. Wire them into the [STALENESS_SIGNALS] placeholder as structured objects, not raw text. Set [LOW_CONFIDENCE_THRESHOLD] based on your product's risk tolerance: 0.6 is a reasonable starting point for most applications, but raise it to 0.8 for regulated domains where overconfidence is more dangerous than underconfidence. The [EXAMPLES] placeholder should contain at least three few-shot examples: one where staleness is absent, one where it is moderate, and one where it is severe enough to trigger suppression. If your domain requires human review before suppressing an output, set [RISK_LEVEL] to high and add a post-processing rule that routes suppress_output recommendations to a review queue rather than blocking the response automatically. Test the prompt against a golden dataset of known-stale and known-fresh context pairs before deployment, and monitor the correlation between adjusted confidence and actual output correctness in production traces.

This prompt is designed to sit after your primary answer-generation step and before the response is shown to the user. In a typical RAG or copilot pipeline, you generate an answer with an initial confidence score, then pass the answer, the context used, and any staleness signals into this prompt to produce a calibrated confidence value and a user-facing uncertainty statement. The prompt does not decide whether to refresh context—it only adjusts confidence and generates caveats based on staleness signals you've already detected. This separation keeps the confidence adjustment logic testable and auditable without coupling it to the retrieval or generation steps.

Wiring the prompt into your application: Call this prompt as a secondary model invocation after your primary answer generation. The input payload should include [ORIGINAL_ANSWER], [ORIGINAL_CONFIDENCE] (a float 0.0–1.0), [CONTEXT_USED] (the retrieved passages or session state the answer relied on), and [STALENESS_SIGNALS] (structured flags from your staleness detection system—such as document age, user correction events, or topic shift indicators). The prompt returns a JSON object with adjusted_confidence, staleness_factors, and user_facing_caveat. Parse this output and attach the adjusted_confidence to your internal logging while appending the user_facing_caveat to the response shown to the user. Validation: Before surfacing the output, validate that adjusted_confidence is a float between 0.0 and 1.0, that adjusted_confidence <= original_confidence (staleness should never increase confidence), and that user_facing_caveat is a non-empty string under 200 characters. If validation fails, fall back to the original confidence and a generic caveat like "Some context may be outdated." Log the validation failure for debugging.

Retry and model selection: This is a low-latency, structured-output task. Use a fast model (GPT-4o-mini, Claude Haiku, or equivalent) with response_format set to json_object or a strict JSON mode. Set temperature=0 to minimize variance in confidence scores. If the output fails JSON parsing or validation, retry once with the same input. If the retry also fails, log the failure, use the original confidence unchanged, and surface a generic caveat. Do not retry more than once—the user is waiting. Observability: Log every invocation with original_confidence, adjusted_confidence, the staleness signals provided, and whether validation passed. This log becomes your dataset for evaluating whether adjusted confidence correlates with actual output correctness. Periodically sample logged outputs and compare adjusted_confidence against human-annotated correctness labels to measure calibration quality.

What to avoid: Do not use this prompt as your only staleness safeguard. It adjusts confidence downward but does not prevent stale answers from reaching users. Pair it with a context refresh trigger that re-queries sources when staleness signals exceed a threshold. Do not show raw staleness_factors to users—the user_facing_caveat field exists specifically to provide a concise, non-alarming explanation. Finally, never allow adjusted_confidence to exceed original_confidence. If the model outputs a higher adjusted value, treat it as a validation failure and fall back to the original score.