Context Refresh Decision Prompt for RAG Pipelines

This prompt is designed for a single, high-stakes decision point inside a production RAG pipeline: should the system refresh its retrieved context before generating an answer? It is not a general staleness detector, a re-query generator, or a user-facing explanation tool. Its sole job is to weigh staleness signals, query criticality, refresh cost, and latency budget to output a binary REFRESH or NO_REFRESH decision with a structured rationale and a priority score. The ideal user is an AI engineer or platform operator who manages a refresh budget—typically a limited number of expensive API calls, database reads, or slow vector searches per session—and needs a cost-aware gate that prevents unnecessary refreshes while catching critical staleness before it produces a confident wrong answer.

Use this prompt when you have a RAG system that caches retrieved context across turns and you need a programmatic decision point before incurring the cost of a live re-retrieval. The prompt expects structured inputs: the original user query, a staleness assessment from an upstream detector, the age and source type of the current context, the estimated refresh cost in milliseconds or tokens, and the session's remaining latency budget. It is appropriate for multi-turn copilots, support agents referencing knowledge bases, and any assistant where context freshness directly impacts answer accuracy. Do not use this prompt when you have no refresh budget constraints (just re-retrieve every time), when the staleness signal is already a hard expiration (bypass the decision and refresh), or when the user has explicitly requested a live re-check (honor the user intent directly).

The prompt's value is in preventing two costly failure modes: refresh waste (re-retrieving context that is still valid, burning budget and adding latency for no accuracy gain) and stale-through (letting outdated context produce a confident but incorrect answer that erodes user trust). Before wiring this into your harness, ensure you have a calibrated staleness detection signal upstream. A weak or noisy staleness signal will cause this decision prompt to either over-refresh or under-refresh, regardless of how well the decision logic is tuned. The next section provides the copy-ready template with all required placeholders. After adapting it, proceed to the implementation harness section to add validation, logging, and cost-tracking around every refresh decision.

This prompt template is the core decision engine for a context refresh gate in a production RAG pipeline. It receives structured staleness signals, the original query, retrieval metadata, and cost parameters, then outputs a binary refresh decision with reasoning and a priority score. The template is designed to be called before every assistant response that depends on previously retrieved context, acting as a cost-aware circuit breaker that prevents both unnecessary API calls and stale answers.

textCopy
You are a context refresh decision engine for a RAG pipeline. Your job is to decide whether previously retrieved context must be refreshed before generating an assistant response. You balance staleness risk against refresh cost and latency budget.

## INPUTS

[QUERY]
The user's current question or request.

[RETRIEVED_CONTEXT]
The previously retrieved context that may be stale. Include source metadata: retrieval timestamp, source type, and any version or publication date.

[STALENESS_SIGNALS]
Structured signals indicating possible staleness. Include:
- Time since retrieval (seconds)
- Domain half-life category (realtime, hourly, daily, weekly, stable)
- User correction flags (if user contradicted prior context)
- Source volatility score (0.0-1.0, where 1.0 means highly volatile)
- External event triggers (e.g., breaking news, system outage, policy change)

[QUERY_CRITICALITY]
- Criticality level: low | medium | high | critical
- Impact of wrong answer: minor_inconvenience | user_mistrust | business_loss | safety_risk
- User-facing: true | false

[REFRESH_COST]
- Estimated refresh latency (ms)
- Estimated token cost for re-retrieval and re-processing
- Remaining context window budget (tokens)
- Remaining latency budget for this turn (ms)

[OUTPUT_SCHEMA]
{
  "refresh_decision": boolean,
  "priority_score": number (0.0-1.0, where 1.0 means must refresh),
  "reasoning": string (concise explanation of the decision),
  "staleness_risk": "low" | "medium" | "high" | "critical",
  "cost_impact": {
    "estimated_added_latency_ms": number,
    "estimated_added_tokens": number,
    "budget_exceeded": boolean
  },
  "stale_items": [
    {
      "context_snippet": string,
      "staleness_reason": string,
      "suggested_action": "refresh" | "drop" | "flag_to_user"
    }
  ],
  "user_facing_caveat": string | null
}

[CONSTRAINTS]
- If query_criticality is "critical" and staleness_risk is "high" or "critical", always refresh regardless of cost.
- If refresh would exceed the latency budget by more than 20%, do not refresh unless criticality is "critical".
- If the context window budget has less than 15% remaining, prefer dropping stale items over refreshing.
- If user correction flags indicate direct contradiction of retrieved context, treat staleness_risk as at least "high".
- For domain_half_life "realtime" and retrieval_age > 60 seconds, staleness_risk is at least "medium".
- If refresh_decision is false but staleness_risk is "high" or "critical", user_facing_caveat must be populated with a concise warning.
- Reasoning must cite specific signals, not generic statements.
- Priority_score must reflect both staleness severity and query criticality, not staleness alone.

[EXAMPLES]
Example 1: Low-risk skip
Query: "What's the capital of France?"
Retrieved context age: 7 days, domain: stable, source volatility: 0.0
Criticality: low, impact: minor_inconvenience
Decision: false, priority: 0.05, reasoning: "Stable factual knowledge with no staleness signals and minimal impact of error."

Example 2: Critical refresh
Query: "Is the patient's latest lab result within normal range?"
Retrieved context age: 4 hours, domain: hourly, source volatility: 0.6
Criticality: critical, impact: safety_risk
User correction: patient mentioned "new test results came in this morning"
Decision: true, priority: 0.95, reasoning: "Safety-critical query with user signal of newer data and moderate source volatility."

Example 3: Budget-constrained skip
Query: "Summarize the project status."
Retrieved context age: 30 minutes, domain: hourly, source volatility: 0.3
Criticality: medium, impact: user_mistrust
Latency budget remaining: 200ms, refresh cost: 350ms
Decision: false, priority: 0.4, reasoning: "Moderate staleness risk but refresh exceeds latency budget by 75%. Flagging to user instead."
user_facing_caveat: "Some project data may be up to 30 minutes old."

Adapt this template by adjusting the [CONSTRAINTS] thresholds to match your application's risk tolerance and cost profile. The domain half-life categories should map to your actual knowledge sources—a stock ticker system needs different categories than a legal document store. The output schema includes a user_facing_caveat field because refusing to refresh does not mean hiding staleness from the user; transparency preserves trust even when the system makes a cost-driven decision to proceed with potentially stale data. Before deploying, run this prompt against a labeled evaluation set where ground-truth refresh decisions are known, measuring both unnecessary refresh rate and missed staleness rate.

The Context Refresh Decision Prompt acts as a cost-aware gate before expensive re-retrieval or re-ranking operations. It should be invoked after the initial retrieval but before any refresh action, receiving the current query, the age and source of the retrieved context, and the estimated cost of a refresh. The harness must treat this prompt as a binary classifier with a priority score, not a conversational agent. Model choice matters: a fast, cheaper model (e.g., GPT-4o-mini, Claude Haiku) is usually sufficient for this classification task, reserving more expensive models for the actual answer generation step. The prompt should be called synchronously in the RAG pipeline's critical path, so latency must be budgeted—target sub-500ms response times by keeping the input context concise and the output schema minimal.

The implementation must include strict output validation and a retry budget. Parse the model's JSON response and validate that refresh_decision is exactly true or false, priority_score is a float between 0.0 and 1.0, and reasoning is a non-empty string. If validation fails, retry once with the same prompt and an appended error message. If the second attempt also fails, default to refresh_decision: true to avoid serving stale context silently—this fail-open posture prioritizes correctness over cost savings. Log every decision, including the input staleness signals, the model's reasoning, and the final action taken, to a structured logging system (e.g., OpenTelemetry spans or application logs). This trace data is essential for measuring cost-aware accuracy: you need to calculate the percentage of refreshes that were necessary (precision) and the percentage of stale contexts that were caught (recall).

For high-stakes domains like healthcare, legal, or finance, insert a human review step when the model's priority_score falls in an ambiguous middle range (e.g., 0.4–0.6) or when the reasoning indicates conflicting signals. The harness should route these cases to a review queue rather than making an automatic decision. Additionally, implement a kill switch: if the refresh API or vector database is experiencing latency spikes, the harness should temporarily bypass the decision prompt and either serve the cached context with a staleness caveat or escalate to the user, depending on the product's risk tolerance. Do not let a refresh decision prompt become a hidden single point of failure in your retrieval pipeline.

Before deploying, build an evaluation harness that replays production queries with known-stale and known-fresh context pairs. Measure the decision prompt's precision and recall against ground truth, and track the cost impact of false positives (unnecessary refreshes) and false negatives (missed staleness). Set acceptance thresholds for both metrics before promoting the prompt to production. Monitor these metrics continuously after deployment, as changes to the underlying knowledge base freshness patterns or user query distributions can silently degrade the prompt's effectiveness over time.