Stale Question Re-Evaluation Prompt

Long-running chat sessions accumulate unanswered questions. Some become irrelevant as the conversation evolves—the user finds the answer themselves, the context shifts, or the assistant's subsequent response makes the original question moot. This prompt evaluates a single pending question against the conversation that followed it, producing a structured decision: still relevant, implicitly answered, or ready for retirement. It is designed for state management middleware that needs to clean up stale open items before re-surfacing them to the user or passing them to a human agent. The ideal user is an AI engineer or backend developer building a pending-question tracker that holds items older than a configurable number of turns and requires an automated decision before re-queuing or discarding them.

Use this prompt when your system maintains a queue of unanswered questions and you need to filter out items that no longer require attention. The prompt requires three inputs: the original question text, the conversation turns that occurred after the question was asked, and a staleness threshold expressed in number of turns. It outputs a classification label, a confidence score, and a brief rationale that can be logged for audit purposes. The confidence score enables automated retirement decisions above a configurable threshold while routing low-confidence cases for human review. This is not a prompt for detecting new questions or for generating answers—it assumes the question has already been identified and stored by upstream detection logic.

Do not use this prompt when the conversation is shorter than the staleness threshold, when the question was asked in the most recent turn, or when you need to answer the question rather than classify its relevance. It is also inappropriate for real-time interruption scenarios where latency constraints prevent a full re-evaluation pass. For high-stakes domains such as healthcare or legal support, always route retirement decisions with confidence scores below 0.9 to human review rather than automating disposal. The prompt works best as part of a scheduled cleanup job that runs periodically on the pending-question store, not as a synchronous step in the user-facing response path.

This prompt template is designed to be dropped into a state management middleware layer. It takes a single pending question and the conversation turns that occurred after it was asked, then determines whether the question is still relevant, has been implicitly answered, or should be retired. The output is a structured JSON decision with a confidence score, enabling automated retirement of stale questions without losing track of unresolved ones.

textCopy
You are a conversation state auditor. Your job is to evaluate whether a previously unanswered question is still relevant given the conversation that followed.

## INPUTS

**Pending Question:**
[PENDING_QUESTION]

**Conversation Since Question Was Asked:**
[SUBSEQUENT_CONVERSATION]

**Current Conversation Phase:**
[CURRENT_PHASE]

## EVALUATION CRITERIA

1. **Implicitly Answered**: Did the assistant or another participant provide information that addresses the question, even if not framed as a direct answer?
2. **Context Shift**: Has the conversation moved to a completely different topic, making the question irrelevant to the current context?
3. **User Abandonment**: Did the user explicitly drop the topic, change the subject, or show disinterest in the answer?
4. **Temporal Staleness**: Is the question time-sensitive in a way that makes it obsolete now?
5. **Dependency Invalidation**: Has new information made the question's premise invalid or the answer no longer useful?

## OUTPUT SCHEMA

Return a JSON object with the following fields:

```json
{
  "question_id": "string - identifier for the pending question",
  "relevance_decision": "still_relevant | implicitly_answered | stale_context_shift | user_abandoned | temporally_stale | premise_invalidated",
  "confidence": 0.0-1.0,
  "rationale": "string - brief explanation citing specific evidence from the conversation",
  "evidence_turn_references": ["array of turn indices or timestamps supporting the decision"],
  "recommended_action": "re_surface | retire | hold_for_re_evaluation | escalate_to_human",
  "retirement_safety_check": {
    "would_user_be_harmed_by_retirement": true/false,
    "explanation": "string - if true, explain the potential harm"
  }
}

CONSTRAINTS

• Do not retire a question if the user would be harmed by losing the answer.
• If confidence is below 0.7, recommend hold_for_re_evaluation or escalate_to_human.
• Cite specific turn references when providing evidence.
• If the question relates to safety, compliance, or a blocking dependency, default to still_relevant unless there is explicit evidence of resolution.
• Do not infer abandonment from silence alone; look for explicit topic shifts or user statements.

EXAMPLES

Example 1: Implicitly Answered Pending Question: "What's the SLA for premium support?" Subsequent Conversation: "...and as I mentioned earlier, premium support guarantees a 4-hour response time for all P1 issues..." Decision: implicitly_answered, confidence 0.95

Example 2: Context Shift Pending Question: "Can you explain the pricing model?" Subsequent Conversation: User says "Actually, let's switch gears. I need help with the API integration instead." followed by 10 turns about API endpoints. Decision: stale_context_shift, confidence 0.85

Example 3: Still Relevant Pending Question: "Is the data migration going to affect our production environment?" Subsequent Conversation: 5 turns discussing migration timeline but never addressing production impact. Decision: still_relevant, confidence 0.90

Adaptation guidance: Replace [PENDING_QUESTION] with the exact text of the unanswered question from your tracking system. [SUBSEQUENT_CONVERSATION] should contain all turns after the question was asked, formatted with speaker labels and turn indices for traceability. [CURRENT_PHASE] is optional but helps contextualize whether the conversation has moved on. The output schema is designed for programmatic consumption—wire the recommended_action field directly into your state management logic. For high-stakes domains like healthcare or finance, set the confidence threshold higher (0.85+) and always require human review for retirement decisions flagged by retirement_safety_check.

Next steps: After copying this template, pair it with a pending question detection prompt to build a complete tracking loop. Test against conversations with known dropped questions to calibrate your confidence thresholds. Log every retirement decision with its rationale and evidence references for auditability. If you observe false retirements, add domain-specific examples to the few-shot section and tighten the constraints around your specific failure modes.

The Stale Question Re-Evaluation Prompt is designed to run as a background evaluation step, not as part of the primary user-facing response generation. In a production chat system, this prompt should be triggered by a state management middleware layer whenever a pending question has aged beyond a configurable threshold (e.g., 5 turns or 3 minutes of silence). The middleware retrieves the original question, its turn context, and all subsequent conversation turns, then assembles the prompt with these inputs. The output is a structured decision object that the application consumes to either keep the question active, retire it silently, or surface a re-evaluation to the user.

The implementation must include a confidence threshold gate on the retirement_confidence field. For high-confidence retirement decisions (e.g., confidence >= 0.9), the system can automatically remove the question from the pending queue and log the retirement reason. For medium-confidence decisions (0.7–0.9), route the question to a human review queue or a secondary LLM judge for confirmation before retirement. For low-confidence decisions (< 0.7), keep the question active and re-evaluate after additional turns. Never auto-retire questions where the model indicates the question was merely forgotten rather than resolved—those require a repair turn instead. Log every retirement decision with the original question, the conversation segment used for evaluation, the model's reasoning, and the confidence score for auditability.

Model choice matters here. This prompt requires strong instruction-following and structured output discipline. Use a model with reliable JSON mode (e.g., GPT-4o, Claude 3.5 Sonnet) and enforce the output schema with your API's structured output feature rather than relying on prompt-level format instructions alone. For cost optimization, consider running this evaluation on a smaller, fine-tuned model if you have sufficient labeled examples of stale vs. relevant questions. Retry logic should handle malformed JSON by re-prompting once with the raw output and a repair instruction. If the second attempt fails, default to keeping the question active and flag for human review. Tool integration is minimal here—this prompt is a pure reasoning step, but its output should feed into your conversation state manager, which may trigger other prompts like the Conversation Repair Prompt or the Pending Action Reminder Generation Prompt.

Testing this prompt requires a golden dataset of conversation segments with known stale and still-relevant questions. Build eval cases that include: questions implicitly answered through subsequent turns, questions made irrelevant by topic shifts, questions the user explicitly abandoned, and questions that remain genuinely open. Measure precision (did we retire only truly stale questions?) and recall (did we catch all stale questions?). The cost of false negatives (keeping stale questions) is context pollution and user annoyance; the cost of false positives (retiring relevant questions) is broken trust and dropped user needs. Weight your eval metrics accordingly—false positives should be penalized more heavily. Run regression tests against this dataset whenever you update the prompt, the model version, or the retirement confidence threshold.