Multi-Turn Retention Scoring Prompt

This prompt is for chat platform teams that need explainable, auditable pruning decisions before context windows overflow. Instead of dropping turns by recency alone, it scores each turn on five dimensions: task relevance, information novelty, user sentiment, correction presence, and temporal decay. The output is scored turn metadata that downstream pruning logic can consume deterministically. Use this when your pruning strategy must be defensible to product reviewers, compliance auditors, or when you need inter-annotator agreement metrics to justify automated retention decisions.

The ideal user is an AI engineer or platform developer building a multi-turn chat system where context window economics directly control cost, latency, and reliability. You need a structured scoring layer that separates the judgment of turn value from the mechanical act of dropping turns. This prompt is not a pruning action itself—it produces a ranked, scored artifact that your application code can use to make deterministic keep/drop decisions with configurable thresholds. Required context includes the full conversation history, the current user task or goal, and any system-level retention policies. The prompt works best when you have already defined your context budget and need a principled way to allocate it, rather than relying on brittle heuristics like "keep last N turns."

Do not use this prompt when you need real-time, sub-50ms pruning decisions—the scoring pass adds latency and is designed for asynchronous or pre-request execution. Do not use it as a substitute for session summarization; scored turns still consume tokens, and this prompt helps you decide what to keep, not how to compress. Avoid this approach when your conversation history is shorter than your context window, as the scoring overhead provides no benefit. Finally, do not deploy this in production without establishing inter-annotator agreement baselines and defining clear score thresholds for your keep/discard/summarize actions. Without those thresholds, the scores are interesting metadata but not actionable pruning logic.

This prompt template implements a multi-dimensional retention scoring system for conversation turns. It evaluates each turn on task relevance, information novelty, user sentiment, correction presence, and temporal decay, producing structured metadata that downstream pruning logic can consume deterministically. The template is designed to be model-agnostic and requires you to supply your own conversation history, scoring weights, and output schema preferences.

textCopy
You are a conversation turn retention scorer. Your job is to evaluate each turn in a multi-turn conversation and assign retention scores across multiple dimensions. You must output structured, machine-readable metadata for each turn. Do not summarize the conversation or provide qualitative commentary.

## INPUT

Conversation history with turn IDs and timestamps:
[CONVERSATION_HISTORY]

Scoring configuration:
- Weights: [SCORING_WEIGHTS]
- Thresholds: [RETENTION_THRESHOLDS]
- Temporal decay factor: [DECAY_FACTOR]
- Current reference timestamp: [REFERENCE_TIMESTAMP]

## SCORING DIMENSIONS

For each turn, compute the following scores on a 0.0 to 1.0 scale:

1. **task_relevance**: How central this turn is to the user's primary task or goal. Turns that define goals, state constraints, or make decisions score higher than social niceties or off-topic digressions.

2. **information_novelty**: How much new information this turn introduces relative to prior turns. A turn that repeats previously stated facts scores low. A turn that introduces a new constraint, preference, or fact scores high.

3. **user_sentiment**: The emotional signal in the user's turn. Score 0.0 for neutral, 0.5 for mild positive or negative sentiment, and 1.0 for strong sentiment (frustration, urgency, satisfaction). Assistant turns without user sentiment should default to 0.0.

4. **correction_presence**: Whether this turn contains a user correction of a prior assistant statement. Score 1.0 if the user explicitly corrects or contradicts the assistant. Score 0.5 if the user clarifies or refines a prior understanding. Score 0.0 if no correction is present.

5. **temporal_decay**: A computed score based on the time difference between the turn timestamp and the reference timestamp, using the provided decay factor. More recent turns score higher. Formula: exp(-decay_factor * time_difference_in_hours).

6. **composite_retention_score**: Weighted sum of the five dimension scores using the provided weights. This is the primary signal for pruning decisions.

## OUTPUT SCHEMA

Return a JSON object with this exact structure:

{
  "turns": [
    {
      "turn_id": "string",
      "scores": {
        "task_relevance": 0.0,
        "information_novelty": 0.0,
        "user_sentiment": 0.0,
        "correction_presence": 0.0,
        "temporal_decay": 0.0,
        "composite_retention_score": 0.0
      },
      "retention_decision": "retain" | "summarize" | "discard",
      "decision_rationale": "Brief one-sentence explanation referencing the dominant scoring dimension."
    }
  ],
  "scoring_metadata": {
    "weights_used": {},
    "thresholds_applied": {},
    "decay_factor": 0.0,
    "reference_timestamp": "string",
    "total_turns_evaluated": 0
  }
}

## RETENTION DECISION RULES

- If composite_retention_score >= [RETAIN_THRESHOLD], mark as "retain".
- If composite_retention_score >= [SUMMARIZE_THRESHOLD] but below [RETAIN_THRESHOLD], mark as "summarize".
- If composite_retention_score < [SUMMARIZE_THRESHOLD], mark as "discard".
- Override rule: Any turn with correction_presence >= 0.5 must be marked "retain" regardless of composite score.

## CONSTRAINTS

- Do not modify the conversation content.
- Do not skip any turn. Every turn in the input must appear in the output.
- Use the exact output schema. No additional fields.
- If a turn is from the assistant and has no user sentiment, set user_sentiment to 0.0.
- Apply the temporal decay formula exactly as specified.

To adapt this template, replace [CONVERSATION_HISTORY] with your structured turn data—each turn needs a unique ID, a speaker role, the message text, and an ISO 8601 timestamp. Set [SCORING_WEIGHTS] as a JSON object mapping each dimension name to its weight (all weights should sum to 1.0). Define [RETENTION_THRESHOLDS] as an object with retain and summarize numeric thresholds. The [DECAY_FACTOR] controls how aggressively older turns are penalized—start with 0.01 for hourly decay and tune based on your session duration patterns. The [REFERENCE_TIMESTAMP] should be the current time or the time of the most recent turn, depending on whether you want decay relative to now or relative to conversation end.

Before deploying, validate that your conversation history format matches the expected turn structure. If your chat platform uses a different schema, add a preprocessing step to normalize turn IDs, roles, and timestamps before passing data to this prompt. The override rule for correction turns is a safety mechanism—test it with conversations containing explicit user corrections to confirm the model respects the override consistently. For high-stakes pruning decisions, route turns scored near the threshold boundaries to human review rather than relying solely on the composite score.

The Multi-Turn Retention Scoring Prompt is designed to be a pre-pruning step, not the pruning action itself. In a production harness, this prompt runs as a stateless evaluation call that receives a batch of conversation turns and returns scored metadata. The application layer then uses these scores to make deterministic pruning decisions—retaining turns above a configurable threshold, summarizing mid-scoring turns, and discarding low-scoring turns. This separation of scoring from action is critical: the model provides judgment, but the application enforces the retention policy, ensuring reproducibility and auditability.

To wire this into an application, implement a scoring service that accepts a list of turn objects (each with turn_id, speaker, content, timestamp, and optional prior_scores) and the prompt template. The service should: (1) assemble the prompt with the turn batch and scoring dimensions; (2) call the model with response_format set to a strict JSON schema matching the expected output—an array of scored turn objects with fields for turn_id, task_relevance, information_novelty, user_sentiment, correction_presence, temporal_decay, and retention_priority; (3) validate the response against the schema and retry once on parse failure with a repair prompt that includes the raw output and schema; (4) log the full prompt, response, and scores to your observability pipeline for later audit and eval comparison. For high-throughput systems, batch turns into groups of 10-20 to balance latency against per-turn scoring granularity. Use a fast, cost-effective model (e.g., GPT-4o-mini or Claude Haiku) for this scoring step, reserving larger models for the downstream generation tasks that consume the pruned context.

Validation and failure handling are essential because scoring errors cascade into context loss. Implement a post-scoring validator that checks: all turn_id values match the input batch, scores are within the defined numeric ranges, and retention_priority is a valid enum value. If validation fails, retry with the same prompt plus the validation error message appended as a [CONSTRAINTS] update. After two failed retries, fall back to a rule-based scoring heuristic (e.g., recency-weighted baseline) and flag the session for human review if the conversation is high-stakes. For regulated or compliance-sensitive deployments, route sessions where the model's correction_presence score exceeds a threshold to a human review queue before pruning, since discarding user corrections is a top failure mode. Store scored turn metadata alongside the conversation record so that downstream debugging can reconstruct why a turn was retained or discarded.

Integration with the pruning pipeline should be explicit and testable. After scoring, the application's pruning controller reads the retention_priority field and applies policy: critical turns are always retained verbatim, high turns are retained unless budget is exhausted, medium turns are candidates for summarization, and low turns are discarded. The pruning controller should emit structured logs showing which turns were kept, summarized, or dropped, along with the scores that drove each decision. Before deploying any threshold change, run the scoring prompt against a golden dataset of conversations with human-annotated retention judgments and measure inter-annotator agreement between model scores and human labels. Track precision and recall of retained critical turns over time, and set alerts if the rate of user corrections after pruning increases—this signals that the scoring prompt is undervaluing correction turns and needs recalibration or threshold adjustment.