Salience Scoring Prompt for Conversation Turns

This prompt is designed for AI engineers and platform teams building deterministic context pruning and compression pipelines for long-running conversational AI systems. The core job-to-be-done is to programmatically rank every turn in a conversation history by its future utility, enabling the system to discard low-value turns when context windows are exceeded. You should use this prompt when you need a repeatable, auditable scoring mechanism that goes beyond simple recency-based eviction, and when you require a structured output that can be consumed directly by your application's context assembly logic. The ideal user is someone who already has a conversation history stored as structured turns and needs to make a hard decision about what to keep before making an LLM call.

This prompt is not a general-purpose summarizer or a replacement for a vector database. Do not use it when you need to retrieve information from external knowledge bases (that's a RAG retrieval job) or when you simply need a prose summary of the conversation (use a session summarization prompt instead). It is also not suitable for real-time, per-token streaming decisions; it operates on complete, discrete turns. The prompt assumes you have already segmented the conversation into turns with clear speaker roles. If your chat logs are unstructured or lack speaker attribution, you must preprocess them before invoking this salience scorer. The output is a ranked list with scores, not a compressed history—you will need a separate pruning step in your application code to act on these scores.

Before integrating this prompt, you must define your salience rubric based on your application's specific needs. A support bot might prioritize turns containing error codes and resolution steps, while a research copilot might prioritize hypothesis statements and data references. The prompt template includes a [RUBRIC] placeholder for this reason. Start by annotating 50–100 real conversations with your own salience judgments to establish a baseline, then use the inter-rater reliability checks described in the evaluation harness to calibrate the model's scoring against your human-annotated ground truth. Never deploy this prompt without a validation layer that checks for missing turns, score distribution anomalies, and schema compliance.

This template provides the core instruction set for a model to act as a salience scorer. It is designed to be dropped into a larger application harness where conversation history is fed in programmatically. The prompt uses square-bracket placeholders for all dynamic inputs, ensuring you can swap in different rubrics, conversation logs, and output schemas without rewriting the core logic. The goal is to produce a deterministic, ranked list of turns that your application can use to prune low-value context before the next LLM call.

textCopy
You are a conversation salience scorer. Your task is to evaluate each turn in a multi-turn conversation and assign a salience score based on a defined rubric. The output will be used to deterministically prune the conversation history, retaining only the most critical turns for future context.

## INPUT

### Conversation History
[CONVERSATION_HISTORY]

### Salience Rubric
[RUBRIC]

### Output Schema
[OUTPUT_SCHEMA]

## INSTRUCTIONS

1.  **Parse the Input:** Read the full [CONVERSATION_HISTORY]. Each turn is a discrete unit of analysis.
2.  **Apply the Rubric:** For each turn, evaluate its content against every dimension in the [RUBRIC]. A turn's score should reflect its decision relevance, factual density, and future utility. Do not score based on politeness or length alone.
3.  **Generate Structured Output:** Produce a JSON object conforming to the [OUTPUT_SCHEMA]. The output must include a `scored_turns` array where each object contains the original `turn_id`, a `score`, and a `rationale`.
4.  **Rank and Justify:** The `scored_turns` array must be sorted in descending order by `score`. The `rationale` field must cite specific evidence from the turn and the rubric dimension that most influenced the score.

## CONSTRAINTS

- Do not modify the original turn text in the output.
- If a turn is completely empty or contains only a greeting with no substantive content, assign it a score of 0.
- If the rubric is ambiguous for a specific turn, explain the ambiguity in the `rationale` and assign a conservative (lower) score.
- Output only the valid JSON object. Do not include any text outside the JSON structure.

To adapt this template, start by defining your [RUBRIC]. A strong rubric for a support copilot might include dimensions like 'Contains a Decision,' 'States a Constraint,' and 'Introduces a New Fact.' For a research assistant, you might prioritize 'States a Hypothesis' and 'Cites a Source.' The [OUTPUT_SCHEMA] should be a strict JSON schema you validate against in your application code. A common failure mode is scoring turns in isolation; ensure your rubric instructs the model to consider a turn's importance relative to the entire conversation. After generating scores, always run a validation check to confirm all turn_ids from the input are present in the output and that no scores fall outside your defined range. For high-stakes pruning, implement an inter-rater reliability check by running the prompt twice with a slightly different [RUBRIC] ordering and flagging sessions where the top-k turns diverge significantly for human review.

The salience scoring prompt is designed to operate as a deterministic filter inside a context pruning pipeline, not as a standalone chat interaction. In production, you will call this prompt once per conversation turn batch (or per full session) before the context window budget is exceeded. The output is a ranked list of turns with salience scores and a recommended retention cutoff. Your application layer should consume this JSON output and prune turns below the threshold before assembling the next model request. This prompt is not a replacement for summarization; it decides what to keep, not how to compress what remains.

Integration pattern: Wrap the prompt in a function that accepts a list of conversation turns and a token budget. Each turn should include the speaker role, message text, and a turn index. The function should: (1) format the turns into the [TURN_LIST] placeholder, (2) inject the [SALIENCE_RUBRIC] (your domain-specific scoring dimensions such as decision relevance, factual density, future actionability, and unresolved question presence), (3) set the [TOKEN_BUDGET] to the number of turns or tokens you can retain, and (4) call the model with response_format set to JSON mode using the provided [OUTPUT_SCHEMA]. Validate the response against the schema before pruning. If validation fails, retry once with the error message appended to the prompt. If the retry also fails, log the failure and fall back to a safe default: retain the most recent N turns and any turns containing explicit user corrections or unresolved questions.

Model selection and latency: Use a fast, instruction-following model for this task (GPT-4o-mini, Claude Haiku, or equivalent). Salience scoring is a high-frequency, cost-sensitive operation that runs before every major context assembly step. Avoid using large reasoning models here; the rubric is structured enough that a smaller model can produce reliable scores. Set a strict timeout (under 2 seconds) and implement a circuit breaker: if the scoring step fails or times out more than three times in a rolling window, skip scoring and retain all turns within the budget, emitting a monitoring alert. Human review gate: For high-stakes domains (healthcare, legal, finance), implement a sampling-based review where 5% of scored batches are sent to a human reviewer who compares the model's salience rankings against the rubric. Track inter-rater agreement over time and trigger a prompt revision if agreement drops below 0.8.

Logging and observability: Log every scoring call with the input turn count, output retention count, validation status, retry count, and latency. Attach the scored turn IDs to downstream model requests so you can trace whether pruned turns contained information that later proved necessary. If users frequently re-ask questions that were pruned, your salience rubric or threshold is too aggressive. What to avoid: Do not use this prompt to score turns one at a time; the model needs the full conversation context to assess relative salience. Do not prune turns before human-visible conversation review is complete if the session is subject to audit. And do not treat the salience score as a confidence measure for downstream reasoning; it is a ranking signal for pruning decisions only.