Session Graceful Degradation Prompt Under Load

This prompt is for platform engineers and SREs who need to programmatically reduce the fidelity of an active conversation session when system resources—such as context window budget, API rate limits, or compute capacity—are critically constrained. The core job-to-be-done is to decide which elements of the session state (dialogue history, pending actions, retrieved documents, user preferences) to compress, defer, or drop entirely, while preserving the minimum context required for the assistant to continue its core workflow without catastrophic failure. The ideal user is a developer building a production chat product who needs a deterministic, auditable degradation plan instead of an opaque model crash or a generic context truncation that silently breaks multi-turn logic.

Use this prompt when you have a real-time resource constraint signal (e.g., token budget exceeded, latency threshold breached, or a load-shedding event) and you need a structured degradation plan as a machine-readable output. The prompt requires a snapshot of the current session state, a list of active workflow constraints, and a criticality map that defines which functions must be preserved. It is not a substitute for upstream context budgeting, prompt caching, or model routing. Do not use this prompt for routine session summarization or for deciding what to include in a standard context window—those are separate, non-degraded workflows. This prompt is explicitly for emergency triage under load, where the cost of inaction is a dropped session or an unresponsive system.

The output is a structured degradation plan, not a conversational response to the user. It must be integrated into an application harness that validates the plan against a schema, checks for unacceptable degradation (e.g., dropping a required compliance acknowledgment or an active payment step), and applies the plan to the session state before the next model call. If the degradation plan flags a 'critical workflow break' risk, the harness should escalate to a human operator or queue the session for a graceful pause rather than executing a destructive state mutation. Always log the full degradation decision for auditability, and never apply a degradation plan that the prompt itself marks as 'unacceptable.'

This template is designed to be dropped into a production AI harness that monitors system resource consumption (e.g., context window utilization, latency budgets, or token-per-second limits). When a threshold is breached, this prompt instructs the model to produce a structured degradation plan. The plan must prioritize preserving critical workflow continuity—such as pending actions, unresolved questions, and active constraints—while identifying non-essential context elements that can be safely compressed, deferred, or dropped. The output is a machine-readable action plan, not a user-facing message.

textCopy
You are a session state manager for a production AI assistant operating under resource constraints. The current session has exceeded its allocated [RESOURCE_BUDGET] for [RESOURCE_TYPE, e.g., context window tokens, latency]. You must produce a degradation plan to reduce state fidelity while preserving critical workflow continuity.

## Current Session State
[FULL_SESSION_STATE]

## Critical Workflow Elements (Must Preserve)
[CRITICAL_ELEMENTS_SCHEMA]

## Degradation Actions Available
- COMPRESS: Summarize or truncate the element to reduce its token footprint.
- DEFER: Remove the element from active context but store it in [DEFERRAL_STORE] for later retrieval.
- DROP: Permanently remove the element. Only allowed if it has no impact on active tasks or user commitments.

## Output Schema
Produce a JSON object strictly conforming to this structure:
{
  "degradation_plan": [
    {
      "element_id": "string",
      "element_type": "string",
      "action": "COMPRESS | DEFER | DROP",
      "rationale": "string (explain why this action was chosen over others)",
      "compressed_content": "string | null (required if action is COMPRESS)",
      "deferral_key": "string | null (required if action is DEFER)",
      "user_impact_assessment": "string (describe how this degradation might affect the user experience)"
    }
  ],
  "aggregate_impact_summary": "string (a concise summary of the overall user impact)",
  "critical_path_intact": "boolean (true if all critical workflow elements remain functional after degradation)"
}

## Constraints
- Do not degrade any element tagged as [CRITICAL_TAG] in the session state.
- If degrading an element would break a core workflow (as defined in [CRITICAL_ELEMENTS_SCHEMA]), you must set `critical_path_intact` to false and explain why in the rationale.
- Prefer COMPRESS over DEFER, and DEFER over DROP, unless the element is explicitly marked as low-priority.
- The `user_impact_assessment` must be specific. Do not use generic phrases like "may affect user experience."

## Examples
[FEW_SHOT_EXAMPLES]

To adapt this template for your application, replace the square-bracket placeholders with your system's specific data structures. [FULL_SESSION_STATE] should be your serialized conversation object, including turn history, tool outputs, and active variables. [CRITICAL_ELEMENTS_SCHEMA] must define what constitutes a critical workflow in your domain—for example, an unsubmitted order in a commerce bot or an unresolved patient symptom in a clinical triage assistant. [FEW_SHOT_EXAMPLES] should include at least one example where critical_path_intact is false to teach the model when to refuse degradation. The [DEFERRAL_STORE] placeholder should be replaced with the actual name of your database or cache table so the plan references a real target for deferred state. Before deploying, validate that the output JSON conforms to the schema using a strict validator in your application layer, and log every degradation plan for auditability.

The degradation prompt is not a standalone utility; it is a control-plane decision node that must be called before the main conversation context is assembled for the next turn. In practice, this means your application server or orchestration layer should invoke this prompt when a resource threshold is breached—typically when the estimated token count for the next request exceeds a configured budget, when latency percentiles cross a defined ceiling, or when an upstream context retrieval step returns more data than the window can safely hold. The prompt receives a structured snapshot of the current session state, resource constraints, and criticality markers, and it returns a degradation plan that your application then executes by pruning, summarizing, or deferring context elements before calling the primary model.

Wire the prompt into a pre-call hook or middleware function that runs after context retrieval and before the main model invocation. The function should: (1) calculate the current context budget utilization and compare it against configurable thresholds; (2) if the threshold is breached, serialize the session state into the [SESSION_STATE] input block—including turn history summaries, pending actions, unresolved questions, active tool outputs, and user preference markers; (3) call the degradation prompt with the serialized state and the [RESOURCE_CONSTRAINTS] block specifying token budget, latency target, and any hard requirements; (4) parse the returned JSON degradation plan and validate it against a schema that requires retained_elements, compressed_elements, deferred_elements, dropped_elements, and a user_impact_assessment with a severity score; (5) apply the plan by reconstructing the context payload according to the retention and compression instructions; and (6) log the degradation decision, the resource conditions that triggered it, and the impact assessment for observability. If the degradation plan fails schema validation, fall back to a safe default strategy—such as retaining only the most recent N turns and the active pending actions—and log the validation failure as a warning.

Retry logic should be minimal for this prompt because it runs on a latency-sensitive path. Configure a single retry with exponential backoff only for transient failures like timeouts or 5xx errors. Do not retry on schema validation failures or impact severity scores that exceed your configured threshold; instead, escalate to the fallback strategy and surface the event to your observability pipeline. For high-risk sessions—such as those involving financial transactions, clinical workflows, or compliance-bound interactions—add a human-review gate when the degradation plan's user_impact_assessment.severity is high or critical. The review gate should pause the session, present the degradation plan and the original context to an operator, and require explicit approval before the degraded context is sent to the main model. This prevents silent degradation from breaking core workflow continuity in regulated domains.

Model choice matters here. The degradation prompt requires strong instruction-following and structured output reliability, not creative reasoning. Use a fast, cost-efficient model with strong JSON mode support—such as Claude 3.5 Haiku, GPT-4o-mini, or a fine-tuned small model if you have sufficient degradation examples. Avoid using your largest, most expensive model for this control-plane decision because the degradation prompt runs on the critical path and its latency directly adds to the user's perceived response time. If you are already operating under load, adding a slow degradation step compounds the problem. Cache the degradation prompt's system instructions as a static prefix to reduce per-call token costs, and consider batching degradation decisions for multiple concurrent sessions if your architecture supports it.

Observability is non-negotiable. Emit structured logs containing the session identifier, the resource conditions that triggered degradation, the full degradation plan, the impact severity score, and whether the plan was applied automatically or escalated for human review. Track degradation frequency, severity distribution, and the specific context elements most frequently compressed or dropped. These metrics will reveal whether your context assembly strategy, retrieval pipeline, or session management policies need adjustment. If degradation events cluster around specific session types or user workflows, investigate whether those workflows can be optimized at the source rather than relying on runtime degradation as a permanent fix. The degradation prompt is a safety valve, not a design target—persistent high-severity degradation signals a need for architectural changes upstream.