Trace Summarization for Stakeholders Prompt

Engineering leads and AI program managers use this prompt to convert a verbose, technical production trace into a structured executive summary. The core job-to-be-done is preserving diagnostic signal—key events, failure points, user impact, and recommended actions—while removing implementation details like raw span IDs, token counts, and function call arguments that obscure business impact for non-engineering stakeholders. The ideal user is someone who understands the trace data well enough to validate the summary but needs a reliable first draft before communicating to a VP, director, or client.

This prompt requires a complete, pre-assembled trace context as input. You must provide the raw trace data in [TRACE_DATA], a description of the system's purpose in [SYSTEM_CONTEXT], and the stakeholder audience type in [AUDIENCE_TYPE] (e.g., 'VP of Product', 'Client Success Manager', 'Director of Operations'). The prompt works best with single-session traces where the user journey, tool calls, and final output are captured end-to-end. Do not use this prompt for aggregate trend analysis across multiple sessions, for real-time incident response where latency is critical, or for traces where the root cause is already known and documented—those workflows require different prompts from the incident trace correlation or root-cause isolation families.

This prompt belongs in a workflow with a mandatory human review step before the summary reaches any stakeholder. The model can misclassify the severity of a failure, omit a subtle user-impact signal, or recommend an action that requires engineering judgment to validate. After generating the summary, route it through a review queue where an engineer or program manager confirms the accuracy of the failure classification, the completeness of the user impact description, and the appropriateness of the recommended actions. If the trace involves a safety incident, a data leakage event, or a regulated domain, escalate the review to include a compliance or trust-and-safety reviewer before distribution.

This prompt template is designed to be pasted directly into your AI orchestration layer or workbench. It converts a raw, multi-span production trace into a structured, stakeholder-ready summary. The summary focuses on the session's key events, identified failure points, user impact, and recommended next steps, stripping away low-level debugging detail while retaining the evidence needed for decision-making. Before using this template, ensure you have a complete, chronological trace log that includes user input, retrieval steps, tool calls, model outputs, and any error metadata.

textCopy
You are an AI observability analyst. Your task is to convert a verbose production trace into an executive summary for engineering leads and program managers.

## INPUT
[TRACE_DATA]

## OUTPUT_SCHEMA
Return a JSON object with the following structure:
{
  "session_id": "string",
  "summary": "A 3-5 sentence plain-language summary of what happened in this session from the user's perspective.",
  "key_events": [
    {
      "timestamp": "ISO 8601 string",
      "event_type": "user_input | retrieval | tool_call | model_output | error | fallback | refusal",
      "description": "One sentence describing the event and its outcome.",
      "success": "boolean"
    }
  ],
  "failure_points": [
    {
      "event_index": "integer (index of the failing event in key_events)",
      "failure_type": "tool_error | retrieval_gap | hallucination | timeout | refusal | format_error | other",
      "root_cause_hypothesis": "A concise, evidence-based hypothesis for why this failure occurred.",
      "user_impact": "high | medium | low | none"
    }
  ],
  "user_impact_summary": "A 1-2 sentence assessment of the overall impact on the user's task or experience.",
  "recommended_actions": [
    {
      "action": "A specific, actionable recommendation (e.g., 'Investigate retrieval index for document ID X', 'Review tool schema for function Y').",
      "priority": "critical | high | medium | low",
      "owner_suggestion": "The engineering role or team best suited to investigate (e.g., 'Search Team', 'Platform SRE')."
    }
  ],
  "diagnostic_signal_preserved": [
    "A list of 3-5 key diagnostic details (IDs, error codes, specific spans) that must be preserved for engineering follow-up."
  ]
}

## CONSTRAINTS
1. **Preserve Diagnostic Signal:** Do not omit error codes, tool names, span IDs, or specific document IDs mentioned in the trace. List them in `diagnostic_signal_preserved`.
2. **Be Concise:** The `summary` must be understandable by a non-engineering stakeholder. Avoid jargon in the summary field.
3. **Evidence-Based:** Every `root_cause_hypothesis` must reference a specific event or error from the trace. Do not speculate without evidence.
4. **Focus on Failures:** If the trace contains no errors or anomalies, state that clearly and set `failure_points` to an empty array.
5. **Actionable Output:** Every `recommended_action` must be specific enough to be turned into a Jira ticket or investigation task.

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", add a `human_review_required` boolean field to the output and set it to `true`. Include a `reviewer_notes` field with specific questions a human reviewer must answer before the summary is distributed.

To adapt this template for your environment, replace the [TRACE_DATA] placeholder with your raw trace object. The trace should be pre-processed into a chronological, text-based format. If your observability platform uses a specific taxonomy for event types or failure modes, update the event_type and failure_type enums in the OUTPUT_SCHEMA to match. The [RISK_LEVEL] variable should be set dynamically by your harness based on the trace context—for example, set it to "high" if the trace involves a payment tool, PII, or a safety filter activation. This triggers the additional human review fields, ensuring that high-stakes summaries are not distributed without oversight.

This prompt is designed to sit downstream of a trace reconstruction step. Before calling the summarization prompt, assemble a complete trace object from your observability store (span data, tool-call logs, retrieval metadata, model responses, and user feedback signals). The prompt expects a structured [TRACE_JSON] input containing chronological events, not raw log lines. If your traces are stored across multiple systems (e.g., LangSmith for LLM calls, CloudWatch for infrastructure metrics, and a separate feedback database), build a pre-processing step that merges these into a single timeline with consistent event types before invoking the prompt.

Wire the prompt into a review workflow with three stages: pre-validation, generation, and post-validation. In pre-validation, check that the trace contains required fields (session ID, timestamps, event types, and at least one model interaction). Reject traces shorter than 3 events or older than your retention window. For generation, use a model with strong instruction-following and long-context handling (GPT-4o, Claude 3.5 Sonnet, or Gemini 1.5 Pro). Set temperature=0.1 to preserve factual consistency across trace events. In post-validation, run the output through a schema check that verifies the summary contains all required sections: Executive Summary, Key Events Timeline, Failure Points, User Impact Assessment, and Recommended Actions. If any section is missing or below a minimum character threshold, retry once with a more explicit constraint in the prompt. Log every generation attempt with the trace ID, model version, token counts, and validation results for auditability.

For high-severity incidents or regulated domains, insert a human approval gate after generation. Route summaries with failure_points.severity == 'critical' or user_impact.affected_users > [THRESHOLD] to a review queue before distribution. Store the approved summary alongside the raw trace for future comparison. Avoid using this prompt for real-time alerting—it is designed for post-hoc review and stakeholder reporting, not for triggering pager notifications. If you need real-time trace evaluation, pair this with a separate monitoring prompt that checks trace patterns against SLO thresholds before summarization.