Test Suite Health Dashboard Prompt Template

Engineering managers and test infrastructure leads use this prompt to synthesize raw test suite signals into a structured, dashboard-ready JSON health summary. The primary job-to-be-done is converting disparate data points—flakiness rates, execution duration trends, coverage gaps, and failure hotspots—into a single, repeatable payload that can feed internal dashboards, weekly reliability reports, or automated alerting pipelines. This prompt assumes you have already collected the input data from CI systems, coverage tools, and test history databases. It does not perform root cause analysis or suggest specific fixes; pair it with the Flaky Test Root Cause Analysis or Test Gap Detection prompts for deeper investigation.

The ideal user is someone who owns test infrastructure health and needs a consistent, machine-readable summary rather than ad-hoc manual analysis. Required inputs include aggregated test execution data (pass/fail/flaky counts, duration percentiles, historical trends), coverage metrics (line, branch, and function coverage with deltas), and failure hotspot identification (modules or test files with elevated failure rates). The prompt works best when these inputs are pre-computed and provided as structured context—it is a synthesis engine, not a raw log parser. Do not use this prompt when you need real-time incident response, root cause identification, or specific code-level fix recommendations; those workflows require different prompt designs with deeper diagnostic reasoning.

Before using this prompt, ensure your input data pipeline is stable and the metrics you feed in are accurate. The output JSON includes severity scores and trend indicators that downstream systems may treat as authoritative, so garbage in means garbage out. Validate the output schema against your dashboard's expected contract, and consider adding a lightweight eval step that checks for required fields, valid severity ranges, and trend direction consistency. If the health summary will trigger automated alerts or paging, route the output through a human review step or a confirmation threshold before acting on it. For teams just starting with test health dashboards, begin with a subset of metrics (flakiness rate and duration trend) before expanding to the full schema.

This template is designed to be copied directly into your prompt management system, IDE, or orchestration layer. It accepts raw test execution data, coverage reports, and historical flakiness records, then produces a dashboard-ready JSON payload with severity scores, trend indicators, and actionable hotspots. Every placeholder is enclosed in square brackets—replace them with your actual data sources, constraints, and output requirements before use.

textCopy
You are a test suite health analyst. Your task is to synthesize the provided test execution data, coverage reports, and flakiness history into a structured health dashboard payload.

## INPUT
- Test execution results (last [TIME_PERIOD]): [EXECUTION_DATA]
- Coverage report: [COVERAGE_DATA]
- Flakiness history (last [FLAKINESS_WINDOW]): [FLAKINESS_DATA]
- Failure hotspot map: [HOTSPOT_DATA]
- Execution duration trends: [DURATION_DATA]
- Team ownership mapping: [OWNERSHIP_DATA]

## OUTPUT SCHEMA
Return a single JSON object with this exact structure:
{
  "dashboard_summary": {
    "overall_health_score": number (0-100),
    "health_trend": "improving" | "stable" | "degrading",
    "generated_at": "ISO 8601 timestamp",
    "data_window": {
      "start": "ISO 8601 timestamp",
      "end": "ISO 8601 timestamp"
    }
  },
  "flakiness": {
    "current_rate": number (percentage),
    "trend": "improving" | "stable" | "degrading",
    "severity": "critical" | "warning" | "healthy",
    "top_flaky_tests": [
      {
        "test_name": string,
        "failure_rate": number,
        "owner_team": string,
        "last_failure": "ISO 8601 timestamp"
      }
    ]
  },
  "coverage": {
    "line_coverage_pct": number,
    "branch_coverage_pct": number,
    "trend": "improving" | "stable" | "degrading",
    "severity": "critical" | "warning" | "healthy",
    "largest_gaps": [
      {
        "module": string,
        "uncovered_lines": number,
        "risk_level": "high" | "medium" | "low",
        "owner_team": string
      }
    ]
  },
  "execution_duration": {
    "median_duration_seconds": number,
    "p95_duration_seconds": number,
    "trend": "improving" | "stable" | "degrading",
    "severity": "critical" | "warning" | "healthy",
    "slowest_suites": [
      {
        "suite_name": string,
        "median_duration_seconds": number,
        "owner_team": string
      }
    ]
  },
  "failure_hotspots": [
    {
      "module_or_area": string,
      "failure_count": number,
      "associated_teams": [string],
      "dominant_failure_type": string,
      "recommended_action": string
    }
  ],
  "recommendations": [
    {
      "priority": "P0" | "P1" | "P2",
      "category": "flakiness" | "coverage" | "duration" | "hotspot",
      "description": string,
      "owner_team": string,
      "expected_impact": string
    }
  ]
}

## CONSTRAINTS
- All severity scores must be data-driven: use thresholds from [SEVERITY_THRESHOLDS].
- Trend indicators must compare current window against the previous equivalent window.
- If any data source is missing or incomplete, set the corresponding severity to "warning" and include a `data_quality_note` field in that section.
- Do not fabricate numbers. If a metric cannot be computed, use null and add a `computation_note`.
- Limit top_flaky_tests, largest_gaps, slowest_suites, and failure_hotspots to at most [MAX_ITEMS_PER_SECTION] entries each.
- Recommendations must be actionable and tied to specific data signals, not generic advice.

## RISK LEVEL
[RISK_LEVEL]

## EXAMPLES
[EXAMPLES]

To adapt this template, start by replacing the data-source placeholders with your actual pipeline outputs—CI test result JSON, coverage XML or LCOV reports, and flakiness tracking database extracts. Set [SEVERITY_THRESHOLDS] to match your team's SLAs (for example, flakiness above 5% might be "critical" while below 1% is "healthy"). If you are integrating this into a production dashboard, wire the output JSON directly into your monitoring frontend and add a validation step that rejects malformed payloads before they reach the display layer. For high-risk environments where dashboard data drives on-call decisions, always include a human review gate before publishing severity scores that could trigger automated alerts or paging.

This prompt is designed to be called by an automated pipeline step or a monitoring service, not a one-off chat. The typical integration point is a scheduled job that collects raw test data from CI systems (e.g., JUnit XML, pytest JSON reports, or a test analytics API), assembles the [TEST_SUITE_DATA] payload, calls the model, validates the JSON output, and writes the result to a dashboard database or alerting system. The prompt expects aggregated statistics—flakiness rates, duration trends, coverage gaps, and failure hotspots—so the calling application must pre-compute these from raw test logs before invoking the model. Do not pass raw log lines directly; the model performs synthesis and severity scoring, not log parsing.

Wire the prompt into a function with the following guardrails. Input assembly: Build the [TEST_SUITE_DATA] object with required fields: suite_name, time_range, total_tests, flaky_test_rate, avg_duration_seconds, duration_trend (one of improving, stable, degrading), coverage_gap_modules (array of {module, uncovered_lines, risk}), and failure_hotspots (array of {test_file, failure_count, last_failure, error_signature}). Model choice: Use a model with strong JSON mode and structured output support (GPT-4o, Claude 3.5 Sonnet, or equivalent). Enable strict JSON mode or supply the output schema as a tool definition. Validation: After receiving the response, validate the JSON against the expected schema—check that severity_score is an integer 1–5, trend_indicators contains only allowed enum values (worsening, stable, improving), and all required fields are present. If validation fails, retry once with the validation errors appended to [CONSTRAINTS]. Logging: Log the input payload, model response, validation result, and any retries to your observability stack. This is critical for debugging drift when the model changes its output shape or when upstream test data quality degrades.

For production deployment, add two additional safeguards. First, implement a stale-data check before calling the model: if the input data is older than the configured freshness window (e.g., 24 hours for a daily dashboard), skip the model call and emit a data_stale status to the dashboard instead of synthesizing misleading health scores. Second, add a human-review gate if the severity score is 4 or 5 (critical or high risk). Rather than auto-publishing a red dashboard, route the output to a notification channel (Slack, PagerDuty) with the raw data and model-generated summary for a human to confirm before the dashboard updates. This prevents false alarms from data anomalies or model misinterpretation from triggering unnecessary incident response. Avoid wiring this prompt directly to an alerting system without these gates—the model can misjudge severity when input data contains outliers or when test suites undergo legitimate restructuring.