Flaky Test History Trend Analysis Prompt

QA leads and test infrastructure engineers use this prompt when they need to move beyond individual flaky test tickets and understand systemic trends. The core job-to-be-done is distinguishing a genuine regression in test stability from normal statistical noise. This requires at least two weeks of daily aggregated data—including pass/fail counts, flakiness rates, and associated metadata like deployment markers or infrastructure change events. The prompt is designed to ingest structured time-series data, not raw logs; sibling prompts in the flaky test detection group handle log extraction and single-test diagnosis.

The ideal input is a CSV or JSON dataset with daily granularity, containing columns for date, total_executions, total_failures, unique_flaky_tests, and optional deployment_ids or infra_change_tags. The prompt expects you to provide this data in the [INPUT] placeholder. It also requires a [CONTEXT] block describing known environmental factors (e.g., 'CI migration on March 10', 'New test suite added March 15') and an [OUTPUT_SCHEMA] defining the desired report structure. Without this structured input, the model will produce plausible but ungrounded trend analysis—a common failure mode when users paste raw logs or anecdotal observations instead of aggregated metrics.

Do not use this prompt for real-time failure classification, single-test root cause analysis, or immediate CI pipeline triage. Those workflows require different prompts optimized for narrow diagnostic reasoning. This prompt is also inappropriate when you lack sufficient historical data; with fewer than 10 data points, trend detection is statistically unreliable and the model may overfit to noise. If you need to quarantine a specific flaky test or diagnose a race condition, use the sibling prompts in this group. After running this analysis, the next step is typically to drill into the highest-severity trends using the Flaky Test Severity Scoring Rubric Prompt or to initiate root cause investigation on specific regressions identified in the trend report.

The following prompt template is designed to be dropped into your AI harness. It expects structured historical flaky test data, a time window, and optional infrastructure change logs. The model is instructed to perform statistical analysis, detect anomalies, and produce a structured trend report with alert thresholds. All placeholders use square brackets and must be replaced with real data before execution. The output schema is explicitly defined to ensure the response can be parsed and ingested by downstream monitoring dashboards or alerting systems.

textCopy
You are a QA analytics system analyzing historical flaky test data.

Your task is to detect trends, regressions, seasonal patterns, and correlations with infrastructure changes or deployments.

## INPUT DATA

### Flaky Test History
[FLAKY_TEST_HISTORY_JSON]

Each record contains:
- test_name: string
- test_suite: string
- execution_timestamp: ISO 8601
- status: "flaky" | "pass" | "fail"
- failure_reason: string | null
- execution_duration_ms: integer
- environment: string
- branch: string

### Infrastructure Change Log (Optional)
[INFRASTRUCTURE_CHANGE_LOG_JSON]

Each record contains:
- change_timestamp: ISO 8601
- change_type: "deployment" | "config_change" | "dependency_update" | "infra_scaling" | "other"
- description: string
- affected_services: string[]

### Analysis Parameters
- Time Window Start: [START_DATE]
- Time Window End: [END_DATE]
- Granularity: [GRANULARITY] (e.g., "daily", "weekly")
- Alert Threshold: [ALERT_THRESHOLD] (flakiness rate increase percentage that triggers an alert)
- Statistical Significance Level: [SIGNIFICANCE_LEVEL] (e.g., 0.05)

## CONSTRAINTS

- Do not fabricate data points. If data is insufficient for a conclusion, state that explicitly.
- Flag any analysis that relies on fewer than [MIN_SAMPLE_SIZE] data points as low-confidence.
- Distinguish between correlation and causation. Do not claim causation without strong temporal and mechanistic evidence.
- If infrastructure change logs are not provided, skip correlation analysis and note the limitation.

## OUTPUT SCHEMA

Return a single JSON object with this exact structure:

{
  "report_metadata": {
    "analysis_window": { "start": string, "end": string },
    "granularity": string,
    "total_test_executions": integer,
    "total_flaky_occurrences": integer,
    "overall_flakiness_rate": number,
    "generated_at": string
  },
  "trend_analysis": {
    "direction": "increasing" | "decreasing" | "stable" | "inconclusive",
    "trend_description": string,
    "statistical_significance": {
      "p_value": number | null,
      "test_used": string | null,
      "is_significant": boolean,
      "confidence_note": string
    },
    "rate_of_change_per_period": number | null
  },
  "seasonal_patterns": [
    {
      "pattern_description": string,
      "period": string,
      "confidence": "high" | "medium" | "low",
      "evidence_summary": string
    }
  ],
  "regressions_detected": [
    {
      "test_name": string,
      "test_suite": string,
      "regression_start_date": string,
      "previous_flakiness_rate": number,
      "current_flakiness_rate": number,
      "severity": "critical" | "high" | "medium" | "low",
      "likely_trigger": string | null
    }
  ],
  "infrastructure_correlations": [
    {
      "change_description": string,
      "change_timestamp": string,
      "correlated_test_suites": string[],
      "correlation_strength": "strong" | "moderate" | "weak" | "none",
      "evidence": string,
      "recommended_action": string
    }
  ],
  "alert_thresholds_breached": [
    {
      "threshold_name": string,
      "current_value": number,
      "threshold_value": number,
      "breach_severity": "critical" | "warning",
      "recommended_action": string
    }
  ],
  "top_flaky_tests": [
    {
      "test_name": string,
      "test_suite": string,
      "flakiness_rate": number,
      "total_flaky_runs": integer,
      "trend": "worsening" | "improving" | "stable"
    }
  ],
  "limitations": [string],
  "recommendations": [
    {
      "priority": integer,
      "category": "quarantine" | "fix" | "investigate" | "monitor" | "infrastructure_change",
      "description": string,
      "affected_tests": string[],
      "expected_impact": string
    }
  ]
}

## EXAMPLES

### Example Input (Abbreviated)
Flaky Test History: [{"test_name":"login_timeout_test","test_suite":"auth","execution_timestamp":"2025-03-01T10:00:00Z","status":"flaky","failure_reason":"timeout","execution_duration_ms":35000,"environment":"staging","branch":"main"}]
Infrastructure Change Log: [{"change_timestamp":"2025-03-01T08:00:00Z","change_type":"config_change","description":"Increased auth service timeout from 30s to 60s","affected_services":["auth-service"]}]

### Example Output (Abbreviated)
{"report_metadata":{"analysis_window":{"start":"2025-03-01","end":"2025-03-07"},"granularity":"daily","total_test_executions":1200,"total_flaky_occurrences":45,"overall_flakiness_rate":3.75,"generated_at":"2025-03-08T00:00:00Z"},"trend_analysis":{"direction":"increasing","trend_description":"Flakiness rate increased from 2.1% to 3.75% over the analysis window.","statistical_significance":{"p_value":0.02,"test_used":"Mann-Kendall","is_significant":true,"confidence_note":"Trend is statistically significant at the 0.05 level."},"rate_of_change_per_period":0.23},"seasonal_patterns":[],"regressions_detected":[{"test_name":"login_timeout_test","test_suite":"auth","regression_start_date":"2025-03-01","previous_flakiness_rate":0.5,"current_flakiness_rate":12.0,"severity":"high","likely_trigger":"Config change: Increased auth service timeout from 30s to 60s"}],"infrastructure_correlations":[{"change_description":"Increased auth service timeout from 30s to 60s","change_timestamp":"2025-03-01T08:00:00Z","correlated_test_suites":["auth"],"correlation_strength":"strong","evidence":"Auth suite flakiness rate increased from 0.5% to 12% immediately following the config change. No other suites showed similar changes.","recommended_action":"Revert auth service timeout config change and re-evaluate test timeout thresholds."}],"alert_thresholds_breached":[{"threshold_name":"Overall flakiness rate increase > 50%","current_value":78.6,"threshold_value":50,"breach_severity":"critical","recommended_action":"Initiate flaky test quarantine process for auth suite."}],"top_flaky_tests":[{"test_name":"login_timeout_test","test_suite":"auth","flakiness_rate":12.0,"total_flaky_runs":18,"trend":"worsening"}],"limitations":["Infrastructure change log only covers auth-service. Other service changes may be unrecorded.","Sample size for auth suite is small (150 executions)."],"recommendations":[{"priority":1,"category":"infrastructure_change","description":"Revert auth service timeout config change and validate test stability.","affected_tests":["login_timeout_test"],"expected_impact":"Expected to resolve auth suite flakiness regression."},{"priority":2,"category":"quarantine","description":"Quarantine login_timeout_test until root cause is resolved.","affected_tests":["login_timeout_test"],"expected_impact":"Reduces CI noise and prevents defect masking."}]}

To adapt this template, replace the placeholders with your actual data. The [FLAKY_TEST_HISTORY_JSON] should be a JSON array of test execution records from your CI system or test management platform. The [INFRASTRUCTURE_CHANGE_LOG_JSON] is optional but strongly recommended for correlation analysis; source it from your deployment pipeline, configuration management system, or incident management tool. Adjust [ALERT_THRESHOLD] and [SIGNIFICANCE_LEVEL] based on your team's risk tolerance. The [MIN_SAMPLE_SIZE] constraint prevents the model from drawing conclusions from statistically meaningless data. Before wiring this into production, validate the output JSON against the schema using a JSON Schema validator. For high-severity regressions, route the output to a human reviewer before triggering automated quarantine actions.

Do not skip the infrastructure change log if you want correlation analysis. Without it, the model will only detect trends and regressions but cannot attribute them to specific changes. If your flaky test data does not include failure_reason or execution_duration_ms, remove those fields from the input schema description to avoid confusing the model. Always test this prompt with a known historical period where you already understand the ground truth—compare the model's output against your own analysis to calibrate trust before relying on it for alerting.

This prompt is designed to operate as a scheduled analysis step within a CI observability or test analytics platform, not as a one-off chat interaction. The primary integration point is a data pipeline that aggregates flaky test execution records from your test reporting system (e.g., JUnit XML, TestNG, Allure, or a vendor API) over a configurable lookback window. Before invoking the LLM, the application layer must pre-process raw test data into the structured [HISTORICAL_FLAKY_TEST_DATA] format the prompt expects: a time-series of test executions with test identifiers, outcomes, timestamps, environment metadata, and any known infrastructure change events. This pre-processing step is critical because the prompt's trend detection and correlation logic depends on clean, normalized input rather than raw log parsing.

The implementation harness should wrap the LLM call in a validation and retry loop. After receiving the model's output, parse the JSON response and validate it against the expected [OUTPUT_SCHEMA] before surfacing results. Key validation checks include: verifying that all trend periods have non-null direction and confidence fields, confirming that statistical_significance values are within 0.0-1.0 range, and ensuring that alert_thresholds contain actionable numeric thresholds rather than vague descriptions. If validation fails, retry with the same input and an additional [CONSTRAINTS] note describing the specific validation error. Limit retries to 2 attempts before falling back to a partial report with an analysis_incomplete flag. Log every invocation—including input hash, model version, output, validation results, and retry count—for auditability and prompt performance tracking over time.

For high-stakes decisions such as quarantining tests from the CI pipeline or rolling back a deployment based on flakiness trends, this prompt's output must pass through a human review gate. The harness should render the trend report in a review UI that highlights: periods where flakiness rate exceeded the recommended alert threshold, correlation scores between infrastructure changes and flakiness spikes, and any statistical significance indicators below 0.95. The reviewer can then approve, reject, or annotate findings before the report triggers automated actions. This human-in-the-loop step is essential because trend correlation does not prove causation, and false positives in flakiness alerts can erode trust in the test suite. Avoid wiring this prompt directly to automated quarantine or rollback actions without review.