Linting Rule Effectiveness Audit Prompt

Platform teams accumulate lint rules over years. Some rules catch critical bugs before they reach production. Others generate noise, slow down CI pipelines, and train developers to ignore warnings. This prompt audits a single lint rule against its violation history, false positive rate, and fix rate to produce a data-driven keep, reconfigure, or remove recommendation. The primary job-to-be-done is replacing gut-feel rule curation with evidence-based decisions that improve developer experience without sacrificing safety. The ideal user is a platform engineer, DevSecOps lead, or engineering manager running a quarterly tooling retrospective or preparing for a lint configuration overhaul.

Use this prompt when you have aggregated rule statistics available: violation count over time, false positive rate (percentage of violations developers marked as invalid or suppressed with justification), and fix rate (percentage of violations that resulted in a code change within a defined window). The prompt requires these three data points as minimum inputs. If you lack historical data on false positive rates or fix rates, run the False Positive Detection and Suppression Justification Prompt first to build the evidence base from raw violation logs and suppression comments. Do not use this prompt for brand-new rules with fewer than 30 days of violation history—the sample size is too small for meaningful analysis. Do not use it for rules that are legally or contractually required; compliance-mandated rules are non-negotiable regardless of developer friction.

The prompt produces a structured recommendation with an effectiveness score, a decision category (keep, reconfigure, remove), and a justification grounded in the provided statistics. It also flags data-quality concerns such as insufficient sample size or contradictory signals. Wire the output into your lint config review process as one input among several: combine it with developer survey feedback, incident postmortems that reference specific rules, and your team's explicit quality policies. The prompt is a decision-support tool, not an automated rule removal engine. Always require human review before disabling or reconfiguring a rule that affects production CI gates.

This template is designed to be executed once per linting rule under review. It expects structured input about the rule's violation history, false positive rate, and fix rate over a defined time window. The output is a scored recommendation that platform teams can use to decide whether to keep, reconfigure, or remove the rule from their CI pipeline. The prompt enforces data-driven justification and explicitly asks the model to flag when evidence is insufficient for a confident recommendation.

textCopy
You are an auditor evaluating the effectiveness of a single linting rule in a production codebase. Your goal is to produce a data-driven recommendation: keep, reconfigure, or remove the rule.

## INPUT DATA
- Rule ID: [RULE_ID]
- Rule Name: [RULE_NAME]
- Linter: [LINTER_NAME]
- Evaluation Window: [START_DATE] to [END_DATE]
- Total Violations Triggered: [TOTAL_VIOLATIONS]
- Confirmed False Positives: [FALSE_POSITIVES]
- Violations Fixed Within 30 Days: [FIXED_VIOLATIONS]
- Violations Suppressed with Justification: [SUPPRESSED_VIOLATIONS]
- Violations Still Open After 90 Days: [OPEN_VIOLATIONS]
- Rule Severity Level: [SEVERITY]
- Team Convention Alignment: [ALIGNMENT_DESCRIPTION]
- Additional Context: [ADDITIONAL_CONTEXT]

## OUTPUT SCHEMA
Return a JSON object with these fields:
- rule_id: string
- effectiveness_score: number (0.0 to 1.0, where 1.0 is highly effective)
- false_positive_rate: number (0.0 to 1.0)
- fix_rate: number (0.0 to 1.0, fixed violations / (total violations - false positives))
- recommendation: "keep" | "reconfigure" | "remove"
- confidence: "high" | "medium" | "low"
- justification: string (3-5 sentences explaining the recommendation with specific data points)
- suggested_reconfiguration: string | null (if reconfigure, describe the change; otherwise null)
- risks_of_removal: string (what could go wrong if this rule is removed)
- evidence_gaps: string[] (list any missing data that would improve confidence)

## SCORING GUIDELINES
- Effectiveness score should weigh fix rate more heavily than raw violation count.
- A rule with high false positive rate (>0.4) should lean toward reconfigure or remove unless the true positives catch critical issues.
- A rule with low fix rate (<0.3) and low severity should lean toward remove.
- A rule aligned with documented team conventions should be preserved even if metrics are moderate.
- If total violations are fewer than 10 in the evaluation window, set confidence to "low" and note insufficient data.

## CONSTRAINTS
- Do not recommend removal of security-critical rules (severity "critical" or "error") without a compensating control.
- If recommendation is "reconfigure," the suggested_reconfiguration field must be specific and actionable.
- Base all conclusions on the provided data. Do not invent violation counts or trends.
- If data is contradictory or incomplete, lower the confidence score and explain why in evidence_gaps.

Adapt this template by adjusting the scoring guidelines to match your team's risk tolerance. For example, teams with stricter quality gates might lower the acceptable false positive threshold from 0.4 to 0.2. The output schema is designed to feed directly into a dashboard or decision log, so keep the JSON structure stable across rule audits. Before running this prompt at scale, validate the output against a few manually reviewed rules to calibrate the scoring thresholds to your repository's actual patterns.

This prompt is designed to run as a batch audit job, not a real-time interactive query. The typical integration pattern is a scheduled pipeline (weekly or per-sprint) that fetches rule violation history from your linting database or CI artifact store, assembles the prompt context, calls the model, validates the output, and writes results to a dashboard or audit log. Because the prompt produces a keep/reconfigure/remove recommendation that may change team tooling, the output should always pass through a human review step before any linter configuration is modified in production.

Input assembly requires three data sources wired together before the prompt is called: (1) the rule definition and configuration from your linter config file, (2) violation history counts and trends from your CI or static analysis database (e.g., SonarQube, CodeClimate, SARIF archives), and (3) fix-rate and suppression-rate data from your version control history or issue tracker. If any of these sources is missing or incomplete, the prompt should receive an explicit [DATA_GAPS] note rather than silently producing a recommendation on partial data. Output validation should check that the response contains a parseable effectiveness score (0.0–1.0), a clear recommendation enum (keep, reconfigure, remove), and at least one data-backed justification. Use a schema validator or a lightweight LLM judge to confirm these fields exist and are well-formed before the result enters the review queue.

Retry logic should be minimal here. If validation fails, retry once with the validation errors appended as [CORRECTION_FEEDBACK]. If the second attempt also fails, flag the rule for manual audit rather than looping. Model choice favors models with strong structured output and reasoning capabilities (GPT-4o, Claude 3.5 Sonnet, or equivalent). Avoid smaller or faster models for this task because the prompt requires multi-factor trade-off reasoning across false positive rates, fix rates, and team impact—areas where weaker models tend to produce shallow or inconsistent recommendations. Logging should capture the rule ID, the effectiveness score, the recommendation, the model version, and the reviewer's final decision for auditability. Store these in a dedicated lint_rule_audit table or log stream, not buried in general application logs.

Human review workflow: route all remove recommendations and any reconfigure recommendations with an effectiveness score below 0.4 to a platform team member for approval. keep recommendations with scores above 0.7 can be auto-accepted if your team has established trust in the audit pipeline, but log the decision for retrospective review. Never auto-apply configuration changes from this prompt without human confirmation—removing a lint rule that catches rare but critical security patterns can introduce risk that the effectiveness score alone may not surface. Wire the review step into your existing ticketing system or a lightweight approval UI rather than building a custom review interface.