Observability Policy Health Check Prompt

Use this prompt when your team needs to audit the health of an AI system's observability policies. It is designed for AI governance teams, trust and safety engineers, and platform architects who must verify that behavioral logging, violation detection, and audit trail mechanisms are functioning as intended. This prompt produces a structured health assessment that measures logging coverage gaps, detection accuracy rates, false-positive patterns, and early indicators of policy drift. Run this prompt during scheduled governance reviews, after system prompt changes, or when audit trail completeness is questioned.

This prompt is a diagnostic tool for existing policy effectiveness, not a replacement for runtime monitoring infrastructure. It expects you to provide production trace samples, current system instructions, and your defined observability policies as input. The output is a structured report that identifies specific gaps—such as unlogged tool calls, inconsistent refusal documentation, or degrading policy adherence over long conversations—and scores their severity. Wire this into a scheduled review pipeline where outputs are compared against previous runs to detect negative trends before they become compliance failures.

Do not use this prompt for real-time violation detection or for authoring new observability policies. It will not catch violations as they happen, nor will it design logging schemas from scratch. Avoid running it on synthetic or toy data; the assessment is only as useful as the production traces you provide. If your traces are incomplete or unrepresentative, the health check will produce a false sense of security. Always pair the output with a human review step, especially when severity scores indicate high-risk gaps that could affect audit readiness or regulatory posture.

This prompt template is designed to be copied directly into your AI governance workflow. It instructs the model to act as an observability policy auditor, comparing your defined logging requirements against actual production traces to identify coverage gaps, detection accuracy issues, and policy drift indicators. The template uses square-bracket placeholders that you must replace with your organization's specific observability policy document, sampled trace data, and review thresholds before execution.

textCopy
You are an observability policy auditor conducting a periodic health check of an AI system's behavioral logging and audit trail coverage.

Your task is to compare the defined observability policy against actual production traces and produce a structured health assessment.

## OBSERVABILITY POLICY
[OBSERVABILITY_POLICY_DOCUMENT]

## PRODUCTION TRACE SAMPLES
[TRACE_SAMPLES]

## REVIEW PARAMETERS
- Review period: [REVIEW_PERIOD_START] to [REVIEW_PERIOD_END]
- Minimum logging coverage threshold: [COVERAGE_THRESHOLD_PERCENT]
- False-positive rate tolerance: [FALSE_POSITIVE_TOLERANCE_PERCENT]
- Drift detection sensitivity: [DRIFT_SENSITIVITY_LEVEL]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "health_assessment": {
    "overall_status": "healthy|degraded|critical",
    "summary": "One-paragraph executive summary of findings",
    "review_period": {
      "start": "ISO date",
      "end": "ISO date"
    }
  },
  "logging_coverage": {
    "score_percentage": number,
    "status": "pass|warn|fail",
    "covered_event_types": ["list of event types with adequate logging"],
    "gap_event_types": [
      {
        "event_type": "string",
        "policy_requirement": "string",
        "observed_coverage": "string",
        "severity": "high|medium|low",
        "remediation": "string"
      }
    ]
  },
  "detection_accuracy": {
    "false_positive_rate_percentage": number,
    "false_negative_indicators": ["list of suspected missed detections"],
    "status": "pass|warn|fail"
  },
  "policy_drift_indicators": [
    {
      "indicator": "string",
      "trend": "increasing|decreasing|stable",
      "severity": "high|medium|low",
      "evidence": "string referencing specific trace patterns"
    }
  ],
  "recommendations": [
    {
      "priority": "critical|high|medium|low",
      "finding": "string",
      "action": "string",
      "owner_suggestion": "string"
    }
  ]
}

## CONSTRAINTS
- Only flag gaps where you have clear evidence from the provided trace samples.
- Do not fabricate trace data or invent policy violations not present in the samples.
- If trace samples are insufficient to assess a policy requirement, mark it as "insufficient_data" rather than pass or fail.
- Cite specific trace IDs or timestamps when identifying drift indicators.
- Distinguish between systemic gaps (policy requirement never logged) and intermittent gaps (policy requirement sometimes missed).

## RISK_LEVEL
[RISK_LEVEL]

If RISK_LEVEL is "high", flag any coverage gap or drift indicator at "medium" severity or above as requiring immediate human review before the assessment is considered final.

After pasting this template, replace each square-bracket placeholder with your actual data. The [OBSERVABILITY_POLICY_DOCUMENT] should contain the full text of your logging policy, including required event types, fields, and retention rules. The [TRACE_SAMPLES] should include a representative sample of production traces spanning the review period — ensure you include both normal interactions and edge cases. Set [RISK_LEVEL] to "high" for compliance-critical systems where false negatives carry regulatory consequences, or "standard" for internal tooling reviews. Run this prompt with a model that supports structured JSON output and validate the response against the schema before acting on recommendations.

The Observability Policy Health Check Prompt is not a one-off analysis tool; it is designed to be embedded into a recurring governance cadence. The primary integration point is a scheduled audit pipeline that runs the prompt against a batch of production trace samples, policy definitions, and prior health check reports. The output should be treated as a structured audit artifact, not a conversational response. This means the implementation harness must enforce strict JSON schema validation on every run, log the raw prompt, the model response, and the validation result, and route failures to a human review queue before the health check is considered complete.

To wire this into an automated pipeline, construct the prompt request by assembling the required inputs: [OBSERVABILITY_POLICY_DOCUMENT], [PRODUCTION_TRACE_SAMPLE], [PRIOR_HEALTH_CHECK_REPORT], and [EVALUATION_CRITERIA]. The trace sample should be a representative slice of recent production interactions, including tool calls, refusals, guardrail activations, and multi-turn conversations. Use a model with strong JSON mode and low temperature (0.1–0.2) to maximize structural reliability. After receiving the response, validate it against the expected [OUTPUT_SCHEMA]—which includes fields like logging_coverage_score, detection_accuracy, false_positive_rate, policy_drift_indicators, and coverage_gaps. If validation fails, retry once with the validation error message appended to the prompt as a correction hint. If the retry also fails, log the failure and escalate to a human reviewer with the raw response and validation context attached.

For high-stakes governance environments, do not treat this prompt's output as authoritative without human sign-off. The health check report should be stored in an audit evidence repository alongside the prompt version, model version, trace sample metadata, and validation result. Implement a trend analysis harness that compares the current health check against prior reports to detect degradation over time—this is where the policy_drift_indicators field becomes actionable. If drift exceeds a configurable threshold, trigger an alert to the AI governance team. Avoid running this prompt on live production traffic; it is a batch analysis tool. Finally, version-lock the prompt template and the output schema together so that any change to one forces a review of the other, preventing silent schema mismatches in your audit pipeline.