Log Sampling Strategy Decision Prompt

This prompt is for SREs and platform engineers who need to design a log sampling strategy that balances cost control with the need to retain forensic evidence for debugging, security incidents, and compliance. The ideal user has a defined set of services, a known log volume problem, and specific error paths or security events that must never be sampled out. The prompt helps you produce a structured sampling strategy with rules, rates, and retention policies, rather than a one-size-fits-all percentage.

Use this prompt when you have telemetry data showing unsustainable log volume growth and you need to make intentional trade-offs about what to keep, what to sample, and what to drop. The prompt requires concrete inputs: a list of services, their current log volumes, known high-value log categories (error stacks, authentication events, data mutations), and any compliance or retention requirements. It is not designed for real-time metric generation or trace sampling, which require different statistical models like tail-based sampling or reservoir sampling. Do not use this prompt if you are designing a metrics pipeline, a distributed tracing strategy, or an alerting architecture—those are separate design problems with their own constraints.

Before running this prompt, gather your service inventory, volume estimates, and a list of 'never-sample' events. The prompt will ask you to define sampling rules per service or log category, specify retention windows, and document the rationale for each decision. The output should be a document you can review with your team and implement in your log aggregation pipeline. If your environment is regulated, plan for a human review step before the strategy is applied to production. After you have a strategy, the next step is to implement it in your observability pipeline and validate that critical events are still captured during a controlled rollout.

This prompt template is designed to be pasted directly into your AI harness. It accepts a description of your system's traffic profile, error characteristics, and business constraints, and returns a structured sampling strategy. The strategy includes specific rules, sampling rates, and retention policies. All dynamic inputs are represented as square-bracket placeholders, such as [SERVICE_PROFILE] and [COST_CONSTRAINTS]. Replace these with your concrete data before execution.

markdownCopy
You are a senior SRE architect designing a log sampling strategy for a production service. Your goal is to minimize log volume and cost while preserving the ability to debug failures, investigate security incidents, and meet compliance requirements.

## Input Data
- **Service Profile:** [SERVICE_PROFILE]
- **Traffic Pattern:** [TRAFFIC_PATTERN]
- **Error Characteristics:** [ERROR_CHARACTERISTICS]
- **Security Event Profile:** [SECURITY_EVENT_PROFILE]
- **Compliance Requirements:** [COMPLIANCE_REQUIREMENTS]
- **Cost Constraints:** [COST_CONSTRAINTS]
- **Current Log Volume (GB/day):** [CURRENT_LOG_VOLUME]

## Output Schema
Return a JSON object with the following structure:
{
  "strategy_name": "string",
  "overview": "string (brief summary of the approach)",
  "sampling_rules": [
    {
      "rule_name": "string",
      "selector": "string (e.g., log level, service, endpoint, error code)",
      "sampling_rate": "number (0.0 to 1.0, where 1.0 means keep all)",
      "rationale": "string (why this rate was chosen)"
    }
  ],
  "retention_policy": {
    "hot_storage_days": "number",
    "cold_storage_days": "number",
    "rationale": "string"
  },
  "forensic_safeguards": [
    "string (specific mechanism to ensure critical events are not lost, e.g., '100% sampling for 5xx errors')"
  ],
  "estimated_cost_savings": "string (qualitative or quantitative estimate)",
  "risks_and_mitigations": [
    {
      "risk": "string",
      "mitigation": "string"
    }
  ]
}

## Constraints
- [CONSTRAINTS]

## Evaluation Criteria
A valid strategy will be judged on these criteria:
1.  **Error Path Coverage:** Does it guarantee 100% capture of server-side errors (5xx)?
2.  **Security Event Coverage:** Does it guarantee 100% capture of defined security events (e.g., auth failures)?
3.  **Cost Reduction:** Does it project a meaningful reduction in log volume?
4.  **Debugability:** Does it preserve enough context (e.g., by sampling entire traces or request paths) to debug a sampled transaction?
5.  **Compliance:** Does the retention policy meet the stated requirements?

Generate the strategy.

To adapt this template, replace each bracketed placeholder with a concise, factual description. The [SERVICE_PROFILE] should describe the service's role and criticality. [CONSTRAINTS] is where you inject hard rules, such as 'Must not sample any logs from the payments service' or 'Use only infrastructure-level sampling agents.' After generating the output, validate the JSON structure and then apply the evaluation criteria manually or with a separate LLM-as-judge step to ensure the strategy is sound before implementation.

This prompt is designed to be called from an internal platform tool or a CI/CD pipeline stage that gates observability configuration changes. The application harness should collect the required inputs—service catalog metadata, traffic profiles, error budget status, and current telemetry costs—from your observability backend (e.g., Grafana, Datadog, Honeycomb) and your incident management system before assembling the prompt. Do not leave these fields to manual copy-paste; stale data produces sampling strategies that either overspend or drop forensic evidence. The harness should also inject a unique request_id and timestamp into the [CONTEXT] block so every generated strategy is traceable to the exact inputs that produced it.

After the model returns a sampling strategy, run a structured output validator that checks for required fields: sampling_rules (each with selector, rate, and rationale), retention_policy, error_path_coverage, and security_event_coverage. If the validator fails, retry once with the validation errors appended to the [CONSTRAINTS] block. If the second attempt also fails, route the request to a human review queue with the partial output and validation log attached. For model choice, use a model with strong JSON mode and long-context support (e.g., gpt-4o or claude-3-opus) because the prompt includes detailed service profiles and cost data that must not be truncated. Set temperature=0.2 to balance consistency with the nuanced trade-offs sampling decisions require.

Log every generation attempt—inputs, outputs, validation results, and the final approved strategy—to an audit table. This is critical because sampling decisions directly affect incident forensics; when a production issue is investigated six months later, the team must be able to explain why certain log streams were sampled at specific rates. Before the strategy is applied to production log pipelines, require a human approval step for any rule that samples error-level logs or security-relevant events at less than 100%. The approval UI should display the generated strategy alongside the input context, the model's rationale, and a diff against the current sampling configuration. Never auto-apply sampling changes without this gate.