Auto-Scaling Policy Design Prompt

This prompt is designed for Site Reliability Engineers (SREs) and cloud engineers who need to produce a rigorous, production-ready auto-scaling policy specification. The job-to-be-done is translating a service's traffic pattern, performance constraints, and cost tolerance into a concrete set of scale-out triggers, scale-in cooldowns, metric selections, and over-provisioning guardrails. The ideal user brings a deep understanding of the target workload's behavior, including its initialization latency, statefulness, and downstream dependency limits, and needs a structured output that can be directly reviewed and translated into infrastructure-as-code configurations like AWS Auto Scaling groups, Kubernetes HorizontalPodAutoscalers, or Terraform modules.

Use this prompt when you are designing a new scaling policy from scratch or auditing an existing one for oscillation risk and cold-start latency. It is most effective when you can provide concrete context: the target metric (CPU, memory, custom metrics like request queue depth), the expected traffic envelope (steady state, diurnal peaks, and known flash-mob events), the service's boot time, and any hard limits on maximum instance count driven by cost or quota. The prompt forces explicit reasoning about the interaction between scale-out aggressiveness and scale-in caution, which is the primary source of production instability. Do not use this prompt for simple static provisioning decisions, for services with entirely predictable batch workloads that don't require dynamic scaling, or as a substitute for load testing. The output is a design document, not a real-time controller; it must be tested against simulated traffic before production deployment.

After generating the policy specification, the next step is to validate it against historical metrics and failure injection scenarios. Pay particular attention to the prompt's output on 'oscillation risk'—the most common production failure mode is a policy that scales out too quickly in response to a transient spike and then scales in too quickly during the cooldown, creating a costly and destabilizing cycle. The guardrails section of the output should be directly translated into maximum and minimum instance counts in your orchestration system. If the prompt identifies a conflict between cost constraints and latency SLOs, resolve that conflict with human judgment before implementing the policy; the model can surface the trade-off but cannot make the business decision.

The following prompt template is designed to produce a complete auto-scaling policy specification from a description of your workload, infrastructure, and operational requirements. It forces the model to reason about scale-out triggers, scale-in cooldowns, metric selection, and over-provisioning guardrails before generating the final output. Use this template as the starting point for your own implementation, replacing each square-bracket placeholder with concrete values from your environment.

textCopy
You are an SRE architect specializing in cloud auto-scaling policy design. Your task is to produce a complete, implementation-ready auto-scaling policy specification based on the provided workload context.

## WORKLOAD CONTEXT
[WORKLOAD_DESCRIPTION]

## INFRASTRUCTURE DETAILS
- Compute platform: [COMPUTE_PLATFORM]
- Current instance/container specifications: [INSTANCE_SPECS]
- Orchestration layer: [ORCHESTRATION_TOOL]
- Target region(s): [REGIONS]

## CONSTRAINTS
- Maximum allowed instance count: [MAX_INSTANCES]
- Minimum required instance count: [MIN_INSTANCES]
- Cost budget constraints: [COST_CONSTRAINTS]
- Cold-start latency tolerance (seconds): [COLD_START_TOLERANCE]
- Regulatory or compliance requirements: [COMPLIANCE_REQUIREMENTS]

## AVAILABLE METRICS
[List the metrics available for scaling decisions, e.g., CPU utilization, request latency p95, queue depth, custom application metrics]
[METRICS_LIST]

## OUTPUT SCHEMA
Produce a JSON object with the following structure:
{
  "policy_name": "string",
  "version": "string",
  "scale_out": {
    "triggers": [
      {
        "metric": "string",
        "threshold": "number",
        "duration_seconds": "number",
        "cooldown_seconds": "number",
        "increment_type": "fixed_or_percent",
        "increment_value": "number"
      }
    ],
    "stabilization_window_seconds": "number"
  },
  "scale_in": {
    "triggers": [
      {
        "metric": "string",
        "threshold": "number",
        "duration_seconds": "number",
        "cooldown_seconds": "number",
        "decrement_type": "fixed_or_percent",
        "decrement_value": "number"
      }
    ],
    "stabilization_window_seconds": "number"
  },
  "over_provisioning_guardrails": {
    "max_headroom_percent": "number",
    "predictive_scaling_enabled": "boolean",
    "schedule_overrides": [
      {
        "cron_expression": "string",
        "min_instances": "number",
        "reason": "string"
      }
    ]
  },
  "oscillation_prevention": {
    "hysteresis_margin_percent": "number",
    "minimum_steady_state_duration_seconds": "number",
    "flapping_detection_window_seconds": "number",
    "max_scale_events_per_window": "number"
  },
  "failure_modes": [
    {
      "scenario": "string",
      "detection": "string",
      "mitigation": "string"
    }
  ],
  "assumptions": ["string"],
  "risks": ["string"]
}

## EVALUATION CRITERIA
Before finalizing the policy, verify:
1. Scale-out triggers activate before user-facing degradation occurs.
2. Scale-in cooldowns prevent premature termination during traffic dips.
3. Oscillation prevention mechanisms are sufficient for the workload's traffic pattern volatility.
4. Cold-start latency is accounted for in stabilization windows.
5. Over-provisioning guardrails prevent cost runaway without sacrificing availability.
6. At least two distinct failure modes are identified with detection and mitigation steps.

## INSTRUCTIONS
1. Analyze the workload description and identify the dominant scaling signals.
2. Select the most appropriate metrics from the available list; explain why any were excluded.
3. Design scale-out and scale-in triggers with specific thresholds justified by the workload context.
4. Configure cooldowns and stabilization windows that account for cold-start latency.
5. Add oscillation prevention parameters appropriate to the traffic pattern.
6. Identify at least two failure modes specific to this workload.
7. Output ONLY the JSON object. No explanatory text outside the JSON.

To adapt this template for your own use, replace each bracketed placeholder with concrete values from your environment. The [WORKLOAD_DESCRIPTION] should include traffic patterns, request characteristics, statefulness, and any known scaling anomalies. The [METRICS_LIST] should enumerate only metrics your observability stack actually exposes—do not invent metrics the system cannot measure. If your platform does not support predictive scaling or schedule overrides, remove those fields from the output schema rather than leaving them with placeholder values. For high-risk production workloads, add a human approval step before the generated policy is applied to any environment.

This prompt is designed to be integrated into a platform engineering or SRE workflow where infrastructure as code (IaC) and observability data are the primary inputs. The implementation harness should treat the LLM as a policy drafting engine, not an autonomous controller. The generated policy specification must pass through a validation layer that checks for structural completeness, metric existence, and threshold sanity before it can be applied to a Terraform module, CloudFormation stack, or Kubernetes HPA manifest. The harness should assemble the prompt context from live data sources: pull current CPU/memory utilization from CloudWatch or Prometheus, extract latency percentiles from your APM, and inject the target resource's constraints from your IaC repository. This ensures the prompt reasons over real data rather than hypotheticals.

The integration pipeline should follow a validate → simulate → approve → apply pattern. After the LLM returns the policy specification, a deterministic validator must check that every referenced metric (e.g., CPUUtilization, RequestCountPerTarget) exists in your monitoring system and that scale-in cooldowns are not shorter than your application's graceful shutdown period. Next, run the proposed policy through a lightweight simulation using historical traffic patterns to detect oscillation risk—if the scale-out threshold triggers a scale-in within the cooldown window more than 10% of the time in simulation, flag the policy for human review. The harness should log the full prompt, raw LLM response, validation results, and simulation outcome to an audit trail before any apply step. For model choice, prefer a model with strong structured output support (e.g., GPT-4o, Claude 3.5 Sonnet) and set response_format to json_schema using the output schema defined in this playbook. Implement a retry wrapper with up to 3 attempts if the output fails schema validation, appending the validation error to the retry prompt.

Do not wire this prompt directly to an auto-apply pipeline. The output is a design artifact that requires a human or an approval workflow to sign off before infrastructure changes are made. A dangerous failure mode is a policy that looks structurally valid but contains a scale-out threshold set to 10% CPU, causing constant flapping and cost overruns. Your harness should include a cost-impact estimator that projects the maximum instance count under the proposed policy against current traffic patterns and flags any projection exceeding a configurable budget threshold. For teams using OpenPolicyAgent or similar policy-as-code frameworks, the harness can translate the LLM output into a Rego rule that gates the Terraform apply, ensuring no scaling policy reaches production without passing the validation and simulation checks.