How to Build a Carbon-Aware AI Compute Orchestrator

These APIs provide the real-time data on the carbon emissions of electricity generation in a specific grid region. Your orchestrator uses this as the primary signal for decision-making.

• Electricity Maps offers historical, real-time, and forecasted carbon intensity data via a REST API.
• WattTime provides similar data with a focus on marginal emissions and automated generation control (AGC) signals.
• Integration involves polling or subscribing to these APIs to map your compute locations (e.g., us-west-2) to a grid region (e.g., CAISO).

The core mechanism for implementing carbon-aware logic is by extending or replacing the default Kubernetes scheduler. You create custom scheduling plugins that score nodes based on carbon intensity, not just resource availability.

• Kube-scheduler framework allows you to write custom Score and Filter plugins in Go.
• Karpenter, a node provisioning tool, can be extended with custom provisioner logic to launch nodes in low-carbon regions.
• The scheduler evaluates the carbon forecast for the expected job duration, shifting workloads to greener regions or times.

This is the core optimization strategy: delaying non-urgent compute to align with periods of high renewable energy availability (e.g., midday solar).

• Define workload classes: Label jobs as delay-tolerant (batch training, model fine-tuning) or latency-sensitive (real-time inference).
• Implement a priority queue: Delay-tolerant jobs are placed in a queue that is only drained when carbon intensity falls below a defined threshold.
• Use spot instances: Combine with cloud spot markets to run workloads when renewable supply is high and electricity prices are low.

Service Level Objectives (SLOs) define the operational guardrails for your carbon-aware system. They create a measurable contract between sustainability goals and performance.

• Carbon Efficiency SLO: "95% of delay-tolerant workloads shall run when grid carbon intensity is below 200 gCO₂eq/kWh."
• Performance Guardrail SLO: "No workload shall be delayed more than 6 hours from its submission time."
• These SLOs are monitored using metrics like carbon_intensity_at_execution and workload_delay_duration exported to Prometheus.

The end-to-end system architecture connects the data sources, decision engine, and execution plane.

• Data Ingestion Layer: Polls carbon APIs and stores time-series data.
• Decision Engine: A custom controller that evaluates pending workloads against carbon forecasts and SLOs.
• Execution Plane: The extended Kubernetes scheduler and Karpenter provisioners that enact the scheduling decisions.
• Observability: Dashboards showing carbon savings, job delays, and SLO compliance, often built with Grafana.