Power Budgeting

Thermal Design Power (TDP) is the maximum sustained power, in watts, that a chip's cooling system is designed to dissipate under a defined worst-case workload. It is a critical budget ceiling.

• Serves as a power cap: The NPU's average power consumption must not exceed TDP to prevent thermal runaway.
• Defines cooling requirements: Dictates the necessary heat sink, fan, or liquid cooling solution.
• Not peak power: Short-term power spikes (PL2/PL4 states) can exceed TDP briefly but are averaged over a time window.

The Power Delivery Network (PDN)—comprising voltage regulators, package traces, and on-chip power grids—imposes hard electrical constraints on current delivery and voltage stability.

• Current carrying capacity: Metal traces and solder bumps have finite IR drop limits; exceeding them causes voltage sag and timing failures.
• Voltage regulator response: The Voltage Regulator Module (VRM) must respond to transient load steps (di/dt) within nanoseconds to prevent crashes.
• Decoupling capacitance: On-package and on-die capacitors are budgeted to supply instantaneous current, limiting the rate of power draw changes.

Total chip power is the sum of dynamic power (from transistor switching) and leakage power (static current). Budgeting requires managing their inverse relationship.

• Dynamic Power (P_dyn): Scales with activity (α), capacitance (C), voltage squared (V²), and frequency (f). Lowering V/f reduces P_dyn but increases task latency.
• Leakage Power (P_leak): Increases exponentially with lower transistor threshold voltage (Vt) and higher temperature. Aggressive voltage scaling to save dynamic power can increase leakage.
• The optimization target: Find the voltage/frequency operating point that minimizes total power (P_dyn + P_leak) for a given performance target.

The physical package and cooling solution limit heat dissipation, creating a thermal budget that directly maps to the power budget.

• Junction Temperature (Tj): The die's hottest spot must stay below the semiconductor's maximum rating (e.g., 125°C). Thermal throttling activates if exceeded.
• Thermal Resistance (θJA): The total resistance from junction to ambient air (°C/W). A lower θJA allows a higher power budget for the same Tj.
• Hotspot-driven budgeting: Power cannot be uniformly distributed; localized power density (W/mm²) on compute arrays creates hotspots that dictate the global limit.

Power budgeting is not merely about staying under a cap but maximizing computational efficiency within that cap, measured as Performance per Watt.

• Primary efficiency metric: Often measured in inferences per second per watt (inf/sec/W) or TOPS/W (Tera Operations Per Second per Watt).
• Drives architectural choices: Influences decisions on dataflow, precision (INT8 vs. FP16), and sparsity utilization to maximize throughput within the wattage envelope.
• The Pareto frontier: The budget defines the optimal trade-off curve between absolute performance (TOPS) and efficiency (TOPS/W).

Static budgets are enforced and optimized at runtime by hardware and firmware controllers that react to dynamic workloads.

• Running Average Power Limit (RAPL): A hardware interface that enforces power caps over a moving time window, allowing brief bursts.
• Dynamic Thermal Management (DTM): Monitors on-die thermal sensors and triggers Dynamic Voltage and Frequency Scaling (DVFS) or workload migration to cool hotspots.
• Power-Aware Scheduling: The OS or runtime assigns tasks to NPU cores based on their current P-states and thermal headroom to stay within the system-wide budget.

Thermal Design Power (TDP) is a chip manufacturer's specification, expressed in watts, that represents the maximum amount of heat a processor is expected to generate under its maximum theoretical workload. It is the primary input for designing a cooling system and serves as the baseline for a system's power budget. For NPUs, TDP defines the sustained thermal envelope within which the accelerator must operate.

• Not Peak Power: TDP is typically lower than absolute peak power, which can be reached in short bursts.
• Cooling Target: System designers select heat sinks and fans capable of dissipating at least the TDP value.
• Budget Anchor: In a multi-component system, the TDP of the CPU, GPU, and NPU are summed to define the total platform power budget.

Dynamic Voltage and Frequency Scaling (DVFS) is the primary runtime technique for enforcing a power budget. It dynamically adjusts a processor's operating voltage and clock frequency in response to real-time workload demands and thermal conditions.

• Power Control Knob: Reducing frequency (f) and voltage (V) provides a cubic reduction in dynamic power (P ∝ C * V² * f).
• OS-Controlled: Managed by the operating system governor, which moves the core between Performance States (P-States).
• Budget Enforcement: When power or temperature limits are approached, DVFS is used to throttle performance to stay within the allocated budget.

Power gating is a circuit-level technique that completely shuts off power to an idle functional block using header or footer switches. This eliminates both dynamic power and leakage power (static power) for that block.

• Ultimate Savings: Provides the highest power savings for unused circuitry, essential for meeting low idle power budgets.
• Wake-up Latency: Turning a gated block back on incurs a time penalty due to power rail stabilization.
• Fine-Grained Control: Modern NPUs use hierarchical power gating, from entire cores down to individual execution units and memory banks.

Running Average Power Limit (RAPL) is a hardware interface (primarily on Intel platforms) that allows software to monitor and enforce power consumption limits for specific domains (e.g., CPU package, DRAM) over a configurable time window.

• Hardware Enforcement: Limits are enforced in hardware, making them reliable and low-overhead.
• Budget Partitioning: Enables sophisticated power budgeting by allocating different limits to the CPU, GPU, and NPU within a shared thermal envelope.
• Telemetry: Provides accurate, real-time power readings, which are critical for validating power budget models and adaptive control systems.

Performance per watt is the fundamental efficiency metric for evaluating systems under a power budget. It measures the useful computational work (e.g., inferences per second, FLOPS) delivered per unit of electrical power consumed (watt).

• Key Comparison Metric: Used to compare different NPU architectures, process nodes, and algorithm implementations.
• Optimization Target: The goal of power budgeting and associated techniques is to maximize performance per watt within the system's constraints.
• Trade-off Analysis: Informs decisions on how to allocate the fixed power budget—for example, running fewer cores at a higher frequency versus more cores at a lower frequency.

The Power Delivery Network (PDN) is the physical infrastructure—including voltage regulators, motherboard planes, package interconnects, and on-chip power grids—that delivers clean, stable power from the wall outlet to every transistor on the NPU.

• Budget Constraint: The PDN has a maximum current delivery capability. A power budget must respect this limit to avoid catastrophic voltage droop (IR drop).
• Integrity is Key: Power integrity analysis ensures the voltage remains within specification during rapid current transients caused by NPU workload changes.
• Design Complexity: High-performance NPUs require sophisticated, low-impedance PDNs with many layers of decoupling capacitors to meet aggressive power budgets.