Hardware Heterogeneity

Central Processing Units (CPUs) are versatile, sequential processors that handle control logic, data preprocessing, and hosting lighter-weight models. In a heterogeneous system, they are often used for:

• Orchestration and scheduling of requests across accelerators.
• Pre- and post-processing tasks (e.g., tokenization, detokenization).
• Serving smaller models where latency is less critical or cost-per-token must be minimized. Their strength lies in flexibility, but they are generally less efficient than specialized hardware for dense matrix operations common in neural network inference.

GPUs are massively parallel accelerators optimized for the matrix and tensor operations fundamental to deep learning. They are the workhorse for large model inference. Key characteristics include:

• High throughput for batched requests.
• Support for mixed-precision computation (FP16, BF16, INT8) to boost speed and reduce memory usage.
• Generational differences (e.g., NVIDIA's Ampere vs. Hopper) that significantly affect performance-per-dollar. Cost-aware scheduling must consider GPU memory capacity, generational efficiency, and the cost of keeping them powered versus using cheaper alternatives for suitable tasks.

Neural Processing Units (NPUs) or AI Accelerators (e.g., Google TPUs, AWS Inferentia, Groq LPUs) are application-specific integrated circuits (ASICs) designed from the ground up for neural network inference. They offer:

• Extreme latency and/or throughput for specific model architectures and data types.
• Higher performance-per-watt compared to general-purpose GPUs for targeted workloads.
• Potential vendor lock-in due to custom software stacks and compilation tools. Their efficiency makes them cost-effective for high-volume, predictable inference patterns but may lack the flexibility of GPUs for rapidly evolving model architectures.

The scheduler is the critical software brain of a heterogeneous system. It makes real-time decisions on where to place each inference request based on a multi-objective cost function. It evaluates:

• Hardware-specific cost-per-token (factoring in instance price, utilization, and energy).
• Model compatibility with available hardware (e.g., kernel support, quantization).
• Current load and queuing delays across all processors.
• Request priority and Service Level Objectives (SLOs). Advanced schedulers use reinforcement learning to continuously optimize routing decisions, minimizing total cost of ownership while meeting latency guarantees.

A unified software layer (e.g., via runtimes like ONNX Runtime, TensorFlow Serving, or Triton Inference Server) provides a common interface for models to execute across different hardware backends. This component is essential for operationalizing heterogeneity. It handles:

• Model compilation and optimization for each target processor (e.g., converting to TensorRT for NVIDIA GPUs, compiling for AWS Neuron for Inferentia).
• Providing a consistent API for client applications, abstracting the underlying hardware complexity.
• Managing model versions and configurations across the disparate hardware pool. Without this layer, managing deployments and cost-aware routing across heterogeneous hardware becomes prohibitively complex.

This component provides the observability needed for financial governance. It collects fine-grained metrics to attribute costs accurately and inform scheduling decisions. It tracks:

• Resource utilization (GPU/CPU/NPU usage, memory consumption) per model and request.
• Actual inference latency and throughput on each hardware type.
• Direct cloud costs or energy consumption per hardware instance.
• Generates cost attribution reports for teams, projects, or users. This data feeds the cost-aware scheduler and provides CTOs and engineering managers with the visibility required for budgeting, chargeback models, and validating the ROI of heterogeneous infrastructure.

A common scenario where an organization operates a mix of GPU architectures (e.g., NVIDIA's A100, H100, and L40S) within the same cluster. Cost-aware schedulers must decide whether to route a latency-sensitive request to a premium H100 or a batch job to a more cost-effective A100. This requires understanding the performance-per-dollar and performance-per-watt of each chip generation for specific model types.

For small language models (SLMs) or highly optimized tasks, a modern CPU (e.g., AWS Graviton, Intel Sapphire Rapids) can be more cost-efficient than a GPU. Heterogeneous systems use intelligent routing to send these workloads to CPU-only instances, freeing expensive accelerators for larger models. This is critical for high-throughput, low-latency services where keeping a model resident on a GPU is wasteful.

Cloud providers deploy proprietary AI accelerators like AWS Inferentia or Google Cloud TPUs alongside traditional GPUs. These Application-Specific Integrated Circuits (ASICs) offer superior performance and cost efficiency for inference on supported model frameworks. A heterogeneous orchestrator must compile and route models to the optimal silicon, managing separate software stacks and kernel libraries for each accelerator type.

Heterogeneity extends geographically. A system may deploy:

• Edge NPUs (e.g., in smartphones, IoT gateways) for real-time, privacy-sensitive inference.
• Regional data centers with mid-tier GPUs for aggregation.
• Central cloud with high-end accelerators for complex batch processing. The orchestrator decides where to execute based on data gravity, latency SLOs, and transfer costs, creating a cost-optimal compute continuum.

A cost-optimized cluster blends expensive on-demand instances with deeply discounted but interruptible spot (AWS) or preemptible (GCP) instances. The orchestrator must place fault-tolerant, checkpointable batch inference jobs on spot instances and stateful, latency-critical services on stable ones. This requires dynamic workload migration and checkpointing strategies to handle instance revocation.

Advanced systems leverage heterogeneity within a single request. Techniques like vLLM's PagedAttention allow the KV Cache to be partially stored in cheaper, abundant CPU RAM, while computation occurs on the GPU. This drastically increases the effective batch size a single GPU can handle by overcoming GPU memory limits, optimizing aggregate throughput and cost-per-token.

The core software component that manages model deployment and request routing in a heterogeneous environment. It makes real-time decisions on workload placement, selecting the most cost-effective hardware (e.g., an older GPU for a latency-tolerant task, a new NPU for a critical path) based on dynamic performance profiles and pricing. Key functions include:

• Service Discovery: Maintaining a registry of available hardware and loaded models.
• Cost-Aware Scheduling: Using a scoring function that evaluates latency, throughput, and dollar cost per token.
• Health Monitoring: Evicting or rescheduling workloads from failing or degraded nodes.

The fundamental unit of financial measurement for LLM inference, calculated as the expense to generate a single output token. In a heterogeneous cluster, this cost varies dramatically by hardware type. For example:

• A high-end H100 GPU may have a lower cost-per-token for large batches due to superior throughput.
• An AWS Inferentia2 chip may offer a better cost-per-token for specific model architectures it's optimized for.
• A CPU instance might have the worst cost-per-token for generative tasks but be optimal for small, frequent classification models. Schedulers use real-time cost-per-token estimates to route requests, making it the primary metric for hardware selection.

The practice of selecting cloud compute instances with the precise combination of vCPUs, GPU memory, and accelerator type needed for a specific model and traffic pattern. In heterogeneous environments, this extends beyond single-instance choice to fleet composition. Strategies include:

• Profiling: Benchmarking a model across different instance types (e.g., g5.xlarge vs. p4d.24xlarge) to build a performance/cost matrix.
• Mixed Fleets: Deploying a combination of instance families (e.g., some with A10G GPUs for medium workloads, some with T4 GPUs for light workloads) to match demand granularly.
• Avoiding Overprovisioning: Preventing the costly mistake of using a high-memory instance for a model that fits in a much smaller memory footprint.

The central engineering decision process when allocating workloads in a heterogeneous system. Every hardware choice involves a balance between inference speed (latency/throughput) and financial expense. The tradeoff curve is not linear; moving a workload from a CPU to a mid-tier GPU yields a massive performance gain for a modest cost increase, while moving from a high-end to a cutting-edge GPU may offer minor gains at extreme cost. The orchestrator's policy defines the acceptable operating point on this curve for different request classes (e.g., user-facing vs. batch processing).

A deployment strategy that distributes model serving across compute resources from multiple cloud providers (e.g., AWS, Google Cloud, Azure, CoreWeave). This is a strategic extension of hardware heterogeneity, introducing provider-level diversity to achieve:

• Cost Optimization: Leveraging spot instances and price differences across providers for the same hardware class.
• Resilience: Avoiding regional or provider-wide outages.
• Vendor Leverage: Mitigating vendor lock-in by maintaining operational capability on multiple platforms. It requires an orchestrator that can manage credentials, networking, and consistent deployment across different cloud APIs and service paradigms.

The automated process of adding or removing compute instances from a serving fleet in response to traffic changes. In a heterogeneous context, autoscaling policies must decide not just how many, but what type of instances to launch. Advanced implementations use:

• Predictive Scaling: Based on workload prediction to provision slower-to-initialize hardware (e.g., GPU instances) before a forecasted spike.
• Cost-Aware Scaling: Evaluating the current spot instance market and on-demand prices across instance families to select the cheapest compatible hardware for the anticipated load.
• Granular Scaling Groups: Maintaining separate scaling groups for different hardware types, allowing the fleet composition to adapt elastically.