Multi-Instance GPU (MIG)

MIG creates physically isolated GPU instances at the hardware level, not through software abstraction. Each instance receives dedicated streaming multiprocessors (SMs), L2 cache slices, and memory bandwidth. This prevents noisy-neighbor interference where one workload's memory access pattern degrades another's performance. Unlike software virtualization, MIG's isolation extends to error containment—a fault in one instance cannot propagate to others sharing the same physical die.

Each MIG instance provides deterministic throughput and latency by dedicating a fixed fraction of the GPU's resources. For edge inference workloads with strict latency budgets, this ensures predictable tail latency. Key characteristics:

• Dedicated memory bandwidth: No contention for DRAM access
• Fixed SM allocation: Compute cores are not time-sliced
• Isolated cache hierarchy: L2 cache partitions prevent eviction interference
• Predictable execution: Ideal for real-time inference serving with SLAs

MIG enables simultaneous execution of heterogeneous inference workloads on a single physical GPU. A single A100 or H100 can be partitioned to serve:

• A large language model on a 40GB instance
• A vision transformer on a 20GB instance
• Multiple lightweight classification models on 10GB instances

Each tenant operates independently with its own CUDA context, allowing different users, frameworks, and model architectures to coexist without resource contention.

GPU instances can be dynamically reconfigured without rebooting the host system. Supported instance profiles on the A100 include:

• 7x 10GB instances (1 SM each)
• 3x 20GB instances (2 SMs each)
• 1x 40GB instance (4 SMs)
• Mixed configurations combining different profile sizes

This flexibility allows operators to reshape GPU resources in response to changing workload demands, maximizing utilization across diurnal traffic patterns.

For edge deployments where physical space and power are constrained, MIG dramatically increases inference density. A single GPU can replace multiple discrete accelerators, reducing:

• Power consumption: One GPU vs. multiple lower-end devices
• Physical footprint: Critical for telco edge cabinets and MEC servers
• Management complexity: Single firmware, driver, and monitoring surface

This density is essential for Multi-access Edge Computing (MEC) deployments serving multiple tenants from a single hardware node.

Modern inference servers like Triton Inference Server natively support MIG by binding model instances to specific GPU partitions. Configuration involves:

• Specifying MIG device UUIDs in model configuration
• Assigning compute instance (CI) and GPU instance (GI) pairs
• Enabling concurrent model execution across isolated instances

This integration allows orchestration platforms to treat each MIG slice as an independent compute resource, enabling fine-grained scheduling and maximizing hardware utilization for dynamic batching workloads.

The foundational abstraction layer that Multi-Instance GPU builds upon. Unlike traditional virtualization with a full GPU driver stack, MIG operates at the hardware level to carve a single physical GPU into multiple isolated instances. Each instance has its own dedicated compute cores, memory, and memory bandwidth, ensuring deterministic Quality of Service without the overhead of a hypervisor. This contrasts with time-slicing approaches where processes compete for shared resources.

A model slicing strategy that directly leverages MIG's hardware guarantees. By aligning the computational requirements of an inference task with a specific GPU Instance, the system can provide strict latency and throughput guarantees. Key considerations include:

• Profile Selection: Matching the model's memory footprint to a specific MIG profile (e.g., 1g.5gb vs. 2g.10gb)
• Isolation: Preventing a noisy neighbor on one instance from degrading the performance of another
• Deterministic Execution: Ensuring that a critical inference task always has access to its allocated memory bandwidth

An execution model that distributes inference workloads across diverse processing units. Multi-Instance GPU adds a new dimension to this paradigm by allowing a single physical GPU to present itself as multiple logical accelerators with different capabilities. An orchestrator can treat each MIG instance as an independent compute resource alongside CPUs and NPUs, enabling fine-grained workload scheduling based on:

• Performance per Watt: Assigning smaller models to smaller instances
• Hardware Affinity: Pinning specific inference tasks to dedicated compute slices
• Resource Fragmentation: Maximizing utilization by filling available MIG slices

A server-side optimization that groups individual inference requests into optimal batch sizes. When combined with Multi-Instance GPU, dynamic batching operates within the isolated confines of a single GPU Instance. This provides a critical advantage: the batching logic for one model does not compete for memory or compute with another model's batching logic. Each instance can independently maximize its hardware utilization without causing unpredictable latency spikes for co-located services, a common failure mode in time-sliced GPU sharing.

A cross-platform inference accelerator that optimizes and executes machine learning models across diverse hardware backends. ONNX Runtime can leverage Multi-Instance GPU through its execution provider framework. By binding a session to a specific MIG device, developers can achieve:

• Hardware Isolation: Ensuring a model's execution context is confined to its allocated GPU Instance
• Cross-Platform Consistency: Using the same API to target MIG partitions, full GPUs, or CPUs
• Graph Optimization: Applying transformations that are aware of the memory and compute limits of the target MIG slice