Spatial Partitioning of GPUs using Nvidia MIG

In the prior blogs, we discussed why GPUs are managed differently in Kubernetes, how the GPU Operator helps streamline management and various strategies to share GPUs on Kubernetes. In 2020, Nvidia introduced Multi-Instance GPU (MIG) that takes GPU sharing to a different level.

In this blog, we will start by reviewing some common industry use cases where MIG is used and then dive deeper into how MIG is configured and used.

Common Use Cases

Let's look at some common use cases and applications for GPU Sharing using Nvidia MIG.

1. GPU Cloud

GPU Cloud providers can use MIG to provide GPU resources to multiple customers simultaneously. Instead of allocating an entire GPU to a single user, they can partition GPUs and offer this to their users that only require a slice of the full GPU. This reduces costs for users running smaller-scale workloads and maximizes the provider’s GPU utilization.

2. Inference for Multiple Applications

Inference workloads generally do not require the full power of a GPU. With MIG, multiple models or applications can be run concurrently on the same GPU. For example,

• A video streaming platform could run multiple real-time tasks (e.g. object detection, facial recognition, and background noise suppression, on different MIG instances).
• A hedge fund could use MIG to run multiple financial models concurrently (one model to analyze market sentiment and the second one to predict stock prices using historical data).

3. Multi-Tenancy for Enterprises

A large research lab can use MIG in a multi-tenant Kubernetes environment, where multiple data scientists are running various machine learning workloads. Each scientist’s pod in the Kubernetes cluster can be assigned a different MIG instance, ensuring fair and isolated access to GPU resources without needing to allocate an entire GPU to each workload.

4. Healthcare

Many hospitals or research institutions run ML models for medical image analysis (e.g. detect tumors from MRI scans, analyze CT images for abnormalities). MIG enables running different models for multiple patients on the same GPU simultaneously, speeding up the diagnosis. Each model can be provided a dedicated portion of the GPU, providing faster results.

Concepts

MIG enables a single physical GPU to be split into multiple independent instances. Each instance operates with its own memory, cache and compute cores, effectively allowing different workloads to run concurrently on separate GPU partitions. With MIG, GPUs such as NVIDIA A100 etc, can be securely partitioned up to seven separate GPU instances.

A MIG device consists of the 3-tuple (Top level GPU + A GPU Instance created inside that GPU + A Compute Instance created inside that GPU Instance).

• A GPU instance is the first level of partitioning on a MIG capable GPU. It creates a wrapper around a fixed set of memory slices, a fixed set of compute slices, and a fixed set of "other" GPU engines (such as jpeg encoders, dma engines, etc.)

• A Compute Instance is the second level of partitioning on a MIG capable GPU. It takes some subset of the compute slices inside a GPU Instance and bundles them together. Each Compute Instance has dedicated access to the compute slices it has bundled together, but shared access to its wrapping GPU Instance's memory and "other" GPU Engines.

This gives us a MIG device naming convention ("[GPU Instance slice count]g.[total memory]gb") that refers to a MIG device. For example, 3g.20gb

Partitioning Strategy

Three strategies are supported with MIG allowing for flexible use of the GPUs.

None

Use this strategy when MIG mode is not enabled on your GPU. This is the default mode and no changes are required to existing pod manifests.

Single

With this strategy, a node exposes a single type of MIG device across all its GPUs. The logic behind this strategy is to avoid the fragmentation of jobs across different nodes. Pods will need to use the nodeSelector to specify the exact model of GPU or MIG device that is required. In single mode, the GPU is partitioned, and users should specify the MIG profile being used.

Mixed

With the mixed strategy, a node can expose different MIG device types and GPUs. Applications need to request a specific MIG device by specifying both the number of compute instances and the total amount of memory provided by the device type (e.g. 2g.10GB). To request a MIG device from the mixed strategy, users must change their YAML specs to request the corresponding resource type.

MIG Profiles

MIG offers predefined profiles that specify how much of the GPU’s resources (e.g., compute units, memory) are allocated to each instance. MIG does not allow GPU instances to be created with an arbitrary number of GPU slices. On a given GPU, you can create multiple GPU instances from a mix and match of the supported profiles. Some profiles may allocate more GPU cores or memory, while others might create smaller, more numerous instances. MIG Profiles follow a naming convention (M:xg:ygb) where

• xg refers to the number of GPU compute units (SMs) and
• ygb refers to the memory allocated in gigabytes.

Shown below is a table of available MIG profiles for a Nvidia A100 TensorCore GPU

Considerations

Listed below are considerations that users need to factor in when using MIG. These are valid at the time of writing this blog and may change as Nvidia updates MIG's capabilities.

1. Available only on specific GPU models like A30, A100 (PCIE/SXM4), H100 (PCIE/SXM5/GH200), and H200 (SXM5)
2. Designed primarily for computing tasks (i.e. no graphical applications)
3. Best suited for servers with GPUs of the same model.
4. Requires the latest version of the GPU drivers.
5. Reconfiguration requires unloading all computing tasks from memory and stopping applications.
6. GPU to GPU P2P (i.e. PCIe or NVLink) is not supported
7. Best suited for Linux

Summary

NVIDIA MIG (Multi-Instance GPU) allows partitioning a single GPU into multiple isolated instances, enabling better resource utilization by running different workloads concurrently on a single GPU. This improves efficiency by tailoring GPU resources to the specific needs of diverse applications, such as training, inference, and development. Shown below is a table that maps common MIG device types to types of workload.

In the next blog, we will discuss how organizations can provide their users (data scientists and ML engineers) self-service access to GPU-enabled workspaces from a shared pool of GPU resources.