AWS SageMaker Splits GPUs Into Up to 7 Partitions via MIG

The smallest unit of GPU you could rent on a SageMaker endpoint used to be one whole accelerator. That changed on November 6, when AWS published a post from Shantanu Tripathi, Apoorva Chandra and Dmitry Soldatkin introducing GPU partitioning on real-time endpoints via NVIDIA Multi-Instance GPU. Each partition receives a fixed share of the GPU's compute units and a dedicated block of its memory, with isolation enforced in hardware.

The partition ceilings vary by chip: up to seven on the A100 (40 GB) in P4d instances and the H100 (80 GB) in P5, up to four on the RTX PRO 6000 Blackwell (96 GB) in G7e, and up to two on the RTX PRO 4500 Blackwell (32 GB) in G7. AWS writes that 'every model keeps its own partition and stays loaded, with no load penalty and no eviction,' and that in their testing, carving accelerators added 'roughly 17 seconds to provisioning.'

Salesforce helped validate the capability. 'MIG would allow us to consolidate workloads safely, significantly improve GPU utilization, reduce inference costs, and maintain predictable performance and strong isolation,' said Richard Chang, Software Engineering Architect at Salesforce, in the post.

The fine print narrows the audience. 'Single-model endpoints, serverless endpoints, and heterogeneous instance pools cannot be partitioned.' Layouts are set at endpoint creation. Models that need more than one GPU still take whole devices. The concrete workload AWS lands on is an 8B model at FP16 that 'comfortably serves an 8,000-token context on a 40 GB partition,' alongside fine-tuned variants, embedding models, rerankers, and classifiers. The release drops into a crowded quarter for inference plumbing: our inference tracker has logged 131 stories in the past 90 days.