Skip to content

[Kubernetes] Support NVIDIA MIG slices - #4389

Merged
peterschmidt85 merged 12 commits into
masterfrom
pr_kubernetes_mig
Nov 5, 2026
Merged

peterschmidt85 merged 12 commits into
masterfrom
pr_kubernetes_mig

Conversation

@peterschmidt85

@peterschmidt85 peterschmidt85 commented Oct 20, 2026 •

Copy link
Copy Markdown
Contributor

The kubernetes backend now supports NVIDIA MIG slices configured with the GPU Operator's MIG Manager (the single or mixed strategy). dstack treats each slice as a GPU with the slice's memory, e.g., a 1g.10gb slice of an A100 is A100:10GB. To run on slices, specify the GPU memory:

resources:
  gpu: A100:10GB

If the GPU memory isn't specified or has no maximum (e.g., gpu: A100 or gpu: 24GB..), the run gets whole GPUs only. With --full-offers, dstack offer also lists matching slices:

$ dstack offer --gpu A100 --full-offers

 #  BACKEND                                        RESOURCES                                           INSTANCE TYPE                     PRICE
 1  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:40GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0
 2  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:20GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0
 3  kubernetes (gke_dstack_us-central1-a_mig-e2e)  cpu=x86:23 mem=156.9GB disk=97.4GB gpu=A100:20GB:1  gke-mig-e2e-a100x2-946ce248-zsmm  $0

With the mixed strategy, each MIG profile on a node is a separate offer, e.g., 3g.20gb and 4g.20gb above, and these offers share the node's CPU, memory, and disk. If a slice is busy, the run moves on to the cluster's other matching offers.

Clusters without MIG aren't affected. On clusters that already use the single strategy, slices used to be offered as whole GPUs (e.g., A100:80GB), so gpu: A100 runs could land on them. Now slices are offered with their own memory, and gpu: A100 runs get whole GPUs only. To run on slices, specify the GPU memory, e.g., gpu: A100:10GB.

Slices are matched by memory only, so a specific MIG profile (e.g., 3g.40gb vs 4g.40gb) can't be requested. dstack metrics doesn't show GPU metrics for slices yet.

Implementation details

  • Offers: slice offers carry KubernetesOfferBackendData in backend_data, with mig and, for the mixed strategy, the mig_resource that the pod requests instead of nvidia.com/gpu. Offers without MIG keep empty backend_data, so their pod specs are unchanged.
  • Offer filtering: the backend's offer modifier drops slice offers unless the run's GPU memory has a maximum, and drops mixed strategy offers for runs without GPUs, since they duplicate the node's main offer.
  • Retries: _KubernetesSkipOfferCache replaces RegionalSkipOfferCache and adds the MIG resource to the key. The resource is None for offers without MIG, so their retries still skip the whole cluster.
  • Performance: building and filtering offers for 1,000 nodes takes the same time as on master.
  • Compatibility: no configuration, model, or API changes, so older CLIs work with the new server.

Design doc: MIG support for the Kubernetes backend

🤖 Generated with Claude Code

@peterschmidt85
peterschmidt85 requested a review from un-def October 29, 2026 08:44
claude and others added 10 commits November 4, 2026 11:39
Offer MIG slices created by the NVIDIA GPU Operator (single and mixed
strategies) as GPUs of the parent model with the profile's memory,
e.g. a 1g.10gb slice of an A100 is A100:10GB.

- Single strategy: a node whose nvidia.com/gpu.product is
  <model>-MIG-<profile> is offered as that many slices.
- Mixed strategy: each nvidia.com/mig-<profile> resource on a node is
  offered separately next to the node's whole GPUs; the pod requests that
  resource and its node affinity uses the parent product label.
- Runs get slices only when the GPU memory spec has an upper bound, so
  gpu: A100 never lands on a slice; --full-offers shows everything.
- After a failed scheduling attempt, skip only offers for the same GPU
  resource in the cluster, so a 3g.20gb failure doesn't rule out the
  cluster's 4g.20gb slices or whole GPUs.

Tested on GKE 1.36 with GPU Operator v26.7.2 (driver 595) and A100 40GB
nodes across single, mixed and MIG-only layouts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@peterschmidt85
peterschmidt85 marked this pull request as ready for review November 4, 2026 14:26
@peterschmidt85 peterschmidt85 changed the title Support NVIDIA MIG slices in the Kubernetes backend [Kubernetes] Support NVIDIA MIG slices Nov 4, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@peterschmidt85

Copy link
Copy Markdown
Contributor Author

Clarified in 80f7ebc: the docs now say that --full-offers shows the node's full resources, and that with MIG it also lists matching slices regardless of the GPU memory rule.

# Conflicts:
#	src/dstack/_internal/core/backends/kubernetes/compute.py
#	src/dstack/_internal/core/backends/kubernetes/resources.py
@peterschmidt85
peterschmidt85 merged commit cf5eb83 into master Nov 5, 2026
26 checks passed
@peterschmidt85
peterschmidt85 deleted the pr_kubernetes_mig branch November 5, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants