Repository navigation
[Kubernetes] Support NVIDIA MIG slices - #4389
Merged
Merged
Conversation
Offer MIG slices created by the NVIDIA GPU Operator (single and mixed strategies) as GPUs of the parent model with the profile's memory, e.g. a 1g.10gb slice of an A100 is A100:10GB. - Single strategy: a node whose nvidia.com/gpu.product is <model>-MIG-<profile> is offered as that many slices. - Mixed strategy: each nvidia.com/mig-<profile> resource on a node is offered separately next to the node's whole GPUs; the pod requests that resource and its node affinity uses the parent product label. - Runs get slices only when the GPU memory spec has an upper bound, so gpu: A100 never lands on a slice; --full-offers shows everything. - After a failed scheduling attempt, skip only offers for the same GPU resource in the cluster, so a 3g.20gb failure doesn't rule out the cluster's 4g.20gb slices or whole GPUs. Tested on GKE 1.36 with GPU Operator v26.7.2 (driver 595) and A100 40GB nodes across single, mixed and MIG-only layouts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EDgQBSHAm1rxrhSUXWziy3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
peterschmidt85
force-pushed
the
pr_kubernetes_mig
branch
from
November 4, 2026 14:26
8f5fea6 to
0f9a785
Compare
peterschmidt85
marked this pull request as ready for review
November 4, 2026 14:26
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
Author
|
Clarified in 80f7ebc: the docs now say that |
un-def
approved these changes
Nov 5, 2026
# Conflicts: # src/dstack/_internal/core/backends/kubernetes/compute.py # src/dstack/_internal/core/backends/kubernetes/resources.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
kubernetesbackend now supports NVIDIA MIG slices configured with the GPU Operator's MIG Manager (thesingleormixedstrategy).dstacktreats each slice as a GPU with the slice's memory, e.g., a1g.10gbslice of an A100 isA100:10GB. To run on slices, specify the GPU memory:If the GPU memory isn't specified or has no maximum (e.g.,
gpu: A100orgpu: 24GB..), the run gets whole GPUs only. With--full-offers,dstack offeralso lists matching slices:With the
mixedstrategy, each MIG profile on a node is a separate offer, e.g.,3g.20gband4g.20gbabove, and these offers share the node's CPU, memory, and disk. If a slice is busy, the run moves on to the cluster's other matching offers.Clusters without MIG aren't affected. On clusters that already use the
singlestrategy, slices used to be offered as whole GPUs (e.g.,A100:80GB), sogpu: A100runs could land on them. Now slices are offered with their own memory, andgpu: A100runs get whole GPUs only. To run on slices, specify the GPU memory, e.g.,gpu: A100:10GB.Slices are matched by memory only, so a specific MIG profile (e.g.,
3g.40gbvs4g.40gb) can't be requested.dstack metricsdoesn't show GPU metrics for slices yet.Implementation details
KubernetesOfferBackendDatainbackend_data, withmigand, for themixedstrategy, themig_resourcethat the pod requests instead ofnvidia.com/gpu. Offers without MIG keep emptybackend_data, so their pod specs are unchanged.mixedstrategy offers for runs without GPUs, since they duplicate the node's main offer._KubernetesSkipOfferCachereplacesRegionalSkipOfferCacheand adds the MIG resource to the key. The resource isNonefor offers without MIG, so their retries still skip the whole cluster.Design doc: MIG support for the Kubernetes backend
🤖 Generated with Claude Code