Repository navigation
[NVIDIA][AgentX] Retune GB300 DSV4.1-Flash Dynamo + SGLang HiCache disagg points on the 20261030 nightly / [NVIDIA][AgentX] 在 20261030 nightly 上重新调整 GB300 DSV4.1-Flash Dynamo + SGLang HiCache 分离式点位 - #4673
Open
weireweire wants to merge 9 commits into
Open
weireweire wants to merge 9 commits into
weireweire wants to merge 9 commits into
Conversation
Contributor
|
Thanks for the contribution!
中文感谢你的贡献！
|
weireweire
added a commit
that referenced
this pull request
Oct 30, 2026
Add 5P bs48 c3615, 6P bs64 c4370 and 7P bs96 c5540 with the decode cap at the P/D equilibrium batch, and 7P c5800 and 9P c7480 at cap 192 with total_requests decode balancing. Drop 8P c4096. Each point's bound recipe matches the measured run's resolved config apart from the unused Mooncake services.
…ys to the 20261030 nightly nightly-dev-cu13-20261030-302b78cc (index digest d4b7701d) bundles SGLang 52604e1d46, 181 commits after 586355a8, and Dynamo 1.7.0.dev20261029. Recipes are unchanged; the points are being re-measured on this image before the sweep.
In 3600 s runs on 586355a8, 7P BS 96 c5540 gave 1,031,199 tok/s/GPU at p90 interactivity 155.2 and 7P BS 192 c5800 gave 1,029,432 at 127.1.
weireweire
force-pushed
the
weireweire/dsv41flash-gb300-sgl-disagg-pareto
branch
from
November 2, 2026 02:39
8542d92 to
b8510e8
Compare
weireweire
changed the title
…t covers Root cause: #4679 added agg TP2/EP2 c16 back to fill the interactivity gap between agg TP4 c16 and disagg 5P c2884, at ~58k tok/s/GPU. Fix: add 3 x DEP2 prefill + DEP16 decode at c1024 (decode CUDA graph batch 20, max-running-requests 320, extra slots 30) and remove override_tp2_c16 and its search-space entry. Validation: 302b78cc fast run with this recipe: ~320k tok/s/GPU projected full at conv p90 288-294 (agg TP2/EP2 c16 published: 57,670 @ 292.4) and E2E-normalized p90 107.9 vs 91.4.
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
…tivity range Root cause: the disagg key carried neighbouring points that add little to the curve (5P c3615 next to 5P c2884, 9P batch 160 next to 9P batch 192, and the new 3P c1024 next to 3P c1228). Fix: drop override_5p1d_dep2_dep16_b48_c3615, override_9p1d_dep2_dep16_b160_c7480 and override_3p1d_dep2_dep16_b20_c1024 with their search-space entries, and rewrite this PR's changelog entry for the final point set. Validation: rule 18(g) on mean interactivity and tput per GPU with the 302b 3600 s fulls: published 9P batch 160 lies under the 7P c5540 - 9P batch 192 line (+0.6%), agg TP2/EP2 c16 under the agg TP4 c16 - 3P c1228 line, and 8P c4096 under the 5P - 6P line; 5P bs48 cost 0.6% of curve at mean 199.7.
weireweire
added
the
full-sweep-enabled
…-gb300-sgl-disagg-pareto # Conflicts: # inferencex-e2e/perf-changelog.yaml
weireweire
marked this pull request as ready for review
November 2, 2026 09:44
weireweire
requested review from
Ankur-singh,
RohitNagraj,
kedarpotdar-nv,
nvpohanh and
wzhao18
as code owners
November 2, 2026 09:44
Contributor
|
Claude finished @weireweire's task in 32s —— View job Review
LGTM - no blocking issues found What I checked
|
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
I reviewed this PR and found no bugs. nvidia-master.yaml and the gb300 srt-slurm recipes have designated CODEOWNERS, so one of them should still look over the retuned points.
What was reviewed:
- Every srt-recipe reference in the two master-config keys points to an override that exists. Prefill num-worker (6/7/9) matches the recipe workers, and the node counts (3/4/inherited 5) have room for those DEP2 workers.
- The removed overrides (override_tp2_c16, override_8p1d_dep2_dep16_c4096, b160_c7480) are no longer referenced anywhere. Image digests, the dynamo-router version and the recipe identity.frameworks pins agree across both keys.
- The perf-changelog block is appended only at the end, lists both config keys, and says the base extra-slots of 256 still applies, which matches the recipe.
Extended reasoning...
The change is benchmark config only: two GB300 DSV4.1-Flash Dynamo + SGLang srt-slurm variant recipes, their matching nvidia-master.yaml search-space rows and image/router pins, and one appended perf-changelog entry. It touches no security-sensitive code. The recipe and master config agree, and no dangling override references remain. I deferred instead of approving because .github/CODEOWNERS assigns nvidia-master.yaml and the gb300 srt-slurm recipe paths to specific NVIDIA owners. The tuning choices (decode batch caps, removed points, total_requests balancing) are also performance decisions those owners should confirm.
Contributor
|
View unofficial run (performance): https://inferencex.semianalysis.com/inference?unofficialRun=41418638862 View unofficial run (accuracy): https://inferencex.semianalysis.com/evaluation?unofficialRun=41418638862 |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Move the GB300 DeepSeek-V4.1-Flash Dynamo + SGLang agg and prefill-HiCache disagg keys to digest-pinned
lmsysorg/sglang:nightly-dev-cu13-20261030-302b78cc(SGLang0.0.0.dev1+g52604e1d4, Dynamo1.7.0.dev20261029), and keep one point per interactivity range.Based on main after #4595 and #4679. The commits change
disagg-dep16-hicache-variants.yaml,agg-variants.yaml, the two keys' search spaces and image pins, and add one changelog entry that lists both keys, so the whole Dynamo + SGLang curve comes from this PR's sweep.Disagg decode is always one DEP16 worker. Removed: agg TP2/EP2 c16 (from #4679), 8 × DEP2 c4096 and 9 × DEP2 BS 160 c7480.
Why these points:
total_requestsbalancing lets decode take every request prefill delivers.total_requestskeeps the per-rank running counts within 2-7 of each other (11-26 withtotal_tokens); the step is paced by the busiest rank.Removed published points (rule 18(g), mean interactivity and throughput per GPU; agg TP4 c16, 3 × DEP2 c1228 and the removed points as published, the other kept points from our 3600 s runs below):
Throughput gain at the same p90 interactivity (tok/s/GPU). Our points were measured on our GB300 cluster on
302b78ccwith the bound recipes of this PR and the AgentX client, not in this PR's sweep. The vLLM column linearly interpolates the currently published GB300 DeepSeek-V4.1-Flash Dynamo + vLLM and vLLM points on their Pareto frontier at our p90 interactivity. A blank cell means no gain.At p90 97.4, the vLLM frontier lies between 1,011,514 @ 75.5 and 993,521 @ 145.6, which gives 1,005,893. At p90 157.5 it lies between 993,521 @ 145.6 and 954,146 @ 196.5, which gives 984,316.
The 3 × DEP2 c1228 point keeps #4679's settings; on
302b78cca 1200 s run gave about 371,000 tok/s/GPU after the 1200 s → 3600 s factor, at p90 interactivity 281.2 and E2E-normalized p90 91.1, matching the published 383,313 @ 283.3 within the fast-run projection.Validation details
All runs used
nightly-dev-cu13-20261030-302b78ccthrough srt-slurm on GB300 with the AgentX client (reasoning_efforthigh, temperature 1) and the golden acceptance length 3.51. Each run's resolved recipe matches the recipe thatinfx generatebinds for this PR, apart from cluster NIC names, scheduler directives and local image/model packaging of the same digest.中文
将 GB300 DeepSeek-V4.1-Flash Dynamo + SGLang 的 agg 与 prefill-HiCache 分离式 key 换到固定 digest 的
nightly-dev-cu13-20261030-302b78cc（SGLang0.0.0.dev1+g52604e1d4、Dynamo1.7.0.dev20261029），每个交互性区间只保留一个点。基于 #4595 与 #4679 合入后的 main。 本 PR 新增一条同时列出两个 key 的 changelog，让整条 Dynamo + SGLang 曲线都来自本 PR 的 sweep。
total_requestsDP 均衡）。decode 都是一个 DEP16 worker。AI model disclosure
aws/anthropic/eccn-claude-opus-5-5) planned the point set from the P/D rate analysis, ran the cluster experiments through delegated agents on the same model, wrote the recipe overrides, master config, changelog entry and this description, and ran independent reviews of the diff through separate delegated agents on the same model.