Skip to content

Allow retries with type: dynamo router - #4474

Merged
un-def merged 1 commit into
masterfrom
pr_allow_dynamo_retry
Nov 10, 2026
Merged

un-def merged 1 commit into
masterfrom
pr_allow_dynamo_retry

Conversation

@un-def

@un-def un-def commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Retry was rejected because workers got the router's internal IP once, via DSTACK_ROUTER_INTERNAL_IP, and a retried router could come back at a different address. Since #4402, Dynamo services run etcd in a dedicated group referenced via replica IP interpolation (#4177), and #4473 removed the variable, so nothing depends on the router's address anymore.

Retry was rejected because workers got the router's internal IP once,
via DSTACK_ROUTER_INTERNAL_IP, and a retried router could come back at
a different address. Since #4402, Dynamo services run etcd in a
dedicated group referenced via replica IP interpolation (#4177), and
#4473 removed the variable, so nothing depends on the router's address
anymore.
@un-def
un-def merged commit 73a6dc6 into master Nov 10, 2026
26 checks passed
@un-def
un-def deleted the pr_allow_dynamo_retry branch November 10, 2026 10:03
un-def added a commit that referenced this pull request Nov 12, 2026
In Dynamo 1.5.1, a vLLM prefill worker that loses etcd hangs on exit:
its teardown never shuts down the EngineCore child process, so Python
blocks joining it (ai-dynamo/dynamo#16401, fixed in 1.6.0). On 1.6.2,
the frontend and both workers exit once their etcd leases expire.

Since #4474, services with the `dynamo` router accept `retry`. Without
it, a failed replica, such as etcd, fails the whole run.

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant