Skip to content

fix(transport): retain terminal cause for startup failures (#1441) - #1444

Open
ip2a wants to merge 29 commits into
modelcontextprotocol:mainfrom
ip2a:fix/1441-terminal-cause
Open

ip2a wants to merge 29 commits into
modelcontextprotocol:mainfrom
ip2a:fix/1441-terminal-cause

Conversation

@ip2a

@ip2a ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor

What

Fixes #1441 — a streamable-http startup handshake failure was reported as WorkerQuitReason::Fatal with the real typed error replaced by a synthetic TransportChannelClosed, and the terminal reason had no retained carrier, so an observer whose own timeout already fired lost the cause entirely.

Implements options B and C from @jokemanfire's triage comment on #1441.

B — typed error in the terminal reason

  • Add WorkerQuitReason::StartupFailed { error, context } (additive; the enum is #[non_exhaustive]), carrying the real typed error instead of a synthetic placeholder.
  • Startup failures now log at warn (they are ordinary lifecycle events — peer unreachable, TLS failure, auth challenge — not internal faults), addressing the 500+ spurious ERROR lines.
  • streamable_http_client.rs: both the startup-POST Err arm and expect_initialized now produce StartupFailed with the real error. When the in-flight send() receiver has already gone away (its own timeout fired), the typed error is retained in the quit reason; when the ack succeeds, the real error's text is still captured into the quit reason's context so the cause is never masked.

C — retain the terminal reason for late observers

  • WorkerTransport now retains the worker's terminal reason in a watch slot, written exactly once when the worker task quits, so the reason survives the single fire-and-forget ack delivery window.
  • New provided Transport::quit_status() (default None — stdio/io/ws/custom transports are unaffected) returns the cause as an owned DynamicTransportError.
  • QuitReason gains an additive Transport(DynamicTransportError) variant; the service loop reads the retained status after transport.close() so waiting() / close() callers learn why the connection ended rather than an undifferentiated Closed.
  • Normal termination (cancel, clean transport close, handler termination, idle timeout) is not reported as Transport — only abnormal termination carries a cause.

Notes for reviewers

  • StartupFailed's error stays the typed E (per the triage discussion), not a generic parameter on QuitReason — the cause crosses the service API boundary as DynamicTransportError, which keeps QuitReason non-generic and non-breaking. The typed error itself is not Clone, so it lives in exactly one place: the in-flight ack, or the retained quit reason when the ack receiver is gone. This matches the report's "exactly one delivery attempt" concern by adding the retained carrier.
  • Added a regression test (startup_failure_is_retained_in_quit_status): a worker whose startup fails retains its cause in quit_status even with no close() pending at the time of death.

Verification

  • cargo test -p rmcp --features client,transport-streamable-http-client-reqwest --lib: 370 passed, 0 failed (incl. the new regression test)
  • cargo check -p rmcp --all-features: clean
  • cargo clippy: no new warnings
  • cargo fmt --check: clean

@ip2a
ip2a requested a review from a team as a code owner November 11, 2026 01:24
@github-actions github-actions Bot added T-core Core library changes T-transport Transport layer changes labels Nov 11, 2026
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 11, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ✅ Completed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-11-11T04:43:12.211302Z 9133157 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@github-actions

Copy link
Copy Markdown
Contributor

@jokemanfire is on triage rotation this week and is the reviewer for this one.

Labels come from the PR labeler. Please review the change.

Automated triage. Labels are a first pass and the assigned maintainer has the final say.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ce07d77f97

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service.rs Resolved
Comment thread crates/rmcp/src/transport/worker.rs Resolved
@github-actions github-actions Bot added the T-service Service layer changes label Nov 11, 2026
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Both review findings addressed in the latest push:

Surface retained status during initialization — expect_next_message now reads transport.quit_status() when the stream ends mid-handshake, so a startup failure's retained cause reaches ClientInitializeError::ConnectionClosed instead of a bare "connection closed". This covers the path where serve_inner is never created (init did not succeed).

Preserve the typed error in quit_status — quit_status no longer stringifies. The retained Arc<WorkerQuitReason<E>> is boxed into DynamicTransportError via a RetainedQuitReason<E> wrapper whose source() exposes the inner typed error. Callers can downcast to RetainedQuitReason<E> to recover the typed WorkerQuitReason<E> (and thus E), and source-only data such as an auth challenge survives the source chain (is_authorization_required keeps working). Regression test now asserts the downcast and the inner error kind.

370 lib tests pass, all-features check clean, clippy/fmt clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d944818fcc

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service/client.rs Resolved
Comment thread crates/rmcp/src/transport/worker.rs Resolved
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Second round addressed in the latest push:

Publish quit status before the receive channel closes — the worker task now holds a clone of to_handler_tx until after the watch write, so a concurrently polled receive() cannot observe EOF and snapshot quit_status() as None. EOF is now ordered after the terminal status is visible.

Return the retained startup failure as a transport error — expect_next_message now returns ClientInitializeError::TransportError carrying the owned DynamicTransportError cause (not a stringified ConnectionClosed), so the caller can inspect its source and downcast.

Keep the typed error until the receiver consumes it — the startup error is now shared via Arc on both sides: StartupFailed holds Arc<E>, and the ack carries the same value as StreamableHttpError::StartupShared (a dyn Error Arc). A receiver dropped after a successful queue no longer destroys the only copy — the typed error survives in the quit reason. The shared ack's #[source] keeps the source chain intact, so auth-challenge detection (is_authorization_required) still reaches the inner AuthRequired error.

370 lib tests pass (640 with auth), all-features check clean, clippy/fmt clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3e22abf26a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service/client.rs Resolved
@github-actions github-actions Bot added the T-test Testing related changes label Nov 11, 2026
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Third round addressed in the latest push:

Preserve startup error variants for live senders — the shared ack now uses SharedStartupError<E>, which retains the concrete Arc<StreamableHttpError<E>> and an erased handle to the same allocation for Error::source(). StreamableHttpError::startup_error() provides the equivalent typed extraction path: it transparently returns the original HttpStatus, AuthRequired, or other variant for inspection. auth_challenge() uses the same unwrapping, and the existing 429 regression now verifies status and Retry-After through this accessor. The typed value remains retained if the ack receiver is dropped.

Recover quit status from failed fallback sends — Auto fallback errors now recover quit_status() when the original startup failure closes the fallback request queue. Recovery also awaits transport closure when the status is not visible yet, ordering the read after the worker publishes its terminal state instead of reintroducing the EOF/status race. A regression test drives discovery timeout → fallback send failure and asserts the retained typed cause replaces the synthetic queue-closed error.

Validation: 794 all-feature lib tests pass, the 10 streamable HTTP status metadata tests pass, strict all-target Clippy and rustfmt are clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8a2d9c6ec3

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service/client.rs Resolved
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Fourth round addressed in the latest push:

Preserve the underlying startup error in Display — StartupShared now renders startup handshake failed: <original error>, and SharedStartupError delegates its own Display to the original typed error. Logs and enclosing ClientInitializeError retain actionable details such as HTTP 429; the status metadata test now asserts the underlying status is present in the rendered error.

Avoid closing every fallback transport to poll worker status — Transport now has a provided supports_quit_status() capability query (default false), implemented by WorkerTransport. Fallback recovery only awaits close() for worker-backed transports whose retained status is not visible yet. Transports using the default quit_status() keep their original error and are not closed or delayed. Added a regression test asserting both behaviors.

Validation: 795 all-feature lib tests pass, both fallback regression tests pass, the 10 HTTP status metadata tests pass, and strict all-target Clippy/rustfmt are clean.

@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 7cd491fc3b

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service/client.rs Resolved
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Fifth round addressed in the latest push:

Recover retained errors after legacy notification sends — fixed at the transport source rather than broadening client-side recovery. The legacy notifications/initialized POST and response processing are now treated as part of startup. On failure, the worker sends the original typed error through the notification responder as StartupShared and returns StartupFailed holding the same Arc, so both a live ordinary serve() caller and a late observer retain HTTP/TLS/auth metadata. No synthetic TransportChannelClosed is substituted.

Added an integration regression for plain ClientLifecycleMode::Initialize: initialize succeeds, the initialized notification receives 429, and the caller downcasts the transport error, reads HttpStatus { status: 429, retry_after: Some("17") } through startup_error(), and sees HTTP 429 in Display.

Validation: 795 all-feature lib tests, all 5 discover/startup integration tests, and all 10 HTTP status metadata tests pass. Strict Clippy passes for both all-target/all-feature and the enabled startup integration configuration; rustfmt is clean.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 9a8a93c455

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Comment thread crates/rmcp/src/service/client.rs Resolved
@ip2a

ip2a commented Nov 11, 2026

Copy link
Copy Markdown
Contributor Author

Sixth round addressed in the latest push:

Preserve live errors during fallback recovery — fallback recovery now replaces only the synthetic channel-closed error. Worker and Transport expose an additive is_closed_error() classifier (default false); WorkerTransport delegates to its worker, and both streamable HTTP worker implementations identify their synthetic closed variant. A real live StartupShared/HttpStatus error is returned unchanged and remains directly downcastable; only a channel-closed placeholder is recovered from quit_status(). The close-and-wait synchronization is likewise restricted to that placeholder path.

The initialized-notification 429 regression now runs in both plain Initialize and Auto legacy-fallback modes. The Auto case unwraps LegacyFallbackFailed, then successfully downcasts the fallback error to StreamableHttpClientTransport<reqwest::Client> and reads status 429 plus Retry-After: 17, proving retained-status recovery no longer launders a live typed error.

Validation: 795 all-feature lib tests, all 6 startup integration cases, and all 10 HTTP metadata tests pass. Strict Clippy passes for all targets/features and the enabled startup integration configuration; rustfmt is clean.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

T-core Core library changes T-service Service layer changes T-test Testing related changes T-transport Transport layer changes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

streamable-http client: startup failures are reported as WorkerQuitReason::Fatal with the real error replaced by a synthetic "Transport channel closed"

1 participant