streamable-http client: startup failures are reported as WorkerQuitReason::Fatal with the real error replaced by a synthetic "Transport channel closed"

streamable-http client: startup failures are reported as `WorkerQuitReason::Fatal` with the real error replaced by a synthetic "Transport channel closed"

Describe the bug

While integrating rmcp 3.9.0 into mcpstore (a Rust MCP
client/manager), we traced a production incident — connect storms leaving client instances stuck
without terminal state — through the streamable-http client lifecycle, and found that the
transport worker's terminal-error contract has two structural gaps.

1. Ordinary startup failures are laundered into `WorkerQuitReason::Fatal`

In StreamableHttpClientWorker::run (rmcp 3.9.0, src/transport/streamable_http_client.rs,
the Err(err) arm of the startup POST), the most ordinary lifecycle event there is — server
unreachable, TLS failure, even a 401 OAuth challenge — is reported as:

Err(err) => {
    let msg = format!("{:?}", err);
    let _ = responder.send(Err(err));
    return Err(WorkerQuitReason::fatal(
        StreamableHttpError::TransportChannelClosed, // synthetic; real error demoted to `msg`
        msg,
    ));
}

The real error is discarded from the typed position and survives only as a Debug-formatted
string in the quit context. src/transport/worker.rs:249 then logs it at ERROR:

ERROR rmcp::transport::worker: worker quit with fatal: Transport channel closed, when Client(reqwest::Error { kind: Request, url: "https://mcp.linear.app/mcp", source: hyper_util::client::legacy::Error(Connect, TimedOut) })

A few days of integration testing on our side produced 500+ of these ERROR lines — every
single one a routine ConnectionRefused or Connect TimedOut. Two costs:

Operators and alerting cannot distinguish "peer unreachable" (expected, retryable) from a
genuine internal fault. The sibling problem was already fixed on the server side in LocalSessionWorker idle keep-alive timeout emits tracing::error! for normal zombie-session reaping #817
(normal zombie reaping logged as tracing::error!).
The typed error contract is broken: anything observing the worker's quit reason
programmatically sees TransportChannelClosed, never the cause.

Note that a 401 challenge — the normal trigger of an OAuth login flow — is laundered the same
way (when AuthRequired(AuthRequiredError { ... })).

2. The terminal error has exactly one delivery attempt and no retention

Transport::send() on the worker adapter awaits the worker's ack, and the ack is only sent
after the HTTP POST completes (responder.send(Ok(())) in the Ok arm of the same match).
On failure the error is delivered via let _ = responder.send(Err(err)) — fire-and-forget, one
attempt, to whoever happens to still be awaiting that particular send() future. If the awaiting
side is gone at that moment — its own timeout fired, a select! arm won, Auto mode's discover
timeout (the #1142 fix) moved on to legacy_startup — the cause is silently dropped, and that
consumer only ever sees its own timeout error. Nobody else can recover it: it is not retained in
the quit reason (see 1) or anywhere else.

Context: why we hit this

During a 26-service connect storm, instances whose handshake failed against unreachable
endpoints got stuck in starting for hours with no terminal failure recorded. The root cause on
our side was unbounded cleanup around a failed handshake (we have since budgeted the whole
attempt), but tracing the incident through rmcp's lifecycle paths showed that termination and
error-delivery guarantees are emergent — dependent on every waiter happening to be parked on a
channel the dying worker owns — rather than specified:

RunningService::close() / cancel() await the service task with no bound; the bounded
variant (close_with_timeout) exists but is the alternative, not the default.
Auto mode bounds the discover phase (10s, the ClientLifecycleMode::Auto hangs on a server that silently ignores server/discover #1142 fix) but the legacy_startup fallback
that follows is unbounded (serve_client_with_ct_inner, src/service/client.rs).
The project keeps patching individual unbounded paths — SESSION_CLEANUP_TIMEOUT,
close_with_timeout, StreamableHttp client: transparent session re-init (HTTP 404) orphans in-flight requests, hanging call_tool forever #912, streamable-http-client: RequestHandle::cancel hangs before response stream starts #1193 — which suggests boundedness is treated site-by-site rather
than as an invariant.

Expected behavior

The worker's terminal reason should tell the truth and terminate deterministically:

Preserve the typed error. A failed startup POST should quit as an expected-failure reason
(e.g. WorkerQuitReason::StartupFailed(E), or at minimum Fatal carrying E) and log at an
appropriate level. The synthetic TransportChannelClosed should be reserved for actual
channel-closure anomalies.
Make worker termination a first-class lifecycle signal. The service layer should observe
worker death deterministically (e.g. selecting on the worker's join handle or a watch of
worker state) instead of relying on channel-drop cascades, and RunningService should expose
connection state (cause included) so a consumer can observe "dead: X" without calling into the
service.
Bounded by construction. Every await whose completion depends on a network peer —
handshake stages, close/cleanup — should carry a default budget: close() itself with
close_with_timeout semantics, and an overall handshake budget on
serve_client_with_lifecycle, so the unbounded variants become the opt-in rather than the
default.

We are happy to follow up with a PR — mcpstore wraps every rmcp client in its own supervision,
so we have a strong interest in seeing 1 and 3 land.

Version

rmcp 3.9.0 (crates.io release, 2026-11-05); code references are from that release.
Client: StreamableHttpClientTransport (reqwest) + serve_client_with_lifecycle with
ClientLifecycleMode::Auto / Discover.
macOS (not load-bearing; the paths involved are OS-independent).

@jokemanfire

@jokemanfire is on triage rotation this week and is the reviewer for this one.

Codex labeled this bug / P2 (T-transport, T-service). Please confirm or correct those labels.

_{Automated triage. Labels are a first pass and the assigned maintainer has the final say.}

Checked both structural gaps against main (f2b0ddf) rather than the 3.9.0 release. Both are still present, and the second one is a layer worse than the report describes.

1. The typed error exists and is dropped at the quit-reason boundary — crates/rmcp/src/transport/streamable_http_client.rs:1203-1210:

Err(err) => {
    let msg = format!("{:?}", err);
    let _ = responder.send(Err(err));
    return Err(WorkerQuitReason::fatal(
        StreamableHttpError::TransportChannelClosed,
        msg,
    ));
}

One nuance worth keeping: the typed value is not lost before delivery — it goes to whichever Transport::send() future is parked on that ack. It is the terminal reason that replaces it, and that reason is the only channel a later observer has. err is already StreamableHttpError<C::Error> (post_message_with_max_sse_event_size returns it; Client(E) is generic at :223), so carrying it is a one-line change, not a plumbing job.

The log side is crates/rmcp/src/transport/worker.rs:243-250, where every WorkerQuitReason::Fatal maps to tracing::error! with no classification — the same shape as #817 on the server side.

2. RunningService cannot report the cause at all — QuitReason is { Cancelled, Closed, JoinError } (crates/rmcp/src/service.rs:1224-1228) and the service loop turns "transport.receive() returned None" into QuitReason::Closed (service.rs:1445-1451). So waiting() / close() never surface the worker's Fatal { error, context }, even though they own the join handle. Item 2 of the report therefore needs a QuitReason carrier, not just a worker-side variant. Both enums are #[non_exhaustive] (worker.rs:20, service.rs:1223), so adding variants is additive.

3. Boundedness, confirmed — only the discover probe is wrapped (crates/rmcp/src/service/client.rs:868-870); the legacy_startup fallback is unwrapped on both the Legacy outcome (:888-895) and the timeout arm (:903). close() (service.rs:1148) awaits the join handle unbounded while close_with_timeout (:1168) exists alongside it.

Option set, as I read the code (maintainers should pick, I am not proposing a shape as settled):

A. Minimal, no new variants. fatal(err, "process initialize response") in the arm above, plus classify the log in worker.rs (transport / HTTP-status class → warn, everything else stays error). Removes the laundering and the 500-ERROR-lines symptom; does nothing for item 2 beyond what the parked waiter already gets.
B. Additive terminal reason. WorkerQuitReason::StartupFailed(E) and a carrier in QuitReason. The second is the awkward one: QuitReason is not generic over E, so either the cause is stored as a message plus kind (cheap, loses the typed contract) or the enum grows a generic parameter (breaking, touches every waiting() caller). That choice is above my pay grade as a drive-by contributor.
C. Retain the terminal reason. Store it on the worker/service (a OnceLock or watch of "dead, cause = X") so a consumer whose own timeout already fired can still read it. This is the part the starting-for-hours symptom actually needs; A and B only help the observer that was still parked when the worker died.

Before suggesting A I checked whether TransportChannelClosed is load-bearing: only err_closed() (streamable_http_client.rs:1140) and a test construct it, and no caller matches on it, so replacing it in this arm does not look like it changes retry behaviour.

What I am not sure about: whether TransportChannelClosed was chosen deliberately so that send() failures look like closure to the service loop (I do not see a dependency — the service loop only matches None from receive()), and whether the log split should land before or with the variant. If A is the shape you want, I can send it as a small PR against transport/{streamable_http_client,worker}.rs; B and C should go to someone who owns the service API.

worker: retain terminal cause, StartupFailed variant, quit_status (mo…

transport: add provided Transport::quit_status (modelcontextprotocol#…

service: surface QuitReason::Transport with terminal cause (modelcont…

streamable-http: report startup failure with real typed error (modelc…

worker: report cause only for abnormal termination; regression test (m…

streamable-http: keep real error text on ack-delivered path (modelcon…

B and C are implemented in #1444.

A few notes on the choices, following the triage:

B: WorkerQuitReason::StartupFailed { error, context } carries the real typed error, and startup failures log at warn instead of error. Both the startup-POST Err arm and expect_initialized use it. On the TransportChannelClosed question: it is still constructed only by err_closed() and tests, and nothing matches on it, so replacing it in this arm does not change retry behaviour — confirmed while implementing.
C: the worker's terminal reason is retained in a watch slot on WorkerTransport, written once when the task quits, and exposed via a new provided Transport::quit_status() (default None, so existing transports are unaffected). The service loop surfaces it as a new additive QuitReason::Transport(DynamicTransportError).

On the "awkward" part you flagged — QuitReason is not generic over E: rather than growing a generic parameter (breaking every waiting() caller) or degrading the cause to a string + kind, the cause crosses the service boundary as the existing DynamicTransportError (Box<dyn Error + Send + Sync> inside). QuitReason stays non-generic and the change is additive.

One design point worth your eyes: the typed error is not Clone, so it lives in exactly one place — the in-flight send() ack, or, when that receiver has already gone away (its own timeout fired), the retained quit reason. When the ack succeeds, the real error's text is still captured into the quit reason's context so the cause is never masked. This keeps the typed contract on the path that has a live receiver, and guarantees a real cause on the path that previously lost it entirely.

Normal termination (cancel, clean transport close, handler termination, idle timeout) is deliberately not reported as QuitReason::Transport — only abnormal termination carries a cause, so a clean peer disconnect is not misreported as a fault.

Regression test included: a worker whose startup fails retains its cause in quit_status even with no close() pending at the time of death. 370 lib tests pass, all-features check clean, clippy/fmt clean.

worker: typed RetainedQuitReason wrapper + downcast test (modelcontex…

client: surface retained startup cause in pre-init expect_next_message (

streamable-http: startup failure retains real typed error (modelconte…

worker: publish quit_status before input channel closes; StartupFaile…

client: pre-init returns TransportError with owned retained cause (mo…

streamable-http: share typed startup error via Arc in ack + quit reas…

Navigation Menu

Repository navigation

streamable-http client: startup failures are reported as WorkerQuitReason::Fatal with the real error replaced by a synthetic "Transport channel closed" #1441

streamable-http client: startup failures are reported as `WorkerQuitReason::Fatal` with the real error replaced by a synthetic "Transport channel closed"

Describe the bug

1. Ordinary startup failures are laundered into `WorkerQuitReason::Fatal`

2. The terminal error has exactly one delivery attempt and no retention

Context: why we hit this

Expected behavior

Version

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

streamable-http client: startup failures are reported as WorkerQuitReason::Fatal with the real error replaced by a synthetic "Transport channel closed" #1441

Description

streamable-http client: startup failures are reported as WorkerQuitReason::Fatal with the real error replaced by a synthetic "Transport channel closed"

Describe the bug

1. Ordinary startup failures are laundered into WorkerQuitReason::Fatal

2. The terminal error has exactly one delivery attempt and no retention

Context: why we hit this

Expected behavior

Version

Activity

github-actions commented on Nov 10, 2026

jokemanfire commented on Nov 10, 2026

ip2a commented on Nov 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

streamable-http client: startup failures are reported as `WorkerQuitReason::Fatal` with the real error replaced by a synthetic "Transport channel closed"

1. Ordinary startup failures are laundered into `WorkerQuitReason::Fatal`