Python: Isolate declarative MCP client sessions by workflow

docs/decisions/0046-python-declarative-request-scope-recovery.md

@@ -0,0 +1,189 @@
+---
+status: proposed
+contact: baywet
+date: 2026-10-15
+deciders: eavanvalkenburg, westey-m
+---
+
+# Preserve declarative request origins with opt-in legacy recovery
+
+## Context and Problem Statement
+
+A declarative workflow can retain requests from several fresh runs. Resuming a
+question, tool approval, or agent external-input request must restore its
+originating workflow scope rather than inherit the most recent run's MCP session.
+Older checkpoints do not retain that origin, so the framework cannot infer which
+session should be resumed.
+
+## Decision Drivers
+
+- All declarative pause types need the same origin and response-batch rules.
+- Existing checkpoints need an explicit recovery path without guessing an origin.
+- Repeated restoration must select the same migrated scope without sharing a newer run's session.
+
+## Considered Options
+
+- Use the current workflow scope for requests without an origin. This needs no
+  host changes and retains apparent continuity, but can attach an older request
+  to an unrelated fresh run.
+- Reject all unknown origins and require restarting the workflow. This avoids
+  guessing, but prevents hosts from recovering otherwise usable pending requests.
+- Automatically assign isolated scopes. This permits recovery without host
+  changes, but silently loses the previous MCP session's server-side state.
+- Require explicit host opt-in to assign stable, isolated scopes. This makes the
+  continuity trade-off visible and permits recovery, but requires host changes
+  and separate resumes for legacy requests.
+
+## Decision Outcome
+
+Choose explicit host opt-in through
+`WorkflowFactory(migrate_legacy_request_scopes=True)`, defaulting to `False`:
+
+- Questions/external input, function approvals, MCP approvals, and agent external
+  input retain their originating scope in the request. Response handlers restore
+  it before continuing. Agent requests also retain their own loop configuration.
+- Factory-built workflows validate all submitted request origins before consuming
+  responses. Mixed-scope batches are rejected without changing request metadata.
+- Checkpoint restores validate ordinary queued response origins before restoring
+  shared or executor state, including embedded child checkpoints. Submitted and
+  ordinary queued responses share batch validation before pending requests are consumed.
+  Wrapped child replies validate selected routes at delivery, after executor restoration.
+- Nested restoration preflights a detached copy of the complete embedded tree
+  with each child's own policy before applying live state or queues. Migration
+  metadata is confined to that candidate; a rejected child leaves earlier
+  siblings, parent executors, and the supplied checkpoint unchanged.
+- Without opt-in, a missing origin produces an actionable recovery error.
+- With opt-in, a missing origin becomes
+  `legacy-request:<framework request ID>`. The distinct namespace never selects
+  the active run's scope; preserving the request ID makes migration stable across
+  workers and repeated restoration. Legacy requests must be resumed separately.
+- Existing valid origins are unchanged. Invalid stored origins are not migrated.
+  Migration does not change header verification or authorize an unverifiable
+  MCP header context.
+- Newly emitted requests and subsequent checkpoints retain the migrated scope.
+  The old MCP session and its server-side history cannot be reconstructed.
+  Hosts requiring that continuity must restart rather than opt in.
+- Legacy agent loop configuration remains available from its old checkpoint
+  slot; new requests own their configuration instead of replacing shared state.
+
+Checkpoint contents and framework request IDs remain trusted host-owned data.
+Migration does not reconstruct state already overwritten in a legacy checkpoint.
+This is a checkpoint-compatibility change with an explicit recovery path, not
+transparent restoration of an old session.
+
+## Nested entry checkpoint replay
+
+Selecting a random child scope only when the child first executes is insufficient:
+the parent's entry checkpoint predates that selection. Rebuilding the graph and
+replaying that checkpoint would select a different child scope. Pre-seeding every
+descendant's mutable state before the parent snapshot would instead couple parent
+checkpoint creation to child initialization.
+
+Select a parent scope before its entry checkpoint and derive each child scope from
+that scope and the binding ID. Agent turns select a stable scope for their session;
+direct fresh runs select new scopes. Declarative response handlers restore the
+originating parent scope before further nested invocation. Child scope selection
+occurs only after the child accepts a fresh run, not before checkpoint validation
+or on response-only continuations.
+
+Declarative executors manually composed with Core `WorkflowBuilder` read the
+active Core scope instead of reusing a stale declarative cache. Factory
+initialization still selects its existing declarative scopes before the entry
+checkpoint; standalone state helpers retain their previous initialization behavior.
+Restored request scopes take precedence over inherited metadata from later runs.
+
+Older parent checkpoints without this scope cannot reconstruct a child's original
+scope if that child had not started. Such parents receive a new isolated scope
+with a warning; subsequent checkpoints retain it. Existing request-origin
+validation and explicit legacy-request migration rules are unchanged.
+
+## Nested response boundaries
+
+Restoring only a child's scope leaves its ancestors on a later fresh run's scope.
+If runs A and B pause in the same child and B is answered before A, a downstream
+binding could therefore select B's MCP session while processing A's continuation.
+
+Retain the originating parent run context for each forwarded child request in
+framework-owned parent shared state. Unlike transient wrapper fields, this state
+is checkpointed with its owning workflow, including at every embedded boundary.
+After a child successfully resumes or cancels a request, restore the saved parent
+context before forwarding outputs or capturing the origins of follow-up requests.
+Release completed origins, preserve still-pending origins across new inputs, and
+reject mixed-parent-scope ordinary response batches before consuming any requests,
+including batches spanning sibling bindings. Cancellation batches obey the same
+parent-scope rule because cancellation can also produce downstream outputs.
+Checkpoint preflight applies the same validation to queued propagated replies
+before replacing live state. Wrapped replies use the delivery boundary below.
+
+Older checkpoints without boundary origins cannot reconstruct those ancestors'
+original scopes. A successful child continuation therefore receives a new isolated
+parent scope with a warning rather than reusing the latest parent's session.
+This does not bypass the child's declarative request-origin checks or explicit
+legacy-request migration policy.
+
+## Ordinary Core requests and live response preflight
+
+Ordinary `WorkflowContext.request_info()` requests can also pause before a child
+binding. Capturing only forwarded child requests leaves these continuations on the
+latest fresh run's scope. Capture their framework-owned run metadata in the same
+checkpointed state, and restore it before Core invokes the response handler,
+including for queued checkpoint replies. Core cancellation handlers restore the
+same metadata. Propagated child responses use this path once; wrapped child
+messages retain their boundary-specific capture and restoration.
+
+Wrapped origins are owned independently by each child binding, not by the
+coordinator's external-request ID. Relaying a child request with that same ID
+therefore retains two independently consumable origins across checkpoints.
+Capturing or releasing an origin stages only that mapping entry in shared state,
+with committed values unchanged until the superstep commits. Discard rolls back
+these entry updates. Full-map validation and detached copies remain checkpoint
+and response-batch operations, not per-request insertion or removal work.
+
+Preflight includes propagated responses already queued for delivery. Repeat
+validation before each live message batch is drained. Cancellation validates ordinary queued replies together with
+the complete selected cancellation set before consuming requests or invoking
+cancellation handlers. Legacy runner contexts use a non-persisting snapshot when
+optional queued-message inspection is unavailable.
+
+Deep-copying all live request payloads to preflight migrations imposes an unrelated
+serialization requirement on non-checkpointed, in-process workflows. Instead,
+response policies validate without mutation and return deferred updates to the
+scope metadata they own. Apply these updates only after every policy in the tree
+has validated. This preserves cross-child rejection atomicity without copying user
+payloads or changing their identity. Stored checkpoint restoration still preflights
+ordinary responses in its detached candidate tree before replacing live state.
+
+## State-dependent wrapped-reply routing
+
+Edge callbacks can read executor fields changed by preceding handlers or restored
+from a checkpoint. Evaluating them during queue or detached-checkpoint preflight
+and caching their decisions changes normal routing behavior. Copying executors
+does not rebind arbitrary callback closures, and speculative handler or restore-hook
+execution can have external side effects. An explicit stable/snapshot-aware routing
+API would impose a new contract on existing graphs.
+
+Instead, finish preceding messages from a source before selecting a wrapped
+reply's targets. Evaluate each edge callback once, validate all selected targets
+together against their own child policies, then deliver that reply. Retain the
+accepted parent scope in checkpoint-backed shared state while the continuation
+has in-flight messages. Later supersteps and mid-run checkpoint replay cannot
+introduce a different scope after earlier origins have been released. Clear this
+guard at quiescence before capturing the final checkpoint, so a later response-only
+run may resume another origin. An accepted new run with no in-flight work also
+resets the guard; restoring an in-flight checkpoint preserves it. Validate stored
+guard metadata before replacing live state.
+Normal message batches
+retain concurrent edge delivery; wrapped replies form delivery barriers, with
+their selected targets still executing concurrently.
+
+This deliberately narrows atomicity for wrapped replies: rejection retains the
+rejected and subsequent undelivered messages, but cannot undo earlier handlers,
+child continuations, external effects, or already restored checkpoint state.
+Recover a partially executed failed run from a known-good checkpoint rather than
+retrying its remaining queue. Ordinary response/cancellation batches and detached
+structural and ordinary-response checkpoint validation retain their preflight
+guarantees. No callback is assumed pure, and no executor is cloned or replayed
+speculatively.
+
+The proposed deciders are Python code owners; maintainer and
+engineering-management approval is still required.

python/packages/core/AGENTS.md

@@ -338,6 +338,37 @@ The vector store API is experimental under the shared `VECTOR_STORES` feature ID
   by that continuation, can atomically restore a supplied checkpoint before cancellation, and returns the resulting
   `WorkflowRunResult`. Cancelling a computer request also cancels the other pending requests in that agent's batch,
   retains already resolved sibling evidence in terminal output, and resets its session before new input.
+- **Nested workflow scopes** - Fresh runs select a checkpoint-backed scope before the entry snapshot.
+  `WorkflowExecutor` derives a child scope from the parent scope and binding ID after the child run is accepted.
+  Agent turns retain their session scope; direct fresh runs select a new scope. Entry replay therefore preserves
+  child scopes even when child execution had not started at the snapshot. Response-only continuations restore
+  their originating scope before further nested invocation. Every `WorkflowContext.request_info()` request
+  retains its run context in checkpointed shared state, including ordinary Core requests before a child binding.
+  Core restores and releases that context before invoking response or cancellation handlers, including queued
+  checkpoint replies. Wrapped child replies use the same helpers at their message boundary; propagated replies
+  are restored only once. Wrapped origins belong to their binding independently of external requests, even when
+  a coordinator relays the same request ID. Origin insertion and removal stage individual mapping entries;
+  committed snapshots remain unchanged until commit and discard rolls back pending changes.
+  Ordinary response and cancellation batches reject mixed parent scopes before consuming requests.
+  Wrapped child replies select routes immediately before delivery, after preceding handlers on their source
+  have completed, and validate all selected targets together using each child's own policy. Predicates run once
+  per delivery, not during checkpoint or queue preflight. Fan-in aggregates are not individual wrapped replies.
+  Selected replies cannot introduce a different parent scope while a continuation has in-flight messages.
+  The accepted scope is checkpoint-backed across supersteps and clears at quiescence before the final
+  checkpoint, so a later response-only run can select another origin. A rejected reply
+  and subsequent messages remain queued, but earlier handlers and child continuations are not rolled back.
+  Recover a partially executed failed run from a known-good checkpoint rather than retrying its remaining queue.
+- **Nested checkpoint restoration** - Stored and embedded restores preflight a detached checkpoint tree before
+  replacing live state or queues. `WorkflowExecutor` delegates its async private validation hook to its child's runner,
+  so ordinary queued responses use each child's own policy and migration metadata stays in the candidate snapshot.
+  Wrapped-reply policies run only at delivery after executor restoration; rejection does not undo that restoration.
+- **Response policy staging** - Live response validation does not copy request payloads. Policies validate
+  without mutation and may return deferred metadata updates; all policies in the workflow tree must succeed
+  before those updates are applied. This preserves in-process non-copyable payloads and atomic legacy-scope
+  migration without imposing checkpoint serialization requirements on ordinary requests.
+- **Runner context compatibility** - Queued-message inspection is an optional capability, not a required
+  `RunnerContext` member. Contexts without it use their non-persisting `build_checkpoint` snapshot so response
+  validation still includes queued replies without draining messages or consuming pending requests.
 - **`WorkflowBuilder`** - Fluent API for building workflows, including explicit
   `output_from` / `intermediate_output_from` selection for caller-facing emissions. `output_from`
   is an allow-list for **Workflow Output**; unselected executor payloads are hidden unless