Skip to content

A lease renew right after a gateway restart answers UNKNOWN_LEASE and the client drops the lease #983

Open
Open
A lease renew right after a gateway restart answers UNKNOWN_LEASE and the client drops the lease#983

Description

@V3RON
V3RON
opened on Oct 25, 2026

What happened

After a gateway restart (including a self-update), a lease.renew that arrives before the lease's worker has reported answers UNKNOWN_LEASE. The gateway looks every id up only in its lease index, which is empty until each worker reports (src/gateway/lease-index.ts:14-20, src/gateway/fleet-coordinator.ts:1410-1418). CLI and MCP renewal treat UNKNOWN_LEASE as final (src/lease-policy/index.ts:187-189) and drop the lease.

The device stays held on the worker until its TTL. Once the index is rebuilt, the requester's next lease request is refused with REQUESTER_ALREADY_LEASED, naming the orphan.

What you expected

ADR 0005 §16 and §30: clients "simply resume renewing" after a gateway restart. A renew in the rebuild window gets a retryable error (WORKER_UNREACHABLE), and the lease survives.

How to reproduce

  1. Gateway with one worker; an agent leases a device through the gateway.
  2. Restart the gateway (or let it self-update).
  3. Renew the lease within roughly 15–40 s, before the worker's first report: UNKNOWN_LEASE, and the client drops the lease.

Environment

main at 7a2f0b3. Found by reading code in the #412 review, not by a user report.

Logs and evidence

From the #412 review, finding R1. Proposed fix there: answer WORKER_UNREACHABLE for an id the index does not hold while the fleet is rebuilding. The window ends when every known worker has reported. ADR 0005 decision 5 says known worker ids are persisted, but only drained ids are (src/gateway/drain-store.ts:20). The fallback is a fixed 60 s window. The same window also fixes R3, R5 and R8.

Written by an agent.

Activity

  1. V3RON
    added
    bug:newReported. An agent may triage it (rule 8), and nothing more.
    on Oct 25, 2026
  2. V3RON
    added a parent issue on Oct 25, 2026
  3. V3RON
    added
    bug:triage
    and removed
    bug:newReported. An agent may triage it (rule 8), and nothing more.
    on Oct 25, 2026
  4. V3RON
    self-assigned this
    on Oct 25, 2026

  5. V3RON commented on Oct 25, 2026

    @V3RON
    V3RON
    on Oct 25, 2026
    ContributorAuthor

    Reproduction

    Confirmed for generated ids (<workerId>.<workerLeaseId>). Branch bug/983-repro:

    src/gateway/dispatcher.test.ts › "right after a gateway restart, renewing a generated lease id answers WORKER_UNREACHABLE until its worker reports, then renews the lease"

    pnpm exec vitest run --project unit src/gateway/dispatcher.test.ts -t "right after a gateway restart"

AssertionError: before the worker reconnects: expected DispatchError: Unknown lease: wrk_1.lease… to match object { code: 'WORKER_UNREACHABLE' }
-   "code": "WORKER_UNREACHABLE",
+   "code": "UNKNOWN_LEASE",

    Cases: worker not reconnected; reconnected, leases not reported; reported (renew reaches the worker).

    Root cause

    Defect, for generated ids. #requireEntry (src/gateway/fleet-coordinator.ts:1410) looks the id up only in the in-memory index and throws UNKNOWN_LEASE on a miss. The index holds a worker's leases only after #onViewsChanged (:1458) sees its first running report. ADR 0005 §16 (still binding for generated ids, per ADR 0020) and §30 say clients just keep renewing. Caller-chosen ids are a Decision: ADR 0020 says a miss answers UNKNOWN_LEASE at once. Changing that needs a superseding ADR.

    Simplest fix

    In #requireEntry, on a miss for an id containing . (chosen ids cannot, LEASE_ID_PATTERN): if that worker has not reported since start (absent from #lastReconciledLeases) and the gateway started under 60 s ago, throw WORKER_UNREACHABLE with { workerId }. Renew, release and exec all get it.

    Alternatives rejected

    • Route by splitting the id (§16): forwards before the owner is known, skipping §27a.
    • 60 s window for every id: breaks ADR 0020's bare-id answer.
    • Persist known worker ids: more state, still needs a timeout.

    Risk

    • dispatcher.test.ts "…UNKNOWN_LEASE for renew/release/exec against an id it never issued" uses wrk_ghost.lease_1 on a fresh gateway: move its clock past the window.
    • A lease that expired during the outage is retried until its deadline instead of dropped at once.
    • Update the Known gap "A lease survives a gateway restart…" in src/gateway/AGENTS.md and the comment at src/gateway/lease-index.ts:14-20.

    Written by an agent.

  6. V3RON
    removed their assignment
    on Oct 25, 2026

  7. V3RON commented on Oct 25, 2026

    @V3RON
    V3RON
    on Oct 25, 2026
    ContributorAuthor

    Maintainer: held at bug:triage for now; no delivery yet.

    Written by an agent.

  8. V3RON
    removed
    bug:triage
    on Oct 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions