Skip to content

A disconnected worker's expired lease locks its requester out of the fleet for up to 24 h #984

Closed
#992
Closed
A disconnected worker's expired lease locks its requester out of the fleet for up to 24 h#984
#992
Assignees
V3RON
Labels
bug:readyAn agent may fix it.

Description

@V3RON
V3RON
opened on Oct 25, 2026

What happened

A gateway lease on a worker that disconnects (the Mac sleeps or drops off the network) stays in the gateway's lease index for as long as the worker's view is kept: 24 h after last seen (src/gateway/worker-registry.ts:442-457, :566-577). The requester limit counts it regardless of connection state (src/gateway/lease-index.ts:152-165, src/gateway/fleet-coordinator.ts:311-318). Release answers WORKER_UNREACHABLE (fleet-coordinator.ts:1365-1369).

So once the lease's TTL has passed, the agent is locked out of the whole fleet with REQUESTER_ALREADY_LEASED until the worker returns or 24 h pass. lease.list also keeps showing the lease past its deadline.

What you expected

A lease past its last reported ttlDeadline on a disconnected worker no longer counts against the requester limit. No renew can have reached it through this gateway. The agent can lease another device.

How to reproduce

  1. Gateway with two workers; an agent leases a device on worker A.
  2. Disconnect worker A (sleep the Mac).
  3. Wait past the lease's TTL, then request a new device: REQUESTER_ALREADY_LEASED, even though worker B has capacity.

Environment

main at 7a2f0b3. Found by reading code in the #412 review, not by a user report.

Logs and evidence

From the #412 review, finding R2. Proposed fix there: skip such an entry for the requester limit, and keep it for routing.

Written by an agent.

Activity

  1. V3RON
    added
    bug:newReported. An agent may triage it (rule 8), and nothing more.
    on Oct 25, 2026
  2. V3RON
    added a parent issue on Oct 25, 2026
  3. V3RON
    added
    bug:triage
    and removed
    bug:newReported. An agent may triage it (rule 8), and nothing more.
    on Oct 25, 2026
  4. V3RON
    self-assigned this
    on Oct 25, 2026

  5. V3RON commented on Oct 25, 2026

    @V3RON
    V3RON
    on Oct 25, 2026
    ContributorAuthor

    Reproduction

    Confirmed. A defect: ADR 0005 §6 already treats a disconnected worker's lease past its deadline as expired, and the requester limit ignores that.

    src/gateway/fleet-coordinator.test.ts, "grants a requester a new lease once its only lease, on a disconnected worker, is past its last reported deadline, and keeps that lease routable to its worker (#984)", on bug/984-repro. One ms before the deadline the request is refused (control). One ms after it, the request still fails:

    pnpm exec vitest run --project unit src/gateway/fleet-coordinator.test.ts -t "#984"

AssertionError: expected RequesterAlreadyLeasedError: Requester ag… to match object { lease: { id: 'wrk_b.lse_1', …(1) } }
+   "existingLeaseId": "wrk_a.lse_old",
 ❯ src/gateway/fleet-coordinator.test.ts:1872:27

    Root cause

    src/gateway/fleet-coordinator.ts:312 passes every index entry (leaseIndex.held, lease-index.ts:152) to requesterLimitDecision. The entry (lease-index.ts:26) has no deadline and no connection state, so nothing at admission can see that the lease is past its deadline on a worker that cannot be reached. The only place that checks this is retention (worker-registry.ts:574), and it drops the entry only after 24 h.

    Simplest fix

    In #refuseIfAlreadyLeased, leave out a held lease whose view is disconnected and whose reported ttlDeadline <= now. Use the same comparison as pruneExpired. Keep the index entry, so ADR §28 and the gateway rule "never decides a lease is gone" still hold. A throwaway version of this made the whole file pass (247/247). Update line 65 of src/gateway/AGENTS.md.

    Alternatives rejected

    • Drop the entry at the deadline: the lease can no longer be routed, which breaks ADR §28.
    • Store ttlDeadline in FleetLeaseEntry: a second copy of view state that every renew must keep in sync.
    • Shorter retention: does not fix it, it only shortens the lockout.

    Risk

    • When the worker comes back, the requester briefly holds two leases, until the worker's lease.expired arrives.
    • The worker's clock and the gateway's clock may disagree. Retention already accepts this.
    • A view that disconnected while starting has no leases list, so its lease still counts.

    Written by an agent.

  6. V3RON
    removed their assignment
    on Oct 25, 2026

  7. V3RON commented on Oct 25, 2026

    @V3RON
    V3RON
    on Oct 25, 2026
    ContributorAuthor

    Decision

    Maintainer accepted the triage report's proposed fix: for the requester limit, skip a lease whose worker is disconnected and whose last reported deadline has passed. Keep the entry in the index for routing (ADR 0005 §28). The review should weigh the three risks the report lists.

    Written by an agent.

  8. V3RON
    added
    bug:readyAn agent may fix it.
    and removed
    bug:triage
    on Oct 25, 2026
  9. V3RON
    self-assigned this
    on Oct 25, 2026
  10. V3RON
    added 4 commits that reference this issue on Oct 25, 2026

    docs(gateway): the requester limit skips an expired lease on a discon…

    b2950a0

    test: straddle the #984 deadline, cover a connected worker, fail loudly

    69408df

    docs(gateway): an expired lease on a disconnected worker does not cou…

    a89d7ac

    test: a lease its disconnected worker's last view does not list still…

    3d5f29f
  11. V3RON
    closed this as completedin #992on Oct 25, 2026
  12. V3RON
    added a commit that references this issue on Oct 25, 2026

    fix(gateway): an expired lease on a disconnected worker no longer blo…

    a0ee5d6

  13. V3RON commented on Oct 25, 2026

    @V3RON
    V3RON
    on Oct 25, 2026
    ContributorAuthor

    Delivery run

    Merged: #992 (#984)

    Written by an agent.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bug:readyAn agent may fix it.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions