usage: make the stream self-sufficient for billing and close the fence gaps

docs/cluster.md

@@ -299,8 +299,8 @@ checkpoint store and keeps each claim's registry row on the latest snapshot:
   starts none, and a round whose registry is down fails before it captures.
 - **Restore.** A lost node's replica is adopted like an archive: the first
   `Lookup` that reaches another node after the lease expires takes the row,
-  records the claim as archived on the replica with the adopter pool's
-  retention window, and the next call wakes it
+  records the claim as archived on the replica with the row's retention deadline,
+  or the adopter pool's window when it has none, and the next call wakes it
   through the archive wake, which restores the guest with its memory and
   processes. The claim resumes from its last committed replica: everything
   after it, and the connections open at the time, are lost: up to the longer of
@@ -330,15 +330,18 @@ checkpoint store and keeps each claim's registry row on the latest snapshot:
   checkpoint behind, the same residue as a node that never returns.
   A restart keeps the node's own live row for a running claim and takes the checkpoint a crashed round left on
   it; a claim whose row another node holds, or whose row a finished adopter
-  consumed, is dropped with its VM. A replica row has no retention deadline: the
-  hourly sweep never purges it, because no round refreshes it while the claim is
-  hibernated, and retention starts when the claim is archived or adopted. The
+  consumed, is dropped with its VM. A live claim's replica row has no retention
+  deadline: the hourly sweep never purges it, because no round refreshes it while
+  the claim is hibernated, and retention starts when the claim is archived or
+  adopted. A row the node holds with no claim, as after reconcile dropped it
+  because the VM record was gone, gains the retention deadline on that sweep: the claim stays adoptable
+  until it passes, and `archive_delete_after_seconds` 0 keeps the row. The
   rows of a node that never returns therefore stay until a client adopts them, a
   bounded residue of one row and up to two checkpoints per claim: the row's, and
   a committed pending replica or a previous replica the node died before
   deleting. Their metas stay live, so offline reclamation keeps them.
   They are archive checkpoints (`"archive": true` in their meta) that no
-  registry row names, which an operator can delete. Apart from that and the
+  registry row names, which an operator can delete. Apart from these and the
   lease rule the sweep is unchanged.
 - **Self-fencing.** A node whose renewal is overdue by 12 seconds (the lease
   length minus a 3 second margin, counted from the start of the last renewal
@@ -373,7 +376,9 @@ checkpoint store and keeps each claim's registry row on the latest snapshot:
   claim's first replica included: the agent relay
   (including parked keep-alive relays, whose later RPCs never pass the
   resolution gate), the exec call, port relays and preview requests. A client
-  retries and meets the 503 until the lease is back or the claim moved.
+  retries and meets the 503 until the lease is back or the claim moved. Neither
+  an API call nor the idle sweep starts a hibernate or a wake of such a claim
+  while the fence is closed, so an idle claim stays billed until the fence lifts.
   Streams of claims that are not replicated stay open.
 - **Availability.** The design chooses consistency over availability for
   replicated claims. A registry outage longer than the fence threshold (12

docs/deploy.md

@@ -841,8 +841,10 @@ only; computing durations or prices stays with the consumer.
       vm          text        NOT NULL DEFAULT '',
       key         text        NOT NULL DEFAULT '',
       template    text        NOT NULL DEFAULT '',
+      net         text        NOT NULL DEFAULT '',
       size        text        NOT NULL DEFAULT '',
       tenant      text        NOT NULL DEFAULT '',
+      deadline    timestamptz,
       volumes     text[],
       volumes_rw  text[],
       children    text[],

docs/sandboxd-api.md

@@ -491,7 +491,8 @@ archives. When to hibernate is the caller's policy — the
 node only provides the transition. 204 on success, 404 unknown id or wrong
 token, 409 on the egress lane or when volumes are attached (neither kind of
 sandbox hibernates; see [egress](egress.md)), 429 while the disk is under
-[`disk_reserve`](deploy.md).
+[`disk_reserve`](deploy.md), 503 `lease lost; retry` for a replicated claim that
+is not archived while its node's lease is overdue, even one already hibernated.
 
 ## POST /v1/sandboxes/{id}/wake
 
@@ -505,7 +506,8 @@ while stored, and waking it starts a fresh lease of the length the claim was gra
 [failed](deploy.md#dead-vmms-and-host-restarts) sandbox is cold-booted from
 its own disk (memory is lost) and its restart budget resets. 204 on
 success, 404 unknown id or wrong token, 409 a failed egress-lane or volume
-sandbox, or one whose VM record is gone, 500 when the cold boot itself fails.
+sandbox, or one whose VM record is gone, 500 when the cold boot itself fails,
+503 `lease lost; retry` for a replicated claim while its node's lease is overdue.
 
 ## POST /v1/sandboxes/{id}/renew
 
@@ -1013,13 +1015,18 @@ truth is the usage journal below.
 
 Always on: every lifecycle transition appends one JSONL event to
 `<data_dir>/usage.jsonl` — `{"t": <RFC3339>, "ev":
-"claim|hibernate|wake|fork|checkpoint|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed",
-"id": "sb_…", "vm": "sbx-…"}` plus `key` (the pool key's stable hash, claim
-events), `template` and `size` (the claim's template and size tier, claim
-events, so a consumer can price a sandbox by its shape without the claim
-record, which is gone after release), `tenant` (the owning tenant, on every
-event of a tenant's claim), `children` (fork) and `ref` (the promoted
-template / checkpoint id, the egress host, or a `vmm_failed` reason). An egress event that injected credentials names
+"claim|renew|hibernate|wake|fork|checkpoint|adopt|promote|release|reap|archive|unarchive|archive_delete|egress|egress_bytes|vmm_exit|vmm_restart|vmm_failed",
+"id": "sb_…"}` plus `vm` (the VM's name; absent on `renew`, `adopt`, `archive`,
+`unarchive`, `archive_delete`, `egress`, `egress_bytes`, and the `reap` or
+`release` of an archived claim), `key` (the pool key's stable hash, claim
+events), `template`, `net` and `size` (the claim's template, lane and size
+tier, claim events, so a consumer can price a sandbox by its shape without the
+claim record, which is gone after release; `size` sets the CPU and memory,
+and `template` is a label, since promote lets the caller name it), `deadline` (the deadline in force after this event: `claim`,
+`renew`, `wake` and `unarchive` carry the lease, `archive` and `adopt` the retention
+end, or nothing for an archive kept forever, and `reap` the deadline it ended under), `tenant` (the owning tenant, on every event of a
+tenant's claim), `children` (fork) and `ref` (the promoted template / checkpoint
+id, the egress host, a `vmm_failed` reason, or why reconcile ended a claim). An egress event that injected credentials names
 them in `secret` (`gh,claim:<entry>`, never a value). An egress event routed through an
 [upstream proxy](egress.md#upstream-proxies) carries `upstream` (its `host:port`), and with
 `egress_usage_bytes` on, `egress_bytes` events add the connection's payload `tx`/`rx` bytes. A volume claim also carries
@@ -1028,16 +1035,30 @@ the subset of those names claimed `rw`, so billing can discriminate write
 access (mounts and host paths are not billing dimensions). The file rotates at
 64 MiB keeping one `.1` backup, so a tailing collector never loses a window
 silently. A last line a crash cut short is ended with a newline when the node
-reopens the journal, so it never swallows the next event. Folding rules: billable compute seconds per sandbox =
-Σ(claim→release/reap) − Σ(hibernate→wake) − Σ(vmm_exit→vmm_restart/release/reap);
-`vmm_exit` is stamped when the node first finds the VMM gone, so an outage
-before sandboxd runs again (a host that was down) stays billed up to that
-point, and `vmm_failed` is informational; hibernated storage seconds =
-Σ(hibernate→wake) minus the archived span; archived storage seconds =
-Σ(archive→unarchive/archive_delete), a cheaper store tier than a hibernated
-VM's RAM. An interval left open by a crash clamps to the claim's
-deadline; reconcile records nothing for the claims it drops (those whose VM
-record is gone), so a restart leaves those intervals without a terminal event. The `vm` name joins
+reopens the journal, so it never swallows the next event. Folding rules: fold a sandbox's events of all nodes in `t` order.
+An event ends the current span at its `t` and enters a state: `claim`, `wake`,
+`unarchive` and `vmm_restart` enter `running`, `hibernate` enters `hibernated`,
+`archive` enters `archived`, and `vmm_exit`, `archive_delete`, `release` and
+`reap` enter `ended`. Other events, `vmm_failed` included, change no state, and
+`reap`, `adopt` and `archive_delete` have the exceptions below.
+Billable compute, hibernated storage and archived storage seconds are the time
+in `running`, `hibernated` and `archived`, a cheaper store tier than a hibernated
+VM's RAM. `vmm_exit` is stamped when the node first finds the VMM gone, so an
+outage before sandboxd runs again (a host that was down) stays billed up to that
+point. A `reap` instead ends the current span at min(`t`, its `deadline`), or `t`
+with none, because `t` can follow the true end by the reaper's tick, the VM's
+teardown or the node's downtime. Outside `archived`, an `archive_delete` ends the
+current span at min(`t`, the last `deadline`), because a purge can follow a claim
+that has no end event by a whole retention window. An `adopt` names a claim
+a node took over, from a lost node or from its own lost record, with `ref` the
+checkpoint it holds. In `archived` it changes nothing; otherwise it ends the
+current span at min(`t`, the last `deadline` before it) and enters `archived` at
+its `t`. A span is never shorter than zero. A span still open ends at min(now,
+the last `deadline`), and one entered by an `archive` or `adopt` without a
+`deadline` stays open. A claim reconcile drops because its VM record is gone
+ends with a `reap` whose `ref` is `vm_gone`, and one that cannot re-lock its
+egress NIC with `quarantined`; a claim another node holds is dropped with no
+event, and neither has a replicated claim dropped while the registry is down. The `vm` name joins
 cocoon's machine-level metering ledger for audit cross-checks. With
 `meta_store.usage`, the node also lands every line in the cell's
 [shared usage table](deploy.md#shared-usage-table), read through
@@ -1049,12 +1070,14 @@ Auth: root only (tenant tokens get 403). Reads the raw usage events of every
 node of the cell from the [shared usage table](deploy.md#shared-usage-table).
 
 - `GET /v1/usage?tenant=<name>&since=<time>&until=<time>&after=<cursor>&limit=<n>` →
-  `{"events": [{"cursor": 41, "node": "n1", "t": "2026-10-30T08:00:00.123456Z", "ev": "claim", "id": "sb_…", "vm": "sbx-…", "key": "…", "template": "…", "size": "large", "tenant": "acme"}], "next": 41}`.
+  `{"events": [{"cursor": 41, "node": "n1", "t": "2026-10-30T08:00:00.123456Z", "ev": "claim", "id": "sb_…", "vm": "sbx-…", "key": "…", "template": "…", "net": "none", "size": "large", "tenant": "acme", "deadline": "2026-10-30T09:00:00.123456Z"}], "next": 41}`.
 - **Events:** each is one journal line with its keys as in
   [usage.jsonl](#usage-journal-usagejsonl), `t` in UTC to the microsecond, plus
   `cursor` (its place in the table; one node's events follow its journal
-  order) and `node` (the node that recorded it). A `claim` line written by
-  v0.1.20 or earlier has no `template` or `size`.
+  order) and `node` (the node that recorded it). A journal written by
+  v0.1.20 or earlier has no `renew` or `adopt` events and no `deadline`, a claim
+  reconcile dropped has no end event, and its `claim` lines have no `template`,
+  `net` or `size`.
 - **Filters:** `tenant` keeps one tenant's events; `since` (inclusive) and
   `until` (exclusive) are RFC 3339 times compared with `t`. Each is optional.
   A journal written by v0.1.20 or earlier stamps `tenant` only on `claim`,

sandboxd/pool/archive.go

@@ -150,7 +150,6 @@ func (m *Manager) archive(ctx context.Context, sb *types.Sandbox) error {
 	js := m.store.set(sb)
 	m.mu.Unlock()
 	if saveErr := m.store.commit(js); saveErr != nil {
-		// roll back so memory matches the still-durable hibernated record
 		m.mu.Lock()
 		var rb claimSnapshot
 		// a Release that landed meanwhile already deleted the claim; re-setting it would resurrect it
@@ -166,6 +165,8 @@ func (m *Manager) archive(ctx context.Context, sb *types.Sandbox) error {
 		m.abandonArchive(ctx, sb, epoch, ck.ID)
 		return fmt.Errorf("archive %s: persist claims: %w", sb.ID, saveErr)
 	}
+	m.counters.archives.Add(1)
+	m.recordUsage(ctx, usage.Event{Event: "archive", ID: sb.ID, Reference: ck.ID, Tenant: sb.Tenant, Deadline: sb.Deadline})
 	// disarm under Transition so a wake right after the release is not clobbered by a late disarm
 	m.out.Disarm(sb.ID, true)
 	sb.Transition.Unlock()
@@ -175,12 +176,10 @@ func (m *Manager) archive(ctx context.Context, sb *types.Sandbox) error {
 	if replica != "" {
 		m.deleteOrphanArchiveCk(ctx, replica)
 	}
-	m.counters.archives.Add(1)
-	m.recordUsage(ctx, usage.Event{Event: "archive", ID: sb.ID, Reference: ck.ID, Tenant: sb.Tenant})
 	return nil
 }
 
-// wakeArchived restores an archived claim into a fresh local VM; caller holds Transition.
+// Caller holds Transition.
 func (m *Manager) wakeArchived(ctx context.Context, sb *types.Sandbox) (string, error) {
 	// egress never archives; a corrupt archived egress claim must fail closed.
 	if sb.Key.Net == types.NetEgress {
@@ -221,7 +220,7 @@ func (m *Manager) wakeArchived(ctx context.Context, sb *types.Sandbox) (string,
 	}
 	live, saved := m.commitWake(ctx, sb, built.VMName, built.VsockSocket)
 	if !live || !saved {
-		m.destroy(ctx, built.VMName) // released, or persist rolled back to archived
+		m.destroy(ctx, built.VMName)
 		if !live {
 			return "", ErrUnknownSandbox
 		}
@@ -245,11 +244,10 @@ func (m *Manager) wakeArchived(ctx context.Context, sb *types.Sandbox) (string,
 		return "", ErrUnknownSandbox
 	}
 	m.counters.unarchives.Add(1)
-	m.recordUsage(ctx, usage.Event{Event: "unarchive", ID: sb.ID, Reference: ck, Tenant: sb.Tenant})
+	m.recordUsage(ctx, usage.Event{Event: "unarchive", ID: sb.ID, Reference: ck, Tenant: sb.Tenant, Deadline: m.deadlineOf(sb)})
 	return built.VsockSocket, nil
 }
 
-// commitWake swaps an archived record back to a live VM if the claim is still live.
 func (m *Manager) commitWake(ctx context.Context, sb *types.Sandbox, vmName, sock string) (live, saved bool) {
 	m.mu.Lock()
 	live = m.claimed[sb.ID] == sb

sandboxd/pool/claim.go

@@ -40,7 +40,6 @@ type ClaimOptions struct {
 	EgressClass string
 }
 
-// apply stamps the options a claim carries for its life.
 func (o ClaimOptions) apply(sb *types.Sandbox) {
 	sb.Tenant, sb.ClaimRef, sb.Metadata, sb.OnExpire, sb.NoEgress = o.Tenant, o.ClaimRef, o.Metadata, o.OnExpire.Or(""), o.NoEgress
 	sb.EgressClass = o.EgressClass
@@ -198,6 +197,7 @@ func (m *Manager) Renew(ctx context.Context, id string, cred Cred, ttl time.Dura
 		return time.Time{}, fmt.Errorf("renew %s: persist claims: %w", id, err)
 	}
 	sb.Touch()
+	m.recordUsage(ctx, usage.Event{Event: "renew", ID: id, Tenant: sb.Tenant, Deadline: deadline})
 	return deadline, nil
 }
 
@@ -320,7 +320,7 @@ func (m *Manager) releaseResolved(ctx context.Context, id string, sb *types.Sand
 	m.mu.Unlock()
 	if saveErr := m.store.commit(js); saveErr != nil {
 		m.mu.Lock()
-		m.addClaimLocked(sb) // roll back so memory matches the still-durable claim
+		m.addClaimLocked(sb)
 		m.restoreVolumeHolds(sb, volumeHolds)
 		delete(m.pendingCks, ck)
 		rb := m.store.set(sb)
@@ -382,7 +382,6 @@ func (m *Manager) admitVolumes(tenant string, volumes []resolvedVolume) ([]types
 	return applied, nil
 }
 
-// admitClaim refuses a busy or retired volume or an over-quota tenant before any VM is built.
 func (m *Manager) admitClaim(tenant string, volumes []resolvedVolume, applied []types.Volume) error {
 	m.mu.Lock()
 	defer m.mu.Unlock()
@@ -506,7 +505,8 @@ func (m *Manager) finalizeBatch(ctx context.Context, sbs []*types.Sandbox, ttl t
 		m.recordUsage(ctx, usage.Event{
 			Event: "claim",
 			ID:    sb.ID, VMName: sb.VMName,
-			KeyHash: sb.Key.Hash(), Template: sb.Key.Template, Size: string(sb.Key.Size), Tenant: sb.Tenant,
+			KeyHash: sb.Key.Hash(), Template: sb.Key.Template, Net: string(sb.Key.Net), Size: string(sb.Key.Size),
+			Tenant: sb.Tenant, Deadline: sb.ClaimedAt.Add(clampTTL(ttl)),
 			Volumes: types.VolumeNames(sb.Volumes), VolumesRW: types.VolumeRWNames(sb.Volumes),
 		})
 	}
@@ -544,6 +544,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 	type victim struct {
 		action                       reapAction
 		id, vmName, snap, ck, tenant string
+		deadline                     time.Time
 		sb                           *types.Sandbox
 		holds                        []types.Volume
 	}
@@ -563,7 +564,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 				unrows = append(unrows, sb)
 			}
 		case sb.ArchiveCk != "":
-			expired = append(expired, victim{action: reapPurge, id: id, ck: sb.ArchiveCk, tenant: sb.Tenant, sb: sb})
+			expired = append(expired, victim{action: reapPurge, id: id, ck: sb.ArchiveCk, tenant: sb.Tenant, deadline: sb.Deadline, sb: sb})
 			purges = append(purges, sb.ArchiveCk)
 		case m.archivesAtDeadline(sb):
 			// archive instead of destroy, kept in m.claimed for archive() to move
@@ -577,7 +578,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 			}
 			expired = append(expired, victim{action: action, id: id, sb: sb})
 		default:
-			expired = append(expired, victim{action: reapDestroy, id: id, vmName: sb.VMName, snap: sb.HibernateSnap, tenant: sb.Tenant, sb: sb, holds: m.cancelVolumeRun(sb)})
+			expired = append(expired, victim{action: reapDestroy, id: id, vmName: sb.VMName, snap: sb.HibernateSnap, tenant: sb.Tenant, deadline: sb.Deadline, sb: sb, holds: m.cancelVolumeRun(sb)})
 			m.dropClaimLocked(sb)
 			dropped = append(dropped, id)
 		}
@@ -634,7 +635,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 			if v.action == reapArchive || v.action == reapPause {
 				delete(m.archiving, v.id) // never removed from m.claimed
 			} else {
-				m.addClaimLocked(v.sb) // roll back so memory matches the still-durable claim
+				m.addClaimLocked(v.sb)
 				m.restoreVolumeHolds(v.sb, v.holds)
 				delete(m.pendingCks, v.ck)
 				restored = append(restored, v.sb)
@@ -660,7 +661,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 			m.purgeArchiveCk(ctx, v.id, v.ck, v.tenant)
 			m.untrack(m.pendingCks, v.ck)
 			m.counters.reaps.Add(1)
-			m.recordUsage(ctx, usage.Event{Event: "reap", ID: v.id, Tenant: v.tenant})
+			m.recordUsage(ctx, usage.Event{Event: reapEvent, ID: v.id, Tenant: v.tenant, Deadline: v.deadline})
 			logger.Infof(ctx, "purged archived sandbox %s", v.id)
 		case reapArchive:
 			logSweepResult(ctx, logger, m.archive(ctx, v.sb), "archived expired sandbox "+v.id, "archive expired sandbox "+v.id)
@@ -672,7 +673,7 @@ func (m *Manager) reapOnce(ctx context.Context) {
 			m.out.Disarm(v.id, m.removeOrRetry(ctx, v.vmName, v.id, td))
 			m.dropSnap(ctx, v.snap)
 			m.counters.reaps.Add(1)
-			m.recordUsage(ctx, usage.Event{Event: "reap", ID: v.id, VMName: v.vmName, Tenant: v.tenant})
+			m.recordUsage(ctx, usage.Event{Event: reapEvent, ID: v.id, VMName: v.vmName, Tenant: v.tenant, Deadline: v.deadline})
 			logger.Infof(ctx, "reaped expired sandbox %s (%s)", v.id, v.vmName)
 		}
 	})
@@ -736,7 +737,13 @@ func (m *Manager) claim(id, token string) (*types.Sandbox, bool) {
 	return m.authed(id, token)
 }
 
-// authed looks up a claim by id and token; callers hold m.mu.
+func (m *Manager) deadlineOf(sb *types.Sandbox) time.Time {
+	m.mu.Lock()
+	defer m.mu.Unlock()
+	return sb.Deadline
+}
+
+// Callers hold m.mu.
 func (m *Manager) authed(id, token string) (*types.Sandbox, bool) {
 	sb := m.claimed[id]
 	if sb == nil || subtle.ConstantTimeCompare([]byte(sb.Token), []byte(token)) != 1 {