Skip to content

usage: make the stream self-sufficient for billing and close the fence gaps - #410

Merged
CMGS merged 7 commits into
mainfrom
usage-deadline-net
Nov 2, 2026
Merged

CMGS merged 7 commits into
mainfrom
usage-deadline-net

Conversation

@CMGS

@CMGS CMGS commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Follow-up to #409. A consumer could not bound a billed interval from the usage stream alone: the fold clamped open intervals to the claim's deadline, but no event carried it, Renew moved it silently, reconcile ended claims silently, and a node's loss billed compute with no VM. This makes the stream self-sufficient and rewrites the fold as a state table.

Events

  • claim, wake and unarchive carry deadline, the lease in force. archive and adopt carry the retention end, or nothing for an archive kept forever. Renew records renew. reap carries the deadline it ended under. Claim events carry net, so all three pool key axes read without the hash.
  • adopt (new): a node took over a claim. A replica is taken of a running claim, so after node loss the stream showed compute until the adopter's release or reap, up to the retention end.
  • Reconcile records a reap with ref: vm_gone or quarantined, and the wake a crash lost. With the registry down it records no vm_gone for a replicated claim, which may be running elsewhere. A failed claims save records nothing, and the next boot does.
  • archive lands under Transition right after the commit. It used to follow the teardown, so a wake during teardown journaled unarchive before archive, and the archived span never closed.

Fold

A state table replaces the formula plus exceptions, which read two ways. reap settles at min(t, its deadline), release at its t, so the teardown of an expired claim is not billed. adopt outside archived ends the span at min(t, last deadline) and enters archived at its t. Outside archived, archive_delete ends the span at min(t, last deadline). template is documented as a label, because promote lets the caller name it: price compute by size. Disk stays out: nothing bills by disk size today.

Fence

A fenced node no longer hibernates or wakes a replicated claim: it answers ErrLeaseLost (503), as the data plane does, through the API and the idle sweep. Renew stays open on purpose, since refusing it during a registry outage expires leases and widens the outage. An idle replicated claim therefore stays billed until the fence lifts, bounded by the outage.

Replica row of a dropped claim

A vm_gone claim stays adoptable through its replica row. The row had no deadline, so it and its checkpoint stayed forever if the client never returned. The hourly sweep now gives a row this node holds with no claim the pool's retention deadline, and the existing purge reclaims it. archive_delete_after_seconds: 0 keeps it. The checkpoint held during retention is not billed.

Hot-path cost: the claim path gains two fields on the existing journal line, with no new lock or syscall. Renew gains one journal append per call (123 bytes), so frequent renews shorten the 64 MiB rotation and the outage the journal can ride out. wake, unarchive and adopt take m.mu once more, off the claim path.

Upgrade note: sandboxd_usage is only created when missing, so a table made by a pre-release build needs the columns before this runs. Until then landing fails and GET /v1/usage answers 503. The journal catches up after the ALTER, for lines still in usage.jsonl and its .1 backup. No released version has this table.

ALTER TABLE sandboxd_usage
    ADD COLUMN IF NOT EXISTS template text NOT NULL DEFAULT '',
    ADD COLUMN IF NOT EXISTS net      text NOT NULL DEFAULT '',
    ADD COLUMN IF NOT EXISTS size     text NOT NULL DEFAULT '',
    ADD COLUMN IF NOT EXISTS deadline timestamptz;

Accepted, no code

  • Two concurrent renews of one sandbox can journal in reverse. The next renew or wake corrects it, and a reap no longer depends on it.
  • An unarchive could precede its adopt, but a wake must fetch and boot within the adopter's one commit.
  • A stale node can still journal vmm_exit/vmm_restart if a VMM dies during a partition after adoption. A guard would delay the real vmm_exit of a claim never adopted.

Verification at the final commit: make go-lint on a cold GOLANGCI_LINT_CACHE 10 × 0 issues. (5 modules × 2 GOOS, pinned v2.13.2, fmt --diff included); asl -forwarder=false clean on linux and darwin; go test -race ./... in sandboxd with SANDBOXD_TEST_PG_URL set against postgres:17-alpine; the fold text run through an interpreter on 13 sequences, each with one reading. Mutants over every new field, event, gate, tag and order are killed, except a log-only branch that changes no behaviour.

Hardware, two bare-metal nodes in one cell with shared PostgreSQL and S3, on the real engine and guest image:

  • Fields and an expiry reap: reap carries the lease, and its t was 0.97 s past it, unbilled.
  • Archive: claim,hibernate,archive,unarchive,release.
  • Fence, database stopped: hibernate and wake 503 and journal nothing, renew 200, and both work after the lift.
  • Node loss (SIGKILL of sandboxd on a replicated claim): adoption 13.8 s after the kill, claim,adopt,unarchive,release across both nodes, with a file written before the kill read back after the wake. Fold 42.7 s of compute, 21.8 s archived.
  • Node loss, never woken: claim,adopt,archive_delete,reap, purged at the 120 s retention.
  • VM record gone: claim,reap(vm_gone),adopt,unarchive,release with the file read back, and an unclaimed row purged at retention, claim,reap,archive_delete. This leg ran the final commit with the hourly store sweep shortened to 60 s, and nothing else changed.
  • Journal lines equal table rows on both nodes, no duplicates.
  • Interleaved pr,main,main,pr, n=8 per arm, large tier, at the commit before the last amend (its later changes are off these paths): claim 1.1/1.0/1.1/1.1 ms, renew 0.5/0.4/0.4/0.5, hibernate 2373/2424/2363/2232, wake 152/156/161/164, release 186/186/184/194.

Not run on hardware: the quarantined reap (no egress bridge), the wake a crash lost, reconcile with the registry down, the idle sweep under a fence, a keep-forever pool and real host power loss. Unit tests cover the first five.

CMGS added 7 commits November 2, 2026 12:56
A consumer could not bound an interval a crash left open: the folding rule clamps to the claim's deadline, but no event carried it, renew recorded nothing, and reconcile ended claims silently. Events that move the deadline now state it, renew records an event, and reconcile records a reap with the reason for a claim it drops or quarantines. Claim events also carry the net lane, so all three pool key axes read without the hash.
ClaimedAt plus the clamped lease is the stamped deadline and nothing moves ClaimedAt, so the claim path needs no captured slice. The reap reasons each have one call site, and quarantineClaim runs only inside Reconcile, before anything can race its forget. The docs state that an archive without a deadline leaves its span unbounded.
The archive event followed the teardown, outside Transition, so a wake during teardown journaled unarchive before archive: the archived span never closed and the retention end overrode the new lease. The event now lands under Transition right after the commit, which orders it against unarchive and survives a crash during teardown.
A replica is taken of a running claim, so after node loss the stream showed compute until the adopter's release or reap, up to the retention end if never woken. An adopt event ends that span and opens the archived one. The fold now closes every reap at min(t, last deadline), since t can follow the true end by the teardown, the downtime or the retention, and unarchive closes open hibernate and vmm_exit spans, which the rule had left out.
A fenced node may have lost the claim to an adopter, yet the idle sweep and the hibernate API still snapshotted its VM and journaled hibernate, which split billing between the adopter's compute and hibernated storage. Hibernate now answers ErrLeaseLost under the same gate the data plane uses; the idle sweep retries once the fence lifts.
A replica row has no deadline, and the sweep purges an own row with no claim only past one. A claim reconcile dropped because its VM record was gone therefore kept its row and checkpoint forever if the client never came back. The hourly sweep now gives that row the pool's retention deadline: the claim stays adoptable until then, and the existing purge reclaims it. A compare-and-set on the epoch keeps this safe against an adoption, and a failed write retries on the next sweep. A keep-forever pool keeps the row.
Review round. A fenced node refused hibernate but still woke a replicated claim, which restored its VM and journaled a wake a stale fold counted as compute; Wake now answers ErrLeaseLost. Renew stays open, since refusing it during a registry outage expires leases and widens the outage.

A reap now carries the deadline it ended under, so it settles without searching the stream and a lost renew line cannot shorten it. Reconcile records no vm_gone for a replicated claim when the registry did not answer, since it may be running elsewhere, and records the wake a crash lost, which left compute unbilled. The fold is a state table: the formula plus exceptions read two ways, and read literally billed compute across an adoption.

Cleanups: startRowRetention returns its error, the ticker and egress frame take the modern forms, tests share a restart helper, and comments that restate code go.
@CMGS CMGS changed the title usage: state the deadline, the net lane and the end of a dropped claim usage: make the stream self-sufficient for billing and close the fence gaps Nov 2, 2026
@CMGS
CMGS merged commit 5c18daa into main Nov 2, 2026
1 check passed
@CMGS
CMGS deleted the usage-deadline-net branch November 2, 2026 07:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant