Repository navigation
usage: make the stream self-sufficient for billing and close the fence gaps - #410
Merged
Merged
Conversation
A consumer could not bound an interval a crash left open: the folding rule clamps to the claim's deadline, but no event carried it, renew recorded nothing, and reconcile ended claims silently. Events that move the deadline now state it, renew records an event, and reconcile records a reap with the reason for a claim it drops or quarantines. Claim events also carry the net lane, so all three pool key axes read without the hash.
ClaimedAt plus the clamped lease is the stamped deadline and nothing moves ClaimedAt, so the claim path needs no captured slice. The reap reasons each have one call site, and quarantineClaim runs only inside Reconcile, before anything can race its forget. The docs state that an archive without a deadline leaves its span unbounded.
The archive event followed the teardown, outside Transition, so a wake during teardown journaled unarchive before archive: the archived span never closed and the retention end overrode the new lease. The event now lands under Transition right after the commit, which orders it against unarchive and survives a crash during teardown.
A replica is taken of a running claim, so after node loss the stream showed compute until the adopter's release or reap, up to the retention end if never woken. An adopt event ends that span and opens the archived one. The fold now closes every reap at min(t, last deadline), since t can follow the true end by the teardown, the downtime or the retention, and unarchive closes open hibernate and vmm_exit spans, which the rule had left out.
A fenced node may have lost the claim to an adopter, yet the idle sweep and the hibernate API still snapshotted its VM and journaled hibernate, which split billing between the adopter's compute and hibernated storage. Hibernate now answers ErrLeaseLost under the same gate the data plane uses; the idle sweep retries once the fence lifts.
A replica row has no deadline, and the sweep purges an own row with no claim only past one. A claim reconcile dropped because its VM record was gone therefore kept its row and checkpoint forever if the client never came back. The hourly sweep now gives that row the pool's retention deadline: the claim stays adoptable until then, and the existing purge reclaims it. A compare-and-set on the epoch keeps this safe against an adoption, and a failed write retries on the next sweep. A keep-forever pool keeps the row.
Review round. A fenced node refused hibernate but still woke a replicated claim, which restored its VM and journaled a wake a stale fold counted as compute; Wake now answers ErrLeaseLost. Renew stays open, since refusing it during a registry outage expires leases and widens the outage. A reap now carries the deadline it ended under, so it settles without searching the stream and a lost renew line cannot shorten it. Reconcile records no vm_gone for a replicated claim when the registry did not answer, since it may be running elsewhere, and records the wake a crash lost, which left compute unbilled. The fold is a state table: the formula plus exceptions read two ways, and read literally billed compute across an adoption. Cleanups: startRowRetention returns its error, the ticker and egress frame take the modern forms, tests share a restart helper, and comments that restate code go.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #409. A consumer could not bound a billed interval from the usage stream alone: the fold clamped open intervals to the claim's deadline, but no event carried it,
Renewmoved it silently, reconcile ended claims silently, and a node's loss billed compute with no VM. This makes the stream self-sufficient and rewrites the fold as a state table.Events
claim,wakeandunarchivecarrydeadline, the lease in force.archiveandadoptcarry the retention end, or nothing for an archive kept forever.Renewrecordsrenew.reapcarries the deadline it ended under. Claim events carrynet, so all three pool key axes read without the hash.adopt(new): a node took over a claim. A replica is taken of a running claim, so after node loss the stream showed compute until the adopter'sreleaseorreap, up to the retention end.reapwithref: vm_goneorquarantined, and thewakea crash lost. With the registry down it records novm_gonefor a replicated claim, which may be running elsewhere. A failed claims save records nothing, and the next boot does.archivelands underTransitionright after the commit. It used to follow the teardown, so a wake during teardown journaledunarchivebeforearchive, and the archived span never closed.Fold
A state table replaces the formula plus exceptions, which read two ways.
reapsettles at min(t, itsdeadline),releaseat itst, so the teardown of an expired claim is not billed.adoptoutsidearchivedends the span at min(t, last deadline) and entersarchivedat itst. Outsidearchived,archive_deleteends the span at min(t, last deadline).templateis documented as a label, because promote lets the caller name it: price compute bysize. Disk stays out: nothing bills by disk size today.Fence
A fenced node no longer hibernates or wakes a replicated claim: it answers
ErrLeaseLost(503), as the data plane does, through the API and the idle sweep.Renewstays open on purpose, since refusing it during a registry outage expires leases and widens the outage. An idle replicated claim therefore stays billed until the fence lifts, bounded by the outage.Replica row of a dropped claim
A
vm_goneclaim stays adoptable through its replica row. The row had no deadline, so it and its checkpoint stayed forever if the client never returned. The hourly sweep now gives a row this node holds with no claim the pool's retention deadline, and the existing purge reclaims it.archive_delete_after_seconds: 0keeps it. The checkpoint held during retention is not billed.Hot-path cost: the claim path gains two fields on the existing journal line, with no new lock or syscall.
Renewgains one journal append per call (123 bytes), so frequent renews shorten the 64 MiB rotation and the outage the journal can ride out.wake,unarchiveandadopttakem.muonce more, off the claim path.Upgrade note:
sandboxd_usageis only created when missing, so a table made by a pre-release build needs the columns before this runs. Until then landing fails andGET /v1/usageanswers 503. The journal catches up after theALTER, for lines still inusage.jsonland its.1backup. No released version has this table.Accepted, no code
reapno longer depends on it.unarchivecould precede itsadopt, but a wake must fetch and boot within the adopter's one commit.vmm_exit/vmm_restartif a VMM dies during a partition after adoption. A guard would delay the realvmm_exitof a claim never adopted.Verification at the final commit:
make go-linton a coldGOLANGCI_LINT_CACHE10 ×0 issues.(5 modules × 2 GOOS, pinned v2.13.2,fmt --diffincluded);asl -forwarder=falseclean on linux and darwin;go test -race ./...insandboxdwithSANDBOXD_TEST_PG_URLset againstpostgres:17-alpine; the fold text run through an interpreter on 13 sequences, each with one reading. Mutants over every new field, event, gate, tag and order are killed, except a log-only branch that changes no behaviour.Hardware, two bare-metal nodes in one cell with shared PostgreSQL and S3, on the real engine and guest image:
reap:reapcarries the lease, and itstwas 0.97 s past it, unbilled.claim,hibernate,archive,unarchive,release.claim,adopt,unarchive,releaseacross both nodes, with a file written before the kill read back after the wake. Fold 42.7 s of compute, 21.8 s archived.claim,adopt,archive_delete,reap, purged at the 120 s retention.claim,reap(vm_gone),adopt,unarchive,releasewith the file read back, and an unclaimed row purged at retention,claim,reap,archive_delete. This leg ran the final commit with the hourly store sweep shortened to 60 s, and nothing else changed.pr,main,main,pr, n=8 per arm,largetier, at the commit before the last amend (its later changes are off these paths): claim 1.1/1.0/1.1/1.1 ms, renew 0.5/0.4/0.4/0.5, hibernate 2373/2424/2363/2232, wake 152/156/161/164, release 186/186/184/194.Not run on hardware: the
quarantinedreap (no egress bridge), the wake a crash lost, reconcile with the registry down, the idle sweep under a fence, a keep-forever pool and real host power loss. Unit tests cover the first five.