[CMGS]

What

• meta_store.usage: true (opt-in, restart only) makes each node land its usage journal in one shared PostgreSQL table, sandboxd_usage, scoped by cell like the archive rows.
• Root-only GET /v1/usage?tenant=&since=&until=&after=&limit= returns the raw events of every node of the cell, in cursor order with cursor paging. A node without meta_store.usage answers 501 no shared usage store configured; a database outage answers 503 usage store down; retry.
• Without meta_store.usage nothing changes: no table, no goroutine, no route behaviour beyond the 501.

Why

An outside billing service needs every node's events in one place. In a cluster one sandbox's events span nodes (an adopted archive), and today the only source is a per-node usage.jsonl. The table holds raw events only; durations and prices stay with the consumer.

Design

• Shipper, not writer. usage.Ship reads usage.jsonl and its .1 backup every 5 s and lands new lines, up to 1000 per transaction, until it catches up. journal.append is untouched.
• At most once per line. Each row keeps the line's SHA-256, and (cell, node, line_sha256) is unique: a land skips a line whose digest already landed for its node (INSERT … ON CONFLICT DO NOTHING, batched in one transaction). One shipper per node lands its lines in journal order, so the node's newest row by cursor is its resume point; it lives in PG and cannot diverge from the rows. At boot and after any failed land the node reads that row and resumes after that line; in steady state it re-checks one line at a remembered offset (a rename keeps the bytes, so rotation keeps the hint valid). When neither file still holds the marked line (a crash lost a tail that had already landed, or the journal was reset), it re-reads both files and the digest key lands only what is new. A restart, a retry or a commit whose reply was lost lands no line twice.
• Gap-free paging. A cell's lands take a transaction-scoped advisory lock before the insert (COPY), so cursor (an identity column) commits in order. A consumer polling with after=<last cursor> never skips a row that commits later, even with several nodes landing at once or a node backfilling after an outage.
• Columns are the journal keys plus cell, cursor, node, line_sha256. Indexes: PK (cell, cursor) for paging, (cell, node, cursor) for the resume query, unique (cell, node, line_sha256) for idempotence, (cell, tenant, cursor) for the tenant filter, (cell, t) for since/until. The schema follows the other pg packages (metastore.Open runs the embedded DDL only when the table is missing).
• Node name: the mesh node id, else the host name.
• Opt-in flag (meta_store.usage, not on by default with meta_store): an upgrade must not add a table, a CREATE requirement or write load to a shared database sized for tenants and pools.
• NUL in ref: a SOCKS domain from a guest can carry \x00, which PostgreSQL text rejects, and one such line would block the node's stream. It lands as U+FFFD (test proves the failure without it). Invalid UTF-8 is already replaced at journal write.
• Torn journal tail (pool/journal.go): opening a journal whose last line a crash cut short ends that line with a newline first, so the first append after the restart is a line of its own instead of being merged into an undecodable fragment.
• Tenant on every claim-scoped event: hibernate, wake, fork, checkpoint, promote, release and reap now carry the claim's tenant (additive in usage.jsonl), so tenant=X returns everything needed to fold that tenant's durations.
• usage.Event: the line type moves from pool to usage (first commit, mechanical) so the writer and the shipper share one schema.

Hot-path cost

Zero. Claim, release, hibernate and wake append to the journal exactly as before (the first commit only renames the type). Landing runs in one background goroutine that only reads the journal files; a database outage never reaches the claim path. Off the claim path, a node's first land after boot hashes the journal once to find its mark (bounded by the 2 × 64 MiB the journal keeps).

Test evidence

• Unit (usage): resume across restarts, rotation while lines are unlanded, store down then back with a rotation in between, a lost reply, two rotations during an outage, a reset journal, an undecodable line and a torn last line, a backlog past one batch, and the ticker loop under synctest.
• PG-gated (usage/pg, real postgres:17-alpine, the CI image): cursor order and mark, every filter and paging, a land that skips a landed digest, cell isolation, the NUL case, the land waiting on the cell lock, the indexes, and Ship end to end across restarts and a rotation. The CI PG step now runs ./usage/... and ./archives/... and still fails on any --- SKIP. Local run: 46 PASS, 0 SKIP, 0 FAIL across ./tenants/... ./poolset/... ./usage/... plus the pool meta-store tests.
• API (server): root-only (403 tenant, 401 none), 501 without a store, query pass-through, next on a full page only, {"events":[]} when empty, 400 on each bad parameter, 503 without leaking the cause.
• Gates: GOWORK=off make go-lint (5 modules × linux/darwin), make go-test, asl -forwarder=false ./... on both GOOS. Locally the e2e module lint hits a staticcheck SA4023 panic on macOS that reproduces on main (fb70ec7) unchanged; CI on Linux is the gate for that module.

Testbed evidence

The runs below predate the removal of seq (c3744ab); their (node, seq) counts read as rows per node in cursor order.

Bare-metal test host, isolated cocoon root, cocoon v0.7.1, rt:24.04, a throwaway PostgreSQL 17.11, sandboxd built from this branch (before the rebase onto #406, which only touches engine/).

• Claim, hibernate, wake and release cycles land every event: 21 journal lines → 21 rows, 21 distinct (node, seq), 21 distinct digests; GET /v1/usage paged by 4 returns them in cursor order.
• Restart: rows unchanged after the restart, new events continue at seq 22 with no duplicate.
• PG killed (pg_ctl stop -m immediate) mid-run: 8 more cycles ran unaffected (claim p50 4.4 ms, hibernate 310 ms, wake 111 ms, release 161 ms), one warning logged, /v1/usage answered 503. After PG returned the 32 backlogged events landed within ~2 s and the node logged usage events land again.
• A/B hot path, meta_store.usage on/off interleaved on,off,off,on, 10 cycles each (p50 ms): claim 0.6 / 4.6 / 4.8 / 4.5 (warm/clone bimodal, max 155–158 in every arm), hibernate 312 / 315 / 307 / 319, wake 112 / 116 / 116 / 117, release 173 / 176 / 167 / 176. No arm difference.
• Lines journaled while the flag was off landed when it was turned back on. Final state after 6 restarts: 225 journal lines = 225 rows = 225 distinct (node, seq) = 225 distinct digests.
• Tenant filter: a tenant claim shows under tenant=acme; a tenant token gets 403; a since/until split at one event's time partitions all events (113 + 114 = 227).
• Kit, VMs and database removed afterwards.

Review fixes (Codex round 1)

Each fix is its own commit with a test that fails on its parent:

• pool: end a crash-torn journal line before the next append — TestJournalKeepsTheFirstAppendAfterATornLine drives the real newJournal + append; on the parent the append merges into the fragment.
• pool: stamp the claim's tenant on every claim-scoped usage event — TestEveryClaimScopedUsageEventCarriesTheTenant runs claim, hibernate, wake, fork, checkpoint, promote, release and reap for a tenant claim; on the parent 7 events lack the tenant.
• usage: land a journal line at most once per node by its digest — PG-gated TestShipLandsNoLineTwiceAfterTheJournalLosesItsLandedTail (6 rows instead of 4 on the parent) and TestLandSkipsALandedLine.
• docs: count every shared table's query pool in the connection budget — the pool set, archive registry and usage table each open a pool sized by the DSN's pool_max_conns.

Hardware re-run on the same host at head 2955632 (fresh kit, PG 17.11, removed afterwards). verify.py hashed every journal line and compared against the table:

• Baseline with a tenant: 24 lines = 24 rows; the tenant's 3 claims have 12 rows, all with the tenant.
• SIGKILL of sandboxd mid-cycle, restart, more cycles: 51 lines = 51 rows, 51 distinct digests, 51 distinct (id, ev, t).
• Torn tail: SIGKILL, then the last (already landed) line cut in half, then restart. The fragment stays alone on line 51, and the first event after the restart (a claim) is the next whole line and lands. 59 rows = 58 decodable lines + the torn line's earlier row; no duplicates.
• Lost landed tail: SIGKILL, then the last 3 landed lines dropped, then restart and 5 cycles. 79 rows = 75 decodable lines + 4 earlier rows no longer in the journal; 79 distinct digests and 79 distinct (id, ev, t); seq increasing. A clean restart after that kept 83 = 79 + 4 with no duplicate.
• In every step: 0 decodable journal lines missing from the table.

Review round

• docs: usage table retention and pre-v0.1.21 tenant stamping — sandboxd never deletes a usage row; a prune must keep each node's newest row, since a node that finds none lands its whole journal again (checked on PostgreSQL 17: keep-newest prune → only the new line lands; prune-all → all 4 journal lines land again). GET /v1/usage notes that journals from v0.1.20 or earlier stamp the tenant only on claim, archive, egress and vmm events.
• pool: embed the request frame in the audit frame — auditFrame repeated requestFrame's nine fields; promoted-field literals keep every call site, and the audit JSON is byte-identical (checked on both sides of the change).
• server: one page-limit parser for the tenant and usage lists — both lists share the documented limit contract.
• usage: open the journal backup inline — os.Open already returns a nil file with fs.ErrNotExist; the wrapper and its gosec suppression go.
• pool: one usage journal decoder for the tests — five tests decoded usage.jsonl by hand.
• ci: run the archive registry tests against PostgreSQL — without SANDBOXD_TEST_PG_URL the 20 archives/pg and archivestest cases skip, and the PG step did not list them, so CI never ran them. They pass on PostgreSQL 17.
• review: drop two comments that restate the usage wiring.

Round delta: prod −17, test −35, comments added 0. Gates: make go-lint 10 × 0 issues. (5 modules × linux/darwin) plus fmt --diff clean; asl clean on both GOOS; go test -race -count=1 green in all 5 modules; the CI PG step plus ./archives/... on PostgreSQL 17: 84 PASS, 0 SKIP; Python SDK ruff format --check, ruff check, pytest 269 passed.

• usage: drop the seq column; a node resumes from its newest cursor — seq repeated each node's cursor order, skipped numbers where a landed line was skipped, and cost a column, a CHECK and a unique index on every land. The resume query now runs on (cell, node, cursor) (EXPLAIN: index scan backward on sandboxd_usage_node).
• usage: drop the empty-event guard — no writer produces a decodable line without ev.
• sdk/python: name the controller test's home volume once.

Gates after these three: make go-lint 10 × 0 issues. (cold lint cache; a warm cache hits the known staticcheck SA4023 panic in e2e), asl clean on both GOOS, go test -race -count=1 green in all 5 modules, the CI PG step plus ./archives/... on PostgreSQL 17: 83 PASS, 0 SKIP, Python 269 passed.

Two-node cluster on hardware (head aeb0e94)

Two bare-metal test hosts, one sandboxd each (isolated cocoon roots, cocoon v0.7.1, rt:24.04 sha256:4231957e9247), joined in one gossip mesh with a cluster key, one meta_store cell with usage: true, and PostgreSQL 17.11 on the first host; the second node lands over a WireGuard link (~10 ms RTT). verify.py hashes every journal line of both nodes and compares it with line_sha256 in the table.

• Mesh: each node lists the other as its peer, sandboxd_config_digest_mismatch 0; a SIGKILLed node drops out of the peer's view in 6.8 s and is back 1.0 s after its restart.
• Concurrent load on both nodes (12 root + 4 tenant cycles each) while a consumer pages GET /v1/usage?after=&limit=7, alternating between the nodes: 128 rows, the consumer saw all 128 exactly once and in table order. Each node: 64 journal lines = 64 rows, cursor order = journal order (the property seq used to carry).
• Either node serves both nodes' events; tenant=acme returns 32 events (claim, hibernate, wake, release × 8, both nodes); tenant token 403, no token 401, limit=0 400; a since/until split at one event's t partitions all rows (65 + 63 = 128).
• Graceful restart of one node, SIGKILL of the other mid-cycle, then a lost landed tail (the last 3 landed lines cut from the journal, restart, more cycles, clean restart): no line landed twice, the 3 cut rows stay, no decodable line missing.
• PostgreSQL stopped for 20 s while both nodes cycled: claims unaffected (claim p50 0.7 ms), GET /v1/usage 503 on both, one warning per node, both backfilled within 1–3 s of the restart.
• A/B on one node, meta_store.usage on, off, off, on × 8 cycles (p50 ms): claim 0.7 / 0.7 / 0.7 / 0.7, hibernate 394 / 399 / 400 / 394, wake 122 / 115 / 115 / 124, release 162 / 172 / 168 / 175. No arm difference; lines journaled while off landed when it was turned back on.
• Final: 380 decodable journal lines, 383 rows (the 3 cut on purpose), 0 duplicate digests, 0 duplicate events, 0 missing. Kits, VMs and the database removed afterwards.

Docs

deploy.md (config row, a "Shared usage table" section with the DDL, exactly-once, order, outage and rename rules), sandboxd-api.md (GET /v1/usage, the root-only list, a pointer from the journal section), cluster.md (the usage journal bullet points at the table), index.md, README.md.

[tonicmuroq]

GET /v1/usage?tenant= only returns the events that carry a tenant: claim, archive/unarchive/archive_delete, egress/egress_bytes and vmm_*. hibernate, wake, release, reap, fork, checkpoint and promote are journaled without one, so a consumer that filters by tenant gets a tenant's claims but none of the hibernate/wake/release events it needs to compute compute seconds, with no sign anything is missing.

Either stamp sb.Tenant on those events at journal time (the sandbox is at hand at each call site), or say in sandboxd-api.md that the filter matches only tenant-stamped events and a billing consumer should read every event and join on id to the claim's tenant. The first seems better: the filter is then complete for new events, and the doc only needs a note about events journaled before the change.

[CMGS]

Done in 8ea1870: hibernate, wake, fork, checkpoint, promote, release and reap now carry the claim's tenant, and TestEveryClaimScopedUsageEventCarriesTheTenant checks every claim-scoped event. 3eea392 adds the note to GET /v1/usage: a journal written by v0.1.20 or earlier stamps the tenant only on claim, archive, unarchive, archive_delete, egress, egress_bytes and vmm_*, so a consumer joins its other events on id to the claim.