usage: expose an event-time watermark for final billing settlement #412

Problem

The shared usage stream has a durable cursor, but no event-time completeness boundary. A billing consumer cannot tell whether a historical interval is final or whether another node still has older events queued in its local journal.

This came up while reviewing https://github.com/simular-ai/box/pull/15 against the usage changes in #409 and #410. These are reproduced consumer failure cases, not a claim of a production billing incident.

The current path is:

1. A lifecycle operation calls recordUsage, which timestamps the event and appends it to local usage.jsonl.
2. main starts usage.Ship in a separate goroutine. It drains the journal every five seconds and retries failed uploads.
3. Each node lands its own batches in the cell's shared PostgreSQL usage table. Cursors follow commit order, not event t order across nodes.
4. cocoon-api's meter actively polls GET /v1/usage and folds the rows it reads. Events are not pushed to the meter.

Code references at 5c18daa:

• Local event recording
• Background shipper startup
• Shipper and retries
• Usage API and folding contract

Draining all currently visible rows only establishes cursor completeness for committed rows. It does not establish that all events before a particular t have landed. A fixed settlement delay does not solve an arbitrarily long upload outage.

Examples

A late reap changes historical compute without changing the current state

For one replicated sandbox, with a compute deadline later than the following times:

Suppose A's claim has landed, but A's reap remains queued. B's events land first. At 00:05, folding the visible rows yields four compute minutes: 00:00–00:02 and 00:03–00:05. Once A's reap lands, folding all rows in t order yields three minutes: 00:00–00:01 and 00:03–00:05.

Ignoring an event older than the current state prevents the late reap from stopping B's running sandbox, but it does not remove the extra historical minute. Applying it as the latest state is also wrong. The consumer needs to know when the history is complete before final billing, or support corrections after billing.

A late renew can fill a window that appeared complete

A claim initially has a deadline at 00:02. A renew at 00:01 extends it to 00:07, but the renew is queued locally. A consumer polling the table can initially see only two billable minutes. Reporting that partial five-minute window as final prevents the later renew from filling it safely with the same immutable billing request/key.

There is a similar case when a reap is reported, then a later poll discovers an adopt/unarchive that resumed compute inside the already-reported window. These consumer cases are why a completeness boundary would help, beyond just protecting the latest state from stale events.

Possible solution: a usage event-time watermark

Have sandboxd publish a conservative per-node boundary describing its upload backlog. The earliest timestamp among not-yet-landed events is a useful starting point. Expose the stronger contract to consumers as a watermark W:

Every event from this producer with t < W has been committed to the shared usage table, and the producer will not later introduce another event with t < W.

The boundary is exclusive. An event exactly at W remains preliminary. A window [start, end) can be finalized when end <= W.

For a cell, take the minimum watermark over all relevant producers. The meter could then:

1. Poll and drain usage by cursor as today.
2. Retain/recompute preliminary history in event-time order, including late events.
3. Report only complete windows ending at or before the cell watermark and the meter's normal settlement boundary.
4. Freeze those reported windows; newer history remains preliminary.

This replaces an arbitrary cache duration with an explicit completeness boundary. It does not require billing to edit past charges if the boundary is trustworthy. A watermark alone does not replace the consumer's event-time fold.

Details that need a defined contract

• Publication ordering: advance a watermark only after every event it covers has committed. A failed or partial upload must leave the boundary behind the remaining backlog.
• Concurrent producers and an empty queue: observing no pending lines must not simply mean W = now or infinity. Account for events being timestamped/appended concurrently. A producer barrier or journal sequence/checkpoint may be needed to prove the prefix is complete, including after restart.
• Reading consistency: associate the boundary with a committed cursor barrier or equivalent snapshot contract. A consumer must not drain an older view of the rows and then finalize using a newer watermark without also reading all rows covered by it. This also needs to work across pagination.
• Idle nodes: provide a safe heartbeat/barrier so an idle producer can advance its boundary without inventing lifecycle events.
• Membership and identity: include every producer that may still backfill historical events, not just currently reachable nodes. Define node restarts/incarnations, joining, removal, and legacy nodes without watermark support. A missing/unknown watermark cannot silently mean complete.
• Clock behavior: define how the producer prevents a later event timestamp from falling below a published watermark, including clock rollback. The timestamp of the oldest queued event alone is insufficient without this guarantee.
• Lost or unreadable backlog: journal rotation/loss, malformed skipped lines, or append failures must not silently become proof that history is complete. Define how such a gap is exposed.

These are requirements for a meaningful boundary, not a request to implement a specific table or endpoint shape. One possible interface is a cell watermark plus per-node progress/status and a cursor barrier on the existing usage API or a companion endpoint.

Offline nodes: an explicit tradeoff remains

A disconnected node cannot safely advance its watermark. If it has older unshipped events, the cell watermark must remain behind them. The immediate effect is delayed final billing, not speculative final charges.

Permanent loss needs a policy decision. Options include recovering the journal and draining it, or explicitly sealing a producer/ownership epoch under a documented lease/fencing billing rule. Removing a node from the live registry or expiring its node lease does not, by itself, prove that it has no older queued events. If an epoch is sealed without recovering its backlog, define how subsequent old events are treated and acknowledge that final charges then follow the sealing rule rather than a perfect reconstruction of local history.

There is no finite buffering duration that guarantees both exact event-time billing and progress through an unbounded producer outage without either this kind of finalization rule or billing adjustments.

Suggested acceptance cases

• B's adopt/unarchive lands before A's earlier reap: no affected window is declared final until A's covered backlog is committed; the final fold has the correct historical total and B remains running.
• A delayed renew fills a preliminary window, which is reported once with the final quantity.
• Failed uploads, multi-batch drains, concurrent appends and watermark reads across pagination never expose a boundary ahead of committed, consumer-readable events.
• Idle producers advance safely; restarting producers and backward clock movement cannot introduce an event below an already-published boundary.
• An offline producer holds the boundary; permanent removal/sealing has an explicit, tested policy.

The request is to expose and document when usage history becomes final, while preserving the existing cursor-based polling API.