Skip to content

review: evacuate stop, moved claims, docs and moves - #438

Open
CMGS wants to merge 70 commits into
mainfrom
review/a1104
Open

CMGS wants to merge 70 commits into
mainfrom
review/a1104

Conversation

@CMGS

@CMGS CMGS commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Whole-repo audit on main after #436 and #437. Each fix is its own commit with a regression test where one is cheap.

Main carries these until this lands

Do not release or deploy main before this PR:

  • POST /v1/drain?evacuate=1 can run on an uncordoned node (427d1a2b). Evacuate waits for warm VMs to go before it starts its run. A DELETE /v1/drain in that wait finds nothing to stop, and the run then moves every claim off the node.
  • A running claim that moved keeps its egress doors (def8bdc2). forgetAdopted removes the VM with no sandbox id, so the removal never disarms. The doors, listeners and goroutines stay until restart, one set per moved claim, unlogged.

Other fixes

  • sdk: Close follows a sandbox that moved off a retired node. It looked it up only on a 404, so once the old node was gone, the claim stayed on the adopter and every retry hit the dead address.
  • pool: a tier claim woken past its lease before the export gets a fresh lease. It used to be destroyed at the next reap tick, while the same call a moment later restored it.
  • pool: a fenced node captures no replicated claim (fork, promote, checkpoint answer 503).
  • pool: consumeArchive reads the row epoch under the manager lock.
  • egress: an upstream dying mid-body truncates the guest's response, for declared-length and chunked bodies.
  • outbound: ArmProxy closes a displaced door before it binds.
  • engine: restore and clone get the dump budget, since a clone restores memory like a restore. Accepted cost: a hung refill clone holds a refill slot for up to 18 minutes, bounded and loud.
  • sandboxd: a preview listener that cannot bind is fatal.
  • silkd: a client hang-up reaps an attached language server; git_branch list names an unborn branch.

Cleanup

  • Non-logic moves in sandboxd/pool: golden.go, provision.go, reap.go, archive_delete.go; ckMarkers, recordLocks, healVetoes and the sweeps field own their state; the pool type joins its methods. types.go splits into env.go and volume.go.
  • Rust borrows what it only reads (BootCfg<'a>, Cow env parsing); dead code removed.
  • Python and Go SDK tidying, shared test helpers, py.typed.
  • Docs match the code: API errors listed once under Guest errors, the capture fence, the restore budget, moved claims, the first wake after an evacuation.

Worth reading closely: silkd/src/lsp.rs, because a stray frame on a language server stream now closes it, matching pty. One log tag changes: pool.NewManager to pool.loadTemplates for template seeding.

Numbers against main

Prod +2185 -2287, net -102. Tests +723 -277, net +446. Docs +251 -190. Comment lines in Go, Rust and Python: 2106 to 2042.

Hot path: no new work on the claim path; the moves rename fields only. A/B on hardware, base f0f2bc4c vs this head, interleaved with the order swapped, sandboxd and driver pinned to one NUMA node:

warm   8 rounds  T-B p50 +0.0015 ms [-0.0068, +0.0097]  p90 -0.0016 ms [-0.0158, +0.0127]  median p50 0.590 vs 0.592
clone  6 rounds  T-B p50 -0.311 ms  [-3.175, +2.553]    p90 -0.437 ms  [-2.305, +1.431]    median p50 50.344 vs 49.567

Verification

Head ac739053:

lint 0-issues lines: 10
gotest rc=0
asl -forwarder=false rc=0 on darwin and linux, all modules
rust darwin silkd passed 158 failed 0 ignored 3
rust linux silkd passed 159 failed 0 ignored 3
rust darwin boot/init passed 15 failed 0 ignored 0
rust linux boot/init passed 17 failed 0 ignored 0
py sdk/python 276 passed, sdk/openai 10 passed, sdk/langchain 15 passed

Hardware, one bare-metal node, sandboxd source same as this head, cocoon v0.7.5, all PASS: smoke through the new init and silkd; LSP hang-up, data_end and stray frame; clone path; capture fence; egress mid-body cut; archive cycle with purge and a restart with live, hibernated and archived claims; drain and uncordon; evacuate with failover off.

Not run on hardware: the evacuate stop race, the door close and the tier wake lease, which are proved by tests that fail without the fix; the evacuation hand-off, last run for #437; a real language server, since the legs use manifest stand-ins.

Open and known

  • A sandboxd restart during a hibernate dump leaves the guest paused while it reads running, exec answers 502, and a retried hibernate recovers it. cocoon dies on SIGPIPE and will not ignore it, so sandboxd does not adopt that snapshot.
  • Once cocoon's capture cancel ships, dumpTermGrace (3 min) must exceed its wait for resume (10 min).

CMGS added 30 commits November 4, 2026 08:29
The pump only watched the server's stdout, so a silent server that ignored stdin EOF pinned its memory after the client left. A request frame during the stream ends it the same way, as a pty does.
rev-parse --abbrev-ref fails on an unborn HEAD and printed HEAD, while git_status on the same repo reports the branch. A detached HEAD still reads HEAD.
claim_env keeps its one read buffer and parses into borrowed pairs, so an exec, pty or session drops 2N String allocations. BootCfg and IpParam borrow the cmdline, read_manifest returns its text, apply_config sets literals, pty and upload signatures take plain borrows, and scan_serials formats a path only when it reads it. Tests changed only where a literal's type or a temporary's lifetime named the old owned form.
Child::id is Some right after spawn, a session body is never empty, Path::file_name is never empty, abort on a finished task is a no-op, and the exec rebinds did nothing. Table::new and Broker::new only aliased Default; git hand-rolled error_frame; the imds consts and handle, DEFAULT_INIT and PROTO_VERSION had no outside user; the ignored bench asserted nothing.
Function-local statics and consts move to the top-of-file block, and the one-caller proxy_vars and silkd_net wrappers go with them; test helpers move above the tests and private below public.
Module narratives go to one line, the imds token fact moves to its const, a history comment goes, make_controlling_tty becomes safe because it has no memory precondition, and find's at_root stops shadowing the root param.
The package is fully annotated and mypy-strict, but the wheel carried no marker, so a
consumer's type checker treated every cocoonsandbox import as untyped (reveal_type gave
Any). Add the marker and the setuptools package-data entry; a scratch wheel now lists
cocoonsandbox/py.typed and mypy on a consumer reveals cocoonsandbox.sandbox.Sandbox.
The arm re-spelled serveStateless byte for byte.
…line comments

Own-line comments start uppercase per /code Comment Style; multi-line godocs on unexported names cut to one line.
…block

Layout only: /code File Organization puts utility funcs below the type's methods.
Same verdict as the hand-rolled path test on 15 inputs.
…-level Request

The proxy address was declared in four drivers. Dial returns RelayConn so portsmoke calls CloseWrite directly. PortRelay.Request never read its receiver. credsmoke's expectPlaceholder locals stop shadowing the entry and header consts.
At v0.1.21 "this release" read as every upgrade needing a drain, a fresh data dir and an emptied bucket. Releases come from the first tag holding each commit: #81 v0.1.6 (the pool key's engine axis went), #364 v0.1.18, #59 v0.1.5, #69 v0.1.5 and #72 v0.1.6, #171 and #169 v0.1.11.
…ull ref

A pulled image is keyed by its whole ref, so template base:24.04 fails the golden build with image not found, as the README already fixed in #425. The deploy sample's cluster_key and the egress sample's missing egress_ca failed config.Load.
…n and a dump

A wake of a failed egress-lane or volume claim answers 409, tenants get 403 on two more root routes, golden dumps queue, an idle default acts on running claims at the next sweep, and an adopted replica always wakes on a read; none of these had a line.
A role without CREATE could not enable archive_failover from the docs: the other three shared tables have a pre-create DDL and this one had none.
The checkpoint_store row repeated the Reclaiming S3 storage section, which takes the facts only the row carried (lineage scope, marker order, re-stamped manifests); the TTL, replica-loss, tenant-outage and cluster-invariant passages become links to the pages that own them.
… digest covers secrets

An operator rebuilding data_dir from config.json lost runtime-volumes.json silently, and gauge 0 read as a check of two secrets a keyless digest never covers.
…red usage table

Stated as hard constraints these ruled out what deploy ships: a TLS edge for SDK clients, egress_internal_allow, and meta_store.usage.
…int list as the server does

The mint endpoint never answers the 503 and 429 its paragraph listed: the browser request does. A template another node deleted stays in tplSet, so the cluster page's never-fires claim was false for that entry.
Wake, agent, ports and exec carried the same four tails. Restore failures, 409s, release, hibernate and the preview listener differ, so they keep their own lines.
Go Exec runs the argv given, with no implicit shell; every control-plane verb returns APIError; the checkpoint option refusals and a fork child's env were incomplete.
Go-specific and Python-specific names stay. The settled-entry sentence stays on the Go page, its only source.
CMGS added 29 commits November 4, 2026 08:29
Pure move: the env types, limits and patterns go to env.go and the volume types, modes and mount validation to volume.go, matching the existing env_test.go and volume_test.go. envView and namedVar stay beside the Sandbox that owns them. No line changes beyond imports and gofmt alignment.
relay dropped the copy error, so net/http wrote the terminating chunk and the guest read a partial body as complete. Hijack and close the conn on a copy error, which leaves the chunked stream unterminated. Success costs one nil check.
The close unlinks the socket path, so a re-arm that bound first left the guest with no door while Route reported relay. A re-arm with no policy left now closes the door too, which makes the Disarm in restartLocked dead. The claim path pays one map delete under a lock it already holds.
A dump may take 15 minutes while an eager-copy restore of the same image had 2, so a slow page cache could fail the wake it had just made possible. cocoon gives vm.restore the same limit as vm.snapshot. Clone and cold boot keep the command budget.
ReadSlice caps a line at the reader's buffer, IsGlobalUnicast already excludes link-local and multicast, UnixListener.Close already unlinks, marshalCA's public key is its PEM key's, and interDER copied interCert.Raw.
The only production egress.New takes a claim as its router, so the two methods move to the egress test file.
parseECKey said intermediate key while parsing the root, so IssueIntermediate reported root key: intermediate key. validHostName takes the package's maxHostLen for the same DNS limit.
…oxy.go's other address helpers

installca.go kept the generic guest write and exec helpers every other engine file uses; installOutputMax becomes guestOutputMax since it caps every guest exec.
…udget

cocoon's vm clone and vm restore share restoreVM and its 10 minute memory transfer, and an archived wake and a fork go through clone. A hung refill clone now holds its slot for up to 18 minutes, which stays bounded and loud.
…o, VM provisioning to provision.go, destroyAll and dropSnap beside destroy, runBounded to pool.go with its callers across the package
…ates in template.go and seedVolumes in volume_catalog.go, keeping the order
…atalogSnapshot and the holder bookkeeping move to volume_catalog.go as volumeCatalog methods
…d recLock's doc says it returns the lock without taking it
… retry chain moves with it to archive_delete.go
Evacuate waits for the warm VMs to go before it starts its run. A DELETE /v1/drain in that wait found nothing to stop, and the run then moved every claim off an uncordoned node.
…h lease

The reaper archives a hibernated claim of a tier pool at its lease end, and the same call a moment after the export restores it with a fresh lease. A call in between woke it with the past deadline, so the next reap tick destroyed it.
forgetAdopted removes the VM with no sandbox id, so the removal never disarms. The doors, their listeners and their goroutines stayed until the next restart, one set per moved claim, unlogged.
Close looked the sandbox up only on a 404 from its old owner. Once that node is retired, the first release fails to connect, so the claim stayed on the adopter until its deadline, and every retry hit the same dead address. A connection failure now looks it up too, and keeps its error when no node answers.
@CMGS CMGS changed the title review: whole-repo audit after #436 and #437: evacuate stop, moved claims, docs and moves review: evacuate stop, moved claims, docs and moves Nov 4, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant