Skip to content

pool: one rule per live volume invariant, whole-repo audit - #398

Merged
CMGS merged 5 commits into
mainfrom
review/volume-boundary-audit
Oct 29, 2026
Merged

CMGS merged 5 commits into
mainfrom
review/volume-boundary-audit

Conversation

@CMGS

@CMGS CMGS commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Whole-repo audit after #380–#397, with the live volume design points from the #382/#383 reviews settled.

Removed

  • The live-writer replica policy: ErrReplicaPresent (503 + Retry-After), ErrReplicatingPool, ErrReplicaStranded, liveWriterErrorLocked at two call sites, claimStore.hasCheckpointReferences, and the two log exceptions for them. The existing rule already covers the case: a claim holding volumes is never replicated, publication rechecks holds (replicate and putRow), and the scheduler retires the replica on its next pass (TestReplicaClaimThatGainsVolumesStopsBeingReplicated). A read-only live addition already took that path; refusing only writers had no basis.
  • The temporary writer hold around the live dirty-marker unlink. Its race needs a detach to stall between its final check and an unlink across a whole release (VM removal) plus a new writer admission.

Unified

  • One clean-unmount predicate (cleanMount: whole filesystem, writable mount and superblock options, no other mount of the device) for live detach and release. Release reads mountinfo once, takes the device from the mount entry and unmounts (two guest calls); live detach keeps its device lookup because it must refuse unplugging a device mounted elsewhere. Before, release cleared the marker after any successful umount. Tests: TestReleaseKeepsMarkerAfterUncleanUnmount, TestUnmountVolumeReportsCleanWritableUnmount.
  • Bug found on hardware, also on main: after an ext4 error with errors=remount-ro, current guest kernels report the superblock as rw,emergency_ro, not ro, so both release and live detach cleared the marker of a failed filesystem. emergency_ro now counts as read-only.

Simplified

Kept, with reasons

  • Persisted attach_only: the wire needs it for GET-to-PUT, and claimDTO shares types.Volume; a separate type costs more than the 5-line load migration.
  • putRow's holds guard: it is the one place every replica row publish passes, so reputArchive cannot republish a volume-bearing row.

Hot path: claims and releases without writable mounts are unchanged (A/B on the dev box: no-volume release 15.48 vs 15.45 ns, volume admit+release 3.584 vs 3.598 µs, p > 0.4). A release with writable mounts makes two guest calls instead of one.

Hardware (cocoon-test2 and cocoon-test1, cocoon v0.7.4 isolated roots, rt:24.04 sha256:03e8bed49d4c, main 5e9cded vs this branch, arms interleaved):

  • sandboxd-e2e.sh with the read-only, writable and live volume legs passes on every arm on both hosts (23.9–24.4 s on test2).
  • Runtime catalog PUT/DELETE 204 on every arm.
  • trigger_fs_error on a writable volume (superblock rw,emergency_ro): main clears the dirty marker on release and on live removal on both hosts; this branch keeps it in both paths on both hosts.
  • Writable release, test2, n=60 per arm: main p50 196.6 / p90 214.5 ms, branch 195.8 / 208.9 ms. An earlier build of this branch that reused live detach's full check measured +19 ms p50; that is what the release split fixes.
  • Live add / remove, n=20 per arm: 215 / 151 ms main, 224 / 151 ms branch (noise; the live path is unchanged).

Audit scope: the 33 production files the hygiene ledger listed as changed since their last clean read, plus #396. store/s3 and metastore (#384), silkd pty.rs/sysutil.rs (AsyncFd), and #396 (no_rx_splice) needed no change; comments added since the last read are all one-line docs on exported items or SAFETY lines.

Net: prod −24, test −100, docs −5.

Gates: golangci-lint v2.13.2 0 issues. ×10 (5 modules, linux and darwin), fmt clean, go test -race in every Go module, 257 Python tests, asl shows only the five forwarders main already has.

CMGS added 5 commits October 29, 2026 15:50
Drop the live-writer replica policy (three errors, a 503 retry contract and a checkpoint-reference probe): a claim holding volumes is already never replicated, publication rechecks holds, and the scheduler retires the replica on its next pass. Release and live detach now share the engine's clean-unmount rule, so a writable superblock remounted read-only keeps the dirty marker on release too. Drop the temporary writer hold around the live marker unlink; it guarded a detach stalled between its final check and an unlink across a whole release and new writer admission.
Inline the live-addition hold helper back into stageVolumeTarget now that it carries no replica check, return SetVolumes' target as is since ValidateVolumes copies it, and name the shared unmount helper for both of its callers.
On current guest kernels an ext4 error with errors=remount-ro leaves the superblock options at rw,emergency_ro instead of ro, so release and live detach cleared the dirty marker of a filesystem that had failed. Found on cocoon-test2 with trigger_fs_error.
…ker hold

The marshal test fed invalid UTF-8 that the HTTP decoder already rejects, and the marker-unlink test came with the temporary writer hold this branch removes; a failed unlink only keeps the marker.
Release reused live detach's device lookup and post-unmount rescan, six guest calls per writable mount, and cost cocoon-test2 +19 ms p50 per writable release. Release only needs the clean verdict, so it reads mountinfo once, takes the device from the mount entry, and checks options, superblock and other mounts of that device with the predicate live detach shares.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant