feat: fold Codex-compressed rollouts, fold on a tight disk, and fix mount descriptor exhaustion

CHANGELOG.md

@@ -20,6 +20,40 @@ pre-1.0, so a minor version may contain compatibility changes.
   integration tests in addition to compiling every package. The required check
   keeps its existing `windows-compile` name. This is not a full Desktop GUI or
   power-loss qualification.
+- Codex-compressed rollouts are folded too. Codex now compresses every rollout
+  idle for seven days to a sibling `rollout-*.jsonl.zst` (zstd level 3) and
+  deletes the `.jsonl`, which pulled sessions out of the store one by one: on
+  a real machine 1,288 managed sessions left this way, the saved figure fell
+  from 69% to 38%, and 3,623 `.zst` files held 24.2 GB that CodexFold never
+  touched. The service now folds each settled `.zst` in place: it proves that
+  libzstd 1.5.7 (the version Codex links, built in through
+  `github.com/DataDog/zstd`) re-encodes the decoded content to the file's exact
+  bytes, folds the content into the store, writes a marker attribute and
+  releases the file's data blocks with a hole punch. Path, inode, size, mode,
+  times and other attributes stay. The mount re-encodes the original bytes on
+  read, refuses writes, hides the marker, and fails with EIO rather than serve
+  the empty blocks if it cannot. All 3,623 real files reproduced byte for byte.
+  On the same 3,280 sessions the store needs 12.5 GB where Codex's per-file
+  zstd needs 18.2 GB, because forks and subagents repeat long prefixes.
+  A dry run prices every file first, and a file is folded only when that
+  costs under 80% of its `.zst`: a rollout that shares nothing with the store
+  folds larger than Codex's own zstd (one real 205 MB rollout: 83.5 MB folded
+  against a 68.6 MB `.zst`), so it stays as it is. Folding follows the
+  automatic-folding policy and never runs alongside an enrollment cycle. Manifests whose file Codex deleted are released only after
+  two scans fifteen minutes apart; a file that lost its marker is found again by
+  inode and re-marked instead.
+- `fs native-zst status` reports folded and unfolded files. `fs native-zst
+  restore --all --apply` writes the original bytes back; run it, with
+  automatic folding disabled, before installing a CodexFold build without this
+  support, which would otherwise serve the empty blocks.
+
+### Changed
+
+- The storage figures (`storage.json`, the saved percentage) now count the
+  native session tree as well as the store: logical bytes are all session
+  content, decoded, and physical bytes are everything on disk. Before, native
+  `.jsonl` and `.zst` files were left out, which hid how much space Codex's
+  sessions really used.
 
 ### Fixed
 
@@ -65,6 +99,63 @@ pre-1.0, so a minor version may contain compatibility changes.
   repaired is logged and served unchanged instead of aborting startup.
 - `fs validate-native --repair` applies the same repair by hand while the
   filesystem service is stopped. Without the flag the command stays read-only.
+- Deleting sessions now shrinks the store. Maintenance only rebuilt the pack
+  when a top-level manifest was newer than it, so a session Codex or the user
+  deleted kept its objects in the pack indefinitely. A manifest the published
+  pack still carries but the store no longer has now triggers a compacting
+  build, and nested manifests count as new.
+- Folding works on a nearly full disk. Every operation must leave the 5 GiB
+  free-space reserve at its peak, and space it frees only counts afterwards,
+  so pessimistic peaks blocked the very work that frees space:
+  - A Codex `.zst` is folded straight from its decoder instead of staging its
+    decoded content (up to 2.7 GB for one real rollout). When space is short a
+    dry run prices the fold exactly, so a file whose content the store already
+    holds costs almost nothing. Such a fold may run below the reserve, down to
+    1 GiB free, when its peak is under 64 MiB and it frees at least four times
+    that; files that do not fit yet are retried, not given up.
+  - A pack build no longer copies the whole pack when objects were dropped.
+    It rewrites only the old pack files whose dead and raw blocks make it
+    worthwhile, best payoff per copied byte first and only as many as fit the
+    free space, and hard-links the rest. A file with no live block is dropped
+    without copying anything. Dead pack bytes trigger another build once there
+    is room to make progress.
+  - Session enrollment priced every rollout at three times its raw size, so
+    on a tight disk a large session was never enrolled even when it was a
+    fork whose content the store already held. When that price does not fit,
+    the planner now asks for the exact price: a dry-run fold that writes
+    nothing, cached by file identity and pack generation, charged twice (loose
+    objects and the next pack). `codexfold fold --apply` measures the same way
+    when space is short, and pack builds price loose objects by their
+    compressed size on disk instead of their raw size.
+- Loose objects are compressed at the pack builder's level through one shared
+  encoder instead of a new encoder, with its own worker goroutines and
+  window, per object.
+- Pack builds entropy-code small blocks. `SpeedDefault` gave up on blocks whose
+  only redundancy is entropy (base64 payloads, encrypted reasoning) and a 2:1
+  cut-off kept other small blocks raw: 7.07 GB of a real 14.83 GB pack. Builds
+  use `SpeedBetterCompression`, keep a block raw only when compression does not
+  shrink it, and recompress raw blocks during compaction; about 2.1 GB of that
+  pack goes away. Decoding cost is unchanged.
+- Mount reads no longer fail with "Too many open files" while several folded
+  `.zst` files are read at once. The FSKit extension ran at launchd's
+  256-descriptor limit, and a folded file used about fourteen of them because
+  it was handed a shared read window that primed nine prefetch channels on
+  open. Folded files now get no shared window. The extension raises its
+  descriptor limit at start (1,200 files held open through a real mount put
+  1,438 descriptors on it without an error) and caps prefetch channels at 48
+  across all open files. The extension change needs the App build 117 install,
+  not just a daemon live update.
+- A folded `.zst` from which APFS kept a few extents allocated was re-proved
+  and re-punched every thirty seconds. It is now left alone until it changes,
+  and only bytes actually released are counted as freed.
+- A managed session's native snapshot left behind after deletion no longer
+  blocks readiness. The mount hides that path on purpose, but the canonical
+  readiness probe waited for it. As a result, automatic enrollment reported
+  `mount-warming` for every session (1,626 on a real machine) and enrolled
+  nothing, and `fs service start` failed its health gate. The probe now skips
+  the routes of deletion tombstones. A failed native FSKit start also left the
+  mount attached with no backend, so every read in it failed; it now reclaims
+  the mount the way an explicit stop does.
 
 ## [0.4.0-beta.2] - 2026-10-01
 

go.mod

@@ -3,6 +3,7 @@ module github.com/samekind/codexfold
 go 1.26
 
 require (
+	github.com/DataDog/zstd v1.5.7
 	github.com/Microsoft/go-winio v0.6.2
 	github.com/jchv/go-webview2 v0.0.0-20260205173254-56598839c808
 	github.com/klauspost/compress v1.19.0

go.sum

@@ -1,3 +1,5 @@
+github.com/DataDog/zstd v1.5.7 h1:ybO8RBeh29qrxIhCA9E8gKY6xfONU9T6G6aP9DTKfLE=
+github.com/DataDog/zstd v1.5.7/go.mod h1:g4AWEaM3yOg3HYfnJ3YIawPnVdXJh9QME85blwSAmyw=
 github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
 github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
 github.com/cpuguy83/go-md2man/v2 v2.0.6/go.mod h1:oOW0eioCTA6cOiMLiUPZOpcVxMig6NIQQ7OS05n1F4g=

internal/cli/fold.go

@@ -54,6 +54,17 @@ func newFoldCommand() *cobra.Command {
 				defer recordIndex.Close()
 				options.RecordIndex = recordIndex
 			}
+			// On a nearly full disk the default price, every source byte new,
+			// rejects folds that add little. Measure what this one adds first.
+			if options.Apply && options.NewBytesEstimate == 0 {
+				if info, err := os.Stat(session.RolloutPath); err == nil && foldSpaceIsTight(options.StoreDir, info.Size()) {
+					newBytes, err := dryRunFoldBytes(command.Context(), options.StoreDir, toFoldSession(session), options)
+					if err != nil {
+						return err
+					}
+					options.NewBytesEstimate = max(newBytes, 1)
+				}
+			}
 			result, err := fold.Fold(command.Context(), toFoldSession(session), options)
 			if err != nil {
 				return err

internal/cli/fs.go

@@ -21,13 +21,15 @@ import (
 
 	"github.com/samekind/codexfold/internal/cdc"
 	"github.com/samekind/codexfold/internal/codex"
+	"github.com/samekind/codexfold/internal/codexzstd"
 	"github.com/samekind/codexfold/internal/compat"
 	"github.com/samekind/codexfold/internal/enroll"
 	"github.com/samekind/codexfold/internal/fold"
 	"github.com/samekind/codexfold/internal/fsctl"
 	"github.com/samekind/codexfold/internal/fskitproto"
 	"github.com/samekind/codexfold/internal/fskitstatus"
 	"github.com/samekind/codexfold/internal/mountfs"
+	"github.com/samekind/codexfold/internal/nativezst"
 	"github.com/samekind/codexfold/internal/pack"
 	"github.com/samekind/codexfold/internal/scan"
 	"github.com/samekind/codexfold/internal/service"
@@ -121,6 +123,7 @@ func newFSCommand() *cobra.Command {
 	command.AddCommand(newFSStatusCommand())
 	command.AddCommand(newFSDoctorCommand())
 	command.AddCommand(newFSValidateNativeCommand())
+	command.AddCommand(newFSNativeZstCommand())
 	command.AddCommand(newFSCompatibilityCommand())
 	command.AddCommand(newFSCompatibilityImportCommand())
 	command.AddCommand(newFSBenchmarkCommand())
@@ -718,9 +721,15 @@ func newFSServeCommand() *cobra.Command {
 				}
 			}
 			filesystem := mountfs.New()
+			var nativeZstSource *nativezst.Source
 			if canonicalNamespace {
 				filesystem = mountfs.NewCanonical()
 				filesystem.SetNativeRoot(nativeRoot)
+				// Serve Codex .zst files folded into the store; a build that
+				// cannot re-encode them reports EIO, never their empty blocks.
+				nativeZstSource = nativezst.NewSource(store, storePacks{})
+				defer nativeZstSource.Close()
+				filesystem.SetNativeZstSource(nativeZstSource)
 				if err := filesystem.SetManagedStore(store); err != nil {
 					return fmt.Errorf("recover managed file names: %w", err)
 				}
@@ -1328,6 +1337,10 @@ func newFSServeCommand() *cobra.Command {
 				if runtime.GOOS == "windows" {
 					storageStatusPath = filepath.Join(store, "fs", "status", "storage.json")
 				}
+				storageScan := storage.Scan
+				if canonicalNamespace {
+					storageScan = withNativeFootprint(nativeRoot, storage.Scan)
+				}
 				storageStatusDone = startStorageStatusReporter(
 					ctx,
 					command.ErrOrStderr(),
@@ -1336,7 +1349,7 @@ func newFSServeCommand() *cobra.Command {
 					10*time.Minute,
 					storageMaintenanceDone,
 					storageStatusRefresh,
-					storage.Scan,
+					storageScan,
 					fskitstatus.Write,
 				)
 				activityCounter = &mountfs.IOActivityCounter{}
@@ -1345,6 +1358,17 @@ func newFSServeCommand() *cobra.Command {
 			namespaceMaintenanceDone := startNamespaceRetirement(ctx, filesystem, store, func() bool {
 				return namespaceFoldingEnabled(flags, enrollmentInterval, runtime.GOOS == "windows")
 			}, command.ErrOrStderr())
+			var nativeZstDone <-chan struct{}
+			if nativeZstSource != nil && codexzstd.Supported() {
+				nativeZstDone = startNativeZstFolding(ctx, filesystem, nativeZstSource, store, func() bool {
+					return namespaceFoldingEnabled(flags, enrollmentInterval, false)
+				}, func() {
+					select {
+					case storageStatusRefresh <- struct{}{}:
+					default:
+					}
+				}, command.ErrOrStderr())
+			}
 			watcherDone := make(chan struct{})
 			go func() {
 				defer close(watcherDone)
@@ -1380,6 +1404,9 @@ func newFSServeCommand() *cobra.Command {
 			}
 			<-runtimeMemoryMaintenanceDone
 			<-namespaceMaintenanceDone
+			if nativeZstDone != nil {
+				<-nativeZstDone
+			}
 			<-enrollmentDone
 			var nativeWatcherErr error
 			if nativeWatcherDone != nil {

internal/cli/fs_enroll.go

@@ -584,6 +584,9 @@ func runPeriodicEnrollment(ctx context.Context, flags enrollmentFlags, interval
 				}
 			}
 		}()
+		// Folding Codex .zst files adds manifests; one appearing between this
+		// cycle's pack build and its loose retirement fails the pack-only proof.
+		storeMaintenanceMu.Lock()
 		result, err := runServiceEnrollmentCycle(cycleCtx, cycleFlags, enrollmentApplyHooks{
 			onManagedCount: func(count int) {
 				progressMu.Lock()
@@ -607,6 +610,7 @@ func runPeriodicEnrollment(ctx context.Context, flags enrollmentFlags, interval
 				return !resolveEnrollmentControl(flags, interval).Enabled
 			},
 		})
+		storeMaintenanceMu.Unlock()
 		cycleCancel()
 		<-watchDone
 		if report != nil {
@@ -987,7 +991,7 @@ func buildEnrollmentPlan(ctx context.Context, flags enrollmentFlags) (enroll.Pla
 	mountHealthy := mountHealthProbe(mount) == nil
 	readiness := canonicalNamespaceReadiness{}
 	if flags.canonicalNamespace && mountHealthy {
-		readiness = enrollmentCanonicalNamespaceReadinessProbe(home, mount, nativeRoot)
+		readiness = enrollmentCanonicalNamespaceReadinessProbe(home, store, mount, nativeRoot)
 	}
 	guard, err := storage.DefaultGuard(store)
 	if err != nil {
@@ -1015,6 +1019,7 @@ func buildEnrollmentPlan(ctx context.Context, flags enrollmentFlags) (enroll.Pla
 		// occurs between those decisions. Reuse one inventory for this plan;
 		// fold/pack/apply retain their fresh per-operation budget checks.
 		Budget: &storage.OnceInventoryGuard{Guard: guard},
+		Price:  enrollmentDryRunPricer(store),
 	}
 	plan, err := enroll.Build(ctx, input)
 	if err != nil || len(plan.Selected) == 0 {
@@ -1310,25 +1315,80 @@ func enrollmentHasManifestsNewerThanPack(store string) (bool, error) {
 		return false, err
 	}
 	packPresent := err == nil
-	entries, err := os.ReadDir(filepath.Join(store, "manifests"))
-	if errors.Is(err, os.ErrNotExist) {
-		return false, nil
-	}
-	if err != nil {
-		return false, err
-	}
-	for _, entry := range entries {
+	// Manifests below manifests/ (folded Codex .zst files) are live too.
+	pending := false
+	live := make(map[string]struct{})
+	err = filepath.WalkDir(filepath.Join(store, "manifests"), func(path string, entry fs.DirEntry, walkErr error) error {
+		if walkErr != nil {
+			return walkErr
+		}
 		if entry.IsDir() || filepath.Ext(entry.Name()) != ".json" {
-			continue
+			return nil
 		}
+		relative, err := filepath.Rel(store, path)
+		if err != nil {
+			return err
+		}
+		live[filepath.ToSlash(relative)] = struct{}{}
 		if !packPresent {
-			return true, nil
+			pending = true
+			return nil
 		}
 		info, err := entry.Info()
 		if err != nil {
-			return false, err
+			return err
 		}
 		if info.ModTime().After(current.ModTime()) {
+			pending = true
+		}
+		return nil
+	})
+	if errors.Is(err, os.ErrNotExist) {
+		return false, nil
+	}
+	if err != nil || pending || !packPresent {
+		return pending, err
+	}
+	// A manifest the pack still carries but the store no longer has (a
+	// deleted session, a released .zst) leaves its objects in the pack until
+	// a build compacts them away. Without this nothing ever repacks after
+	// deletions alone.
+	if dropped, err := packHasDroppedManifests(store, live); err != nil || dropped {
+		return dropped, err
+	}
+	return packWasteWorthCompacting(store), nil
+}
+
+// packWasteWorthCompacting reports dead pack bytes that a build could now
+// free. On a nearly full disk a build rewrites only what fits, so the rest
+// is picked up by later cycles once space allows, not retried in vain.
+func packWasteWorthCompacting(store string) bool {
+	files, wasted, err := pack.GenerationWaste(store)
+	if err != nil || wasted < max(int64(512<<20), files/10) {
+		return false
+	}
+	limits, err := storage.LoadLimits(store)
+	if err != nil {
+		return false
+	}
+	available, err := storage.AvailableBytes(store)
+	return err == nil && available-limits.FreeSpaceReserveBytes >= 1<<30
+}
+
+func packHasDroppedManifests(store string, live map[string]struct{}) (bool, error) {
+	generation, err := pack.CurrentGeneration(store)
+	if err != nil {
+		return false, nil
+	}
+	catalog, err := pack.ReadRecoveryCatalog(filepath.Join(store, "packs", generation))
+	if err != nil {
+		return false, nil
+	}
+	for _, file := range catalog.Files {
+		if !strings.HasPrefix(file.Path, "manifests/") {
+			continue
+		}
+		if _, ok := live[file.Path]; !ok {
 			return true, nil
 		}
 	}