[ThomasWaldmann]

The files cache (files.<suffix> in the local cache directory) was stored as plaintext msgpack: file sizes, timestamps, inode numbers, chunk lists and, with borg create --digests, unkeyed digests of the whole file contents. Its only protection was a checksum without a key in the cache config. So anyone who could read the cache directory could read that metadata, and anyone who could write to it could make borg treat changed files as unchanged.

This PR protects the files cache with the repository key, like the index/ and cache/ objects in the repository (#9819, #10235).

Fixes #5041.

Changes

Commit 1: sealed streams (borg.crypto.sealed_stream)

A reusable layer that cuts a byte stream into 1 MiB frames and puts every frame into the key's envelope. Frames are encrypted and authenticated in the encrypting modes, authenticated only in the authenticated-* modes.

stream := frame+
frame  := flag (1 byte: 0 = more, 1 = last) + length (4 bytes LE) + envelope
AAD    := caller context + frame number (8 bytes BE) + flag

• The reader rejects:
  • changed, reordered, dropped or duplicated frames;
  • a stream that ends before its last frame;
  • a "last" flag on an earlier frame;
  • data after the last frame;
  • an envelope larger than the frame size limit.
• Not detected: replacing a whole stream with an older stream that has the same context. Each user of the layer decides whether that matters.
• Every frame is sealed with encrypt_oneshot(): its own random session id, IV 0. That makes the layer thread-safe. It costs about 3 µs extra per frame, so about 3 ms per GiB, and nothing extra when reading.
• Why frames rather than one envelope: one envelope needs the whole serialized cache in memory 2-3 times while writing it. That measured +3 GiB peak RSS for a 1 GiB cache, at Cache.close(), when the in-memory files cache and chunks index are still loaded.
• The encrypt_oneshot() docstring is reworded: frames of about 1 MiB are fine, millions of chunks are not.

Commit 2: the files cache uses it

• Context: b"borg-files-cache\0" + repository id + the file name of the files cache. A files cache copied from another repository (with the same key material) or from another archive series is rejected.
• A files cache that fails the verification is ignored with a warning; the exit code stays the same. Then it is rebuilt from the previous archive of the series, as for a missing files cache. A plaintext files cache from an older borg 2 beta takes this path once.
• Putting back an older sealed copy of a files cache is not detected, but harmless. Its entries were correct when written, and entries that refer to chunks no longer in the repository are dropped when loading.
• The [integrity] checksum section of the cache config is removed (IntegrityCheckedFile stays for reading borg 1.x repos). Nothing in the config changes after its creation any more, so Cache.close() no longer rewrites it, and the "Saving cache config" progress message is gone.
• File names are unchanged: files.<store hash of the series name>.
• Docs:
  • data-structures.rst: new "Sealed streams" section, files cache on-disk format, the checksumming section (now only about borg 1.x repos);
  • security.rst;
  • the cache security FAQ (it said the cache is not encrypted);
  • the frontends.rst progress example.

Measurements

Serialization layer only, 1M files cache entries (98 MiB on disk), on a ramdisk:

	write	read	extra peak RSS
master (IntegrityCheckedFile)	1.25 s	0.39 s	0 MiB
this PR (sealed stream, AES-OCB)	0.89 s	0.36 s	5-7 MiB

Writing got faster, because IntegrityCheckedFile hashed every small write() separately, while the new writer collects data into 1 MiB frames first.

Tests

• testsuite/crypto/sealed_stream_test.py: round trips at and around the frame boundaries, the frame size limit, and the one-off sessions.
  • Error cases: tampering in each region of a frame, truncation, trailing data, reordered or duplicated frames, a forged "last" flag, a context mismatch, and writer misuse.
  • Run with AES-OCB, ChaCha20-Poly1305 and both authenticated-* key types.
• cache_test.py: a round trip over many frames, an empty cache, a missing cache (no warning), and that no plaintext is left in the file.
  • Rejected files caches: tampered, from another series, from another repository, plaintext, and authentic but invalid content.
  • The cache config has no [integrity] section and is not rewritten by close().
• create_cmd_test.py: a corrupted files cache gives a warning and rc 0, and the unchanged file is still found unchanged.

Local run on macOS: full suite green, except test_migrate_lock_alive[archiver], a FUSE mount timeout that also fails locally without this change.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 97.41379% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.25%. Comparing base (f3c82cd) to head (185c5a3).
⚠️ Report is 37 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/borg/cache.py	92.30%	2 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10497      +/-   ##
==========================================
+ Coverage   89.06%   89.25%   +0.18%     
==========================================
  Files         103      104       +1     
  Lines       19685    19893     +208     
  Branches     3081     3118      +37     
==========================================
+ Hits        17533    17756     +223     
+ Misses       1490     1478      -12     
+ Partials      662      659       -3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.