Skip to content

test(volume-cache): force distinct mtime on replaced source - #667

Open
deanq wants to merge 1 commit into
mainfrom
deanq/fix-volume-cache-mtime-flake
Open

deanq wants to merge 1 commit into
mainfrom
deanq/fix-volume-cache-mtime-flake

Conversation

@deanq

@deanq deanq commented Nov 1, 2026

Copy link
Copy Markdown
Member

Problem

test_hydrate_rejects_source_replaced_between_ranges fails intermittently with assert restored == 0 (assert 1 == 0). Observed on Python 3.14 in CI on #665 (run 41127886501, attempts 2 and 3) and on the fix/preserve-empty-endpoint-input branch. It passed in the main runs checked. The test came from #621.

Cause

_file_identity compares only (st_size, st_mtime_ns); it skips the inode on purpose because of FUSE mounts (see its docstring). The test replaces the mirror file with random bytes of the same size, so detection depends only on mtime. sync() copies the source mtime onto the mirror with os.utime, and the replacement is written a few milliseconds later. On a filesystem with coarse timestamps both can land on the same mtime. The swap is then not detected and hydrate publishes the file (restored == 1).

Fix

Test-only. After writing the replacement, set its mtime to the original's plus one second, with a one-line comment. Production identity logic is unchanged: a replacement with the same size and the same mtime remains an accepted blind spot, and widening the identity (for example adding the inode) would reintroduce the FUSE false rejections the docstring describes.

Verification

  • Natural flake did not reproduce locally: macOS APFS has nanosecond mtimes. 0/200 failures before the fix, 0/200 after (Python 3.14.6, single test in a loop).
  • Mechanism reproduced: with the replacement's mtime forced equal to the original's (before the fix), the test fails with assert 1 == 0, the same assertion as CI. _file_identity of two same-size files with equal mtime compares equal; with mtime +1s it differs.
  • Full test file: 118 passed.
  • make quality-check: 2181 passed, coverage gate satisfied.

The hydrate source identity is (st_size, st_mtime_ns). The replacement
in this test has the same size as the file it replaces, so detection
depended on timestamp granularity.
@deanq
deanq marked this pull request as ready for review November 1, 2026 15:13
@deanq
deanq requested review from a team and KAJdev and a balanced review from Copilot November 2, 2026 01:56

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are reported for the test-only adjustment that gives the replacement file a distinct modification time.

No open issues 🎉

What changed in this PR

Stabilizes the VolumeCache source-replacement test by ensuring the replacement has a distinct modification time, avoiding intermittent failures on filesystems with coarse timestamps.

Changes:

  • Sets the replacement file’s mtime to the original’s plus one second.
  • Documents why the explicit timestamp is needed; production identity logic remains unchanged.
File Description
tests/​test_serverless/​test_utils/​test_rp_volume_cache.py Forces a distinct replacement mtime in the between-ranges source-replacement test.

Review effort: Balanced

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants