Repository navigation
fix(media-use): serialize concurrent preference records - #5366
miguel-heygen merged 17 commits into
Conversation
jrusso1020
left a comment
There was a problem hiding this comment.
Thanks, this fixes #5333 and I'd like to land it. I'll approve as soon as the conflict below is resolved.
What I checked:
prefs-store.concurrent.test.mjs5/5 passes at head, and 17/17 together with the existing prefs-store tests. With main'sprefs-store.mjsswapped back in, all 5 new tests fail in 5 of 5 runs. The ready barrier keeps the 8-writer race tight, and the held-lock cases are deterministic.- An out-of-tree probe ran 25 trials of 8 parallel
node prefs.mjs recordprocesses (2 projects × 4 keys). Main had 8 ENOENT exits, 18 lost project keys and 14 lost user sightings, with 13/25 trials losing data. This head had no failures, no lost keys or sightings, and no leftover.tmp/.lockfiles. - The standalone and CLI
prefs-store.mjscopies are byte-identical, and the manifest hash at head matches the tree.
Before merge: the branch conflicts with main only in skills-manifest.json. Please rebase and rerun bun run --cwd packages/cli gen:skills-manifest, since main's media-use hash has moved since your base.
Non-blocking notes:
prefs-store.mjs:79-80: thefinallyrmSync(tmp, { force: true })is unguarded. An EPERM/EBUSY there, most likely on Windows, would mask the original error or fail a write that already succeeded. Wrapping it in try/catch would be safer.withFileLocknever takes over a lock. Arecordkilled inside the critical section leaves.media/preferences.json.lock, and later records for that project fail with "Settings are locked…" until someone deletes the file. That matches the helper's config.json policy and your abandoned-lock test, so I'm fine with it. It may be worth a mention in the prefs docs, and maybe.media/*.lockin the project's ignore guidance, since.media/is committed.
This supersedes #5460, which uses the same lock but only patches the standalone copy.
— Rames
jrusso1020
left a comment
There was a problem hiding this comment.
Approving at e4c6891f. Thanks for working through both notes. The conflict is resolved and both non-blocking points are addressed.
Since my last review at 4b778034:
- Merged current
main(d5cdf418). The only conflict wasskills-manifest.json, andgen-skills-manifest.ts --checknow reports it in sync (21 skills).mainhadn't touchedprefs-store.mjs,config-lock.mjsormemory.mdsince your base, so the merge didn't change the fix. prefs-store.mjs:79-82: thefinallycleanup is now wrapped in try/catch. I checked the four new EPERM/EBUSY tests inprefs-store.test.mjsby putting the unguardedrmSyncback. All four failed: two because a successful write turned into a failure, and two because the cleanup error hid theENOSPC. With your change they pass.skills/media-use/references/memory.md: it now documents recovery from an abandonedpreferences.json.lockand the.media/*.lockignore guidance. The claim that "a locked personal tier does not fail the project record" matchesrecordPreference, which catches the error fromrecordUserSighting(prefs-store.mjs:191-195).
At this head, the CLI and standalone prefs-store.mjs files are byte-identical, and prefs-store.test.mjs plus prefs-store.concurrent.test.mjs pass 21/21.
CI: the one red job, CLI: npx shim (ubuntu-latest), was cancelled because bun install hung for 10 minutes. Its macOS and Windows runs passed, so it only needs a rerun.
— Rames
|
Thanks for this, @user-github-me. This PR now has merge conflicts with |
Preview approval requiredThis pull request comes from a fork, so Mintlify won't build a preview until someone with write access approves commit |
miguel-heygen
left a comment
There was a problem hiding this comment.
Thanks for rebasing this. Since the earlier approval was on an older commit, I compared the fix itself across the two heads: both prefs-store.mjs copies, the tests and the skill reference are unchanged line for line, and the merge keeps main's newer media-home handling around them. The only new piece is the "Remember confirmed preferences" section in the media guide, and it matches the code: the lock file name and recovery steps, the .media/*.lock ignore advice, and the fact that a locked personal tier does not fail the project record.
CI has no failures at this head, including the Windows lanes, and the branch merges cleanly with main.
Verdict: APPROVE
Reasoning: The previously approved fix is unchanged after the rebase, the new docs section is accurate, and CI is green with no conflicts.
miguel-heygen
left a comment
There was a problem hiding this comment.
Re-checked at 6acf696b after the merge with main. Compared with f072486a, the only change in this PR's own diff is the regenerated media-use hash in skills-manifest.json, which is expected since main changed other media-use files. Both prefs-store.mjs copies, the tests and the docs are identical, and "Skills: manifest in sync" passes at this head. No failing checks so far; a few jobs were still running when I looked.
Verdict: APPROVE
Reasoning: The approved fix is unchanged; the new commits only bring in main and refresh the manifest hash, which CI confirms is in sync.
|
This was approved, but it conflicts with main again, most likely the skills manifest hash. Could you rebase on current main, regenerate the manifest, and push? We'll re-approve at the new head. — Rames |
miguel-heygen
left a comment
There was a problem hiding this comment.
Re-checked at e46f584 (the changes since the earlier approvals are the rebase and the regenerated manifest). I ran prefs-store.concurrent.test.mjs and prefs-store.test.mjs: 21 passed. With both prefs-store.mjs copies put back to the merge base, 5 fail, so the tests guard the change.
Verdict: APPROVE
Reasoning: same fix as the approved head, tests fail without it, CI green.
Merge 2 more upstream commits (7d6c02f..acd0bc3: heygen-com#5366, heygen-com#3943). Regenerate skills-manifest.json for the media-use hash conflict. Record the 10 open PRs heygen-com#6430-heygen-com#6439 in docs/DECISIONS.md (none adopted; heygen-com#6436 cannot be copied because the fork wraps authored scripts in Function()) and advance the commit watermark to acd0bc3 and the PR watermark to heygen-com#6439. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Concurrent preference records can overwrite each other before a confirmed preference reaches the personal tier. Serialize each read-modify-write with the existing per-file lock so every project preference and personal sighting survives. Clean up temporary files without hiding the original write error.
The packaged and standalone copies stay identical. The media guide and skill reference explain abandoned-lock recovery, transient lock files, and why a locked personal tier leaves the project record intact.
Fixes #5333.
Validation:
f21c3e3eand regenerated the seven-skill manifest. The previously reviewed code, tests, guide and reference remain byte-identical.