Repository navigation
fix(rules): roll back a restore or rollback that fails mid-write - #653
Merged
Merged
Conversation
rule restore and rule rollback wrote their rule in place, so an IO error part-way could leave the new rule file beside the old fixtures while the command failed with INTERNAL_ERROR. They now write through the staged writePreparedRules that rule create uses: a failed move puts the previous version back and the command exits RULE_RESTORE_FAILED. A verified set that is not a writable rule exits RULE_RESTORE_MISMATCH. The directory is replaced whole, so the stale-entry notice branch is gone, and the in-place writeServedRule and writePreparedRule are removed. Fixes #636
Member
Author
|
@claude /review |
Contributor
|
Claude finished @theCodeDrift's task in 3m 39s —— View job Review completeReview mode: incremental — read 0 prior review thread(s) before reviewing.
CorrectnessTraced the new failure path by hand against
Minor, non-blocking observations
Tests / spec
No blocking issues found. |
Member
Author
Thanks. On the two minor observations:
— AI Coding Agent |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
rule restoreandrule rollbackwrote their one rule in place (writeServedRule→writeDeliveredFileSet): delivered files first, then a purge of entries the set did not name. An IO error part-way (a full disk,EACCESon one file) left.taskless/rules/<engine>/<id>/holding a mix of revisions while the command failed withINTERNAL_ERROR.Measured on
mainwith the new test: a rollback over a version whose fixture file is read-only overwrote the rule file with the new revision, then failed on the fixture withINTERNAL_ERROR: EACCES ... tests/fail/case/case.ts. The rule was left at the new revision with the old fixture.rule create/rule improvestopped having this problem in #635, which left restore/rollback on the old path on purpose.What changed
recover.tswrite()now callsprepareServedRule, thenwritePreparedRules(cwd, [prepared], …), the same staged pathrule createuses. The rule is written in full under.taskless/.tmp-delivery-*, then renamed over its directory, with any version already there renamed aside first and renamed back if the move fails.RULE_RESTORE_FAILED(new code intypes/errors.tsand therecover-rulerecipe's error table, topic v5). The message is the staged writer's, which names the errno and says.taskless/rules/is as it was, or names where the previous version was kept if the rollback itself could not finish. This is a new code rather thanRULE_RESTORE_MISMATCHbecause the two need different responses: a mismatch means the bytes were wrong and retrying will not help, while this one means the bytes were right and the command should be run again once the filesystem is fixed.RULE_RESTORE_MISMATCH(an unsafe fixture path, an incomplete set). Fixtures carry no signature, so such a set gets through verification and was only refused byprepareServedRule, as a plainError, so it came out asINTERNAL_ERROR.RULE_UNSUPPORTEDpasses through unchanged.writeServedRuleandwritePreparedRuleare deleted. No production code called them after this change. The tests that usedwriteServedRuleas an entry point now go through a test-only helper (test/support/write-served-rule.ts) that does prepare +writePreparedRules, so those tests now exercise the path the commands actually use.The
PurgeIncompleteErrorbranch: removed, nothing replaces itThat branch meant "the rule is on disk, but a stale entry beside it could not be removed", and returned success with a notice. Under rename-based delivery the old directory is moved aside as a whole and removed with the staging directory, so that state no longer exists. The served set either is the directory, or the directory is what it was before. If the purge ever did throw inside staging (it writes into a fresh directory, so it has nothing to purge),
writePreparedRuleswould treat it like any other write error and roll back. A note onPurgeIncompleteErrorsays so.writeDeliveredFileSet's purge andPurgeIncompleteErrorthemselves are kept. They are now reachable only through that function directly, and the twodeliver.test.tscases that pin the purge call it directly. Removing them, along with the symlink handling that only matters when writing over an existing directory, is a reasonable follow-up. I left it out to keep this PR to the write path.Behaviour differences (same as #635 accepted for
rule create)describeUnsafePath, which the in-place purge skipped.Residual window (not atomic)
Each
renameis atomic on one filesystem, but the pair of renames (old version aside, new version in) is not. A process killed outright (SIGKILL, power loss) between them runs no rollback. The rule can then be absent, with its previous version under.taskless/.tmp-delivery-*, and nothing moves it back automatically. Nothing is fsynced. This is documented onwrite()and in the new spec requirement.Testing
rule-recovery.test.ts, driven through the realrule restore --json/rule rollback --jsonagainst the stubbed v2 service:.taskless/rules/sg/made0o555, so the move fails with a realEACCES): restore of an edited rule, restore of a missing rule, and rollback over the current version with a straynotes.mdbeside it. Each assertsRULE_RESTORE_FAILED, anEACCESmessage, exit code 1,.taskless/rules/byte-identical, and no.tmp-delivery-*left behind. Skipped as root and on Windows.tests/../../escape.tsfixture isRULE_RESTORE_MISMATCH, and nothing is written.main'ssrc/, all four new IO tests fail: the threeRULE_RESTORE_FAILEDcases getINTERNAL_ERROR, or success for the edited-rule case, since the in-place write gets through a read-only engine directory, and the read-only-fixture case gets the half-write described above.restore-rollback-rolls-back-io-failure, archived here. MODIFIEDA delivered rule replaces its directory(drops the stale-entry sentence, adds one scenario) andRecovery commands report under --json(addsRULE_RESTORE_FAILED). Both keep their titles and restate every scenario. ADDEDA restore or rollback that cannot be written leaves the rule as it was. Pre-archive check:cli-generated-rule-delivery25 → 26 scenarios,cli-rule-recovery27 → 32, none dropped.patchchangeset.pnpm typecheck,pnpm lint,pnpm --filter @taskless/cli test(3065 passed, 11 skipped) all pass.Fixes #636