Skip to content

feat(verify): report one status field per rule - #629

Merged
theCodeDrift merged 2 commits into
mainfrom
feat/627-verify-status-field
Nov 8, 2026
Merged

theCodeDrift merged 2 commits into
mainfrom
feat/627-verify-status-field

Conversation

@theCodeDrift

Copy link
Copy Markdown
Member

What changed

A rule's verify result had four outcomes (passed, failed after its tests ran, failed before they ran, refused by the execution policy) spread across three fields: ok, ran and refused. Each consumer combined them with its own precedence, and commands/verify.ts did not check ran at all. A consumer reading ok alone could report a rule that never ran as passing.

This adds one discriminator, computed once by the producer:

  • status: "passed" | "failed" | "not-run" | "refused" on every RuleVerification and in the verify --json schema (schemas/rule-check.ts), described as the field to branch on.
  • One constructor, verification() in rules/inspect.ts, builds every RuleVerification. It derives ok, ran and refused from status, so ok: true with ran: false (or ok: true with refused) has no spelling any more.
  • Consumers read status: commands/verify.ts (marks, summary counts, exit code) and report/draft.ts (the report's test status) no longer reference ran or refused. Behavior is unchanged: not-run still counts as a failure (exit 1, ✗, report fail), and refused stays its own outcome (○, exit 2, report refused).
  • Additive: ok, ran and refused are still emitted for existing --json consumers; their schema descriptions now say they are derived from status.
  • validate runs no tests and does not emit status; the schema says so.
  • Recipes: validate-rule (topic v10), create-sg-rule (v13) and create-vale-rule (v20) name status.
  • OpenSpec: verify-status-field adds the requirement "Verify reports one outcome per rule" to cli-rule-validation (ADDED, so no existing scenario is touched) and is archived in this PR. Single-PR delivery shape.
  • Changeset: patch for @taskless/cli (0.y.z, additive field).

How it was tested

  • New test/verify-status.test.ts: for all four outcomes, ok/ran/refused agree with status, refused is absent unless refused, and status survives the --json output schema.
  • End to end against the built CLI: status is passed and failed for sg rules, not-run for an sg rule validate rejects (exit 1) and for a Vale rule stopped at validate, refused for a runtime rule without --dangerously-run-scripts, failed for a runtime rule whose fail case did not fire, and absent from validate --json.
  • pnpm typecheck, pnpm lint and pnpm --filter @taskless/cli test (154 files, 2976 tests) pass. The archived spec keeps all 61 prior scenarios and adds 5.

Fixes #627

@theCodeDrift

Copy link
Copy Markdown
Member Author

@claude /review

@github-actions

github-actions Bot commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Claude finished @theCodeDrift's task in 2m 37s —— View job


Review complete

Review mode: incremental — read 0 prior review thread(s) before reviewing. .prior-review.json held no review threads, review summaries, or top-level comments beyond the triggering @claude /review comment itself, so this is the first review of this PR: the whole diff was assessed from scratch, nothing to carry forward as addressed/still-open.

  • Read prior review state (.prior-review.json)
  • Fetch PR diff, commits, file list
  • Read changed source files in full where needed
  • Review diff for correctness, security, performance, test adequacy, clarity
  • Post inline comments for concrete issues (none found)
  • Post final top-level summary

Assessment

This is a clean, well-scoped PR. I read packages/cli/src/rules/inspect.ts, src/commands/verify.ts, src/report/draft.ts, and src/schemas/rule-check.ts in full, plus the updated recipes and tests.

Correctness

  • verification() (inspect.ts:220) is genuinely the sole constructor for RuleVerification — confirmed no other file in src/ builds one by hand (ValeRuleVerification in rules/vale/verify.ts is an unrelated internal type with the same-ish name). ok/ran/refused are derived purely from status, so the ok: true, ran: false combination really has no spelling anymore.
  • verdictOf() (commands/verify.ts:86) correctly branches on "status" in result to tell RuleValidation (no status) from RuleVerification (always has it), and folds not-run into failed while keeping refused distinct — preserving the exit-code/mark semantics the proposal describes (exit 1 for not-run, exit 2 only for refused).
  • report/draft.ts's testStatus() switch over status is a faithful simplification of the old refused !== undefined / ok && ran combination, with a clear comment on why failed and not-run both map to "fail" for the report.
  • Grepped for other readers of .ran/.refused on verification results — found none outside the two updated consumers, consistent with the PR's claim.

Schema

  • status is correctly .optional() (absent, not undefined, for validate), and the ok/ran/refused field descriptions were updated to say they're derived — good additive-change hygiene for existing --json consumers.

Docs/recipes

  • validate-rule.md, create-sg-rule.md, create-vale-rule.md topic versions were bumped and the example payloads/tables updated consistently with the new field. Checked other recipes that mention verify --json (improve-rule.md) — they don't name specific fields, so no update was needed there.

Tests

  • verify-status.test.ts covers all four outcomes' derived-field agreement and schema round-tripping. validate-verify-commands.test.ts and runtime-fixture-runner.test.ts additions assert status end-to-end for the not-run/refused/failed/passed cases described in the proposal, matching the new spec scenarios.

No correctness, security, or performance issues found, and no concrete inline comments to post.

@theCodeDrift

Copy link
Copy Markdown
Member Author

Re: @github-actions — "Claude finished @theCodeDrift's task in 2m 37s … Review complete"
#629 (comment)

Read; the review raised no findings, so there is nothing to change.

— AI Coding Agent

…refused

Each verify --json rule result now carries status (passed, failed, not-run, refused), built by one constructor that derives ok, ran and refused from it. commands/verify.ts and report/draft.ts read status instead of combining the old fields. Additive for existing --json consumers; validate reports no status.
@theCodeDrift
theCodeDrift force-pushed the feat/627-verify-status-field branch from 9a2802b to 9840916 Compare November 8, 2026 05:40
@theCodeDrift
theCodeDrift merged commit af51538 into main Nov 8, 2026
4 checks passed
@theCodeDrift
theCodeDrift deleted the feat/627-verify-status-field branch November 8, 2026 05:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Give verify results one status field instead of ok × ran × refused

1 participant