feat(verify): report one status field per rule

.changeset/verify-status-field.md

@@ -0,0 +1,5 @@
+---
+"@taskless/cli": patch
+---
+
+`verify --json` now reports a `status` on each rule: `passed`, `failed`, `not-run` (the rule failed before its tests ran) or `refused` (the execution policy declined to run them). Branch on it instead of combining `ok`, `ran` and `refused`, where reading `ok` alone could report a rule that never ran as passing. The change is additive: `ok`, `ran` and `refused` are still reported and are now derived from `status`, so they always agree with it. `validate` runs no tests and does not report `status`.

openspec/changes/archive/2026-11-07-verify-status-field/proposal.md

@@ -0,0 +1,27 @@
+## Why
+
+A rule's `verify` result has four outcomes (passed, failed after its tests ran, failed before they ran, refused by the execution policy) spread across three fields: `ok`, `ran` and `refused`. Every consumer combines them with its own precedence, and a consumer that reads `ok` alone can report a rule that never ran as passing. The encoding also admits combinations no producer should emit, such as `ok: true` with `ran: false` (#627).
+
+## What Changes
+
+- Each `verify --json` rule result carries `status`, one of `passed`, `failed`, `not-run` or `refused`, computed once by the producer.
+- `ok`, `ran` and `refused` stay, derived from `status` by one constructor, so they can no longer disagree with it. The change is additive for existing `--json` consumers.
+- `verify`'s marks, summary counts and exit code, and the report draft's test status, read `status` instead of combining the old fields.
+- `validate` runs no tests and does not report `status`.
+- The `validate-rule`, `create-sg-rule` and `create-vale-rule` recipes name `status` as the field to branch on.
+
+## Capabilities
+
+### New Capabilities
+
+None.
+
+### Modified Capabilities
+
+- `cli-rule-validation`: `verify` reports one outcome field per rule.
+
+## Impact
+
+- `packages/cli/src/rules/inspect.ts`, `src/schemas/rule-check.ts`, `src/commands/verify.ts`, `src/report/draft.ts`, and three recipes.
+- Exit codes, top-level `ok` and every existing field are unchanged.
+- Delivery shape: **single PR**. The field, its consumers and the spec only make sense together, and the diff fits one review.

openspec/changes/archive/2026-11-07-verify-status-field/specs/cli-rule-validation/spec.md

@@ -0,0 +1,46 @@
+## ADDED Requirements
+
+### Requirement: Verify reports one outcome per rule
+
+Under `--json`, each rule result from `verify` SHALL carry a `status` naming its outcome, one of `passed`, `failed`, `not-run` or `refused`:
+
+- `passed`: the rule's tests ran and passed.
+- `failed`: the rule's tests ran and something failed.
+- `not-run`: the rule failed before its tests ran, because `validate` rejected it or its fixtures could not be read, and the execution policy did not refuse it.
+- `refused`: the execution policy declined to run the rule's tests.
+
+`ok`, `ran` and `refused` SHALL remain on the result and SHALL be derived from `status`: `ok` is true exactly when `status` is `passed`, `ran` is true exactly when `status` is `passed` or `failed`, and `refused` is present exactly when `status` is `refused`. The CLI's own readers of a verification (the human marks, the summary counts, the exit code and the report draft) SHALL read `status` rather than combining the other three.
+
+`validate` runs no tests and SHALL NOT report `status`.
+
+**Rationale.** Three fields for four outcomes let every consumer invent its own precedence, and let a consumer that reads `ok` alone report a rule that never ran as passing. One field computed by the producer removes the choice. Deriving the old fields from it, rather than setting them alongside it, keeps them consistent for the consumers that already read them.
+
+#### Scenario: A passing rule reports passed
+
+- **WHEN** `verify --json` runs a rule whose tests run and pass
+- **THEN** the rule's `status` SHALL be `passed`
+- **AND** its `ok` SHALL be `true` and its `ran` SHALL be `true`
+
+#### Scenario: A failing rule reports failed
+
+- **WHEN** `verify --json` runs a rule whose tests run and a case fails
+- **THEN** the rule's `status` SHALL be `failed`
+- **AND** its `ok` SHALL be `false` and its `ran` SHALL be `true`
+
+#### Scenario: A rule validate rejected reports not-run
+
+- **WHEN** `verify --json` runs a rule that `validate` rejects
+- **THEN** the rule's `status` SHALL be `not-run`
+- **AND** its `ok` SHALL be `false`, its `ran` SHALL be `false`, and it SHALL carry no `refused`
+- **AND** `verify` SHALL exit 1
+
+#### Scenario: A refused rule reports refused
+
+- **WHEN** `verify --json` runs a runtime rule without `--dangerously-run-scripts`
+- **THEN** the rule's `status` SHALL be `refused`
+- **AND** its `ok` SHALL be `false`, its `ran` SHALL be `false`, and `refused` SHALL carry the reason
+
+#### Scenario: Validate reports no status
+
+- **WHEN** `validate --json` reports a rule
+- **THEN** the rule's result SHALL NOT carry `status`

openspec/changes/archive/2026-11-07-verify-status-field/tasks.md

@@ -0,0 +1,14 @@
+## 1. Status field
+
+- [x] 1.1 Add `status` to `RuleVerification`, built by one constructor that derives `ok`, `ran` and `refused` from it
+- [x] 1.2 Describe `status` in the `rule-check` schema as the field to branch on, verify-only
+- [x] 1.3 Read `status` in `commands/verify.ts` and `report/draft.ts`
+
+## 2. Recipes
+
+- [x] 2.1 Name `status` in `validate-rule`, `create-sg-rule` and `create-vale-rule`
+
+## 3. Tests
+
+- [x] 3.1 Assert `ok`, `ran` and `refused` agree with `status` for all four outcomes
+- [x] 3.2 Assert `status` end to end for a pass, a failure, a rule validate rejected, a refused runtime rule, and its absence from `validate`

openspec/specs/cli-rule-validation/spec.md

@@ -520,3 +520,48 @@ A trap that ast-grep itself refuses (a keyword or Markdown inline construct name
 - **WHEN** `verify` runs against a rule whose only finding from `validate` is a trap notice, and whose fixtures pass
 - **THEN** the rule SHALL be reported as passing
 - **AND** the notice SHALL be reported on it
+
+### Requirement: Verify reports one outcome per rule
+
+Under `--json`, each rule result from `verify` SHALL carry a `status` naming its outcome, one of `passed`, `failed`, `not-run` or `refused`:
+
+- `passed`: the rule's tests ran and passed.
+- `failed`: the rule's tests ran and something failed.
+- `not-run`: the rule failed before its tests ran, because `validate` rejected it or its fixtures could not be read, and the execution policy did not refuse it.
+- `refused`: the execution policy declined to run the rule's tests.
+
+`ok`, `ran` and `refused` SHALL remain on the result and SHALL be derived from `status`: `ok` is true exactly when `status` is `passed`, `ran` is true exactly when `status` is `passed` or `failed`, and `refused` is present exactly when `status` is `refused`. The CLI's own readers of a verification (the human marks, the summary counts, the exit code and the report draft) SHALL read `status` rather than combining the other three.
+
+`validate` runs no tests and SHALL NOT report `status`.
+
+**Rationale.** Three fields for four outcomes let every consumer invent its own precedence, and let a consumer that reads `ok` alone report a rule that never ran as passing. One field computed by the producer removes the choice. Deriving the old fields from it, rather than setting them alongside it, keeps them consistent for the consumers that already read them.
+
+#### Scenario: A passing rule reports passed
+
+- **WHEN** `verify --json` runs a rule whose tests run and pass
+- **THEN** the rule's `status` SHALL be `passed`
+- **AND** its `ok` SHALL be `true` and its `ran` SHALL be `true`
+
+#### Scenario: A failing rule reports failed
+
+- **WHEN** `verify --json` runs a rule whose tests run and a case fails
+- **THEN** the rule's `status` SHALL be `failed`
+- **AND** its `ok` SHALL be `false` and its `ran` SHALL be `true`
+
+#### Scenario: A rule validate rejected reports not-run
+
+- **WHEN** `verify --json` runs a rule that `validate` rejects
+- **THEN** the rule's `status` SHALL be `not-run`
+- **AND** its `ok` SHALL be `false`, its `ran` SHALL be `false`, and it SHALL carry no `refused`
+- **AND** `verify` SHALL exit 1
+
+#### Scenario: A refused rule reports refused
+
+- **WHEN** `verify --json` runs a runtime rule without `--dangerously-run-scripts`
+- **THEN** the rule's `status` SHALL be `refused`
+- **AND** its `ok` SHALL be `false`, its `ran` SHALL be `false`, and `refused` SHALL carry the reason
+
+#### Scenario: Validate reports no status
+
+- **WHEN** `validate --json` reports a rule
+- **THEN** the rule's result SHALL NOT carry `status`

packages/cli/src/agent/create-sg-rule.md

@@ -1,4 +1,4 @@
-# Topic: create-sg-rule     (CLI v%(CLI_VERSION)s / topic v12)
+# Topic: create-sg-rule     (CLI v%(CLI_VERSION)s / topic v13)
 
 %(ONE_COMMAND_PER_CALL)s
 
@@ -151,7 +151,7 @@ whole rule.
    Both answer in the same shape:
    ```json
    {"ok":true,"rules":[{"engine":"sg","ruleId":"no-eval",
-    "ok":true,"errors":[],"ran":true}]}
+    "status":"passed","ok":true,"errors":[],"ran":true}]}
    ```
    - `ok: true` → go to step 8.
    - `ok: false` → read `errors` and fix. Repeat up to 3 times.

packages/cli/src/agent/create-vale-rule.md

@@ -1,4 +1,4 @@
-# Topic: create-vale-rule     (CLI v%(CLI_VERSION)s / topic v19)
+# Topic: create-vale-rule     (CLI v%(CLI_VERSION)s / topic v20)
 
 %(ONE_COMMAND_PER_CALL)s
 
@@ -980,7 +980,7 @@ it.
 
    ```json
    {"ok":true,"rules":[{"engine":"vale","ruleId":"no-simply",
-    "ok":true,"errors":[],"ran":true}]}
+    "status":"passed","ok":true,"errors":[],"ran":true}]}
    ```
 
    `ok` is the answer. `errors` names what failed, one string per

packages/cli/src/agent/validate-rule.md

@@ -1,4 +1,4 @@
-# Topic: validate-rule     (CLI v%(CLI_VERSION)s / topic v9)
+# Topic: validate-rule     (CLI v%(CLI_VERSION)s / topic v10)
 
 %(ONE_COMMAND_PER_CALL)s
 
@@ -53,8 +53,8 @@ A runtime rule's fixtures execute its `check.ts`, so they run only when
 you pass `--dangerously-run-scripts`. Nothing else runs them: `verify`
 makes no network request and does not reconcile, so a signature the
 service blessed does not count here. Without the flag, that rule prints `○` rather than a tick, names the reason
-and the flag, and is reported as `"ran": false` with a `refused` string
-saying why. It is counted under "did not run" rather than among the
+and the flag, and is reported as `"status": "refused"` with a `refused`
+string saying why. It is counted under "did not run" rather than among the
 rules verified.
 
 `verify` exits 0 only when every rule you asked for was verified:
@@ -65,8 +65,8 @@ rules verified.
 | `1`  | at least one rule failed                              | `false`        |
 | `2`  | nothing failed, but at least one rule did not run     | `false`        |
 
-Exit `2` is not a broken rule. Read `ran` on each rule to see which ones
-did not run, and rerun them with the flag if you mean to execute them.
+Exit `2` is not a broken rule. Read `status` on each rule to see which
+ones did not run, and rerun them with the flag if you mean to execute them.
 A runtime rule that did not run is not a rule that passed, and treating
 the two alike is the defect this reporting exists to prevent.
 
@@ -102,18 +102,21 @@ Both commands answer in the same shape, one entry per rule:
 
 - `ok` at the top is true only when every rule passed.
 - `errors` is one string per problem, written to be read by a person.
-- `verify` adds `ran`, which is false when `validate` failed and the tests
-  never executed, and false for a runtime rule the execution policy
-  refused. The second case also carries `refused` with the reason, and
-  is the one where `ok` alone would mislead you.
+- `verify` adds `status`, the field to branch on. It is one of
+  `passed` (the tests ran and passed), `failed` (they ran and something
+  failed), `not-run` (`validate` failed, so the tests never executed),
+  or `refused` (the execution policy declined to run them, with the
+  reason in `refused`). `validate` runs no tests and reports no
+  `status`. The older `ran` and `refused` fields are still there and
+  agree with it, but `ok` alone would mislead you on a refused rule.
 - `notice` is something true about the rule that is not a failure, and
   it is reported on a pass as well. For a Vale rule it carries what
   Vale wrote to stderr on a run that still exited zero, such as the
   `W101` warning about an assignment placed above any matcher: the rule
   validates, Vale ignores it, and only this field says so. For an `sg`
   rule it names the silent traps from `create-sg-rule`'s Quirks section
   that the rule's shape matches, such as `$$$` beside a comma. Read it
-  before `ok`, the same as `ran`.
+  before `ok`, the same as `status`.
 
 A green Vale `verify` also assumes Vale read every fixture. A fixture
 whose front matter Vale cannot parse is excluded from the run and

packages/cli/src/commands/verify.ts

@@ -74,6 +74,22 @@ function printFindings(findings: FixtureFinding[]): void {
   }
 }
 
+/**
+ * How a result counts toward the marks, the summary and the exit code.
+ *
+ * `verify` results carry `status`, which is read as given apart from one fold:
+ * a rule that failed before its tests ran (`not-run`) counts as a failure,
+ * because something about it is broken and the errors say what. Only a refusal
+ * is held apart. `validate` runs no tests and has no `status`, so `ok` is its
+ * whole answer.
+ */
+function verdictOf(
+  result: RuleValidation | RuleVerification
+): "passed" | "failed" | "refused" {
+  if (!("status" in result)) return result.ok ? "passed" : "failed";
+  return result.status === "not-run" ? "failed" : result.status;
+}
+
 /**
  * The shared body of `validate` and `verify`.
  *
@@ -189,10 +205,8 @@ async function runOverPath(options: {
   // its own marker below, and has its own exit code, because an exit code is all
   // a CI step or an agent reads, and a rule nothing exercised must not read the
   // same as a rule that passed.
-  const isRefused = (result: RuleValidation | RuleVerification): boolean =>
-    "refused" in result && result.refused !== undefined;
-  const refused = results.filter((result) => isRefused(result));
-  const failed = results.filter((result) => !result.ok && !isRefused(result));
+  const refused = results.filter((result) => verdictOf(result) === "refused");
+  const failed = results.filter((result) => verdictOf(result) === "failed");
   const judged = results.length - refused.length;
 
   if (json) {
@@ -211,7 +225,9 @@ async function runOverPath(options: {
       // `○` is neither tick nor cross on purpose: the rule was not verified, and
       // a reader scanning the column has to be able to see that at a glance.
       // The reason prints below it, from `errors`, and names what would run it.
-      const mark = isRefused(result) ? "○" : result.ok ? "✓" : "✗";
+      const verdict = verdictOf(result);
+      const mark =
+        verdict === "refused" ? "○" : verdict === "passed" ? "✓" : "✗";
       console.log(`${mark} ${result.engine}/${result.ruleId}`);
       for (const error of result.errors) {
         console.log(`    ${error}`);
@@ -233,7 +249,7 @@ async function runOverPath(options: {
       // Under the errors, because the errors say which fixture is wrong and
       // these say what the rule reported about it. A refused run has nothing
       // to show: nothing ran.
-      if (!result.ok && !isRefused(result)) {
+      if (verdict === "failed") {
         printFindings(findingsOf(result));
       }
     }

packages/cli/src/report/draft.ts

@@ -138,9 +138,22 @@ async function testStatus(
   engine: "sg" | "vale",
   id: string
 ): Promise<TestStatus> {
-  const verification = await verifyOneRule(cwd, { engine, ruleId: id });
-  if (verification.refused !== undefined) return "refused";
-  return verification.ok && verification.ran ? "pass" : "fail";
+  const { status } = await verifyOneRule(cwd, { engine, ruleId: id });
+  switch (status) {
+    case "passed": {
+      return "pass";
+    }
+    case "refused": {
+      return "refused";
+    }
+    // A rule that failed before its tests ran is reported as failing, not as
+    // `not-run`: the report's `not-run` means nothing tried to test it, and
+    // here something did and found it broken.
+    case "failed":
+    case "not-run": {
+      return "fail";
+    }
+  }
 }
 
 /** Every rule under `.taskless/rules/`, with its tests run. */