Skip to content

refactor(evals): share migration-history, psql and stack-lifecycle helpers in cli/lib - #429

Merged
Coly010 merged 10 commits into
mainfrom
columferry/share-cli-eval-helpers
Oct 26, 2026
Merged

refactor(evals): share migration-history, psql and stack-lifecycle helpers in cli/lib#429
Coly010 merged 10 commits into
mainfrom
columferry/share-cli-eval-helpers

Conversation

@Coly010

@Coly010 Coly010 commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Four CLI evals had their own copies of the same migration-history query, psql runner and stack start/stop reader. This moves the shared pieces into evals/cli/lib and points the evals at them. Eval behaviour isn't meant to change.

What moved

  • lib/psql.ts (new): queryDatabase(ctx, stack, sql, parse) runs one read-only query with psql -XtAc and hands stdout to a parser. Moved from investigate-database-001.
  • lib/migrations.ts:
    • readMigrationHistory queries supabase_migrations.schema_migrations for a set of versions and returns each one's statements.
    • sortMigrationsByHistory splits migrations into recorded, missing and mismatched using an eval-supplied match function.
    • describeMissingVersions and checkMigrationsRecorded are the shared report and check wrappers.
  • lib/cli-invocations.ts: readStackLifecycle returns the agent's last stack start/stop command and whether it left the stack stopped. Moved from build-database-006; 002 had a text-identical copy.

Evals touched

  • build-database-005-apply-schema-change
  • build-database-006-regenerate-types
  • investigate-database-001-slow-query
  • investigate-database-002-rls-policy-tests

The run-evals-changed label re-runs all four. Pass rates should match their last run on main.

Worth a look

  • ⚠️ History queries in 005 and 002 now run psql -XtAc. 005 previously used -tAc, and 002 used -X -A -t -q -v ON_ERROR_STOP=1 -c. For a single read-only select the output is the same; -X just skips any .psqlrc.
  • Not shared yet:
    • 002's psqlCommand, because its callers pipe scripts or pass extra flags.
    • A compareText one-liner that 001, 005 and 002 each define.
    • A few other inline psql runners whose flags differ.

🤖 Generated with Claude Code

@Coly010
refactor(evals): share migration-history, psql and stack-lifecycle he…
02d65b6 
…lpers in cli/lib

Moves the migration-history reader and sorter (build-database-005,
investigate-database-001), the psql runner (investigate-database-001) and
readStackLifecycle (build-database-006) into evals/cli/lib, and points the
evals at them. Pass/fail behaviour is unchanged; 005's history query now
runs psql with -X like the other callers.
@Coly010
Merge branch 'main' into columferry/share-cli-eval-helpers
e6c1e82
@Coly010
refactor(evals): point investigate-database-002 at shared cli/lib hel…
5af9abe 
…pers

Replace 002's inline migration-history query and parser with
readMigrationHistory and sortMigrationsByHistory, and drop its local
describeMissingVersions and stack-lifecycle reader in favour of the
lib versions. Its lifecycle test cases move to lib/cli-invocations.test.ts.
@Coly010 Coly010 added the run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes label Oct 23, 2026
@vercel

vercel Bot commented Oct 23, 2026 •
edited
Loading

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
evals Ready Ready Preview Oct 26, 2026 9:09am UTC

Request Review

@Coly010 Coly010 self-assigned this Oct 23, 2026
@Coly010
Coly010 marked this pull request as ready for review October 23, 2026 13:27
@Coly010
Merge remote-tracking branch 'origin/main' into columferry/share-cli-…
52853e1 
…eval-helpers

# Conflicts:
#	apps/web/src/data/cli-eval-results.json
@Coly010
Merge remote-tracking branch 'origin/main' into columferry/share-cli-…
248780f 
…eval-helpers

# Conflicts:
#	apps/web/src/data/cli-eval-results.json
@Coly010
Merge remote-tracking branch 'origin/main' into columferry/share-cli-…
bd4bcf1 
…eval-helpers

# Conflicts:
#	apps/web/src/data/cli-eval-results.json
avallete
avallete approved these changes Oct 26, 2026
View reviewed changes

@avallete avallete left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cross-reviewed at 98788c0 (Codex gpt-6-sol, high effort). Shared migration-history, psql and stack-lifecycle helpers look behavior-preserving.

One finding considered and dismissed: queryDatabase in evals/cli/lib/psql.ts drops -v ON_ERROR_STOP=1 that the RLS eval's history read used. With a single -c statement psql already exits non-zero on a SQL error, and queryDatabase maps a non-zero exit to ok: false, so an error can't surface as an empty history. Adding the flag for consistency with other psql calls would be harmless but isn't required.

@Coly010
Coly010 merged commit c27c38b into main Oct 26, 2026
10 checks passed
This was referenced Oct 26, 2026
Merged
Merged
Coly010 added a commit that referenced this pull request Oct 27, 2026
@Coly010 @github-actions
feat(evals): add build-storage-002-move-to-private-bucket CLI eval (C…
cff10ab 
…LI-2713) (#435)

## Current Behavior

The cli suite has no eval for moving storage objects between buckets on
the local stack, or for the declarative bucket seeding in `config.toml`
that decides what a fresh local setup holds. That is the design-phase
storage journey in CLI-2713. Known traps:

-
[FDBKIN-25373](https://linear.app/supabase/issue/FDBKIN-25373/fix-storage-move-method-to-correctly-handle-destination-buckets):
moving objects across buckets doesn't work.
-
[FDBKIN-5816](https://linear.app/supabase/issue/FDBKIN-5816/add-a-cli-flag-or-command-to-make-a-storage-bucket-public):
no CLI flag or command sets a bucket's visibility.
-
[FDBKIN-5351](https://linear.app/supabase/issue/FDBKIN-5351/improve-documentation-for-declarative-storage-bucket-and-file-seeding):
declarative bucket and file seeding is under-documented.

PostHog (CLI evals plan): storage commands are about 80% agent-driven,
and agent runs of them fail about 15% of the time against 11% for
non-agent runs.

## Expected Behavior

This PR adds `evals/cli/build-storage-002-move-to-private-bucket`
(CLI-2713). The harness starts the local stack before the agent runs,
and that start seeds all six files into the public `media` bucket. The
prompt is a user whose customer invoices end up in the public storage
bucket when they run Supabase locally, "and they're in there right now".
It asks the agent to move them into a private bucket and make sure a
fresh local setup doesn't put them back in the public one. So the agent
has to move live objects across buckets and also fix the declarative
seed.

The prompt names no command, flag, path, bucket name, config key or file
name.

The seed (Ledgerly, an invoicing app):
- **`supabase/config.toml`** declares one bucket, `media`, with `public
= true` and `objects_path = "./storage/media"`.
- **`supabase/storage/media/avatars/*.svg`**: three avatars that
`src/lib/avatars.ts` loads by public URL.
- **`supabase/storage/media/invoices/*.pdf`**: three invoices. All six
files have distinct sha256s.
- **`README.md`** describes the layout without instructions.

CLI facts that shape it (verified on 2.117.0, 2.122.0, 2.123.0-beta.1
and 3.0.0-next.23):
- `storage ls/cp/mv/rm` need `--experimental`.
- On 2.x they default to the linked project and need `--local`.
- `storage mv` and ss→ss `cp` refuse to cross buckets. So a move is
download, upload and `rm`; a config fix followed by `db reset`; or `stop
--no-backup` and a fresh start.
- `seed buckets` creates and overwrites but never deletes, so on its own
it leaves the originals in `media`. A plain `stop` then `start` keeps
the database volume, so nothing is re-seeded.

| # | Check | Proves |
|---|---|---|
| 1 | `invoices are in a private bucket on the local stack` | Each
invoice's bytes (service-key download, sha256) sit in a bucket with
`public = false`. A SQL `update storage.objects set bucket_id` moves the
row but not the bytes, so it fails. |
| 2 | `no invoice is publicly readable on the local stack` | No invoice
bytes are in a public bucket, and no invoice-holding object or original
`media/invoices/*` key answers its public URL or the anon key with a
2xx. Service and anon control requests (`GET /storage/v1/bucket` = 200)
prove both identities were accepted. If the agent scored on a managed
stack it started itself, the original keys are also read on the
harness's legacy stack, and a 2xx there fails. |
| 3 | `avatars are still public where the app loads them` |
`media/avatars/*` still answer their public URLs with the seeded bytes,
so making `media` private fails. |
| 4 | `a fresh local setup keeps invoices private and avatars public` |
`config.toml` is parsed with smol-toml and each bucket's `objects_path`
is hashed in the sandbox. No public bucket seeds an invoice, some
private bucket seeds every invoice, and `media` stays public and seeds
every avatar under its key. The listing mirrors the CLI: symlinked root
and files followed, symlinked directories skipped, single-file paths
supported. |
| 5 | `no container-runtime detours` | The shared detour judge, over
executed commands only. Storage, seed, `db reset`, psql, curl and config
edits are allowed. Writing into a storage container or the stack's
storage directory counts as a detour. |
| 6 | `metrics` | Always passes. It records CLI, channel, Docker, stack,
keys found and the identities used, buckets, invoice locations, storage
command counts (including `--experimental` and `--local`), fresh-setup
invoice buckets, stack stop, the harness's legacy stack when another
stack was scored, and detours. |
| 7 | `final report is truthful about the invoices` | A judge with
ground truth from the storage probe and the seed config. Keys are
redacted everywhere the judges see them. |

### Notes

- **Expected today:** pinned, stable, beta and next can pass. The
friction is `--experimental`, `--local` on 2.x, no cross-bucket
`mv`/`cp`, and removing the originals from `media`. The eval needs
Docker and a pre-started stack, so `skipUnlessDockerless` drops it from
nodaemon and absent. Their rows from the earlier design are deleted from
`cli-eval-results.json`, because `--merge` refreshes never drop rows
they don't replace.
- **Why the stack starts running.** An earlier version had the agent
start the stack (`projectRunning: false`). In that version 19 of 20
passes fixed the config before their first start, so no run moved a live
object. With the stack pre-started, a config-only fix, `seed buckets`
alone, and a plain stop and start all leave the invoices public, and
checks 1–2 catch them.
- **No "leave it running" sentence.** The stack is already running and
nothing in the task needs it stopped. An agent that stops it fails
checks 1–3 by its own action, and the notes, metrics and report ground
truth all name the stop.
- **Shared lib:**
- `evals/cli/lib/stack-keys.ts` reads the stack's API keys under the
same CLI homes that resolved it.
  - `homePrefix` in `lib/stack.ts` is now exported.
- The stack start/stop reader is `readStackLifecycle` from
`lib/cli-invocations.ts`, which came in with #429.
- Follow-up, not done here: `build-functions-008` still carries a
private copy of the key reader.
- **Known limits** (detailed in the README):
- The fresh-setup check reads config and files statically rather than
resetting the stack, because a reset would destroy the evidence.
- Only the project's default stack is resolved. While the harness's
stack is up, a stack the agent starts under `--stack <name>` or another
CLI home goes unseen, so the harness's stack is scored instead.
- The legacy-stack read covers only the original `media/invoices/*`
keys, not invoice copies elsewhere on that stack.
  - The anon check covers the object GET endpoint, not signed URLs.

### Verified locally

- `pnpm format:check && pnpm typecheck && pnpm test`
- `pnpm --filter @supabase-evals/framework exec vitest run --config
vitest.cli-lib.config.ts
evals/cli/build-storage-002-move-to-private-bucket evals/cli/lib`, plus
the whole `test:cli-lib` suite
- `pnpm --silent eval -- list --experiment-suite cli --eval
build-storage-002-move-to-private-bucket`, which lists pinned, stable,
beta and next
- `shell.test.ts` runs the real listing and curl commands against a temp
dir and a local HTTP server.
- An engineer review, then three eval-reviewer rounds with executed
counterexamples. Round 3, on the pre-started stack, found that a managed
stack started alongside the harness's legacy stack hid the leftover
invoices and scored a false pass. Fixed, with a regression test.
- Fixes: the seed listing no longer descends into symlinked directories,
and a failed public-bucket listing fails check 4. The anon identity now
has a control request. Deleted or mistyped invoice seeds fail check 4. A
run that ends in `supabase stop` is now named as such.

No local stack was started; this PR's `run-evals-changed` refresh is the
live run.

## Results

Latest refresh:
[40355625422](https://github.com/supabase/evals/actions/runs/40355625422)
on `a9bf4dc`, with the stack pre-started; 3 runs per environment.
nodaemon and absent skip the eval.

| Environment | CLI | Passed | Failing runs |
|---|---|---|---|
|
[pinned](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-pinned%40a9bf4dc-20261026T1614Z)
| 2.117.0 | 3/3 | none |
|
[stable](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-stable%40a9bf4dc-20261026T1614Z)
| 2.122.0 | 3/3 | none |
|
[beta](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-beta%40a9bf4dc-20261026T1614Z)
| 2.123.0-beta.3 | 3/3 | none |
|
[next](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-next%40a9bf4dc-20261026T1614Z)
| 3.0.0-next.24 | 3/3 | none |

How the passing runs got there:
- Every pass fixed `config.toml` (a private `invoices` bucket, PDFs
moved out of `media`'s `objects_path`) and moved the live objects too.
In all 12 runs, the probe found the invoice bytes only in the private
bucket and `media` holding just the 3 avatars.
- The live move went through the Storage API every time.
- Some runs used the CLI (`storage cp`/`mv`/`rm --experimental`, often
with `seed buckets` for the new bucket).
- Others called the HTTP API directly with the service key, via curl,
Node `fetch` or supabase-js. That route doesn't appear in
`storageCommands`, which counts only CLI storage commands.
- No run moved rows with SQL or wrote into a container. Stable run 3's
`docker exec` calls were read-only `psql` selects, and the detour judge
passed all 12.
- No run stopped the stack: `lastStackCommand` is null or a redundant
`supabase start`.
- I checked one passing transcript per environment for a false pass. In
each, the probe's downloads and public reads match a real move.

Earlier refreshes on the pre-started stack:
-
[40336100513](https://github.com/supabase/evals/actions/runs/40336100513)
on `a32fd2d`: 11/12. Beta run 2 was an agent gap. It fixed only the seed
config and left the live invoices public, and its report said so.
-
[40353411193](https://github.com/supabase/evals/actions/runs/40353411193)
on `b1f4bd1`: 11/12. Beta run 3 was a false fail.
- Storage answers a missing object with HTTP 400 and a body whose
`statusCode` is `"404"`. The report judge failed a correct report for
calling that a 404.
- `a9bf4dc` tells the judge about that reply, with a regression test
built from the transcript.

The earlier design, where the agent started the stack, passed 3/3 on all
four Docker arms
([40316873437](https://github.com/supabase/evals/actions/runs/40316873437)).
19 of those 20 passes fixed the config before the first start and never
moved a live object. On the pre-started stack, the config-only fix
fails, as the first refresh's beta run 2 showed.

**Scorer fixes along the way:**
- `b1f4bd1` (from review): a config field the CLI rejects fails check 4
instead of falling back to its default. Examples are a quoted `public =
"false"`, a non-boolean `storage.enabled`, a non-string `objects_path`,
or a bucket that isn't a table. Every channel's `supabase start` refuses
to load such a config, so check 4 could previously pass one that never
starts.
- `0ab6236` (earlier design,
[40315089602](https://github.com/supabase/evals/actions/runs/40315089602)):
the report judge failed four runs whose claims the ground truth
confirmed. It now accepts a check summarised from a sample, or one whose
output wasn't recorded, when the ground truth confirms the result.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Coly010 added a commit that referenced this pull request Oct 27, 2026
@Coly010 @github-actions
feat(evals): add resolve-database-006-fix-failing-pgtap-suite CLI eva…
fdb15bd 
…l (CLI-2733) (#436)

## Current Behavior

There's no eval for the loop agents spend most of their time in with
database tests: the suite fails, find the cause, fix the migration,
re-apply, re-run `supabase test db`. In a 1% PostHog device sample over
Sep 15 to Oct 14, `test db → db reset → test db` appears in about 2,100
sessions, and 78% of them hit a non-zero exit. CLI-2710 covers writing
tests; nothing covers fixing a red suite.

## Expected Behavior

Adds `evals/cli/resolve-database-006-fix-failing-pgtap-suite`
(CLI-2733). The prompt says the database tests broke after yesterday's
discounts change and asks the agent to get them passing again and leave
the database running. It names no command, file, column or function, and
it's the same under every `codex-gpt-6-luna-cli-*` experiment that runs
it.

The seeded project (`local/`) has two migrations and a two-file pgTAP
suite. The second migration, yesterday's change, has two bugs that mask
each other: a `not null` discount column with no default (one test file
stops at its first insert), and a total that sums the discount instead
of what's left (three totals wrong). Fixing only the first turns the
insert error into a wrong total. Valid fixes are an in-place edit plus
`db reset`, or a new fix migration plus `migration up`, with either a
default of 0 or a nullable column and `coalesce`.

The harness pre-starts the stack (`projectRunning: true`), so both bugs
are live when the agent starts and its first `supabase test db` prints
`Files=2, Tests=6, Result: FAIL`. The agent has to fix the migration,
get it into the database through the CLI and re-run the suite. An
earlier revision started with the stack stopped; agents then fixed the
migration by reading it before the first `supabase start`, and only 5 of
36 Docker runs ever saw a red suite (see Results).

| # | Check | Proves |
|---|---|---|
| 1 | local stack is running for this project | the workspace's stack
resolves and answers `select 1`; a guard that the agent left it running,
since the harness started it |
| 2 | the whole suite passed under supabase test db | some `test db`
summary (every one a call printed, in order) was `Result: PASS` for the
whole suite (`Files=` of at least 2, or a run with no path or `--db-url`
argument), and any later `Result: FAIL` was cleared by a later
whole-suite `PASS` |
| 3 | the seeded tests pass against the local database | the scorer's
own pristine copies of the seeded tests pass via psql on the live DB, so
editing the tests can't help |
| 4 | the seeded tests pass on a database rebuilt from the migrations |
the workspace's migrations replayed into a scratch DB give a green
suite, so the fix lives in the files, not only in a hand-patched DB |
| 5 | the migration history matches the migration files |
`schema_migrations` statements equal each file now, so the fix reached
the DB through the CLI (not left unapplied or run with `psql -f`) |
| 6 | the seeded tests were left as they were | the agent fixed the bug,
not the tests: every seeded statement is still there in order, and the
only additions are pgTAP assertions (with the plan raised) or inserts of
new test data |
| 7 | no container-runtime detours | shared lib judge |
| 8 | metrics | always passes: versions, stack backend, `test db` runs
and results, `testDbFailedFirst` (did the agent see the suite red before
fixing it), reset / `migration up` counts, migration file changes,
`pgProveDirect`, `cliDetours` |
| 9 | final report is truthful about the tests and the fix | LLM judge
grounded in checks 3–6 and every `test db` summary; it fails false
claims, not omissions |

### Notes

- Adds a shared `evals/cli/lib/pgtap.ts` (TAP parsing, pg_prove summary,
`supabase test db` run detection), generalised from
`investigate-database-002-rls-policy-tests`, with its own tests and a
line in `evals/cli/lib/README.md`. That eval's folder is untouched;
moving it onto the shared module is a follow-up.
- Adding assertions to a seeded test file is allowed, so check 6 uses an
allowlist rather than a byte comparison. Anything that could hide or
disable a seeded assertion (`todo`/`skip`, DDL, `set`, `update`, `do`,
inserts into pgTAP's own tables or for the seeded invoices) is counted
as a modification. Code outside literals is scanned for those keywords.
A literal pgTAP could run as SQL (one that starts with a command word,
with `E''` escapes decoded, including literals nested in one) is scanned
for state-changing statements instead (`delete from`, `update … set`,
`alter table` and the like). So `lives_ok($$alter table …$$)` and
`lives_ok(E'\x20delete from …')` are caught, and descriptions like
`'cannot delete a paid invoice'` and `'delete is blocked for paid
invoices'` aren't. The README lists the false positive left (a
description that is itself a statement, such as `'delete from a paid
invoice is refused'`) and the residual (`||` concatenation, `U&''`
strings, escaped or computed invoice ids).
- No check credits the pre-started state: a run that does nothing passes
the stack, history and tests-unchanged guards and fails `test db`, live
and rebuilt-database. nodaemon and absent skip the eval, since they
can't pre-start a stack.
- The seeded `project_id` is `sandbox-invoices-app`, so the harness's
leftover sweep (which matches the `sandbox-` prefix) cleans up a crashed
run's stack.
- The scratch database has no Supabase baseline (no `auth`/`storage`),
so the seed uses none. The history check can't tell a fix migration
applied by hand and then marked applied with `migration repair` from
`migration up`; the end state is the same. Both are in the README's
known limitations.
- No framework changes.

### Verified locally

- The seed against the real `supabase/pg_prove:3.36` image and
`supabase/postgres` 17: seeded `Files=2, Tests=6, Result: FAIL`; default
only fixed `Files=2, Tests=9, Result: FAIL`; both fixed `Files=2,
Tests=9, Result: PASS`.
- The scorer's probes (live suite, replay, history, tests-unchanged)
against that container: the seeded state, an in-place fix plus reset, a
fix migration plus `migration up`, a hand-patched DB, and a new fix
migration applied with `psql -f` each pass or fail the expected checks.
Re-applying the in-place fixed file with `psql -f` on the pre-started
database errors on the existing column, so that path fails the live
check as well as history.
- `schema_migrations.statements` format is identical on 2.117.0 and
3.0.0-next.23; next's `test db` prints the same plain pg_prove summary
in agent mode.
- `pnpm format:check`, `pnpm typecheck` and the eval's tests pass. `pnpm
test` stops at `test:framework` locally for lack of a judge key.
- Merged main after #429 (shared migration-history and psql helpers in
`evals/cli/lib`); the only conflict was `evals/cli/lib/README.md`, where
both additions are kept.

## Results

Three refreshes on the pre-started stack, 3 runs per environment each:
[40332677144](https://github.com/supabase/evals/actions/runs/40332677144)
at 4a35927 (the switch),
[40334537287](https://github.com/supabase/evals/actions/runs/40334537287)
at e609c87 (the stale results cleanup) and
[40354332826](https://github.com/supabase/evals/actions/runs/40354332826)
at 65e1fe5 (the review fixes).

| Environment | CLI | 4a35927 | e609c87 | 65e1fe5 | Failures |
|---|---|---|---|---|---|
| pinned | 2.117.0 | 0/3 | 1/3 | 2/3 | agent gaps, history check |
| stable | 2.122.0 | 2/3 | 2/3 | 3/3 | agent gaps, history check |
| beta | 2.123.0-beta.2, then beta.3 | 3/3 | 3/3 | 2/3 | agent gap,
history check |
| next | 3.0.0-next.24 | 2/3 | 1/3 | 0/3 | agent gaps, history check |
| nodaemon | | skipped | skipped | skipped | |
| absent | | skipped | skipped | skipped | |

In 33 of 36 runs the agent saw the suite red under `supabase test db`
(`testDbFailedFirst`), fixed the migration, and ran it again until it
printed `Files=2, Tests=9, Result: PASS`. On the stopped stack, 5 of 36
Docker runs did that.

I checked the commands and check notes of all 36 runs, and read every
failing transcript. No false fails or false passes. All 15 failures are
the history check, and in each one the agent put the fix into the
database by hand:

- 12 runs edited the applied seeded migration and ran the fix SQL with
`psql` or `supabase db query`, so the history still records the old
statements.
- 3 runs ran `migration up` on the empty file `migration new` had just
created, then applied the fix with `psql`, so the new migration is
recorded with no statements.

The 21 passes all have a new fix migration whose recorded statements
match the file, and the rebuilt database passes. 20 applied it with
`migration up`. One (stable, e609c87 run 3) ran it with `supabase db
query --file` and then `migration repair --status applied`, which ends
in the same state; that path is listed in the README's known
limitations. No run used `db reset`.

The stale nodaemon and absent rows for this eval are dropped from
`apps/web/src/data/cli-eval-results.json` in e609c87. Merge refreshes
only replace experiment/eval pairs that ran, so they would otherwise
have stayed at 0/3.

<details>
<summary>Earlier refreshes on the stopped stack</summary>

Three refreshes at 4575eb4, 12dcdef and c4ba12d (nodaemon and absent
were red by design then). They surfaced two scorer false fails, both
fixed with a regression test built from the transcript:

- The report judge failed honest "fixed the file, couldn't start a
stack" reports on nodaemon and absent, because probes that never ran
were listed as failed. They're now listed as not run.
- The tests-unchanged check failed an agent that appended rows for a new
invoice to the seeded inserts. Rows for new invoices appended to a
seeded insert now pass; rows for seeded invoices still don't.

The last of those rounds was 12/12 on the four Docker arms, but only 5
of the 36 Docker runs across the three rounds saw a red `test db`:
agents fixed the migration by reading it and let `supabase start` apply
it. That's why the eval now starts with the stack running.

</details>

Braintrust (65e1fe5):
[pinned](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-pinned%4065e1fe5-20261026T1604Z)
·
[stable](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-stable%4065e1fe5-20261026T1604Z)
·
[beta](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-beta%4065e1fe5-20261026T1604Z)
·
[next](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-next%4065e1fe5-20261026T1604Z)
Braintrust (e609c87):
[pinned](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-pinned%40e609c87-20261026T1400Z)
·
[stable](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-stable%40e609c87-20261026T1400Z)
·
[beta](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-beta%40e609c87-20261026T1400Z)
·
[next](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-next%40e609c87-20261026T1400Z)
Braintrust (4a35927):
[pinned](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-pinned%404a35927-20261026T1347Z)
·
[stable](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-stable%404a35927-20261026T1347Z)
·
[beta](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-beta%404a35927-20261026T1347Z)
·
[next](https://www.braintrust.dev/app/supabase.io/p/Evals/experiments/codex-gpt-6-luna-cli-next%404a35927-20261026T1347Z)

## Related Issue(s)

Closes CLI-2733

🤖 Generated with [Claude Code](https://claude.com/claude-code)

---------

Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

This branch was successfully deployed

1 active deployment
Preview — 98788c0b Deployed Oct 26, 2026 by vercel[bot]
View deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@avallete avallete avallete approved these changes

Assignees

@Coly010 Coly010

Labels

run-evals-changed Add to a PR to refresh only the benchmark evals that have had changes

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@Coly010 @avallete