Skip to content

Reuse cited measurements in Yardstick comparisons - #8382

Open
loopx-agent wants to merge 4 commits into
mainfrom
codex/yardstick-reuse-cited-measurements
Open

loopx-agent wants to merge 4 commits into
mainfrom
codex/yardstick-reuse-cited-measurements

Conversation

@loopx-agent

@loopx-agent loopx-agent commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A Yardstick decision can explicitly cite a prior measurement on its linked question. When preview and authoritative results were written separately, the paired comparison only read findings that carried a Yardstick decision. A previously recorded, explicitly cited preview could disappear from the comparison even when both evaluator profiles measured the same candidates.

Change

  • Include cited measurements by the same agent on the linked question in the existing comparison.
  • Apply the evidence selection after normal measurement and execution-attribution validation. Keep prior decisions under their original work and target scope.
  • Keep exact profile and candidate pairing, optional decision capture, and the existing readback contract.

Compatibility and validation

Current head 5f9a6a02d incorporates main a3b589610 and composes with #8380. The comparison selects cited evidence before ranking and scopes the proxy to the two candidates paired by the authoritative profile.

  • The focused regression fails on main without the cited-measurement support and when either the evidence or candidate filter is removed. It covers cited sibling and uncited evidence, different authors, questions and input revisions, and prior decisions on another target.
  • 40 focused TypeScript tests, 28 CLI and architecture tests, and control-plane typechecking passed.
  • Isolated SQLite replay through the ordinary CLI surfaced the ranking reversal on the 12 KB opening page. It rejected both candidate and input-revision mismatches.
  • The bilingual RFC gate, quick premerge canary, and strict MkDocs documentation build passed locally.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@mergify

mergify Bot commented Nov 8, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 8, 2026
Select cited evidence and goal-matched proxy candidates before ranking.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
@mergify mergify Bot removed the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 8, 2026

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机 / Motivation

已启用 Yardstick 的研究者需要在同一候选对上核对代理排序与权威结果。之前单独写入的代理测量即使明确引用,也进不了当前配对。当前提交恢复了此前已记录且匹配 profile 的配对,但允许后写入的 finding 改变旧引用所指的结果。新结果会出现在下一轮首屏,影响下一步用哪种测量。当前评审边界是已启用 Yardstick 的关联问题中对既有测量的复用。C13 的真实模型采用与完整研究成本留待独立验收。

改动思路 / Approach

保留 Explore 的校验与配对 owner,并约束引用对应的来源时间。当前评审边界是已启用 Yardstick 的关联问题中对既有测量的复用。已有归档仍可用于审计。发现后写或互相冲突的引用时,需要由同一 owner 保持原绑定或明确显示证据未知。

具体改动 / Changes

评审 5f9a6a0 对 a3b5896 的完整 5 个文件,+70/−12。现有规范的 C6 和 C12 路径通过,C7 和 C11 的来源范围出现回归。C13 的范围是实际下一步决策。新版双语新增条款都写明复用较早记录的测量。 基线 RFC

关键代码讲解

  • projectWorkYardstickCue 对带决策的 finding 保留原工作和目标过滤。无决策的新增分支只校验关联问题、Agent 和相同 evidence ref,未核对对应 finding 在本次决策时是否已存在。 来源筛选
  • projectExploreMeasurementFeedback 继续先校验 measurement 和 execution observation,再按引用与候选过滤。这里的 append order 会把后来写入的同名记录选为最新结果。历史读取与关闭配置仍使用原路径。 测量排序

对主干的风险 / Risks

P2 · 后写 finding 可改变已提交决策引用的测量。 在同一问题先记录 proxy A=8、B=6。再提交一条引用这两项结果且 outcome A=20、B=14 的 Yardstick 决策。通过普通 refresh-state 和原结算追加一条无决策的 proxy B=99,沿用 result:preview-b、相同 Agent 和 profile。File 与 SQLite 都接受该记录。当前决策 ID 未变,详细读回却用新 finding 的 99 替换原 6,proxy_order_reversed 从 false 变成 true,下一轮 12 KB 首屏也出现排序反转。基线在两种 provider 都保持无反转提示。

请将无决策的引用来源限制在当前决策当时已记录的证据范围内,使用现有的提交或记录顺序,给不确定的来源保留未知。重跑同一 File/SQLite 反例并保留此前合法引用的正例。定向验证还确认更换 input revision 后不生成配对。22 项 CLI 测试、72 项相关 TypeScript 测试、类型检查、双语 RFC 检查及 standard premerge 的 4 项直接检查和 14 项自动检查通过。

我的整体评价 / Verdict

当前提交需要修改引用来源的时间边界。修复后请用同一组数据检查历史配对、后写冲突和首屏建议,再确认关闭 Yardstick 时仍可正常读取原有测量记录。

English verdict: REQUEST_CHANGES for 5f9a6a0. A later plain finding that repeats an old evidence reference changes the proxy ranking of an unchanged decision and flips the next-turn cue on both File and SQLite.

// A decision can cite the agent's earlier measurements on this linked
// question without changing their provenance or assigning earlier work.
if (row.work_yardstick === undefined) {
return cited.size > 0 && row.agent_id === capture.binding.agent_id && Array.isArray(row.evidence_refs)

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 将引用限制在决策当时的来源

这个分支会接纳决策提交后写入的普通 finding。实际 CLI 对照先写 proxy A=8、B=6,再提交引用它们的决策(outcome A=20、B=14),最后追加沿用 result:preview-b 的 proxy B=99。两种 provider 都把最新 finding 的 99 计入原决策,下一轮首屏因此从排序一致翻成反转。请用现有来源顺序约束已引用的测量,保留合法的较早记录,并在 File 和 SQLite 复测晚写同名引用。

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant