Skip to content

Retain Official benchmark workers after confirmed failure - #8404

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-official-failure-retention
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-official-failure-retention

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

When the native Official solver still exited nonzero after its automatic resumes, the runner removed its worker container. This could erase unsubmitted work and the session needed to investigate or resume the run.

With the existing --preserve-failed-worker option, native feedback now keeps the exact stopped worker after a confirmed final solver failure. Intermediate retries and grading drain still run normally. The option stays off by default, and successful exits still clean up. The receipt identifies the preserved source; it does not certify that a resumable checkpoint was exported.

Validation: 353 focused tests passed. Docker tests used the pinned SForge provider and an independent native evaluator to verify automatic retry, one accepted grade, final worker retention, normal and flag-off cleanup, and retention when writing the receipt fails.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

Approval conclusion (author-owned PR; GitHub blocks formal self-approval)

动机 / Motivation

Official benchmark 在自动续跑后确认失败时,默认清理会一起删除原 worker 和未提交工作。对照重放:同一个 runner 先失败、自动续跑,再完成一次 native 评分和最终归档。显式启用后,当前提交保留确认失败的原 worker,原评分记录仍可读。此次使用真实 runner 与本地 native HTTP evaluator,Docker 传输和任务是合成夹具。真实 Docker 现场和后续续跑资格仍需单独验收。

改动思路 / Approach

复用现有 native 返回证据和最终 cleanup 入口。范围只包括显式启用的 Official/native 路径与已有停止核验。后一次 solver 派发清除上一段证据,只有确认的最终非零返回进入保留分支;其余沿原清理。保留收据区分停止确认、尝试捕获与恢复资格。

具体改动 / Changes

评审 ad706ed,PR 基线 c44eb15,6 个路径,+202/−27;实际主干 71190b6 的 #8402、#8403 改动与这些路径不重叠,合并树干净。对照原地续跑和证据边界:explicit-official-admission、retain-after-final-grade、uncertain-stop-and-receipt 已复测,installed-restore-qualification 仍独立。

关键代码讲解

  • CLI 只为 Official/native 和 agent-only 评分的显式选项启用。其他 profile 与原 checkpoint 路径保留自己的限制。
  • backend 从实际 solver 命令和 SForge 的结构化退出信息取证;最终清理 才请求停止。保留收据 不授予恢复资格。

对主干的风险 / Risks

固定 SForge 的实际 CLI 对照,两版各 7 个用例,都执行一次自动续跑、接收一次合成评分,并在归档和评分完成后才停止 worker。候选只保留已确认失败且显式启用的原 worker 和未提交字节;关闭选项、成功、未观察到终止或超时仍删除。停止失败时收据标为未确认,收据写入失败时无收据,两者都保留现场。宽套件 409 项通过、12 项 Docker 用例跳过;唯一缺少 Linux guest 解释器的失败在同环境基线也复现。预合并的 5 项直接和 19 项自动检查通过,benchmark 人工门仍单独保留。

我的整体评价 / Verdict

认可当前提交。真实 CLI 已完成启用、评分、停止与结果读回的隔离对照;保留现场不等于恢复验证。此次未复跑作者报告的真实 Docker 组合,也未启动真实 benchmark 或付费模型。

English verdict: APPROVE. Explicit Official retention preserves the same worker only after confirmed final failure and grading drain. Default cleanup and the separate checkpoint/restore boundary are preserved.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant