Skip to content

Keep resumed scheduler state in benchmark failure checkpoints - #8400

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-terminal-scheduler-phases
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-terminal-scheduler-phases

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

When a retained benchmark worker fails again after an in-place continuation, the default terminal capture still includes only the original scheduler state. That omits the latest failure and any intermediate phase state selected by the scheduler.

Reuse the scheduler's existing phase naming rule in the terminal component builder. An operator can bind the admitted deadlines in predecessor order, keeping the root and those exact phase states without capturing the control directory. Fresh captures retain their current inputs. Recovery still requires the stopped worker and its Host witness.

Validation: 85 tests passed in the Linux guest with the pinned native evaluator. The Docker failure test has the real scheduler select two continuation phases, then verifies that the native terminal checkpoint captures both and the original state, preserves scoring opportunity, and excludes an unrelated control file. RFC diff check passed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机 / Motivation

需要保留原地续跑失败证据的 EdgeBench 操作者,依赖终态检查点回读当时的调度状态。用真实 scheduler 创建两段续跑状态并在第二段保留失败。取两版生产调用实际传入的参数生成请求,两个请求都包含根状态、零个 phase 文件。本地执行了实际调度器与请求校验。本机 Docker daemon 不可用。

改动思路 / Approach

候选复用 phase_state_name,在 terminal_components 接到截止时间时追加精确文件。显式传入两段时,组件顺序和调度器选出的文件一致,重复与非法时钟会被拒绝。调度器阶段文件继续由原有恢复逻辑管理,导出的组件交给现有 checkpoint worker。控制目录整体仍在捕获范围之外。

具体改动 / Changes

评审提交 67e5f18785cfc20500392eb82c8a5ad3e89aebb2,PR 基线 c44eb156d947198b9b5e36e1f9444afbbf9b205b,4 个文件,+71/−13。已核对现有终态捕获约定,clock-naming-parity 和 bounded-component-selection 已验证。正常入口的 shipped-terminal-phase-binding 缺失,full-native-boundary 仍需在能运行本地 Docker 的环境验收。文档要求调用者按前序顺序传入截止时间,测试通过新增参数注入两个阶段。实际入口在 solver 启动前创建组件清单,负责终态捕获的监督器直接使用这份清单。终态捕获约定

关键代码讲解

对主干的风险 / Risks

[P2] 正常终态捕获会漏掉续跑后的失败阶段。 显式参数能生成正确的组件清单。新增 Docker 测试也直接传列表,跳过了生产调用。最小修复是让失败保留入口从 scheduler 已准入的阶段链构建组件,在现有写入屏障内核验,并通过正常入口测试封存回读。

30 项相关测试通过,12 项可选验证跳过。标准预合并的 5 项直接及 19 项自动检查通过,仍有一项 benchmark 人工门槛。

我的整体评价 / Verdict

请先把阶段链接入实际失败保留路径。当前运行入口产生的检查点仍无法独立读出最新阶段的失败记录,排查仍依赖原停止 worker。后续恢复仍以该 worker 和对应 Host 见证为依据。

English verdict: REQUEST_CHANGES. At 67e5f18, the production terminal-preservation caller passes no phase deadlines, so a resumed failure still exports zero phase files. Bind the scheduler-owned phase chain at the shipped capture boundary.

components.extend({
"id": f"terminal_scheduler_phase_{index}",
"source_path": str(root.with_name(phase_state_name(root, deadline))),
} for index, deadline in enumerate(scheduler_phase_deadlines, start=1))

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] 让失败保留入口传入已准入的 scheduler phase 链

这里仅在调用方提供列表时追加阶段。唯一生产调用 benchmark.edgebench.run 仍执行 terminal_components(declaration, task.cwd)。本地用真实调度器创建两段续跑并保留最后一次失败后,两版生产调用都生成零个阶段组件。新 Docker 用例直接传列表,无法覆盖此缺口。请从实际 scheduler 阶段链绑定捕获组件,并从正常失败保留入口验证根状态、前序阶段和最新失败均可回读。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant