Skip to content

Bind benchmark checkpoints to the resumed phase clock - #8398

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-retained-phase-clock
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/benchmark-retained-phase-clock

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

When an interrupted benchmark worker resumes in place, its original clock stays on disk as recovery evidence. The checkpoint supervisor previously always read that original path. A later checkpoint could therefore record an exhausted allowance while the resumed scheduler was still within its admitted phase.

Allow the host supervisor to bind the active phase clock once and pass that same path to held and terminal captures. The normal fresh-worker path stays the default. The scheduler and native evaluator continue to own the deadline and scoring opportunities.

Validation: 27 focused tests passed in the Linux guest with the pinned native evaluator. The Docker tests keep a deliberately expired original clock alongside a separate active clock, then verify both capture paths seal the active value and preserve the original bytes. RFC diff check passed.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning_effort=max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

动机 / Motivation

原地续跑保留旧阶段时钟作为恢复凭证,恢复后的 scheduler 则使用单独获准的新时限。此时再捕获 checkpoint,记录的剩余额度应该来自这次续跑。评审覆盖已发布的续跑入口、正常 runner 的 supervisor 构造、held capture 和失败后的 terminal preservation。这次对照为同一保留 worker 提供已过期的原文件和独立的新时钟。当前改动使两个适配器都能接收新文件,但续跑入口没有把新时钟交给它们。评审走实际入口和捕获请求,Docker daemon 与 evaluator seal 由夹具模拟。

改动思路 / Approach

复用已有的 phase_deadline_path 请求字段和捕获 owner。显式选择一份物理时钟后,held 和 terminal 请求都用这份文件,原文件及 evaluator 机会和冷却仍由原 owner 管理。改动范围限于 benchmark 适配器构造和捕获请求。缺少的是在已有续跑路径上绑定已经获准的新时钟。

具体改动 / Changes

评审 3535876,基线 c44eb15,6 个路径,+49/−14。对照原地继续的入口契约:same-active-clock 的续跑入口尚未接线,capture-routing 和 fresh-default-and-source-retention 的适配器行为已验证。

关键代码讲解

对主干的风险 / Risks

P2 · 受支持的原地续跑仍无法让自动捕获使用新时钟。 同一份保留 worker 夹具放入已过期的原文件和剩余 800 秒的新文件。实际 scheduler 入口选择新阶段,正常 runner 构造的 supervisor 却在基线和候选上都让 held、terminal 请求读到旧文件和 0 秒。候选显式给构造函数传入新路径后,两条请求都读到 800 秒;新建阶段在两版都读到默认文件的 400 秒,原文件字节均不变。把当前已核准的时钟接到现有续跑和捕获交接处,并从该入口覆盖两个请求;保留旧证据和未继续时的默认值。

我的整体评价 / Verdict

请求修改续跑接线。独立对照检查了两个捕获请求、归档读取和阶段预算计算。两版各 4 个对照用例及 43 个现有显式恢复用例通过;候选的相关源码套件 448 项通过、45 项跳过。standard premerge 的 5 项直接检查和 19 项自动检查通过,benchmark 人工门单独保留。本次没有复跑作者报告的 Linux/Docker seal。

English verdict: REQUEST_CHANGES. Both adapters honor an explicit alternate clock, but the shipped in-place continuation still leaves automatic capture on the expired default. Wire the verified active clock into that caller while preserving the original witness.

def __init__(self, controller_file: Path, destination: Path, retention_policy: Path | None = None,
*, retention_enabled: bool = True, terminal_components: list[dict[str, str]] | None = None,
terminal_receipt: Path | None = None):
terminal_receipt: Path | None = None, phase_deadline_path: str = _PHASE_DEADLINE):

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2:原地续跑没有把新时钟交给这两个捕获路径。 当前唯一的非测试构造点 edgebench.run 没传这个参数;runtime.scheduler 用新 epoch 选择续跑状态,但没有对应的 supervisor 绑定。用两份物理时钟重放,续跑获准 800 秒后,正常构造出的 held 和 terminal 请求仍从旧文件读到 0 秒;只有手动给构造函数传入新路径,两条请求才读到 800 秒。请在现有续跑的捕获交接中绑定已核准的新时钟,并从该入口验证两种请求及原文件字节不变。

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant