Skip to content

fix: return from exhausted API fallback - #8342

Open
loopx-agent wants to merge 1 commit into
mainfrom
codex/api-quota-fallback-20261108
Open

loopx-agent wants to merge 1 commit into
mainfrom
codex/api-quota-fallback-20261108

Conversation

@loopx-agent

Copy link
Copy Markdown
Collaborator

Summary

  • Derive native API readiness from the current key candidate set rather than from whether any key is configured.
  • Stop projecting an active API route as api_active when every key is cooling or exhausted.
  • Return to an available subscription account when the API pool is unavailable, while keeping api_fallback enabled.
  • Bump the provider to 0.15.4 and cover the recovery path with policy and loopback integration tests.

Root cause

The Watcher treated a saved key pool as ready after billing exhaustion. Policy selection then preserved api_active when no subscription was immediately selectable, even though no API key could actually serve a request.

Validation

  • node --test packages/loopx-codex-provider-routing/smoke/native-policy.test.mjs packages/loopx-codex-provider-routing/smoke/openai-policy.test.mjs
  • uv run --python 3.12 --with pytest pytest tests/presentation/test_openai_fallback.py tests/presentation/test_openai_relay.py packages/loopx-codex-provider-routing/tests/test_api_stream_outcome.py tests/presentation/test_native_account_store.py -q
  • Results: 17 policy tests passed; 70 Python tests and 28 subtests passed.

Ordinary 429 responses retain temporary cooldowns. Only structured billing/quota failures participate in exhaustion handling. Validation uses synthetic keys and loopback/fake upstream transport; it does not claim a real paid-quota test.

Treat API readiness as current key eligibility instead of configuration presence, and stop projecting an exhausted pool as active. Add recovery coverage for returning to a subscription while api_fallback remains enabled.

Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>

@loopx-agent loopx-agent left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewer: model_agent, gpt-6.1-sol, OpenAI, runtime_reported, reasoning effort max

Request changes conclusion (author-owned PR; GitHub blocks formal self-review)

发现 / Findings

  1. [P2] 已选 API 路径耗尽后,自动回切订阅多等 4 分钟。 当两个订阅最近一次都显示耗尽、已经选中的 API 又没有可用 key 时,native_policy.ts:213 跌入通常给订阅使用的等待分支,重查间隔从 60 秒变为 300 秒。隔离 wheel 中,429 后订阅 B 已在原定重查前恢复:基线在第 60 秒回切,当前提交直到第 300 秒才回切;第 60 秒连订阅观测都被跳过。普通 429 的 30 秒 key 冷却到期后,观察器也仍显示不可用,直到第 300 秒才恢复报告并重新执行 relay 准备。保留准确的 all_unavailable,在已选 API 且 fallback 仍启用时维持原有 60 秒重查上限,允许更早的订阅重置唤醒。请加入不调用 Refresh 的时钟测试。
  2. [P2] 订阅额度读取期间 key 失效,仍会切进已知不可用的 API。 native.py:463 在外部订阅观测之前读取 key;观测返回后仍用旧布尔值。用真实 key store 在 observe 回调中写入最后一个 key 的结构化 quota 429,返回新鲜的订阅耗尽结果时,候选数已是零,当前提交仍执行了一次 API slot 切换。该场景是旧缺陷未覆盖的窗口。把候选读取放在外部观测之后、选择之前;若准备阶段还引入等待,在其有意义的效果边界复核。用同样的观测中失效夹具断言不准备 relay、不切换。
  3. [P2] 合并后的 wheel 仍叫 0.15.4,普通升级会保留旧路由。 pyproject.toml:7 与 native manifest 使用的 0.15.4 已被 #8394 合入 main。当前 main 与无冲突试合并树都产出同名 0.15.4 wheel,但实际观察器不同。隔离安装 main wheel 后,pip install --upgrade 指向试合并 wheel 返回同版本已安装,读回仍走旧的不可用 API 接续。新建不可变环境可直接装入新产物,但无法按版本区分两者。请基于最新 main 同步分发及两份 manifest 版本,重跑从 main wheel 到新 wheel 的安装和路由读回。

动机 / Motivation

开启 API 接续的 Codex 用户在订阅耗尽后,希望继续已有会话,并在订阅恢复时自动返回。旧版把配置了 key 当成 API 可用:所有 key 已冷却时仍能切到 API,状态也会继续说正在使用 API。当前提交能拒绝切到已经确认无可用 key 的 API,并在已选 API 不可用时给出准确的等待状态。此评审范围是路由资格、后台恢复和安装读回;真实扣费、正在运行的请求回放和新账号权限不在范围内。仍需补齐自动恢复时序、观测期间的 key 变化以及升级产物的版本区分。

改动思路 / Approach

沿用现有分工:Python 读私有 key 状态和订阅观测,TypeScript 决定路由,native owner 继续负责切换。当前 PR 应完成候选资格和不可用状态修正,同时保持已有回切时序,并交付可区分的安装版本。保留已合适的 key 候选 owner;只修改状态标签会让进入不可用 API 的旧行为继续发生。把可用性和下次观察时间分开,避免为显示等待状态而放慢恢复。订阅优先、手动选择、现有重启和空闲条件都继续归原 owner。

具体改动 / Changes

关键代码讲解

  • NativeAccounts.tick 由查看有没有已存 key 改为查看当前候选。无已知可用候选时不会正常进入 API;但它在读取订阅状态前就冻结了这一判断。
  • decide 先选择已恢复订阅,再处理已选 API。API 不可用时走已有的 all_unavailable,这同时改变了 retry_at。原有账号选择结果、开关和等待字段仍在同一个状态对象中。
  • Watcher._observe_once 按这个时间唤醒。受保护的 tick 会在 retry_at 到期前直接返回旧投影,因此即时刷新测试可以通过,后台仍错过原有的回切时间。App 已有相应文案,显示等待与显示当前所选槽位是两件事。

规范对照:packages/loopx-codex-provider-routing/NATIVE.md,版本 5f78e87。固定版本。NATIVE-FALLBACK-AUTHORITY 已覆盖独立 opt-in 和新鲜订阅耗尽;NATIVE-SUBSCRIPTION-PRIORITY 已覆盖新鲜可用订阅在选择时优先,后台发现该状态的时序退化见发现 1;NATIVE-REQUEST-BOUNDARY 沿用原 relay 的请求归属和不重放保证。NATIVE-INSTALLED-PROVIDER 的独立 wheel 和 doctor 可运行,升级版本冲突见发现 3。

对主干的风险 / Risks

同一组隔离观测 原基线 当前提交
初次准备 API 时,唯一 key 已知耗尽 切到 API 留在订阅槽位,未准备 relay
API 已选,429 后订阅恢复 60 秒回切 300 秒回切
订阅观察中途写入最后一个 key 的额度拒绝 切到 API 仍切到 API
未启用 API 接续或订阅证据未知 无 API 切换 无 API 切换

PASS:17 项 TypeScript policy 测试,70 项 Python 测试及 28 项子测试;从基线、当前提交、main 和试合并树分别安装的 wheel 均通过 doctor。对照调用实际的 Watcher、native TS bridge、私有 key store 和 wheel;原生槽位及额度响应使用合成 transport,未访问真实账号或付费模型。FAIL:后台重查时序、观察期间已知拒绝和同版本升级读回。UNVERIFIED:按当前策略未观察远端 CI,未启动实际 Codex App 或真实模型请求。

语义与 CI 对齐

复用既有 all_unavailable 合适,但它不能把已选 API 的重查从一分钟无提示地拉长到五分钟。将 key 判断留在现有 owner,保留原恢复边界,用普通 tick 和受控时钟重跑两个 429 分支。只在隔离试合并产物中把取样移到订阅观测之后、保留 60 秒上限并改用 0.15.5,对照分别恢复了拒绝、回切和升级结果;这些修复未出现在评审提交。

我的整体评价 / Verdict

应该保留基于候选的路由判断与准确状态。当前提交让自动恢复变慢,仍能在一个可复现的观测窗口切错路由,并且无法通过普通同版本升级送达已有安装,因此请求修改。检查了 App 的现有状态消费者和标签;没有布局变化,本次未做浏览器视觉检查。现有状态名和权限边界保持原样,无需再加一套状态或配置。修复后请用不触发手动刷新、无真实扣费的三组对照复验,再核对新提交。

English verdict: REQUEST_CHANGES at a887e6f. Preserve timely recovery, recheck key eligibility after subscription observation, and ship a distinct upgrade version.

@mergify

mergify Bot commented Nov 9, 2026

Copy link
Copy Markdown

This pull request has merge conflicts with main and cannot be merged
until they are resolved. Please rebase or merge the base branch, @loopx-agent.

Choose the remote for the base repository, not an out-of-date fork.
For a fork clone, first inspect git remote -v; upstream must point
to https://github.com/loopx-project/loopx.git. If it is absent, add it
with git remote add upstream https://github.com/loopx-project/loopx.git.
Then run:

git fetch upstream
git rebase upstream/main
# Resolve each conflict, git add the resolved files, then git rebase --continue.
git push --force-with-lease origin HEAD

For a same-repository clone whose origin points to
https://github.com/loopx-project/loopx.git, use origin instead of
upstream for fetch/rebase. If you prefer merging the base, use
git merge <base-remote>/main and push normally.

Keep the DCO Signed-off-by trailer on every commit when you rebase.
https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify Bot added the needs-rebase Mergify: the pull request has merge conflicts with its base branch label Nov 9, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

needs-rebase Mergify: the pull request has merge conflicts with its base branch

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant