Skip to content

fix(ai): investigations finish inside their clock and close out inside the context limit - #2719

Merged
Makisuo merged 3 commits into
mainfrom
triage/chat-agent-budgets
Nov 16, 2026
Merged

Makisuo merged 3 commits into
mainfrom
triage/chat-agent-budgets

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

Fixes two production error issues in maple-chat, both from investigations:

  • AgentPolicyError: Agent exceeded its 10m duration limit (322 events in 30 days, still firing today)
  • ContextBudgetError: Compaction could not fit the next model prompt inside the 128000 token context target

What the traces show (2026-11-11..16)

  • 45 of about 267 investigations (17%) ran their pass out of clock. Pull request reviews never did.
  • In those passes the model streamed for 90% of the time: 9-17 model calls at about 45 output tokens a second, reasoning growing to 3-5k tokens per call as context grows. Tools took 3-113s in total. No tool loop.
  • The 90s attempt cap fired 261 times. The cut call was thrown away and the low-effort retry still took 83s at the median (p95 150s), so one step cost about 3 minutes.
  • The engine's run-status warning comes at 80% (8 minutes). One such call fits in the 2 minutes left, so the pass could not submit in time.
  • All 38 ContextBudgetError spans in 14 days come from the close-out that runs after a stopped pass. Its first prompt was 123k of 128k tokens (39k system prompt plus 200k characters of evidence). submit_diagnosis landed, then the next round no longer fit (Run failed after its submission landed).

Changes

  • genai-spans.ts: a call past its attempt cap keeps going while the retry starts, and whichever answers first is used. Failing before the cap still fails at once, and the 4-minute call timeout still applies.
  • budgets.ts: investigation passes get 15 minutes. The warning then comes at 12, which leaves room for two slow calls. A close-out runs on its own clock of at most 5 minutes (closeOutAgent), so pass plus close-out plus margin stays within TURN_STALE_MS (25 minutes). Pull request reviews and replies keep their budgets.
  • close-out.ts: the close-out evidence drops from 200k to 120k characters (about 50k tokens), which leaves room for the round after submission.

This stays one agent turn ending in submit_diagnosis.

Verification

  • bunx vitest run src/platform/model-call-span.test.ts src/chat/budgets.test.ts src/chat/autonomous-phases.test.ts src/chat/close-out.test.ts src/chat/investigation.test.ts: 50 passed.
  • New tests:
    • keeps the first attempt past the cap and delivers it when it beats the retry: fails on main, passes here.
    • runs an investigation's close-out on its own short clock, not the pass's
    • leaves a close-out room for the round after it submits: fails at 200k.
    • The TURN_STALE_MS bound in budgets.test now uses pass plus the close-out clock.
  • oxlint -c .oxlintrc.effect.json passes on the changed files.

MODEL_ATTEMPT_CAP's comment in Llm.ts still says "cut". I left Llm.ts alone because #2715 edits it.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

…e the context limit

- A model call past its 90s attempt cap keeps running while the low-effort
  retry starts, and whichever answers first wins. Cutting it threw away 90s,
  and the retry took 83s more at the median.
- Investigation passes get 15 minutes. At 10, 17% of passes ran out of clock
  after 9-17 model calls at about 45 tokens a second.
- A close-out runs on its own 5-minute clock, so pass plus close-out stays
  under the turn watchdog.
- The close-out's evidence drops to 120k characters. At 200k it opened at
  123k of 128k tokens and the round after submit_diagnosis no longer fit.
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 15 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: ccc2cfd3-2133-4b78-8b05-3005b42e71c5
Commits

Reviewing files that changed from the base of the PR and between a032c8b and a7978bb.

Files selected for processing: 8

  • apps/ai/src/chat/autonomous-phases.test.ts
  • apps/ai/src/chat/autonomous-phases.ts
  • apps/ai/src/chat/budgets.test.ts
  • apps/ai/src/chat/budgets.ts
  • apps/ai/src/chat/close-out.test.ts
  • apps/ai/src/chat/close-out.ts
  • apps/ai/src/platform/genai-spans.ts
  • apps/ai/src/platform/model-call-span.test.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

Devin Review

Comment thread apps/ai/src/platform/genai-spans.ts Outdated
Comment on lines +518 to +520
Effect.race(
Fiber.join(first),
Stream.runCollect(attempts.retry(attempt(options, 2))),

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Failed responses cancel healthy retries

After the cap, Effect.race accepts an original response ending with finish.reason === "error" and cancels the retry. These provider failures complete Stream.runCollect successfully, so a retry that can recover the call never finishes.

Learn more

Provider failures can arrive as response data rather than Effect failures. The existing provider-failure fixture in model-call tests exercises a normally completed stream with an error finish reason. Collecting that stream succeeds, so the race treats it as a winning answer and interrupts the other request. Previously, the original request stopped at the cap and could not cancel its replacement this way.

Example: The original request streams reasoning for 100 seconds, then returns a finish part with reason error. Its retry starts at 90 seconds and would return a valid answer at 110 seconds. The original request wins at 100 seconds and cancels that retry.

Recommended fix: Validate each collected response before admitting it as a successful race result. Treat error parts and error finish reasons as failed attempts, allowing the other attempt to continue within the existing call timeout. Keep this response-specific validation outside the generic helper or pass it an explicit validation callback. Add a regression test with a late error finish on the original attempt and a later valid retry.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed: past the cap, a first attempt whose finish reason is error no longer wins the race; it waits for the retry and is only replayed if the retry fails too. Test added (fails without the fix).

@maple-review-bot

maple-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. No test covers a first attempt that fails with a typed error past the cap; every other path the change adds is pinned.
1 note · tests covered · risk medium

Past its attempt cap the first model call is no longer cut: the low-effort retry races it and the first answer wins, while investigation passes move to 15 minutes and the close-out runs on its own 5-minute clock with 120k characters of evidence. Safe to merge; the one earlier note on PhaseTurn.agent still stands.

  • retriedPastCap keeps the first attempt streaming and races the retry beside it
  • A first attempt answering with finish.reason: error past the cap waits for the retry
  • INVESTIGATION_BUDGET moves to 15 minutes; CLOSE_OUT_MAX_DURATION caps the close-out at 5
  • CLOSE_OUT_EVIDENCE_CHARS drops to 120k, leaving the round after submission room

Still open from earlier reviews

a7978bb · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit b1c0fa6 into main Nov 16, 2026
25 checks passed
@Makisuo
Makisuo deleted the triage/chat-agent-budgets branch November 16, 2026 16:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant