Skip to content

Avoid worker-pool waits for fresh async credential cache hits - #2061

Open
itsrayeedscodespace wants to merge 1 commit into
anthropics:mainfrom
itsrayeedscodespace:perf/async-cached-credentials
Open

itsrayeedscodespace wants to merge 1 commit into
anthropics:mainfrom
itsrayeedscodespace:perf/async-cached-credentials

Conversation

@itsrayeedscodespace

Copy link
Copy Markdown

Fresh cached credentials on AsyncAnthropic currently require a worker-pool round trip when the credential provider is synchronous. Unrelated blocking work can therefore stall a request before it reaches the HTTP transport, even though its token is already cached and needs no refresh.

Return fresh cached tokens under the existing cache lock before dispatching work. Cache misses, the advisory/mandatory windows, per-request (expires_at=0) providers and invalidation continue through the existing worker path. Provider calls remain off the event loop. Tokens with no expiry retain their existing caching behavior.

The regression uses AsyncAnthropic.models.list(), MockTransport and a real single-worker default executor. After priming the cache, an unrelated task occupies the worker until the request finishes. On upstream both fresh-token cases time out at the five-second test guard; with the change both requests complete before the worker is released. The test owns its loop/executor and releases the blocker in finally.

Validation on Apple M5/macOS arm64, Python 3.11.18:

  • New cases on unchanged upstream: 2 failed, 7 passed.
  • Full tests/lib/test_credentials.py: 456 passed, 2 skipped.
  • Added boundary checks at 121/120/60/30/0/-1 seconds remaining, invalidation/force-refresh and per-request tokens; synchronous provider execution stays off-loop.
  • A separate public-client Trio/AnyIO reproduction also blocks on upstream's occupied worker pool and completes with the change, with one provider call in both cases.
  • Changed-file Ruff lint/format, Pyright (zero diagnostics), and git diff --check passed.

Seven warmed samples, alternating order, synthetic credentials and in-memory transport (no live API calls):

Case Before After
Cached lookup, future expiry 27.24 µs 0.18 µs
models.list(), future expiry, free worker 172.29 µs 128.52 µs
models.list(), future expiry, worker occupied for 100 ms 101.86 ms 0.73 ms

The last case demonstrates isolation from unrelated worker contention; it is an intentionally constrained executor, not a production throughput claim. With the normal network path, network latency can dominate the small scheduling savings. The fast path briefly acquires the existing mutex and reads the configured clock on the loop, just as the async-provider path already does.

Searched open/closed TokenCache, async_get_token, credential/cache/executor/worker reports and local commit history. #2058 changes fallback after advisory-refresh failure; closed #1841 concerned invalidation during refresh. Their diffs do not implement this fresh-cache path. Workload identity provider proposals #1903/#2010 address a different provider interface.

AI assistance was used for implementation, reproduction and verification.

@itsrayeedscodespace
itsrayeedscodespace requested a review from a team as a code owner November 12, 2026 03:56

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant