Repository navigation
Avoid worker-pool waits for fresh async credential cache hits - #2061
Open
itsrayeedscodespace wants to merge 1 commit into
Open
itsrayeedscodespace wants to merge 1 commit into
itsrayeedscodespace wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fresh cached credentials on
AsyncAnthropiccurrently require a worker-pool round trip when the credential provider is synchronous. Unrelated blocking work can therefore stall a request before it reaches the HTTP transport, even though its token is already cached and needs no refresh.Return fresh cached tokens under the existing cache lock before dispatching work. Cache misses, the advisory/mandatory windows, per-request (
expires_at=0) providers and invalidation continue through the existing worker path. Provider calls remain off the event loop. Tokens with no expiry retain their existing caching behavior.The regression uses
AsyncAnthropic.models.list(), MockTransport and a real single-worker default executor. After priming the cache, an unrelated task occupies the worker until the request finishes. On upstream both fresh-token cases time out at the five-second test guard; with the change both requests complete before the worker is released. The test owns its loop/executor and releases the blocker infinally.Validation on Apple M5/macOS arm64, Python 3.11.18:
tests/lib/test_credentials.py: 456 passed, 2 skipped.git diff --checkpassed.Seven warmed samples, alternating order, synthetic credentials and in-memory transport (no live API calls):
models.list(), future expiry, free workermodels.list(), future expiry, worker occupied for 100 msThe last case demonstrates isolation from unrelated worker contention; it is an intentionally constrained executor, not a production throughput claim. With the normal network path, network latency can dominate the small scheduling savings. The fast path briefly acquires the existing mutex and reads the configured clock on the loop, just as the async-provider path already does.
Searched open/closed TokenCache, async_get_token, credential/cache/executor/worker reports and local commit history. #2058 changes fallback after advisory-refresh failure; closed #1841 concerned invalidation during refresh. Their diffs do not implement this fresh-cache path. Workload identity provider proposals #1903/#2010 address a different provider interface.
AI assistance was used for implementation, reproduction and verification.