Repository navigation
Avoid quadratic copying when decoding large batch results - #2060
Open
itsrayeedscodespace wants to merge 1 commit into
Open
itsrayeedscodespace wants to merge 1 commit into
itsrayeedscodespace wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Large batch-result records repeatedly copy the growing JSONL buffer because both decoders append each chunk to immutable bytes.
BaseAPIResponsefeeds these decoders 64-byte chunks, so decoding a long completion has quadratic copying cost.Use
bytearray.extend()and clear the buffer after decoding each record in both sync and async paths.json.loadsaccepts bytearray directly. The public API and chunking remain unchanged.On Apple M5 CPU, Python 3.11.18, httpx2 2.16.0 and Pydantic 2.14.3, measured through
client.messages.batches.results()/ its async counterpart using MockTransport, real response chunking and typed results. Medians of five warmed calls, alternating baseline/candidate order; no live API/network calls:The 1 MiB sync samples ranged 122.2–124.8 ms before and 3.17–3.46 ms after. Small-record costs were effectively unchanged. This measures SDK CPU work with synthetic payloads; network-limited throughput is not claimed.
Validation: 44 tests passed across JSONL decoders and GA/beta batch-results API paths; Ruff lint/format, changed-file Pyright (0 errors), and
git diff --checkpassed. Added fragmented Unicode/escaped-newline records at 1/64/4096-byte boundaries, with and without a trailing newline and with a following record to check buffer reset.Searched open/closed JSONL, JSONLDecoder, bytearray, quadratic/buffer PRs and commit history. This addresses record buffering, separate from the SSE accumulation work in #1663/#2022, newline handling in #1778 and error cleanup in #1932/#2007/#2049. Those changes touch related code but solve different problems. This generated/persisted module may need a generator-side equivalent.
AI assistance was used for implementation and verification.