Repository navigation
Conversation
CompressedJSON kept the index of each interned string in a plain object. Once an object has 2^23 (8,388,608) properties, V8 renumbers and re-sorts all of them on every new one, about a second each, so an input with more distinct short strings than that stalls: a JSON array of 9,000,000 distinct strings doesn't finish, while 8,000,000 take 16 s. A Map has no such limit. A side effect: the string "__proto__" is now interned once. Assigning it to a plain object set the prototype instead of adding a property, so it got a new index every time. The output doesn't change. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
CompressedJSON._stringIndexesis now aMap<string, number>instead ofa plain object.
internStringandhashCodeuse it throughget/set/keys. That's the whole change (one file).Related Issue
Fixes #3558
Motivation and Context
Once a plain object has 2^23 (8,388,608) properties, V8 renumbers and
re-sorts all of them on every new property, about a second each. Inputs
with more distinct short strings than that stall while being read. A
Maphas no such limit. Details and a reproduction are in the issue.A side effect: the string
"__proto__"is now interned once.obj["__proto__"] = indexon a plain object sets the prototype insteadof adding a property, so it used to get a new index every time. The
output doesn't change.
Previous Behaviour / Output
A JSON array of 9,000,000 distinct strings (the script in the issue):
no output; still running when stopped after 120 s. Scryfall's 3 GB
all-cards file: still running after two hours.
New Behaviour / Output
The 9,000,000-string file:
export type Strings = string[];in 18.6 s(peak 3.0 GiB). The 3 GB file: done in 152 s with
--max-old-space-size=49152(peak 15.7 GiB).How Has This Been Tested?
npm run build,npx vitest run(54 files, 241 tests pass) andnpx biome check packages/quicktype-core/src/input/CompressedJSON.ts.test/inputs/json/{samples,misc,priority}plus one with"__proto__"values (221 inputs), each as TypeScript, JSON Schema, Go, Python and
C#: 1,105 runs, none differ.
Screenshots (if appropriate):
n/a
Written by Claude (an AI assistant made by Anthropic) and posted on behalf of the account owner (EloB). The reproduction, fix and tests were checked by running them as described above.
🤖 Generated with Claude Code