Skip to content

Split quoted attributes without the lookahead regex - #252

Merged
daler merged 1 commit into
daler:v0.15rcfrom
DylanCosto:perf-linear-attribute-splitting
Oct 21, 2026
Merged

daler merged 1 commit into
daler:v0.15rcfrom
DylanCosto:perf-linear-attribute-splitting

Conversation

@DylanCosto

Copy link
Copy Markdown

The quoted_semicolon_patterns regexes split on a separator only if the rest of the string has balanced quotes. That lookahead rescans the remaining text at every separator, so parsing is quadratic in the attribute length. It gets slow for GTF records with many attributes or quoted values that contain semicolons.

This replaces the regex with _split_on_unquoted_separator, which splits naively and then rejoins pieces until the number of quotes to the right is even. That's the same condition the lookahead checks, including for unbalanced quotes, but each piece is scanned once.

Rough timings on synthetic files:

  • iterating 2,000 GTF records with 40 attributes: 1.36 s → 0.15 s
  • building an in-memory db from 200 GTF records with 200 attributes: 4.6 s → 0.11 s
  • GFF3 records with only a few attributes: about the same

@daler

daler commented Oct 21, 2026

Copy link
Copy Markdown
Owner

This is great, thanks! Will target v0.15rc with this.

xref #212 #215

@daler
daler changed the base branch from master to v0.15rc October 21, 2026 14:11
@daler
daler merged commit e041e0f into daler:v0.15rc Oct 21, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants