Skip to content

perf(flat): single-pass tokenized flank and window assembly - #432

Merged
d-laub merged 4 commits into
mainfrom
perf/ref-flanks-single-pass
Nov 20, 2026
Merged

d-laub merged 4 commits into
mainfrom
perf/ref-flanks-single-pass

Conversation

@d-laub

@d-laub d-laub commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #431 (merged). Originally stacked on it; now targets main.

What

Single-pass, tokenize-while-copying assembly for variant flanks and windows in src/variants/windows.rs. Output is byte-identical; only speed changes.

Reference path (the default, so this speeds up every existing user):

  • Before: fetch_windows read [start-L, end+L) into a ragged byte buffer. Then slice_flanks copied it into f5 and f3, an interleave copied it again, and tokenize made a fourth pass.
  • Now: ref_flanks reads ref[start-L, start) and ref[end, end+L) straight into the interleaved [f5|f3] output and applies the LUT while copying. It never reads the deleted span of long deletions.
  • ref_window is one tokenizing read of [start-L, end+L).
  • alt_window reuses hap_windows with reference token flanks: f5·ALT·f3, with empty-ALT deletions re-anchored to ref[start] as before.

Haplotype path (from #431): haplotype_flanks now LUT-maps its short per-row buffer once and copies token slices. This drops the final full-size tokenize pass.

Benchmark

Rust kernel microbenchmark, 524k variants, best of 5, times in ms.

Density L Output main ref ref (this PR) hap (this PR)
~1/300 bp 16 flank_tokens 80.3 19.6 29.3
64 flank_tokens 150.2 49.5 68.0
256 flank_tokens 497.8 153.9 188.1
16 windows 99.5 50.4 60.0
64 windows 189.2 127.7 141.8
256 windows 646.8 417.2 419.4
~1/8 bp 16 flank_tokens 40.7 12.8 21.7
64 flank_tokens 151.0 44.6 50.9
256 flank_tokens 479.7 147.9 160.0
16 windows 65.2 36.0 51.8
64 windows 192.1 119.2 116.3
256 windows 636.1 409.0 383.8

Reference flank_tokens got 2.8–4.1x faster and reference windows 1.3–2.0x faster. Haplotype flanks are now 0.94–1.7x the time of reference flanks; #431's "faster than reference" comparison was against the unfused path. The docs in faq.md and SKILL.md are corrected to match.

Tests

  • The old unfused helpers (fetch_windows, slice_flanks, anchor_empty_del_alts, assemble_alt_window) are kept as #[cfg(test)] oracles.
  • A new randomized equivalence test runs 240 cases each for u8 and i32 LUTs, with L ∈ {1, 4, 9}. Cases span 2 contigs, starts at both contig edges, deletions past the contig end, and empty-ALT deletions. It asserts:
    • fused flank_tokens == unfused output;
    • ref_window and alt_window data and offsets == unfused output, for all 4 ref/alt mode combinations;
    • haplotype tokens == tokenize(haplotype bytes).
  • Full pytest tests: 1358 passed, 0 failed.
  • cargo test --lib: 144 passed.
  • clippy: no new warnings.

🤖 Generated with Claude Code

Reference flanks were read via fetch_windows into a ragged byte buffer,
copied into separate f5/f3 arrays, interleaved, then tokenized: 3-4
passes over every flank byte. They are now read straight from the
reference into the interleaved [f5|f3] layout with the LUT applied while
copying (ref_flanks), and the deleted span of long deletions is skipped.
ref_window is one tokenizing read of [start-L, end+L); alt_window reuses
hap_windows with token flanks.

haplotype_flanks now maps its per-row buffer through the LUT once and
copies token slices, dropping the final full-size tokenize pass.

Kernel microbenchmark (524k variants): reference flank_tokens 2.8-3.4x
faster, windows 1.3-2.0x faster; haplotype flanks 1.4-3x faster than on
main's reference path. The unfused helpers stay as #[cfg(test)] oracles
for a randomized equivalence test.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@d-laub
d-laub changed the base branch from feat/personal-flanks to main November 20, 2026 18:38
@d-laub
d-laub merged commit 889700e into main Nov 20, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant