Repository navigation
perf(flat): single-pass tokenized flank and window assembly - #432
Merged
Merged
Conversation
Reference flanks were read via fetch_windows into a ragged byte buffer, copied into separate f5/f3 arrays, interleaved, then tokenized: 3-4 passes over every flank byte. They are now read straight from the reference into the interleaved [f5|f3] layout with the LUT applied while copying (ref_flanks), and the deleted span of long deletions is skipped. ref_window is one tokenizing read of [start-L, end+L); alt_window reuses hap_windows with token flanks. haplotype_flanks now maps its per-row buffer through the LUT once and copies token slices, dropping the final full-size tokenize pass. Kernel microbenchmark (524k variants): reference flank_tokens 2.8-3.4x faster, windows 1.3-2.0x faster; haplotype flanks 1.4-3x faster than on main's reference path. The unfused helpers stay as #[cfg(test)] oracles for a randomized equivalence test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ckerlab/GenVarLoader into perf/ref-flanks-single-pass
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #431 (merged). Originally stacked on it; now targets
main.What
Single-pass, tokenize-while-copying assembly for variant flanks and windows in
src/variants/windows.rs. Output is byte-identical; only speed changes.Reference path (the default, so this speeds up every existing user):
fetch_windowsread[start-L, end+L)into a ragged byte buffer. Thenslice_flankscopied it into f5 and f3, an interleave copied it again, andtokenizemade a fourth pass.ref_flanksreadsref[start-L, start)andref[end, end+L)straight into the interleaved[f5|f3]output and applies the LUT while copying. It never reads the deleted span of long deletions.ref_windowis one tokenizing read of[start-L, end+L).alt_windowreuseshap_windowswith reference token flanks:f5·ALT·f3, with empty-ALT deletions re-anchored toref[start]as before.Haplotype path (from #431):
haplotype_flanksnow LUT-maps its short per-row buffer once and copies token slices. This drops the final full-size tokenize pass.Benchmark
Rust kernel microbenchmark, 524k variants, best of 5, times in ms.
mainrefReference flank_tokens got 2.8–4.1x faster and reference windows 1.3–2.0x faster. Haplotype flanks are now 0.94–1.7x the time of reference flanks; #431's "faster than reference" comparison was against the unfused path. The docs in
faq.mdandSKILL.mdare corrected to match.Tests
fetch_windows,slice_flanks,anchor_empty_del_alts,assemble_alt_window) are kept as#[cfg(test)]oracles.tokenize(haplotype bytes).pytest tests: 1358 passed, 0 failed.cargo test --lib: 144 passed.🤖 Generated with Claude Code