Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/source/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Both are sparse columnar variant archives from [`genoray`](https://github.com/mc

`.svar2` is Phase-1 scope: a handful of combinations (`annotated` haplotypes, `min_af`/`max_af`, `VarWindowOpt(ref="allele")`, fixed-length haplotype-realigned tracks, spliced `variant-windows`, and `variants`/`variant-windows` output with jitter) aren't wired yet and raise `NotImplementedError` rather than silently mis-computing. Haplotype and `variants` output support splicing, `var_filter="exonic"`, and negative-strand reverse-complementation. `"variant-windows"` output, `unphased_union` (for both `"variants"` and `"variant-windows"`), and `var_fields`-selected store INFO/FORMAT fields (also for both, when the `.svar2` was written with them) are also supported. See the `genvarloader` skill's `.svar2` section or `docs/source/format.md` for the full list. Everything else — haplotypes, tracks, and variants/variant-windows at any supported jitter/output-length combination — is byte-identical between the two backends.

One documented difference in raw output: for a pure deletion, `with_seqs("variants")` on a `.svar` dataset reports the VCF anchor base as ALT (e.g. `b"G"` for `GTA>G`), while a `.svar2` dataset reports the atomized empty ALT (`b""`) — a genoray `.svar2` format convention, not a bug. Reconstructed haplotypes are unaffected; only `RaggedVariants.alt` differs (and `FlatVariantWindows.alt`/`.alt_window` for `"variant-windows"`), and only for pure-deletion records. `ref_window` is byte-identical between the two backends.
One documented difference in raw output: for a pure deletion, the bare ALT allele from a `.svar` dataset is the VCF anchor base (e.g. `b"G"` for `GTA>G`), while a `.svar2` dataset reports the atomized empty ALT (`b""`), which is how genoray's `.svar2` format stores pure deletions. This affects only the bare allele bytes, `RaggedVariants.alt` from `with_seqs("variants")` and the `alt` field of `"variant-windows"` with `VarWindowOpt(alt="allele")`, and only for pure-deletion records. The flanked windows do not differ: `ref_window` and `alt_window` are byte-identical between the two backends, and `alt_window` is the reference with that variant applied (for `GTA>G`, `flank5 . G . flank3`), because gvl restores the anchor base of an empty-ALT deletion when it assembles the window. Reconstructed haplotypes are byte-identical too.

## Can I build a dataset in parallel shards and merge them?

Expand Down
25 changes: 15 additions & 10 deletions docs/source/format.md
Original file line number Diff line number Diff line change
Expand Up @@ -149,16 +149,21 @@ If none match, a `FileNotFoundError` is raised naming the expected `.svar2` base
suggesting `svar2=`. After resolution, the fingerprint (`Svar2Fingerprint`, above) is verified;
a mismatch raises `ValueError` and lists both expected and observed values.

## `.svar2` variants ALT convention

For a pure deletion (e.g. VCF `GTA>G`), decoding `with_seqs("variants")` yields different raw
ALT bytes depending on the backing store: `.svar` reports the VCF anchor base (`b"G"`), while
`.svar2` reports the atomized empty ALT (`b""`) — a genoray `.svar2` format convention, not a
bug. Both stores consume the ALT identically when reconstructing haplotype sequence, so
`with_seqs("haplotypes")` / `with_seqs("annotated")` output is byte-identical between the two
backends; only `RaggedVariants.alt` differs, and only for pure-deletion records. The same holds
for `with_seqs("variant-windows")`: `ref_window` is byte-identical between the backends, while the
`alt`/`alt_window` fields differ only for pure-deletion records (the same empty-vs-anchor ALT).
## `.svar2` pure-deletion ALT bytes

genoray's `.svar2` stores an atomized pure deletion (e.g. VCF `GTA>G`) at the VCF anchor
position with an empty ALT (`b""`); `.svar` keeps the VCF anchor base (`b"G"`). The variant's
position, `ilen` and reference span (anchor plus deleted bases) are the same in both stores.

- Bare ALT bytes differ: `RaggedVariants.alt` from `with_seqs("variants")`, and the `alt` field
of `with_seqs("variant-windows")` with `VarWindowOpt(alt="allele")`, are `b""` on `.svar2` and
`b"G"` on `.svar`, for pure-deletion records only.
- Flanked windows do not differ. `ref_window` reads `[start-L, end+L)` from the reference, and
`alt_window` is `flank5 . ALT . flank3` with the anchor base restored for an empty-ALT
deletion, so both are byte-identical between the backends and `alt_window` equals the
reference with that one variant applied.
- Haplotypes (`with_seqs("haplotypes")` / `"annotated"`) are byte-identical: reconstruction
also keeps the anchor (`ref[pos]`) for an empty-ALT deletion.

## `.svar2` Phase-1 unsupported combinations

Expand Down
51 changes: 47 additions & 4 deletions python/genvarloader/_dataset/_flat_flanks.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,39 @@ def _assemble_alt_windows(f5, f3, alt_data, alt_seq_off, flank_len):
return out, out_off


def _anchor_empty_del_alts(
alt_data: NDArray[np.uint8],
alt_seq_off: NDArray[np.int64],
ilens: NDArray[np.int32],
rw_data: NDArray[np.uint8],
rw_off: NDArray[np.int64],
flank_len: int,
) -> tuple[NDArray[np.uint8], NDArray[np.int64]]:
"""Re-insert the anchor base into empty-ALT pure deletions before alt-window assembly.

SVAR2 stores an atomized pure deletion with an empty ALT at the VCF anchor
position, so its ``[start, end)`` span is anchor + deleted bases, the same as
SVAR1's ``GTA>G``. Haplotypes keep the anchor (``ref[pos]``), so the alt window
must too. The anchor is the first byte of the variant's ref-window span,
``rw_data[rw_off[i] + L]``. Returns the inputs unchanged when nothing needs
re-anchoring. Mirrors Rust ``anchor_empty_del_alts``.
"""
alt_data = np.asarray(alt_data, np.uint8)
alt_seq_off = np.asarray(alt_seq_off, np.int64)
ilens = np.asarray(ilens, np.int32)
alt_len = np.diff(alt_seq_off)
needs = (alt_len == 0) & (ilens < 0)
if not needs.any():
return alt_data, alt_seq_off
idx = np.flatnonzero(needs)
anchors = np.asarray(rw_data, np.uint8)[
np.asarray(rw_off, np.int64)[idx] + flank_len
]
new_data = np.insert(alt_data, alt_seq_off[idx], anchors)
new_off = lengths_to_offsets(alt_len + needs, np.int64)
return new_data.astype(np.uint8, copy=False), new_off


def compute_ref_window(
reference,
v_contigs: NDArray[np.integer],
Expand Down Expand Up @@ -153,14 +186,21 @@ def compute_alt_window(
lut: NDArray,
row_offsets: NDArray[np.int64],
) -> "_FlatWindow":
"""Alt window = tokenized ``flank5 . alt . flank3`` assembly."""
"""Alt window = tokenized ``flank5 . alt . flank3`` assembly.

An empty-ALT pure deletion (SVAR2) gets its anchor base back first, so the
window equals the haplotype around the variant (see ``_anchor_empty_del_alts``).
"""
starts = np.asarray(starts, np.int32)
ilens = np.asarray(ilens, np.int32)
ends = starts - np.minimum(ilens, 0) + 1
rw = reference.fetch(v_contigs, starts - flank_len, ends + flank_len)
f5, f3 = _slice_flanks(
rw.data.view(np.uint8), np.asarray(rw.offsets, np.int64), flank_len
) # each (n, L)
rw_data = rw.data.view(np.uint8)
rw_off = np.asarray(rw.offsets, np.int64)
f5, f3 = _slice_flanks(rw_data, rw_off, flank_len) # each (n, L)
alt_data, alt_seq_off = _anchor_empty_del_alts(
alt_data, alt_seq_off, ilens, rw_data, rw_off, flank_len
)
alt_bytes, alt_off = _assemble_alt_windows(
np.ascontiguousarray(f5).reshape(-1),
np.ascontiguousarray(f3).reshape(-1),
Expand Down Expand Up @@ -223,6 +263,9 @@ def compute_windows(

# alt window: flank5 . alt . flank3 from sliced flanks.
f5, f3 = _slice_flanks(data, rw_off, flank_len)
alt_data, alt_seq_off = _anchor_empty_del_alts(
alt_data, alt_seq_off, ilens, data, rw_off, flank_len
)
alt_bytes, alt_off = _assemble_alt_windows(
np.ascontiguousarray(f5).reshape(-1),
np.ascontiguousarray(f3).reshape(-1),
Expand Down
4 changes: 2 additions & 2 deletions skills/genvarloader/SKILL.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Unlike `.svar` (whose read path builds an interval search tree + a per-read dens

(`FlankSample` insertion-fill for tracks spanning multiple contigs in one query is now supported and byte-identical to the `.svar` backend — issue #267.)

**`variants`/`variant-windows` ALT bytes differ from `.svar` for pure deletions (format convention, not a bug).** For a pure deletion (e.g. VCF `GTA>G`), `with_seqs("variants")` on a `.svar` dataset yields the VCF anchor base as ALT (`b"G"`), while a `.svar2` dataset yields the atomized empty ALT (`b""`) — this is how genoray's `.svar2` format represents pure deletions. The same convention carries into `with_seqs("variant-windows")`: `ref_window` is byte-identical between `.svar`/`.svar2`, but `alt`/`alt_window` differ for pure-deletion records for the same reason. Reconstructed **haplotypes are byte-identical** between the two backends (both consume the ALT identically when building sequence); only the raw allele/window bytes differ for pure-deletion records. See `docs/source/faq.md`.
**Bare ALT bytes differ from `.svar` for pure deletions; windows do not.** For a pure deletion (e.g. VCF `GTA>G`), `with_seqs("variants")` on a `.svar` dataset yields the VCF anchor base as ALT (`b"G"`), while a `.svar2` dataset yields the atomized empty ALT (`b""`), which is how genoray's `.svar2` format stores pure deletions. The bare `alt` of `variant-windows` with `VarWindowOpt(alt="allele")` differs the same way. The flanked windows are byte-identical between the backends: `ref_window` reads the reference, and `alt_window` restores the anchor base of an empty-ALT deletion, so it is `flank5 . G . flank3` on both (the reference with that variant applied). Reconstructed **haplotypes are byte-identical** too. See `docs/source/faq.md`.

Symbolic/breakend variants are rejected the same as `.svar`, but for `.svar2` the rejection happens **upstream, at `.svar2` conversion time** (the store format cannot represent them) — a `.svar2` must be built from an already-filtered source; gvl cannot re-filter a materialized `.svar2` any more than it can a materialized `.svar`.

Expand Down Expand Up @@ -482,7 +482,7 @@ See `docs/source/format.md` for the full schema, versioning, and SVAR-link detai
- **`gvl.Table` is a core public API.** No extra install required. It uses a Rust COITrees overlap engine and is CI-covered. Import it as `gvl.Table` (re-exported from the top-level package).
- **Symbolic / breakend variants are rejected, not skipped.** Remove them before `gvl.write` — e.g. `bcftools view -e 'ALT~"<" || ALT~"\["'` (drop SVs and breakends), or construct the genoray reader with `filter=genoray.exprs.is_biallelic & ~genoray.exprs.is_symbolic & ~genoray.exprs.is_breakend`. SVAR inputs must be built from an already-filtered source, since gvl validates but cannot re-filter a materialized `.svar`. For `.svar2` the same variants are rejected **upstream at `.svar2` conversion time** (genoray), not at `gvl.write` time — the store format cannot represent them at all.
- **`.svar2` has a Phase-1 unsupported-combination matrix.** `min_af`/`max_af`, `annotated` haplotypes, `VarWindowOpt(ref="allele")`, splicing or `var_filter="exonic"` with non-haplotype/track output, reverse-complemented or fixed-length haplotype-realigned tracks, `variants`/`variant-windows` output with jitter (`max_jitter>0` at write or `jitter>0` at read), and `extend_to_length=False` at write time all raise `NotImplementedError` on a `.svar2`-backed dataset instead of silently mis-computing. Plain haplotypes support splicing, exonic filtering, and negative-strand reverse-complementation. `with_seqs("variant-windows")`, `unphased_union`, `var_fields`-selected store INFO/FORMAT fields (on both `"variants"` and `"variant-windows"`), and multi-contig `FlankSample` track fills (issue #267) are also supported for `.svar2`. See "`.svar2` — the read-bound sparse variant format" above.
- **`.svar2` `variants`/`variant-windows` ALT bytes differ from `.svar` for pure deletions.** `.svar` keeps the VCF anchor base (`b"G"` for `GTA>G`); `.svar2` decodes the atomized empty ALT (`b""`). Reconstructed haplotypes are byte-identical either way; `ref_window` is also byte-identical — only raw ALT/`alt_window` bytes differ for pure-deletion records.
- **`.svar2` bare ALT bytes differ from `.svar` for pure deletions.** `.svar` keeps the VCF anchor base (`b"G"` for `GTA>G`); `.svar2` decodes the atomized empty ALT (`b""`) in `variants` output and `VarWindowOpt(alt="allele")`. Haplotypes, `ref_window` and `alt_window` are byte-identical either way (`alt_window` restores the anchor).
- Opening a genotypes-only dataset without a `reference=` defaults to the `"variants"` view (`RaggedVariants`), not `"haplotypes"`. Only `"variants"` is available without a reference; `with_seqs("haplotypes" | "annotated" | "reference")` raises `ValueError` if no reference was provided.
- `with_insertion_fill` raises unless the dataset has both haplotypes AND tracks active.
- `min_af` / `max_af` are supported for `.svar` only. A non-SVAR-backed dataset (plain BCF/PGEN) raises `RuntimeError`; a `.svar2`-backed dataset raises `NotImplementedError` specifically.
Expand Down
102 changes: 95 additions & 7 deletions src/variants/windows.rs
Original file line number Diff line number Diff line change
Expand Up @@ -90,6 +90,47 @@ pub fn assemble_alt_window(
(Array1::from_vec(out), Array1::from_vec(out_off))
}

/// Re-insert the anchor base into empty-ALT pure deletions before alt-window
/// assembly. SVAR2 stores an atomized pure DEL with an empty ALT (genoray keeps
/// no anchor in the key) at the VCF anchor position, so `[start, end)` spans the
/// anchor plus the deleted bases exactly as for SVAR1 (`GTA>G`). Haplotype
/// reconstruction keeps the anchor (`ref[pos]`), so the alt window must too, or
/// `flank5 . alt . flank3` silently drops a base and disagrees with both the
/// haplotype and the SVAR1 alt window. The anchor is the first byte of each
/// variant's ref-window span, `rw_data[rw_off[i] + L]`.
///
/// Returns `None` when no variant needs re-anchoring (the common SVAR1 case), so
/// callers keep the gathered buffers without a copy. Mirrors
/// `_anchor_empty_del_alts` in `_dataset/_flat_flanks.py`.
pub fn anchor_empty_del_alts(
alt_data: ArrayView1<u8>,
alt_seq_off: ArrayView1<i64>,
ilens_v: ArrayView1<i32>,
rw_data: ArrayView1<u8>,
rw_off: ArrayView1<i64>,
flank_len: usize,
) -> Option<(Array1<u8>, Array1<i64>)> {
let n = alt_seq_off.len() - 1;
let needs = |i: usize| alt_seq_off[i + 1] == alt_seq_off[i] && ilens_v[i] < 0;
if !(0..n).any(needs) {
return None;
}
let alt_s = alt_data.as_slice().expect("anchor_empty_del_alts: alt_data must be contiguous");
let rw_s = rw_data.as_slice().expect("anchor_empty_del_alts: rw_data must be contiguous");
let mut out: Vec<u8> = Vec::with_capacity(alt_s.len() + n);
let mut off: Vec<i64> = Vec::with_capacity(n + 1);
off.push(0);
for i in 0..n {
if needs(i) {
out.push(rw_s[rw_off[i] as usize + flank_len]);
} else {
out.extend_from_slice(&alt_s[alt_seq_off[i] as usize..alt_seq_off[i + 1] as usize]);
}
off.push(out.len() as i64);
}
Some((Array1::from_vec(out), Array1::from_vec(off)))
}

/// Fetch the per-variant reference window `[start-L, end+L)` into one flat
/// buffer, with `ends = starts - min(ilen, 0) + 1`. Returns `(data, rw_off)`
/// where `rw_off` are per-variant byte boundaries (len `n+1`). Reuses
Expand Down Expand Up @@ -251,14 +292,15 @@ pub fn assemble_windows_mode<Tok: Copy>(

// One fused fetch if either side needs a window read.
let need_fetch = ref_mode == 1 || alt_mode == 1;
let fetched = if need_fetch {
let (fetched, ilens_sel) = if need_fetch {
let (starts_v, ilens_v) = gather_starts_ilens(v_idxs, v_starts, ilens);
Some(fetch_windows(
let f = fetch_windows(
v_contigs, starts_v.view(), ilens_v.view(), flank_len, reference, ref_offsets,
pad_char,
))
);
(Some(f), Some(ilens_v))
} else {
None
(None, None)
};

// ref side (ordered first to match Python field insertion order).
Expand All @@ -278,13 +320,20 @@ pub fn assemble_windows_mode<Tok: Copy>(
if alt_mode == 1 {
let (rw_data, rw_off) = fetched.as_ref().expect("alt window needs a fetch");
let (f5, f3) = slice_flanks(rw_data.view(), rw_off.view(), l);
let (alt_bytes, alt_off) = assemble_alt_window(
f5.view(),
f3.view(),
let ilens_v = ilens_sel.as_ref().expect("alt window needs gathered ilens");
let anchored = anchor_empty_del_alts(
alt_data.view(),
alt_seq_off.view(),
ilens_v.view(),
rw_data.view(),
rw_off.view(),
l,
);
let (a_data, a_off) = match &anchored {
Some((d, o)) => (d.view(), o.view()),
None => (alt_data.view(), alt_seq_off.view()),
};
let (alt_bytes, alt_off) = assemble_alt_window(f5.view(), f3.view(), a_data, a_off, l);
let tok = tokenize(alt_bytes.view(), lut);
tok_bufs.push(("alt_window", tok, alt_off));
} else if alt_mode == 2 {
Expand Down Expand Up @@ -355,6 +404,45 @@ mod tests {
assert_eq!(off.to_vec(), vec![0i64, 3, 7]);
}

#[test]
fn test_anchor_empty_del_alts() {
// L=1. var0: SNP alt "A"; var1: SVAR2 pure DEL (ilen -2, empty ALT) whose
// ref window is [9 | 30 31 32 | 8] -> anchor 30; var2: empty ALT but
// ilen 0 (not a DEL) stays empty.
let alt_data = arr1(&[65u8]);
let alt_seq_off = arr1(&[0i64, 1, 1, 1]);
let ilens = arr1(&[0i32, -2, 0]);
let rw = arr1(&[1u8, 2, 3, 9, 30, 31, 32, 8, 4, 5, 6]);
let rw_off = arr1(&[0i64, 3, 8, 11]);
let (d, o) = anchor_empty_del_alts(
alt_data.view(),
alt_seq_off.view(),
ilens.view(),
rw.view(),
rw_off.view(),
1,
)
.expect("a pure DEL needs re-anchoring");
assert_eq!(d.to_vec(), vec![65u8, 30]);
assert_eq!(o.to_vec(), vec![0i64, 1, 2, 2]);

// SVAR1-style anchored DEL: nothing to do, no copy.
let alt_data = arr1(&[30u8]);
let alt_seq_off = arr1(&[0i64, 1]);
let ilens = arr1(&[-2i32]);
let rw = arr1(&[9u8, 30, 31, 32, 8]);
let rw_off = arr1(&[0i64, 5]);
assert!(anchor_empty_del_alts(
alt_data.view(),
alt_seq_off.view(),
ilens.view(),
rw.view(),
rw_off.view(),
1,
)
.is_none());
}

#[test]
fn test_fetch_windows() {
use ndarray::Array1 as A1;
Expand Down
Loading
Loading