Repository navigation
Reading Blackrock recordings from fsspec URLs: is remote reading in scope for rawio? #1931
Description
We convert Blackrock recordings (NSx, NEV, PTP) that live in cloud blob storage. Today that means downloading each file first. We have a working implementation of BlackrockRawIO that reads straight from an fsspec URL (abfs://, s3://, ...) with byte-range reads, and we would like to upstream what you are willing to take. Before opening PRs we would like to know whether remote reading is something rawio should support at all, because neo is built around local files and memory maps and nothing in rawio reads remote files today (the only fsspec use I found is storage_options for kerchunk in xarray_utils.py).
What we have
BlackrockRawIO(filename="abfs://container/rec", storage_options={...}). A name containing://goes throughfsspec.core.url_to_fs; anything else behaves exactly as now.fsspecstays an optional import (aneo[remote]extra).- Local files keep the
np.memmappath unchanged. The remote path copies byte ranges instead, because a remote file cannot be memory-mapped. - Headers, signal chunks (standard and PTP) and
.nevpackets are read as byte ranges. A.nevis scanned in pieces instead of being read whole. - On every Blackrock test file in
ephy_testing_data, remote and local reads agree: headers, all signal chunks, timestamps, spikes, waveforms and events, including.nevscans in pieces as small as one packet. (Tests use an fsspec filesystem over the same local files, so CI needs no credentials.) - Memory: opening and streaming a 2.43 GB
.nevthrough fsspec peaked at 0.32 GB, against 4.63 GB for the existing local code path.
Proposed split, smallest first
- Generic file access (small, reusable by other rawios): helpers in
rawio/utils.pyto read records and strided values from an open file, plus an optional-fsspec opener. No change toBaseRawIO. - Blackrock streaming: chunked reads without mmap, PTP over a URL,
.nevstreaming. Blackrock-specific, most of the code. iter_spikes(): new public API that streams every unit's spikes in one pass. The per-unit methods each have to read the whole.nev, so a remote file would otherwise be read once per unit. This is the part we expect to need the most discussion.
Independent of this, we have sent or are sending small bug fixes as separate PRs (a .nev over 4 GiB failing under NumPy 2: #1929; the has_buffer_description_api() check: #1930).
Questions
- Is remote reading in scope for rawio, or do you see it as out of scope for neo?
- If it is: should it be opt-in per rawio (what we did for Blackrock), or a
BaseRawIOfeature (astorage_optionsargument, with file access going through the base class)? Per-rawio is much smaller, since 21 rawio modules usenp.memmap,np.fromfileormmapdirectly, but it leaves other formats to adopt it one by one. - Would you accept
iter_spikes()as new rawio API? If not, would you prefer we keep it out of neo? - Is the extra maintenance of a remote code path inside
blackrockrawio.py(about 700 added lines, tested against local files) acceptable? If not, we will maintain it in a fork and only send the generic pieces.
Happy to split or drop any part.
Activity
zm711 commentedon Oct 24, 2026
@cboulay thanks for bringing this up. I think traditionally we've been doing most remote reading through spikeinterface rather than directly in Neo. What do you think @h-mayorquin and @alejoe91 ?