Skip to content

Reading Blackrock recordings from fsspec URLs: is remote reading in scope for rawio? #1931

Open

Description

@cboulay

We convert Blackrock recordings (NSx, NEV, PTP) that live in cloud blob storage. Today that means downloading each file first. We have a working implementation of BlackrockRawIO that reads straight from an fsspec URL (abfs://, s3://, ...) with byte-range reads, and we would like to upstream what you are willing to take. Before opening PRs we would like to know whether remote reading is something rawio should support at all, because neo is built around local files and memory maps and nothing in rawio reads remote files today (the only fsspec use I found is storage_options for kerchunk in xarray_utils.py).

What we have

  • BlackrockRawIO(filename="abfs://container/rec", storage_options={...}). A name containing :// goes through fsspec.core.url_to_fs; anything else behaves exactly as now. fsspec stays an optional import (a neo[remote] extra).
  • Local files keep the np.memmap path unchanged. The remote path copies byte ranges instead, because a remote file cannot be memory-mapped.
  • Headers, signal chunks (standard and PTP) and .nev packets are read as byte ranges. A .nev is scanned in pieces instead of being read whole.
  • On every Blackrock test file in ephy_testing_data, remote and local reads agree: headers, all signal chunks, timestamps, spikes, waveforms and events, including .nev scans in pieces as small as one packet. (Tests use an fsspec filesystem over the same local files, so CI needs no credentials.)
  • Memory: opening and streaming a 2.43 GB .nev through fsspec peaked at 0.32 GB, against 4.63 GB for the existing local code path.

Proposed split, smallest first

  1. Generic file access (small, reusable by other rawios): helpers in rawio/utils.py to read records and strided values from an open file, plus an optional-fsspec opener. No change to BaseRawIO.
  2. Blackrock streaming: chunked reads without mmap, PTP over a URL, .nev streaming. Blackrock-specific, most of the code.
  3. iter_spikes(): new public API that streams every unit's spikes in one pass. The per-unit methods each have to read the whole .nev, so a remote file would otherwise be read once per unit. This is the part we expect to need the most discussion.

Independent of this, we have sent or are sending small bug fixes as separate PRs (a .nev over 4 GiB failing under NumPy 2: #1929; the has_buffer_description_api() check: #1930).

Questions

  1. Is remote reading in scope for rawio, or do you see it as out of scope for neo?
  2. If it is: should it be opt-in per rawio (what we did for Blackrock), or a BaseRawIO feature (a storage_options argument, with file access going through the base class)? Per-rawio is much smaller, since 21 rawio modules use np.memmap, np.fromfile or mmap directly, but it leaves other formats to adopt it one by one.
  3. Would you accept iter_spikes() as new rawio API? If not, would you prefer we keep it out of neo?
  4. Is the extra maintenance of a remote code path inside blackrockrawio.py (about 700 added lines, tested against local files) acceptable? If not, we will maintain it in a fork and only send the generic pieces.

Happy to split or drop any part.

Activity

  1. zm711 commented on Oct 24, 2026

    @zm711
    Contributor

    @cboulay thanks for bringing this up. I think traditionally we've been doing most remote reading through spikeinterface rather than directly in Neo. What do you think @h-mayorquin and @alejoe91 ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions