Skip to content

Prefetch neighbor bookkeeping and whole vectors in HNSW search - #1070

Closed
dor-forer wants to merge 1 commit into
mainfrom
dor-forer-hnsw-search-prefetch
Closed

dor-forer wants to merge 1 commit into
mainfrom
dor-forer-hnsw-search-prefetch

Conversation

@dor-forer

@dor-forer dor-forer commented Oct 1, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

HNSW search on Arm is bound by memory latency, not by distance computation. This PR changes how a node's neighbor list is scanned so the cache misses overlap instead of stalling one after another. Search results are unchanged.

processCandidate (top-k search and insertion) and processCandidate_RangeSearch now share one helper, forEachUnvisitedNeighbor, which scans the list in two passes:

  1. Prefetch the visited tag and the metadata entry of every neighbor in the list.
  2. In batches of 64, filter out visited and in-process neighbors (marking the rest visited), then compute distances in link order while prefetching every cache line of each vector 4 candidates ahead.

Before this change, the scan prefetched only the first 64-byte line of the next neighbor's vector and the next neighbor's tag, and never prefetched the metadata flags read by isInProcess. Locking is unchanged: the node's links lock is still held across the whole scan.

Why

Profile on Arm Neoverse-V2 (dorer-arm), standalone SQ8 HNSW index, dbpedia 1M x 768, ef_runtime=200, k=10, one core, query loop only (perf gated to the query phase):

  • Backend memory-stall cycles were 59% of all cycles.
  • processCandidate took 43% of cycles and the distance kernel 50%. Inside processCandidate, the hottest instructions were the isInProcess(id) flag load (about 17% of total cycles; the packed 9-byte ElementMetaData array is about 9 MB and was never prefetched) and the visited-tag check (about 14%; a 2 MB array prefetched one neighbor ahead). These shares are derived from sampled cycles: function share times instruction share within it.
  • Only the first 64-byte line of each vector was prefetched, while a 780-byte SQ8 vector spans 13 lines.

Results

Single-thread TopK latency, median of 3 interleaved rounds, 2,000 to 20,000 distinct queries per run, process pinned to one core. Same prebuilt index (dbpedia-cosine-dim768-M64-efc512-sq8.hnsw_v5, identical md5 on both machines), fp32 queries.

Arm Neoverse-V2 (dorer-arm)

ef_runtime / k main this PR speedup
10 / 10 0.168 ms 0.141 ms 1.20x
200 / 10 1.500 ms 1.162 ms 1.29x
100 / 100 0.872 ms 0.669 ms 1.30x
500 / 500 3.142 ms 2.415 ms 1.30x

x86 Xeon Platinum 8375C (dorer-intel)

ef_runtime / k main this PR speedup
10 / 10 0.132 ms 0.135 ms 0.98x
200 / 10 1.291 ms 1.258 ms 1.03x
100 / 100 0.736 ms 0.744 ms 0.99x
500 / 500 2.809 ms 2.643 ms 1.06x

Speedup is main latency divided by PR latency, so values below 1.00x are slower. On x86 the change is roughly neutral. At ef_runtime=100 the rounds overlap, so 0.99x is within noise. The ef_runtime=10 slowdown is small but consistent across the three rounds (0.134 to 0.136 ms vs 0.130 to 0.133 ms).

Prefetch lookahead sweep (Arm, ef_runtime=200, earlier prototype of the same scan): 1 = 0.937 ms, 2 = 0.885 ms, 4 = 0.873 ms, 8 = 0.893 ms, 16 = 0.930 ms.

A prototype that also moved the element flags into a separate dense 1-byte array was within noise of this PR (1.175 vs 1.164 ms at ef_runtime=200 on Arm), so this PR keeps ElementMetaData as is and prefetches its entry instead.

Correctness

  • The scan visits, marks, and scores exactly the same candidates in the same order as before. The benchmark driver checksums the result ids and scores of 1,000 queries: identical to main for every configuration on both machines.
  • make unit_test on dorer-arm: 100% tests passed, 0 tests failed out of 3140.

Not covered

  • Only single-thread, standalone HNSW TopK was timed. Range search and insertion use the same helper but were not benchmarked. Tiered indexes under concurrent writes were not benchmarked.
  • One dataset and dimension (768). Vectors with very few cache lines get less from the vector prefetch.
  • Unit tests ran on Arm only; x86 was benchmarked but its unit tests were not run in this round (CI covers it).
  • Not measured: issuing the tag and metadata prefetches per batch instead of for the whole list up front, which matters only for lists longer than 64 links.

🤖 Generated with Claude Code


Note

Medium Risk
Refactors the hottest HNSW graph-scan path used by TopK and range search; behavior should be identical but any mistake in visit filtering or ordering could affect query results.

Overview
HNSW neighbor expansion now goes through a shared forEachUnvisitedNeighbor helper instead of duplicated loops in processCandidate and processCandidate_RangeSearch.

The helper keeps the same visit order and locking (caller still holds the node links lock). It first prefetches every neighbor’s visited tag and ElementMetaData (for isInProcess), then processes links in batches of 64: skip visited/in-process nodes, mark the rest visited, prefetch all cache lines of each candidate vector with a 4-neighbor lookahead, and invoke the caller’s lambda for distance/heaps work.

This replaces the old one-ahead prefetch of a single vector cache line and drops the special-cased “last neighbor” loop. Search semantics are intended to be unchanged; the goal is lower memory latency on Arm (and roughly neutral on x86).

Reviewed by Cursor Bugbot for commit d686f2b. Bugbot is set up for automated code reviews on this repo. Configure here.

Split neighbor expansion in processCandidate and processCandidate_RangeSearch
into a shared two-pass scan: prefetch the visited tags and metadata of the
whole neighbor list, filter visited/in-process neighbors in batches, then
compute distances while prefetching every cache line of the vector four
candidates ahead. Results are unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer

Copy link
Copy Markdown
Collaborator Author

Benchmark results at 1M: ARM and x86, with an ablation

Bottom line: this scan is faster on both ARM and x86 and returns the same results, so it does not need an ARM-only guard. All of the gain comes from two parts working together: dropping visited neighbors first, then prefetching every line of each remaining vector 4 candidates ahead. Pass 1 (prefetching tags and metadata for the whole list) adds nothing on either machine and can be removed.

Setup

  • Data and index: SQ8 HNSW on DBpedia 1M x 768, k=10.
  • Query loop: 5 batches of 500 queries, one thread pinned to one core, data in /dev/shm.
  • Comparison: the same graph, built once and loaded into every build.
  • Runs: 3 interleaved runs per build.
  • Results: the returned labels matched in every run.
  • Coverage: only the top-k processCandidate scan was ported and measured. Range search was not measured.

Query time (ms per query, mean of 3 runs)

ef ARM old ARM #1070 ARM faster by x86 old x86 #1070 x86 faster by
40 0.216 0.176 18% 0.172 0.166 3%
80 0.359 0.299 17% 0.294 0.278 5%
160 0.644 0.512 20% 0.522 0.490 6%
320 1.155 0.917 21% 0.957 0.893 7%
  • Hosts: ARM is Neoverse-V2. x86 is a Xeon Platinum 8375C VM.
  • Recall@10: the same for both versions: 95.66 / 97.62 / 98.74 / 99.04%.
  • x86 noise: the x86 gain is small. At ef80 and ef160 the run means do not overlap with the old loop's: 0.278..0.280 vs 0.292..0.295 ms, and 0.487..0.492 vs 0.520..0.525 ms.

Which part gives the gain (ef160, change vs the old loop)

Variant ARM x86
This PR as is -20.4% -6.1%
Without pass 1 (no tag/metadata prefetch of the whole list) -21.3% -5.9%
Vector prefetched 1 ahead instead of 4 -21.5% +1.4% (no gain)
Only the first line of each vector prefetched -4.8% +1.8% (no gain)
No vector prefetch at all +17.8% not run
Old loop + whole-vector prefetch, no filter first not run +6.0%
Old loop + pass 1 only -2.9% (noise) not run

What the table shows:

  • Whole vector, not one line. An SQ8 vector is 784 B, about 13 cache lines. The old loop prefetches only the first line. Prefetching only the first line here loses almost all of the gain.
  • Filter first. Prefetching whole vectors without dropping visited neighbors first wastes bandwidth on vectors that are never scored. On x86 that is 4 to 8% slower than the old loop.
  • 4 ahead. On ARM, 1 ahead is as good as 4. On x86 only 4 ahead helps. Keep 4.
  • Pass 1. It changes nothing measurable on either host.

ARM counters (perf stat, gated to the timed batches, per query)

ef160 cycles instructions IPC memory-stall share of cycles L1D refills
Old loop 1747k 2736k 1.57 58% 34.8k
This PR 1593k 2939k 1.84 40% 15.6k
  • Instructions: about 7% more instructions run.
  • Waiting on memory: the share of cycles spent waiting on memory drops from 58% to 40%.
  • x86: the VM has no hardware PMU, so x86 has timing only.

Suggestion

Keep the filter-first batches and the whole-vector prefetch 4 ahead, drop pass 1, and enable it on all platforms. Measuring range search separately would also be worth doing, since this PR changes it too.

@dor-forer

Copy link
Copy Markdown
Collaborator Author

Larger M: does scanning many already-seen neighbors cost more than it saves?

Bottom line: no. At M16, M32 and M64 the scan in this PR is 17 to 23% faster than the old loop at every ef from 40 to 1280. The gain does not shrink as the share of already-seen neighbors grows; at M64 it is slightly larger. Pass 1 (prefetching the tag and metadata of every neighbor) still buys nothing, so dropping it remains the suggestion.

Setup

  • Data and index: SQ8, DBpedia 1M x 768 (999,500 base vectors plus 500 held-out queries), k=10, efConstruction 200.
  • Graphs: one graph per M, built once with the old loop and loaded by every build.
  • Hardware: ARM Neoverse-V2, one thread pinned to one core.
  • Runs: 5 batches of 500 queries, 3 interleaved runs per build.
  • Results: the returned labels were identical across all builds and runs, for every M.
  • Counting build: a separate build counted, per query, the nodes expanded, the links scanned and the links that survive the visited filter.

Results (ms per query, mean of 3 runs)

M ef links per expanded node already seen old this PR without pass 1
16 40 22.9 26% 0.229 0.188 (-17.7%) 0.189 (-17.5%)
16 160 22.9 39% 0.677 0.558 (-17.5%) 0.538 (-20.5%)
16 1280 22.9 55% 4.133 3.299 (-20.2%) 3.264 (-21.0%)
32 40 33.9 27% 0.313 0.260 (-16.8%) 0.253 (-19.2%)
32 160 34.1 41% 0.938 0.738 (-21.3%) 0.736 (-21.5%)
32 1280 34.4 59% 5.504 4.261 (-22.6%) 4.274 (-22.3%)
64 40 39.2 27% 0.355 0.280 (-21.1%) 0.283 (-20.4%)
64 160 39.5 41% 1.061 0.842 (-20.7%) 0.830 (-21.8%)
64 640 40.1 54% 3.445 2.652 (-23.0%) 2.667 (-22.6%)
64 1280 40.2 60% 6.187 4.873 (-21.2%) 4.774 (-22.8%)

The other ef values (80, 320 and 640 at every M) follow the same pattern.

What the numbers show

  • Real nodes hold far fewer links than M allows. Layer 0 has room for 2M links, so 32, 64 and 128. On this data an expanded node has about 23, 34 and 40.
  • "Mostly seen" peaks around 60%. The share of scanned links that were already visited grows with ef, from about 27% at ef40 to 55-60% at ef1280.
  • A seen neighbor costs less in this PR than in the old loop. Here a seen neighbor costs a 2-byte tag check in the filter pass. In the old loop it also costs a prefetch of the first 64 bytes of its vector, which is never used. So more seen neighbors means more wasted work for the old loop, not for this scan.
  • Pass 1 is not needed. Without it the scan is equal or slightly faster: up to 3 points at M16 and within noise at M32 and M64.

A stress run at M64 with ef 2560, 5120 and 10240, to push the seen share higher, is still running. I will add it here when it finishes.

@dor-forer

Copy link
Copy Markdown
Collaborator Author

Stress follow-up: M64 at very high ef (up to 74% of neighbors already seen)

Bottom line: even when about three quarters of the scanned neighbors were already visited, this scan is still 21 to 24% faster than the old loop. At the highest seen share, pass 1 starts to cost measurable time, which is one more reason to drop it.

The setup is the same as the comment above: the M64 graph, SQ8 1M, ARM Neoverse-V2, one core, 3 interleaved runs. The returned labels were identical across builds and runs.

ef links per expanded node already seen old (ms) this PR without pass 1
2560 40.3 65% 11.358 8.633 (-24.0%) 8.673 (-23.6%)
5120 40.3 70% 20.213 15.750 (-22.1%) 15.524 (-23.2%)
10240 40.4 74% 36.110 28.659 (-20.6%) 27.956 (-22.6%)
  • The gain eases slightly as the seen share rises: -24% at 65% seen, -20.6% at 74%. With fewer neighbors left to score, there is less vector fetching to hide.
  • The old loop is still slower. It keeps paying a vector prefetch for every seen neighbor, so the PR's scan stays ahead.
  • Pass 1 now has a cost. At ef10240, dropping it is 2 points faster, and the run means do not overlap: 27.86..28.09 ms without it vs 28.50..28.98 ms with it.

Recall is saturated at these ef values (99.12%), so this only stress-tests the scan. No real workload would run with these settings.

@codecov

codecov Bot commented Nov 16, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.54%. Comparing base (f2e097a) to head (d686f2b).
⚠️ Report is 20 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1070      +/-   ##
==========================================
+ Coverage   97.51%   97.54%   +0.02%     
==========================================
  Files         142      157      +15     
  Lines        9251    10005     +754     
==========================================
+ Hits         9021     9759     +738     
- Misses        230      246      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@GuyAv46 GuyAv46 closed this Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants