Repository navigation
Conversation
Split neighbor expansion in processCandidate and processCandidate_RangeSearch into a shared two-pass scan: prefetch the visited tags and metadata of the whole neighbor list, filter visited/in-process neighbors in batches, then compute distances while prefetching every cache line of the vector four candidates ahead. Results are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Benchmark results at 1M: ARM and x86, with an ablationBottom line: this scan is faster on both ARM and x86 and returns the same results, so it does not need an ARM-only guard. All of the gain comes from two parts working together: dropping visited neighbors first, then prefetching every line of each remaining vector 4 candidates ahead. Pass 1 (prefetching tags and metadata for the whole list) adds nothing on either machine and can be removed. Setup
Query time (ms per query, mean of 3 runs)
Which part gives the gain (ef160, change vs the old loop)
What the table shows:
ARM counters (perf stat, gated to the timed batches, per query)
SuggestionKeep the filter-first batches and the whole-vector prefetch 4 ahead, drop pass 1, and enable it on all platforms. Measuring range search separately would also be worth doing, since this PR changes it too. |
Larger M: does scanning many already-seen neighbors cost more than it saves?Bottom line: no. At M16, M32 and M64 the scan in this PR is 17 to 23% faster than the old loop at every ef from 40 to 1280. The gain does not shrink as the share of already-seen neighbors grows; at M64 it is slightly larger. Pass 1 (prefetching the tag and metadata of every neighbor) still buys nothing, so dropping it remains the suggestion. Setup
Results (ms per query, mean of 3 runs)
The other ef values (80, 320 and 640 at every M) follow the same pattern. What the numbers show
A stress run at M64 with ef 2560, 5120 and 10240, to push the seen share higher, is still running. I will add it here when it finishes. |
Stress follow-up: M64 at very high ef (up to 74% of neighbors already seen)Bottom line: even when about three quarters of the scanned neighbors were already visited, this scan is still 21 to 24% faster than the old loop. At the highest seen share, pass 1 starts to cost measurable time, which is one more reason to drop it. The setup is the same as the comment above: the M64 graph, SQ8 1M, ARM Neoverse-V2, one core, 3 interleaved runs. The returned labels were identical across builds and runs.
Recall is saturated at these ef values (99.12%), so this only stress-tests the scan. No real workload would run with these settings. |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1070 +/- ##
==========================================
+ Coverage 97.51% 97.54% +0.02%
==========================================
Files 142 157 +15
Lines 9251 10005 +754
==========================================
+ Hits 9021 9759 +738
- Misses 230 246 +16 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Summary
HNSW search on Arm is bound by memory latency, not by distance computation. This PR changes how a node's neighbor list is scanned so the cache misses overlap instead of stalling one after another. Search results are unchanged.
processCandidate(top-k search and insertion) andprocessCandidate_RangeSearchnow share one helper,forEachUnvisitedNeighbor, which scans the list in two passes:Before this change, the scan prefetched only the first 64-byte line of the next neighbor's vector and the next neighbor's tag, and never prefetched the metadata flags read by
isInProcess. Locking is unchanged: the node's links lock is still held across the whole scan.Why
Profile on Arm Neoverse-V2 (dorer-arm), standalone SQ8 HNSW index, dbpedia 1M x 768, ef_runtime=200, k=10, one core, query loop only (
perfgated to the query phase):processCandidatetook 43% of cycles and the distance kernel 50%. InsideprocessCandidate, the hottest instructions were theisInProcess(id)flag load (about 17% of total cycles; the packed 9-byteElementMetaDataarray is about 9 MB and was never prefetched) and the visited-tag check (about 14%; a 2 MB array prefetched one neighbor ahead). These shares are derived from sampled cycles: function share times instruction share within it.Results
Single-thread TopK latency, median of 3 interleaved rounds, 2,000 to 20,000 distinct queries per run, process pinned to one core. Same prebuilt index (
dbpedia-cosine-dim768-M64-efc512-sq8.hnsw_v5, identical md5 on both machines), fp32 queries.Arm Neoverse-V2 (dorer-arm)
x86 Xeon Platinum 8375C (dorer-intel)
Speedup is main latency divided by PR latency, so values below 1.00x are slower. On x86 the change is roughly neutral. At ef_runtime=100 the rounds overlap, so 0.99x is within noise. The ef_runtime=10 slowdown is small but consistent across the three rounds (0.134 to 0.136 ms vs 0.130 to 0.133 ms).
Prefetch lookahead sweep (Arm, ef_runtime=200, earlier prototype of the same scan): 1 = 0.937 ms, 2 = 0.885 ms, 4 = 0.873 ms, 8 = 0.893 ms, 16 = 0.930 ms.
A prototype that also moved the element flags into a separate dense 1-byte array was within noise of this PR (1.175 vs 1.164 ms at ef_runtime=200 on Arm), so this PR keeps
ElementMetaDataas is and prefetches its entry instead.Correctness
mainfor every configuration on both machines.make unit_teston dorer-arm: 100% tests passed, 0 tests failed out of 3140.Not covered
🤖 Generated with Claude Code
Note
Medium Risk
Refactors the hottest HNSW graph-scan path used by TopK and range search; behavior should be identical but any mistake in visit filtering or ordering could affect query results.
Overview
HNSW neighbor expansion now goes through a shared
forEachUnvisitedNeighborhelper instead of duplicated loops inprocessCandidateandprocessCandidate_RangeSearch.The helper keeps the same visit order and locking (caller still holds the node links lock). It first prefetches every neighbor’s visited tag and
ElementMetaData(forisInProcess), then processes links in batches of 64: skip visited/in-process nodes, mark the rest visited, prefetch all cache lines of each candidate vector with a 4-neighbor lookahead, and invoke the caller’s lambda for distance/heaps work.This replaces the old one-ahead prefetch of a single vector cache line and drops the special-cased “last neighbor” loop. Search semantics are intended to be unchanged; the goal is lower memory latency on Arm (and roughly neutral on x86).
Reviewed by Cursor Bugbot for commit d686f2b. Bugbot is set up for automated code reviews on this repo. Configure here.