Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions poc/pipnn-arm/BASELINE_BUILD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
# VecSim ARM baseline build

The baseline was built on `dorer-arm-big` in `/home/ubuntu/pipnn-poc-20261114`, at detached commit `54a841ae841828d53a482a826d6ace7753bc5450`. The commands below are run on that host unless noted otherwise. The build used GCC 12.3.0, CMake `Release`, tests and test helpers disabled, and SVS disabled. CMake selected the ARM SIMD configurations supported by the host.

Create the worktree from the existing local clone and initialize the pinned SVS submodule from local objects:

```sh
mkdir -p /home/ubuntu/pipnn-poc-20261114
git -C /home/ubuntu/wt/arm-pr6 worktree add --detach /home/ubuntu/pipnn-poc-20261114/vecsim 54a841ae841828d53a482a826d6ace7753bc5450
git -C /home/ubuntu/pipnn-poc-20261114/vecsim submodule update --init --no-fetch deps/ScalableVectorSearch
```

The pinned revision's SVS-disabled build requires the minimal guard patch saved at `/home/ubuntu/pipnn-poc-20261114/vecsim-svs-off.patch`. It guards SVS includes and shared-pool calls; it does not modify HNSW or SIMD code. Apply it to a fresh worktree before building:

```sh
git -C /home/ubuntu/pipnn-poc-20261114/vecsim apply /home/ubuntu/pipnn-poc-20261114/vecsim-svs-off.patch
```

Configure and build the library, with at most four jobs:

```sh
cmake -S /home/ubuntu/pipnn-poc-20261114/vecsim -B /home/ubuntu/pipnn-poc-20261114/vecsim-build -DCMAKE_BUILD_TYPE=Release -DVECSIM_BUILD_TESTS=OFF -DVECSIM_BUILD_TEST_HELPERS=OFF -DUSE_SVS=OFF
cmake --build /home/ubuntu/pipnn-poc-20261114/vecsim-build --target VectorSimilarity -j4 > /home/ubuntu/pipnn-poc-20261114/vecsim-build.log 2>&1
```

The host approval marker was supplied as a leading prefix to the local SSH command: `VECSIM_BUILD_MACHINE_CONFIRMED=1 ssh dorer-arm-big '…'`. The source is `vecsim_bench.cpp` in this directory and was copied to the remote root before compilation. Link the driver against the exact library dependencies:

```sh
cd /home/ubuntu/pipnn-poc-20261114
g++ -O3 -std=c++20 -Ivecsim/src vecsim_bench.cpp -Wl,--start-group vecsim-build/src/VecSim/libVectorSimilarity.a vecsim-build/src/VecSim/spaces/libVectorSimilaritySpaces.a vecsim-build/src/VecSim/spaces/libVectorSimilaritySpaces_no_optimization.a vecsim-build/_deps/cpu_features-build/libcpu_features.a -Wl,--end-group -lpthread -ldl -o vecsim_bench
```

Copy `smoke_data.py` from this directory to the remote root, then generate the small brute-force reference and run the smoke test:

```sh
cd /home/ubuntu/pipnn-poc-20261114
python3 smoke_data.py smoke
./vecsim_bench smoke/base.fbin smoke/query.fbin smoke/truth.ibin 16 200 1 200
```

The smoke run used 200 base vectors and 12 queries, with dimension 16, and reported recall@10 of 1.0. Its timings are not benchmark results. The C API uses the HNSW constructor's fixed seed of 100; the driver's `repeat` argument is a run identifier, not a seed.
33 changes: 33 additions & 0 deletions poc/pipnn-arm/PHASE2_PLAN.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Phase 2: DBpedia100k pruning screen

Continue on the already selected dorer-arm-big, using unchanged binaries and
prepared input. Preserve the first-phase evidence. This is exploratory tuning
on the existing queries, not validation on unseen queries.

1. Verify host idleness, free disk, binary/data identities and effective flags.
2. Screen PiPNN alpha 1.0, 1.1, 1.4 with one tree, retaining alpha 1.2 as reference.
Also test alpha 1.2 with two trees to distinguish pruning from candidate coverage.
3. Test HNSW M16 efConstruction=100 twice alongside existing efConstruction=200.
4. At measured recall >=95% and >=99%, select the PiPNN setting with the lowest
p95 among settings with construction time no greater than half the fastest
feasible HNSW baseline and resident memory <=125% of it. If none qualifies,
report that and repeat the best latency candidate within the HNSW construction
budget. Repeat each selected new configuration once; retain both targets.
5. Compare repeat medians/ranges and memory, including peak build RSS. Obtain
separate review against raw evidence and document the decision.

Plan review found no execution blocker. Construction boundaries and compiler
arithmetic flags differ, so the 2x ratio is only a screening proxy, not an
end-to-end or algorithm-isolated claim. Fixed randomness does not test graph-seed
variation. Search effort uses the existing finite sweep with no interpolation.
Parameter tuning is the only change in this phase; aligned compiler settings,
common ready-to-query instrumentation and larger-scale runs remain separate work.

## Selection adjustment after screening

The first HNSW efConstruction=100 run takes 59.6 seconds, below every PiPNN
candidate (minimum 60.0 seconds). Therefore no candidate meets even the fallback
construction budget. Retain that failure explicitly. Repeat alpha=1.1 anyway
to measure the potentially useful near-equal-build, high-recall tradeoff; do not
present it as passing the original selection gate. Alpha=1.2 remains the best
95%-recall PiPNN point and already has two runs. No further parameter expansion.
89 changes: 89 additions & 0 deletions poc/pipnn-arm/PHASE2_RESULTS.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# DBpedia100k parameter screen on ARM

Decision: retain PiPNN alpha=1.1 with one tree as a high-recall research candidate.
Do not replace HNSW. None of the screened PiPNN settings meets the 2x construction
screening target against the cheaper HNSW configuration. Pruning tuning did not
remove the 95%-recall latency disadvantage against HNSW efConstruction=200.

This phase changes only parameters on the same ARM host, binaries, prepared
100k x 1536 FP32 inputs and 1000 queries. It adds seven complete runs: four new
PiPNN configurations, a second run of the selected alpha=1.1 configuration, and
two HNSW M16/efConstruction=100 runs. The existing evidence remains unchanged in
`evidence/`; the combined snapshot is in `phase2-evidence/`.

## Measured tradeoff

| Configuration | Runs | Build s | p95 >=95% recall (ms) | p95 >=99% recall (ms) |
|---|---:|---:|---:|---:|
| hnsw M16-efC100 | 2 | 59.41 | 0.715 | 4.566 |
| hnsw M16-efC200 | 2 | 119.23 | 0.633 | 2.233 |
| hnsw M32-efC200 | 2 | 147.27 | 0.794 | 2.235 |
| pipnn alpha1.0-trees1 | 1 | 60.04 | 0.820 | 2.424 |
| pipnn alpha1.1-trees1 | 2 | 62.58 | 0.768 | 1.850 |
| pipnn alpha1.2-trees1 | 2 | 65.08 | 0.765 | 2.078 |
| pipnn alpha1.2-trees2 | 1 | 126.73 | 0.822 | 1.854 |
| pipnn alpha1.4-trees1 | 1 | 66.53 | 0.817 | 2.264 |

Construction seconds are an imperfect proxy: PiPNN includes graph serialization
but excludes loading that graph for queries; HNSW is immediately queryable after
its reported construction. These are not common end-to-end readiness times.
PiPNN also retains native/fast-math build flags unlike the VecSim core. Results
compare these builds, not algorithms under identical compiler arithmetic policy.

Query values are the median of per-run p95 at a measured search effort meeting
the recall threshold in every available repeat. A single-run screen is labeled
explicitly. The sweep is finite: notably, HNSW efConstruction=100 reaches only
98.9% recall at efRuntime=400, then 99.34% at 800. Untested efforts in between
could reduce its reported >=99% latency. Do not treat these sampled points as
the exact optimal frontier. Recall also differs slightly among qualifying points.

## What changed our understanding

- Reducing HNSW efConstruction from 200 to 100 roughly halves construction time.
That removes the general PiPNN build-speed advantage on this dataset, but costs
substantial query latency at the tested 99%-recall point.
- Stronger PiPNN pruning (alpha=1.0) saves a few build seconds but requires wider
searches. Less aggressive pruning (alpha=1.4) adds edges without improving the
measured latency. Alpha=1.1 gives a better measured 99%-recall tradeoff.
- Two trees improve candidate coverage and attain 99% recall with a smaller beam.
However, their approximately 127-second build is about twice the single-tree
cost, and the selected alpha=1.1 single-tree candidate has similar query latency.
- Reported query-process residency is about
649 MB for PiPNN versus 653–654 MB for HNSW M16. Peak construction RSS is about
1232 MB versus 1268–1269 MB. These include runtime/input-buffer overhead.
Every PiPNN setting reports exactly the same RSS values despite differing
graph file sizes. This phase did not isolate the reason for that equality;
do not use these snapshots to rank memory efficiency among pruning settings.

Selected alpha=1.1 repeats built in 62.45–62.71 seconds. At beam 200, both reached recall 0.9901; p95 ranged 1.839–1.861 ms. Compared with HNSW efConstruction=100, its median construction proxy is 5.3% longer, its >=95% p95 is 7.4% higher, and its >=99% p95 is 59.5% lower at the tested efforts. Compared with HNSW efConstruction=200, its construction proxy is 47.5% lower and its >=99% p95 is 17.1% lower, but its >=95% p95 is 21.3% higher.

## Verification and next step

The runner refused overwrites and competing known benchmark/build processes.
The host had about 20 GiB free and was idle before the screen. Source inspection
confirmed alpha and replica forwarding to the builder; output reports the tree
count and graph degrees. Both drivers check result uniqueness and compute recall
against the existing exhaustive reference. The PiPNN driver also checks returned
ID bounds; the VecSim driver checks ground-truth ID bounds. Fixed/default graph randomness
and two repeats establish timing repeatability only, not seed robustness.

Parameter selection reused the evaluation queries, so there is no unseen-query
validation claim. The plan's fallback construction budget also failed after the
cheaper HNSW result; this is recorded in [the plan](PHASE2_PLAN.md). The selected
PiPNN setting was repeated to quantify the near-equal-build high-recall tradeoff,
not to relabel that gate as passed.

Recommend the next validation target be >=99% recall: use untouched queries,
denser search-effort sweeps near the threshold, aligned arithmetic flags and a
common ready-to-query boundary before moving to million-vector/multithreaded
measurements. Online updates/deletes and production integration remain untested.
No production HNSW source was changed.

Reproduce the combined summary with:

```sh
python3 poc/pipnn-arm/summarize.py --evidence poc/pipnn-arm/phase2-evidence
```

Exact commands, return codes, timings and query observations are saved in
`phase2-evidence/results/`; executable/input hashes are in its `metadata.json`.
40 changes: 40 additions & 0 deletions poc/pipnn-arm/PIPNN_BUILD.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
# PiPNN ARM build

This records the successful proof-of-concept build on `dorer-arm-big`, using GCC 12.3.0 and CMake Release (`-O3 -DNDEBUG -march=native -ffast-math`). The upstream revision and resolved submodule and FetchContent revisions are in [metadata.txt](metadata.txt). FetchContent uses moving branches for several dependencies; a fresh configure may therefore resolve different revisions unless the recorded revisions are supplied separately.

## Fresh checkout and build

Run on the ARM host, with the patch and `pipnn_query.cpp` copied from this directory. Replace `ARTIFACT_DIR` and `WORK_DIR` with their absolute paths. The top-level submodules are initialized without `--recursive`: the MIRAGE checkout has a stale nested submodule entry that makes recursive initialization fail.

```sh
git clone https://github.com/ParAlg/PiPNN.git "$WORK_DIR"
cd "$WORK_DIR"
git checkout 443a328dc71a3b7be753d52ca2b3c9a4ba77f3fd
git submodule update --init
git apply "$ARTIFACT_DIR/arm-port.patch"
cp "$ARTIFACT_DIR/pipnn_query.cpp" algorithms/PipNN/pipnn_query.cpp
cmake -S . -B build -DCMAKE_BUILD_TYPE=Release
cmake --build build --target neighbors-pipnn_FLOAT_T_EUCLIDEAN -j4
cmake --build build --target pipnn_query -j4
```

The patch combines three changes: ARM-compatible headers and aligned allocation; a PiPNN-only CMake configuration; and a CMake target linking the separate query driver to Parlay, Eigen, Abseil and Highway. The build restriction is not an upstream requirement or part of the algorithm: it skips unrelated targets, including MIRAGE, whose configuration otherwise failed on this host because BLAS was unavailable. The query source is a separate file and must be copied as shown. Eigen and Highway remain in the PiPNN build.

Configuration fetches dependencies from the network, including Eigen, Parlay, Abseil, Highway and Google Benchmark. The top-level submodule checkout of Parlay is distinct from the version actually fetched by CMake; use the FetchContent revisions in the metadata to identify the tested build.

The resulting executables are `build/algorithms/PipNN/neighbors-pipnn_FLOAT_T_EUCLIDEAN` and `build/algorithms/PipNN/pipnn_query`. Set `PARLAY_NUM_THREADS` at runtime. The query driver accepts a graph file, base fbin, query fbin, ground-truth file, k, and one or more beam sizes:

```sh
PARLAY_NUM_THREADS=1 build/algorithms/PipNN/pipnn_query \
GRAPH BASE.fbin QUERY.fbin TRUTH.bin 10 10 32 64 100
```

Ground truth contains two 32-bit integers (`nq`, `k`), then `nq*k` row-major 32-bit IDs, then `nq*k` row-major float32 distances. The driver warms the first 100 queries per beam, measures each query separately, and checks result count, uniqueness and bounds before reporting recall and latency percentiles. Its index-process RSS snapshot is taken after loading the graph and base vectors and before loading queries and ground truth; file sizes are reported separately.

## Checks performed

- ARM scalar L2, inner-product and norm fallbacks matched a double reference for dimensions 1 through 100 using the recorded release flags. The test is in `distance_check.cpp`; its output is in `distance_check.log`.
- The builder produced a graph for 512 random 8-dimensional vectors using EigenKNN and two workers.
- The query driver returned recall@10 of 1.0 on 20 independent queries against a 128-point, 13-dimensional fully connected graph, with beam 128 and independently computed brute-force ground truth. Its uniqueness and bounds checks passed. This check did not independently verify result ordering.

The remote proof-of-concept directory contains the configure, build and smoke logs. These checks establish basic portability and query correctness for the cases tested, not benchmark parity with the paper.
25 changes: 25 additions & 0 deletions poc/pipnn-arm/PIPNN_LICENSE.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
Upstream: https://github.com/ParAlg/PiPNN
Revision: 443a328dc71a3b7be753d52ca2b3c9a4ba77f3fd
Applies to upstream code represented in arm-port.patch.

MIT License

Copyright (c) 2023 magdalendobson

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
81 changes: 81 additions & 0 deletions poc/pipnn-arm/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
# PiPNN ARM feasibility experiment

This directory is an experimental benchmark bundle, not a VectorSimilarity
backend or a production HNSW change. Start with [phase 2 findings](PHASE2_RESULTS.md)
for the current conclusion, then [initial findings](RESULTS.md) for both datasets.
Build instructions are in [BASELINE_BUILD.md](BASELINE_BUILD.md) and
[PIPNN_BUILD.md](PIPNN_BUILD.md). Builds use pinned revisions, not this PR's base.

The runner and data preparation scripts preserve the original ARM host paths.
To rerun elsewhere, adapt those paths and provide the datasets referenced in
`prepare_data.py`; vectors, binaries and graph files are not included. The JSON
summary can be regenerated locally from the included text logs without datasets.
`evidence/` preserves phase 1; `phase2-evidence/` is the combined snapshot, so
some raw logs deliberately appear in both. See [upstream attribution](PIPNN_LICENSE.txt)
for the PiPNN patch and its source.

Status: ARM build and correctness smoke checks passed. Initial 100k-vector
screening is complete; see [results and limitations](RESULTS.md).
The follow-up [1536D parameter screen](PHASE2_RESULTS.md) identifies a useful
high-recall PiPNN setting and a cheaper HNSW baseline.
This experiment evaluates a candidate replacement for VectorSimilarity HNSW.
Static build/search results cannot establish support for online updates/deletes.

## Host and isolation

- User-selected host: `dorer-arm-big`, 16 Neoverse-V2 cores, 123 GiB RAM.
- Remote experiment: `/home/ubuntu/pipnn-poc-20261114`.
- PiPNN upstream: `443a328dc71a3b7be753d52ca2b3c9a4ba77f3fd`.
- VectorSimilarity: clean detached worktree at `54a841ae841828d53a482a826d6ace7753bc5450`.
- Main working checkout and its existing modifications are untouched.
- Baseline uses FP32 L2, single-value HNSW, Release build, fixed C API seed 100.

Before setup, root disk had 4.6 GiB available. Cleanup removed only ELF binaries
and static archives from these unused build trees, following process-reference
checks, retaining repositories, sources, datasets, logs, and reports:

- `/home/ubuntu/wt/hnsw-search-prefetch/bin`
- `/home/ubuntu/wt/mod-18279/bin`
- `/home/ubuntu/sq4-gate-1103/bin`

Available disk increased to 21 GiB. Logical removed file sizes total 18,708,642,004
bytes; this differs from physical disk reclaimed because of sparse files and
filesystem accounting. New experiment setup subsequently consumes some space.

## Inputs and measurement boundaries

`prepare_data.py` prepares 100k DEEP vectors (96 dimensions) and 100k DBpedia
embeddings (1536 dimensions) from existing host datasets. DBpedia is normalized
to FP32 unit vectors and searched with squared L2, approximately preserving cosine
ranking (rounding can affect close ties). Ground truth uses these FP32 L2 inputs.
Raw and normalized exact-content overlaps are excluded. Near-identical queries
and ambiguous top-10 distance ties are excluded and counted in each manifest.
DEEP has 999 evaluation queries; DBpedia has 1000. Ground truth is recomputed
against the actual subset using an exhaustive float64 scan and direct-distance
reranking of shortlisted candidates.

Initial screening uses one build thread and one query thread for both algorithms.
Inputs are prepared FP32 fbin files; common offline data preparation and ground
truth generation are excluded from measured build costs. Build timing, file
loading, and graph serialization must be reported with their respective scope.
Query timing excludes recall evaluation; reported QPS is inverse mean serial
query time, not concurrent batch throughput. Both drivers warm the first 100
queries before each search-effort sweep.

HNSW reports tracked index allocation and resident process memory after releasing
its extra raw input copy, before loading queries/truth. PiPNN reports resident
process memory after loading graph/vectors, before loading queries/truth. These
RSS values include runtime overhead; graph file size is not resident index size.

Compare actual operating points attaining at least 95% and 99% recall@10, never
interpolate unattained targets. Initial proposed gate per dataset/target: at least
2x faster construction, at most 10% p95 query-latency regression and at most 25%
resident-index-process-memory increase. Report measured outcomes separately and
do not call a static winner a production-ready HNSW replacement. Repeat promising
configurations; parameter choices and random graph variation must be disclosed.

## Sources

- Paper: https://arxiv.org/abs/2602.21247
- Authors' implementation: https://github.com/ParAlg/PiPNN
- Follow-up research (not implemented here): https://arxiv.org/abs/2611.05690
Loading
Loading