Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 9 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,13 +8,13 @@
Starting with version 8.0, RediSearch and this vector similarity library is an integral part of Redis. See https://github.com/redis/redis

This repo exposes a C API for using vector similarity search.
Allows creating indices of vectors and searching for top K similar to some vector in two methods: brute force, and by using the hnsw algorithm (probabilistic).
Allows creating vector indexes and searching for the top K similar vectors using exact brute force or approximate graph search.

The API header files are `vec_sim.h` and `query_results.h`, which are located in `src/VecSim`.

## Algorithms

All of the algorithms in this library are designed to work inside RediSearch and support the following features:
The Flat and HNSW algorithms are designed to work inside RediSearch and support the following features:
1. In place insert, delete, and update vectors in the index.
2. KNN queries - results can be ordered by score or ID.
3. Iterator interface for consecutive KNN queries.
Expand All @@ -30,6 +30,13 @@ Brute force comparison of the query vector `q` with the stored vectors. Vectors
### HNSW
Modified implementation of [hnswlib](https://github.com/nmslib/hnswlib). Modified to accommodate the above feature set.

### PiPNN

An opt-in bulk-built graph backend for FLOAT32/L2 single-value indexes. Mutations
invalidate its graph and queries use exact search until an explicit rebuild.
See [PiPNN lifecycle, supported operations, and validation](docs/pipnn.md) before
integrating it; tiered indexes and Redis Search schema integration are not included.

## Metrics
We support three popular distance metrics to measure the degree of similarity between two vectors:

Expand Down
53 changes: 53 additions & 0 deletions docs/benchmarks/pipnn/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
# Native PiPNN ARM screen — 2026-11-15

This screen measures the native backend in this change, not the upstream PiPNN
executable used in [the earlier PoC](https://github.com/RedisAI/VectorSimilarity/pull/1122).
The native implementation builds faster in these runs, but HNSW gives materially
lower query latency at comparable recall. These results do **not** support
replacing HNSW for query-heavy workloads.

## Conditions

- `dorer-arm-big`: aarch64, 16 CPUs, Neoverse-V2; GCC 12.3.0.
- CMake `RelWithDebInfo` (`-O2 -g -DNDEBUG`), normal SVS-enabled build;
both backends use the same library and distance dispatch.
- One build and one query thread; backends run serially with no competing build.
- One build per backend/dataset, fixed seed 100. This is a screening measurement,
not a repeated or statistically characterized benchmark.
- PiPNN defaults: degree 32, leaf 128, fanout 2, hash bits 12, candidate limit 80,
alpha 1.1. HNSW: M=16, efConstruction=100.
- 100,000 vectors per dataset. DEEP has 96 dimensions and 999 queries; normalized
DBpedia has 1,536 dimensions and 1,000 queries. Inputs and exact ground truth are
the prepared datasets described in the earlier PoC. DBpedia unit normalization
makes squared L2 approximately preserve cosine ranking, subject to FP32 rounding.
- Build time includes creation and insertion, plus explicit graph construction
for PiPNN. File I/O and ground-truth preparation are excluded.
- Each ef sweep warms up the first 100 queries. p95 measures the query call,
excluding result inspection and recall computation.

## Results

These are the lowest measured ef points exceeding 95% recall in the coarse sweep.
Recall is not identical between rows; no interpolation is used.

| Dataset | Backend | Build (s) | ef | Recall@10 | p95 (ms) | Index (MB) | Peak build RSS (MB) |
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: |
| DBpedia 1536D | PiPNN | 21.326 | 800 | 96.17% | 3.180 | 635.6 | 1362.4 |
| DBpedia 1536D | HNSW | 60.003 | 100 | 96.92% | 0.847 | 647.7 | 1268.4 |
| DEEP 96D | PiPNN | 2.587 | 400 | 97.61% | 0.831 | 57.5 | 210.0 |
| DEEP 96D | HNSW | 12.628 | 50 | 96.92% | 0.109 | 69.8 | 117.6 |

MB is decimal. Index memory is allocator accounting; peak RSS includes the input
matrix, process overhead, and temporary build storage. PiPNN's smaller final
index does not imply smaller peak build memory.

The complete ef sweeps, including 99% recall points where reached, are in
[the raw output](arm-native-2026-11-15.txt). PiPNN did not reach 99% on DBpedia
within this sweep (maximum ef 1600 reached 98.24%).

For reproduction, build `pipnn_comparison` and follow the commands in
[the backend documentation](../../pipnn.md). Use ef values
`50,100,200,400,800,1600` and the query counts above. Before considering deployment,
repeat the screen on the target workload and assess the complete rebuild lifecycle,
peak memory, and serialized-call integration requirement. No Redis Search schema
or tiered integration is included in this change.
28 changes: 28 additions & 0 deletions docs/benchmarks/pipnn/arm-native-2026-11-15.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
backend=pipnn n=100000 dim=1536 queries=1000 metric=L2 type=FLOAT32 seed=100 max_degree=32 leaf_size=128 fanout=2 hash_bits=12 candidate_limit=80 alpha=1.1 build_seconds=21.3256435 creation_insert_seconds=0.28626524 graph_build_seconds=21.0393783 index_bytes=635556728 build_peak_rss_bytes=1362378752
backend=pipnn ef=50 recall10=0.6911 p50_us=279.601 p95_us=414.402 serial_query_qps=3454.80207 peak_rss_bytes=1362378752
backend=pipnn ef=100 recall10=0.8023 p50_us=462.763 p95_us=628.957 serial_query_qps=2110.40427 peak_rss_bytes=1362378752
backend=pipnn ef=200 recall10=0.8804 p50_us=838.702 p95_us=1073.17 serial_query_qps=1184.183 peak_rss_bytes=1362378752
backend=pipnn ef=400 recall10=0.9295 p50_us=1479.26 p95_us=1821.179 serial_query_qps=676.595371 peak_rss_bytes=1362378752
backend=pipnn ef=800 recall10=0.9617 p50_us=2659.499 p95_us=3180.417 serial_query_qps=379.050232 peak_rss_bytes=1362378752
backend=pipnn ef=1600 recall10=0.9824 p50_us=5034.395 p95_us=5886.945 serial_query_qps=200.977983 peak_rss_bytes=1362378752
backend=hnsw n=100000 dim=1536 queries=1000 metric=L2 type=FLOAT32 seed=100 M=16 ef_construction=100 build_seconds=60.0027098 creation_insert_seconds=60.0027092 graph_build_seconds=5.73e-07 index_bytes=647694644 build_peak_rss_bytes=1268420608
backend=hnsw ef=50 recall10=0.931 p50_us=403.208 p95_us=506.731 serial_query_qps=2478.42923 peak_rss_bytes=1268420608
backend=hnsw ef=100 recall10=0.9692 p50_us=690.182 p95_us=847.021 serial_query_qps=1472.03628 peak_rss_bytes=1268420608
backend=hnsw ef=200 recall10=0.9807 p50_us=1223.222 p95_us=1500.39 serial_query_qps=835.288335 peak_rss_bytes=1268420608
backend=hnsw ef=400 recall10=0.989 p50_us=2215.058 p95_us=2693.607 serial_query_qps=464.052971 peak_rss_bytes=1268420608
backend=hnsw ef=800 recall10=0.9934 p50_us=3848.446 p95_us=4715.636 serial_query_qps=266.484313 peak_rss_bytes=1268420608
backend=hnsw ef=1600 recall10=0.9962 p50_us=6662.637 p95_us=8310.268 serial_query_qps=150.830566 peak_rss_bytes=1268420608
backend=pipnn n=100000 dim=96 queries=999 metric=L2 type=FLOAT32 seed=100 max_degree=32 leaf_size=128 fanout=2 hash_bits=12 candidate_limit=80 alpha=1.1 build_seconds=2.58716228 creation_insert_seconds=0.040176588 graph_build_seconds=2.54698569 index_bytes=57529208 build_peak_rss_bytes=209981440
backend=pipnn ef=50 recall10=0.802102102 p50_us=120.33 p95_us=159.593 serial_query_qps=8198.44787 peak_rss_bytes=209981440
backend=pipnn ef=100 recall10=0.89029029 p50_us=208.736 p95_us=263.74 serial_query_qps=4777.3922 peak_rss_bytes=209981440
backend=pipnn ef=200 recall10=0.944744745 p50_us=372.53 p95_us=459.254 serial_query_qps=2703.53 peak_rss_bytes=209981440
backend=pipnn ef=400 recall10=0.976076076 p50_us=687.242 p95_us=831.266 serial_query_qps=1478.07696 peak_rss_bytes=209981440
backend=pipnn ef=800 recall10=0.991291291 p50_us=1238.211 p95_us=1599.289 serial_query_qps=797.539677 peak_rss_bytes=209981440
backend=pipnn ef=1600 recall10=0.997097097 p50_us=2302.216 p95_us=2756.505 serial_query_qps=435.03809 peak_rss_bytes=209981440
backend=hnsw n=100000 dim=96 queries=999 metric=L2 type=FLOAT32 seed=100 M=16 ef_construction=100 build_seconds=12.6275737 creation_insert_seconds=12.6275732 graph_build_seconds=4.7e-07 index_bytes=69761400 build_peak_rss_bytes=117645312
backend=hnsw ef=50 recall10=0.969169169 p50_us=85.69 p95_us=108.746 serial_query_qps=11802.7651 peak_rss_bytes=117645312
backend=hnsw ef=100 recall10=0.98968969 p50_us=153.594 p95_us=194.854 serial_query_qps=6572.82143 peak_rss_bytes=117645312
backend=hnsw ef=200 recall10=0.997297297 p50_us=265.36 p95_us=339.757 serial_query_qps=3802.86748 peak_rss_bytes=117645312
backend=hnsw ef=400 recall10=0.9996997 p50_us=494.223 p95_us=632.8 serial_query_qps=2052.7578 peak_rss_bytes=117645312
backend=hnsw ef=800 recall10=1 p50_us=915.594 p95_us=1152.396 serial_query_qps=1114.93286 peak_rss_bytes=117645312
backend=hnsw ef=1600 recall10=1 p50_us=1673.679 p95_us=2102.27 serial_query_qps=609.500333 peak_rss_bytes=117645312
144 changes: 144 additions & 0 deletions docs/pipnn.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,144 @@
# PiPNN backend

PiPNN is an opt-in, explicitly built graph index inspired by
[the PiPNN paper](https://arxiv.org/abs/2602.21247).
It does not replace HNSW or change existing index defaults.

This implementation supports finite FLOAT32 vectors, squared L2 distance, and
one vector per label. Other data types, metrics, multi-value indexes, and tiered
indexes are rejected. It is a VectorSimilarity library backend; this change
does not add a Redis Search schema option.

## Lifecycle

```cpp
VecSimParams params{};
params.algo = VecSimAlgo_PIPNN;
params.algoParams.pipnnParams.type = VecSimType_FLOAT32;
params.algoParams.pipnnParams.dim = 128;
params.algoParams.pipnnParams.metric = VecSimMetric_L2;
VecSimIndex *index = VecSimIndex_New(&params);
// Check index != nullptr; insert 128-component vectors with VecSimIndex_AddVector.
// Check insertion return values: 1 inserted, 0 replaced, -1 failed.
if (index && VecSimIndex_BuildGraph(index)) {
// TopKQuery now searches the graph. Check replies for nullptr before use.
}
if (index) VecSimIndex_Free(index);
```

Before building, TopK queries use exact search. Successful insertions,
overwrites, deletions, updates, and relabels invalidate the graph. Queries then
use exact search until the caller explicitly invokes `VecSimIndex_BuildGraph`.
No query rebuilds the index. A failed build leaves vector storage available
for exact queries. `VecSimIndex_IsGraphReady` exposes the current state.
An allocation failure during a mutation can also invalidate the graph while
preserving stored vectors and labels; check readiness after a failed operation.

Range queries and batch iteration always use exact search. A dirty index can
therefore have substantially higher query latency than a built graph. This
lifecycle is intended for bulk ingestion followed by read-heavy operation;
frequent writes need an explicit rebuild policy at the integration layer.

Callers must serialize **all** operations on an index, including queries:
inherited search-mode bookkeeping is mutable. Do not mutate or free an index
while a batch iterator is live. There is no background builder or tiered
integration in this version.

## Construction and search

The builder partitions vectors using sampled leaders and overlapping buckets,
collects two local neighbors per point per leaf, merges candidates by residual
direction hashes, and applies distance-based pruning. Candidate reservoirs and
final graph rows have explicit size limits. A directed successor cycle reserves
one edge per vertex to preserve graph reachability, including duplicate data.

This is a native implementation, not a vendored copy of the paper's software.
It uses existing VectorSimilarity distance kernels, Gaussian projections,
overlap at the first two partition levels, bounded recursion, and deterministic
fallback leaves for non-shrinking partitions. It currently has no GEMM-based
leaf kernel or parallel construction. These choices require independent recall,
latency, and build-time evaluation; upstream PoC measurements do not establish
this backend's performance. A fixed seed reproduces construction within the same
standard-library and numerical environment; cross-platform bitwise identity is
not promised.

TopK uses bounded beam search. Set `pipnnRuntimeParams.efRuntime` in
`VecSimQueryParams`, or resolve `EF_RUNTIME` through the existing parameter API.
Higher values usually improve recall at increased query cost. Graph results
are approximate; exact fallback, range, and batch results are exhaustive subject
to timeout. Timeouts use the existing query reply code.

## Parameters

Zero selects the default for optional fields, including seed.

| Parameter | Default | Constraint |
| --- | ---: | --- |
| `maxDegree` | 32 | At least 2 |
| `leafSize` | 128 | At least 2 |
| `fanout` | 2 | 2–32 |
| `hashBits` | 12 | 1–32 |
| `candidateLimit` | 80 | At least `maxDegree` |
| `alpha` | 1.1 | Finite, at least 1 |
| `efRuntime` | 100 | Positive after default resolution |
| `seed` | 100 | Nonzero to override |
| `blockSize` | 1024 | Positive after default resolution |

Internal IDs and count are bounded by the existing 32-bit ID representation.
Parameter-dependent allocation products are checked. Memory estimates include
the final graph but not peak build scratch: candidate storage is proportional
to `count * min(candidateLimit, count - 1)`, with additional projection and
partition storage. Large fanout and leaf sizes can substantially increase work.

## Persistence

`VecSimIndex_SavePiPNN` writes a versioned little-endian file containing resolved
parameters, labels, vectors, graph readiness, and graph data. It writes a
temporary file in the destination directory, flushes it, and renames it over
the destination only after successful writing. Parent-directory fsync is not
performed, so this is not a promise of rename durability after power loss.

`VecSimIndex_LoadPiPNN` validates lengths, parameters, finite vectors, unique
labels, graph bounds, duplicate edges, connectivity edges, and a corruption
checksum. It returns null on failure. The checksum is not authentication.
Dirty indexes stay dirty after loading; built indexes retain their graph.
Files are specific to this backend and cannot be loaded as HNSW files.

## Validation

Build and run `test_pipnn` with the repository's normal CMake test configuration.
The suite covers exact fallback, graph recall, mutations, label width, range and
batch queries, timeouts, invalid inputs, duplicate vectors, allocation cleanup,
and persistence corruption. Production rollout also requires workload-specific
recall/latency/build-memory validation and integration tests in the embedding
application.

ARM validation on 2026-11-15 passed 245 selected Common, BruteForce, Allocator,
and PiPNN CTest cases with GCC 12.3 and SVS enabled. All 25 PiPNN cases also
passed Clang 21 ASan/UBSan with leak detection. Function-type sanitization was
disabled because the existing CPU-feature C/C++ dispatch triggers that check.
Clang was used after GCC 12 hit a sanitizer compiler error in an existing BF16
SVE kernel.
The SVS-enabled GCC Release library also builds with both `VECSIM_BUILD_TESTS`
and `VECSIM_BUILD_TEST_HELPERS` disabled.

The [initial native ARM screen](benchmarks/pipnn/README.md) found faster builds but
slower queries than HNSW at comparable recall on both tested datasets. It does not
establish this backend as a production HNSW replacement.

Build the optional native comparison runner with
`cmake --build <build-dir> --target pipnn_comparison -j4`, then run:

```sh
<build-dir>/benchmark/pipnn_comparison base.fbin query.fbin truth.ibin pipnn 50,100,200,400,800 1000
<build-dir>/benchmark/pipnn_comparison base.fbin query.fbin truth.ibin hnsw 50,100,200,400,800 1000
```

The matrix files have two little-endian uint32 dimensions followed by row-major
FLOAT32 vectors or uint32 ground-truth IDs. Ground truth must contain at least
10 neighbors per query. The runner uses PiPNN defaults and HNSW M=16,
efConstruction=100, seed=100. Build timing includes index creation, insertion,
and PiPNN graph construction. Peak build RSS includes the input matrix; index
bytes come from allocator accounting. Compare latency at matched recall, not
merely at the same ef value. Run each backend in a separate process on an idle
machine and repeat before making a performance decision.
3 changes: 3 additions & 0 deletions src/VecSim/CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,9 @@ endif()

add_library(VectorSimilarity ${VECSIM_LIBTYPE}
index_factories/brute_force_factory.cpp
index_factories/pipnn_factory.cpp
algorithms/pipnn/pipnn.cpp
algorithms/pipnn/pipnn_serialization.cpp
index_factories/hnsw_factory.cpp
index_factories/tiered_factory.cpp
index_factories/svs_factory.cpp
Expand Down
9 changes: 5 additions & 4 deletions src/VecSim/algorithms/brute_force/bf_batch_iterator.h
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
#include <iostream>
#include <cmath>
#include <functional>
#include <memory>

using std::pair;

Expand Down Expand Up @@ -106,7 +107,7 @@ void BF_BatchIterator<DataType, DistType>::swapScores(

template <typename DataType, typename DistType>
VecSimQueryReply *BF_BatchIterator<DataType, DistType>::heapBasedSearch(size_t n_res) {
auto rep = new VecSimQueryReply(this->allocator);
auto rep = std::make_unique<VecSimQueryReply>(this->allocator);
DistType upperBound = std::numeric_limits<DistType>::lowest();
vecsim_stl::max_priority_queue<DistType, labelType> TopCandidates(this->allocator);
// map vector's label to its index in the scores vector.
Expand Down Expand Up @@ -136,12 +137,12 @@ VecSimQueryReply *BF_BatchIterator<DataType, DistType>::heapBasedSearch(size_t n
TopCandidates.pop();
}
swapScores(TopCandidatesIndices, rep->results.size());
return rep;
return rep.release();
}

template <typename DataType, typename DistType>
VecSimQueryReply *BF_BatchIterator<DataType, DistType>::selectBasedSearch(size_t n_res) {
auto rep = new VecSimQueryReply(this->allocator);
auto rep = std::make_unique<VecSimQueryReply>(this->allocator);
size_t remaining_vectors_count = this->scores.size() - this->scores_valid_start_pos;
// Get an iterator to the effective first element in the scores array, which is the first
// element that hasn't been returned in previous iterations.
Expand All @@ -162,7 +163,7 @@ VecSimQueryReply *BF_BatchIterator<DataType, DistType>::selectBasedSearch(size_t
}
// Update the valid results start position after returning the results.
this->scores_valid_start_pos += rep->results.size();
return rep;
return rep.release();
}

template <typename DataType, typename DistType>
Expand Down
Loading
Loading