diff --git a/README.md b/README.md index 650cbfaa4..b4c11829f 100644 --- a/README.md +++ b/README.md @@ -8,13 +8,13 @@ Starting with version 8.0, RediSearch and this vector similarity library is an integral part of Redis. See https://github.com/redis/redis This repo exposes a C API for using vector similarity search. -Allows creating indices of vectors and searching for top K similar to some vector in two methods: brute force, and by using the hnsw algorithm (probabilistic). +Allows creating vector indexes and searching for the top K similar vectors using exact brute force or approximate graph search. The API header files are `vec_sim.h` and `query_results.h`, which are located in `src/VecSim`. ## Algorithms -All of the algorithms in this library are designed to work inside RediSearch and support the following features: +The Flat and HNSW algorithms are designed to work inside RediSearch and support the following features: 1. In place insert, delete, and update vectors in the index. 2. KNN queries - results can be ordered by score or ID. 3. Iterator interface for consecutive KNN queries. @@ -30,6 +30,13 @@ Brute force comparison of the query vector `q` with the stored vectors. Vectors ### HNSW Modified implementation of [hnswlib](https://github.com/nmslib/hnswlib). Modified to accommodate the above feature set. +### PiPNN + +An opt-in bulk-built graph backend for FLOAT32/L2 single-value indexes. Mutations +invalidate its graph and queries use exact search until an explicit rebuild. +See [PiPNN lifecycle, supported operations, and validation](docs/pipnn.md) before +integrating it; tiered indexes and Redis Search schema integration are not included. + ## Metrics We support three popular distance metrics to measure the degree of similarity between two vectors: diff --git a/docs/benchmarks/pipnn/README.md b/docs/benchmarks/pipnn/README.md new file mode 100644 index 000000000..6d9b25ee1 --- /dev/null +++ b/docs/benchmarks/pipnn/README.md @@ -0,0 +1,53 @@ +# Native PiPNN ARM screen — 2026-11-15 + +This screen measures the native backend in this change, not the upstream PiPNN +executable used in [the earlier PoC](https://github.com/RedisAI/VectorSimilarity/pull/1122). +The native implementation builds faster in these runs, but HNSW gives materially +lower query latency at comparable recall. These results do **not** support +replacing HNSW for query-heavy workloads. + +## Conditions + +- `dorer-arm-big`: aarch64, 16 CPUs, Neoverse-V2; GCC 12.3.0. +- CMake `RelWithDebInfo` (`-O2 -g -DNDEBUG`), normal SVS-enabled build; + both backends use the same library and distance dispatch. +- One build and one query thread; backends run serially with no competing build. +- One build per backend/dataset, fixed seed 100. This is a screening measurement, + not a repeated or statistically characterized benchmark. +- PiPNN defaults: degree 32, leaf 128, fanout 2, hash bits 12, candidate limit 80, + alpha 1.1. HNSW: M=16, efConstruction=100. +- 100,000 vectors per dataset. DEEP has 96 dimensions and 999 queries; normalized + DBpedia has 1,536 dimensions and 1,000 queries. Inputs and exact ground truth are + the prepared datasets described in the earlier PoC. DBpedia unit normalization + makes squared L2 approximately preserve cosine ranking, subject to FP32 rounding. +- Build time includes creation and insertion, plus explicit graph construction + for PiPNN. File I/O and ground-truth preparation are excluded. +- Each ef sweep warms up the first 100 queries. p95 measures the query call, + excluding result inspection and recall computation. + +## Results + +These are the lowest measured ef points exceeding 95% recall in the coarse sweep. +Recall is not identical between rows; no interpolation is used. + +| Dataset | Backend | Build (s) | ef | Recall@10 | p95 (ms) | Index (MB) | Peak build RSS (MB) | +| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | +| DBpedia 1536D | PiPNN | 21.326 | 800 | 96.17% | 3.180 | 635.6 | 1362.4 | +| DBpedia 1536D | HNSW | 60.003 | 100 | 96.92% | 0.847 | 647.7 | 1268.4 | +| DEEP 96D | PiPNN | 2.587 | 400 | 97.61% | 0.831 | 57.5 | 210.0 | +| DEEP 96D | HNSW | 12.628 | 50 | 96.92% | 0.109 | 69.8 | 117.6 | + +MB is decimal. Index memory is allocator accounting; peak RSS includes the input +matrix, process overhead, and temporary build storage. PiPNN's smaller final +index does not imply smaller peak build memory. + +The complete ef sweeps, including 99% recall points where reached, are in +[the raw output](arm-native-2026-11-15.txt). PiPNN did not reach 99% on DBpedia +within this sweep (maximum ef 1600 reached 98.24%). + +For reproduction, build `pipnn_comparison` and follow the commands in +[the backend documentation](../../pipnn.md). Use ef values +`50,100,200,400,800,1600` and the query counts above. Before considering deployment, +repeat the screen on the target workload and assess the complete rebuild lifecycle, +peak memory, and serialized-call integration requirement. No Redis Search schema +or tiered integration is included in this change. diff --git a/docs/benchmarks/pipnn/arm-native-2026-11-15.txt b/docs/benchmarks/pipnn/arm-native-2026-11-15.txt new file mode 100644 index 000000000..f26b43b58 --- /dev/null +++ b/docs/benchmarks/pipnn/arm-native-2026-11-15.txt @@ -0,0 +1,28 @@ +backend=pipnn n=100000 dim=1536 queries=1000 metric=L2 type=FLOAT32 seed=100 max_degree=32 leaf_size=128 fanout=2 hash_bits=12 candidate_limit=80 alpha=1.1 build_seconds=21.3256435 creation_insert_seconds=0.28626524 graph_build_seconds=21.0393783 index_bytes=635556728 build_peak_rss_bytes=1362378752 +backend=pipnn ef=50 recall10=0.6911 p50_us=279.601 p95_us=414.402 serial_query_qps=3454.80207 peak_rss_bytes=1362378752 +backend=pipnn ef=100 recall10=0.8023 p50_us=462.763 p95_us=628.957 serial_query_qps=2110.40427 peak_rss_bytes=1362378752 +backend=pipnn ef=200 recall10=0.8804 p50_us=838.702 p95_us=1073.17 serial_query_qps=1184.183 peak_rss_bytes=1362378752 +backend=pipnn ef=400 recall10=0.9295 p50_us=1479.26 p95_us=1821.179 serial_query_qps=676.595371 peak_rss_bytes=1362378752 +backend=pipnn ef=800 recall10=0.9617 p50_us=2659.499 p95_us=3180.417 serial_query_qps=379.050232 peak_rss_bytes=1362378752 +backend=pipnn ef=1600 recall10=0.9824 p50_us=5034.395 p95_us=5886.945 serial_query_qps=200.977983 peak_rss_bytes=1362378752 +backend=hnsw n=100000 dim=1536 queries=1000 metric=L2 type=FLOAT32 seed=100 M=16 ef_construction=100 build_seconds=60.0027098 creation_insert_seconds=60.0027092 graph_build_seconds=5.73e-07 index_bytes=647694644 build_peak_rss_bytes=1268420608 +backend=hnsw ef=50 recall10=0.931 p50_us=403.208 p95_us=506.731 serial_query_qps=2478.42923 peak_rss_bytes=1268420608 +backend=hnsw ef=100 recall10=0.9692 p50_us=690.182 p95_us=847.021 serial_query_qps=1472.03628 peak_rss_bytes=1268420608 +backend=hnsw ef=200 recall10=0.9807 p50_us=1223.222 p95_us=1500.39 serial_query_qps=835.288335 peak_rss_bytes=1268420608 +backend=hnsw ef=400 recall10=0.989 p50_us=2215.058 p95_us=2693.607 serial_query_qps=464.052971 peak_rss_bytes=1268420608 +backend=hnsw ef=800 recall10=0.9934 p50_us=3848.446 p95_us=4715.636 serial_query_qps=266.484313 peak_rss_bytes=1268420608 +backend=hnsw ef=1600 recall10=0.9962 p50_us=6662.637 p95_us=8310.268 serial_query_qps=150.830566 peak_rss_bytes=1268420608 +backend=pipnn n=100000 dim=96 queries=999 metric=L2 type=FLOAT32 seed=100 max_degree=32 leaf_size=128 fanout=2 hash_bits=12 candidate_limit=80 alpha=1.1 build_seconds=2.58716228 creation_insert_seconds=0.040176588 graph_build_seconds=2.54698569 index_bytes=57529208 build_peak_rss_bytes=209981440 +backend=pipnn ef=50 recall10=0.802102102 p50_us=120.33 p95_us=159.593 serial_query_qps=8198.44787 peak_rss_bytes=209981440 +backend=pipnn ef=100 recall10=0.89029029 p50_us=208.736 p95_us=263.74 serial_query_qps=4777.3922 peak_rss_bytes=209981440 +backend=pipnn ef=200 recall10=0.944744745 p50_us=372.53 p95_us=459.254 serial_query_qps=2703.53 peak_rss_bytes=209981440 +backend=pipnn ef=400 recall10=0.976076076 p50_us=687.242 p95_us=831.266 serial_query_qps=1478.07696 peak_rss_bytes=209981440 +backend=pipnn ef=800 recall10=0.991291291 p50_us=1238.211 p95_us=1599.289 serial_query_qps=797.539677 peak_rss_bytes=209981440 +backend=pipnn ef=1600 recall10=0.997097097 p50_us=2302.216 p95_us=2756.505 serial_query_qps=435.03809 peak_rss_bytes=209981440 +backend=hnsw n=100000 dim=96 queries=999 metric=L2 type=FLOAT32 seed=100 M=16 ef_construction=100 build_seconds=12.6275737 creation_insert_seconds=12.6275732 graph_build_seconds=4.7e-07 index_bytes=69761400 build_peak_rss_bytes=117645312 +backend=hnsw ef=50 recall10=0.969169169 p50_us=85.69 p95_us=108.746 serial_query_qps=11802.7651 peak_rss_bytes=117645312 +backend=hnsw ef=100 recall10=0.98968969 p50_us=153.594 p95_us=194.854 serial_query_qps=6572.82143 peak_rss_bytes=117645312 +backend=hnsw ef=200 recall10=0.997297297 p50_us=265.36 p95_us=339.757 serial_query_qps=3802.86748 peak_rss_bytes=117645312 +backend=hnsw ef=400 recall10=0.9996997 p50_us=494.223 p95_us=632.8 serial_query_qps=2052.7578 peak_rss_bytes=117645312 +backend=hnsw ef=800 recall10=1 p50_us=915.594 p95_us=1152.396 serial_query_qps=1114.93286 peak_rss_bytes=117645312 +backend=hnsw ef=1600 recall10=1 p50_us=1673.679 p95_us=2102.27 serial_query_qps=609.500333 peak_rss_bytes=117645312 diff --git a/docs/pipnn.md b/docs/pipnn.md new file mode 100644 index 000000000..2d468db64 --- /dev/null +++ b/docs/pipnn.md @@ -0,0 +1,144 @@ +# PiPNN backend + +PiPNN is an opt-in, explicitly built graph index inspired by +[the PiPNN paper](https://arxiv.org/abs/2602.21247). +It does not replace HNSW or change existing index defaults. + +This implementation supports finite FLOAT32 vectors, squared L2 distance, and +one vector per label. Other data types, metrics, multi-value indexes, and tiered +indexes are rejected. It is a VectorSimilarity library backend; this change +does not add a Redis Search schema option. + +## Lifecycle + +```cpp +VecSimParams params{}; +params.algo = VecSimAlgo_PIPNN; +params.algoParams.pipnnParams.type = VecSimType_FLOAT32; +params.algoParams.pipnnParams.dim = 128; +params.algoParams.pipnnParams.metric = VecSimMetric_L2; +VecSimIndex *index = VecSimIndex_New(¶ms); +// Check index != nullptr; insert 128-component vectors with VecSimIndex_AddVector. +// Check insertion return values: 1 inserted, 0 replaced, -1 failed. +if (index && VecSimIndex_BuildGraph(index)) { + // TopKQuery now searches the graph. Check replies for nullptr before use. +} +if (index) VecSimIndex_Free(index); +``` + +Before building, TopK queries use exact search. Successful insertions, +overwrites, deletions, updates, and relabels invalidate the graph. Queries then +use exact search until the caller explicitly invokes `VecSimIndex_BuildGraph`. +No query rebuilds the index. A failed build leaves vector storage available +for exact queries. `VecSimIndex_IsGraphReady` exposes the current state. +An allocation failure during a mutation can also invalidate the graph while +preserving stored vectors and labels; check readiness after a failed operation. + +Range queries and batch iteration always use exact search. A dirty index can +therefore have substantially higher query latency than a built graph. This +lifecycle is intended for bulk ingestion followed by read-heavy operation; +frequent writes need an explicit rebuild policy at the integration layer. + +Callers must serialize **all** operations on an index, including queries: +inherited search-mode bookkeeping is mutable. Do not mutate or free an index +while a batch iterator is live. There is no background builder or tiered +integration in this version. + +## Construction and search + +The builder partitions vectors using sampled leaders and overlapping buckets, +collects two local neighbors per point per leaf, merges candidates by residual +direction hashes, and applies distance-based pruning. Candidate reservoirs and +final graph rows have explicit size limits. A directed successor cycle reserves +one edge per vertex to preserve graph reachability, including duplicate data. + +This is a native implementation, not a vendored copy of the paper's software. +It uses existing VectorSimilarity distance kernels, Gaussian projections, +overlap at the first two partition levels, bounded recursion, and deterministic +fallback leaves for non-shrinking partitions. It currently has no GEMM-based +leaf kernel or parallel construction. These choices require independent recall, +latency, and build-time evaluation; upstream PoC measurements do not establish +this backend's performance. A fixed seed reproduces construction within the same +standard-library and numerical environment; cross-platform bitwise identity is +not promised. + +TopK uses bounded beam search. Set `pipnnRuntimeParams.efRuntime` in +`VecSimQueryParams`, or resolve `EF_RUNTIME` through the existing parameter API. +Higher values usually improve recall at increased query cost. Graph results +are approximate; exact fallback, range, and batch results are exhaustive subject +to timeout. Timeouts use the existing query reply code. + +## Parameters + +Zero selects the default for optional fields, including seed. + +| Parameter | Default | Constraint | +| --- | ---: | --- | +| `maxDegree` | 32 | At least 2 | +| `leafSize` | 128 | At least 2 | +| `fanout` | 2 | 2–32 | +| `hashBits` | 12 | 1–32 | +| `candidateLimit` | 80 | At least `maxDegree` | +| `alpha` | 1.1 | Finite, at least 1 | +| `efRuntime` | 100 | Positive after default resolution | +| `seed` | 100 | Nonzero to override | +| `blockSize` | 1024 | Positive after default resolution | + +Internal IDs and count are bounded by the existing 32-bit ID representation. +Parameter-dependent allocation products are checked. Memory estimates include +the final graph but not peak build scratch: candidate storage is proportional +to `count * min(candidateLimit, count - 1)`, with additional projection and +partition storage. Large fanout and leaf sizes can substantially increase work. + +## Persistence + +`VecSimIndex_SavePiPNN` writes a versioned little-endian file containing resolved +parameters, labels, vectors, graph readiness, and graph data. It writes a +temporary file in the destination directory, flushes it, and renames it over +the destination only after successful writing. Parent-directory fsync is not +performed, so this is not a promise of rename durability after power loss. + +`VecSimIndex_LoadPiPNN` validates lengths, parameters, finite vectors, unique +labels, graph bounds, duplicate edges, connectivity edges, and a corruption +checksum. It returns null on failure. The checksum is not authentication. +Dirty indexes stay dirty after loading; built indexes retain their graph. +Files are specific to this backend and cannot be loaded as HNSW files. + +## Validation + +Build and run `test_pipnn` with the repository's normal CMake test configuration. +The suite covers exact fallback, graph recall, mutations, label width, range and +batch queries, timeouts, invalid inputs, duplicate vectors, allocation cleanup, +and persistence corruption. Production rollout also requires workload-specific +recall/latency/build-memory validation and integration tests in the embedding +application. + +ARM validation on 2026-11-15 passed 245 selected Common, BruteForce, Allocator, +and PiPNN CTest cases with GCC 12.3 and SVS enabled. All 25 PiPNN cases also +passed Clang 21 ASan/UBSan with leak detection. Function-type sanitization was +disabled because the existing CPU-feature C/C++ dispatch triggers that check. +Clang was used after GCC 12 hit a sanitizer compiler error in an existing BF16 +SVE kernel. +The SVS-enabled GCC Release library also builds with both `VECSIM_BUILD_TESTS` +and `VECSIM_BUILD_TEST_HELPERS` disabled. + +The [initial native ARM screen](benchmarks/pipnn/README.md) found faster builds but +slower queries than HNSW at comparable recall on both tested datasets. It does not +establish this backend as a production HNSW replacement. + +Build the optional native comparison runner with +`cmake --build --target pipnn_comparison -j4`, then run: + +```sh +/benchmark/pipnn_comparison base.fbin query.fbin truth.ibin pipnn 50,100,200,400,800 1000 +/benchmark/pipnn_comparison base.fbin query.fbin truth.ibin hnsw 50,100,200,400,800 1000 +``` + +The matrix files have two little-endian uint32 dimensions followed by row-major +FLOAT32 vectors or uint32 ground-truth IDs. Ground truth must contain at least +10 neighbors per query. The runner uses PiPNN defaults and HNSW M=16, +efConstruction=100, seed=100. Build timing includes index creation, insertion, +and PiPNN graph construction. Peak build RSS includes the input matrix; index +bytes come from allocator accounting. Compare latency at matched recall, not +merely at the same ef value. Run each backend in a separate process on an idle +machine and repeat before making a performance decision. diff --git a/src/VecSim/CMakeLists.txt b/src/VecSim/CMakeLists.txt index ab0ee5482..8a35b5f1a 100644 --- a/src/VecSim/CMakeLists.txt +++ b/src/VecSim/CMakeLists.txt @@ -24,6 +24,9 @@ endif() add_library(VectorSimilarity ${VECSIM_LIBTYPE} index_factories/brute_force_factory.cpp + index_factories/pipnn_factory.cpp + algorithms/pipnn/pipnn.cpp + algorithms/pipnn/pipnn_serialization.cpp index_factories/hnsw_factory.cpp index_factories/tiered_factory.cpp index_factories/svs_factory.cpp diff --git a/src/VecSim/algorithms/brute_force/bf_batch_iterator.h b/src/VecSim/algorithms/brute_force/bf_batch_iterator.h index 5477db21a..4c0dd21a7 100644 --- a/src/VecSim/algorithms/brute_force/bf_batch_iterator.h +++ b/src/VecSim/algorithms/brute_force/bf_batch_iterator.h @@ -18,6 +18,7 @@ #include #include #include +#include using std::pair; @@ -106,7 +107,7 @@ void BF_BatchIterator::swapScores( template VecSimQueryReply *BF_BatchIterator::heapBasedSearch(size_t n_res) { - auto rep = new VecSimQueryReply(this->allocator); + auto rep = std::make_unique(this->allocator); DistType upperBound = std::numeric_limits::lowest(); vecsim_stl::max_priority_queue TopCandidates(this->allocator); // map vector's label to its index in the scores vector. @@ -136,12 +137,12 @@ VecSimQueryReply *BF_BatchIterator::heapBasedSearch(size_t n TopCandidates.pop(); } swapScores(TopCandidatesIndices, rep->results.size()); - return rep; + return rep.release(); } template VecSimQueryReply *BF_BatchIterator::selectBasedSearch(size_t n_res) { - auto rep = new VecSimQueryReply(this->allocator); + auto rep = std::make_unique(this->allocator); size_t remaining_vectors_count = this->scores.size() - this->scores_valid_start_pos; // Get an iterator to the effective first element in the scores array, which is the first // element that hasn't been returned in previous iterations. @@ -162,7 +163,7 @@ VecSimQueryReply *BF_BatchIterator::selectBasedSearch(size_t } // Update the valid results start position after returning the results. this->scores_valid_start_pos += rep->results.size(); - return rep; + return rep.release(); } template diff --git a/src/VecSim/algorithms/pipnn/pipnn.cpp b/src/VecSim/algorithms/pipnn/pipnn.cpp new file mode 100644 index 000000000..fe03ebfd3 --- /dev/null +++ b/src/VecSim/algorithms/pipnn/pipnn.cpp @@ -0,0 +1,624 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#include "pipnn.h" + +#include +#include +#include +#include +#include +#include + +namespace { +BFParams storageParams(const PiPNNParams ¶ms) { + return {.type = params.type, + .dim = params.dim, + .metric = params.metric, + .multi = false, + .initialCapacity = params.initialCapacity, + .blockSize = params.blockSize}; +} + +size_t checkedProduct(size_t a, size_t b) { + if (b && a > std::numeric_limits::max() / b) { + throw std::length_error("PiPNN allocation size overflow"); + } + return a * b; +} + +struct Candidate { + float distance; + idType id; + uint32_t hash; +}; + +bool nearer(const Candidate &a, const Candidate &b) { + return a.distance < b.distance || (a.distance == b.distance && a.id < b.id); +} + +// The flat iterator's base receives null. A failed constructor therefore cannot +// free a copied query that is still owned by its caller. +class PiPNNBatchIterator final : public BFS_BatchIterator { + MemoryUtils::unique_blob query_; + + VecSimQueryReply_Code calculateScores() override { + this->index_label_count = this->index->indexLabelCount(); + vecsim_stl::vector> staged(this->allocator); + staged.reserve(this->index_label_count); + for (size_t i = 0; i < this->index_label_count; ++i) { + if (VECSIM_TIMEOUT(this->getTimeoutCtx())) { + return VecSim_QueryReply_TimedOut; + } + const idType id = static_cast(i); + const float score = this->index->calcDistanceForQuery( + this->index->getDataByInternalId(id), this->getQueryBlob()); + staged.emplace_back(score, this->index->getVectorLabel(id)); + } + // Publish only a complete scan. A timeout or allocation failure leaves scores empty, + // so a subsequent batch retries the full scan rather than using a partial prefix. + this->scores = std::move(staged); + return VecSim_QueryReply_OK; + } + +public: + PiPNNBatchIterator(MemoryUtils::unique_blob &query, const PiPNNIndex *index, + VecSimQueryParams *params, const std::shared_ptr &allocator) + : BFS_BatchIterator(nullptr, index, params, allocator), + query_(std::move(query)) {} + + const void *getQueryBlob() const override { return query_.get(); } + + VecSimQueryReply *getNextResults(size_t n, VecSimQueryReply_Order order) override { + try { + if (order != BY_ID && order != BY_SCORE) { + return nullptr; + } + if (n == 0) { + return new VecSimQueryReply(this->allocator); + } + return BFS_BatchIterator::getNextResults(n, order); + } catch (...) { + // A failure must not propagate through the batch C API. The flat search stages + // its allocations before committing the next batch, so the iterator can be retried. + return nullptr; + } + } +}; + +struct Bucket { + vecsim_stl::vector ids; + size_t depth; + Bucket(const std::shared_ptr &allocator, size_t depth) + : ids(allocator), depth(depth) {} +}; +} // namespace + +PiPNNIndex::PiPNNIndex(const PiPNNParams *params, const AbstractIndexInitParams &abstractInitParams, + const IndexComponents &components) + : PiPNNIndex(storageParams(*params), params, abstractInitParams, components) {} + +PiPNNIndex::PiPNNIndex(const BFParams &storageParams, const PiPNNParams *params, + const AbstractIndexInitParams &abstractInitParams, + const IndexComponents &components) + : Base(&storageParams, abstractInitParams, components), params_(*params), + graph_(this->allocator), degrees_(this->allocator) {} + +void PiPNNIndex::validateVector(const void *blob) const { + if (!blob) { + throw std::invalid_argument("PiPNN requires a non-null FP32 vector"); + } + // Test the representation so finite validation also works in builds using -ffast-math. + auto bytes = static_cast(blob); + for (size_t i = 0; i < this->dim; ++i) { + uint32_t bits; + std::memcpy(&bits, bytes + i * sizeof(float), sizeof(bits)); + if ((bits & 0x7f800000U) == 0x7f800000U) { + throw std::invalid_argument("PiPNN requires finite FP32 components"); + } + } +} + +void PiPNNIndex::invalidateGraph() { + graphReady_ = false; + vecsim_stl::vector emptyGraph(this->allocator); + vecsim_stl::vector emptyDegrees(this->allocator); + graph_.swap(emptyGraph); + degrees_.swap(emptyDegrees); +} + +int PiPNNIndex::addVector(const void *vector_data, labelType label) { + validateVector(vector_data); + auto existing = this->labelToIdLookup.find(label); + if (this->count == std::numeric_limits::max() && + existing == this->labelToIdLookup.end()) { + throw std::length_error("PiPNN internal ID capacity exhausted"); + } + auto prepared = this->preprocessForStorage(vector_data); + invalidateGraph(); + if (existing != this->labelToIdLookup.end()) { + this->vectors->updateElement(existing->second, prepared.get()); + return 0; + } + + // Stage metadata growth so a failure cannot leave size and capacity inconsistent. + vecsim_stl::vector expanded(this->allocator); + if (this->indexSize() == this->indexCapacity()) { + if (this->indexCapacity() > std::numeric_limits::max() - this->blockSize) { + throw std::length_error("PiPNN metadata capacity overflow"); + } + expanded.resize(this->indexCapacity() + this->blockSize); + std::copy_n(this->idToLabelMapping.begin(), this->count, expanded.begin()); + expanded[this->count] = label; + } + const idType id = this->count; + this->labelToIdLookup.emplace(label, id); + try { + this->vectors->addElement(prepared.get(), id); + } catch (...) { + this->labelToIdLookup.erase(label); + throw; + } + if (!expanded.empty()) { + this->idToLabelMapping.swap(expanded); + } else { + this->setVectorLabel(id, label); + } + ++this->count; + return 1; +} + +int PiPNNIndex::deleteVector(labelType label) { + const auto found = this->labelToIdLookup.find(label); + if (found == this->labelToIdLookup.end()) { + return 0; + } + invalidateGraph(); + const idType removed = found->second; + const idType last = this->count - 1; + if (removed != last) { + const labelType movedLabel = this->getVectorLabel(last); + this->vectors->updateElement(removed, this->getDataByInternalId(last)); + this->setVectorLabel(removed, movedLabel); + this->labelToIdLookup.find(movedLabel)->second = removed; + } + this->vectors->removeElement(last); + this->labelToIdLookup.erase(found); + --this->count; + // Retain metadata capacity: deletion must not allocate to shrink it. Empty vector-data + // blocks are still released by the storage container. + return 1; +} + +int PiPNNIndex::deleteVectorById(labelType label, idType id) { + // The single-value flat contract resolves the vector by label, including this entry point. + return deleteVector(label); +} + +std::unordered_map> +PiPNNIndex::deleteVectorAndGetUpdatedIds(labelType label) { + std::unordered_map> updated; + const auto found = this->labelToIdLookup.find(label); + if (found == this->labelToIdLookup.end()) { + return updated; + } + const idType removed = found->second; + const idType last = this->count - 1; + if (removed != last) { + updated.emplace(removed, std::make_pair(last, this->getVectorLabel(last))); + } + deleteVector(label); + return updated; +} + +VecSimUpdateCode PiPNNIndex::updateVectors(labelType label, const void *new_blobs, size_t n) { + if (n > 1) { + return VecSimUpdate_MultiNotSupported; + } + if (n == 1) { + addVector(new_blobs, label); + } else { + deleteVector(label); + } + return VecSimUpdate_OK; +} + +VecSimRelabelCode PiPNNIndex::relabelVector(labelType old_label, labelType new_label) { + if (old_label == new_label) { + return VecSimRelabel_SameLabel; + } + const auto old = this->labelToIdLookup.find(old_label); + if (old == this->labelToIdLookup.end()) { + return VecSimRelabel_OldLabelMissing; + } + if (this->labelToIdLookup.find(new_label) != this->labelToIdLookup.end()) { + return VecSimRelabel_NewLabelTaken; + } + const idType id = old->second; + invalidateGraph(); + // Insert before removing the old mapping; an allocation failure preserves the label. + // Erase by key because insertion may rehash and invalidate the old iterator. + this->labelToIdLookup.emplace(new_label, id); + this->setVectorLabel(id, new_label); + this->labelToIdLookup.erase(old_label); + return VecSimRelabel_OK; +} + +double PiPNNIndex::getDistanceFrom_Unsafe(labelType label, const void *vector_data) const { + validateVector(vector_data); + return Base::getDistanceFrom_Unsafe(label, vector_data); +} + +VecSimQueryReply *PiPNNIndex::rangeQuery(const void *queryBlob, double radius, + VecSimQueryParams *queryParams) const { + validateVector(queryBlob); + auto query = this->preprocessQuery(queryBlob); + auto reply = std::make_unique(this->allocator); + this->lastMode = RANGE_QUERY; + void *timeoutCtx = queryParams ? queryParams->timeoutCtx : nullptr; + const float bound = static_cast(radius); + for (idType id = 0; id < this->count; ++id) { + if (VECSIM_TIMEOUT(timeoutCtx)) { + reply->code = VecSim_QueryReply_TimedOut; + break; + } + const float score = this->calcDistanceForQuery(this->getDataByInternalId(id), query.get()); + if (score <= bound) { + reply->results.push_back({this->getVectorLabel(id), score}); + } + } + return reply.release(); +} + +VecSimBatchIterator *PiPNNIndex::newBatchIterator(const void *queryBlob, + VecSimQueryParams *queryParams) const { + validateVector(queryBlob); + auto query = this->preprocessQuery(queryBlob, true); + return new (this->allocator) PiPNNBatchIterator(query, this, queryParams, this->allocator); +} + +void PiPNNIndex::buildGraph() { + // Building is transactional: a failed build leaves valid flat storage and exact queries. + invalidateGraph(); + const size_t n = this->count; + vecsim_stl::vector graph(checkedProduct(n, params_.maxDegree), idType{0}, + this->allocator); + vecsim_stl::vector degrees(n, size_t{0}, this->allocator); + if (n < 2) { + graph_.swap(graph); + degrees_.swap(degrees); + graphReady_ = true; + return; + } + + const size_t cap = std::min(params_.candidateLimit, n - 1); + vecsim_stl::vector candidates(checkedProduct(n, cap), this->allocator); + vecsim_stl::vector candidateCounts(n, size_t{0}, this->allocator); + std::mt19937_64 random(params_.seed); + + // Project each point once. Subtracting the source projection from the target projection + // hashes the residual direction, rather than the point's position relative to the origin. + // Double precision avoids overflow for finite FP32 input during these dot products. + const size_t bits = params_.hashBits; + vecsim_stl::vector planes(checkedProduct(bits, this->dim), this->allocator); + std::normal_distribution gaussian(0.0, 1.0); + for (double &component : planes) { + component = gaussian(random); + } + vecsim_stl::vector sketches(checkedProduct(n, bits), 0.0, this->allocator); + for (size_t i = 0; i < n; ++i) { + const float *point = this->getDataByInternalId(static_cast(i)); + for (size_t bit = 0; bit < bits; ++bit) { + double projection = 0; + for (size_t d = 0; d < this->dim; ++d) { + projection += double(point[d]) * planes[bit * this->dim + d]; + } + sketches[i * bits + bit] = projection; + } + } + auto merge = [&](idType source, idType target, float distance) { + uint32_t hash = 0; + for (size_t bit = 0; bit < bits; ++bit) { + if (sketches[size_t(target) * bits + bit] > sketches[size_t(source) * bits + bit]) { + hash |= uint32_t{1} << bit; + } + } + Candidate incoming{distance, target, hash}; + Candidate *row = candidates.data() + size_t(source) * cap; + size_t &size = candidateCounts[source]; + size_t worst = 0; + for (size_t j = 0; j < size; ++j) { + if (row[j].hash == hash) { + if (nearer(incoming, row[j])) { + row[j] = incoming; + } + return; + } + if (nearer(row[worst], row[j])) { + worst = j; + } + } + if (size < cap) { + row[size++] = incoming; + } else if (nearer(incoming, row[worst])) { + row[worst] = incoming; + } + }; + + auto processLeaf = [&](const vecsim_stl::vector &ids, size_t begin, size_t end) { + // PiPNN collects only a small local neighborhood from each overlapping leaf. Both + // directions are considered, while the global reservoir remains candidateLimit wide. + for (size_t i = begin; i < end; ++i) { + Candidate first{std::numeric_limits::infinity(), + std::numeric_limits::max(), 0}; + Candidate second = first; + for (size_t j = begin; j < end; ++j) { + if (i == j) { + continue; + } + Candidate value{this->calcDistance(this->getDataByInternalId(ids[i]), + this->getDataByInternalId(ids[j])), + ids[j], 0}; + if (nearer(value, first)) { + second = first; + first = value; + } else if (nearer(value, second)) { + second = value; + } + } + for (const Candidate &value : {first, second}) { + if (value.id != std::numeric_limits::max()) { + merge(ids[i], value.id, value.distance); + merge(value.id, ids[i], value.distance); + } + } + } + }; + + vecsim_stl::vector pending(this->allocator); + pending.emplace_back(this->allocator, 0); + pending.back().ids.resize(n); + std::iota(pending.back().ids.begin(), pending.back().ids.end(), idType{0}); + while (!pending.empty()) { + Bucket bucket = std::move(pending.back()); + pending.pop_back(); + if (bucket.ids.size() <= params_.leafSize) { + processLeaf(bucket.ids, 0, bucket.ids.size()); + continue; + } + // Sampling a bounded number of leaders keeps assignment linear in the bucket size. + // Overlap is limited to the first two levels, preventing replication at every level. + const size_t leaderCount = + std::min(bucket.ids.size(), + std::min(size_t{512}, + std::max(params_.fanout + 1, + std::max(size_t{3}, bucket.ids.size() / params_.leafSize)))); + const size_t overlap = bucket.depth < 2 ? std::min(params_.fanout, leaderCount) : 1; + vecsim_stl::vector leaders(this->allocator); + leaders = bucket.ids; + // Partial Fisher-Yates samples distinct leaders without shuffling the whole bucket. + for (size_t j = 0; j < leaderCount; ++j) { + const size_t chosen = j + random() % (leaders.size() - j); + std::swap(leaders[j], leaders[chosen]); + } + leaders.resize(leaderCount); + vecsim_stl::vector children(this->allocator); + children.reserve(leaderCount); + for (size_t j = 0; j < leaderCount; ++j) { + children.emplace_back(this->allocator, bucket.depth + 1); + } + vecsim_stl::vector> distances(leaderCount, this->allocator); + for (idType id : bucket.ids) { + for (size_t j = 0; j < leaderCount; ++j) { + distances[j] = {this->calcDistance(this->getDataByInternalId(id), + this->getDataByInternalId(leaders[j])), + j}; + } + std::partial_sort(distances.begin(), distances.begin() + overlap, distances.end()); + for (size_t j = 0; j < overlap; ++j) { + children[distances[j].second].ids.push_back(id); + } + } + for (Bucket &child : children) { + if (child.ids.empty()) { + continue; + } + // Coincident points can choose the same leaders forever. A depth limit and a + // strict shrink check terminate that case using bounded deterministic leaves. + if (bucket.depth >= 4 || + child.ids.size() >= bucket.ids.size() - bucket.ids.size() / 5) { + for (size_t begin = 0; begin < child.ids.size(); begin += params_.leafSize) { + processLeaf(child.ids, begin, + std::min(begin + params_.leafSize, child.ids.size())); + } + } else { + pending.push_back(std::move(child)); + } + } + } + + for (size_t source = 0; source < n; ++source) { + Candidate *row = candidates.data() + source * cap; + std::sort(row, row + candidateCounts[source], nearer); + size_t °ree = degrees[source]; + // Reserve one edge for a directed spanning cycle. This is a connectivity safeguard + // within the configured degree bound, including fully duplicated datasets. + const idType successor = static_cast((source + 1) % n); + for (size_t j = 0; j < candidateCounts[source] && degree < params_.maxDegree - 1; ++j) { + const Candidate &candidate = row[j]; + if (candidate.id == successor) { + continue; + } + bool occluded = false; + for (size_t edge = 0; edge < degree; ++edge) { + const idType selected = graph[source * params_.maxDegree + edge]; + const float between = this->calcDistance(this->getDataByInternalId(selected), + this->getDataByInternalId(candidate.id)); + if (params_.alpha * double(between) <= double(candidate.distance)) { + occluded = true; + break; + } + } + if (!occluded) { + graph[source * params_.maxDegree + degree++] = candidate.id; + } + } + graph[source * params_.maxDegree + degree++] = successor; + } + graph_.swap(graph); + degrees_.swap(degrees); + graphReady_ = true; +} + +VecSimQueryReply *PiPNNIndex::topKQuery(const void *queryBlob, size_t k, + VecSimQueryParams *queryParams) const { + k = std::min(k, size_t(this->count)); + if (k == 0) { + this->lastMode = STANDARD_KNN; + return new VecSimQueryReply(this->allocator); + } + validateVector(queryBlob); + if (!graphReady_) { + auto reply = std::make_unique(this->allocator); + this->lastMode = STANDARD_KNN; + auto query = this->preprocessQuery(queryBlob); + void *timeoutCtx = queryParams ? queryParams->timeoutCtx : nullptr; + using ScoredLabel = std::pair; + vecsim_stl::vector storage(this->allocator); + std::priority_queue> nearest( + std::less{}, std::move(storage)); + for (idType id = 0; id < this->count; ++id) { + if (VECSIM_TIMEOUT(timeoutCtx)) { + reply->code = VecSim_QueryReply_TimedOut; + return reply.release(); + } + const ScoredLabel candidate{ + this->calcDistanceForQuery(this->getDataByInternalId(id), query.get()), + this->getVectorLabel(id)}; + if (nearest.size() < k || candidate < nearest.top()) { + nearest.push(candidate); + if (nearest.size() > k) { + nearest.pop(); + } + } + } + reply->results.resize(nearest.size()); + for (auto it = reply->results.rbegin(); it != reply->results.rend(); ++it) { + std::tie(it->score, it->id) = nearest.top(); + nearest.pop(); + } + return reply.release(); + } + auto reply = std::make_unique(this->allocator); + this->lastMode = STANDARD_KNN; + void *timeoutCtx = queryParams ? queryParams->timeoutCtx : nullptr; + auto query = this->preprocessQuery(queryBlob); + const size_t requestedEf = queryParams && queryParams->pipnnRuntimeParams.efRuntime + ? queryParams->pipnnRuntimeParams.efRuntime + : params_.efRuntime; + const size_t beam = std::min(size_t(this->count), std::max(k, requestedEf)); + using ScoredId = std::pair; + vecsim_stl::vector frontierStorage(this->allocator); + vecsim_stl::vector bestStorage(this->allocator); + std::priority_queue, std::greater> frontier( + std::greater{}, std::move(frontierStorage)); + std::priority_queue> best(std::less{}, + std::move(bestStorage)); + vecsim_stl::unordered_set visited(this->allocator); + auto visit = [&](idType id) { + if (!visited.insert(id).second) { + return; + } + ScoredId candidate{this->calcDistanceForQuery(this->getDataByInternalId(id), query.get()), + id}; + if (best.size() < beam || candidate < best.top()) { + best.push(candidate); + frontier.push(candidate); + if (best.size() > beam) { + best.pop(); + } + } + }; + // Deterministic starting points avoid dependence on the insertion order of a single entry. + const size_t startCount = std::min(size_t{4}, size_t(this->count)); + for (size_t i = 0; i < startCount; ++i) { + if (VECSIM_TIMEOUT(timeoutCtx)) { + reply->code = VecSim_QueryReply_TimedOut; + return reply.release(); + } + visit(static_cast(i * (size_t(this->count) / startCount))); + } + while (!frontier.empty()) { + if (VECSIM_TIMEOUT(timeoutCtx)) { + reply->code = VecSim_QueryReply_TimedOut; + return reply.release(); + } + const ScoredId next = frontier.top(); + frontier.pop(); + if (best.size() == beam && next > best.top()) { + break; + } + for (size_t j = 0; j < degrees_[next.second]; ++j) { + if (VECSIM_TIMEOUT(timeoutCtx)) { + reply->code = VecSim_QueryReply_TimedOut; + return reply.release(); + } + visit(graph_[size_t(next.second) * params_.maxDegree + j]); + } + } + vecsim_stl::vector ordered(this->allocator); + ordered.reserve(best.size()); + while (!best.empty()) { + const auto [score, id] = best.top(); + best.pop(); + ordered.push_back({this->getVectorLabel(id), score}); + } + std::sort(ordered.begin(), ordered.end(), [](const auto &a, const auto &b) { + return a.score < b.score || (a.score == b.score && a.id < b.id); + }); + ordered.resize(std::min(k, ordered.size())); + reply->results = std::move(ordered); + return reply.release(); +} + +VecSimIndexBasicInfo PiPNNIndex::basicInfo() const { + auto info = Base::basicInfo(); + info.algo = VecSimAlgo_PIPNN; + return info; +} + +VecSimIndexDebugInfo PiPNNIndex::debugInfo() const { + VecSimIndexDebugInfo info{}; + info.commonInfo = this->getCommonInfo(); + info.commonInfo.basicInfo = basicInfo(); + return info; +} + +VecSimDebugInfoIterator *PiPNNIndex::debugInfoIterator() const { + const auto info = this->debugInfo(); + // Reserve common, PiPNN and the C API's shared-memory field before publishing the iterator. + auto iterator = std::make_unique(14, this->allocator); + iterator->addInfoField(VecSim_InfoField{ + .fieldName = VecSimCommonStrings::ALGORITHM_STRING, + .fieldType = INFOFIELD_STRING, + .fieldValue = {FieldValue{.stringValue = VecSimCommonStrings::PIPNN_STRING}}}); + this->addCommonInfoToIterator(iterator.get(), info.commonInfo); + iterator->addInfoField( + VecSim_InfoField{.fieldName = VecSimCommonStrings::BLOCK_SIZE_STRING, + .fieldType = INFOFIELD_UINT64, + .fieldValue = {FieldValue{.uintegerValue = this->getBlockSize()}}}); + iterator->addInfoField( + VecSim_InfoField{.fieldName = "GRAPH_READY", + .fieldType = INFOFIELD_UINT64, + .fieldValue = {FieldValue{.uintegerValue = graphReady_}}}); + iterator->addInfoField( + VecSim_InfoField{.fieldName = "MAX_DEGREE", + .fieldType = INFOFIELD_UINT64, + .fieldValue = {FieldValue{.uintegerValue = params_.maxDegree}}}); + return iterator.release(); +} diff --git a/src/VecSim/algorithms/pipnn/pipnn.h b/src/VecSim/algorithms/pipnn/pipnn.h new file mode 100644 index 000000000..509d510f0 --- /dev/null +++ b/src/VecSim/algorithms/pipnn/pipnn.h @@ -0,0 +1,62 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#pragma once + +#include "VecSim/algorithms/brute_force/brute_force_single.h" +#include + +// An explicitly built, single-value FP32/L2 PiPNN graph. Storage, exact range queries and +// batch iteration use the flat index contracts. Callers must serialize all operations, including +// queries and batch iteration, because the inherited query statistics are mutable. +class PiPNNIndex final : public BruteForceIndex_Single { + using Base = BruteForceIndex_Single; + + PiPNNParams params_; + // Each row occupies maxDegree slots, only degrees_[id] of which are initialized edges. + vecsim_stl::vector graph_; + vecsim_stl::vector degrees_; + bool graphReady_ = false; + + PiPNNIndex(const BFParams &storageParams, const PiPNNParams *params, + const AbstractIndexInitParams &abstractInitParams, + const IndexComponents &components); + void invalidateGraph(); + void validateVector(const void *blob) const; + +public: + PiPNNIndex(const PiPNNParams *params, const AbstractIndexInitParams &abstractInitParams, + const IndexComponents &components); + + int addVector(const void *vector_data, labelType label) override; + int deleteVector(labelType label) override; + int deleteVectorById(labelType label, idType id) override; + std::unordered_map> + deleteVectorAndGetUpdatedIds(labelType label) override; + VecSimUpdateCode updateVectors(labelType label, const void *new_blobs, size_t n) override; + VecSimRelabelCode relabelVector(labelType old_label, labelType new_label) override; + double getDistanceFrom_Unsafe(labelType label, const void *vector_data) const override; + + VecSimQueryReply *topKQuery(const void *queryBlob, size_t k, + VecSimQueryParams *queryParams) const override; + using Base::rangeQuery; + VecSimQueryReply *rangeQuery(const void *queryBlob, double radius, + VecSimQueryParams *queryParams) const override; + VecSimBatchIterator *newBatchIterator(const void *queryBlob, + VecSimQueryParams *queryParams) const override; + VecSimIndexBasicInfo basicInfo() const override; + VecSimIndexDebugInfo debugInfo() const override; + VecSimDebugInfoIterator *debugInfoIterator() const override; + + // No query builds a graph implicitly. A successful mutation discards the graph and queries + // fall back to exact scanning until the caller explicitly builds again. + void buildGraph(); + bool isGraphReady() const { return graphReady_; } + void saveIndex(const std::string &path) const; + static PiPNNIndex *loadIndex(const std::string &path, void *logCtx = nullptr); +}; diff --git a/src/VecSim/algorithms/pipnn/pipnn_serialization.cpp b/src/VecSim/algorithms/pipnn/pipnn_serialization.cpp new file mode 100644 index 000000000..8d273e56f --- /dev/null +++ b/src/VecSim/algorithms/pipnn/pipnn_serialization.cpp @@ -0,0 +1,260 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#include "pipnn.h" +#include "VecSim/index_factories/pipnn_factory.h" +#include "VecSim/vec_sim.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +static_assert(sizeof(idType) == 4, "PiPNN format v1 requires 32-bit internal IDs"); +constexpr char magic[] = "VSPIPE01"; +constexpr uint64_t hashBasis = 14695981039346656037ULL; +constexpr uint64_t hashPrime = 1099511628211ULL; + +size_t checkedProduct(size_t a, size_t b) { + if (b && a > SIZE_MAX / b) + throw std::runtime_error("PiPNN file size overflow"); + return a * b; +} +size_t checkedSum(size_t a, size_t b) { + if (a > SIZE_MAX - b) + throw std::runtime_error("PiPNN file size overflow"); + return a + b; +} + +// Integers and IEEE floating-point bit patterns use little-endian encoding. +class Writer { + FILE *file; + uint64_t hash = hashBasis; + +public: + explicit Writer(FILE *f) : file(f) {} + void bytes(const void *data, size_t size, bool checksum = true) { + auto p = static_cast(data); + if (std::fwrite(data, 1, size, file) != size) + throw std::runtime_error("PiPNN write failed"); + if (checksum) + for (size_t i = 0; i < size; ++i) + hash = (hash ^ p[i]) * hashPrime; + } + void integer(uint64_t value, unsigned width = 8, bool checksum = true) { + unsigned char bytes_[8]; + for (unsigned i = 0; i < width; ++i) + bytes_[i] = value >> (8 * i); + bytes(bytes_, width, checksum); + } + void finish() { integer(hash, 8, false); } +}; + +class Reader { + std::ifstream file; + uint64_t hash = hashBasis; + size_t remaining_; + +public: + explicit Reader(const std::string &path) : file(path, std::ios::binary | std::ios::ate) { + if (!file) + throw std::runtime_error("PiPNN open failed"); + auto size = file.tellg(); + if (size < 0 || static_cast(size) > SIZE_MAX) + throw std::runtime_error("PiPNN invalid file length"); + remaining_ = static_cast(size); + file.seekg(0); + } + size_t remaining() const { return remaining_; } + void bytes(void *data, size_t size, bool checksum = true) { + if (size > remaining_) + throw std::runtime_error("PiPNN truncated file"); + file.read(static_cast(data), size); + if (!file) + throw std::runtime_error("PiPNN read failed"); + remaining_ -= size; + auto p = static_cast(data); + if (checksum) + for (size_t i = 0; i < size; ++i) + hash = (hash ^ p[i]) * hashPrime; + } + uint64_t integer(unsigned width = 8, bool checksum = true) { + unsigned char data[8]; + bytes(data, width, checksum); + uint64_t value = 0; + for (unsigned i = 0; i < width; ++i) + value |= uint64_t(data[i]) << (8 * i); + return value; + } + size_t size() { + auto value = integer(); + if (value > SIZE_MAX) + throw std::runtime_error("PiPNN size is not representable"); + return static_cast(value); + } + void finish() { + if (integer(8, false) != hash || remaining_ != 0) + throw std::runtime_error("PiPNN checksum or trailing data error"); + } +}; +} // namespace + +void PiPNNIndex::saveIndex(const std::string &location) const { + static_assert(sizeof(float) == 4 && std::numeric_limits::is_iec559); + static_assert(sizeof(double) == 8 && std::numeric_limits::is_iec559); + if (location.empty()) + throw std::invalid_argument("PiPNN empty save path"); + std::string temporary = location + ".tmp.XXXXXX"; + int fd = mkstemp(temporary.data()); + if (fd < 0) + throw std::runtime_error("PiPNN create temporary file failed"); + FILE *file = fdopen(fd, "wb"); + if (!file) { + close(fd); + unlink(temporary.c_str()); + throw std::runtime_error("PiPNN open temporary file failed"); + } + try { + Writer out(file); + out.bytes(magic, 8); + out.integer(1, 4); + out.integer(params_.dim); + out.integer(params_.blockSize); + out.integer(params_.maxDegree); + out.integer(params_.leafSize); + out.integer(params_.fanout); + out.integer(params_.hashBits); + out.integer(params_.candidateLimit); + out.integer(std::bit_cast(params_.alpha)); + out.integer(params_.efRuntime); + out.integer(params_.seed); + out.integer(this->indexSize()); + out.integer(graphReady_ ? 1 : 0, 4); + for (size_t i = 0; i < this->indexSize(); ++i) { + out.integer(this->getVectorLabel(i)); + const float *data = this->getDataByInternalId(i); + for (size_t d = 0; d < params_.dim; ++d) + out.integer(std::bit_cast(data[d]), 4); + } + if (graphReady_) { + if (degrees_.size() != this->indexSize() || + graph_.size() != checkedProduct(this->indexSize(), params_.maxDegree)) + throw std::runtime_error("PiPNN invalid graph shape"); + for (auto degree : degrees_) + out.integer(degree); + for (auto edge : graph_) + out.integer(edge, 4); + } + out.finish(); + if (fflush(file) || fsync(fileno(file))) + throw std::runtime_error("PiPNN flush failed"); + int result = fclose(file); + file = nullptr; + if (result) + throw std::runtime_error("PiPNN close failed"); + if (rename(temporary.c_str(), location.c_str())) + throw std::runtime_error("PiPNN atomic rename failed"); + } catch (...) { + if (file) + fclose(file); + unlink(temporary.c_str()); + throw; + } +} + +PiPNNIndex *PiPNNIndex::loadIndex(const std::string &location, void *logCtx) { + Reader in(location); + char signature[8]; + in.bytes(signature, sizeof(signature)); + if (std::memcmp(signature, magic, 8) || in.integer(4) != 1) + throw std::runtime_error("PiPNN unsupported file format"); + PiPNNParams params{}; + params.type = VecSimType_FLOAT32; + params.metric = VecSimMetric_L2; + params.dim = in.size(); + params.blockSize = in.size(); + params.maxDegree = in.size(); + params.leafSize = in.size(); + params.fanout = in.size(); + params.hashBits = in.size(); + params.candidateLimit = in.size(); + params.alpha = std::bit_cast(in.integer()); + params.efRuntime = in.size(); + params.seed = in.integer(); + const size_t count = in.size(); + const auto ready = in.integer(4); + if (ready > 1 || count > std::numeric_limits::max()) + throw std::runtime_error("PiPNN invalid index size or state"); + params = PiPNNFactory::ResolveParams(params); + const size_t graphSize = ready ? checkedProduct(count, params.maxDegree) : 0; + size_t expected = checkedProduct(count, checkedSum(8, checkedProduct(params.dim, 4))); + expected = checkedSum(expected, ready ? checkedProduct(count, 8) : 0); + expected = checkedSum(expected, checkedProduct(graphSize, 4)); + expected = checkedSum(expected, 8); + if (in.remaining() != expected) + throw std::runtime_error("PiPNN invalid payload length"); + + VecSimParams config{}; + config.algo = VecSimAlgo_PIPNN; + config.algoParams.pipnnParams = params; + config.logCtx = logCtx; + std::unique_ptr index( + static_cast(PiPNNFactory::NewIndex(&config)), VecSimIndex_Free); + if (!index) + throw std::runtime_error("PiPNN index creation failed"); + vecsim_stl::vector data(index->getAllocator()); + data.resize(params.dim); + for (size_t i = 0; i < count; ++i) { + const auto label = in.integer(); + if (label > std::numeric_limits::max()) + throw std::runtime_error("PiPNN label is not representable"); + for (size_t d = 0; d < params.dim; ++d) { + data[d] = std::bit_cast(static_cast(in.integer(4))); + if (!std::isfinite(data[d])) + throw std::runtime_error("PiPNN nonfinite vector"); + } + if (index->addVector(data.data(), static_cast(label)) != 1) + throw std::runtime_error("PiPNN duplicate label in file"); + } + if (ready) { + index->degrees_.resize(count); + index->graph_.resize(graphSize); + for (auto °ree : index->degrees_) { + degree = in.size(); + if (degree > params.maxDegree || degree >= count) + throw std::runtime_error("PiPNN invalid degree"); + } + for (auto &edge : index->graph_) { + edge = static_cast(in.integer(4)); + } + for (size_t i = 0; i < count; ++i) { + bool hasSuccessor = count <= 1; + for (size_t j = 0; j < index->degrees_[i]; ++j) { + const auto edge = index->graph_[i * params.maxDegree + j]; + if (edge >= count || edge == i) + throw std::runtime_error("PiPNN invalid edge"); + hasSuccessor |= edge == (i + 1) % count; + for (size_t k = 0; k < j; ++k) + if (index->graph_[i * params.maxDegree + k] == edge) + throw std::runtime_error("PiPNN duplicate edge"); + } + if (!hasSuccessor) + throw std::runtime_error("PiPNN missing connectivity edge"); + } + } + in.finish(); + index->graphReady_ = ready != 0; + return index.release(); +} diff --git a/src/VecSim/index_factories/index_factory.cpp b/src/VecSim/index_factories/index_factory.cpp index 16459a091..ae59d77ef 100644 --- a/src/VecSim/index_factories/index_factory.cpp +++ b/src/VecSim/index_factories/index_factory.cpp @@ -10,14 +10,17 @@ #include "index_factory.h" #include "hnsw_factory.h" #include "brute_force_factory.h" +#include "pipnn_factory.h" #include "tiered_factory.h" #include "svs_factory.h" namespace VecSimFactory { VecSimIndex *NewIndex(const VecSimParams *params) { VecSimIndex *index = NULL; - std::shared_ptr allocator = VecSimAllocator::newVecsimAllocator(); try { + if (!params) { + return nullptr; + } switch (params->algo) { case VecSimAlgo_HNSWLIB: { index = HNSWFactory::NewIndex(params); @@ -28,6 +31,10 @@ VecSimIndex *NewIndex(const VecSimParams *params) { index = BruteForceFactory::NewIndex(params); break; } + case VecSimAlgo_PIPNN: { + index = PiPNNFactory::NewIndex(params); + break; + } case VecSimAlgo_TIERED: { index = TieredFactory::NewIndex(¶ms->algoParams.tieredParams); break; @@ -44,11 +51,16 @@ VecSimIndex *NewIndex(const VecSimParams *params) { } size_t EstimateInitialSize(const VecSimParams *params) { + if (!params) { + return SIZE_MAX; + } switch (params->algo) { case VecSimAlgo_HNSWLIB: return HNSWFactory::EstimateInitialSize(¶ms->algoParams.hnswParams); case VecSimAlgo_BF: return BruteForceFactory::EstimateInitialSize(¶ms->algoParams.bfParams); + case VecSimAlgo_PIPNN: + return PiPNNFactory::EstimateInitialSize(¶ms->algoParams.pipnnParams); case VecSimAlgo_TIERED: return TieredFactory::EstimateInitialSize(¶ms->algoParams.tieredParams); case VecSimAlgo_SVS:; // empty statement if svs not available @@ -58,11 +70,16 @@ size_t EstimateInitialSize(const VecSimParams *params) { } size_t EstimateElementSize(const VecSimParams *params) { + if (!params) { + return SIZE_MAX; + } switch (params->algo) { case VecSimAlgo_HNSWLIB: return HNSWFactory::EstimateElementSize(¶ms->algoParams.hnswParams); case VecSimAlgo_BF: return BruteForceFactory::EstimateElementSize(¶ms->algoParams.bfParams); + case VecSimAlgo_PIPNN: + return PiPNNFactory::EstimateElementSize(¶ms->algoParams.pipnnParams); case VecSimAlgo_TIERED: return TieredFactory::EstimateElementSize(¶ms->algoParams.tieredParams); case VecSimAlgo_SVS:; // empty statement if svs not available diff --git a/src/VecSim/index_factories/pipnn_factory.cpp b/src/VecSim/index_factories/pipnn_factory.cpp new file mode 100644 index 000000000..bd0234ac4 --- /dev/null +++ b/src/VecSim/index_factories/pipnn_factory.cpp @@ -0,0 +1,157 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#include "VecSim/index_factories/pipnn_factory.h" +#include "VecSim/algorithms/pipnn/pipnn.h" +#include "VecSim/index_factories/brute_force_factory.h" +#include "VecSim/index_factories/components/components_factory.h" +#include "VecSim/index_factories/factory_utils.h" +#include +#include +#include +#include +#include + +namespace PiPNNFactory { +namespace { +size_t CheckedAdd(size_t a, size_t b) { + if (a > std::numeric_limits::max() - b) { + throw std::overflow_error("PiPNN allocation size overflow"); + } + return a + b; +} +size_t CheckedMultiply(size_t a, size_t b) { + if (b && a > std::numeric_limits::max() / b) { + throw std::overflow_error("PiPNN allocation size overflow"); + } + return a * b; +} +BFParams ToBFParams(const PiPNNParams ¶ms) { + return {.type = params.type, + .dim = params.dim, + .metric = params.metric, + .multi = false, + .initialCapacity = params.initialCapacity, + .blockSize = params.blockSize}; +} +} // namespace + +PiPNNParams ResolveParams(const PiPNNParams &input) { + if (input.type != VecSimType_FLOAT32 || input.metric != VecSimMetric_L2 || input.multi || + input.dim == 0) { + throw std::invalid_argument("PiPNN requires FLOAT32, L2, positive dimension, single-value"); + } + PiPNNParams params = input; + if (!params.blockSize) { + params.blockSize = DEFAULT_BLOCK_SIZE; + } + if (!params.maxDegree) { + params.maxDegree = 32; + } + if (!params.leafSize) { + params.leafSize = 128; + } + if (!params.fanout) { + params.fanout = 2; + } + if (!params.hashBits) { + params.hashBits = 12; + } + if (!params.candidateLimit) { + params.candidateLimit = 80; + } + if (params.alpha == 0.0) { + params.alpha = 1.1; + } + if (!params.efRuntime) { + params.efRuntime = 100; + } + if (!params.seed) { + params.seed = 100; + } + constexpr size_t max_id = std::numeric_limits::max(); + if (params.maxDegree < 2 || params.maxDegree > max_id || params.leafSize < 2 || + params.leafSize > max_id || params.fanout < 2 || params.fanout > 32 || + params.hashBits > 32 || params.candidateLimit < params.maxDegree || + params.candidateLimit > max_id || params.efRuntime > max_id || params.blockSize > max_id || + params.initialCapacity > max_id || !std::isfinite(params.alpha) || params.alpha < 1.0) { + throw std::invalid_argument("Invalid PiPNN tuning parameters"); + } + + // Check all parameter-dependent allocation products before dispatching or estimating. + // Include allocator headers/alignment and metadata, not just the vector payload. + const size_t overhead = CheckedAdd(VecSimAllocator::getAllocationOverheadSize(), 64); + const size_t vector_bytes = CheckedMultiply(params.dim, sizeof(float)); + CheckedAdd(CheckedMultiply(params.blockSize, vector_bytes), overhead); + CheckedAdd(CheckedMultiply(params.blockSize, sizeof(labelType)), overhead); + CheckedAdd(CheckedMultiply(params.maxDegree, sizeof(idType)), overhead); + CheckedAdd(CheckedMultiply(params.candidateLimit, sizeof(std::pair)), overhead); + CheckedAdd(CheckedMultiply(params.efRuntime, sizeof(std::pair)), overhead); + CheckedAdd(CheckedMultiply(CheckedMultiply(params.fanout, params.dim), sizeof(float)), + overhead); + CheckedAdd(CheckedMultiply(CheckedMultiply(params.hashBits, params.dim), sizeof(float)), + overhead); + if (params.initialCapacity) { + const size_t rounded = CheckedAdd(params.initialCapacity, params.blockSize - 1) / + params.blockSize * params.blockSize; + if (rounded > max_id) { + throw std::invalid_argument("PiPNN capacity exceeds internal ID range"); + } + CheckedMultiply(rounded, vector_bytes); + CheckedMultiply(CheckedMultiply(rounded, params.maxDegree), sizeof(idType)); + } + return params; +} + +VecSimIndex *NewIndex(const VecSimParams *params) { + if (!params || params->algo != VecSimAlgo_PIPNN) { + throw std::invalid_argument("PiPNN factory requires PiPNN parameters"); + } + const auto resolved = ResolveParams(params->algoParams.pipnnParams); + const auto init = VecSimFactory::NewAbstractInitParams(&resolved, params->logCtx, false); + init.allocator->setThrowOnAllocationFailure(); + // Reserve the index allocation before constructing components. Its guard also releases + // the storage if any part of the index constructor fails. + auto storage = init.allocator->allocate_unique(sizeof(PiPNNIndex)); + unsigned char alignment = 0; + auto dist_func = spaces::GetDistFunc(resolved.metric, resolved.dim, &alignment); + std::unique_ptr> calculator( + new (init.allocator) DistanceCalculatorCommon(init.allocator, dist_func)); + // PiPNN only accepts L2, so this container does not need normalization preprocessors. + std::unique_ptr preprocessors( + new (init.allocator) PreprocessorsContainerAbstract(init.allocator, alignment)); + const IndexComponents components{calculator.release(), preprocessors.release()}; + // VecSimIndexAbstract adopts both components, including cleanup if its storage allocation + // fails. A completed base destructor owns them if a later constructor fails. + auto *index = ::new (storage.get()) PiPNNIndex(&resolved, init, components); + storage.release(); + return index; +} + +size_t EstimateInitialSize(const PiPNNParams *params) { + if (!params) { + throw std::invalid_argument("Missing PiPNN parameters"); + } + const auto resolved = ResolveParams(*params); + const auto bf = ToBFParams(resolved); + const size_t base = BruteForceFactory::EstimateInitialSize(&bf); + return CheckedAdd(base - sizeof(BruteForceIndex_Single), sizeof(PiPNNIndex)); +} + +size_t EstimateElementSize(const PiPNNParams *params) { + if (!params) { + throw std::invalid_argument("Missing PiPNN parameters"); + } + const auto resolved = ResolveParams(*params); + const auto bf = ToBFParams(resolved); + // The built graph stores one degree and a fixed-width row of IDs per node. + size_t size = BruteForceFactory::EstimateElementSize(&bf); + size = CheckedAdd(size, sizeof(size_t)); + return CheckedAdd(size, CheckedMultiply(resolved.maxDegree, sizeof(idType))); +} +} // namespace PiPNNFactory diff --git a/src/VecSim/index_factories/pipnn_factory.h b/src/VecSim/index_factories/pipnn_factory.h new file mode 100644 index 000000000..993cab25d --- /dev/null +++ b/src/VecSim/index_factories/pipnn_factory.h @@ -0,0 +1,20 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#pragma once + +#include "VecSim/vec_sim.h" + +namespace PiPNNFactory { +// Resolve zero-valued defaults and validate every public setting. Throws invalid_argument or +// overflow_error on invalid input. Shared by construction, estimates, and native loading. +PiPNNParams ResolveParams(const PiPNNParams ¶ms); +VecSimIndex *NewIndex(const VecSimParams *params); +size_t EstimateInitialSize(const PiPNNParams *params); +size_t EstimateElementSize(const PiPNNParams *params); +} // namespace PiPNNFactory diff --git a/src/VecSim/memory/vecsim_malloc.cpp b/src/VecSim/memory/vecsim_malloc.cpp index e51a46abb..4ab151329 100644 --- a/src/VecSim/memory/vecsim_malloc.cpp +++ b/src/VecSim/memory/vecsim_malloc.cpp @@ -34,12 +34,19 @@ void VecSimAllocator::setMemoryFunctions(VecSimMemoryFunctions memFunctions) { } void *VecSimAllocator::allocate(size_t size) { + if (size > (SIZE_MAX >> 1) - allocation_header_size) { + if (throwOnFailure) + throw std::bad_alloc(); + return nullptr; + } auto ptr = static_cast(vecsim_malloc(size + allocation_header_size)); if (ptr) { this->allocated += size + allocation_header_size; *ptr = {size, false}; return ptr + 1; } + if (throwOnFailure) + throw std::bad_alloc(); return nullptr; } @@ -48,6 +55,11 @@ void *VecSimAllocator::allocate_aligned(size_t size, unsigned char alignment) { return allocate(size); } + if (size > (SIZE_MAX >> 1) - allocation_header_size - alignment) { + if (throwOnFailure) + throw std::bad_alloc(); + return nullptr; + } size += alignment; // Add enough space for alignment. auto ptr = static_cast(vecsim_malloc(size + allocation_header_size)); if (ptr) { @@ -61,6 +73,8 @@ void *VecSimAllocator::allocate_aligned(size_t size, unsigned char alignment) { // Return the aligned pointer. return ptr + allocation_header_size + offset; } + if (throwOnFailure) + throw std::bad_alloc(); return nullptr; } @@ -102,6 +116,11 @@ void VecSimAllocator::free_allocation(void *p) { } void *VecSimAllocator::callocate(size_t size) { + if (size > (SIZE_MAX >> 1) - allocation_header_size) { + if (throwOnFailure) + throw std::bad_alloc(); + return nullptr; + } size_t *ptr = (size_t *)vecsim_calloc(1, size + allocation_header_size); if (ptr) { @@ -109,6 +128,8 @@ void *VecSimAllocator::callocate(size_t size) { *ptr = size; return ptr + 1; } + if (throwOnFailure) + throw std::bad_alloc(); return nullptr; } @@ -123,9 +144,17 @@ std::unique_ptr VecSimAllocator::allocate_unique return {ptr, Deleter(*this)}; } -void *VecSimAllocator::operator new(size_t size) { return vecsim_malloc(size); } +void *VecSimAllocator::operator new(size_t size) { + if (void *ptr = vecsim_malloc(size)) + return ptr; + throw std::bad_alloc(); +} -void *VecSimAllocator::operator new[](size_t size) { return vecsim_malloc(size); } +void *VecSimAllocator::operator new[](size_t size) { + if (void *ptr = vecsim_malloc(size)) + return ptr; + throw std::bad_alloc(); +} void VecSimAllocator::operator delete(void *p, size_t size) { vecsim_free(p); } void VecSimAllocator::operator delete[](void *p, size_t size) { vecsim_free(p); } diff --git a/src/VecSim/memory/vecsim_malloc.h b/src/VecSim/memory/vecsim_malloc.h index 5dd1bc080..d30d24655 100644 --- a/src/VecSim/memory/vecsim_malloc.h +++ b/src/VecSim/memory/vecsim_malloc.h @@ -12,6 +12,8 @@ #include #include #include +#include +#include struct VecSimAllocator { // Allow global vecsim memory functions to access this class. @@ -22,6 +24,7 @@ struct VecSimAllocator { private: std::atomic_uint64_t allocated; + bool throwOnFailure = false; // Static member that indicates each allocation additional size. static size_t allocation_header_size; @@ -33,6 +36,8 @@ struct VecSimAllocator { public: static std::shared_ptr newVecsimAllocator(); + // Set during factory initialization, before the allocator is shared with index components. + void setThrowOnAllocationFailure() { throwOnFailure = true; } void *allocate(size_t size); void *allocate_aligned(size_t size, unsigned char alignment); void *callocate(size_t size); @@ -145,7 +150,12 @@ struct VecsimSTLAllocator { return *this; } - T *allocate(size_t size) { return (T *)this->vecsim_allocator->allocate(size * sizeof(T)); } + T *allocate(size_t size) { + if (size > std::numeric_limits::max() / sizeof(T)) { + throw std::bad_array_new_length(); + } + return (T *)this->vecsim_allocator->allocate(size * sizeof(T)); + } void deallocate(T *ptr, size_t size) { this->vecsim_allocator->deallocate(ptr, size * sizeof(T)); diff --git a/src/VecSim/query_results.cpp b/src/VecSim/query_results.cpp index 845a980a7..a60562502 100644 --- a/src/VecSim/query_results.cpp +++ b/src/VecSim/query_results.cpp @@ -67,8 +67,9 @@ extern "C" void VecSimQueryReply_IteratorReset(VecSimQueryReply_Iterator *iterat /********************** batch iterator API ***************************/ VecSimQueryReply *VecSimBatchIterator_Next(VecSimBatchIterator *iterator, size_t n_results, VecSimQueryReply_Order order) { - assert((order == BY_ID || order == BY_SCORE) && - "Possible order values are only 'BY_ID' or 'BY_SCORE'"); + if (order != BY_ID && order != BY_SCORE) { + return nullptr; + } return iterator->getNextResults(n_results, order); } diff --git a/src/VecSim/query_results.h b/src/VecSim/query_results.h index be5f630a3..b1de66791 100644 --- a/src/VecSim/query_results.h +++ b/src/VecSim/query_results.h @@ -117,6 +117,8 @@ typedef struct VecSimBatchIterator VecSimBatchIterator; * @param order enum - determine the returned results order (by id or by score). * @return List of (at most) new n_results vectors which are the "nearest neighbours" to the * underline query vector in the iterator. + * Returns NULL for an unsupported order. A PiPNN iterator also returns NULL on allocation + * failure; the iterator remains valid for retry. */ VecSimQueryReply *VecSimBatchIterator_Next(VecSimBatchIterator *iterator, size_t n_results, VecSimQueryReply_Order order); diff --git a/src/VecSim/utils/vec_utils.cpp b/src/VecSim/utils/vec_utils.cpp index bb3e23cde..9089dbe79 100644 --- a/src/VecSim/utils/vec_utils.cpp +++ b/src/VecSim/utils/vec_utils.cpp @@ -24,6 +24,7 @@ const char *VecSimCommonStrings::FLAT_STRING = "FLAT"; const char *VecSimCommonStrings::HNSW_STRING = "HNSW"; const char *VecSimCommonStrings::TIERED_STRING = "TIERED"; const char *VecSimCommonStrings::SVS_STRING = "SVS"; +const char *VecSimCommonStrings::PIPNN_STRING = "PIPNN"; const char *VecSimCommonStrings::TYPE_STRING = "TYPE"; const char *VecSimCommonStrings::FLOAT32_STRING = "FLOAT32"; @@ -198,6 +199,8 @@ const char *VecSimAlgo_ToString(VecSimAlgo vecsimAlgo) { return VecSimCommonStrings::TIERED_STRING; case VecSimAlgo_SVS: return VecSimCommonStrings::SVS_STRING; + case VecSimAlgo_PIPNN: + return VecSimCommonStrings::PIPNN_STRING; } return NULL; } diff --git a/src/VecSim/utils/vec_utils.h b/src/VecSim/utils/vec_utils.h index 25a35208e..543b15ddb 100644 --- a/src/VecSim/utils/vec_utils.h +++ b/src/VecSim/utils/vec_utils.h @@ -24,6 +24,7 @@ struct VecSimCommonStrings { static const char *HNSW_STRING; static const char *TIERED_STRING; static const char *SVS_STRING; + static const char *PIPNN_STRING; static const char *TYPE_STRING; static const char *FLOAT32_STRING; diff --git a/src/VecSim/vec_sim.cpp b/src/VecSim/vec_sim.cpp index a9aeb6b7a..1f186fd69 100644 --- a/src/VecSim/vec_sim.cpp +++ b/src/VecSim/vec_sim.cpp @@ -16,9 +16,12 @@ #include "VecSim/vec_sim_adhoc_bf_ctx.h" #include "VecSim/types/bfloat16.h" #include "VecSim/algorithms/svs/svs_utils.h" +#include "VecSim/algorithms/pipnn/pipnn.h" #include #include #include +#include +#include #include "memory.h" // Ensure labelType is compatible with size_t for C API boundary. @@ -75,10 +78,10 @@ static VecSimResolveCode _ResolveParams_EFRuntime(VecSimAlgo index_type, VecSimR VecsimQueryType query_type, const char **err_msg) { long long num_val; - // EF_RUNTIME is a valid parameter only in HNSW algorithm. - if (index_type != VecSimAlgo_HNSWLIB) { + // EF_RUNTIME controls the graph search breadth. + if (index_type != VecSimAlgo_HNSWLIB && index_type != VecSimAlgo_PIPNN) { return SetResolveErr(err_msg, VecSimParamResolverErr_UnknownParam, - "%s is only valid for HNSW indexes", + "%s is only valid for HNSW and PiPNN indexes", VecSimCommonStrings::HNSW_EF_RUNTIME_STRING); } // EF_RUNTIME is invalid for range query @@ -87,7 +90,9 @@ static VecSimResolveCode _ResolveParams_EFRuntime(VecSimAlgo index_type, VecSimR "%s is not a valid parameter for range queries", VecSimCommonStrings::HNSW_EF_RUNTIME_STRING); } - if (qparams->hnswRuntimeParams.efRuntime != 0) { + auto &ef_runtime = index_type == VecSimAlgo_PIPNN ? qparams->pipnnRuntimeParams.efRuntime + : qparams->hnswRuntimeParams.efRuntime; + if (ef_runtime != 0) { return SetResolveErr(err_msg, VecSimParamResolverErr_AlreadySet, "%s was specified more than once", VecSimCommonStrings::HNSW_EF_RUNTIME_STRING); @@ -98,7 +103,7 @@ static VecSimResolveCode _ResolveParams_EFRuntime(VecSimAlgo index_type, VecSimR VecSimCommonStrings::HNSW_EF_RUNTIME_STRING); } - qparams->hnswRuntimeParams.efRuntime = (size_t)num_val; + ef_runtime = (size_t)num_val; return VecSimParamResolver_OK; } @@ -300,29 +305,71 @@ extern "C" size_t VecSimIndex_EstimateInitialSize(const VecSimParams *params) { } extern "C" int VecSimIndex_AddVector(VecSimIndex *index, const void *blob, size_t label) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->addVector(blob, label); + } catch (...) { + return -1; + } + } return index->addVector(blob, label); } extern "C" int VecSimIndex_DeleteVector(VecSimIndex *index, size_t label) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->deleteVector(label); + } catch (...) { + return -1; + } + } return index->deleteVector(label); } extern "C" VecSimRelabelCode VecSimIndex_RelabelVector(VecSimIndex *index, size_t old_label, size_t new_label) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->relabelVector(old_label, new_label); + } catch (...) { + return VecSimRelabel_Failed; + } + } return index->relabelVector(old_label, new_label); } extern "C" VecSimUpdateCode VecSimIndex_UpdateVectors(VecSimIndex *index, size_t label, const void *new_blobs, size_t n) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->updateVectors(label, new_blobs, n); + } catch (...) { + return VecSimUpdate_Failed; + } + } return index->updateVectors(label, new_blobs, n); } extern "C" double VecSimIndex_GetDistanceFrom_Unsafe(VecSimIndex *index, size_t label, const void *blob) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->getDistanceFrom_Unsafe(label, blob); + } catch (...) { + return std::numeric_limits::quiet_NaN(); + } + } return index->getDistanceFrom_Unsafe(label, blob); } extern "C" size_t VecSimIndex_EstimateElementSize(const VecSimParams *params) { + if (!params || params->algo == VecSimAlgo_PIPNN) { + try { + return VecSimFactory::EstimateElementSize(params); + } catch (...) { + return SIZE_MAX; + } + } return VecSimFactory::EstimateElementSize(params); } @@ -428,10 +475,10 @@ extern "C" VecSimResolveCode VecSimIndex_ResolveParams(VecSimIndex *index, VecSi "%s is irrelevant for the '%s' hybrid policy", VecSimCommonStrings::BATCH_SIZE_STRING, VECSIM_POLICY_ADHOC_BF); } - // Also, 'ef_runtime' is meaning less in AD-HOC policy, since it doesn't involve search in HNSW - // graph. - if (qparams->searchMode == HYBRID_ADHOC_BF && index_type == VecSimAlgo_HNSWLIB && - qparams->hnswRuntimeParams.efRuntime > 0) { + // EF_RUNTIME is irrelevant to AD-HOC policy, which does not search the graph. + if (qparams->searchMode == HYBRID_ADHOC_BF && + ((index_type == VecSimAlgo_HNSWLIB && qparams->hnswRuntimeParams.efRuntime > 0) || + (index_type == VecSimAlgo_PIPNN && qparams->pipnnRuntimeParams.efRuntime > 0))) { return SetResolveErr(err_msg, VecSimParamResolverErr_InvalidPolicy_AdHoc_With_EfRuntime, "%s is irrelevant for the '%s' hybrid policy", VecSimCommonStrings::HNSW_EF_RUNTIME_STRING, VECSIM_POLICY_ADHOC_BF); @@ -445,6 +492,20 @@ extern "C" VecSimResolveCode VecSimIndex_ResolveParams(VecSimIndex *index, VecSi extern "C" VecSimQueryReply *VecSimIndex_TopKQuery(VecSimIndex *index, const void *queryBlob, size_t k, VecSimQueryParams *queryParams, VecSimQueryReply_Order order) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + if (order != BY_ID && order != BY_SCORE) { + return nullptr; + } + std::unique_ptr results(index->topKQuery(queryBlob, k, queryParams)); + if (results && order == BY_ID) { + sort_results_by_id(results.get()); + } + return results.release(); + } catch (...) { + return nullptr; + } + } assert((order == BY_ID || order == BY_SCORE) && "Possible order values are only 'BY_ID' or 'BY_SCORE'"); VecSimQueryReply *results; @@ -459,6 +520,16 @@ extern "C" VecSimQueryReply *VecSimIndex_TopKQuery(VecSimIndex *index, const voi extern "C" VecSimQueryReply *VecSimIndex_RangeQuery(VecSimIndex *index, const void *queryBlob, double radius, VecSimQueryParams *queryParams, VecSimQueryReply_Order order) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + if ((order != BY_ID && order != BY_SCORE) || radius < 0 || std::isnan(radius)) { + return nullptr; + } + return index->rangeQuery(queryBlob, radius, queryParams, order); + } catch (...) { + return nullptr; + } + } if (order != BY_ID && order != BY_SCORE) { throw std::runtime_error("Possible order values are only 'BY_ID' or 'BY_SCORE'"); } @@ -479,6 +550,21 @@ extern "C" VecSimIndexDebugInfo VecSimIndex_DebugInfo(VecSimIndex *index) { } extern "C" VecSimDebugInfoIterator *VecSimIndex_DebugInfoIterator(VecSimIndex *index) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + std::unique_ptr iterator(index->debugInfoIterator()); + if (!iterator) { + return nullptr; + } + iterator->addInfoField( + VecSim_InfoField{.fieldName = VecSimCommonStrings::SHARED_MEMORY_STRING, + .fieldType = INFOFIELD_UINT64, + .fieldValue = {FieldValue{.uintegerValue = VecSim_GetSharedMemory()}}}); + return iterator.release(); + } catch (...) { + return nullptr; + } + } VecSimDebugInfoIterator *infoIterator = index->debugInfoIterator(); // Append the process-wide shared memory total. This field is not emitted by // any algorithm's own debugInfoIterator(); it is injected here at the C API @@ -505,6 +591,13 @@ extern "C" VecSimIndexStatsInfo VecSimIndex_StatsInfo(VecSimIndex *index) { extern "C" VecSimBatchIterator *VecSimBatchIterator_New(VecSimIndex *index, const void *queryBlob, VecSimQueryParams *queryParams) { + if (index && index->basicInfo().algo == VecSimAlgo_PIPNN) { + try { + return index->newBatchIterator(queryBlob, queryParams); + } catch (...) { + return nullptr; + } + } return index->newBatchIterator(queryBlob, queryParams); } @@ -551,3 +644,62 @@ extern "C" void VecSimIndex_AdhocBfCtx_GetExactDistances(VecSimAdhocBfCtx *ctx, double *distances_out, size_t count) { ctx->getExactDistances(labels, distances_out, count); } + +namespace { +const PiPNNIndex *AsPiPNNIndex(const VecSimIndex *index) { + if (!index) { + return nullptr; + } + const auto info = index->basicInfo(); + if (info.algo != VecSimAlgo_PIPNN || info.isTiered) { + return nullptr; + } + return dynamic_cast(index); +} +} // namespace + +extern "C" int VecSimIndex_BuildGraph(VecSimIndex *index) { + try { + auto *pipnn = const_cast(AsPiPNNIndex(index)); + if (!pipnn) { + return 0; + } + pipnn->buildGraph(); + return 1; + } catch (...) { + return 0; + } +} + +extern "C" bool VecSimIndex_IsGraphReady(const VecSimIndex *index) { + try { + const auto *pipnn = AsPiPNNIndex(index); + return pipnn && pipnn->isGraphReady(); + } catch (...) { + return false; + } +} + +extern "C" int VecSimIndex_SavePiPNN(const VecSimIndex *index, const char *location) { + try { + const auto *pipnn = AsPiPNNIndex(index); + if (!pipnn || !location || !*location) { + return 0; + } + pipnn->saveIndex(location); + return 1; + } catch (...) { + return 0; + } +} + +extern "C" VecSimIndex *VecSimIndex_LoadPiPNN(const char *location, void *logCtx) { + try { + if (!location || !*location) { + return nullptr; + } + return PiPNNIndex::loadIndex(location, logCtx); + } catch (...) { + return nullptr; + } +} diff --git a/src/VecSim/vec_sim.h b/src/VecSim/vec_sim.h index b51442d04..9ad00795e 100644 --- a/src/VecSim/vec_sim.h +++ b/src/VecSim/vec_sim.h @@ -47,7 +47,7 @@ size_t VecSimIndex_EstimateInitialSize(const VecSimParams *params); * combined growth of both tiers or total index memory. * @param params index configurations (initial size, data type, dimension, metric, algorithm and the * algorithm-related params). - * @return The estimated single vector memory consumption, considering the parameters. + * @return Estimated single vector memory consumption; PiPNN returns SIZE_MAX for invalid params. */ size_t VecSimIndex_EstimateElementSize(const VecSimParams *params); @@ -57,13 +57,38 @@ size_t VecSimIndex_EstimateElementSize(const VecSimParams *params); */ void VecSimIndex_Free(VecSimIndex *index); +/** + * @brief Build the PiPNN graph for the current vectors. Returns 1 on success, 0 on failure. + * Only standalone PiPNN indexes are supported; tiered PiPNN is unsupported. + * PiPNN requires callers to serialize all operations, including queries, because inherited + * search-mode bookkeeping is mutable. Writes invalidate the graph; queries use exact search + * until a successful rebuild. Do not mutate an index while a batch iterator is live. + */ +int VecSimIndex_BuildGraph(VecSimIndex *index); + +/** @brief Return whether a standalone PiPNN index has a graph for its current vectors. + * Returns false for NULL or a different backend. Requires synchronization against writes/builds. + */ +bool VecSimIndex_IsGraphReady(const VecSimIndex *index); + +/** @brief Save a standalone PiPNN index in its native format. Returns 1 on success, 0 on failure. + * The caller must synchronize against writes and graph builds for the duration of the save. + */ +int VecSimIndex_SavePiPNN(const VecSimIndex *index, const char *location); + +/** @brief Load a native PiPNN index. Returns NULL on invalid input or any load failure. + * The caller owns the returned index and releases it with VecSimIndex_Free. + */ +VecSimIndex *VecSimIndex_LoadPiPNN(const char *location, void *logCtx); + /** * @brief Add a vector to an index. * @param index the index to which the vector is added. * @param blob binary representation of the vector. Blob size should match the index data type and * dimension. * @param label the label of the added vector - * @return the number of new vectors inserted (1 for new insertion, 0 for override). + * @return The number of new vectors inserted (1 for new insertion, 0 for override). + * PiPNN returns -1 if the input is invalid or storage fails. */ int VecSimIndex_AddVector(VecSimIndex *index, const void *blob, size_t label); @@ -71,7 +96,8 @@ int VecSimIndex_AddVector(VecSimIndex *index, const void *blob, size_t label); * @brief Remove a vector from an index. * @param index the index from which the vector is removed. * @param label the label of the removed vector - * @return the number of vectors removed (0 if the label was not found) + * @return The number of vectors removed (0 if the label was not found). + * PiPNN returns -1 if deletion fails. */ int VecSimIndex_DeleteVector(VecSimIndex *index, size_t label); @@ -92,6 +118,7 @@ int VecSimIndex_DeleteVector(VecSimIndex *index, size_t label); * @param old_label the label currently holding the vector(s). * @param new_label the label to move them to. * @return `VecSimRelabel_OK` if the label was moved, otherwise the reason it was not. + * PiPNN reports `VecSimRelabel_Failed` if the operation fails. */ VecSimRelabelCode VecSimIndex_RelabelVector(VecSimIndex *index, size_t old_label, size_t new_label); @@ -136,6 +163,7 @@ VecSimUpdateCode VecSimIndex_UpdateVectors(VecSimIndex *index, size_t label, con * absent. * @note Quantized indexes allocate and preprocess a temporary query blob on each call. */ +// PiPNN returns NaN for invalid FP32 input or an operation failure. double VecSimIndex_GetDistanceFrom_Unsafe(VecSimIndex *index, size_t label, const void *blob); /** @@ -196,6 +224,8 @@ VecSimResolveCode VecSimIndex_ResolveParams(VecSimIndex *index, VecSimRawParam * * (which is the distance according to the index metric) of every result through * VecSimQueryReply_Iterator. */ +// PiPNN returns NULL for invalid FP32 input, invalid order, or allocation failure. +// Check for NULL before calling VecSimQueryReply accessors. VecSimQueryReply *VecSimIndex_TopKQuery(VecSimIndex *index, const void *queryBlob, size_t k, VecSimQueryParams *queryParams, VecSimQueryReply_Order); @@ -212,6 +242,8 @@ VecSimQueryReply *VecSimIndex_TopKQuery(VecSimIndex *index, const void *queryBlo * (which is the distance according to the index metric) of every result through * VecSimQueryReply_Iterator. */ +// PiPNN returns NULL for invalid FP32 input, invalid radius/order, or allocation failure. +// Check for NULL before calling VecSimQueryReply accessors. VecSimQueryReply *VecSimIndex_RangeQuery(VecSimIndex *index, const void *queryBlob, double radius, VecSimQueryParams *queryParams, VecSimQueryReply_Order); /** @@ -240,6 +272,7 @@ VecSimIndexStatsInfo VecSimIndex_StatsInfo(VecSimIndex *index); * * @param index this index to return its info. * @return VecSimDebugInfoIterator* An iterable containing the index general and specific meta-data. + * PiPNN returns NULL on allocation failure; check before using iterator accessors. */ VecSimDebugInfoIterator *VecSimIndex_DebugInfoIterator(VecSimIndex *index); @@ -253,6 +286,7 @@ VecSimDebugInfoIterator *VecSimIndex_DebugInfoIterator(VecSimIndex *index); * @param queryParams run time params for the search, which are algorithm-specific. * @return Fresh batch iterator */ +// PiPNN returns NULL for invalid FP32 input or allocation failure. VecSimBatchIterator *VecSimBatchIterator_New(VecSimIndex *index, const void *queryBlob, VecSimQueryParams *queryParams); diff --git a/src/VecSim/vec_sim_common.h b/src/VecSim/vec_sim_common.h index 1d7cf7aa3..dc4f9ed8f 100644 --- a/src/VecSim/vec_sim_common.h +++ b/src/VecSim/vec_sim_common.h @@ -78,7 +78,13 @@ typedef enum { } VecSimQuantType; // Algorithm type/library. -typedef enum { VecSimAlgo_BF, VecSimAlgo_HNSWLIB, VecSimAlgo_TIERED, VecSimAlgo_SVS } VecSimAlgo; +typedef enum { + VecSimAlgo_BF, + VecSimAlgo_HNSWLIB, + VecSimAlgo_TIERED, + VecSimAlgo_SVS, + VecSimAlgo_PIPNN +} VecSimAlgo; typedef enum { VecSimOption_AUTO = 0, @@ -141,7 +147,8 @@ typedef enum { VecSimRelabel_OldLabelMissing, // `old_label` is not in the index VecSimRelabel_NewLabelTaken, // `new_label` is already in the index VecSimRelabel_SameLabel, // `old_label` and `new_label` are equal - VecSimRelabel_Unsupported // this index type does not implement relabeling + VecSimRelabel_Unsupported, // this index type does not implement relabeling + VecSimRelabel_Failed // the index could not complete relabeling } VecSimRelabelCode; // Outcome of `updateVectors`. The refusals are told apart because a caller acts on them @@ -206,6 +213,27 @@ typedef struct { size_t blockSize; } BFParams; +/** PiPNN is an opt-in, in-memory backend for FLOAT32/L2 single-value indexes. + * Zero-valued tuning fields select the defaults documented below. + * initialCapacity is deprecated and does not reserve storage, as in FLAT. + */ +typedef struct { + VecSimType type; + size_t dim; + VecSimMetric metric; + size_t initialCapacity; + size_t blockSize; // Default: DEFAULT_BLOCK_SIZE. + bool multi; // Must be false. + size_t maxDegree; // At least 2; default: 32. + size_t leafSize; // At least 2; default: 128. + size_t fanout; // Between 2 and 32; default: 2. + size_t hashBits; // At most 32; default: 12. + size_t candidateLimit; // At least maxDegree; default: 80. + double alpha; // Finite and at least 1; default: 1.1. + size_t efRuntime; // Default: 100. + uint64_t seed; // Default: 100. +} PiPNNParams; + typedef enum { VecSimSvsQuant_NONE = 0, // No quantization. VecSimSvsQuant_Scalar = 1, // 8-bit scalar quantization @@ -290,6 +318,7 @@ typedef union { BFParams bfParams; TieredIndexParams tieredParams; SVSParams svsParams; + PiPNNParams pipnnParams; } AlgoParams; struct VecSimParams { @@ -332,6 +361,10 @@ typedef struct { double epsilon; // Epsilon parameter for HNSW graph accuracy/latency for range search. } HNSWRuntimeParams; +typedef struct { + size_t efRuntime; // Zero selects the index default. +} PiPNNRuntimeParams; + typedef struct { size_t efRuntime; // EF parameter for HNSW graph accuracy/latency for search. double epsilon; // Epsilon parameter for HNSW graph accuracy/latency for range search. @@ -377,6 +410,7 @@ typedef struct { HNSWRuntimeParams hnswRuntimeParams; HNSWDiskRuntimeParams hnswDiskRuntimeParams; SVSRuntimeParams svsRuntimeParams; + PiPNNRuntimeParams pipnnRuntimeParams; }; size_t batchSize; VecSearchMode searchMode; diff --git a/src/VecSim/vec_sim_index.h b/src/VecSim/vec_sim_index.h index fbdab3a67..9b0f73881 100644 --- a/src/VecSim/vec_sim_index.h +++ b/src/VecSim/vec_sim_index.h @@ -151,8 +151,16 @@ struct VecSimIndexAbstract : public VecSimIndexInterface { // so only the first vector in every block is guaranteed to be aligned. Aligning every // vector would require padding `storedDataSize` up to a multiple of the alignment, which // is a separate change. - this->vectors = new (this->allocator) DataBlocksContainer( - this->blockSize, this->storedDataSize, this->allocator, this->getStorageAlignment()); + try { + this->vectors = new (this->allocator) DataBlocksContainer( + this->blockSize, this->storedDataSize, this->allocator, this->getStorageAlignment()); + } catch (...) { + // A failed constructor does not run this class's destructor. Components have + // already been adopted, so release them before propagating the storage failure. + delete indexCalculator; + delete preprocessors; + throw; + } } /** diff --git a/src/VecSim/vec_sim_tiered_index.h b/src/VecSim/vec_sim_tiered_index.h index 9566d5ea5..41f06a663 100644 --- a/src/VecSim/vec_sim_tiered_index.h +++ b/src/VecSim/vec_sim_tiered_index.h @@ -716,6 +716,7 @@ VecSimIndexDebugInfo VecSimTieredIndex::debugInfo() const { info.tieredInfo.backendInfo.svsInfo = backendInfo.svsInfo; break; case VecSimAlgo_BF: + case VecSimAlgo_PIPNN: case VecSimAlgo_TIERED: assert(false && "Invalid backend algorithm"); } diff --git a/tests/benchmark/CMakeLists.txt b/tests/benchmark/CMakeLists.txt index 75e3bb50d..3d338a963 100644 --- a/tests/benchmark/CMakeLists.txt +++ b/tests/benchmark/CMakeLists.txt @@ -5,6 +5,10 @@ project(VectorSimilarity_Benchmark) set(CMAKE_CXX_STANDARD 20) +# Explicitly requested comparison runner; it needs no benchmark/gtest runtime. +add_executable(pipnn_comparison EXCLUDE_FROM_ALL pipnn_comparison.cpp) +target_link_libraries(pipnn_comparison VectorSimilarity) + set(CMAKE_CXX_FLAGS_DEBUG "-g") set(CMAKE_CXX_FLAGS_RELEASE "-O3") include_directories(../) diff --git a/tests/benchmark/pipnn_comparison.cpp b/tests/benchmark/pipnn_comparison.cpp new file mode 100644 index 000000000..ee9f012b3 --- /dev/null +++ b/tests/benchmark/pipnn_comparison.cpp @@ -0,0 +1,281 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ +#include "VecSim/vec_sim.h" +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { +using Clock = std::chrono::steady_clock; +using IndexPtr = std::unique_ptr; +using ReplyPtr = std::unique_ptr; +using IteratorPtr = + std::unique_ptr; +constexpr size_t k = 10; + +// fbin/ibin have two little-endian uint32 dimensions followed by a dense row-major payload. +static_assert(std::endian::native == std::endian::little); +static_assert(sizeof(float) == 4 && std::numeric_limits::is_iec559); + +template +struct Matrix { + uint32_t rows = 0; + uint32_t dim = 0; + std::vector values; + const T *row(size_t i) const { return values.data() + i * dim; } +}; + +template +Matrix readMatrix(const char *path) { + std::ifstream input(path, std::ios::binary | std::ios::ate); + if (!input) { + throw std::runtime_error(std::string("Cannot open matrix: ") + path); + } + const std::streamoff file_size = input.tellg(); + input.seekg(0); + Matrix matrix; + input.read(reinterpret_cast(&matrix.rows), sizeof(matrix.rows)); + input.read(reinterpret_cast(&matrix.dim), sizeof(matrix.dim)); + const uint64_t count = uint64_t(matrix.rows) * matrix.dim; + if (!input || !matrix.rows || !matrix.dim || file_size < 8 || + count > (std::numeric_limits::max() - 8) / sizeof(T) || + count > matrix.values.max_size() || + count > uint64_t(std::numeric_limits::max()) / sizeof(T) || + uint64_t(file_size) != 8 + count * sizeof(T)) { + throw std::runtime_error(std::string("Invalid matrix header or file size: ") + path); + } + matrix.values.resize(static_cast(count)); + input.read(reinterpret_cast(matrix.values.data()), + static_cast(count * sizeof(T))); + if (!input) { + throw std::runtime_error(std::string("Short matrix read: ") + path); + } + if constexpr (std::is_same_v) { + // Inspect bits so validation also works when the library is built with fast-math. + for (float value : matrix.values) { + if ((std::bit_cast(value) & 0x7f800000U) == 0x7f800000U) { + throw std::runtime_error(std::string("Non-finite FP32 matrix component: ") + path); + } + } + } + return matrix; +} + +size_t positiveInteger(std::string_view value) { + size_t result = 0; + const auto parsed = std::from_chars(value.data(), value.data() + value.size(), result); + if (value.empty() || parsed.ec != std::errc{} || parsed.ptr != value.data() + value.size() || + !result) { + throw std::runtime_error("Expected a positive integer: " + std::string(value)); + } + return result; +} + +std::vector parseEfList(std::string_view value) { + std::vector result; + for (;;) { + const size_t comma = value.find(','); + const size_t ef = positiveInteger(value.substr(0, comma)); + if (ef < k || ef > std::numeric_limits::max()) { + throw std::runtime_error("Each ef must be between 10 and UINT32_MAX"); + } + result.push_back(ef); + if (comma == std::string_view::npos) { + return result; + } + value.remove_prefix(comma + 1); + } +} + +uint64_t peakRssBytes() { + rusage usage{}; + if (getrusage(RUSAGE_SELF, &usage) != 0 || usage.ru_maxrss < 0) { + throw std::runtime_error("getrusage failed"); + } +#ifdef __APPLE__ + return static_cast(usage.ru_maxrss); +#else + return static_cast(usage.ru_maxrss) * 1024; +#endif +} + +double seconds(Clock::time_point start, Clock::time_point end) { + return std::chrono::duration(end - start).count(); +} + +ReplyPtr query(VecSimIndex *index, const float *blob, VecSimQueryParams ¶ms) { + ReplyPtr reply(VecSimIndex_TopKQuery(index, blob, k, ¶ms, BY_SCORE), VecSimQueryReply_Free); + if (!reply || VecSimQueryReply_GetCode(reply.get()) != VecSim_QueryReply_OK || + VecSimQueryReply_Len(reply.get()) != k) { + throw std::runtime_error("Query failed or returned fewer than 10 results"); + } + return reply; +} + +size_t countHits(VecSimQueryReply *reply, const uint32_t *truth, size_t base_count) { + IteratorPtr iterator(VecSimQueryReply_GetIterator(reply), VecSimQueryReply_IteratorFree); + if (!iterator) { + throw std::runtime_error("Cannot allocate result iterator"); + } + std::array seen{}; + size_t results = 0, hits = 0; + while (VecSimQueryReply_IteratorHasNext(iterator.get())) { + const int64_t id = VecSimQueryResult_GetId(VecSimQueryReply_IteratorNext(iterator.get())); + if (id < 0 || uint64_t(id) >= base_count || results == k || + std::find(seen.begin(), seen.begin() + results, uint32_t(id)) != + seen.begin() + results) { + throw std::runtime_error("Out-of-range or duplicate result ID"); + } + seen[results++] = static_cast(id); + hits += std::find(truth, truth + k, uint32_t(id)) != truth + k; + } + if (results != k) { + throw std::runtime_error("Result iterator length mismatch"); + } + return hits; +} +} // namespace + +int main(int argc, char **argv) { + try { + if (argc != 6 && argc != 7) { + throw std::runtime_error( + "Usage: pipnn_comparison base.fbin query.fbin truth.ibin pipnn|hnsw " + "ef[,ef...] [query_limit]"); + } + const std::string backend = argv[4]; + if (backend != "pipnn" && backend != "hnsw") { + throw std::runtime_error("Backend must be pipnn or hnsw"); + } + const auto ef_values = parseEfList(argv[5]); + const size_t query_limit = argc == 7 ? positiveInteger(argv[6]) : SIZE_MAX; + auto base = readMatrix(argv[1]); + if (base.rows < k) { + throw std::runtime_error("Base must contain at least 10 vectors"); + } + VecSimParams params{}; + if (backend == "pipnn") { + params.algo = VecSimAlgo_PIPNN; + params.algoParams.pipnnParams = {.type = VecSimType_FLOAT32, + .dim = base.dim, + .metric = VecSimMetric_L2, + .maxDegree = 32, + .seed = 100}; + } else { + params.algo = VecSimAlgo_HNSWLIB; + params.algoParams.hnswParams = {.type = VecSimType_FLOAT32, + .dim = base.dim, + .metric = VecSimMetric_L2, + .M = 16, + .efConstruction = 100}; + } + const auto build_start = Clock::now(); + IndexPtr index(VecSimIndex_New(¶ms), VecSimIndex_Free); + if (!index) { + throw std::runtime_error("Index creation failed"); + } + for (size_t i = 0; i < base.rows; ++i) { + if (VecSimIndex_AddVector(index.get(), base.row(i), i) != 1) { + throw std::runtime_error("Vector insertion failed at row " + std::to_string(i)); + } + } + const auto insert_end = Clock::now(); + if (backend == "pipnn" && + (!VecSimIndex_BuildGraph(index.get()) || !VecSimIndex_IsGraphReady(index.get()))) { + throw std::runtime_error("PiPNN graph build failed"); + } + const auto build_end = Clock::now(); + const auto index_bytes = VecSimIndex_StatsInfo(index.get()).memory; + const auto build_peak_rss = peakRssBytes(); + const size_t base_count = base.rows; + const size_t dimension = base.dim; + std::vector().swap(base.values); + + auto queries = readMatrix(argv[2]); + const auto truth = readMatrix(argv[3]); + if (queries.dim != dimension || truth.rows != queries.rows || truth.dim < k) { + throw std::runtime_error("Query/truth dimensions do not match the base or each other"); + } + const size_t query_count = std::min(queries.rows, query_limit); + for (size_t i = 0; i < query_count; ++i) { + const auto *row = truth.row(i); + for (size_t j = 0; j < k; ++j) { + if (row[j] >= base_count || std::find(row, row + j, row[j]) != row + j) { + throw std::runtime_error("Out-of-range or duplicate ground-truth ID"); + } + } + } + std::cout << std::setprecision(9) << "backend=" << backend << " n=" << base_count + << " dim=" << dimension << " queries=" << query_count + << " metric=L2 type=FLOAT32 seed=100"; + if (backend == "pipnn") { + std::cout << " max_degree=32 leaf_size=128 fanout=2 hash_bits=12 candidate_limit=80" + " alpha=1.1"; + } else { + std::cout << " M=16 ef_construction=100"; + } + // Dataset I/O is excluded. Build RSS includes the base input matrix and process overhead. + std::cout << " build_seconds=" << seconds(build_start, build_end) + << " creation_insert_seconds=" << seconds(build_start, insert_end) + << " graph_build_seconds=" << seconds(insert_end, build_end) + << " index_bytes=" << index_bytes << " build_peak_rss_bytes=" << build_peak_rss + << '\n'; + for (size_t ef : ef_values) { + VecSimQueryParams query_params{}; + if (backend == "pipnn") { + query_params.pipnnRuntimeParams.efRuntime = ef; + } else { + query_params.hnswRuntimeParams.efRuntime = ef; + } + for (size_t i = 0; i < std::min(query_count, 100); ++i) { + query(index.get(), queries.row(i), query_params); + } + std::vector latency_us; + latency_us.reserve(query_count); + size_t hits = 0; + for (size_t i = 0; i < query_count; ++i) { + const auto start = Clock::now(); + auto reply = query(index.get(), queries.row(i), query_params); + const auto end = Clock::now(); + latency_us.push_back( + std::chrono::duration(end - start).count()); + hits += countHits(reply.get(), truth.row(i), base_count); + } + const double query_us = std::accumulate(latency_us.begin(), latency_us.end(), 0.0); + std::sort(latency_us.begin(), latency_us.end()); + // Nearest-rank percentiles: ceil(p * n) - 1, without floating-point rounding. + const size_t p50 = query_count / 2 + query_count % 2 - 1; + const size_t p95 = query_count - query_count / 20 - 1; + std::cout << "backend=" << backend << " ef=" << ef + << " recall10=" << double(hits) / (double(k) * query_count) + << " p50_us=" << latency_us[p50] << " p95_us=" << latency_us[p95] + << " serial_query_qps=" << 1e6 * query_count / query_us + << " peak_rss_bytes=" << peakRssBytes() << '\n'; + } + return 0; + } catch (const std::exception &error) { + std::cerr << "pipnn_comparison: " << error.what() << '\n'; + return 1; + } +} diff --git a/tests/unit/CMakeLists.txt b/tests/unit/CMakeLists.txt index 96226f2ff..770c017fd 100644 --- a/tests/unit/CMakeLists.txt +++ b/tests/unit/CMakeLists.txt @@ -38,6 +38,7 @@ add_executable(test_hnsw ../utils/test_main_with_timeout.cpp ../utils/mock_threa add_executable(test_hnsw_sq4 ../utils/test_main_with_timeout.cpp ../utils/mock_thread_pool.cpp test_hnsw_sq4.cpp test_sq4_distances.cpp test_sq4_components.cpp unit_test_utils.cpp) add_executable(test_hnsw_sq8 ../utils/test_main_with_timeout.cpp ../utils/mock_thread_pool.cpp test_hnsw_sq8.cpp unit_test_utils.cpp) add_executable(test_bruteforce ../utils/test_main_with_timeout.cpp test_bruteforce.cpp test_bruteforce_multi.cpp ../utils/mock_thread_pool.cpp unit_test_utils.cpp) +add_executable(test_pipnn ../utils/test_main_with_timeout.cpp test_pipnn.cpp ../utils/mock_thread_pool.cpp unit_test_utils.cpp) add_executable(test_allocator ../utils/test_main_with_timeout.cpp test_allocator.cpp ../utils/mock_thread_pool.cpp unit_test_utils.cpp) add_executable(test_spaces ../utils/test_main_with_timeout.cpp test_spaces.cpp) add_executable(test_types ../utils/test_main_with_timeout.cpp test_types.cpp) @@ -54,6 +55,7 @@ target_link_libraries(test_hnsw PUBLIC gtest VectorSimilarity) target_link_libraries(test_hnsw_sq8 PUBLIC gtest VectorSimilarity) target_link_libraries(test_hnsw_sq4 PUBLIC gtest VectorSimilarity) target_link_libraries(test_bruteforce PUBLIC gtest VectorSimilarity) +target_link_libraries(test_pipnn PUBLIC gtest VectorSimilarity) target_link_libraries(test_allocator PUBLIC gtest VectorSimilarity) target_link_libraries(test_spaces PUBLIC gtest VectorSimilarity) target_link_libraries(test_common PUBLIC gtest VectorSimilarity) @@ -76,6 +78,7 @@ gtest_discover_tests(test_hnsw) gtest_discover_tests(test_hnsw_sq8) gtest_discover_tests(test_hnsw_sq4) gtest_discover_tests(test_bruteforce) +gtest_discover_tests(test_pipnn) gtest_discover_tests(test_allocator) gtest_discover_tests(test_spaces) gtest_discover_tests(test_common) diff --git a/tests/unit/test_bruteforce.cpp b/tests/unit/test_bruteforce.cpp index c642adfa9..09d62ce5a 100644 --- a/tests/unit/test_bruteforce.cpp +++ b/tests/unit/test_bruteforce.cpp @@ -1280,7 +1280,7 @@ TYPED_TEST(BruteForceTest, brute_force_resolve_params) { ASSERT_EQ(VecSimIndex_ResolveParams(index, rparams.data(), rparams.size(), &qparams, query_type, &err_msg), VecSimParamResolverErr_UnknownParam); - ASSERT_STREQ(err_msg, "EF_RUNTIME is only valid for HNSW indexes"); + ASSERT_STREQ(err_msg, "EF_RUNTIME is only valid for HNSW and PiPNN indexes"); } /** Testing with hybrid query params - cases which are only relevant for BF flat index. **/ // Sending only "batch_size" param is valid. diff --git a/tests/unit/test_pipnn.cpp b/tests/unit/test_pipnn.cpp new file mode 100644 index 000000000..b2d8479ea --- /dev/null +++ b/tests/unit/test_pipnn.cpp @@ -0,0 +1,1107 @@ +/* + * Copyright (c) 2006-Present, Redis Ltd. + * All rights reserved. + * + * Licensed under your choice of the Redis Source Available License 2.0 + * (RSALv2); or (b) the Server Side Public License v1 (SSPLv1); or (c) the + * GNU Affero General Public License v3 (AGPLv3). + */ + +#include "VecSim/vec_sim.h" +#include "unit_test_utils.h" + +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include +#include + +namespace { + +using IndexPtr = std::unique_ptr; +using ReplyPtr = std::unique_ptr; +using BatchPtr = std::unique_ptr; +using Point = std::array; + +VecSimParams pipnnParams(size_t beam = 256) { + VecSimParams params{}; + params.algo = VecSimAlgo_PIPNN; + auto &p = params.algoParams.pipnnParams; + p.type = VecSimType_FLOAT32; + p.dim = Point{}.size(); + p.metric = VecSimMetric_L2; + p.blockSize = 8; + p.maxDegree = 16; + p.leafSize = 8; + p.fanout = 4; + p.hashBits = 4; + p.candidateLimit = 128; + p.alpha = 1.2; + p.efRuntime = beam; + p.seed = 42; + return params; +} + +IndexPtr newIndex(const VecSimParams ¶ms) { + return IndexPtr(VecSimIndex_New(¶ms), VecSimIndex_Free); +} + +IndexPtr newExactIndex() { + BFParams bf{.type = VecSimType_FLOAT32, .dim = Point{}.size(), .metric = VecSimMetric_L2}; + return newIndex(CreateParams(bf)); +} + +std::vector randomPoints(size_t count) { + std::mt19937 generator(73); + std::uniform_real_distribution distribution(-10, 10); + std::vector points(count); + for (auto &point : points) { + for (auto &coordinate : point) { + coordinate = distribution(generator); + } + } + return points; +} + +std::vector> topK(VecSimIndex *index, const Point &query, size_t k) { + std::vector> results; + runTopKSearchTest(index, query.data(), k, [&](size_t label, double score, size_t) { + results.emplace_back(label, score); + }); + return results; +} + +void expectNearest(VecSimIndex *index, const Point &query, size_t expected_label) { + runTopKSearchTest(index, query.data(), 1, [&](size_t label, double score, size_t) { + EXPECT_EQ(label, expected_label); + EXPECT_DOUBLE_EQ(score, 0); + }); +} + +// Each file owns a unique path so persistence tests can run concurrently. +struct TemporaryFile { + std::string path; + + TemporaryFile() { + char name[] = "/tmp/vecsim-pipnn-test-XXXXXX"; + int descriptor = mkstemp(name); + if (descriptor >= 0) { + close(descriptor); + path = name; + } + } + ~TemporaryFile() { + if (!path.empty()) { + unlink(path.c_str()); + } + } +}; + +struct TimeoutGuard { + TimeoutGuard() { + VecSim_SetTimeoutCallbackFunction([](void *) { return 1; }); + } + ~TimeoutGuard() { + VecSim_SetTimeoutCallbackFunction([](void *) { return 0; }); + } +}; + +// These wrappers remain compatible with allocations made before and after an injection. +class FailNthAllocation { + static inline thread_local size_t failAt = 0; + static inline thread_local size_t calls = 0; + static inline thread_local bool failed = false; + static inline thread_local bool trackPointers = false; + static inline thread_local std::unordered_set livePointers; + + static bool shouldFail() { + if (++calls != failAt) { + return false; + } + failed = true; + return true; + } + static void *remember(void *pointer) { + if (trackPointers && pointer) { + livePointers.insert(reinterpret_cast(pointer)); + } + return pointer; + } + static void *allocate(size_t size) { + return shouldFail() ? nullptr : remember(std::malloc(size)); + } + static void *callocate(size_t count, size_t size) { + return shouldFail() ? nullptr : remember(std::calloc(count, size)); + } + static void *reallocate(void *pointer, size_t size) { + if (shouldFail()) { + return nullptr; + } + const auto old_address = reinterpret_cast(pointer); + void *resized = std::realloc(pointer, size); + if (trackPointers && (resized || size == 0)) { + livePointers.erase(old_address); + } + return remember(resized); + } + static void release(void *pointer) { + if (trackPointers && pointer) { + livePointers.erase(reinterpret_cast(pointer)); + } + std::free(pointer); + } + +public: + explicit FailNthAllocation(size_t position, bool track = false) { + failAt = position; + calls = 0; + failed = false; + trackPointers = track; + livePointers.clear(); + VecSim_SetMemoryFunctions({.allocFunction = allocate, + .callocFunction = callocate, + .reallocFunction = reallocate, + .freeFunction = release}); + } + ~FailNthAllocation() { restore(); } + bool wasInjected() const { return failed; } + size_t liveAllocationCount() const { return livePointers.size(); } + static void restore() { + trackPointers = false; + VecSim_SetMemoryFunctions({.allocFunction = std::malloc, + .callocFunction = std::calloc, + .reallocFunction = std::realloc, + .freeFunction = std::free}); + } +}; + +class PiPNNAllocationFailureTest : public ::testing::Test { +protected: + static constexpr size_t storedCount = 16; + static constexpr size_t maxFailurePosition = 2048; + const std::vector points = randomPoints(storedCount + 1); + std::vector> allocators; + + IndexPtr preparedIndex(bool build = true) { + auto index = newIndex(pipnnParams()); + if (!index) { + return index; + } + allocators.push_back(index->getAllocator()); + for (size_t label = 0; label < storedCount; ++label) { + EXPECT_EQ(VecSimIndex_AddVector(index.get(), points[label].data(), label), 1); + } + if (build) { + EXPECT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + return index; + } + + void verifyStoredVectors(VecSimIndex *index, bool relabeled = false) { + for (size_t label = 0; label < storedCount; ++label) { + expectNearest(index, points[label], relabeled && label == 3 ? 99 : label); + } + } + + void TearDown() override { + FailNthAllocation::restore(); + // Kept alive until teardown so fatal assertions also verify allocator cleanup. + for (const auto &allocator : allocators) { + EXPECT_EQ(allocator->getAllocationSize(), sizeof(VecSimAllocator)); + } + } +}; + +} // namespace + +TEST(PiPNNTest, SupportsOnlySingleValueFloat32L2) { + auto params = pipnnParams(); + auto index = newIndex(params); + ASSERT_NE(index, nullptr); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), 0); + + for (auto type : {VecSimType_FLOAT64, VecSimType_FLOAT16, VecSimType_BFLOAT16, VecSimType_INT8, + VecSimType_UINT8}) { + auto unsupported = params; + unsupported.algoParams.pipnnParams.type = type; + EXPECT_EQ(newIndex(unsupported), nullptr); + } + for (auto metric : {VecSimMetric_IP, VecSimMetric_Cosine}) { + auto unsupported = params; + unsupported.algoParams.pipnnParams.metric = metric; + EXPECT_EQ(newIndex(unsupported), nullptr); + } + params.algoParams.pipnnParams.multi = true; + EXPECT_EQ(newIndex(params), nullptr); + params.algoParams.pipnnParams.multi = false; + params.algoParams.pipnnParams.dim = 0; + EXPECT_EQ(newIndex(params), nullptr); +} + +TEST(PiPNNTest, ZeroAlgorithmParametersUseDefaults) { + VecSimParams params{}; + params.algo = VecSimAlgo_PIPNN; + params.algoParams.pipnnParams.type = VecSimType_FLOAT32; + params.algoParams.pipnnParams.dim = Point{}.size(); + params.algoParams.pipnnParams.metric = VecSimMetric_L2; + auto index = newIndex(params); + ASSERT_NE(index, nullptr); + Point point{1, 2, 3, 4}; + EXPECT_EQ(VecSimIndex_AddVector(index.get(), point.data(), 7), 1); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_TRUE(VecSimIndex_IsGraphReady(index.get())); + expectNearest(index.get(), point, 7); +} + +TEST(PiPNNTest, PublicInfoEstimatesAndRuntimeParameters) { + constexpr size_t count = 64; + auto params = pipnnParams(1); + const size_t initial_estimate = VecSimIndex_EstimateInitialSize(¶ms); + const size_t element_estimate = VecSimIndex_EstimateElementSize(¶ms); + EXPECT_GT(initial_estimate, 0); + EXPECT_LT(initial_estimate, SIZE_MAX); + EXPECT_GT(element_estimate, Point{}.size() * sizeof(float)); + EXPECT_LT(element_estimate, SIZE_MAX); + auto invalid = params; + invalid.algoParams.pipnnParams.metric = VecSimMetric_Cosine; + EXPECT_EQ(VecSimIndex_EstimateInitialSize(&invalid), SIZE_MAX); + EXPECT_EQ(VecSimIndex_EstimateElementSize(&invalid), SIZE_MAX); + + auto index = newIndex(params); + auto exact = newExactIndex(); + ASSERT_NE(index, nullptr); + ASSERT_NE(exact, nullptr); + auto points = randomPoints(count); + for (size_t label = 0; label < count; ++label) { + VecSimIndex_AddVector(index.get(), points[label].data(), label); + VecSimIndex_AddVector(exact.get(), points[label].data(), label); + } + auto check_info = [&](bool ready) { + EXPECT_EQ(VecSimIndex_BasicInfo(index.get()).algo, VecSimAlgo_PIPNN); + auto info = VecSimIndex_DebugInfo(index.get()); + EXPECT_EQ(info.commonInfo.basicInfo.algo, VecSimAlgo_PIPNN); + EXPECT_EQ(info.commonInfo.indexSize, count); + std::unique_ptr iterator( + VecSimIndex_DebugInfoIterator(index.get()), VecSimDebugInfoIterator_Free); + ASSERT_NE(iterator, nullptr); + bool found_ready = false; + while (VecSimDebugInfoIterator_HasNextField(iterator.get())) { + auto field = VecSimDebugInfoIterator_NextField(iterator.get()); + if (std::string(field->fieldName) == "GRAPH_READY") { + found_ready = true; + ASSERT_EQ(field->fieldType, INFOFIELD_UINT64); + EXPECT_EQ(field->fieldValue.uintegerValue, ready); + } + if (std::string(field->fieldName) == "ALGORITHM") { + ASSERT_EQ(field->fieldType, INFOFIELD_STRING); + EXPECT_STREQ(field->fieldValue.stringValue, "PIPNN"); + } + } + EXPECT_TRUE(found_ready); + }; + check_info(false); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + check_info(true); + + VecSimQueryParams runtime{}; + VecSimRawParam beam{"EF_RUNTIME", 10, "0", 1}; + const char *error = nullptr; + EXPECT_EQ(VecSimIndex_ResolveParams(index.get(), &beam, 1, &runtime, QUERY_TYPE_KNN, &error), + VecSimParamResolverErr_BadValue); + ASSERT_NE(error, nullptr); + const auto beam_value = std::to_string(count); + beam.value = beam_value.c_str(); + beam.valLen = beam_value.size(); + ASSERT_EQ(VecSimIndex_ResolveParams(index.get(), &beam, 1, &runtime, QUERY_TYPE_KNN, &error), + VecSim_OK); + ASSERT_EQ(runtime.pipnnRuntimeParams.efRuntime, count); + Point query{1.25f, -2.5f, 3.75f, 0.125f}; + auto expected = topK(exact.get(), query, 10); + ASSERT_EQ(expected.size(), 10); + runTopKSearchTest( + index.get(), query.data(), 10, + [&](size_t label, double score, size_t position) { + EXPECT_EQ(label, expected[position].first); + EXPECT_DOUBLE_EQ(score, expected[position].second); + }, + &runtime); +} + +TEST(PiPNNTest, InvalidTuningIsRejected) { + const auto defaults = pipnnParams(); + for (size_t field = 0; field < 8; ++field) { + auto params = defaults; + auto &p = params.algoParams.pipnnParams; + switch (field) { + case 0: + p.maxDegree = 1; + break; + case 1: + p.leafSize = 1; + break; + case 2: + p.fanout = 1; + break; + case 3: + p.hashBits = 33; + break; + case 4: + p.candidateLimit = p.maxDegree - 1; + break; + case 5: + p.alpha = 0.5; + break; + case 6: + p.alpha = std::numeric_limits::quiet_NaN(); + break; + case 7: + p.fanout = 33; + break; + } + EXPECT_EQ(newIndex(params), nullptr) << "Invalid parameter case " << field; + } +} + +TEST(PiPNNTest, LifecycleRejectsNullInputsAndOtherBackends) { + auto index = newIndex(pipnnParams()); + auto exact = newExactIndex(); + ASSERT_NE(index, nullptr); + ASSERT_NE(exact, nullptr); + EXPECT_EQ(VecSimIndex_BuildGraph(nullptr), 0); + EXPECT_FALSE(VecSimIndex_IsGraphReady(nullptr)); + EXPECT_EQ(VecSimIndex_SavePiPNN(nullptr, "/tmp/unused-pipnn-path"), 0); + EXPECT_EQ(VecSimIndex_SavePiPNN(index.get(), nullptr), 0); + EXPECT_EQ(VecSimIndex_SavePiPNN(index.get(), ""), 0); + EXPECT_EQ(VecSimIndex_LoadPiPNN(nullptr, nullptr), nullptr); + EXPECT_EQ(VecSimIndex_LoadPiPNN("", nullptr), nullptr); + EXPECT_EQ(VecSimIndex_BuildGraph(exact.get()), 0); + EXPECT_FALSE(VecSimIndex_IsGraphReady(exact.get())); + EXPECT_EQ(VecSimIndex_SavePiPNN(exact.get(), "/tmp/unused-pipnn-path"), 0); +} + +TEST(PiPNNTest, InvalidVectorsReturnCStatusAndPreserveStoredData) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + const Point valid{1, 2, 3, 4}; + ASSERT_EQ(VecSimIndex_AddVector(index.get(), valid.data(), 7), 1); + for (bool build : {false, true}) { + if (build) { + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + Point nonfinite = valid; + for (float value : + {std::numeric_limits::quiet_NaN(), std::numeric_limits::infinity(), + -std::numeric_limits::infinity()}) { + nonfinite.back() = value; + EXPECT_EQ(VecSimIndex_AddVector(index.get(), nonfinite.data(), 99), -1); + EXPECT_EQ(VecSimIndex_AddVector(index.get(), nonfinite.data(), 7), -1); + EXPECT_EQ(VecSimIndex_UpdateVectors(index.get(), 7, nonfinite.data(), 1), + VecSimUpdate_Failed); + } + EXPECT_EQ(VecSimIndex_AddVector(index.get(), nullptr, 99), -1); + EXPECT_EQ(VecSimIndex_UpdateVectors(index.get(), 7, nullptr, 1), VecSimUpdate_Failed); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), 1); + expectNearest(index.get(), valid, 7); + } +} + +TEST(PiPNNTest, InvalidQueriesReturnNullBeforeAndAfterBuild) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + const Point valid{1, 2, 3, 4}; + Point nan = valid, positive_infinity = valid, negative_infinity = valid; + nan.back() = std::numeric_limits::quiet_NaN(); + positive_infinity.back() = std::numeric_limits::infinity(); + negative_infinity.back() = -std::numeric_limits::infinity(); + const std::array invalid_queries{nullptr, nan.data(), positive_infinity.data(), + negative_infinity.data()}; + ASSERT_EQ(VecSimIndex_AddVector(index.get(), valid.data(), 7), 1); + for (bool build : {false, true}) { + if (build) { + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + for (const void *query : invalid_queries) { + ReplyPtr nearest(VecSimIndex_TopKQuery(index.get(), query, 1, nullptr, BY_SCORE), + VecSimQueryReply_Free); + EXPECT_EQ(nearest, nullptr); + ReplyPtr range(VecSimIndex_RangeQuery(index.get(), query, 10, nullptr, BY_SCORE), + VecSimQueryReply_Free); + EXPECT_EQ(range, nullptr); + BatchPtr batch(VecSimBatchIterator_New(index.get(), query, nullptr), + VecSimBatchIterator_Free); + EXPECT_EQ(batch, nullptr); + EXPECT_TRUE(std::isnan(VecSimIndex_GetDistanceFrom_Unsafe(index.get(), 7, query))); + } + for (double radius : {-1.0, std::numeric_limits::quiet_NaN()}) { + ReplyPtr range( + VecSimIndex_RangeQuery(index.get(), valid.data(), radius, nullptr, BY_SCORE), + VecSimQueryReply_Free); + EXPECT_EQ(range, nullptr); + } + // BY_SCORE_THEN_ID is a public enum value, but is not a supported query order. + ReplyPtr nearest( + VecSimIndex_TopKQuery(index.get(), valid.data(), 1, nullptr, BY_SCORE_THEN_ID), + VecSimQueryReply_Free); + EXPECT_EQ(nearest, nullptr); + ReplyPtr range( + VecSimIndex_RangeQuery(index.get(), valid.data(), 10, nullptr, BY_SCORE_THEN_ID), + VecSimQueryReply_Free); + EXPECT_EQ(range, nullptr); + expectNearest(index.get(), valid, 7); + } +} + +TEST(PiPNNTest, ExactBeforeBuildAndHighRecallAfterBuild) { + constexpr size_t count = 160; + constexpr size_t k = 10; + auto index = newIndex(pipnnParams(count)); + auto exact = newExactIndex(); + ASSERT_NE(index, nullptr); + ASSERT_NE(exact, nullptr); + auto points = randomPoints(count); + for (size_t i = 0; i < points.size(); ++i) { + ASSERT_EQ(VecSimIndex_AddVector(index.get(), points[i].data(), i), 1); + VecSimIndex_AddVector(exact.get(), points[i].data(), i); + } + ASSERT_FALSE(VecSimIndex_IsGraphReady(index.get())); + Point query{1.25f, -2.5f, 3.75f, 0.125f}; + EXPECT_EQ(topK(index.get(), query, k), topK(exact.get(), query, k)); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + ASSERT_TRUE(VecSimIndex_IsGraphReady(index.get())); + + size_t hits = 0; + constexpr size_t queries = 20; + for (size_t q = 0; q < queries; ++q) { + // Offset stored points so nearest-neighbor recall is not only self lookup. + query = points[q * 7]; + query[0] += 0.17f; + query[2] -= 0.31f; + auto reference = topK(exact.get(), query, k); + auto actual = topK(index.get(), query, k); + ASSERT_EQ(actual.size(), k); + for (const auto &[label, score] : actual) { + const auto &point = points.at(label); + double expected_score = 0; + for (size_t d = 0; d < point.size(); ++d) { + expected_score += double(point[d] - query[d]) * (point[d] - query[d]); + } + EXPECT_NEAR(score, expected_score, 1e-3); + hits += std::any_of(reference.begin(), reference.end(), + [label](const auto &result) { return result.first == label; }); + } + } + EXPECT_GE(double(hits) / (queries * k), 0.95); + EXPECT_TRUE(VecSimIndex_IsGraphReady(index.get())); +} + +TEST(PiPNNTest, MutationsInvalidateAndRebuildAfterCompaction) { + auto index = newIndex(pipnnParams()); + auto exact = newExactIndex(); + ASSERT_NE(index, nullptr); + auto points = randomPoints(48); + for (size_t label = 0; label < points.size(); ++label) { + VecSimIndex_AddVector(index.get(), points[label].data(), label); + VecSimIndex_AddVector(exact.get(), points[label].data(), label); + } + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + Point replacement{100, 200, 300, 400}; + EXPECT_EQ(VecSimIndex_AddVector(index.get(), replacement.data(), 3), 0); + VecSimIndex_AddVector(exact.get(), replacement.data(), 3); + EXPECT_FALSE(VecSimIndex_IsGraphReady(index.get())); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), points.size()); + expectNearest(index.get(), replacement, 3); + EXPECT_EQ(topK(index.get(), points[3], 12), topK(exact.get(), points[3], 12)); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + + // Remove a nonfinal slot, forcing the backend to keep moved vectors and labels aligned. + ASSERT_EQ(VecSimIndex_DeleteVector(index.get(), 1), 1); + VecSimIndex_DeleteVector(exact.get(), 1); + EXPECT_FALSE(VecSimIndex_IsGraphReady(index.get())); + expectNearest(index.get(), points.back(), points.size() - 1); + EXPECT_EQ(topK(index.get(), points[1], 12), topK(exact.get(), points[1], 12)); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_TRUE(VecSimIndex_IsGraphReady(index.get())); + expectNearest(index.get(), replacement, 3); + expectNearest(index.get(), points.back(), points.size() - 1); + + Point added{-100, -200, -300, -400}; + EXPECT_EQ(VecSimIndex_AddVector(index.get(), added.data(), 90), 1); + EXPECT_FALSE(VecSimIndex_IsGraphReady(index.get())); + expectNearest(index.get(), added, 90); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + expectNearest(index.get(), added, 90); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), points.size()); + + EXPECT_EQ(VecSimIndex_RelabelVector(index.get(), 90, 91), VecSimRelabel_OK); + EXPECT_FALSE(VecSimIndex_IsGraphReady(index.get())); + expectNearest(index.get(), added, 91); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_EQ(VecSimIndex_UpdateVectors(index.get(), 91, replacement.data(), 1), VecSimUpdate_OK); + EXPECT_FALSE(VecSimIndex_IsGraphReady(index.get())); + EXPECT_EQ(VecSimIndex_DeleteVector(index.get(), 3), 1); + expectNearest(index.get(), replacement, 91); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + expectNearest(index.get(), replacement, 91); +} + +TEST(PiPNNTest, RangeAndBatchRemainExactBeforeAndAfterBuild) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + constexpr size_t count = 17; + for (size_t i = 0; i < count; ++i) { + GenerateAndAddVector(index.get(), 4, i, i); + } + Point query{}; + for (bool build : {false, true}) { + if (build) { + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + runRangeQueryTest( + index.get(), query.data(), 4 * 4.5 * 4.5, + [](size_t label, double score, size_t position) { + EXPECT_EQ(label, position); + EXPECT_DOUBLE_EQ(score, 4 * label * label); + }, + 5, BY_SCORE); + BatchPtr batch(VecSimBatchIterator_New(index.get(), query.data(), nullptr), + VecSimBatchIterator_Free); + ASSERT_NE(batch, nullptr); + size_t seen = 0; + while (VecSimBatchIterator_HasNext(batch.get()) && seen < count) { + size_t start = seen; + size_t expected = std::min(size_t(6), count - seen); + runBatchIteratorSearchTest( + batch.get(), 6, + [&](size_t label, double score, size_t position) { + EXPECT_EQ(label, start + position); + EXPECT_DOUBLE_EQ(score, 4 * label * label); + ++seen; + }, + BY_SCORE, expected); + } + EXPECT_EQ(seen, count); + EXPECT_FALSE(VecSimBatchIterator_HasNext(batch.get())); + VecSimBatchIterator_Reset(batch.get()); + EXPECT_TRUE(VecSimBatchIterator_HasNext(batch.get())); + runBatchIteratorSearchTest(batch.get(), 1, [](size_t label, double score, size_t) { + EXPECT_EQ(label, 0); + EXPECT_DOUBLE_EQ(score, 0); + }); + } +} + +TEST(PiPNNTest, TimeoutPropagatesFromGraphFallbackRangeAndBatch) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + for (size_t i = 0; i < 32; ++i) { + GenerateAndAddVector(index.get(), 4, i, i); + } + Point query{}; + for (bool build : {false, true}) { + if (build) { + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + TimeoutGuard timeout; + ReplyPtr nearest(VecSimIndex_TopKQuery(index.get(), query.data(), 5, nullptr, BY_SCORE), + VecSimQueryReply_Free); + EXPECT_EQ(VecSimQueryReply_GetCode(nearest.get()), VecSim_QueryReply_TimedOut); + ReplyPtr range(VecSimIndex_RangeQuery(index.get(), query.data(), 100, nullptr, BY_SCORE), + VecSimQueryReply_Free); + EXPECT_EQ(VecSimQueryReply_GetCode(range.get()), VecSim_QueryReply_TimedOut); + BatchPtr batch(VecSimBatchIterator_New(index.get(), query.data(), nullptr), + VecSimBatchIterator_Free); + ReplyPtr first(VecSimBatchIterator_Next(batch.get(), 5, BY_SCORE), VecSimQueryReply_Free); + EXPECT_EQ(VecSimQueryReply_GetCode(first.get()), VecSim_QueryReply_TimedOut); + } +} + +TEST(PiPNNTest, BatchRetriesPartialTimeoutAndZeroBatchDoesNotConsumeResults) { + constexpr size_t count = 1101; + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + auto points = randomPoints(count); + for (size_t i = 0; i < count; ++i) { + ASSERT_EQ(VecSimIndex_AddVector(index.get(), points[i].data(), i), 1); + } + const Point query = points.back(); + const auto expected = topK(index.get(), query, count); + struct State { + size_t calls = 0; + bool enabled = true; + } state; + VecSimQueryParams params{}; + params.timeoutCtx = &state; + TimeoutGuard restoreCallback; + VecSim_SetTimeoutCallbackFunction([](void *context) { + auto &state = *static_cast(context); + return state.enabled && ++state.calls > 7 ? 1 : 0; + }); + BatchPtr batch(VecSimBatchIterator_New(index.get(), query.data(), ¶ms), + VecSimBatchIterator_Free); + ASSERT_NE(batch, nullptr); + ReplyPtr invalid(VecSimBatchIterator_Next(batch.get(), 1, BY_SCORE_THEN_ID), + VecSimQueryReply_Free); + EXPECT_EQ(invalid, nullptr); + ReplyPtr empty(VecSimBatchIterator_Next(batch.get(), 0, BY_SCORE), VecSimQueryReply_Free); + ASSERT_NE(empty, nullptr); + EXPECT_EQ(VecSimQueryReply_Len(empty.get()), 0); + EXPECT_EQ(state.calls, 0); + EXPECT_TRUE(VecSimBatchIterator_HasNext(batch.get())); + + ReplyPtr timedOut(VecSimBatchIterator_Next(batch.get(), 10, BY_SCORE), VecSimQueryReply_Free); + ASSERT_NE(timedOut, nullptr); + EXPECT_EQ(VecSimQueryReply_GetCode(timedOut.get()), VecSim_QueryReply_TimedOut); + EXPECT_EQ(VecSimQueryReply_Len(timedOut.get()), 0); + EXPECT_EQ(state.calls, 8); + state.enabled = false; + std::vector> actual; + runBatchIteratorSearchTest(batch.get(), count, [&](size_t label, double score, size_t) { + actual.emplace_back(label, score); + }); + EXPECT_EQ(actual, expected); + EXPECT_FALSE(VecSimBatchIterator_HasNext(batch.get())); +} + +TEST(PiPNNTest, LabelsKeepAllBitsThroughBuildAndMutation) { + if (sizeof(size_t) < sizeof(uint64_t)) { + GTEST_SKIP() << "Requires 64-bit labels"; + } + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + const size_t high_label = (uint64_t(1) << 40) + 7; + Point near{1, 2, 3, 4}, far{10, 20, 30, 40}; + VecSimIndex_AddVector(index.get(), near.data(), high_label); + VecSimIndex_AddVector(index.get(), far.data(), 7); + for (bool build : {false, true}) { + if (build) { + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + } + // The shared topK helper narrows labels to int, so inspect this reply directly. + ReplyPtr reply(VecSimIndex_TopKQuery(index.get(), near.data(), 1, nullptr, BY_SCORE), + VecSimQueryReply_Free); + ASSERT_EQ(VecSimQueryReply_Len(reply.get()), 1); + auto iterator = VecSimQueryReply_GetIterator(reply.get()); + auto result = VecSimQueryReply_IteratorNext(iterator); + EXPECT_EQ(VecSimQueryResult_GetId(result), high_label); + EXPECT_DOUBLE_EQ(VecSimQueryResult_GetScore(result), 0); + VecSimQueryReply_IteratorFree(iterator); + } + EXPECT_EQ(VecSimIndex_DeleteVector(index.get(), high_label), 1); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), 1); + expectNearest(index.get(), far, 7); +} + +TEST(PiPNNTest, EmptyAndIdenticalVectorsBuildAndTerminate) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + Point query{}; + EXPECT_TRUE(topK(index.get(), query, 10).empty()); + for (size_t label = 0; label < 65; ++label) { + VecSimIndex_AddVector(index.get(), query.data(), label); + } + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_TRUE(VecSimIndex_IsGraphReady(index.get())); + runTopKSearchTest(index.get(), query.data(), 20, [](size_t label, double score, size_t) { + EXPECT_LT(label, 65); + EXPECT_DOUBLE_EQ(score, 0); + }); + for (size_t label = 0; label < 65; ++label) { + ASSERT_EQ(VecSimIndex_DeleteVector(index.get(), label), 1); + } + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_TRUE(topK(index.get(), query, 10).empty()); +} + +TEST(PiPNNTest, AllocatorReleasesGraphAndQueryStorage) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + auto allocator = index->getAllocator(); + const size_t empty_size = allocator->getAllocationSize(); + auto points = randomPoints(100); + for (size_t label = 0; label < points.size(); ++label) { + VecSimIndex_AddVector(index.get(), points[label].data(), label); + } + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + EXPECT_GT(allocator->getAllocationSize(), empty_size); + const size_t built_size = allocator->getAllocationSize(); + expectNearest(index.get(), points[12], 12); + EXPECT_EQ(allocator->getAllocationSize(), built_size); + index.reset(); + EXPECT_EQ(allocator->getAllocationSize(), sizeof(VecSimAllocator)); +} + +TEST_F(PiPNNAllocationFailureTest, ConstructorFailuresReleaseAllAllocations) { + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + bool created; + bool injected; + size_t live_allocations; + { + // Unlike operation tests, construction has no surviving allocator to inspect. + FailNthAllocation failure(position, true); + auto index = newIndex(pipnnParams()); + created = index != nullptr; + index.reset(); + injected = failure.wasInjected(); + live_allocations = failure.liveAllocationCount(); + } + EXPECT_EQ(created, !injected); + EXPECT_EQ(live_allocations, 0); + if (!injected) { + completed = true; + break; + } + ++injected_count; + } + EXPECT_TRUE(completed) << "Construction did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); +} + +TEST_F(PiPNNAllocationFailureTest, BuildFailuresPreserveFlatStorageAndCanBeRetried) { + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + auto index = preparedIndex(); + ASSERT_NE(index, nullptr); + int result; + bool injected; + { + FailNthAllocation failure(position); + result = VecSimIndex_BuildGraph(index.get()); + injected = failure.wasInjected(); + } + EXPECT_EQ(result, injected ? 0 : 1); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), storedCount); + EXPECT_EQ(VecSimIndex_IsGraphReady(index.get()), !injected); + verifyStoredVectors(index.get()); + if (!injected) { + completed = true; + break; + } + ++injected_count; + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + verifyStoredVectors(index.get()); + } + EXPECT_TRUE(completed) << "Build did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); +} + +TEST_F(PiPNNAllocationFailureTest, AddFailuresPreserveCountAndExistingVectors) { + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + // The 17th vector crosses the configured block boundary. + auto index = preparedIndex(); + ASSERT_NE(index, nullptr); + int result; + bool injected; + { + FailNthAllocation failure(position); + result = VecSimIndex_AddVector(index.get(), points.back().data(), 99); + injected = failure.wasInjected(); + } + EXPECT_EQ(result, injected ? -1 : 1); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), storedCount + (injected ? 0 : 1)); + verifyStoredVectors(index.get()); + if (!injected) { + expectNearest(index.get(), points.back(), 99); + completed = true; + break; + } + ++injected_count; + EXPECT_TRUE( + std::isnan(VecSimIndex_GetDistanceFrom_Unsafe(index.get(), 99, points.back().data()))); + ASSERT_EQ(VecSimIndex_AddVector(index.get(), points.back().data(), 99), 1); + expectNearest(index.get(), points.back(), 99); + } + EXPECT_TRUE(completed) << "Insertion did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); +} + +TEST_F(PiPNNAllocationFailureTest, RelabelFailuresKeepOriginalLabelAndData) { + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + auto index = preparedIndex(); + ASSERT_NE(index, nullptr); + VecSimRelabelCode result; + bool injected; + { + FailNthAllocation failure(position); + result = VecSimIndex_RelabelVector(index.get(), 3, 99); + injected = failure.wasInjected(); + } + EXPECT_EQ(result, injected ? VecSimRelabel_Failed : VecSimRelabel_OK); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), storedCount); + verifyStoredVectors(index.get(), !injected); + const size_t absent_label = injected ? 99 : 3; + EXPECT_TRUE(std::isnan( + VecSimIndex_GetDistanceFrom_Unsafe(index.get(), absent_label, points[3].data()))); + if (!injected) { + completed = true; + break; + } + ++injected_count; + ASSERT_EQ(VecSimIndex_RelabelVector(index.get(), 3, 99), VecSimRelabel_OK); + verifyStoredVectors(index.get(), true); + } + EXPECT_TRUE(completed) << "Relabel did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); +} + +TEST_F(PiPNNAllocationFailureTest, QueryFailuresReleaseTemporaryStorage) { + for (bool build : {false, true}) { + for (bool range : {false, true}) { + SCOPED_TRACE(build ? "built" : "fallback"); + SCOPED_TRACE(range ? "range" : "topK"); + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + auto index = preparedIndex(build); + ASSERT_NE(index, nullptr); + const auto before = index->getAllocationSize(); + ReplyPtr reply(nullptr, VecSimQueryReply_Free); + bool injected; + { + FailNthAllocation failure(position); + reply.reset(range ? VecSimIndex_RangeQuery(index.get(), points[3].data(), 1e6, + nullptr, BY_SCORE) + : VecSimIndex_TopKQuery(index.get(), points[3].data(), 4, + nullptr, BY_SCORE)); + injected = failure.wasInjected(); + } + EXPECT_EQ(reply == nullptr, injected); + if (reply) { + EXPECT_EQ(VecSimQueryReply_GetCode(reply.get()), VecSim_QueryReply_OK); + EXPECT_EQ(VecSimQueryReply_Len(reply.get()), range ? storedCount : 4); + } + reply.reset(); + EXPECT_EQ(index->getAllocationSize(), before); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), storedCount); + EXPECT_EQ(VecSimIndex_IsGraphReady(index.get()), build); + verifyStoredVectors(index.get()); + if (!injected) { + completed = true; + break; + } + ++injected_count; + } + EXPECT_TRUE(completed) << "Query did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); + } + } +} + +TEST_F(PiPNNAllocationFailureTest, DebugIteratorFailuresReleaseTemporaryStorage) { + auto index = preparedIndex(); + ASSERT_NE(index, nullptr); + const auto before = index->getAllocationSize(); + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + std::unique_ptr iterator( + nullptr, VecSimDebugInfoIterator_Free); + bool injected; + { + FailNthAllocation failure(position); + iterator.reset(VecSimIndex_DebugInfoIterator(index.get())); + injected = failure.wasInjected(); + } + EXPECT_EQ(iterator == nullptr, injected); + if (iterator) { + EXPECT_EQ(VecSimDebugInfoIterator_NumberOfFields(iterator.get()), 14); + } + iterator.reset(); + EXPECT_EQ(index->getAllocationSize(), before); + EXPECT_TRUE(VecSimIndex_IsGraphReady(index.get())); + if (!injected) { + completed = true; + break; + } + ++injected_count; + } + EXPECT_TRUE(completed); + EXPECT_GT(injected_count, 0); +} + +TEST_F(PiPNNAllocationFailureTest, BatchFailuresReleaseIteratorAndQueryStorage) { + for (bool build : {false, true}) { + SCOPED_TRACE(build ? "built" : "fallback"); + bool completed = false; + size_t injected_count = 0; + for (size_t position = 1; position <= maxFailurePosition; ++position) { + SCOPED_TRACE(position); + auto index = preparedIndex(build); + ASSERT_NE(index, nullptr); + const auto before = index->getAllocationSize(); + BatchPtr batch(nullptr, VecSimBatchIterator_Free); + ReplyPtr reply(nullptr, VecSimQueryReply_Free); + bool injected; + { + FailNthAllocation failure(position); + batch.reset(VecSimBatchIterator_New(index.get(), points[3].data(), nullptr)); + if (batch) { + reply.reset(VecSimBatchIterator_Next(batch.get(), 4, BY_SCORE)); + } + injected = failure.wasInjected(); + } + EXPECT_EQ(batch == nullptr || reply == nullptr, injected); + if (reply) { + EXPECT_EQ(VecSimQueryReply_GetCode(reply.get()), VecSim_QueryReply_OK); + EXPECT_EQ(VecSimQueryReply_Len(reply.get()), 4); + } + if (injected && batch && !reply) { + auto expected = topK(index.get(), points[3], 4); + ASSERT_EQ(expected.size(), 4); + runBatchIteratorSearchTest(batch.get(), 4, + [&](size_t label, double score, size_t position) { + EXPECT_EQ(label, expected[position].first); + EXPECT_DOUBLE_EQ(score, expected[position].second); + }); + } + reply.reset(); + batch.reset(); + EXPECT_EQ(index->getAllocationSize(), before); + EXPECT_EQ(VecSimIndex_IndexSize(index.get()), storedCount); + EXPECT_EQ(VecSimIndex_IsGraphReady(index.get()), build); + verifyStoredVectors(index.get()); + if (!injected) { + completed = true; + break; + } + ++injected_count; + } + EXPECT_TRUE(completed) << "Batch did not finish within the failure-position bound"; + EXPECT_GT(injected_count, 0); + } +} + +TEST(PiPNNTest, PersistencePreservesReadyDirtyAndPostLoadMutations) { + auto index = newIndex(pipnnParams()); + auto exact = newExactIndex(); + ASSERT_NE(index, nullptr); + auto points = randomPoints(40); + for (size_t label = 0; label < points.size(); ++label) { + VecSimIndex_AddVector(index.get(), points[label].data(), label); + VecSimIndex_AddVector(exact.get(), points[label].data(), label); + } + TemporaryFile file; + ASSERT_FALSE(file.path.empty()); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + ASSERT_EQ(VecSimIndex_SavePiPNN(index.get(), file.path.c_str()), 1); + IndexPtr loaded(VecSimIndex_LoadPiPNN(file.path.c_str(), nullptr), VecSimIndex_Free); + ASSERT_NE(loaded, nullptr); + EXPECT_TRUE(VecSimIndex_IsGraphReady(loaded.get())); + EXPECT_EQ(VecSimIndex_IndexSize(loaded.get()), points.size()); + EXPECT_EQ(topK(loaded.get(), points[10], 10), topK(index.get(), points[10], 10)); + auto loaded_allocator = loaded->getAllocator(); + + Point replacement{100, 200, 300, 400}; + EXPECT_EQ(VecSimIndex_AddVector(loaded.get(), replacement.data(), 4), 0); + VecSimIndex_AddVector(exact.get(), replacement.data(), 4); + EXPECT_EQ(VecSimIndex_DeleteVector(loaded.get(), 1), 1); + VecSimIndex_DeleteVector(exact.get(), 1); + EXPECT_FALSE(VecSimIndex_IsGraphReady(loaded.get())); + ASSERT_EQ(VecSimIndex_SavePiPNN(loaded.get(), file.path.c_str()), 1); + loaded.reset(); + EXPECT_EQ(loaded_allocator->getAllocationSize(), sizeof(VecSimAllocator)); + loaded.reset(VecSimIndex_LoadPiPNN(file.path.c_str(), nullptr)); + ASSERT_NE(loaded, nullptr); + EXPECT_FALSE(VecSimIndex_IsGraphReady(loaded.get())); + EXPECT_EQ(VecSimIndex_IndexSize(loaded.get()), points.size() - 1); + EXPECT_EQ(topK(loaded.get(), points[1], 10), topK(exact.get(), points[1], 10)); + expectNearest(loaded.get(), replacement, 4); + ASSERT_EQ(VecSimIndex_BuildGraph(loaded.get()), 1); + expectNearest(loaded.get(), replacement, 4); + Point added{-100, -200, -300, -400}; + EXPECT_EQ(VecSimIndex_AddVector(loaded.get(), added.data(), 99), 1); + EXPECT_FALSE(VecSimIndex_IsGraphReady(loaded.get())); + expectNearest(loaded.get(), added, 99); +} + +TEST(PiPNNTest, PersistenceRejectsTruncationAndCorruptHeader) { + auto index = newIndex(pipnnParams()); + ASSERT_NE(index, nullptr); + auto points = randomPoints(20); + for (size_t label = 0; label < points.size(); ++label) { + VecSimIndex_AddVector(index.get(), points[label].data(), label); + } + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + TemporaryFile valid, damaged; + ASSERT_FALSE(valid.path.empty()); + ASSERT_FALSE(damaged.path.empty()); + ASSERT_EQ(VecSimIndex_SavePiPNN(index.get(), valid.path.c_str()), 1); + std::ifstream input(valid.path, std::ios::binary); + std::vector bytes((std::istreambuf_iterator(input)), {}); + ASSERT_GT(bytes.size(), 16); + + for (size_t length : {size_t(0), size_t(1), bytes.size() / 2, bytes.size() - 1}) { + std::ofstream output(damaged.path, std::ios::binary | std::ios::trunc); + output.write(bytes.data(), length); + output.close(); + IndexPtr loaded(VecSimIndex_LoadPiPNN(damaged.path.c_str(), nullptr), VecSimIndex_Free); + EXPECT_EQ(loaded, nullptr) << "Accepted a file truncated to " << length << " bytes"; + } + bytes[0] ^= 0x7f; + std::ofstream output(damaged.path, std::ios::binary | std::ios::trunc); + output.write(bytes.data(), bytes.size()); + output.close(); + IndexPtr loaded(VecSimIndex_LoadPiPNN(damaged.path.c_str(), nullptr), VecSimIndex_Free); + EXPECT_EQ(loaded, nullptr); +} + +TEST(PiPNNTest, PersistenceRejectsMissingSuccessorWithValidChecksum) { + auto params = pipnnParams(); + auto index = newIndex(params); + ASSERT_NE(index, nullptr); + const Point first{1, 2, 3, 4}, second{5, 6, 7, 8}; + ASSERT_EQ(VecSimIndex_AddVector(index.get(), first.data(), 7), 1); + ASSERT_EQ(VecSimIndex_AddVector(index.get(), second.data(), 8), 1); + ASSERT_EQ(VecSimIndex_BuildGraph(index.get()), 1); + TemporaryFile file; + ASSERT_FALSE(file.path.empty()); + ASSERT_EQ(VecSimIndex_SavePiPNN(index.get(), file.path.c_str()), 1); + IndexPtr pristine(VecSimIndex_LoadPiPNN(file.path.c_str(), nullptr), VecSimIndex_Free); + ASSERT_NE(pristine, nullptr); + ASSERT_TRUE(VecSimIndex_IsGraphReady(pristine.get())); + + std::ifstream input(file.path, std::ios::binary); + std::vector bytes((std::istreambuf_iterator(input)), {}); + // V1: magic[8], version[4], ten tuning fields[8], count[8], ready[4], + // then count records of label[8] + dim FP32 coordinates, followed by degrees[8]. + constexpr size_t count = 2; + constexpr size_t header_size = 8 + 4 + 10 * 8 + 8 + 4; + const size_t degree_offset = header_size + count * (8 + Point{}.size() * 4); + ASSERT_GT(bytes.size(), degree_offset + count * 8 + 8); + // A zero-degree row is otherwise structurally valid, but removes node 0's only successor. + std::fill_n(bytes.begin() + degree_offset, 8, char{0}); + uint64_t checksum = 14695981039346656037ULL; + for (size_t i = 0; i < bytes.size() - 8; ++i) { + checksum = (checksum ^ static_cast(bytes[i])) * 1099511628211ULL; + } + for (size_t i = 0; i < 8; ++i) { + bytes[bytes.size() - 8 + i] = static_cast(checksum >> (8 * i)); + } + std::ofstream output(file.path, std::ios::binary | std::ios::trunc); + output.write(bytes.data(), bytes.size()); + output.close(); + ASSERT_TRUE(output.good()); + IndexPtr malformed(VecSimIndex_LoadPiPNN(file.path.c_str(), nullptr), VecSimIndex_Free); + EXPECT_EQ(malformed, nullptr); +}