Skip to content

Add opt-in native PiPNN backend with explicit graph lifecycle - #1123

Draft
dor-forer wants to merge 1 commit into
mainfrom
codex/pipnn-backend
Draft

dor-forer wants to merge 1 commit into
mainfrom
codex/pipnn-backend

Conversation

@dor-forer

Copy link
Copy Markdown
Collaborator

Adds a native, opt-in PiPNN index backend to VectorSimilarity. After bulk insertion, callers explicitly build its graph; TopK queries then use beam search. Mutations invalidate the graph and queries fall back to exact search until rebuilt. Existing index defaults are unchanged.

This is implementation code, separate from the upstream benchmark PoC in #1122.

Scope

  • FLOAT32, squared L2, one vector per label; unsupported configurations are rejected.
  • Factory and C API integration, explicit graph lifecycle, runtime EF_RUNTIME, debug info, and memory estimates.
  • Bounded candidate construction and pruning, with a successor cycle for graph connectivity.
  • Versioned persistence with structural validation, corruption detection, and temporary-file replacement.
  • Allocation-failure handling and rollback, timeout handling, exact range queries and batch iteration.
  • Native comparison runner, raw ARM measurements, and lifecycle documentation.

All calls on an index must be serialized, including queries. Redis Search schema support, tiered indexing, background rebuilds, other metrics/types, and parallel construction are outside this change. Persistence does not fsync the parent directory. The implementation uses native distance kernels and bounded scalar construction; it does not claim performance equivalence with upstream PiPNN.

Performance and rollout

The initial native ARM screen does not justify replacing HNSW. On 100k normalized DBpedia vectors, build time was 21.33 s versus HNSW's 60.00 s, but p95 query latency was 3.18 ms versus 0.85 ms at the lowest measured points above 95% recall. On DEEP, builds were 2.59 s versus 12.63 s and p95 was 0.83 ms versus 0.11 ms. Recall differs slightly between compared points; each backend/dataset had one build and a coarse ef sweep. PiPNN's peak build RSS was higher on both datasets.

Keep HNSW as the default. This draft provides a reviewable backend implementation; rollout still requires workload-specific recall/latency/rebuild-memory validation and embedding-application integration tests. Full conditions and raw results are in docs/benchmarks/pipnn/.

Validation

  • ARM GCC 12.3, standard SVS-enabled build: 245/245 selected Common, BruteForce, Allocator, and PiPNN CTest cases passed.
  • 25 PiPNN cases cover configuration, exact fallback, graph recall, mutations, timeouts and retry, allocation-failure sweeps, debug info, and persistence corruption.
  • ARM Clang 21 ASan/UBSan with leak detection: 25/25 PiPNN tests passed. Function-type sanitization was disabled because the existing CPU-feature C/C++ dispatch triggers that check. GCC 12 sanitizer compilation hit an internal compiler error in the existing BF16 SVE kernel, so Clang was used.
  • Independent artifact-only code review completed; reported issues were fixed and re-reviewed.
  • ARM GCC Release library built successfully with SVS enabled and both VECSIM_BUILD_TESTS and VECSIM_BUILD_TEST_HELPERS disabled.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant