Repository navigation
Conversation
… vectors Distances for multi-vector retrieval in the style of GEM (arXiv:2603.20336) over a document's token set stored as one fixed-width vector of centroid ids ([n, ids..., padding]): - QEmd: exact earth mover's distance over a centroid-pair cost table, solved with POT's network simplex (vendored unmodified in third-party/pot). - QChamfer: quantized Chamfer distance with both sides quantized. - QChamferExactQuery: raw float query tokens against document centroids, with the per-query dot table built in fix_argument. The kernels are instantiated per ISA in multi-arch/x86 like L2 and IP. pre-commit skips the vendored headers so they stay byte-identical. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DefaultBuildAdaptor default-constructs its distance instead of copying the one passed to the builder, which does not compile for QEmd and QChamfer (they hold their centroid table), and PruneStrategy has no primary template. Override both for these functors. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A vector set is a variable-cardinality set of token vectors, as used by late-interaction retrieval. The concept comes first and the only model is a non-owning view over caller-held tokens and offsets; an owning container can follow behind the same concept. The view also models ImmutableMemoryDataset, so existing index code can traverse it. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sum over query tokens of the maximum inner product over document tokens, built on the existing dispatched inner product so every supported element type pair works without new SIMD code. A tile-blocked kernel can replace the body of maxsim_batch behind the same signature. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Pins QEmd, QChamfer, QChamferExactQuery, CentroidTable and the Vamana build-adaptor overloads with hand-computed values before the row layout changes. Also registers the vector-set view and MaxSim tests. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
WithVectorSets pairs the dataset the graph traverses with the vector sets it was derived from. The single_search overload for it runs the ordinary graph search, then scores every candidate in the search buffer with MaxSim against the query set and returns them best-first. Static indexes only: an index with id translation is rejected at compile time. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
QEmd, QChamfer and QChamferExactQuery accept a ContiguousVectorSet next to the padded float row: a set of unsigned centroid ids with one id per token, and a float query set. Both layouts share the per-ISA cores, and the padded path keeps its arithmetic. The view gains the extent member VamanaIndex reads, so a graph can be built and searched over it directly. The id helpers called from the per-ISA structs become members of those structs, so each ISA keeps its own copy of them. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
First compile of the branch. The re-rank overload names its unused index parameter, the re-rank test supplies the prune strategy its local distance needs, and the MaxSim asymmetry test uses sets whose two directions differ. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Moving merge_duplicates into the per-ISA struct made it private and broke an external timing harness that measures it separately. The two overloads are public again; behaviour is unchanged. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
VectorSetData<T, Extent, Alloc> owns the same CSR layout ConstVectorSetView reads: a SimpleData token matrix plus a std::vector<uint64_t> of offsets, so SVS's allocators, alignment, and binary file format carry over unchanged. Construction takes both by value and moves them in, so a caller handing over existing storage pays no copy; copy_from() covers the case where a caller only has borrowed memory. A Blocked allocator is rejected at compile time since it would tear a document's tokens across blocks. Save/load follows SimpleData's protocol: the token matrix nests its own save() under a sub-table, the offsets go to a standalone binary file referenced by metadata. VectorSetDataLoader mirrors VectorDataLoader so auto_assemble can reload a saved index over this dataset; auto_assemble never calls verify_and_set_default_index_parameters (that only runs from the build-parameters constructor), so a custom distance such as QChamfer hits no obstacle there. Validation is factored out of ConstVectorSetView's constructor into a shared detail::validate_vector_set_layout so the view and the owning container cannot drift apart. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Add save()/load() to WithVectorSets, nesting each member under its own sub-table. Both are constrained by a requires-clause checking Primary::save/load and Sets::save/load exist, so a WithVectorSets over a non-owning Sets (e.g. ConstVectorSetView, which has neither) still compiles as a dataset member; the methods simply don't participate for that instantiation. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
QEmd, QChamfer and QChamferExactQuery accepted two token-set layouts: the old fixed-width padded float row and a vector set. Only the vector-set layout survives; the padded-row span overloads, their gem_detail decoders (set_size/set_ids, vector_set_size/vector_set_tokens), and the per-ISA float-pointer overloads they fed are deleted. QEmdImpl::merge_duplicates is private again now that its only caller is the vector-set path; the external gem_time harness that needed it public will be adapted separately, outside this repository. Vector-set callers are unaffected: their arithmetic and dispatch are unchanged. Tests are rewritten to pass the same hand-computed cases as vector sets instead of padded spans, dropping only the padding-is-ignored and layout-comparison cases that tested the removed layout; no expected value changed. A static_assert pins down that a bare std::span can no longer satisfy QChamfer's compute() constraint. BREAKING CHANGE: removes QEmdImpl::compute(span,span), QChamferImpl::compute(span,span), QChamferImpl::decode_ids, QEmdImpl::merge_duplicates(const float*, ...) (now private for the uint32_t overload too), QEmd::compute(span,span), QChamfer::compute(span, span), QChamferExactQuery::fix_argument(span) and QChamferExactQuery::compute(span,span). gem_detail::set_size, set_ids, vector_set_size and vector_set_tokens are also removed. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
QChamferExactQuery::fix_argument built its (K x n_query) dot table with a per-centroid-row loop limited to 2 independent FMA accumulators, which left it latency-bound well short of the AVX-512 FMA ceiling. Factor that build into a standalone inner_product_matrix kernel (generic/AVX2/AVX512, same dispatch mechanism as the other GEM distances) that blocks several A-rows per step so the FMA dependency chain is spread across more accumulators; GEM's build_table is now a direct call into it. The reduction half (compute_decoded) is unchanged in this commit; its cross-TU weak-symbol duplication is fixed separately next. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
gem_detail::qchamfer_exact_reduce was a free inline function called from QChamferExactImpl::compute_decoded in the generic header and from both avx2.cpp and avx512.cpp. Each TU compiled it under different -m flags but emitted the same weak inline symbol, so the linker could keep an arbitrary one; a host lacking AVX-512 could end up executing code built for it. Move the reduction into QChamferExactImpl's primary template body (one source text) and drop its AVX2/AVX512 specializations entirely: the explicit instantiation already in avx2.cpp/avx512.cpp now compiles that same body under each TU's own flags, giving each ISA a distinctly named, non-weak symbol, matching how QEmdImpl/QChamferImpl already do this. Verified via `nm -C` on both object files: no QEmdImpl/QChamferImpl/ QChamferExactImpl/InnerProductMatrixImpl symbol is shared between them anymore. MergeScratch's compiler-generated destructor still is, but it holds no SIMD-sensitive logic, so it is not an ODR hazard. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Extends the inner-product-matrix kernel family with a shared PackedOperand (query packed once, reused across documents) and a column-max output form that keeps the lane-max in registers instead of materializing the m x n table, with a Float16 overload that converts one row-block at a time. MaxSim now follows the fix_argument protocol (like QChamferExactQuery) so the Vamana re-rank extension packs the query once per search instead of once per candidate. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…rProductMatrix MaxSim called InnerProductMatrixImpl<AVX512|AVX2|NONE>::column_max directly through its own three-way switch on InnerProductMatrix::select_isa(), duplicating the kernel's own dispatch and leaking the Impl struct and the ISA enum into a caller. InnerProductMatrix now exposes column_max(float) and column_max(Float16) wrappers that share one private dispatch() helper with compute(); select_isa() is private again. MaxSim calls only the wrapper and keeps a member maxima_ buffer (sized in fix_argument) instead of a thread_local one, since implicit_broadcast=false already gives each search thread its own MaxSim copy. Also tightens the Float16 generic-vs-ISA tolerance in inner_product_matrix.cpp from an unmeasured epsilon(5e-4)/margin(5e-5) to epsilon(2e-4)/margin(1e-4), about 2x the largest summation-order difference (4.7e-5 abs / 3.1e-5 rel) measured by a throwaway probe over the test's own shapes. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ch pair once from_centroids now tiles the K x K ground-cost table over the upper triangle of blocks (including diagonal blocks), computing each tile with InnerProductMatrix::compute and mirroring the transformed value into both T[i][j] and T[j][i] instead of calling it twice. Tiles are scheduled with a DynamicPartition to balance the uneven tile count per row block; the table's storage, load, and operator()(i, j) are unchanged. Symmetry is now guaranteed by construction (one computed float stored at both cells) rather than by two independently computed dot products, so results can differ from the old code in the last bits. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…o InnerProductMatrix Adds svs::BFloat16 overloads of both InnerProductMatrixImpl output forms (build_packed/table and column_max) at every ISA level (generic, AVX2, AVX512), sharing the Float16 overloads' row-block-conversion shape via a templated helper per ISA file; also fills in the previously-missing Float16 overload of the table form since the shared helper makes it free. BFloat16's widening to float32 is an exact zero-extend (bfloat16 is float32's high 16 bits), unlike Float16's lossy half-to-single conversion. PackedOperand gains pack_rounded_to_bf16, rounding B to bfloat16 (round to nearest, ties to even) and storing the result as float32: rounding B is the caller's choice of precision, not a hidden kernel behaviour, so it lives in how B is packed rather than in the kernel itself. svs::BFloat16's own constructor truncates instead of rounding; svs::round_to_bfloat16 (bfloat16.h) adds the correctly-rounded conversion both this and the bf16 A overloads above need to agree on. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…float16 FineCentroids now holds its K x dim matrix as svs::BFloat16 (round to nearest, ties to even) instead of float32: one representation, no float32 copy kept after construction/load, so every ISA level computes on identical values and the upcoming AMX level can consume the same layout directly. The constructor and load() round in parallel (a thread pool each already had access to); load() additionally reads and rounds in row-block chunks so peak memory never holds the full float32 input alongside the full bf16 output. QChamferExactQuery::fix_argument_decoded now packs the query rounded to bf16 (PackedOperand::pack_rounded_to_bf16) and builds the per-query table with InnerProductMatrix::compute_packed against the (already bf16) centroids, so the table is <bf16(centroid), bf16(query)> on every ISA level. CentroidTable::from_centroids rounds both the row and column block of each tile to bf16 before computing, so its K x K table agrees with FineCentroids on the same rounded values; the public signature (float32 centroids in) is unchanged, since rounding now happens per tile instead of requiring a second, already-bf16 input. Test fixtures: all pre-existing hand-computed expectations use only values exactly representable in bfloat16 (1, 0, -1), so none changed. Added: a bf16-rounding section for FineCentroids (incl. a load() round-trip through the unchanged float32 file format), a dedicated non-representable-value case for both CentroidTable::from_centroids and QChamferExactQuery against a naive bf16-rounded reference, and rounded the from_centroids tiling test's random input before building its reference (bf16 rounding error otherwise swamped the existing 1e-4 summation-order margin). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Extends AVX_AVAILABILITY with AMX = 10, leaving the ordinals 3-9 free for a parallel split of AVX512 into AVX512F/AVX512VNNI: nothing in this codebase reads the enum's ordinal or order, only switches and templates on the enumerator itself, so a gap is free. Detection in avx_detection.h follows the existing CPUID-probe style and adds the one-time, thread-safe OS permission request AMX tile use needs (ARCH_REQ_XCOMP_PERM), cached in a function-local static so the per-call cost stays one branch. Build integration in cmake/multi-arch.cmake adds amx.cpp only when the compiler accepts -march=sapphirerapids, gated through a build-wide SVS_AMX_TILE_BF16_SUPPORTED definition so a build without that support still produces a working library with the level compiled out. No existing kernel's dispatch changes: InnerProductMatrix::select_isa and its exhaustive switch gain an AMX branch guarded on SVS_AMX_COMPILED_IN, falling back to the AVX512 case when the level was never compiled in. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adds the AMX tile-dot-product kernel for InnerProductMatrixImpl's table (build_packed) and column-max forms, selected only when AMX is available, A is BFloat16, and B was packed with PackedOperand::pack_rounded_to_bf16 (which now also fills a pair-interleaved transposed bf16 tile layout, built with ordinary scalar code so it needs no -mamx flags of its own, gated on the build-wide SVS_AMX_COMPILED_IN and the runtime availability check). Every other case (float32/Float16 A, or an unrounded B) forwards to the AVX-512 implementation, keeping one source of truth. The kernel processes A 16 rows and B 32 lanes (two n-tiles) at a time, with two independent fp32 accumulator tiles so their dpbf16ps chains overlap; A's final row-block (m not a multiple of 16) gets a second, one-off tile reconfigure, and A's final k-chunk (d not a multiple of 32) is copied through a zero-padded stack buffer so the kernel never reads past the caller's d columns. Tile numbers are literal integers throughout (0-4): gcc's tile intrinsics stringify their register argument, so a named constant would not assemble. _tile_loadd/_tile_stored carry no memory operand or clobber, so the compiler is otherwise free to reorder a preceding store (the tail copy) or a following read (emit's use of the stored tiles) across them; both call sites get an explicit compiler memory barrier. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Sweeps InnerProductMatrixImpl<AMX>'s build_packed and column_max against a double-precision reference and against AVX-512, over m/n/d shapes crossing the 16-row, 16-lane, and 32-element (16-pair) tile boundaries; checks float32/Float16 A and an unrounded B forward to AVX-512 bit for bit; checks the availability probe agrees across concurrent threads; and checks a thread that never itself requested AMX permission can still run the kernel after another thread already has. Skips when amx_tile_bf16_available() is false, so this is a no-op on hardware or a build without the level. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ernel Templates MaxSim on its document element type (default float, so the non-template spelling keeps today's behaviour) instead of branching on it inside fix_argument. A document of the wrong element type now fails to compile via compute's requires-clause, and fix_argument packs the query rounded to bf16 only for MaxSim<svs::BFloat16>, so float/Float16 callers still pack and score unrounded, bit for bit as before. The Vamana re-rank extension now instantiates MaxSim<Sets::const_value_type ::element_type>, so a bfloat16 vector-set store automatically gets the bf16 (and AMX, where available) InnerProductMatrix::column_max path. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…ails Adds a mmap/mprotect-based test that places A so it ends exactly at a PROT_NONE page (catches an overrun), places `out` the same way, and places A at the start of a mapping with a leading PROT_NONE page (catches an underrun), then runs both the table and column-max forms over every tail shape. Covers the AMX bfloat16 tile kernel plus, where available, the AVX-512 and AVX2 float/Float16/bfloat16 kernels. POSIX-only, guarded so the file still compiles on a non-POSIX target; no-ops when the relevant ISA is unavailable. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
gcc 11's _tile_loadd/_tile_stored macros expand to inline asm with no
memory operand and no clobber, so the optimizer believes they neither
read nor write memory: a plain C++ store just before a load, or a
plain C++ read just after a store, through the same address had no
ordering guarantee and could be reordered around the tile
instruction. Two bare `__asm__ volatile("" ::: "memory")` barriers
papered over this; a third spot (configure_tiles' struct writes
before _tile_loadconfig) had no barrier at all and was correct only
by accident of how this compiler happens to schedule it today.
Replaces all three with named wrappers, one per literal tile number
actually used (0-4), built from extended inline asm that names the
memory each operation touches: a real "m"/"=m" operand covering the
full extent where it is a compile-time constant (the 64-byte tile
config; B's always-16x64-byte tile blocks; the local zero-padded tail
buffer; the 16x16 fp32 accumulator stores), or an input operand plus
a "memory" clobber where the extent is runtime-only (A's direct
strided load, whose row pitch depends on the caller's `d`).
Verified by a negative control: compiling this file without the
wrappers (bare intrinsics only, the reviewer's stripped copy) at
-O2/-O3 with and without -flto reproduces the hazard (garbage column
maxima); the same scenario against this commit's amx.cpp, built the
same four ways, matches a double-precision reference at all four.
Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
…hirerapids -march=sapphirerapids also enables AVX512-FP16, AVX512-BF16, AVX-VNNI and other extensions the runtime probe never tests; the compiler was free to use them auto-vectorizing ordinary code in that file, and a hypervisor can expose AMX without them. Extends the SVS_X86 tuple format with an optional 4th+ field of extra compiler options (every existing tuple unchanged: no 4th field, same -march/-mtune as before), and uses it to compile amx.cpp with exactly -march=cascadelake -mtune=cascadelake -mamx-tile -mamx-bf16, matching the avx512 tuple's instruction set plus the two AMX extensions. The object target name now keys off the tuple's name field when extra options are present, since amx and avx512 now share the same `arch` and would otherwise collide on `cascadelake_obj`. The compiler check tests the two AMX flags directly rather than the (now unused) -march=sapphirerapids. amx_tile_bf16_available() now also requires avx_runtime_flags.is_avx512f_supported(): the AMX level's code is compiled at that instruction set and its non-tile paths forward to the AVX-512 kernels directly, so AMX is only meaningful where AVX-512 already is. Verified with objdump that the amx object's instructions are plain AVX-512F/scalar: no vaddph/vcvtph2psx/vpdpbusd or other AVX512-FP16/ AVX-VNNI-only mnemonics appear. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Expands amx_tile_bf16_available()'s comment into a documentation block (exempt from the two-line internal-comment limit) stating two facts no header or source on this machine confirms: where the ARCH_REQ_XCOMP_PERM feature number 18 comes from (the kernel's xstate ABI doc, not any installed header) and that requesting it was confirmed by experiment on this Linux/x86-64 machine, not assumed from documentation; and that a seccomp policy killing on an unlisted syscall terminates the host process at this call. Also states, citing kernel documentation as unverified here, that the request is process-wide (tile state becomes part of every thread's extended processor state) and that the kernel may refuse it if an installed alternate signal stack is too small. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Shared k-means building blocks (centroid assignment/adjustment/split, GEMM-backed distance computation, training-set sampling) now live in svs::kmeans under include/svs/core/kmeans_training.h, so a future multi-vector codebook trainer can reuse them. Function bodies are moved verbatim; the IVF-specific kmeans_training(IVFBuildParameters, ...) overload now forwards to the shared one via a small KMeansTrainingParameters struct. include/svs/index/ivf/common.h re-exports every moved name with using-declarations so existing IVF call sites, tests, bindings and benchmarks compile unchanged. Named the shared header kmeans_training.h rather than kmeans.h: an unrelated legacy generic k-means utility (svs::find_nearest, svs::train, KMeansParameters) already owns include/svs/core/kmeans.h and its always-built test tests/svs/core/kmeans.cpp; reusing that path would have required threading an MKL dependency into the default, IVF/multi-vector-disabled build. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Covers the L2 centroid update formula (initial centroid plus assigned points, divided by count+1 - not a plain mean), unit-norm centroids under IP distance, and centroid_assignment agreeing with a brute-force nearest-centroid oracle for both distances. Registered in tests/CMakeLists.txt under "IVF or multi-vector", matching the new header's MKL dependency. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
SVS_EXPERIMENTAL_ENABLE_MULTI_VECTOR (default OFF) joins SVS_EXPERIMENTAL_ENABLE_IVF as a trigger for pulling in Intel(R) MKL, since both features train a codebook with the shared k-means code. IVF-only gates (tests, benchmark, bindings) are untouched. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
Adds unreachable_nodes (BFS from a root set) and repair_connectivity, which gives every unreachable node one inbound edge from a reachable candidate: a plain append when the candidate has room, else a re-prune, else a forced replacement of one of the candidate's neighbors. A re-prune or replacement is taken only when no neighbor it drops loses its last reachable in-edge; otherwise the repair falls back to an unsafe replacement and counts it. A round that does not reduce the unreachable count stops the repair. Needed because a build whose insert order puts a few nodes first (GEM's representatives) can leave part of the graph unreachable from the roots. Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ves first Adds gem_insert_order (every valid router representative first, deduplicated and ascending, then every other document ascending) and build_gem_graph, which runs VamanaIndex's own two-pass build sequence over that order when the distance carries a router, repairs connectivity from the entry point plus every representative, and reports what it did. Also adds the QEmd overload of extensions::build_entry_points: when the distance carries a router, a node's insert routes through its own centroid-id set the same way GemRoutedInitializer routes a query, merged with the fixed entry points (minus the node itself). Co-Authored-By: Claude Fable 5.2 <noreply@anthropic.com>
QuantizeTimings and all std::chrono use leave codebook.h. train_codebook and quantize are now thin compositions of public phase functions (sample_training_tokens, train_centroids, assign_tokens, build_ground_structures, coarse_clusters); train_centroids's k-means loop takes a compile-time observer (default a no-op) called once per iteration, with zero cost when unused. codebook_timed.h is new and opt-in: it defines QuantizeTimings and quantize_timed/train_codebook_timed, composing the public phases with timers and an observer that records iteration seconds. Nothing in the library includes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…header
Review on PR 478 pointed out an inconsistency: avx2.cpp/avx512.cpp/amx.cpp
previously held only macro-driven explicit instantiations of templates
defined in their headers, but this PR added the InnerProductMatrixImpl
AVX2/AVX512/AMX kernel bodies directly in those .cpp files instead.
Moves every kernel body (and its anonymous-namespace helpers, now named
detail namespaces per ISA: inner_product_matrix_{avx2,avx512,amx}_detail)
into inner_product_matrix.h, next to the specialization each belongs to,
guarded by the same SVS_AVX512_F / SVS_AVX2 / SVS_AMX_TILE_BF16 conditions
that already guarded the specialization declarations. Each .cpp now holds
only includes and explicit instantiation lines, matching euclidean.h's and
gem.h's QEmdImpl/QChamferExactImpl pattern. No kernel logic changed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace the scattered #if SVS_ENABLE_STATS blocks in vamana_build.h and maxsim.h with NullBuildStats/CountingBuildStats (and the maxsim equivalent) collector classes. Both collectors always compile, so the stats path is type-checked in every build; only a std::conditional_t alias on SVS_ENABLE_STATS picks the default, and VamanaBuilder takes the collector as a defaulted template parameter so tests can force CountingBuildStats regardless of the build flag. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Second pass. Shares the GEM centroid/corpus fixtures and adds a generic brute-force top-k oracle plus a degree-bound checker (tests/svs/index/vamana/ gem_test_utils.h, used by all three GEM files and by maxsim_rerank.cpp); drops gem_routing.cpp's "no router" / "every cluster unrepresented" cases (covered by gem_build.cpp's build_entry_points); folds connectivity.cpp's five repair_connectivity cases into one table-driven TEST_CASE; builds one index per TEST_CASE in gem_vector_set.cpp and maxsim_rerank.cpp and runs every behavior as a CATCH_SECTION on it; and removes the build-time routed-vs-unrouted recall comparison from gem_build.cpp's build_gem_graph, keeping only the query-time one in gem_routing.cpp's "Routed search is not worse than unrouted" (the layer that owns search-time routing). Also drops maxsim_rerank.cpp's capacity=40 variant of the MaxSim pool-order check: a tighter search window only re-exercises the generic search buffer's eviction path, which search_buffer.cpp already owns, not anything in the MaxSim re-rank extension under test. Before -> after (lines, after clang-format): build_stats.cpp 139 -> 139 (unchanged, already minimal) connectivity.cpp 274 -> 219 gem_build.cpp 297 -> 170 gem_routing.cpp 394 -> 245 gem_vector_set.cpp 313 -> 202 maxsim_rerank.cpp 687 -> 548 gem_test_utils.h 0 -> 150 (new, shared across all 6 files) Total (6 files + helper, counted together per the <=1050 target) -> 1673, against a 2104-line baseline for the 6 files alone. This still falls short of the <=1050 target. Code-only lines (stripping every blank line and every comment from all 7 files) sum to over 1300, so the remaining gap cannot close through formatting or comment trimming alone: closing it means dropping tested branches, not duplication. Two branches already lost coverage in this pass, noted above; the handback report to the coordinator lists further branches that could be cut with their line cost, for the coordinator to choose from rather than have cut silently. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replace QEmdImpl's vendored-POT solve with a clean-room primal network simplex (include/svs/core/distance/transport_simplex.h): an artificial root node with big-M arcs gives a strongly feasible initial tree (Ahuja, Magnanti & Orlin, "Network Flows", 1993, ch. 11), each pivot's leaving arc is the *last* blocking arc seen walking the cycle from its apex in the entering arc's own direction (Cunningham, "A network simplex method", 1976), which keeps the tree strongly feasible and so guarantees termination independent of how the entering arc is chosen. That lets the entering rule optimize for speed instead: a block-cyclic search over the dense cost matrix (block ~sqrt(ua*ub) arcs, rounded to whole padded rows), picking the most negative reduced cost in the first block that has one and resuming from where the previous search left off, with an AVX-512/AVX2/scalar-templated row scan. No code is reused or paraphrased from POT/LEMON; the algorithm is textbook network simplex, so there is nothing to attribute beyond the existing citation of Bonneel et al.'s EMD problem shape already in the new header's top comment. An earlier version of this solver used northwest-corner initialization and Bland's rule; both are replaced here because northwest-corner is not automatically strongly feasible and Bland's rule (lowest-index entering arc) is too slow on this hot path's highly degenerate, asymmetric problem shapes (up to ~180x32 and 180x180), where it tends to stall rescanning the same low-index rows. QEmdImpl<Avx>::compute_decoded keeps its signature, the ua==ub==1 shortcut, and throw-on-failure behavior; gem_node_id_type and the POT include/pragma/undef block are gone since nothing else used them. Also fixes a stale comment in inner_product_matrix.h that described amx.cpp's build gate as -march=sapphirerapids support when it is actually -mamx-tile/-mamx-bf16 support at -march=cascadelake. Adds a random cross-check test (tests/svs/core/distances/gem.cpp): the NONE path against an independent successive-shortest-paths reference, and every compiled ISA path against NONE, over random, degenerate (equal-cost, single-node-side, wide weight range), and highly degenerate (uniform weights, tiny integer cost range, up to 64x64) problems -- the family that stalls a naive entering rule. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qEMD now solves its own transportation simplex (see the prior commit), so include/svs/third-party/pot/ (POT/LEMON's network_simplex_simple.h, full_bipartitegraph.h, core.h) has no remaining caller. Drops the directory and the .pre-commit-config.yaml exclusion that kept it byte-identical to upstream. No code from POT or LEMON is reused anywhere in this repo after this change, so there is nothing left to attribute beyond what the new solver's header already cites. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reviewer feedback: the GEM router/codebook/vector-set unit tests were way too extensive. Consolidates duplicate setup into shared helpers, merges per-assigner/per-thread-count and per-variant checks into single loops/sections sharing one fixture, folds validation-failure checks into tables, and drops hand-picked route_query cases that were strict subsets of the brute-force reference check already in the suite. Line counts (before -> after, clang-format applied): - tests/svs/core/distances/gem_router.cpp: 302 -> 239 - tests/svs/quantization/gem/codebook.cpp: 563 -> 480 - tests/svs/core/data/vector_set.cpp: 261 -> 222 - tests/svs/core/data/vector_set_view.cpp: 137 -> 124 - total: 1263 -> 1004 (16% fewer) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ans tests Second pass, on top of the first consolidation: one shared type/ISA matrix for build_packed/column_max (float32/Float16/BFloat16 x NONE/AVX2/AVX512), a compacted mmap guard-page RAII helper, one AMX forwarding+tile-kernel TEST_CASE, one AMX-permission TEST_CASE, one kmeans_training TEST_CASE with three sections, and a trimmed MaxSim suite. Also applies a branch-coverage pass: drops checks that exercise no library branch beyond another check on the keep list (AMX's float32/Float16 forwarding is one branch, tested with one type; MaxSim's empty-document and dimension-mismatch checks are one branch in MaxSim::compute, tested with one document type; compute_packed's BFloat16/Float16 overloads share one dispatch call, tested with one type; MaxSim's hand-computed cases were strict subsets of the random-oracle sections and are dropped). Before/after line counts: tests/svs/core/distances/inner_product_matrix.cpp: 1253 -> 637 tests/svs/core/distances/maxsim.cpp: 325 -> 251 tests/svs/core/kmeans_training.cpp: 215 -> 208 total: 1793 -> 1096 (38.9% cut) Still short of the requested >=50% / <=895-line target; see the PR description / handback report for the itemized list of what would have to drop (and its branch-coverage cost) to close the remaining ~200-line gap. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces scattered hand-computed cases with one random oracle per distance (QEmd/QChamfer/QChamferExactQuery) checked against an independent naive reference, each covering duplicate ids and, for QChamferExactQuery, bf16 non-representable values. Consolidates the three per-distance "generic dispatch" sections and the three "dimensions() != 1 throws" sections into one test case each. Compacts reference_transport_cost to a standard min-cost-flow edge-list SPFA and merges the "many equal costs" and "uniform-weight tiny-cost" transport-solver families into one parameterized loop, with lower trial counts (fixed seeds). Lines: 798 -> 725 (9% cut). Short of the requested 50%/400-line target: the transport-solver section (independent reference + the four problem families x per-ISA cross-check, all explicitly requested to stay) is ~195 lines on its own, and the from_centroids tiling test, traits, and Vamana extension test cases are already near-minimal. Getting under 400 would mean dropping one of those explicitly-requested items, which this PR does not do unilaterally. One real coverage loss, restored: QEmd's "identical sets give 0" invariant is not exercised by the random oracle (which never draws a == b), so it stays as its own hand case. Two consolidations are not drop-in equivalent in form (only in the branches they exercise): the old hand-picked worked examples for QChamfer/QEmd are gone, replaced by randomized reference checks, so a reader can no longer eyeball one fixed expected number. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Moves today's big-M network simplex, unchanged in arithmetic, into TransportSimplexReference: scalar pricing only, no Avx parameter. It becomes the AVX_AVAILABILITY::NONE path and the test oracle for the optimized solver that follows. Its helpers live in transport_reference_detail so the two cannot clash. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The balance residual (up to 1e-6 * total) is folded into the last demand entry. When that entry is smaller than the residual, the fold left it at zero or below, which the simplex cannot solve. That input now throws, naming ua, ub, the entry and the residual. Valid inputs are unaffected. The copy, balance check and fold move into prepare_marginals, which the optimized solver shares. It takes a caller tag type so each -march translation unit instantiates its own copy rather than sharing one weak symbol. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…simplex RowMin<Avx>::min_in_row finds the most negative reduced cost in one cost row, over float costs with either float or double potentials. Generic, AVX2 and AVX512 bodies are bit-identical: two separate subtractions (no FMA), ascending scalar lane scan, first minimum wins. Rows are padded to 16 lanes with +inf. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Replaces the per-pivot BFS tree rebuild, which was about 80% of the old solve, with LEMON's incremental spanning-tree update (parent, thread, rev_thread, succ_num, last_succ), ported by way of POT's network_simplex_simple.h. On the sub50k neighbors workload, one thread: 246 us -> 86 us per solve (POT: 103 us). The checksum is unchanged. Structure: - The smaller side becomes the rows. The optimal cost is invariant. - Costs are stored as float. CostFn must return float, so this is exact. - The start is north-west corner, rooted at source 0, with no big-M. - Node ids are uint16_t when max(ua, ub) <= 256, else uint32_t. The solver picks the bucket itself. - Pricing is float (RowMin) on float mirrors of double potentials. - Flows and the final cost are double. - AVX_AVAILABILITY::NONE forwards to TransportSimplexReference. - Every helper is a member of TransportSimplex<Avx>, so no function body becomes a weak symbol shared across -march translation units. Strong feasibility. LEMON's leaving rule (strict < on the i-side, <= on the j-side) keeps every zero-flow tree arc pointing toward the root, so its child is a source. The start meets this: - On a value tie at (i, j), the zero arc is (i+1, j), whose child is the fresh source i+1. - Every arc with a sink child carries a fresh, strictly positive amount. - The final cell uses whichever side is still untouched. No clamp. Pivot gate. Let n = ua+ub, P = (n-1)*max_cost (bounds |potential|), R = recompute_period, eps = DBL_EPSILON. - walk_tol = 2n*eps*(max_cost + P). This bounds |r_fresh - r_exact|: two root-path sums of at most n-1 steps each, plus the two subtractions that form r. Potentials walked from the root equal a fresh recompute bit for bit, and so does RowMin's double r, because the operation order is the same. - drift_tol = 2(n-1)*R*9*eps*P + walk_tol. One pivot moves a tree arc's residual by at most 9*eps*P: sigma costs 5*eps*P and applying it 4*eps*P. A node's error is the sum of residuals along its root path, so the depth factor is n-1, and there are two endpoints. - A candidate enters only if r_drifted < -drift_tol, or, when borderline, r_walked < -walk_tol. So every entering arc is truly negative, and Cunningham's finiteness proof applies. - Otherwise the solver recomputes fresh and prices once in double. Every stop happens on fresh potentials. error_bound. Each term covers a stated rounding source. - 2*walk_tol*total_mass: at the stop, r_exact >= -2*walk_tol, and complementary slackness turns that into a cost gap. - n^2*eps*max_cost*total_mass: before the final sum, the flows are recomputed from the marginals in post-order. Each flow is a signed subtree sum, and subtree sizes sum to at most n^2. - 2(n-1)*eps*max_cost*total_mass: the n-1 multiplies and adds of the final sum. The balance residual is folded into the caller's last demand before orienting, as in the reference. Portions derive from POT 0.9.5 (MIT) and LEMON (Boost Software License 1.0); see the header notices and THIRD-PARTY-PROGRAMS. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…erate shapes
Every ISA result is compared with the NONE (reference) result within
TransportSimplex<Avx>::error_bound plus 2 float ulps for the float
return of compute_decoded. The reference's own double rounding (about
1e-15) is far below that margin. New fixed cases: 100x100, 4x256 and
256x4, 3x200, 5x300 and 300x5 (the uint32 bucket), 1x256 and 256x1,
and 100x100 with all-equal or {0,1} costs and uniform weights.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The SIMD RowMin overloads stored each block of reduced costs and scanned it with a branchy scalar loop, about 40% of solve_impl's self cycles on sub50k. Each block now tests a strict-less mask against the running best and is skipped when the mask is empty. Only an improving block resolves its minimum: the lowest column equal to the block minimum wins, and its exact bits are taken from the lane itself (compress on AVX-512, a stack readback on AVX2), so ties, signed zeros and NaN behave as in the scalar primary template. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
update_potential, find_join_node, the leaving-arc search, change_flow, update_tree_structure and recompute_potentials index std::vector members of the workspace in their loops. A store through one vector's data may alias another vector's pointer member, so the compiler reloaded the base pointers each iteration. The loops now use __restrict locals. Reads, writes and operation order are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dense K x K table is 1 GiB at K=16384, and qEMD's cost fill does scattered lookups into it, which are TLB- and cache-miss bound. It now lives in SimpleData with HugepageAllocator, the same storage as VectorSetData and SimpleGraph. It takes 1 GiB or 2 MiB pages when available and falls back to normal pages otherwise. CentroidTable becomes copyable (deep copy) because SimpleData is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
When ua > ub, TransportSimplex transposes and its cost fill walks ids_a in the inner loop, so every lookup landed in a different 64 KiB table row. A table built by from_centroids is exactly symmetric, since both halves are written from one float. For such a table compute_decoded now swaps the lookup to table(ids_b[j], ids_a[i]), which keeps the row fixed. Results are bit-identical. Tables from the raw constructor or load keep the old lookup. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds TransportSimplex<Avx>::thread_pivot_count(), a thread-local pivot counter that exists only when SVS_ENABLE_STATS is set (SVS_EXPERIMENTAL_ENABLE_STATS). Normal builds compile no extra code. The counter is per ISA instantiation, so a caller must read it through the same Avx as the dispatched solver. It lets a benchmark report pivots / (m + k - 1). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The northwest-corner start now walks columns in an order grouped by each column's cheapest row (col_argmin, ties broken by ascending column id) instead of column-id order, so the start tree is biased toward low-cost arcs before any pivot runs. Strong feasibility still holds, because this is the same NW-corner walk (the existing construction's argument) applied to a relabelled column axis; the walk's argument only ever reasons about *position* (i, p), never about a column's real id j = col_order[p]: - The forced last-row/last-col steps and the final cell are unchanged: they fire on p == k - 1 exactly as they fired on j == k - 1, and the final cell still hands the fresh side's untouched, strictly-positive value to the one remaining arc. - A value tie (supply and demand exhausted together) still adds a zero arc to row i + 1 with a source child, because that branch never reads j except to pass it to add_tree_arc; which column j happens to be is irrelevant to the branch's own logic. - Every other zero-flow arc claim is unaffected: the tie arc is still the only arc add_tree_arc is ever called with flow == 0, since every other call passes min(supply[i], demand[j]) once one side is still positive, or a fresh side's full remaining value. - Permuting columns cannot create a zero sink-child arc: sink-child flow values are exactly the same set of quantities (supply/demand residuals) the identity-order walk would have produced, just assigned to relabelled j's; relabelling a column changes which array slot a value lands in, not the value itself or whether it is positive. - Row identity is untouched (rows still visit 0..m-1 in order, root stays row 0), which is what the proof actually depends on; only columns are relabelled. Extra cost: one compare per cost cell while filling (fill_cost_matrix also tracks col_min/col_argmin), plus an O(m + k) counting sort for col_order. On qEMD solves from a 50k-document subset (K=16384), pivots per (m + k - 1) drop 5-9% in every set-size bucket, and 1-thread solve time drops 2.4-4.4%. Results are unchanged. The start therefore runs unconditionally, with no size gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…check
Adds a section to the random cross-check case: all-equal and {0,1} costs,
uniform and repeated ({1,2,3}) weights, 1xn, nx1 and nxn shapes up to
256x256, plus a cost pattern where row 0 is the cheapest row for every
column (every column in one group). Existing sections draw the same random
numbers as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
svs::kmeans::emeans_train in svs/core/kmeans_emeans.h implements Algorithm 1 of Bessa et al., arXiv:2611.09501: - EMA per-cluster counts and sums (centroid = S / N), with a 1-based batch counter as in Algorithm 1. - A cosine-annealed step from alpha0 down to 0.1 * alpha0. - Batch size and alpha0 from the paper's Table 2 scaling laws when unset, including the /8 batch law above n = 2M. - Dead-cluster revival by splitting every 5 batches at tau = 0.1. - A final pass that sets each centroid to the mean of its last-epoch points. It reuses centroid_assignment, generate_norms and init_centroids from kmeans_training.h. Assignment runs in chunks of at most assignment_chunk rows, so the score buffer is assignment_chunk x k floats rather than B x k. The state (k * d double sums, k counts) and all scratch are allocated once. Revival costs O(k + dead * log k) per round. Deviations and readings where the paper is silent are commented in the code: - the tail batch is trained; - power-of-two rounding is done in log2 space; - the donor pool excludes this round's dead clusters; - delta = 2^-7; - epochs defaults to 2, an SVS choice. The scaling laws were fitted on DEEP (L2, 256 <= k <= 65536). Outside that range they are applied as published. The returned objective is the MSE of the final centroids. Tests cover: determinism; revival of a forced-dead cluster; the scaling laws against hand-computed values; a tail batch; the k > n error and the other parameter checks. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodebookParameters gains kmeans_algorithm (KMeansAlgorithm::Lloyd by default) and emeans_epochs, which defaults to EMeansParameters' own default. When e-means is selected, train_centroids runs emeans_train: - kmeans_iterations is ignored; - minibatch_size caps the assignment chunk; - the objective is reported as a sum, the same convention as the Lloyd path. Default behaviour is unchanged. The faiss 256-points-per-centroid bound now applies only to Lloyd. The coarse-cluster k-means stays Lloyd. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The back-edge prune ran DynamicPartition over the BackedgeBuffer buckets, 1000 consecutive node ids each. With an expensive distance, a few buckets holding hub nodes with thousands of overflow edges ran alone at the end of every batch. The prune now flattens the buckets into one list of overflow nodes. The list is sorted by overflow set size descending, ties by id. Threads claim entries one at a time from an atomic cursor. The per-thread setup (scratch vectors, build adaptor, stats collector) still runs once per thread. Generation and its bucket locks are unchanged. Each node's prune reads only its own adjacency and the data, so the per-node result does not depend on the schedule. Measured on a 50k-document qEMD sub-build (R=24, L=80, alpha=1.2, 112 threads, one NUMA node): 755 s before, 290 s after (2.6x). The build is not deterministic even before the change: two baseline builds share 99.9% of edges. Against that noise, degree statistics and recall@10 are unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CodebookParameters::kmeans_algorithm now defaults to KMeansAlgorithm::EMeans (2 epochs, scaling-law batch size and step). Lloyd stays selectable. The switch was made by the 1M MS MARCO run's criteria: the e-means objective is at most Lloyd's at no more training time, and MRR@10 is within 0.005 of the Lloyd codebook at every search window. The numbers are in the run's decision record. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two one-thread builds of data/test_dataset (L2, R 16, window 32) must give identical adjacency lists. With more threads, back-edge lock order changes the graph, so this is the strongest determinism the static build offers. It also records how 5c650da (one back-edge prune per work unit, largest first) was shown not to change the graph, beyond matching statistics: - With one thread the generation phase has no lock-order effect, so the bucket schedule and the per-node schedule must agree byte for byte. qEMD, 1,000 docs of the 50k subset, R 24, L 80, alpha 1.2, unrouted: 705115b against 705115b plus the 5c650da vamana_build.h, and against HEAD, give graph files that differ only in the 16-byte file UUID. - L2 on data/test_dataset via utils/build_index (window 64, R 32, alpha 1.2, one thread): 705115b, HEAD and a second HEAD run likewise differ only in the UUID. - ThreadSanitizer on a multi-threaded static build at HEAD (2,000 to 200,000 points, degree 2 to 32, 8 to 112 threads): no reports, in the prune phase or elsewhere. TSan sees only races unordered by happens-before, and the per-vertex locks order most pairs, so this is supporting evidence; the one-thread identity carries the proof. - [vamana] (49 cases) and [integration][build][vamana] pass at HEAD. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds enum class TransportStart { NorthWest, CheapestRow }. QEmd takes it as
an optional third constructor argument (default NorthWest) and passes it
through QEmdImpl<Avx>::compute / compute_decoded to TransportSimplex::solve.
The choice is not saved with an index: it changes how fast a solve reaches
the optimum, not the optimum.
NorthWest walks columns in id order and skips the column-min fold, so it
reproduces the solver before 8a648b2. This changes the default:
CheapestRow, hard-wired since 8a648b2, is now opt-in. Both starts share
one strong-feasibility argument, since the walk reasons only about
positions and rows keep their order.
The random cross-check, including its start-degeneracy section, runs under
both starts. On 200k qEMD solves (50k subset, K 16384), pivots per
(m + k - 1) and checksums equal 705115b's for NorthWest and 5c650da's
for CheapestRow on all four datasets. At 1 thread, NorthWest is within 1%
of 705115b, and CheapestRow is 3-5% faster than NorthWest.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.