Repository navigation
SQ4: ARM and x86 SIMD distance kernels - #1103
Merged
Merged
Conversation
dor-forer
force-pushed
the
codex/sq4-simd
branch
from
November 8, 2026 08:09
fc096df to
170fd09
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## codex/sq4-production-backend #1103 +/- ##
================================================================
+ Coverage 97.46% 97.52% +0.05%
================================================================
Files 148 157 +9
Lines 9715 10012 +297
================================================================
+ Hits 9469 9764 +295
- Misses 246 248 +2 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
FP32-query IP and L2 on NEON, SVE/SVE2, AVX2_FMA and AVX512F_BW_VL_VNNI; SQ4-SQ4 IP, L2 and the centered L2 identity on NEON, NEON_DOTPROD, SVE/SVE2 and AVX512F_BW_VL_VNNI. Tails use masked or lane loads, so no kernel reads past the packed vector. The FP32-query IP kernels fall back to the scalar reconstruction on a non-finite result, and the integer SQ4-SQ4 paths are bounded by SQ4_MAX_EXACT_SIMD_DIM. Each tier is tested against the scalar kernel, including guard-page and overflow cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dor-forer
force-pushed
the
codex/sq4-simd
branch
from
November 8, 2026 08:52
170fd09 to
cdc35b3
Compare
…to codex/sq4-simd
…to codex/sq4-simd
2 tasks done
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit b92b87c. Configure here.
The AVX2 SQ4-FP32 IP and L2 tails read the last query floats with maskload. Masked-out lanes of VMASKMOVPS may fault on some processors, so an exact-size query could fault at a page boundary. Load the last full 8-float window that ends at the last coordinate instead, shift the tail codes into the matching top lanes and zero the lanes below. A dim below 8, which the dispatcher never sends here, copies the floats. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts: # tests/unit/test_sq4_distances.cpp
GuyAv46
previously approved these changes
Nov 9, 2026
Add AVX2 stored-to-stored IP and L2 kernels with the existing exact-dimension fallback and VNNI precedence. Share the stored-IP metadata formula across scalar and SQ4 SIMD kernels while preserving its operation order.
Revert 03a445f. On Neoverse V2 with GCC 12.3, the comparison bound regresses SVE2 FP32 IP at D768 by 1.26-1.28% and L2 at D770 by 2.39-2.41% in both link orders. Short-vector gains do not establish a general replacement.
The NEON and SVE SQ4-FP32 kernels handle every dim, and their tails read only the codes, the query floats and metadata. Drop the dim < 16 scalar gate and test the ARM tiers from dim 0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
GuyAv46
approved these changes
Nov 10, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.

Summary
ARM and x86 SIMD kernels for SQ4. Stacked on #1099 (scalar SQ4), which is stacked on #1101. Retarget each PR after the one below it merges.
This PR adds no new behavior: every kernel returns what the scalar SQ4 kernels in #1099 return, within a tested tolerance, and the dispatchers fall back to those kernels below each tier's minimum dimension, on CPUs without the tier, and above
SQ4_MAX_EXACT_SIMD_DIMfor the integer paths.SIMD kernels
dimfloats plus metadata of the query.SQ4_MAX_EXACT_SIMD_DIM.Median ns per distance at D768, single thread, SQ4 vs SQ8 on the same ISA:
Hosts: Neoverse-V2 (SVE vector length 128) and a Xeon 8375C. Results are medians of 5 runs on lightly loaded shared hosts, and are indicative only.
The SQ4-SQ4 L2 row was measured with the identity kernels, which are now the only SQ4-SQ4 L2 kernels.
Benchmarks
bm_spaces_sq4_fp32andbm_spaces_sq4_sq4are registered inbenchmarks.shand the CI benchmark matrix. An HNSW SQ4 index benchmark is left for later, because it needs an uploaded fixture.Main risks and how they are tested
SQ4_MAX_EXACT_SIMD_DIM(derivation inspaces.h), above which the chooser returns the scalar kernel, whose 64-bit sum is exact. The FP32-query IP kernels behave like SQ8's for extreme query values.Commits
SQ4: Add NEON, SVE, AVX2 and AVX-512 SQ4 distance kernels: kernel headers, choosers inspaces/functions/*, dispatch inIP_space.cpp/L2_space.cpp, and the per-tier tests.SQ4: Add SQ4 spaces microbenchmarks:bm_spaces_sq4_fp32,bm_spaces_sq4_sq4,benchmarks.shand the CI benchmark matrix.Testing
cdc35b35); CI checks it, and a host run on ARM and x86 follows after the review. An earlier version of these kernels, with the plain-L2 reconstruction kernels and the IP overflow fallback, passed fullcteston x86 (3396/3396) and ARM (3353/3353).🤖 Generated with Claude Code
Note
Medium Risk
Changes hot-path distance math for quantized indices; wrong SIMD tails or integer overflow would skew search results, though dispatch bounds and broad per-tier tests mitigate this.
Overview
Adds SIMD distance kernels for 4-bit quantized vectors (SQ4-FP32 and SQ4-SQ4) on inner product and L2, matching the existing SQ8 dispatch pattern.
Dispatch in
IP_space.cpp/L2_space.cppno longer always returns scalar: it selects NEON / SVE / SVE2, AVX2+FMA, or AVX-512 VNNI by CPU features, with dimension floors on x86 and scalar fallback aboveSQ4_MAX_EXACT_SIMD_DIMwhen the integer code-dot would overflowuint32.New kernel headers implement nibble unpacking, masked/predicated tails, and L2 via the stored-sum identity (reusing IP code-dot).
Stored_InnerProduct_FromCodeDotis factored out ofquantized_kernels.hfor shared SIMD/scalar completion.CI/benchmarks:
bm-spaces-sq4-fp32andbm-spaces-sq4-sq4in the workflow matrix; microbenchmark binaries andtest_sq4_distancestier tests (scalar parity, guard pages, corruption controls, exact-dim cap).Reviewed by Cursor Bugbot for commit 4fdb250. Bugbot is set up for automated code reviews on this repo. Configure here.