Skip to content

SQ4: ARM and x86 SIMD distance kernels - #1103

Merged
dor-forer merged 15 commits into
codex/sq4-production-backendfrom
codex/sq4-simd
Nov 10, 2026
Merged

dor-forer merged 15 commits into
codex/sq4-production-backendfrom
codex/sq4-simd

Conversation

@dor-forer

@dor-forer dor-forer commented Nov 8, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

ARM and x86 SIMD kernels for SQ4. Stacked on #1099 (scalar SQ4), which is stacked on #1101. Retarget each PR after the one below it merges.

This PR adds no new behavior: every kernel returns what the scalar SQ4 kernels in #1099 return, within a tested tolerance, and the dispatchers fall back to those kernels below each tier's minimum dimension, on CPUs without the tier, and above SQ4_MAX_EXACT_SIMD_DIM for the integer paths.

SIMD kernels

Kernel ARM x86
FP32-query IP and L2 NEON, SVE, SVE2 AVX2+FMA, AVX-512
SQ4-SQ4 IP and L2 (stored-sum identity, as SQ8) NEON, NEON_DOTPROD, SVE, SVE2 AVX-512 VNNI
  • Kernels read only within the blob (codes plus metadata) and dim floats plus metadata of the query.
  • Integer SQ4-SQ4 paths fall back to scalar above SQ4_MAX_EXACT_SIMD_DIM.

Median ns per distance at D768, single thread, SQ4 vs SQ8 on the same ISA:

ARM NEON ARM SVE x86 AVX-512
FP32-query IP 108 vs 126 100 vs 80 52 vs 40
FP32-query L2 143 vs 133 138 vs 117 73 vs 51
SQ4-SQ4 IP 32 vs 33 22 vs 18 14 vs 26
SQ4-SQ4 L2 33 vs 35 23 vs 18 14 vs 29

Hosts: Neoverse-V2 (SVE vector length 128) and a Xeon 8375C. Results are medians of 5 runs on lightly loaded shared hosts, and are indicative only.

The SQ4-SQ4 L2 row was measured with the identity kernels, which are now the only SQ4-SQ4 L2 kernels.

Benchmarks

bm_spaces_sq4_fp32 and bm_spaces_sq4_sq4 are registered in benchmarks.sh and the CI benchmark matrix. An HNSW SQ4 index benchmark is left for later, because it needs an uploaded fixture.

Main risks and how they are tested

  • Reading past the vector. SQ4 has no padding after the codes, so tails use masked loads (AVX-512 masks, SVE predicates) or lane loads (NEON, AVX2). Guard-page tests place each blob and query against an unmapped page.
  • Overflow. The integer SQ4-SQ4 paths are exact up to SQ4_MAX_EXACT_SIMD_DIM (derivation in spaces.h), above which the chooser returns the scalar kernel, whose 64-bit sum is exact. The FP32-query IP kernels behave like SQ8's for extreme query values.
  • Wrong nibble order. Negative controls swap a nibble or shift the metadata and require the error to exceed the tolerance by 100x.

Commits

  1. SQ4: Add NEON, SVE, AVX2 and AVX-512 SQ4 distance kernels: kernel headers, choosers in spaces/functions/*, dispatch in IP_space.cpp / L2_space.cpp, and the per-tier tests.
  2. SQ4: Add SQ4 spaces microbenchmarks: bm_spaces_sq4_fp32, bm_spaces_sq4_sq4, benchmarks.sh and the CI benchmark matrix.

Testing

  • Not yet built at this head (cdc35b35); CI checks it, and a host run on ARM and x86 follows after the review. An earlier version of these kernels, with the plain-L2 reconstruction kernels and the IP overflow fallback, passed full ctest on x86 (3396/3396) and ARM (3353/3353).
  • Per-tier SIMD tests check every tier against scalar over dims 1-130 and 767-769, at blob offsets 0-7 and with a query offset, plus the guard-page, large-offset FP32-query L2 and dispatch-cap tests above. Each tier is reached through the dispatcher by clearing the feature flags of the tiers above it.

🤖 Generated with Claude Code


Note

Medium Risk
Changes hot-path distance math for quantized indices; wrong SIMD tails or integer overflow would skew search results, though dispatch bounds and broad per-tier tests mitigate this.

Overview
Adds SIMD distance kernels for 4-bit quantized vectors (SQ4-FP32 and SQ4-SQ4) on inner product and L2, matching the existing SQ8 dispatch pattern.

Dispatch in IP_space.cpp / L2_space.cpp no longer always returns scalar: it selects NEON / SVE / SVE2, AVX2+FMA, or AVX-512 VNNI by CPU features, with dimension floors on x86 and scalar fallback above SQ4_MAX_EXACT_SIMD_DIM when the integer code-dot would overflow uint32.

New kernel headers implement nibble unpacking, masked/predicated tails, and L2 via the stored-sum identity (reusing IP code-dot). Stored_InnerProduct_FromCodeDot is factored out of quantized_kernels.h for shared SIMD/scalar completion.

CI/benchmarks: bm-spaces-sq4-fp32 and bm-spaces-sq4-sq4 in the workflow matrix; microbenchmark binaries and test_sq4_distances tier tests (scalar parity, guard pages, corruption controls, exact-dim cap).

Reviewed by Cursor Bugbot for commit 4fdb250. Bugbot is set up for automated code reviews on this repo. Configure here.

@codecov

codecov Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 99.35691% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 97.52%. Comparing base (e01beba) to head (4fdb250).

Files with missing lines Patch % Lines
src/VecSim/spaces/IP_space.cpp 94.44% 1 Missing ⚠️
src/VecSim/spaces/L2_space.cpp 94.44% 1 Missing ⚠️
Additional details and impacted files
@@                       Coverage Diff                        @@
##           codex/sq4-production-backend    #1103      +/-   ##
================================================================
+ Coverage                         97.46%   97.52%   +0.05%     
================================================================
  Files                               148      157       +9     
  Lines                              9715    10012     +297     
================================================================
+ Hits                               9469     9764     +295     
- Misses                              246      248       +2     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

dor-forer and others added 2 commits November 8, 2026 10:50
FP32-query IP and L2 on NEON, SVE/SVE2, AVX2_FMA and AVX512F_BW_VL_VNNI;
SQ4-SQ4 IP, L2 and the centered L2 identity on NEON, NEON_DOTPROD,
SVE/SVE2 and AVX512F_BW_VL_VNNI. Tails use masked or lane loads, so no
kernel reads past the packed vector. The FP32-query IP kernels fall back
to the scalar reconstruction on a non-finite result, and the integer
SQ4-SQ4 paths are bounded by SQ4_MAX_EXACT_SIMD_DIM. Each tier is tested
against the scalar kernel, including guard-page and overflow cases.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer dor-forer mentioned this pull request Nov 8, 2026
2 tasks done

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit b92b87c. Configure here.

dor-forer and others added 2 commits November 8, 2026 18:43
The AVX2 SQ4-FP32 IP and L2 tails read the last query floats with
maskload. Masked-out lanes of VMASKMOVPS may fault on some processors, so
an exact-size query could fault at a page boundary. Load the last full
8-float window that ends at the last coordinate instead, shift the tail
codes into the matching top lanes and zero the lanes below. A dim below 8,
which the dispatcher never sends here, copies the floats.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	tests/unit/test_sq4_distances.cpp
GuyAv46
GuyAv46 previously approved these changes Nov 9, 2026
Comment thread src/VecSim/spaces/IP_space.cpp Resolved
Add AVX2 stored-to-stored IP and L2 kernels with the existing exact-dimension fallback and VNNI precedence. Share the stored-IP metadata formula across scalar and SQ4 SIMD kernels while preserving its operation order.
dor-forer and others added 4 commits November 9, 2026 23:44
Revert 03a445f. On Neoverse V2 with GCC 12.3, the comparison bound regresses SVE2 FP32 IP at D768 by 1.26-1.28% and L2 at D770 by 2.39-2.41% in both link orders. Short-vector gains do not establish a general replacement.
The NEON and SVE SQ4-FP32 kernels handle every dim, and their tails read
only the codes, the query floats and metadata. Drop the dim < 16 scalar
gate and test the ARM tiers from dim 0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer
dor-forer requested a review from GuyAv46 November 10, 2026 10:35
@dor-forer
dor-forer added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit f2ddf8e Nov 10, 2026
34 of 36 checks passed
@dor-forer
dor-forer deleted the codex/sq4-simd branch November 10, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants