Skip to content

SQ4: 4-bit scalar quantization for standalone HNSW - #1099

Merged
dor-forer merged 17 commits into
codex/quant-storage-typefrom
codex/sq4-production-backend
Nov 10, 2026
Merged

dor-forer merged 17 commits into
codex/quant-storage-typefrom
codex/sq4-production-backend

Conversation

@dor-forer

@dor-forer dor-forer commented Nov 3, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds VecSimQuant_SQ4 = 2, a packed 4-bit scalar quantizer for standalone HNSW. Stacked on #1101 (the StorageType refactor of the SQ8 pipeline); retarget to main after #1101 merges. SQ8 behavior is unchanged.

Distances in this PR are scalar only. The ARM and x86 SIMD kernels are in the follow-up PR stacked on this one, #1103, so this PR can be reviewed for the feature itself: storage layout, validation, recall and C API safety.

On a 10k DBpedia D768 sample, centered SQ4 has 93.6% exhaustive recall@10 (SQ8: 99.4%) at 400 bytes per vector (SQ8: 784).

What is supported

SQ4
Index Standalone HNSW, single and multi value
Input FP32 only
Metric IP and L2 (Cosine only when vectors are declared pre-normalized, resolved to IP)
Centering Optional mean via HNSWParams.quantParams
Query FP32 (asymmetric)
Save/load Existing HNSW V5 format (quantType, mean)
Python VecSimQuant_SQ4, quantization_mean accepted

The following are rejected, with NULL from NewIndex, SIZE_MAX from EstimateInitialSize, and no exception crossing the C API:

  • non-FP32 input
  • raw Cosine
  • tiered SQ4
  • unknown quant types
  • corrupt snapshots

Storage and distances

  • Storage is ceil(dim/2) code bytes (even coordinate in the low nibble, padding nibble zero), then the SQ8 metadata as unaligned FP32: min, delta, sum, then sum_sq for L2, then x_mean_ip for centered IP.

  • vecsim_types::sq4 (types/sq4.h) overrides the StorageType hooks from Generalize the scalar quantization pipeline over a StorageType #1101: max_code, data_bytes_count, load_code, store_code, and the code-dot loops code_dot and code_dot_fp32, which read each packed byte once and use both nibbles (the stored-to-stored sum is exact in 64-bit integers). The scalar kernels are the generic templates from Generalize the scalar quantization pipeline over a StorageType #1101 instantiated with sq4.

  • FP32-query IP is min*y_sum + delta*sum(code*y), the same identity as SQ8, with the same behavior for extreme query values (no SQ4-specific overflow fallback).

  • FP32-query L2 subtracts the offsets first, as SQ8 does.

  • SQ4-SQ4 IP uses SQ8's stored-sum identity.

  • SQ4-SQ4 L2 uses SQ8's stored-sum identity ||x||^2 + ||y||^2 - 2*IP(x, y), for centered and plain indexes alike.

  • Known limitation: on un-normalized data with a large common offset, plain (uncentered) SQ4 L2 loses up to 9.6 pp HNSW recall@100 with the identity compared with per-element reconstruction; centered SQ4 is unaffected. Use a mean for such L2 data. Tracked in MOD-20999.

  • Every SQ4 distance chooser returns alignment 0: the packed storage and its metadata have no alignment requirement.

  • Components are built with Generalize the scalar quantization pipeline over a StorageType #1101's generic CreateQuantizedIndexComponents<..., sq4>, on both the create and the load path.

Commits

  1. SQ4: Add packed 4-bit scalar quantization for standalone HNSW: the type, scalar distances, factory, preprocessing, serializer, tiered rejection and C++ tests.
  2. SQ4: Expose SQ4 in the Python bindings: the enum, quantization_mean, and the flow test.
  3. SQ4: Apply review decisions: generic components builder, no IP overflow fallback, packed code-dot loops and named nibble constants.

Testing

  • CI basic tests at 2c6c28f6: full unit ctest 3373/3373 passed (the Python step after it failed on a GitHub 504 while downloading SVS). The CI sanitizer job passed at the earlier head f9726022.
  • x86 host (Xeon 8375C, GCC 13.3, release): full ctest 3376/3376 at 3ffff81b, before the single L2 formula and the review decisions.
  • C++ tests: packing and odd dims, independent numeric references for every kernel, large-offset FP32-query L2, non-finite input, SQ8 byte identity, insert/query/range/batch/delete, save/load followed by mutation, duplicate labels, estimator vs measured memory, rejection through the C API, and corrupt-snapshot rejection.
  • Python flow tests and the 10k DBpedia smoke (IP and L2) were run on the combined tree with the SIMD kernels; the scalar kernels are the reference those tests compare against.

🤖 Generated with Claude Code


Note

Medium Risk
Touches HNSW index construction, serialization, and distance kernels—a correctness-sensitive path—but scope is gated by validation, tiered SQ4 is blocked, and coverage is heavy in new unit/flow tests.

Overview
Adds VecSimQuant_SQ4 and wires packed 4-bit scalar quantization through standalone HNSW: create, load/save (V5 quantType), memory estimation, and Python (VecSimQuant_SQ4, quantization_mean).

vecsim_types::sq4 packs two coordinates per byte with the same FP32 metadata layout as SQ8. Scalar L2/IP kernels cover SQ4↔FP32 (L2 uses per-residual accumulation) and SQ4↔SQ4 (generic quantized templates). Distance dispatch sets alignment 0 for packed blobs. QuantPreprocessor handles SQ4 packing; SQ4 L2 query blobs keep zeroed trailing FP32 slots instead of query sums.

Validation / limits: FP32 input and IP/L2 only (Cosine when pre-normalized → IP); tiered SQ4 is rejected; unknown quant types and bad snapshots fail without exceptions across the C API. SQ8 behavior is unchanged.

Reviewed by Cursor Bugbot for commit e01beba. Bugbot is set up for automated code reviews on this repo. Configure here.

@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from 36aa7d7 to d1499fb Compare November 4, 2026 08:50
@dor-forer
dor-forer changed the base branch from codex/hnsw-forward-is-normalized to main November 4, 2026 08:50
@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from d1499fb to 0bf8bdb Compare November 4, 2026 15:27
@dor-forer dor-forer changed the title SQ4: 4-bit scalar quantization for standalone HNSW SQ4: 4-bit scalar quantization for standalone HNSW with SIMD kernels Nov 4, 2026
@dor-forer
dor-forer changed the base branch from main to codex/quant-storage-type November 4, 2026 15:29
@dor-forer
dor-forer added this pull request to stack #1102 November 5, 2026 08:12
@dor-forer
dor-forer marked this pull request as ready for review November 5, 2026 08:48
@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from 168c7e6 to 971d296 Compare November 5, 2026 09:11
@codecov

codecov Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.46%. Comparing base (af9a7da) to head (e01beba).

Additional details and impacted files
@@                     Coverage Diff                      @@
##           codex/quant-storage-type    #1099      +/-   ##
============================================================
+ Coverage                     97.38%   97.46%   +0.08%     
============================================================
  Files                           147      148       +1     
  Lines                          9601     9715     +114     
============================================================
+ Hits                           9350     9469     +119     
+ Misses                          251      246       -5     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@cursor cursor Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.

Fix All in Cursor

❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, have a team admin enable autofix in the Cursor dashboard.

Reviewed by Cursor Bugbot for commit 971d296. Configure here.

Comment thread src/VecSim/spaces/IP/IP_SVE_SQ4_SQ4.h Resolved
@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from 403c48c to 3ffff81 Compare November 8, 2026 07:40
@dor-forer dor-forer changed the title SQ4: 4-bit scalar quantization for standalone HNSW with SIMD kernels SQ4: 4-bit scalar quantization for standalone HNSW Nov 8, 2026
@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from 3ffff81 to 979efab Compare November 8, 2026 08:06
dor-forer and others added 3 commits November 8, 2026 10:42
Adds VecSimQuant_SQ4: FP32 input, IP and L2 (Cosine on pre-normalized
vectors resolves to IP), optional mean centering, standalone HNSW only.
Codes are packed two per byte, followed by the SQ8 metadata. Distances
use the generic scalar kernels from the StorageType split; the FP32-query
IP falls back to per-element reconstruction on a non-finite result.
Unsupported configurations are rejected with NULL, and the existing V5
serializer stores the quant type and mean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Build SQ4 components through the generic CreateQuantizedIndexComponents
wrapper and drop the SQ4-specific factory overloads. Remove the SQ4-FP32
inner product overflow fallback so SQ4 returns the plain distance like
SQ8. Give sq4 packed code dot hooks that read each byte once, and name the
nibble shift bits_per_code.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer
dor-forer force-pushed the codex/sq4-production-backend branch from 979efab to f972602 Compare November 8, 2026 08:52
dor-forer and others added 2 commits November 8, 2026 11:41
The SQ4 mean correction x_mean_ip is summed from each code as it is
generated, in coordinate order with one double accumulator, instead of
in a second pass that re-reads and decodes the packed codes. The stored
bytes and the correction are unchanged. For SQ4 the input-side sum in
find_min_max, which was overwritten, is skipped.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer dor-forer mentioned this pull request Nov 8, 2026
2 tasks done
Keep two initialized FP32 padding slots for SIMD tails and update the query metadata contract and component checks.

Validation on dorer-intel: SQ4 81 passed, 12 expected skips; SQ8 209 passed.
Decode both nibbles from one load, preserve the residual and coordinate accumulation order, and handle odd tails explicitly.

Verify bitwise agreement with the previous scalar arithmetic across odd/even dimensions, nonzero padding, and large offsets.

Validation on dorer-intel: SQ4 82 passed with 12 expected skips; SQ8 209 passed. GCC 13.3 production assembly has one byte load per pair.
dor-forer and others added 3 commits November 8, 2026 15:43
Adapt SQ4 to the storage context components: the SQ4 create path passes
QuantizationInit, the initial-size estimate uses the SQ4 storage type,
sq4 reports its own quantization_type, and tiered creation keeps
rejecting SQ4 now that the generic quantization check is gone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The base reverted the storage context redesign, so this restores the
SQ4 code to its form on the original components API (the tree of
9f3f5b9) and drops the adaptations from the previous merge.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comment thread src/VecSim/types/sq4.h Fixed
The products are at most 225 and never overflowed, but multiplying in the
64-bit accumulator type keeps static analysis from flagging a widening
after a 32-bit multiply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
// padding for SIMD tails that load a final query pair when dim is odd.
const MetadataType padding[2] = {};
memcpy(metadata_dst, padding, sizeof(padding));
} else {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why? and why not in assign_query_metadata?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SQ8’s FP32-query L2 path also does not use these sums. Both kernels accumulate residual squares directly: we found precision loss from cancellation when expanding L2 into squared norms and a dot product. SQ8 preprocessing still computes the unused sums on this path, whereas SQ4 skips them.

The trailing slots currently remain as initialized padding, including for odd SVE query-pair tails in #1103. We opened MOD-21066 to benchmark and test skipping the unused calculations and removing the trailing fields, including any required tail-load changes, before deciding whether to remove them entirely.

Comment thread src/VecSim/types/sq4.h Outdated
namespace vecsim_types {

// Two scalar-quantized coordinates per byte, followed by the same metadata as SQ8.
struct sq4 : sq8 {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why?

Comment thread src/VecSim/types/sq4.h Outdated

static constexpr size_t data_bytes_count(size_t dim) { return dim / 2 + dim % 2; }

static uint8_t load_code(const uint8_t *data, size_t index) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe get()? consider a clearer name

Comment thread src/VecSim/types/sq4.h

// Dot product of two packed code arrays: sum(q1_i * q2_i). Each byte is loaded once and both
// nibbles are used. The 64-bit integer sum is exact for any dimension.
static float code_dot(const uint8_t *a, const uint8_t *b, size_t dim) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

consider mentioning in the comment that only naive uses this?

@dor-forer
dor-forer requested a review from GuyAv46 November 10, 2026 10:35
@dor-forer
dor-forer added this pull request to the merge queue Nov 10, 2026
Merged via the queue into main with commit a262916 Nov 10, 2026
16 checks passed
@dor-forer
dor-forer deleted the codex/sq4-production-backend branch November 10, 2026 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants