Skip to content

MOD-19169: Quantize SQ8 IP and Cosine queries once to 8 bits on AVX-512 VNNI - #1128

Closed
dor-forer wants to merge 3 commits into
mainfrom
dor-forer-MOD-19169-a8-query
Closed

dor-forer wants to merge 3 commits into
mainfrom
dor-forer-MOD-19169-a8-query

Conversation

@dor-forer

Copy link
Copy Markdown
Collaborator

Describe the changes in the pull request

SQ8 HNSW indexes with IP or Cosine now quantize each query once to signed 8-bit codes ("A8"),
with their own offset and step. Each distance is then one u8 x s8 integer dot product
(vpdpbusd, 4 accumulators) plus a short double-precision epilogue. This applies to searches
and to the searches inside graph inserts, for FP32 and FP16 input, with and without the learned
mean.

It is active only when the CPU has AVX-512 F/BW/VL/VNNI and 768 <= dim <= 65,793 (the upper
bound keeps the int32 dot product exact). Everywhere else (smaller dims, CPUs without VNNI,
ARM, L2, SQ4) the components and distances are exactly those of main. The stored format does
not change, so there is no serializer change.

Results on dorer-intel (Ice Lake Xeon 8375C), DBpedia 768, 99,000 base vectors, 1,000 held-out
queries, M=64, efConstruction=512, exact FP32 ground truth:

Mean ef Recall@10 main Recall@10 A8 QPS main QPS A8
off 100 99.25% 99.13% 2,615 4,112 (+57%)
off 200 99.28% 99.17% 1,411 2,247 (+59%)
on 100 99.40% 99.45% 2,592 3,776 (+46%)
on 200 99.48% 99.50% 1,447 2,098 (+45%)

The worst recall@10 drop at ef 50 to 400 is 0.12 points (mean off). QPS is single-threaded and
pinned, median of 3 runs; the main-vs-main noise floor is 0.2% to 3.3%. FP16 queries with the
mean change recall by -0.02 to +0.04 points and gain about 75% QPS.

How it works:

  • IP_SQ8_A8_GetDistFunc returns the VNNI kernel or nullptr; the factory uses that as its only
    switch, so the A8 path is chosen when the index is built or loaded.
  • SQ8A8QueryPreprocessor encodes the query (vectorized AVX-512F encoder for FP32 and FP16
    input, bit-identical codes to the scalar encoder). With a mean it centers the query and writes
    <mean, y> in the same pass; a mean installed later through setMean applies to later queries.
  • SQ8A8QueryDistanceCalculatorWithNorm applies the same mean correction as the stored-to-stored
    path. The new classes add no data members, and static_asserts keep the memory estimates valid.
  • A scalar A8 kernel is the test oracle only; all tiers share one epilogue.

Behaviour to be aware of:

  • Range queries and GetDistanceFrom return A8 estimates, whose error is somewhat larger than
    today's asymmetric path.
  • An index saved on a host without VNNI and loaded on one with it (or the reverse) switches query
    path. The stored bytes are identical; scores differ slightly.

Tests: scalar A8 vs an exact reference; SIMD vs scalar bit-identical for every dim % 64 residual
and the gate edges; encoder bit-identity; component class choice above and below the gate; HNSW
top-K, range, batch iterator, multi-value, GetDistanceFrom and tiered with a late mean, FP32
and FP16, at dims 768, 777 and 1024. The A8 tests skip on hosts without VNNI.

Which issues this PR fixes

  1. MOD-19169

Main objects this PR modified

  1. spaces/IP/IP_AVX512F_BW_VL_VNNI_SQ8_A8.h, spaces/IP/IP_AVX512F_SQ8_A8_Encoder.h,
    spaces/sq8_a8_query_encoder.h, types/sq8_a8.h (new)
  2. index_factories/components/components_factory.h, spaces/computer/preprocessors.h,
    spaces/computer/calculator.h, spaces/IP_space.cpp

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

🤖 Generated with Claude Code

dor-forer and others added 3 commits November 17, 2026 15:59
…eries

An SQ8 IP or Cosine query can be quantized once to signed 8-bit codes on its own grid, so
each distance becomes one uint8 x int8 integer dot product plus a short double epilogue that
folds the query's rounding error into per-query constants.

- types/sq8_a8.h: the query blob layout (int8 codes, zero padding to 8 bytes, QD, QM, QS,
  QK and Y_MEAN_IP as double, P as int64).
- spaces/sq8_a8_query_encoder.h: the scalar encoder for FP32 and FP16 queries.
- quantized_distance_helpers.h: the shared double epilogue, with explicit fma so every tier
  rounds the same way.
- quantized_kernels.h, IP.h/.cpp: the scalar SQ8_A8_InnerProduct reference.
- IP_AVX512F_BW_VL_VNNI_SQ8_A8.h: the vpdpbusd kernel with 4 accumulators and a masked tail,
  dispatched on dim % 64.
- IP_SQ8_A8_GetDistFunc returns nullptr unless AVX-512 F/BW/VL/VNNI is available and
  SQ8_A8_MIN_DIM (768) <= dim <= SQ8_A8_MAX_EXACT_SIMD_DIM (65,793, where the int32 code dot
  stops being exact). Cosine indexes resolve to IP, so there is no Cosine chooser.
- test_spaces.cpp: encoder and scalar kernel against definitions in long double, the AVX-512
  kernel against the scalar one (exact integer dot, bit-identical floats) for every residual
  and the edge dims, and the chooser gates via arch_opt.

Nothing calls the chooser yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The A8 query encoder now runs vectorized and handles mean-centered indexes, so the factory can
quantize SQ8 IP queries once per query at a cost close to today's query preparation.

- sq8_a8_query_encoder.h: SQ8_A8_EncodeQuery takes an optional mean. With one, it encodes
  y - mean (the same FP32 subtraction the stored vectors get) and writes <mean, y> of the raw
  query to the Y_MEAN_IP slot. The three-argument form is unchanged.
- IP_AVX512F_SQ8_A8_Encoder.h: the AVX-512F encoder for FP32 and FP16 input (vcvtph2ps). Each
  lane does the scalar encoder's double operations in the same order, and the range uses the
  same min / max operand order, so the codes equal the scalar encoder's bit for bit, including
  NaN (code -128), infinities and constant queries. Only the summed constants may differ in the
  last bits.
- SQ8_A8_GetQueryEncoder<float|float16>(arch_opt) picks it when the CPU has AVX-512F, and the
  scalar encoder otherwise.
- test_spaces.cpp: AVX-512F against scalar for 14 dims and 17 kinds of query (ties at exact and
  inexact steps, NaN first, in a block and in the tail, infinities, +-FLT_MAX, subnormals), with
  and without a mean, FP32 and FP16; every finite FP16 value; the chooser; and the centered
  encoding against the encoding of y - mean.

Nothing calls the encoder yet.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
SQ8 IP and Cosine HNSW indexes now quantize FP32 and FP16 queries once to 8-bit A8 codes where
IP_SQ8_A8_GetDistFunc has a kernel (AVX-512 F/BW/VL/VNNI, 768 <= dim <= 65,793), for searches
and for graph inserts, with and without the learned mean. Elsewhere nothing changes: FP32 and
FP16 queries stay asymmetric, and L2 and SQ4 are untouched.

- components_factory.h: CreateQuantizedIndexComponents tries the A8 chooser first (SQ8 IP only,
  under if constexpr) and builds the A8 classes when it returns a kernel; otherwise it runs the
  existing asymmetric path unchanged. The shared assembly moved into
  AssembleQuantizedIndexComponents. An optional arch_opt (tests only) reaches the chooser.
- preprocessors.h: SQ8A8QueryPreprocessor derives from QuantPreprocessor, so applySQ8Mean's cast
  still holds. Storage blobs are the base's; the query is the A8 blob (64-byte aligned), encoded
  with the mean as it is when the query arrives, so a mean installed later applies. The encoder
  is chosen once in a function-local static.
- calculator.h: SQ8A8QueryDistanceCalculatorWithNorm derives from DistanceCalculatorWithNorm and
  applies base - x_mean_ip - y_mean_ip + mean_sum_squares to the A8 kernel, in double; the
  stored-to-stored path is the base's. Without a mean the A8 kernel is the plain query function.
- The new classes add no data members; static_asserts in EstimateInitialSize keep the estimates
  valid.
- Tests: the factory's choice above and below the gate and with the CPU forced off; blob layout,
  padding, alignment and ownership; distances against the scalar kernel and against a bound
  derived from the two quantization steps (SQ8A8DistanceTolerance); the deferred mean through
  applySQ8Mean; and the public API at dims 768, 777 (not a multiple of 64, so the tails run) and
  1024 (top-K, range, batch iterator, multi-value, GetDistanceFrom_Unsafe) for standalone indexes,
  and at 768 and 1024 for tiered indexes with a late mean.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer dor-forer closed this Nov 17, 2026
@dor-forer
dor-forer deleted the dor-forer-MOD-19169-a8-query branch November 17, 2026 15:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant