Skip to content

TQ2/TQ3: Replace the dense rotation with a block Hadamard rotation - #1124

Closed
dor-forer wants to merge 1 commit into
codex/tq23-rabitq-draftfrom
dor-forer-tq-hadamard-rotation
Closed

dor-forer wants to merge 1 commit into
codex/tq23-rabitq-draftfrom
dor-forer-tq-hadamard-rotation

Conversation

@dor-forer

@dor-forer dor-forer commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Describe the changes in the pull request

Stacked on #1108. This replaces the dense random rotation in TQ2/TQ3 with a block Hadamard rotation. Each of three rounds applies a random sign flip, then a fast Walsh-Hadamard transform (FWHT) on each power-of-two block, then a random permutation.

  • Block size: the largest power of two that divides the dimension, capped at 1024. At dim 768 that is three blocks of 256.
  • Dense fallback: dimensions whose block would be under 64 (for example 96 or 100) keep the dense rotation unchanged.
  • One plain C++ loop: there are no SIMD kernels or dispatch. The loop only multiplies, adds and subtracts in a fixed order, so results are the same on every machine.
  • Determinism: the signs and permutations come from the existing seed, using only raw mt19937_64 output and 64-bit arithmetic.

Benchmark: dense is #1108 at bc5ed800 and block Hadamard is this PR at 04a42eb8, with the same benchmark source for both.

  • Data: DBpedia 768-dim, cosine through normalized IP; 100,000 base rows and 500 held-out queries; fixture checksum verified.
  • Setup: one pinned core, Release; Xeon 8375C (x86) and Neoverse V2 (ARM).
Dense Block Hadamard
Exhaustive recall@10, TQ2, mean (SD) of codec seeds 42, 1, 2, 3, 4 87.18% (0.36) 87.36% (0.21)
Exhaustive recall@10, TQ3, mean (SD) of the same seeds 92.82% (0.31) 92.76% (0.18)
Query prep, TQ2, x86 / ARM 641 / 61.5 us 6.5 / 5.2 us
Encode per vector, TQ2, x86 / ARM 695 / 434 us 25.7 / 23.0 us
Shared codec allocation, TQ2 2,366,016 B 22,256 B
Tiered HNSW index memory, TQ2 / TQ3 54.6 / 64.3 MB 52.2 / 62.0 MB
HNSW build of 100k rows, TQ2, x86 / ARM 291 / 92 s 156 / 41 s
HNSW whole query, TQ2, ef 40/80/160, x86 1.315 / 1.746 / 2.521 ms 0.657 / 1.081 / 1.853 ms
HNSW whole query, TQ2, ef 40/80/160, ARM 0.261 / 0.381 / 0.619 ms 0.185 / 0.308 / 0.548 ms
HNSW whole query, TQ3, ef 40/80/160, ARM 0.292 / 0.448 / 0.735 ms 0.231 / 0.387 / 0.669 ms
HNSW recall@10, TQ2, seed 42, ef 40/80/160 83.90 / 86.30 / 87.48% 83.72 / 86.12 / 86.88%
HNSW recall@10, TQ3, seed 42, ef 40/80/160 89.04 / 91.30 / 92.44% 88.32 / 90.76 / 92.06%
  • Recall: the 5-seed means are equal within one SD, and the block Hadamard rotation varies less across seeds.
  • HNSW recall: HNSW ran only at seed 42, which is the best of the five dense seeds, so its small gap is not a measured mean.
  • Speed: ARM is the fair speed comparison, because its dense path uses NEON (HNSW queries are 12 to 29% faster for TQ2 and 9 to 21% for TQ3, from ef 160 to ef 40). The x86 dense path is a scalar loop, so most of the x86 gain comes from removing it.
  • Plain loop vs the old kernels: query prep is 6.5 us against 5.4 us with the removed AVX-512 kernels, and 5.2 us against 4.7 us with NEON.
  • Cross-machine results: recall is identical on x86 and ARM. This version also returns exactly the same ids and scores as the earlier kernel version, checked on 200 top-10 results at dim 128 and dim 96.

Tests

  • test_hnsw_turboquant passes 29/29 on dorer-intel and dorer-arm-big in Release, and on dorer-arm-big in Debug.
  • The new tests cover:
    • the block rule and the dense fallback;
    • an independent double-precision Sylvester reference;
    • norm and inner-product preservation;
    • orthonormality and seed determinism;
    • encode and query using the same rotation;
    • save/load round trips;
    • loading a dense blob;
    • rejecting corrupt or truncated block Hadamard state.

Open items

  • Only dim 768 has measured recall. Block-64 dimensions such as 960 use the new rotation without a recall measurement.

Which issues this PR fixes

  1. None. This is a follow-up to Add TQ2/TQ3 as quantized StorageTypes #1108.

Main objects this PR modified

  1. src/VecSim/quantization/turboquant_rotation.h (new), turboquant.h and turboquant_components.h (memory estimate).
  2. tests/unit/test_hnsw_turboquant.cpp.

Mark if applicable

  • This PR introduces API changes
  • This PR introduces serialization changes

Serialization: the TQ fitted-state schema number is now the rotation kind.

  • Schema 1 (dense) is byte-for-byte the Add TQ2/TQ3 as quantized StorageTypes #1108 format.
  • Schema 2 stores the block Hadamard signs and permutations.
  • Loading validates the header, the sign magnitudes and the permutations, and rejects truncation, before it changes any state.

🤖 Generated with Claude Code

Use three rounds of [random sign flip, fast Walsh-Hadamard transform per
power-of-two block, random permutation] instead of the dense random
orthogonal matrix. The block is the largest power of two dividing the
dimension, capped at 1024; dimensions whose block would be under 64 keep
the dense rotation, generated from the same RNG stream as before.

* Add TurboQuantRotation, which owns either the dense matrix or the signs
  and permutations, generated portably from the existing seed with 64-bit
  arithmetic only. The transform is a plain scalar loop.
* The fitted state schema number is now the rotation kind: schema 1 (dense)
  is byte-for-byte the previous format; schema 2 stores the actual signs
  and permutations. Loading validates everything before replacing state.
* Size the components memory estimate for the rotation actually allocated.
* Add tests for the block rule, an independent reference, norm and inner
  product preservation, orthonormality, seed determinism, encode/query
  consistency, save/load round trips, legacy dense loads and corrupt input.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dor-forer

Copy link
Copy Markdown
Collaborator Author

Moved to RediSearch/RediSearch#12353, since VecSim now lives in RediSearch under src/vecsim.

@dor-forer dor-forer closed this Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant