[kamronis]

feat(index): add opt-in quantized vector refinement

Part of #8634.

Problem

Exact vector refinement reads original vectors from the dataset. On remote storage, these reads can dominate query
latency even though multi-bit RQ data is already available in the vector index.

Change

Add an explicit opt-in quantized refinement stage for flat IVF_RQ indexes.

When quantized_refine_factor is configured:

1. Overfetch factor * k candidates using Fast one-bit RQ scoring.
2. Preserve each candidate's IVF partition-local identity.
3. Rerank only those candidates using the index's Accurate stored RQ representation.
4. Return the best k results without reading original vectors for indexed candidates.

For distance_range queries, selected partitions use Accurate RQ search directly
so the range applies before top-k candidate truncation.

Exact refinement remains a separate optional final stage and can be combined with quantized refinement.

The option is exposed through:

• Rust: Scanner::quantized_refine
• Python: quantized_refine_factor
• Java: Query.Builder.setQuantizedRefineFactor

Scope

• Opt-in; default vector-search behavior is unchanged.
• Initially limited to flat IVF_RQ indexes.
• Unsupported index types and invalid factors return descriptive errors.

Validation

Added coverage for:

• RQ4 and RQ8 indexes
• L2, cosine, and dot-product metrics
• Multi-fragment datasets
• Quantized refinement followed by exact refinement
• Candidate identity and selected-candidate scoring
• NULL and non-finite vector cases
• Invalid and unsupported configurations
• Python and Java/JNI bindings

[lance-gatekeeper]

✅ Gate recommendation: approve.

The distance-range fix now applies bounds during Accurate RQ selection, before top-k truncation. Unbounded queries retain index-only Fast-to-Accurate refinement, scoped filtering, common-scale fallback merging, and per-query batch reranking; exact refinement remains separate. The Python signature also restores existing positional argument order.

[Xuanwo]

Approved after review and complete current-head validation. The bounded range path applies Accurate RQ bounds before top-k truncation while preserving segment restrictions, fallback scale handling, batch behavior, exact refinement, and the existing Python positional API.