Skip to content

Raise Elasticsearch dense_vector dims limit to 4096 - #359

Open
inamdarmihir wants to merge 1 commit into
qdrant:masterfrom
inamdarmihir:fix/113-elastic-vector-limit
Open

inamdarmihir wants to merge 1 commit into
qdrant:masterfrom
inamdarmihir:fix/113-elastic-vector-limit

Conversation

@inamdarmihir

Copy link
Copy Markdown

engine/clients/elasticsearch/configure.py refuses any dataset with vector_size > 2048 by raising IncompatibilityError. That limit was the Elasticsearch 8.10 dense_vector maximum (the comment links the 8.10 docs). The benchmark's server image is now elasticsearch:9.5.1, where dense_vector accepts up to 4096 dimensions, so 3072/4096-dim datasets are skipped for no reason. The blocker noted on #113 ("need to update the ES version") no longer applies.

This replaces the literal with a MAX_DENSE_VECTOR_DIMS = 4096 constant and updates the docs link.

Checked against Elastic's source: DenseVectorFieldMapper.MAX_DIMS_COUNT is 2048 at tag 8.10 and 4096 at 8.11, 8.12, 9.3.0 and main; the docs say it "can't exceed 4096". I did not check tag 9.5.1 itself. The OpenSearch client has its own, different limit (1024) and is untouched.

Tests (tests/engine/clients/elasticsearch/test_elasticsearch_configure.py, stubbed ES client, no server needed): 128, 2048, 2049 and 4096 dims are accepted; 4097 and 8192 raise IncompatibilityError without calling indices.create; the existing Distance.DOT rejection still works.

pytest tests/engine/clients/elasticsearch   # 8 passed

Fixes #113

dense_vector supports up to 4096 dims since Elasticsearch 8.11 (the pinned server image is 9.5.1), but the configurator rejected datasets above 2048. Use a named constant for the limit and update the docs link. Add unit tests for the boundary values.

Fixes qdrant#113
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Elastic vector limit should be 4096 instead of 2048

1 participant