Skip to content

Rename Weaviate reserved property names consistently - #360

Open
inamdarmihir wants to merge 1 commit into
qdrant:masterfrom
inamdarmihir:fix/229-weaviate-reserved-properties
Open

inamdarmihir wants to merge 1 commit into
qdrant:masterfrom
inamdarmihir:fix/229-weaviate-reserved-properties

Conversation

@inamdarmihir

Copy link
Copy Markdown

Weaviate 1.29 refuses to create properties named id, _id or _additional ("is a reserved property name", HTTP 422), but the Weaviate client creates one property for every field in the dataset schema, so any dataset with such a field fails at collection creation. #229 asks the tool to handle this. arxiv-titles-384-angular-filters declares an id field, for example.

This adds engine/clients/weaviate/properties.py with one helper, property_name(), and routes every place the client names a property through it: the schema in configure.py, the object payload in upload.py, and all three filter builders (exact match, range, geo) in parser.py. A reserved name is stored as prop_<name>; every other name is untouched.

I chose renaming over skipping because skipping id would drop a field that filtered datasets query on, so the filter would match nothing or error. Renaming keeps those filters working. Because the schema, the objects and the filters all go through the same function, they cannot drift apart. The reserved list (_additional, _id, id) comes from reservedPropertyNames in Weaviate's entities/schema/validation.go at v1.29.1 and main. rename_properties raises ValueError if a rename would collide with an existing key.

Tests (tests/engine/clients/weaviate/test_weaviate_properties.py, no server needed): ordinary names including Id, ID, id_, _idx are untouched; each reserved name maps to a non-reserved one; empty schema and empty/None metadata; input dicts are not mutated; collisions raise; an empty upload batch makes no insert call; configure, upload and parser agree on the property name.

pytest tests/engine/clients/weaviate   # 33 passed
black / isort --profile black          # clean

Not done: no Docker daemon was available, so I did not run this against a real Weaviate container. Weaviate main also rejects names ending in internal suffixes like _searchable; that is not a 1.29 rule and no dataset uses such names, so it is not handled. The prop_ prefix is arbitrary and easy to change.

Fixes #229

Weaviate rejects properties named id, _id and _additional with 422, which broke collection creation for datasets with an `id` field (e.g. arxiv-titles-384-angular-filters). Store such fields under a prefixed name via a shared helper used by the configurator, the uploader and the filter parser, so filters on them keep working.

Fixes qdrant#229
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

weaviate 1.29: tool should ignore properties that are reserved property names

1 participant