Repository navigation
Rename Weaviate reserved property names consistently - #360
Open
inamdarmihir wants to merge 1 commit into
Open
inamdarmihir wants to merge 1 commit into
inamdarmihir wants to merge 1 commit into
Conversation
Weaviate rejects properties named id, _id and _additional with 422, which broke collection creation for datasets with an `id` field (e.g. arxiv-titles-384-angular-filters). Store such fields under a prefixed name via a shared helper used by the configurator, the uploader and the filter parser, so filters on them keep working. Fixes qdrant#229
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Weaviate 1.29 refuses to create properties named
id,_idor_additional("is a reserved property name", HTTP 422), but the Weaviate client creates one property for every field in the dataset schema, so any dataset with such a field fails at collection creation. #229 asks the tool to handle this.arxiv-titles-384-angular-filtersdeclares anidfield, for example.This adds
engine/clients/weaviate/properties.pywith one helper,property_name(), and routes every place the client names a property through it: the schema inconfigure.py, the object payload inupload.py, and all three filter builders (exact match, range, geo) inparser.py. A reserved name is stored asprop_<name>; every other name is untouched.I chose renaming over skipping because skipping
idwould drop a field that filtered datasets query on, so the filter would match nothing or error. Renaming keeps those filters working. Because the schema, the objects and the filters all go through the same function, they cannot drift apart. The reserved list (_additional,_id,id) comes fromreservedPropertyNamesin Weaviate'sentities/schema/validation.goat v1.29.1 and main.rename_propertiesraisesValueErrorif a rename would collide with an existing key.Tests (
tests/engine/clients/weaviate/test_weaviate_properties.py, no server needed): ordinary names includingId,ID,id_,_idxare untouched; each reserved name maps to a non-reserved one; empty schema and empty/Nonemetadata; input dicts are not mutated; collisions raise; an empty upload batch makes no insert call; configure, upload and parser agree on the property name.Not done: no Docker daemon was available, so I did not run this against a real Weaviate container. Weaviate main also rejects names ending in internal suffixes like
_searchable; that is not a 1.29 rule and no dataset uses such names, so it is not handled. Theprop_prefix is arbitrary and easy to change.Fixes #229