Repository navigation
Conversation
…erDecoder FlexBufferDecoder.decode_map always decoded the keys vector of a map with 1-byte offsets. A FlexBuffers map header stores the byte width of its keys vector next to the keys offset, and the builder widens it to 2 bytes once a key string is more than 255 bytes before the vector, for example with long or many keys. Such a map was then decoded into wrong keys or raised UnicodeDecodeError. Read the keys byte width from the map header and use it for the key offsets. The custom options of TFLite_Detection_PostProcess, the only current caller, keep a 1-byte keys vector and decode as before.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
FlexBufferDecoder.decode_mapdecoded the keys vector of a map with 1-byte offsets no matter what the buffer said. A FlexBuffers map header stores the byte width of its keys vector right after the keys offset. The FlexBuffers builder widens that vector to 2 bytes once a key string sits more than 255 bytes before it, which happens with long keys or a few dozen short ones. For such a map the decoder read wrong key offsets and returned garbage keys, or raisedUnicodeDecodeError.This change reads the keys byte width from the map header and uses it for the key offsets. The custom options of
TFLite_Detection_PostProcess, the only current caller, keep a 1-byte keys vector, so they decode exactly as before. That was checked with the standard 11 options and with_output_quantizedand_support_output_type_float_in_quantized_opadded.The new
test_flexbuffer_decoder_wide_key_vectorintests/python/relax/test_frontend_tflite.pyencodes two flat maps withflatbuffers.flexbuffers, one with long keys and one with 40 keys, and checks that they decode back to the original dict.libtvmwas not built locally, so the test body was run as is, withtflite_flexbuffer.pyloaded by path (the module only needs the standard library). Without the fix both cases fail, one withUnicodeDecodeErrorand one with wrong keys. With the fix both pass. A wider comparison againstflexbuffers.Loadsalso matches, covering maps of up to 200 entries and mixed int, float and bool values.pre-commit run --fileson both files passes (ruff check and ruff format). The repository-wide ASF header and file type hooks were skipped in the sparse checkout used here.