Repository navigation
MDEV-41725 cset_narrowing wrong result for non-general_ci utf8mb4 constant - #5981
Open
Omar-Abdo1 wants to merge 1 commit into
Open
Omar-Abdo1 wants to merge 1 commit into
Omar-Abdo1 wants to merge 1 commit into
Conversation
gkodinov
added
the
External Contribution
gkodinov
requested changes
Oct 26, 2026
gkodinov
left a comment
gkodinov
left a comment
Member
There was a problem hiding this comment.
Thank you for your contribution! This is a preliminary review.
Please address the test suite issue. There's also an optional suggestion on re-using some of the code already there.
Omar-Abdo1
force-pushed
the
MDEV-41725-cset-narrowing
branch
2 times, most recently
from
October 26, 2026 13:37
cf85bc9 to
5161c2a
Compare
Omar-Abdo1
force-pushed
the
MDEV-41725-cset-narrowing
branch
from
October 26, 2026 17:54
5161c2a to
fb0ac05
Compare
gkodinov
approved these changes
Oct 27, 2026
gkodinov
left a comment
gkodinov
left a comment
Member
There was a problem hiding this comment.
LGTM. Please stand by for the final review.
…nstant Problem: With cset_narrowing=on, Utf8_narrow::should_do_narrowing() only allowed narrowing for the utf8mb4_general_ci collation. When comparing a utf8mb3 key column with a utf8mb4 constant in any other collation (like utf8mb4_unicode_ci), narrowing was skipped. This caused 4-byte characters to be stored in the index key as '?' instead of U+FFFD, resulting in incorrect empty results. Fix: Allow narrowing for constants in any utf8mb4 collation by comparing the character set name with MY_UTF8MB4. The field must still be utf8mb3_general_ci.
Omar-Abdo1
force-pushed
the
MDEV-41725-cset-narrowing
branch
from
October 28, 2026 14:21
fb0ac05 to
f0671d8
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://jira.mariadb.org/browse/MDEV-41725
Problem
With optimizer_switch='cset_narrowing=on', a utf8mb3_general_ci key column
compared with a utf8mb4 constant can still use its index. To build the
lookup key, the constant is "narrowed": every 4-byte character is replaced
by U+FFFD, which is how utf8mb4_general_ci treats it.
Whether to narrow is decided by Utf8_narrow::should_do_narrowing(). When the
ref key is built (store_key_item), it is called with the constant's own
collation, and it required exactly utf8mb4_general_ci. The check that decides
whether the index may be used (field.cc) uses the comparison collation
instead. With a connection collation of utf8mb4_unicode_ci and a column
compared as utf8mb4_general_ci, the two checks disagree: the index is used,
but narrowing is skipped. The 4-byte character is then written into the key
as '?', the lookup finds no row, and the optimizer reports
"Impossible WHERE noticed after reading const tables". The query returns an
empty result although a matching row exists.
Fix
should_do_narrowing() now accepts a constant in any utf8mb4 collation
(the charset name is compared with "utf8mb4"). The key column must still be
utf8mb3_general_ci and the optimizer switch must still be on. Other
charsets that have a 4-byte maximum length (utf16, utf32, gb18030) are not
affected.
Testing
New test: mysql-test/main/mdev_41725.test
row; before the fix it was "Impossible WHERE" and returned nothing.
utf8mb4_bin and utf8mb4_general_ci constants return the same result with
the index and with IGNORE INDEX.
Also run and passing: main.cset_narrowing, main.ctype_collate,
main.ctype_uca, main.range, main.ctype_utf8mb4, main.ctype_utf8, main.join,
main.select.