Repository navigation
Reproduce Lucene 10.5.2's tokenizer in uax29, term for term - #83
Merged
Merged
Conversation
The reference tokenizer starts an emoji token on every code point with the Extended_Pictographic property, and FiQA's index holds 34 of them that this library dropped. The rule is the property, not the OtherSymbol category: the same corpus holds 143 other So characters (degree sign, black square, numero) that the reference drops, so a category rule would add terms on the other side. The tables are pinned to Unicode 12.1 because the reference grammar declares that version; later revisions reassign code points it already classifies. FiQA: 32 documents differing -> 8; missing 50 -> 16, extra 11 unchanged; total terms 5288596 -> 5288630.
The reference grammar's IDEOGRAPHIC rule emits a single CJK ideograph per token, so a run of them is not a word. FiQA holds nine ideographs; joining 理财 and 雙元貨幣產品 lost eight terms and invented two. The rule is Script=Han minus its word letters, not a block range: U+3005 and U+303B are Word_Break=ALetter and the reference keeps them inside an alphanumeric run, which this library's letter scan already did. Katakana and Hangul runs are untouched. FiQA: 8 documents differing -> 5; missing 16 -> 6, extra 11 -> 8; total terms 5288630 -> 5288637.
The separator table joins a full stop between two letters, and it asks for the class of the characters on each side. A word ending in a combining mark put the mark on that side, and a mark is neither a letter nor a digit, so tɕɛ̝.bʌl split into two terms where the reference holds one. The class now skips the marks that modify the base character, which is what they are: tɕɛ̝ has the class of ɛ. FiQA: 5 documents differing -> 3; missing 6 -> 4, extra 8 -> 4; total terms 5288637 -> 5288635.
The reference's word-break data classes a byte-order mark, a left-to-right mark and a soft hyphen as Format, which a word continues through; the term keeps the character. FiQA holds 199<U+FEFF>0 and bank<U+200E>, both split here before this rule. U+200B is the exception that makes this a word-break rule rather than a category test: it is Format by category and Other by word-break class, and the reference splits on it. A format character still cannot open a word, and the separator class lookup now reads past one, as it reads past a combining mark. FiQA: 3 documents differing -> 1; missing 4 -> 2, extra 4 -> 1; total terms 5288635 -> 5288634.
The reference tokenizer emits a token when its 255-unit input buffer fills and continues the rest as a new token, so FiQA's 468-character blob is indexed as one term of 255 and one of 213. The limit is the scanner's buffer width, so it belongs to the mode that reproduces that scanner: the identifiers analysis runs through the same joining scan and keeps a long identifier whole, pinned by the new scope tests alongside emoji, ideographs and format characters. A cut that would land inside a surrogate pair moves back one unit, matching the scanner's hold-back of a trailing high surrogate. FiQA: 1 document differing -> 0; missing 2 -> 0, extra 1 -> 0; total terms 5288634 -> 5288635, identical to the Lucene 10.5.2 dump.
U+202F is Word_Break=ExtendNumLet in the reference's Unicode 12.1 data, not a separator: it joins on both sides, and a leading one belongs to the token that follows, so FiQA's section sign + U+202F + 1950.6 is one term where this library had 1950.6 alone. The residual was invisible to the plan's diff, which splits on any Unicode whitespace and splits U+202F away. Comparing on ASCII spaces only — the dumps join terms with one — finds it, and the two files are now byte-identical. Only U+202F is handled: it is the only ExtendNumLet member the corpora exercise, and the rest are left until a document needs them. FiQA: 1 document differing -> 0, missing 1 -> 0, extra 1 -> 0; total terms unchanged at 5288635.
FiQA's index is now the reference's: 5,288,635 terms and a per-document stream byte-identical to Lucene 10.5.2's EnglishAnalyzer. The evaluation and backend pages say so, the changelog carries the rules and the one knowingly unimplemented case (multi-code-point emoji sequences, absent from FiQA).
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #83 +/- ##
==========================================
+ Coverage 89.26% 89.28% +0.02%
==========================================
Files 270 271 +1
Lines 21383 21461 +78
Branches 4621 4635 +14
==========================================
+ Hits 19088 19162 +74
+ Misses 1374 1371 -3
- Partials 921 928 +7 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The emoji, ideograph, format-character and ExtendNumLet rules had grown the scan to 129 lines and a complexity CodeFactor flags. Each step is now a named method -- standalone tokens, whether a word begins, leading joiners, the word run -- with the comments moved to the rule they explain. No behaviour change: the FiQA term dump is still byte-identical to Lucene 10.5.2's.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
uax29now produces, for FiQA-2018, exactly the term stream Lucene 10.5.2'sEnglishAnalyzerproduces — byte-identical, all 57,638 documents. FiQA's index holds the published 5,288,635 terms instead of 5,288,596, and the aligned BM25 row is computed on the reference's index rather than on a close one.Evidence
The comparison is a dump of both token streams,
cmpexits 0.--verify-referencestays green: NFCorpus/SciFact/ArguAna fingerprints 637,485 / 838,128 / 969,528 and aligned scores 0.3218 / 0.6789 / 0.3970 unchanged. Full suite with the integration runsettings: 2,999 passed, 32 skipped, 0 failed.api-docs checkgreen; no public API change.The rules, one commit each
So: FiQA has 143 non-emojiSocharacters the reference drops, and the reference starts an emoji token onExtended_Pictographiconly.Script=Hanword letters (U+3005, U+303B).tɕɛ̝.bʌlstays one token.Word_Break=Format/ZWJ in 12.1 plus U+200C) stay inside the word; U+200B isOtherand still splits, as the reference does.ExtendNumLet), the residual the whitespace-based diff had hidden.All of it is gated to
WordSegmentation.UnicodeWordBoundaries;default,porter,englishandidentifiersare unchanged, pinned byAnalysisScopeTests.Known limits, stated
ExtendNumLetis implemented for U+202F only, the member the corpora exercise; the others are left alone rather than extended speculatively.Docs
docs/evaluation.mdanddocs/backends.mdnow say the FiQA index is the reference's, not merely a close one; the changelog carries the rules and the one unimplemented case.