Skip to content

Reproduce Lucene 10.5.2's tokenizer in uax29, term for term - #83

Merged
manuc66 merged 8 commits into
mainfrom
feat/uax29-lucene-parity
Nov 17, 2026
Merged

manuc66 merged 8 commits into
mainfrom
feat/uax29-lucene-parity

Conversation

@manuc66

@manuc66 manuc66 commented Nov 17, 2026

Copy link
Copy Markdown
Owner

What this adds

uax29 now produces, for FiQA-2018, exactly the term stream Lucene 10.5.2's EnglishAnalyzer produces — byte-identical, all 57,638 documents. FiQA's index holds the published 5,288,635 terms instead of 5,288,596, and the aligned BM25 row is computed on the reference's index rather than on a close one.

Evidence

before after
documents differing from Lucene 32 (33 under an exact comparison) 0
missing / extra terms 50 / 11 0 / 0
total terms 5,288,596 5,288,635
FiQA aligned BM25 0.236 (unchanged) 0.236, index identical to the 0.2361 reference

The comparison is a dump of both token streams, cmp exits 0. --verify-reference stays green: NFCorpus/SciFact/ArguAna fingerprints 637,485 / 838,128 / 969,528 and aligned scores 0.3218 / 0.6789 / 0.3970 unchanged. Full suite with the integration runsettings: 2,999 passed, 32 skipped, 0 failed. api-docs check green; no public API change.

The rules, one commit each

  1. Extended_Pictographic symbols as terms (Unicode 12.1, the revision the reference grammar declares). Not every So: FiQA has 143 non-emoji So characters the reference drops, and the reference starts an emoji token on Extended_Pictographic only.
  2. One term per Han ideograph, excluding the two Script=Han word letters (U+3005, U+303B).
  3. Word class read past combining marks, so tɕɛ̝.bʌl stays one token.
  4. Format characters (Word_Break=Format/ZWJ in 12.1 plus U+200C) stay inside the word; U+200B is Other and still splits, as the reference does.
  5. 255-unit token chop, surrogate-safe (Lucene holds a trailing high surrogate back).
  6. U+202F as a word extender (ExtendNumLet), the residual the whitespace-based diff had hidden.

All of it is gated to WordSegmentation.UnicodeWordBoundaries; default, porter, english and identifiers are unchanged, pinned by AnalysisScopeTests.

Known limits, stated

  • Multi-code-point emoji sequences (ZWJ families, flags, keycaps, skin-tone modifiers) are not joined; FiQA contains no such code point, and a test documents the divergence.
  • ExtendNumLet is implemented for U+202F only, the member the corpora exercise; the others are left alone rather than extended speculatively.
  • Hiragana is not split per character (Lucene does); FiQA contains none, and Katakana/Hangul runs match.
  • The tables are pinned to Unicode 12.1 on purpose: that is what the reference grammar declares.

Docs

docs/evaluation.md and docs/backends.md now say the FiQA index is the reference's, not merely a close one; the changelog carries the rules and the one unimplemented case.

The reference tokenizer starts an emoji token on every code point with the
Extended_Pictographic property, and FiQA's index holds 34 of them that this
library dropped. The rule is the property, not the OtherSymbol category: the
same corpus holds 143 other So characters (degree sign, black square, numero)
that the reference drops, so a category rule would add terms on the other side.

The tables are pinned to Unicode 12.1 because the reference grammar declares
that version; later revisions reassign code points it already classifies.

FiQA: 32 documents differing -> 8; missing 50 -> 16, extra 11 unchanged;
total terms 5288596 -> 5288630.
The reference grammar's IDEOGRAPHIC rule emits a single CJK ideograph per
token, so a run of them is not a word. FiQA holds nine ideographs; joining
理财 and 雙元貨幣產品 lost eight terms and invented two.

The rule is Script=Han minus its word letters, not a block range: U+3005 and
U+303B are Word_Break=ALetter and the reference keeps them inside an
alphanumeric run, which this library's letter scan already did. Katakana and
Hangul runs are untouched.

FiQA: 8 documents differing -> 5; missing 16 -> 6, extra 11 -> 8;
total terms 5288630 -> 5288637.
The separator table joins a full stop between two letters, and it asks for the
class of the characters on each side. A word ending in a combining mark put the
mark on that side, and a mark is neither a letter nor a digit, so tɕɛ̝.bʌl split
into two terms where the reference holds one.

The class now skips the marks that modify the base character, which is what they
are: tɕɛ̝ has the class of ɛ. FiQA: 5 documents differing -> 3; missing 6 -> 4,
extra 8 -> 4; total terms 5288637 -> 5288635.
The reference's word-break data classes a byte-order mark, a left-to-right mark
and a soft hyphen as Format, which a word continues through; the term keeps the
character. FiQA holds 199<U+FEFF>0 and bank<U+200E>, both split here before this
rule.

U+200B is the exception that makes this a word-break rule rather than a
category test: it is Format by category and Other by word-break class, and the
reference splits on it. A format character still cannot open a word, and the
separator class lookup now reads past one, as it reads past a combining mark.

FiQA: 3 documents differing -> 1; missing 4 -> 2, extra 4 -> 1;
total terms 5288635 -> 5288634.
The reference tokenizer emits a token when its 255-unit input buffer fills and
continues the rest as a new token, so FiQA's 468-character blob is indexed as
one term of 255 and one of 213. The limit is the scanner's buffer width, so it
belongs to the mode that reproduces that scanner: the identifiers analysis runs
through the same joining scan and keeps a long identifier whole, pinned by the
new scope tests alongside emoji, ideographs and format characters.

A cut that would land inside a surrogate pair moves back one unit, matching the
scanner's hold-back of a trailing high surrogate.

FiQA: 1 document differing -> 0; missing 2 -> 0, extra 1 -> 0;
total terms 5288634 -> 5288635, identical to the Lucene 10.5.2 dump.
U+202F is Word_Break=ExtendNumLet in the reference's Unicode 12.1 data, not a
separator: it joins on both sides, and a leading one belongs to the token that
follows, so FiQA's section sign + U+202F + 1950.6 is one term where this
library had 1950.6 alone.

The residual was invisible to the plan's diff, which splits on any Unicode
whitespace and splits U+202F away. Comparing on ASCII spaces only — the dumps
join terms with one — finds it, and the two files are now byte-identical.

Only U+202F is handled: it is the only ExtendNumLet member the corpora
exercise, and the rest are left until a document needs them.

FiQA: 1 document differing -> 0, missing 1 -> 0, extra 1 -> 0; total terms
unchanged at 5288635.
FiQA's index is now the reference's: 5,288,635 terms and a per-document
stream byte-identical to Lucene 10.5.2's EnglishAnalyzer. The evaluation and
backend pages say so, the changelog carries the rules and the one knowingly
unimplemented case (multi-code-point emoji sequences, absent from FiQA).
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 84.87395% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 89.28%. Comparing base (cb9bedd) to head (10777a8).

Files with missing lines Patch % Lines
src/LexiSharp/Linguistics/Tokenizer.cs 76.31% 6 Missing and 12 partials ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main      #83      +/-   ##
==========================================
+ Coverage   89.26%   89.28%   +0.02%     
==========================================
  Files         270      271       +1     
  Lines       21383    21461      +78     
  Branches     4621     4635      +14     
==========================================
+ Hits        19088    19162      +74     
+ Misses       1374     1371       -3     
- Partials      921      928       +7     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

The emoji, ideograph, format-character and ExtendNumLet rules had grown the
scan to 129 lines and a complexity CodeFactor flags. Each step is now a named
method -- standalone tokens, whether a word begins, leading joiners, the word
run -- with the comments moved to the rule they explain. No behaviour change:
the FiQA term dump is still byte-identical to Lucene 10.5.2's.
@manuc66
manuc66 merged commit 1f145cd into main Nov 17, 2026
17 checks passed
@manuc66
manuc66 deleted the feat/uax29-lucene-parity branch November 17, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants