Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 11 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,17 @@
against 25.7 / 50.0 and 265.0 / 284.4 ms with the gate off. `--keep` leaves the table for
`EXPLAIN`.

- **`uax29` now reproduces Lucene 10.5.2's `StandardTokenizer` term for term.** Under
`WordSegmentation.UnicodeWordBoundaries`, the tokenizer emits Extended_Pictographic symbols
(Unicode 12.1, the revision the reference grammar declares), one term per Han ideograph, keeps
combining marks and format characters inside the word, chops a token at the reference's 255-unit
buffer, and treats U+202F as a word extender. The other analyzers are unchanged, pinned by
`AnalysisScopeTests`. FiQA-2018's index now holds the reference's **5,288,635** terms and its
per-document term stream is **byte-identical** to Lucene 10.5.2's `EnglishAnalyzer`; the
NFCorpus/SciFact/ArguAna fingerprints and aligned scores are unchanged. Multi-code-point emoji
sequences (ZWJ, flags, keycaps, skin tones) are the one knowingly unimplemented rule — FiQA
contains none, and a test pins the divergence.

- **FiQA-2018 as a fourth selectable corpus** (`--dataset fiqa`; 57,638 documents, 648 test
queries). It is not part of `--dataset all` or the pinned comparison: it is the candidate gate's
evidence on a corpus of another nature (financial Q&A, short queries). At pool 1,000 the gate
Expand Down
3 changes: 2 additions & 1 deletion docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -389,7 +389,8 @@ financial Q&A with short queries) is not part of the pinned three. Under this ta
analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer uax29`, linear gains,
k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation
constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the
paper's 0.236: the figure is reproduced. This engine cannot use `uax29`; with its closest
paper's 0.236: the figure is reproduced, and the index is the reference's: 5,288,635 terms, with a
per-document term stream byte-identical to Lucene 10.5.2's `EnglishAnalyzer`. This engine cannot use `uax29`; with its closest
configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The gap was decomposed: a PostgreSQL
configuration with the reference's 33-word stop list and Snowball reads 0.2196, so the stop list is
not the cause (the longer stock list helps) and the remainder is the stemmer and the parser — not
Expand Down
7 changes: 6 additions & 1 deletion docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -529,7 +529,12 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10):
gains, k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted
saturation constant, single precision, depth 1,000 — the in-memory BM25 reads **0.236** against
the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was
published at. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`,
published at. The index is the reference's, not merely a close one: `--fingerprint` reports
5,288,635 terms, and the per-document term stream is byte-identical to Lucene 10.5.2's
`EnglishAnalyzer` over the corpus (Extended_Pictographic symbols, one term per Han ideograph,
combining marks, format characters, the 255-unit token chop, U+202F). Multi-code-point emoji
sequences are the one knowingly unimplemented rule — FiQA contains none, and a test pins the
divergence. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`,
k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate
cost. Where the gap sits was measured rather than assumed: a PostgreSQL configuration carrying the
reference's 33-word stop list and its Snowball stemmer reads **0.2196**, so the stop list is not
Expand Down
357 changes: 263 additions & 94 deletions src/LexiSharp/Linguistics/Tokenizer.cs

Large diffs are not rendered by default.

142 changes: 142 additions & 0 deletions src/LexiSharp/Linguistics/WordBreakClasses.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,142 @@
using System.Text;

namespace LexiSharp.Linguistics;

/// <summary>
/// The Unicode character classes the reference word tokenizer tests directly, which the base class
/// library does not expose: the emoji property, the ideographic script and the word-continuing
/// format class.
/// </summary>
/// <remarks>
/// <para>
/// The tables are pinned to Unicode 12.1 on purpose. The tokenizer this library reproduces — Lucene
/// 10.5.2's <c>StandardTokenizer</c> — is generated from a grammar declared as <c>%unicode 12.1</c>,
/// so its answers are the 12.1 answers even where a later Unicode version reassigns a code point.
/// Two of those reassignments are visible here: U+0600..U+0605 are <c>Format</c> in 12.1 and become
/// <c>Prepend</c> later, and the <c>Extended_Pictographic</c> property was laid out over reserved
/// ranges that later shrank. Using the newer tables would part company with the reference on
/// characters assigned after 12.1 — rare in a corpus, but not impossible in a URL or a name.
/// </para>
/// <para>
/// This is deliberately not a general-purpose Unicode classification API: it exists so that
/// <see cref="WordSegmentation.UnicodeWordBoundaries"/> can emit the same tokens as that grammar.
/// The English and identifier analyses never call it.
/// </para>
/// </remarks>
internal static class WordBreakClasses
{
/// <summary>
/// Whether <paramref name="rune"/> carries the <c>Extended_Pictographic</c> property — the set
/// the reference grammar starts an emoji token on. Not every <c>OtherSymbol</c> is emoji:
/// <c>©</c> and <c>™</c> are, while <c>°</c> and <c>■</c> are not, and the reference drops the
/// latter where it emits the former.
/// </summary>
public static bool IsEmoji(Rune rune)
{
if (rune.Value < 0xA9)
return false;

return InRanges(rune.Value, EmojiRanges);
}

/// <summary>
/// The <c>Extended_Pictographic</c> code point ranges of Unicode 12.1, as adjacent
/// <c>[start, end]</c> pairs sorted ascending.
/// </summary>
private static readonly int[] EmojiRanges =
[
0xA9, 0xA9, 0xAE, 0xAE, 0x203C, 0x203C, 0x2049, 0x2049, 0x2122, 0x2122, 0x2139, 0x2139, 0x2194, 0x2199, 0x21A9, 0x21AA,
0x231A, 0x231B, 0x2328, 0x2328, 0x2388, 0x2388, 0x23CF, 0x23CF, 0x23E9, 0x23F3, 0x23F8, 0x23FA, 0x24C2, 0x24C2, 0x25AA, 0x25AB,
0x25B6, 0x25B6, 0x25C0, 0x25C0, 0x25FB, 0x25FE, 0x2600, 0x2605, 0x2607, 0x2612, 0x2614, 0x2685, 0x2690, 0x2705, 0x2708, 0x2712,
0x2714, 0x2714, 0x2716, 0x2716, 0x271D, 0x271D, 0x2721, 0x2721, 0x2728, 0x2728, 0x2733, 0x2734, 0x2744, 0x2744, 0x2747, 0x2747,
0x274C, 0x274C, 0x274E, 0x274E, 0x2753, 0x2755, 0x2757, 0x2757, 0x2763, 0x2767, 0x2795, 0x2797, 0x27A1, 0x27A1, 0x27B0, 0x27B0,
0x27BF, 0x27BF, 0x2934, 0x2935, 0x2B05, 0x2B07, 0x2B1B, 0x2B1C, 0x2B50, 0x2B50, 0x2B55, 0x2B55, 0x3030, 0x3030, 0x303D, 0x303D,
0x3297, 0x3297, 0x3299, 0x3299, 0x1F000, 0x1F0FF, 0x1F10D, 0x1F10F, 0x1F12F, 0x1F12F, 0x1F16C, 0x1F171, 0x1F17E, 0x1F17F, 0x1F18E, 0x1F18E,
0x1F191, 0x1F19A, 0x1F1AD, 0x1F1E5, 0x1F201, 0x1F20F, 0x1F21A, 0x1F21A, 0x1F22F, 0x1F22F, 0x1F232, 0x1F23A, 0x1F23C, 0x1F23F, 0x1F249, 0x1F3FA,
0x1F400, 0x1F53D, 0x1F546, 0x1F64F, 0x1F680, 0x1F6FF, 0x1F774, 0x1F77F, 0x1F7D5, 0x1F7FF, 0x1F80C, 0x1F80F, 0x1F848, 0x1F84F, 0x1F85A, 0x1F85F,
0x1F888, 0x1F88F, 0x1F8AE, 0x1F8FF, 0x1F90C, 0x1F93A, 0x1F93C, 0x1F945, 0x1F947, 0x1FFFD,
];

/// <summary>
/// Whether <paramref name="rune"/> is a CJK ideograph: a character with the <c>Han</c> script
/// that the reference grammar tokenizes one code point at a time.
/// </summary>
/// <remarks>
/// The exceptions are the script's word letters: U+3005 and U+303B, the ideographic iteration
/// marks, are <c>Script=Han</c> but <c>Word_Break=ALetter</c>, so the reference folds them into
/// an alphanumeric run instead of emitting them alone. U+3007, the ideographic number zero,
/// stays here because it is <c>Word_Break=Other</c>.
/// </remarks>
public static bool IsIdeographic(Rune rune)
{
if (rune.Value < 0x2E80)
return false;

return InRanges(rune.Value, IdeographicRanges);
}

/// <summary>
/// The <c>Script=Han</c> code points of Unicode 12.1 that are not word letters, as adjacent
/// <c>[start, end]</c> pairs sorted ascending.
/// </summary>
private static readonly int[] IdeographicRanges =
[
0x2E80, 0x2E99, 0x2E9B, 0x2EF3, 0x2F00, 0x2FD5, 0x3007, 0x3007, 0x3021, 0x3029, 0x3038, 0x303A, 0x3400, 0x4DB5, 0x4E00, 0x9FEF,
0xF900, 0xFA6D, 0xFA70, 0xFAD9, 0x20000, 0x2A6D6, 0x2A700, 0x2B734, 0x2B740, 0x2B81D, 0x2B820, 0x2CEA1, 0x2CEB0, 0x2EBE0, 0x2F800, 0x2FA1D,
];

/// <summary>
/// Whether <paramref name="rune"/> is a format character the reference's word-break data lets a
/// word continue through: a soft hyphen, a byte-order mark, a left-to-right mark, and the other
/// <c>Word_Break=Format</c> code points.
/// </summary>
/// <remarks>
/// A format character cannot open a word, but it does not end one: a byte-order mark between
/// the digits of <c>1990</c>, or a left-to-right mark after <c>bank</c>, stays inside the term
/// in the reference index. U+200B is deliberately absent — it is <c>Word_Break=Other</c>, and
/// the reference splits a word around it, which this library already did.
/// </remarks>
public static bool IsFormatExtend(Rune rune)
{
if (rune.Value < 0xAD)
return false;

return InRanges(rune.Value, FormatRanges);
}

/// <summary>
/// The <c>Word_Break=Format</c> and <c>Word_Break=ZWJ</c> code points of Unicode 12.1 that are
/// format characters, plus U+200C, whose word-break class is <c>Extend</c>, as adjacent
/// <c>[start, end]</c> pairs sorted ascending.
/// </summary>
private static readonly int[] FormatRanges =
[
0xAD, 0xAD, 0x600, 0x605, 0x61C, 0x61C, 0x6DD, 0x6DD, 0x70F, 0x70F, 0x8E2, 0x8E2, 0x180E, 0x180E, 0x200C, 0x200F,
0x202A, 0x202E, 0x2060, 0x2064, 0x2066, 0x206F, 0xFEFF, 0xFEFF, 0xFFF9, 0xFFFB, 0x110BD, 0x110BD, 0x110CD, 0x110CD, 0x13430, 0x13438,
0x1BCA0, 0x1BCA3, 0x1D173, 0x1D17A, 0xE0001, 0xE0001, 0xE0020, 0xE007F,
];

/// <summary>
/// Whether <paramref name="value"/> falls inside one of the <c>[start, end]</c> pairs in
/// <paramref name="ranges"/>.
/// </summary>
private static bool InRanges(int value, ReadOnlySpan<int> ranges)
{
int low = 0;
int high = (ranges.Length / 2) - 1;

while (low <= high)
{
int middle = (low + high) / 2;

if (value < ranges[middle * 2])
high = middle - 1;
else if (value > ranges[(middle * 2) + 1])
low = middle + 1;
else
return true;
}

return false;
}
}
88 changes: 88 additions & 0 deletions tests/LexiSharp.Tests/AnalysisScopeTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
using LexiSharp.Linguistics;
using Xunit;

namespace LexiSharp.Tests;

/// <summary>
/// The four parity rules added for <see cref="WordSegmentation.UnicodeWordBoundaries"/> — emoji and
/// ideographs as terms, combining and format characters inside a word, and the 255-unit chop — do
/// not reach the other analyses.
/// </summary>
/// <remarks>
/// This is the regression pin for the scope of the change: <c>default</c>, <c>porter</c>,
/// <c>english</c> and <c>identifiers</c> are the analyses every published figure and every caller
/// outside the uax29 comparison uses, and each keeps the behavior it had. The joining path is
/// shared with the identifiers analysis, which is why the chop is gated on the mode rather than on
/// the scan.
/// </remarks>
public class AnalysisScopeTests
{
public static TheoryData<string> Analyses => new() { "default", "porter", "english", "identifiers" };

private static Tokenizer Build(string analyzer) => analyzer switch
{
"default" => new Tokenizer(TokenizerOptions.Default),
"porter" => new Tokenizer(new TokenizerOptions
{
Stemmer = new PorterStemmer(),
RemoveStopWords = true,
}),
"english" => new Tokenizer(new TokenizerOptions
{
Stemmer = new PorterStemmer(),
RemoveStopWords = true,
StopWords = StopWords.EnglishFunction,
KeepSingleCharTerms = true,
}),
"identifiers" => new Tokenizer(new TokenizerOptions
{
WordJoiners = "-_.:/",
KeepSingleCharTerms = true,
FoldDiacritics = false,
}),
_ => throw new ArgumentOutOfRangeException(nameof(analyzer)),
};

[Theory]
[MemberData(nameof(Analyses))]
public void AnEmojiIsNotATermOutsideTheMode(string analyzer)
{
Tokenizer tokenizer = Build(analyzer);

Assert.Equal(["ab", "cd"], tokenizer.Tokenize("ab🤔cd"));
Assert.Empty(tokenizer.Tokenize("✔"));
}

[Theory]
[MemberData(nameof(Analyses))]
public void AnIdeographicRunStaysOneWordOutsideTheMode(string analyzer)
{
Assert.Equal(["理财"], Build(analyzer).Tokenize("理财"));
}

[Theory]
[MemberData(nameof(Analyses))]
public void AFormatCharacterStillSplitsOutsideTheMode(string analyzer)
{
// The ASCII number survives every analysis; the `0` after the byte-order mark is kept by
// `english` and `identifiers`, which keep single-character terms, and dropped by the two
// that do not.
string[] expected = analyzer is "english" or "identifiers" ? ["199", "0"] : ["199"];

Assert.Equal(expected, Build(analyzer).Tokenize("199\uFEFF0"));
}

[Theory]
[MemberData(nameof(Analyses))]
public void ANarrowNoBreakSpaceStillSplitsOutsideTheMode(string analyzer)
{
Assert.Equal(["ab", "cd"], Build(analyzer).Tokenize("ab\u202Fcd"));
}

[Theory]
[MemberData(nameof(Analyses))]
public void ALongTokenIsNotChoppedOutsideTheMode(string analyzer)
{
Assert.Equal([new string('x', 468)], Build(analyzer).Tokenize(new string('x', 468)));
}
}
15 changes: 15 additions & 0 deletions tests/LexiSharp.Tests/CombiningMarkTests.cs
Original file line number Diff line number Diff line change
Expand Up @@ -87,4 +87,19 @@ public void PrecomposedLettersAreUnaffected()
Assert.Equal(["क़"], tokenizer.Tokenize("क़"));
Assert.Equal(["ẋ"], tokenizer.Tokenize("ẋ"));
}

/// <summary>
/// A mark does not change the class of the word that ends with it, so a separator that joins two
/// letters still joins when a mark sits between the last letter and the separator. FiQA's
/// <c>tɕɛ̝.bʌl</c> is one term in the reference; reading the class off U+031D — a mark, neither a
/// letter nor a digit — split it into two and was the whole of that document's difference.
/// </summary>
[Fact]
public void AMarkBeforeTheSeparatorDoesNotChangeTheWordClass()
{
Assert.Equal(["tɕɛ̝.bʌl"], Tokenizing().Tokenize("tɕɛ̝.bʌl"));
Assert.Equal(["ab" + Dot + ".cd"], Tokenizing().Tokenize("ab" + Dot + ".cd"));
Assert.Equal(["ab" + Dot + ":cd"], Tokenizing().Tokenize("ab" + Dot + ":cd"));
Assert.Equal(["12" + Dot + ",34"], Tokenizing().Tokenize("12" + Dot + ",34"));
}
}
89 changes: 89 additions & 0 deletions tests/LexiSharp.Tests/EmojiTokenTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
using LexiSharp.Linguistics;
using Xunit;

namespace LexiSharp.Tests;

/// <summary>
/// Under <see cref="WordSegmentation.UnicodeWordBoundaries"/> an emoji code point is a term of its
/// own, and a symbol that is not emoji is not.
/// </summary>
/// <remarks>
/// The rule is the <c>Extended_Pictographic</c> property, not the <c>OtherSymbol</c> category, and
/// the difference is not academic: FiQA's index holds <c>✔</c>, <c>♦</c>, <c>☏</c> and the emoji,
/// while it holds no <c>°</c>, <c>■</c> or <c>№</c> at all. A rule written on the category emits
/// 143 terms the reference does not have.
/// <para>
/// The code point is emitted whole, so a supplementary-plane emoji is one term rather than two
/// surrogate halves. Multi-code-point emoji sequences — a ZWJ family, a flag pair — are not
/// reproduced; see the remark on <c>WordBreakClasses</c> and the plan's residual note.
/// </para>
/// </remarks>
public class EmojiTokenTests
{
private static Tokenizer Segmenter() => new(new TokenizerOptions
{
KeepSingleCharTerms = true,
FoldDiacritics = false,
WordSegmentation = WordSegmentation.UnicodeWordBoundaries,
});

[Theory]
[InlineData("✔")]
[InlineData("♦")]
[InlineData("☏")]
[InlineData("©")]
[InlineData("®")]
[InlineData("™")]
[InlineData("↔")]
[InlineData("🤔")]
[InlineData("😂")]
[InlineData("👻")]
public void AnEmojiCodePointIsATerm(string symbol)
{
Assert.Equal([symbol], Segmenter().Tokenize(symbol));
}

[Fact]
public void ASupplementaryEmojiIsOneTermNotTwoSurrogateHalves()
{
IReadOnlyList<string> terms = Segmenter().Tokenize("👨‍👩‍👧‍👦 🤔");

// The ZWJ family is four emoji code points plus three joiners and becomes four terms; the
// point of the assertion is that the surrogate pairs are not split.
Assert.Equal(["👨", "👩", "👧", "👦", "🤔"], terms);
}

[Fact]
public void AnEmojiDoesNotExtendTheWordBesideIt()
{
Assert.Equal(["aspirin", "®"], Segmenter().Tokenize("Aspirin®"));
Assert.Equal(["a", "🤔", "b"], Segmenter().Tokenize("a🤔b"));
Assert.Equal(["2024", "©"], Segmenter().Tokenize("2024©"));
}

[Fact]
public void RepeatedEmojiAreOneTermEach()
{
Assert.Equal(["🤔", "🤔", "🤔"], Segmenter().Tokenize("🤔🤔🤔"));
Assert.Equal(["✔", "✔"], Segmenter().Tokenize("✔ ✔"));
}

/// <summary>
/// The other half of the rule: the category is not enough. These characters are
/// <c>OtherSymbol</c> and the reference drops them.
/// </summary>
[Theory]
[InlineData("°")]
[InlineData("¦")]
[InlineData("●")]
[InlineData("■")]
[InlineData("▲")]
[InlineData("№")]
[InlineData("℅")]
[InlineData("│")]
public void ANonEmojiSymbolIsNotATerm(string symbol)
{
Assert.Empty(Segmenter().Tokenize(symbol));
Assert.Equal(["a", "b"], Segmenter().Tokenize("a" + symbol + "b"));
}
}
Loading
Loading