Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,22 @@
## [Unreleased]

### Fixed

- **The evaluation harness measured a rounded run in the engine's tie-break, not in the run reader's.**
The four-decimal rounding reproduces what the reference writes down, and a written run is read back
by `trec_eval`, which ignores the rank column and breaks an exact score tie by document number
**descending** (`comp_sim_docno` in `form_res_rels.c`). The harness kept the engine's order
(document id ascending), so on ArguAna it reported **0.4005** where `trec_eval` reads **0.3970**
from the same file: 5,699 exact duplicate scores, 183 queries with a tie in the top 10, 84 queries
whose nDCG differs. Applying the reader's tie-break to a rounded run makes the two per-query equal,
1,406 of 1,406, and the pin `arguana/uax29-aligned` is re-recorded from 0.4005 to **0.3970** as a
measured value, not a copied one. The replay also retrieves at the reference run's depth (1,000)
rather than 10: a document at rank 11 can enter the top ten by tying the tenth score, and that
boundary alone is worth 0.0004 in the aggregate, across four queries. NFCorpus and SciFact are
unchanged; no configuration had a tie inside their top ten. The table header now names the depth
the metrics are measured at (`MetricDepth.Cutoff`) instead of `--top-k`, which had printed
`nDCG@1000` over an nDCG@10 value.

## [1.0.0] — 2026-11-10

The first stable release. The public API is now versioned by semantic versioning: a breaking
Expand Down
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ aligned to the published figures:
|---|---:|---:|
| NFCorpus | 0.3218 | 0.3218 |
| SciFact | 0.6789 | 0.6789 |
| ArguAna | 0.4005 | 0.3970 |
| ArguAna | 0.3970 | 0.3970 |

Point estimates; the harness also reports paired-bootstrap 95 % CIs. Under the library's own
default analysis and parameters the same scorer reads 0.308 / 0.662 / 0.320 — the difference is
Expand Down Expand Up @@ -113,8 +113,10 @@ linear classifier): [text analysis →](docs/text-analysis.md).
- **net10.0**, **1.0.0**: the public API is stable within a major version. Breaking changes
require a major release, additions are minor and fixes are patches. Pin a major version and
read the [changelog](CHANGELOG.md).
- ArguAna reads 0.4005 against 0.3970 published (0.0035 short; cause not established). An earlier
ArguAna figure was withdrawn and is not reproducible by any code path.
- ArguAna reads 0.3970 against 0.3970 published, equal to four decimals on a run file read by
`trec_eval`; the 0.4005 an earlier version reported was the same ranking under the engine's own
tie-break inside an exact score tie. An earlier ArguAna figure was withdrawn and is not
reproducible by any code path.
- Not every engine/scorer/reranker/merger combination is tested; treat unusual pairings as
supported but unproven until tested on your data.
- SQL backends are measured on ranking for the engines that rank on their own; sparse and fuzzy
Expand Down
62 changes: 48 additions & 14 deletions bench/LexiSharp.Eval/Evaluation.cs
Original file line number Diff line number Diff line change
Expand Up @@ -718,20 +718,27 @@ void Score(int i)
{
EvaluatedQuery evaluated = queries[i];

string[] retrieved = engine.Search(
evaluated.Query.Text,
// The rounding is a reproduction of what the reference writes down, not a scoring
// choice, so it is off unless the run asks for it: on it, the metrics measure a
// different ranking function.
new SearchOptions(
topK,
ExcludedDocumentIds: evaluated.Excluded,
ParseQuerySyntax: parseQuerySyntax)
.WithScoreRounding(scoreRounding
? ScoreRounding.FourDecimals
: ScoreRounding.None))
.Select(result => result.DocumentId)
.ToArray();
IReadOnlyList<SearchResult> hits = engine.Search(
evaluated.Query.Text,
// The rounding is a reproduction of what the reference writes down, not a scoring
// choice, so it is off unless the run asks for it: on it, the metrics measure a
// different ranking function.
new SearchOptions(
topK,
ExcludedDocumentIds: evaluated.Excluded,
ParseQuerySyntax: parseQuerySyntax)
.WithScoreRounding(scoreRounding
? ScoreRounding.FourDecimals
: ScoreRounding.None));

// Rounding merges scores that were distinct into a tie, and a run file is read back by a
// tool whose tie-break is part of the published number. Measuring the engine's order
// instead would score a ranking no run file contains: on ArguAna that is 0.4005 against
// the 0.3970 a reader computes from the same file.
if (scoreRounding)
hits = RunFileOrder(hits);

string[] retrieved = hits.Select(result => result.DocumentId).ToArray();

string[] relevant = evaluated.Graded.Keys.ToArray();

Expand Down Expand Up @@ -801,6 +808,33 @@ void Score(int i)
.ToList());
}

/// <summary>
/// Rewrites a page into the order a run file's reader reconstructs: score descending, then
/// document id descending.
/// </summary>
/// <remarks>
/// <para>
/// This is <c>trec_eval</c>'s <c>comp_sim_docno</c>, which is what turns a run file into a
/// ranking: the file's rank column is ignored, and the scores are sorted with
/// <c>strcmp(p2-&gt;docno, p1-&gt;docno)</c> on an exact tie. The order matters because
/// <see cref="ScoreRounding.FourDecimals"/> deliberately rounds scores onto the same
/// ten-thousandth, so a tie the engine had already broken becomes an exact tie in the file. The
/// walk-down step cannot always separate it either: at an ArguAna score of 118 the float spacing
/// is larger than the one-millionth it subtracts, so the scores stay bit-identical and the reader's
/// tie-break decides.
/// </para>
/// <para>
/// Only the reproduction path calls this. Without rounding, an exact tie is the engine's own and
/// its configured <see cref="TieBreak"/> is the ranking being measured; with rounding, the run file
/// is the object being measured, and the engine order is a ranking that file no longer carries.
/// </para>
/// </remarks>
internal static SearchResult[] RunFileOrder(IReadOnlyList<SearchResult> results) =>
results
.OrderByDescending(result => result.Score)
.ThenByDescending(result => result.DocumentId, StringComparer.Ordinal)
.ToArray();

private static string RunTuned(
ITextIndex index,
IReadOnlyList<EvaluatedQuery> fitQueries,
Expand Down
11 changes: 8 additions & 3 deletions bench/LexiSharp.Eval/Program.cs
Original file line number Diff line number Diff line change
Expand Up @@ -1149,7 +1149,7 @@ void CloseRerankCache()
referenceBm25 ?? new Bm25Parameters(0.9, 0.4), parseQuerySyntax);
}

PrintTable(results, topK, jobs);
PrintTable(results, jobs);

if (bootstrap)
RetrievalBootstrap.Print(results, bootstrapBaseline, bootstrapResamples, bootstrapCorrection);
Expand Down Expand Up @@ -1597,9 +1597,14 @@ private static IEnumerable<double> ParseStrengths(string value)
$"--expand-ranking expects 'count', 'pmi' or 'both', got '{value}'."),
};

private static void PrintTable(IReadOnlyList<ConfigResult> results, int topK, int jobs)
private static void PrintTable(IReadOnlyList<ConfigResult> results, int jobs)
{
var header = new[] { "Config", $"nDCG@{topK}", $"MAP@{topK}", $"MRR@{topK}", $"R@{topK}", "Queries", "Wall" };
// The header names the depth the metrics are measured at, not the depth the run retrieved.
// --top-k 1000 still scores at rank 10 (MetricDepth.Cutoff), and printing "nDCG@1000" over an
// nDCG@10 value is a label a reader has no way to catch: the parity command retrieves 1,000
// documents to compare scores one by one and measures ten.
int depth = MetricDepth.Cutoff;
var header = new[] { "Config", $"nDCG@{depth}", $"MAP@{depth}", $"MRR@{depth}", $"R@{depth}", "Queries", "Wall" };
var widths = new[] { 30, 9, 9, 9, 9, 8, 8 };

Console.WriteLine();
Expand Down
22 changes: 12 additions & 10 deletions bench/LexiSharp.Eval/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -952,22 +952,24 @@ document count, and a metric cutoff.
| + reference analysis, at k1=0.9/b=0.4 | 0.240 |
| + reference analysis, + query-frequency counting, at k1=0.9/b=0.4 | 0.301 |
| + reference analysis, + query-frequency counting, at k1=3.0/b=0.75 | 0.367 |
| + the parity options, at k1=0.9/b=0.4 | 0.4005 |
| + the parity options, at k1=0.9/b=0.4 | 0.3970 |
| published reference, at k1=0.9/b=0.4 | 0.3970 |

Read the third and fourth rows together: the setting is worth +0.061 at the reference's own
parameters and +0.033 at k1=3.0. Those are settings the library exposes. The fifth row adds the four
options a bit-for-bit comparison needs — the document count that carries the field rather than every
indexed document, the reference's scale and precision, its four-decimal rounding, and the query's own
document removed — all off by default. It reads **0.4005** against the published **0.3970**, 0.0035
short, and **that gap is not closed by any configuration tried**.

A parity row in `reference/pinned.json` used to carry 0.3970 as this library's own figure, with a
claim that all 15,466 returned scores matched the reference's on the raw bits. Both are withdrawn:
the parity section is displayed by `--verify-reference` but never replayed, and the figure was the
published one rather than what the documented configuration measures. The reproduction command in
`docs/evaluation.md` now produces 0.4005, and that value is a pinned regression, so the gate replays
it.
document removed — all off by default. It reads **0.3970** against the published **0.3970**: the run
file is read with `trec_eval`'s tie-break (score descending, then document number descending), which
is the convention the published figure was produced under. Scored in the engine's own order the same
ranking reads 0.4005, which is the order inside an exact score tie, not a different result.

A parity row in `reference/pinned.json` used to carry 0.3970 as this library's own figure without
replaying it, with a claim that all 15,466 returned scores matched the reference's on the raw bits.
That row was withdrawn: the parity section is displayed by `--verify-reference` but never replayed,
and the figure was the published one rather than a measurement. The value is now in a pinned
regression (`arguana/uax29-aligned`) that `--verify-reference` replays, and the run-file comparison is
per query.

An earlier figure published for this corpus was withdrawn; it is not reproducible by any code path in
this repository. See the [changelog](https://github.com/manuc66/LexiSharp/blob/main/CHANGELOG.md). The
Expand Down
30 changes: 25 additions & 5 deletions bench/LexiSharp.Eval/ReferenceVerifier.cs
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,13 @@ namespace LexiSharp.Eval;
/// </remarks>
internal static class ReferenceVerifier
{
/// <summary>
/// The depth the reference's run files are written at, and therefore the depth a rounded pin has
/// to retrieve before it can be read back: the reader's tie-break can promote the document at
/// rank 11 into the measured top ten when it ties the tenth score.
/// </summary>
private const int ReferenceRunDepth = 1000;

public static async Task<int> RunAsync(
PinnedReference pinned, string dataBaseDir, bool write, CancellationToken cancellationToken)
{
Expand Down Expand Up @@ -184,14 +191,27 @@ private static double Measure(
// on BEIR queries a straight quote is ordinary prose and a query language collapses it
// (146 of ArguAna's 1,406 test queries carry one). The field exists so the pin says
// which of the two it was recorded under.
var retrieved = engine
// A rounded pin is a run file, and a run file's reader can move a document from beyond
// rank 10 into the top 10 when it ties the tenth score. The published ArguAna run was
// written at depth 1000; retrieving only 10 leaves four queries reading 0.3974 instead of
// the reader's 0.3970, because the document that decides those four is the eleventh.
int depth = pin.ScoreRounding ? ReferenceRunDepth : MetricDepth.Cutoff;

IReadOnlyList<SearchResult> hits = engine
.Search(query.Text, new SearchOptions(
10,
depth,
ExcludedDocumentIds: excluded,
ParseQuerySyntax: pin.QuerySyntax)
.WithScoreRounding(pin.ScoreRounding ? ScoreRounding.FourDecimals : ScoreRounding.None))
.Select(result => result.DocumentId)
.ToArray();
.WithScoreRounding(pin.ScoreRounding ? ScoreRounding.FourDecimals : ScoreRounding.None));

// A rounded pin is replaying a run file, and a run file is ranked by its reader: score
// descending, then document id descending. Measuring the engine's tie-break against a
// published number computed by trec_eval is what made ArguAna read 0.4005 where the same
// ranking read 0.3970 from the file.
if (pin.ScoreRounding)
hits = Evaluation.RunFileOrder(hits);

var retrieved = hits.Select(result => result.DocumentId).ToArray();

total += RetrievalMetrics.NdcgAtK(retrieved, corpus.TestRelevance[query.Id], 10, gain);
}
Expand Down
Loading
Loading