From efec41bba6f9a6cab8670175c5ccb1748b56c8f4 Mon Sep 17 00:00:00 2001 From: manuc66 Date: Tue, 17 Nov 2026 07:30:48 +0100 Subject: [PATCH] Decompose FiQA's analysis gap instead of naming it A PostgreSQL configuration carrying the reference's 33-word stop list and Snowball reads 0.2196, below the stock english configuration's 0.2291: the stop list is not the cause, the longer stock list helps, and the remainder is the stemmer and the parser. The pages now say that rather than leaving the 0.007 attributed to "analysis". --- docs/backends.md | 5 ++++- docs/evaluation.md | 5 ++++- 2 files changed, 8 insertions(+), 2 deletions(-) diff --git a/docs/backends.md b/docs/backends.md index 3326036..2669d85 100644 --- a/docs/backends.md +++ b/docs/backends.md @@ -390,7 +390,10 @@ analysis at k1=0.9/b=0.4. Under the reference's own conventions — `--analyzer k1=0.9/b=0.4, query-term frequency, reference index statistics, score rounding, omitted saturation constant, single precision, depth 1,000 — the in-memory row reads **0.236** against 0.2361 and the paper's 0.236: the figure is reproduced. This engine cannot use `uax29`; with its closest -configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident: +configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**. The gap was decomposed: a PostgreSQL +configuration with the reference's 33-word stop list and Snowball reads 0.2196, so the stop list is +not the cause (the longer stock list helps) and the remainder is the stemmer and the parser — not +replaceable on stock PostgreSQL. The pattern is SciFact's rather than a scientific-corpus accident: the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does not. diff --git a/docs/evaluation.md b/docs/evaluation.md index 3633fb5..8e5add0 100644 --- a/docs/evaluation.md +++ b/docs/evaluation.md @@ -531,7 +531,10 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10): the published **0.2361** (the paper's 0.236): the figure is reproduced at the precision it was published at. The PostgreSQL row cannot use `uax29`; with its closest configuration (`english`, k1=0.9/b=0.4) it reads **0.2291**, the same ~0.007 analysis gap the other corpora show, not a gate - cost. The gate's direction is SciFact's: + cost. Where the gap sits was measured rather than assumed: a PostgreSQL configuration carrying the + reference's 33-word stop list and its Snowball stemmer reads **0.2196**, so the stop list is not + the cause — the longer stock list helps here — and what remains is the stemmer and the parser, + which stock PostgreSQL cannot replace without an extension. The gate's direction is SciFact's: the ungated candidates are chosen by coverage and truncated, the gated ones by idf. - **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable** with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric