Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions docs/backends.md
Original file line number Diff line number Diff line change
Expand Up @@ -383,10 +383,14 @@ candidate set is small enough not to truncate. Lowering the budget trades a litt
NFCorpus for speed; raising it changes nothing here.

**A corpus of another nature.** FiQA-2018 (`--dataset fiqa`; 57,638 documents, 648 test queries,
financial Q&A with short queries) is not part of the pinned three. At the same settings the gate
reads **0.2061** nDCG@10 and the ungated pool **0.1077**, against Anserini's published **0.2361**
for its own analysis. The pattern is SciFact's rather than a scientific-corpus accident: the
ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does not.
financial Q&A with short queries) is not part of the pinned three. Under this table's analysis —
`simple` + `unaccent`, k1=1.5/b=0.75 — the gate reads **0.2061** nDCG@10 and the ungated pool
**0.1077**. That is not comparable with Anserini's published **0.2361**, which is Lucene's English
analysis at k1=0.9/b=0.4: align both (`--analyzer english --reference-bm25 0.9,0.4`,
`--backend-config english --backend-bm25 0.9,0.4`) and the in-memory BM25 reads 0.237 against
0.2361, this engine **0.2291**. The pattern is SciFact's rather than a scientific-corpus accident:
the ungated pool truncates a coverage-ranked candidate set, and the gate's idf-selected one does
not.

**What it costs, in time.** The evaluation harness reports quality and not time on purpose; the
[PostgreSQL lane](benchmarks.md#postgresql-lane-postgres-what-a-search-costs-on-stock-postgresql)
Expand Down
13 changes: 9 additions & 4 deletions docs/evaluation.md
Original file line number Diff line number Diff line change
Expand Up @@ -521,10 +521,15 @@ same qrels (harness defaults, exponential gains, 10 retrieved at 10):
than by idf — on SciFact that truncation is the whole 0.10 difference. The numbers behind the
choice, and what it costs in milliseconds, are on the [backend guide](backends.md#bm25-over-stock-postgresql-postgresbm25searchengine).
- **The gate on a corpus of another nature.** FiQA-2018 is a selectable dataset (`--dataset fiqa`)
but not part of `--dataset all` or the pinned comparison. On its 57,638 documents and 648 test
queries, the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate and **0.1077**
without it at pool 1,000; Anserini's published BM25 is 0.2361 with its own analysis. Same cause as
SciFact: the ungated candidates are chosen by coverage and truncated, the gated ones by idf.
but not part of `--dataset all` or the pinned comparison. Under the library's default analysis —
no stemming, no stop list — the stock-PostgreSQL BM25 row reads **0.2061** nDCG@10 with the gate
and **0.1077** without it at pool 1,000. Those are **not** comparable with Anserini's published
0.2361: that figure is Lucene's English analysis at k1=0.9/b=0.4, and this row is PostgreSQL's
`simple` configuration at 1.5/0.75. Aligning both (`--analyzer english --reference-bm25 0.9,0.4`,
and `--backend-config english --backend-bm25 0.9,0.4` for the SQL row), the in-memory BM25 reads
**0.237** against the published **0.2361**, and the PostgreSQL row reads **0.2291** — the same
~0.007 analysis gap the other corpora show, not a gate cost. The gate's direction is SciFact's:
the ungated candidates are chosen by coverage and truncated, the gated ones by idf.
- **`ts_rank_cd`'s and ParadeDB's scores are PostgreSQL-native and not numerically comparable**
with `Bm25Scorer`; the stock-PostgreSQL BM25 row's are `Bm25Scorer`'s and are. The metric
columns compare either way. The analysis is PostgreSQL's own, not this library's tokenizer, which
Expand Down
Loading