Skip to content

Say what FiQA's default-analysis rows are comparable with - #80

Merged
manuc66 merged 1 commit into
mainfrom
docs/fiqa-aligned-analysis
Nov 17, 2026
Merged

manuc66 merged 1 commit into
mainfrom
docs/fiqa-aligned-analysis

Conversation

@manuc66

@manuc66 manuc66 commented Nov 17, 2026

Copy link
Copy Markdown
Owner

What changed

Two paragraphs, no code. The FiQA rows were measured under the library's default analysis — PostgreSQL simple + unaccent, k1=1.5/b=0.75 — while the published 0.2361 is Lucene's English analysis at k1=0.9/b=0.4. The pages now say so, and give the aligned measurement that closes the gap:

FiQA-2018, nDCG@10 measured published
In-memory BM25, English analysis, k1=0.9/b=0.4 0.237 0.2361
PostgreSQL BM25, english config, k1=0.9/b=0.4, gate 0.2291 0.2361
PostgreSQL BM25, simple config, k1=1.5/b=0.75, gate 0.2061 not comparable
PostgreSQL BM25, simple config, k1=1.5/b=0.75, no gate 0.1077 not comparable

So the published figure is reproduced when the analysis and operating point are aligned (0.237 against 0.2361), and the SQL row is ~0.007 short for the documented reason — PostgreSQL's English configuration is not Lucene's analyzer. The gate is not the cause: the aligned SQL row is a gated row, and on FiQA the gate's candidate set does not truncate.

docs/evaluation.md and docs/backends.md carry the numbers; no behaviour changes.

The FiQA rows were measured under PostgreSQL's simple configuration, and the
published 0.2361 is Lucene's English analysis at k1=0.9/b=0.4. The page now
states that, and the aligned measurement that closes it: in-memory BM25 0.237
against 0.2361, this engine 0.2291, the same ~0.007 analysis gap as the other
corpora.
@codecov-commenter

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.16%. Comparing base (614f642) to head (c09bf97).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #80      +/-   ##
==========================================
- Coverage   89.19%   89.16%   -0.03%     
==========================================
  Files         270      270              
  Lines       21383    21383              
  Branches     4621     4621              
==========================================
- Hits        19073    19067       -6     
- Misses       1388     1391       +3     
- Partials      922      925       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 3e595fa into main Nov 17, 2026
16 checks passed
@manuc66
manuc66 deleted the docs/fiqa-aligned-analysis branch November 17, 2026 06:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants