Skip to content

Name the reference conventions FiQA's aligned figure needs - #81

Merged
manuc66 merged 1 commit into
mainfrom
docs/fiqa-reference-conventions
Nov 17, 2026
Merged

manuc66 merged 1 commit into
mainfrom
docs/fiqa-reference-conventions

Conversation

@manuc66

@manuc66 manuc66 commented Nov 17, 2026

Copy link
Copy Markdown
Owner

What changed

The FiQA paragraphs said the aligned figure was 0.237 against 0.2361. That run aligned the analysis (english) and the operating point (k1=0.9/b=0.4) only — it was not the reference's configuration.

The reference's aligned rows also use:

  • --analyzer uax29 (the published separator rules, apostrophes inside a word, possessives stripped before the stop list)
  • --ndcg-gain linear
  • --query-term-frequency
  • --reference-index-statistics (average over non-empty documents, one-byte length quantisation)
  • --reference-score-rounding
  • --omit-saturation-constant
  • --single-precision-bm25
  • --top-k 1000

With those, FiQA-2018 reads 0.236 nDCG@10 against the published 0.2361 and the paper's 0.236 — the figure is reproduced at the precision it was published at, as on NFCorpus and SciFact.

The PostgreSQL row cannot use uax29: its closest configuration (english, k1=0.9/b=0.4) reads 0.2291, the same ~0.007 analysis gap the other corpora already document — not a gate cost, and not a failure of the engine.

The harness's own closing note still named --analyzer english for published figures; it now names the exact set of aligned flags.

Docs

docs/evaluation.md, docs/backends.md, and the harness's note. No behaviour change.

The previous paragraphs aligned the analysis and the operating point only, and
read 0.237 against 0.2361. The reference's aligned configuration also uses
uax29, linear gains, query-term frequency, its index statistics, its score
rounding, the omitted saturation constant and single precision: with those the
in-memory BM25 reads 0.236 against the published 0.2361 (paper 0.236). The
PostgreSQL row stays 0.2291 because it cannot use uax29, which is the analysis
gap the other corpora already document.

The harness's own closing note still pointed at --analyzer english for the
published figures; it now names the full set of flags the aligned rows use.
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.18%. Comparing base (3e595fa) to head (9042ff2).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #81      +/-   ##
==========================================
- Coverage   89.25%   89.18%   -0.08%     
==========================================
  Files         270      270              
  Lines       21383    21383              
  Branches     4621     4621              
==========================================
- Hits        19086    19071      -15     
- Misses       1375     1389      +14     
- Partials      922      923       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 5bcad44 into main Nov 17, 2026
17 checks passed
@manuc66
manuc66 deleted the docs/fiqa-reference-conventions branch November 17, 2026 06:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants