Repository navigation
Name the reference conventions FiQA's aligned figure needs - #81
Merged
Merged
Conversation
The previous paragraphs aligned the analysis and the operating point only, and read 0.237 against 0.2361. The reference's aligned configuration also uses uax29, linear gains, query-term frequency, its index statistics, its score rounding, the omitted saturation constant and single precision: with those the in-memory BM25 reads 0.236 against the published 0.2361 (paper 0.236). The PostgreSQL row stays 0.2291 because it cannot use uax29, which is the analysis gap the other corpora already document. The harness's own closing note still pointed at --analyzer english for the published figures; it now names the full set of flags the aligned rows use.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #81 +/- ##
==========================================
- Coverage 89.25% 89.18% -0.08%
==========================================
Files 270 270
Lines 21383 21383
Branches 4621 4621
==========================================
- Hits 19086 19071 -15
- Misses 1375 1389 +14
- Partials 922 923 +1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The FiQA paragraphs said the aligned figure was 0.237 against 0.2361. That run aligned the analysis (
english) and the operating point (k1=0.9/b=0.4) only — it was not the reference's configuration.The reference's aligned rows also use:
--analyzer uax29(the published separator rules, apostrophes inside a word, possessives stripped before the stop list)--ndcg-gain linear--query-term-frequency--reference-index-statistics(average over non-empty documents, one-byte length quantisation)--reference-score-rounding--omit-saturation-constant--single-precision-bm25--top-k 1000With those, FiQA-2018 reads 0.236 nDCG@10 against the published 0.2361 and the paper's 0.236 — the figure is reproduced at the precision it was published at, as on NFCorpus and SciFact.
The PostgreSQL row cannot use
uax29: its closest configuration (english, k1=0.9/b=0.4) reads 0.2291, the same ~0.007 analysis gap the other corpora already document — not a gate cost, and not a failure of the engine.The harness's own closing note still named
--analyzer englishfor published figures; it now names the exact set of aligned flags.Docs
docs/evaluation.md,docs/backends.md, and the harness's note. No behaviour change.