Repository navigation
Add FiQA-2018 as the candidate gate's third corpus - #79
Merged
Merged
Conversation
The gate was measured on NFCorpus and SciFact, both scientific and rich in specific terms -- its best case. FiQA-2018 is the other nature: 57,638 documents, 648 test queries, financial Q&A with short queries. At pool 1,000 the gate reads 0.2061 nDCG@10 against 0.1077 with it off; Anserini's published BM25 is 0.2361 with its own analysis. The direction is SciFact's, so the result is the candidate set and not a scientific-corpus accident. It is selectable with --dataset fiqa and deliberately out of --dataset all: the pinned tables were built on the three corpora, and a fourth row that is not pinned must not silently join them. A test pins that choice. Anserini indexes 57,600 of the file's 57,638 documents because 38 have no text; the fingerprint's 3% tolerance reports no difference, and the entry says where the number comes from.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #79 +/- ##
==========================================
- Coverage 89.24% 89.18% -0.07%
==========================================
Files 270 270
Lines 21383 21383
Branches 4621 4621
==========================================
- Hits 19084 19071 -13
- Misses 1376 1389 +13
Partials 923 923 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this adds
FiQA-2018 as a fourth selectable corpus —
--dataset fiqa— and deliberately not part of--dataset allor the pinned comparison. The three tables were built on NFCorpus, SciFact and ArguAna; a corpus that is not pinned must not silently join them. A test pins that choice.The reason for the corpus is the candidate gate's evidence. Both corpora the gate was measured on are scientific and rich in specific terms, which is its best case. FiQA is the other nature: 57,638 documents, 648 test queries, financial Q&A with short queries.
Measured
nDCG@10, stock-PostgreSQL BM25, pool 1,000, same analysis (
simple+unaccent):The direction is SciFact's, so the effect is the candidate set (coverage + truncation vs idf), not a scientific-corpus accident. Reproduce:
Data
BeirDatasetentry; the loader fetches and verifies them like the other three. The corpus is not committed (bench/LexiSharp.Eval/data/is ignored).bench/LexiSharp.Eval/README.mdgains a row with unconfirmed terms, the same treatment NFCorpus and ArguAna already get.Docs
docs/backends.md(BM25 section),docs/evaluation.md(SQL backends),docs/index.md, the harness README and the changelog carry the corpus and the numbers. The "two corpora" caveat becomes "two scientific corpora and one financial, same direction".