Skip to content

Add FiQA-2018 as the candidate gate's third corpus - #79

Merged
manuc66 merged 1 commit into
mainfrom
feat/fiqa-gate-evidence
Nov 16, 2026
Merged

manuc66 merged 1 commit into
mainfrom
feat/fiqa-gate-evidence

Conversation

@manuc66

@manuc66 manuc66 commented Nov 16, 2026

Copy link
Copy Markdown
Owner

What this adds

FiQA-2018 as a fourth selectable corpus — --dataset fiqa — and deliberately not part of --dataset all or the pinned comparison. The three tables were built on NFCorpus, SciFact and ArguAna; a corpus that is not pinned must not silently join them. A test pins that choice.

The reason for the corpus is the candidate gate's evidence. Both corpora the gate was measured on are scientific and rich in specific terms, which is its best case. FiQA is the other nature: 57,638 documents, 648 test queries, financial Q&A with short queries.

Measured

nDCG@10, stock-PostgreSQL BM25, pool 1,000, same analysis (simple + unaccent):

FiQA-2018 gate, budget 1,000 gate off Anserini BM25 (its own analysis)
nDCG@10 0.2061 0.1077 0.2361

The direction is SciFact's, so the effect is the candidate set (coverage + truncation vs idf), not a scientific-corpus accident. Reproduce:

dotnet run --project bench/LexiSharp.Eval -c Release -- --dataset fiqa --no-tuned \
  --backend bm25 --backend-gate rare --backend-connection "Host=...;Database=..."
dotnet run --project bench/LexiSharp.Eval -c Release -- --dataset fiqa --no-tuned \
  --backend bm25 --backend-gate off  --backend-connection "Host=...;Database=..."

Data

  • Download URL and md5 are in the BeirDataset entry; the loader fetches and verifies them like the other three. The corpus is not committed (bench/LexiSharp.Eval/data/ is ignored).
  • Anserini indexes 57,600 of the file's 57,638 documents because 38 have no text. The entry records 57,600/57,600 as published; the harness's 3% fingerprint tolerance reports no difference, and the changelog says where the number comes from.
  • The license table in bench/LexiSharp.Eval/README.md gains a row with unconfirmed terms, the same treatment NFCorpus and ArguAna already get.

Docs

docs/backends.md (BM25 section), docs/evaluation.md (SQL backends), docs/index.md, the harness README and the changelog carry the corpus and the numbers. The "two corpora" caveat becomes "two scientific corpora and one financial, same direction".

The gate was measured on NFCorpus and SciFact, both scientific and rich in
specific terms -- its best case. FiQA-2018 is the other nature: 57,638
documents, 648 test queries, financial Q&A with short queries. At pool 1,000
the gate reads 0.2061 nDCG@10 against 0.1077 with it off; Anserini's
published BM25 is 0.2361 with its own analysis. The direction is SciFact's,
so the result is the candidate set and not a scientific-corpus accident.

It is selectable with --dataset fiqa and deliberately out of --dataset all:
the pinned tables were built on the three corpora, and a fourth row that is
not pinned must not silently join them. A test pins that choice.

Anserini indexes 57,600 of the file's 57,638 documents because 38 have no
text; the fingerprint's 3% tolerance reports no difference, and the entry
says where the number comes from.
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.18%. Comparing base (65c9e34) to head (8b86a27).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #79      +/-   ##
==========================================
- Coverage   89.24%   89.18%   -0.07%     
==========================================
  Files         270      270              
  Lines       21383    21383              
  Branches     4621     4621              
==========================================
- Hits        19084    19071      -13     
- Misses       1376     1389      +13     
  Partials      923      923              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 614f642 into main Nov 16, 2026
17 checks passed
@manuc66
manuc66 deleted the feat/fiqa-gate-evidence branch November 16, 2026 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants