Skip to content

Decompose FiQA's analysis gap instead of naming it - #82

Merged
manuc66 merged 1 commit into
mainfrom
docs/fiqa-analysis-attribution
Nov 17, 2026
Merged

manuc66 merged 1 commit into
mainfrom
docs/fiqa-analysis-attribution

Conversation

@manuc66

@manuc66 manuc66 commented Nov 17, 2026

Copy link
Copy Markdown
Owner

What changed

The FiQA pages attributed the PostgreSQL row's 0.007 to "analysis". It is now decomposed by measurement:

FiQA-2018, nDCG@10, reference conventions value
In-memory BM25, uax29 0.236
In-memory BM25, english (Porter + 33 words) 0.235
PostgreSQL BM25, english (Snowball + 127 words) 0.2291
PostgreSQL BM25, Snowball + the reference's 33-word stop list 0.2196

So the stop list is not the cause: matching the reference's 33 words in PostgreSQL makes it worse, because the longer stock list removes noise on this corpus. What remains is the stemmer (Snowball vs Porter) and the parser, neither replaceable on stock PostgreSQL without an extension. The gap is the portable path's floor, not a gate cost.

The custom configuration used for the last row was created in the test server for this measurement; it is not shipped, and the pages do not present it as a supported option.

Docs

docs/evaluation.md and docs/backends.md carry the decomposition. No behaviour change.

A PostgreSQL configuration carrying the reference's 33-word stop list and
Snowball reads 0.2196, below the stock english configuration's 0.2291: the
stop list is not the cause, the longer stock list helps, and the remainder is
the stemmer and the parser. The pages now say that rather than leaving the
0.007 attributed to "analysis".
@codecov-commenter

codecov-commenter commented Nov 17, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.27%. Comparing base (5bcad44) to head (efec41b).

Additional details and impacted files
@@           Coverage Diff           @@
##             main      #82   +/-   ##
=======================================
  Coverage   89.27%   89.27%           
=======================================
  Files         270      270           
  Lines       21383    21383           
  Branches     4621     4621           
=======================================
  Hits        19090    19090           
  Misses       1373     1373           
  Partials      920      920           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit cb9bedd into main Nov 17, 2026
16 checks passed
@manuc66
manuc66 deleted the docs/fiqa-analysis-attribution branch November 17, 2026 06:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants