Repository navigation
Decompose FiQA's analysis gap instead of naming it - #82
Merged
Merged
Conversation
A PostgreSQL configuration carrying the reference's 33-word stop list and Snowball reads 0.2196, below the stock english configuration's 0.2291: the stop list is not the cause, the longer stock list helps, and the remainder is the stemmer and the parser. The pages now say that rather than leaving the 0.007 attributed to "analysis".
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #82 +/- ##
=======================================
Coverage 89.27% 89.27%
=======================================
Files 270 270
Lines 21383 21383
Branches 4621 4621
=======================================
Hits 19090 19090
Misses 1373 1373
Partials 920 920 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
The FiQA pages attributed the PostgreSQL row's 0.007 to "analysis". It is now decomposed by measurement:
uax29english(Porter + 33 words)english(Snowball + 127 words)So the stop list is not the cause: matching the reference's 33 words in PostgreSQL makes it worse, because the longer stock list removes noise on this corpus. What remains is the stemmer (Snowball vs Porter) and the parser, neither replaceable on stock PostgreSQL without an extension. The gap is the portable path's floor, not a gate cost.
The custom configuration used for the last row was created in the test server for this measurement; it is not shipped, and the pages do not present it as a supported option.
Docs
docs/evaluation.mdanddocs/backends.mdcarry the decomposition. No behaviour change.