Repository navigation
Fix the ArguAna parity figure: read a rounded run as its reader does - #76
Merged
Merged
Conversation
The four-decimal rounding reproduces what the reference writes down, and a written run is read back by trec_eval, which ignores the rank column and breaks an exact score tie by document number descending (comp_sim_docno in form_res_rels.c). The harness kept the engine's order (document id ascending), so on ArguAna it reported 0.4005 where trec_eval reads 0.3970 from the same file: 5,699 exact duplicate scores, 183 queries with a tie in the top 10, 84 queries whose nDCG differs. Applying the reader's tie-break to a rounded run makes the two per-query equal, 1,406 of 1,406. The replay also retrieves at the reference run's depth (1,000) rather than 10: a document at rank 11 can enter the top ten by tying the tenth score, and that boundary alone is worth 0.0004 in the aggregate, across four queries. The pin arguana/uax29-aligned is re-recorded from 0.4005 to 0.3970 as a measured value, not a copied one. NFCorpus and SciFact are unchanged; no configuration had a tie inside their top ten. The table header now names the depth the metrics are measured at (MetricDepth.Cutoff) instead of --top-k, which had printed nDCG@1000 over an nDCG@10 value. A unit test covers the order, and --verify-reference replays 0.3970 against the new pin.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #76 +/- ##
==========================================
- Coverage 89.01% 88.92% -0.10%
==========================================
Files 262 262
Lines 20900 20900
Branches 4531 4531
==========================================
- Hits 18604 18585 -19
- Misses 1383 1399 +16
- Partials 913 916 +3 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was wrong
The evaluation harness rounded scores to four decimals to reproduce what the reference writes into a run file, then kept the engine's tie order. The published figure is read from that file by
trec_eval, which ignores the rank column and breaks an exact score tie by document number descending (comp_sim_docnoinform_res_rels.c). The two orders therefore disagree whenever the rounding creates an exact tie.On the full ArguAna run (1,406 queries, 1,000 deep):
trec_evalorder: 0.396983 (trec_eval9.0.8 prints 0.3970)Per query, against
trec_eval -q -m ndcg_cut.10: 84/1,406 differ under the engine's order, 0/1,406 undertrec_eval's.A second effect accounts for the old 0.0004 residual: the published run is 1,000 deep, and a document at rank 11 can tie the tenth score and be promoted into the measured top ten. A 10-deep replay reads 0.3974; 1,000-deep reads 0.3970.
The fix
Evaluation.RunFileOrder: score descending, then document id descending, applied to a rounded run (the run-file reproduction path).ReferenceVerifier: replayed rounded pins retrieve at the reference run's depth (1,000), not 10.MetricDepth.Cutoff(10) instead of--top-k, which had printednDCG@1000over an nDCG@10 value.arguana/uax29-alignedre-recorded from 0.4005 to 0.3970 as a measured value, not a copied one.Verification
--verify-reference:arguana/uax29-aligned0.3970, expected 0.3970, delta +0.0000. NFCorpus and SciFact unchanged.ARoundedRunIsMeasuredInTheOrderItsReaderReconstructs.