Skip to content

Fix the ArguAna parity figure: read a rounded run as its reader does - #76

Merged
manuc66 merged 1 commit into
mainfrom
fix/arguana-parity-tiebreak
Nov 16, 2026
Merged

manuc66 merged 1 commit into
mainfrom
fix/arguana-parity-tiebreak

Conversation

@manuc66

@manuc66 manuc66 commented Nov 16, 2026

Copy link
Copy Markdown
Owner

What was wrong

The evaluation harness rounded scores to four decimals to reproduce what the reference writes into a run file, then kept the engine's tie order. The published figure is read from that file by trec_eval, which ignores the rank column and breaks an exact score tie by document number descending (comp_sim_docno in form_res_rels.c). The two orders therefore disagree whenever the rounding creates an exact tie.

On the full ArguAna run (1,406 queries, 1,000 deep):

  • 5,699 exact duplicate scores
  • 183 queries with a tie inside the top 10
  • 84 queries whose nDCG changes with the tie order
  • engine order: 0.400517
  • trec_eval order: 0.396983 (trec_eval 9.0.8 prints 0.3970)

Per query, against trec_eval -q -m ndcg_cut.10: 84/1,406 differ under the engine's order, 0/1,406 under trec_eval's.

A second effect accounts for the old 0.0004 residual: the published run is 1,000 deep, and a document at rank 11 can tie the tenth score and be promoted into the measured top ten. A 10-deep replay reads 0.3974; 1,000-deep reads 0.3970.

The fix

  • Evaluation.RunFileOrder: score descending, then document id descending, applied to a rounded run (the run-file reproduction path).
  • ReferenceVerifier: replayed rounded pins retrieve at the reference run's depth (1,000), not 10.
  • The table header names MetricDepth.Cutoff (10) instead of --top-k, which had printed nDCG@1000 over an nDCG@10 value.
  • Pin arguana/uax29-aligned re-recorded from 0.4005 to 0.3970 as a measured value, not a copied one.
  • Docs, harness README, changelog and reproducibility script updated; the landing page now says 0.3970/0.3970 instead of "0.0035 short".

Verification

  • --verify-reference: arguana/uax29-aligned 0.3970, expected 0.3970, delta +0.0000. NFCorpus and SciFact unchanged.
  • New unit test: ARoundedRunIsMeasuredInTheOrderItsReaderReconstructs.
  • The run file is byte-identical before and after the code change; only the metric's reading order changes.

The four-decimal rounding reproduces what the reference writes down, and a
written run is read back by trec_eval, which ignores the rank column and
breaks an exact score tie by document number descending (comp_sim_docno in
form_res_rels.c). The harness kept the engine's order (document id
ascending), so on ArguAna it reported 0.4005 where trec_eval reads 0.3970
from the same file: 5,699 exact duplicate scores, 183 queries with a tie in
the top 10, 84 queries whose nDCG differs. Applying the reader's tie-break
to a rounded run makes the two per-query equal, 1,406 of 1,406.

The replay also retrieves at the reference run's depth (1,000) rather than
10: a document at rank 11 can enter the top ten by tying the tenth score,
and that boundary alone is worth 0.0004 in the aggregate, across four
queries. The pin arguana/uax29-aligned is re-recorded from 0.4005 to 0.3970
as a measured value, not a copied one. NFCorpus and SciFact are unchanged;
no configuration had a tie inside their top ten.

The table header now names the depth the metrics are measured at
(MetricDepth.Cutoff) instead of --top-k, which had printed nDCG@1000 over an
nDCG@10 value. A unit test covers the order, and --verify-reference replays
0.3970 against the new pin.
@codecov-commenter

codecov-commenter commented Nov 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.92%. Comparing base (76b6565) to head (834def0).

Additional details and impacted files
@@            Coverage Diff             @@
##             main      #76      +/-   ##
==========================================
- Coverage   89.01%   88.92%   -0.10%     
==========================================
  Files         262      262              
  Lines       20900    20900              
  Branches     4531     4531              
==========================================
- Hits        18604    18585      -19     
- Misses       1383     1399      +16     
- Partials      913      916       +3     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@manuc66
manuc66 merged commit 8ab690c into main Nov 16, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants