Repository navigation
Conversation
for more information, see https://pre-commit.ci
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #185 so this PR includes the shared CI prerequisites. Review/merge #185 first, then retarget this PR to
main.Fuzzing negatives currently reveal their labels through a forced token position, resetting the random seed per example, and a mark count calculated differently from positive marks. Randomizing positions alone leaves the count shortcut described in https://arxiv.org/abs/2610.37676.
Transfer each retained positive's complete mark set to one negative of the same token length. This makes the empirical distribution of
(window length, marked positions)identical across labels, so every rule based solely on that signature has balanced accuracy 0.5. Save the actual marked indices in classifier output for auditing, and use the configured threshold consistently.Default mode uses target-inactive windows, including neighbour-derived negatives whose stored activations describe a different feature. Active mode uses maximum matching to transfer masks only onto zero-target-activation tokens. Unmatched pairs are omitted with a coverage warning; an empty matching fails explicitly. Active-mode coverage can therefore change, and the new scores should not be compared directly with the old protocol.
Also fix the rendering edge cases: a single eligible token no longer triggers unrelated fallback text, all-zero windows are not highlighted as correct, threshold-boundary tokens cannot be both correct and incorrect, and impossible incorrect marks fail explicitly.
Validation: 17 focused fuzzing/highlighting tests pass independently, including equality of the complete mark distribution across five seeds, unequal counts and lengths, active matching, and threshold edge cases. The combined fix set passes 49 tests; Ruff passes. Tests use real CPU Torch/scorer implementations with only the eager vLLM package initializer bypassed. No LLM scoring benchmark was run; shuffled-explanation controls remain necessary.
The existing collection failure on main is addressed separately by #185.