Skip to content

fix: match fuzzing mark distributions between labels - #186

Open
SrGonao wants to merge 4 commits into
fix/ci-module-cache-isolationfrom
fix/fuzzing-matched-mark-sets
Open

SrGonao wants to merge 4 commits into
fix/ci-module-cache-isolationfrom
fix/fuzzing-matched-mark-sets

Conversation

@SrGonao

@SrGonao SrGonao commented Oct 31, 2026 •

Copy link
Copy Markdown
Collaborator

Stacked on #185 so this PR includes the shared CI prerequisites. Review/merge #185 first, then retarget this PR to main.

Fuzzing negatives currently reveal their labels through a forced token position, resetting the random seed per example, and a mark count calculated differently from positive marks. Randomizing positions alone leaves the count shortcut described in https://arxiv.org/abs/2610.37676.

Transfer each retained positive's complete mark set to one negative of the same token length. This makes the empirical distribution of (window length, marked positions) identical across labels, so every rule based solely on that signature has balanced accuracy 0.5. Save the actual marked indices in classifier output for auditing, and use the configured threshold consistently.

Default mode uses target-inactive windows, including neighbour-derived negatives whose stored activations describe a different feature. Active mode uses maximum matching to transfer masks only onto zero-target-activation tokens. Unmatched pairs are omitted with a coverage warning; an empty matching fails explicitly. Active-mode coverage can therefore change, and the new scores should not be compared directly with the old protocol.

Also fix the rendering edge cases: a single eligible token no longer triggers unrelated fallback text, all-zero windows are not highlighted as correct, threshold-boundary tokens cannot be both correct and incorrect, and impossible incorrect marks fail explicitly.

Validation: 17 focused fuzzing/highlighting tests pass independently, including equality of the complete mark distribution across five seeds, unequal counts and lengths, active matching, and threshold edge cases. The combined fix set passes 49 tests; Ruff passes. Tests use real CPU Torch/scorer implementations with only the eager vLLM package initializer bypassed. No LLM scoring benchmark was run; shuffled-explanation controls remain necessary.

The existing collection failure on main is addressed separately by #185.

@SrGonao
SrGonao changed the base branch from main to fix/ci-module-cache-isolation November 1, 2026 14:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant