Repository navigation
FIX: Apply Cyber's malware scoring question in its default scorer - #4179
Draft
Richard Lundeen (richlundeen) wants to merge 1 commit into
Draft
Richard Lundeen (richlundeen) wants to merge 1 commit into
Richard Lundeen (richlundeen) wants to merge 1 commit into
Conversation
Cyber defined get_override_composite_scorer_questions_path(), but the Scenario base only calls _get_additional_scoring_questions(). The names diverged in b0e91f8 (microsoft#1695), so Cyber's default objective scorer never used malware.yaml and fell back to the registry default scorer or the inverted refusal scorer. Rename the override to the hook the base calls, bump VERSION to 4 since default scoring changes, and test that the default scorer is the malware question AND not-refused composite. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues were reported for the change connecting Cyber’s malware scoring to the default scorer.
No open issues 🎉
What changed in this PR
This PR connects Cyber’s malware scoring question to the scenario base class’s default-scorer hook, so default success scoring requires malware content and no refusal.
Changes:
- Rename Cyber’s hook to
_get_additional_scoring_questions(). - Bump
VERSIONfrom 3 to 4 to prevent reuse of cached results scored under the previous behavior. - Add a regression test checking the malware scorer and inverted-refusal composite.
| File | Description |
|---|---|
tests/unit/scenario/airt/test_cyber.py |
Update the version assertion and verify the default scorer composition. |
pyrit/scenario/scenarios/airt/cyber.py |
Correct the scoring hook name and bump the scenario version. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Cyber's default objective scorer never used its malware question. This PR connects it so that, by default, Cyber counts a response as a success only when it contains concrete malware and isn't a refusal.
get_override_composite_scorer_questions_path(), butScenario._get_default_objective_scorer()only calls_get_additional_scoring_questions(). Nothing called Cyber's method, so the base hook returned()andmalware.yamlwas never loaded._get_default_objective_scorer()override, which built malware AND not-refused, with the class-method hook. Leakage and Scam were converted to the hook's name; Cyber was not._get_additional_scoring_questions(). The base now buildsTrueFalseCompositeScorer(AND, [malware question, inverted refusal]). Like every scenario that declares scoring questions, this takes precedence over a registered default objective scorer and reuses that scorer's chat target.VERSION3 → 4: default scoring changes, souse_cachedwon't match runs scored the old way.main.Other scenarios with scoring questions (Leakage, Scam, AdversarialBenchmark, FigStep) already use the correct name. Cyber is about to be held up as the model scenario in #4175, so this also stops the dead method from being copied.
🤖 Generated with Claude Code