[PHM (PHM-Audit)]

Description

Adds PhysicsConsistencyScorer, a deterministic MessageTrueFalseScorer that flags responses stating physically impossible numbers — speeds above c, temperatures below absolute zero, negative masses or energies (all recognised units incl. kWh/eV/MeV/GeV), probabilities outside [0, 100%], free-fall timings that cannot happen at 1 g, and percentage shares summing above 100%.

Fully local: no LLM call, no network request, no external service. Same input → same verdict on every run. It complements LLM-based scoring (semantic) and regex threat catalogs (attacks) by covering the numeric-claim domain — modelled on GarakExploitationScorer's local-detector shape.

What this adds

• pyrit/score/true_false/physics_consistency_scorer.py — self-contained scorer: pure-function checker core + MessageTrueFalseScorer shell, fail_on parameter ("FAIL" default / "WARN")
• tests/unit/score/test_physics_consistency_scorer.py — 40+ regression cases incl. typographic minus (U+2212), range separators (0-300 °C must not parse as a negative), leading-dot decimals (.5 c), acceleration units (5e8 m/s^2 must not be flagged as a speed), the exact Fahrenheit absolute-zero boundary (float round-trip), zero-second and line-spanning fall claims, and scientific-notation percentage sums
• pyrit/score/__init__.py — package export registration

Related

• Same rule family shipped as a Promptfoo example: promptfoo/promptfoo #11955 (two rounds of automated security review, zero findings; all reviewer edge cases fixed with regression tests)
• ATR's PyRIT dataset loader (FEAT add Agent Threat Rules (ATR) adversarial payload dataset loader #1715) covers threat patterns; this covers the physical-law domain
• Deterministic-repeat check included in the test suite

Happy to adjust naming, API shape, or scope per maintainer guidance.

[PHM (PHM-Audit)]

@microsoft-github-policy-service agree

[Richard Lundeen (richlundeen)]

The local scorer fits the framework, but these four correctness issues need changes.

[PHM (PHM-Audit)]

Thank you Richard Lundeen (@richlundeen) for the precise review — all four issues are real and all four are fixed in fab9b23 (pushed to this branch).

Per point:

1. Negative energy (P2 Migrate PyRIT to open source repository #1) — fixed as suggested. The unconditional energy check is removed; Rule 3 now covers mass only (non-negativity is established there). "The hydrogen ground-state energy is -13.6 eV." is now a PASS and added as a regression, alongside -5 kWh and -1 eV controls.

2. Percentage bounds (P2 Fix link to assets #2) — fixed as suggested. Both the range rule and the share-sum rule are now gated on bounded-share context (probability/likelihood/odds/confidence/share-of/fraction-of phrasing). "The output power increased by 150%." and "The power changed by -5%." are PASS regressions; "We are 150% confident" and "a probability of -5%" remain FAIL; the 60%+70% of traffic WARN case is preserved via the same gate.

3. Signed c velocities (P2 Expand pre-commit hook to all files #3) — fixed as suggested. The c-branch now applies abs(value) before conversion, matching the m/s and km/s branches. "The particle moved at -1.5 c." is a FAIL regression through the public scoring entry point, and "-0.5 c." is the below-threshold signed control.

4. Free-fall parsing (P2 Upgrade transformers to >=4.36.0 to address dependabot alert #4) — fixed with both halves of the suggestion. The capture grammar is restricted to well-formed numbers (so "...", "1.2.3" can no longer enter float()), and conversions go through a defensive _safe_float (malformed/nonfinite → explicit WARN "malformed-fall-claim", never an exception). Malformed-height and malformed-time regressions added; they do not abort scoring under either threshold.

rules_version is bumped to 3 so stored identifiers reflect the semantics change. All 31 parametrized cases pass locally. Of the updated expectations, two flipped as a direct consequence of the energy semantics change (-5 kWh, -1 eV from FAIL to PASS); the remaining updates are the eleven new regressions named above. A negative energy value is never a FAIL on its own — if a negative mass co-occurs in the same text, the mass rule fires (covered by the existing -5 kg / -5 g cases).

One note on scope: the malformed-fall WARN does not change the verdict, only visibility of unparseable claims — if you prefer the strict no-op reading, dropping the WARN is a one-line change.

[Copilot]

🟡 Changes recommended

The scorer incorrectly applies bounds and sums percentages across unrelated claims, causing valid growth statements and separate distributions to be flagged as failures or warnings.

2 open issues

• Scope percentage bounds to each claim's context · New
• Sum only shares belonging to the same distribution · New

What changed in this PR

Adds PhysicsConsistencyScorer, a local, deterministic scorer that complements PyRIT’s model-based scoring by checking numeric physical claims without external services.

Changes:

• Adds physical-limit checks and configurable FAIL/WARN detection thresholds.
• Covers numeric parsing, verdicts, metadata, and repeatability with regression tests.
• Registers the scorer through the package’s lazy exports.

File	Description
tests/​unit/​score/​test_physics_consistency_scorer.py	Adds regression coverage for rules, parsing edge cases, and scorer behavior.
pyrit/​score/​true_false/​physics_consistency_scorer.py	Implements the numeric checker and true/false scorer.
pyrit/​score/​__init__.py	Exposes the scorer through lazy loading and type-checking imports.

_{Review effort: Balanced}

[PHM (PHM-Audit)]

Both Copilot points are addressed in 080afd9 (pushed to this branch) — the fix moves from response-level context to per-claim (sentence-level) context, which addresses both points:

1. Percentage bounds scoped per claim. Rule 4 now evaluates the bounded-share context of the sentence each percentage lives in. The mixed case "The output power increased by 150%. The probability of success is 50%." is now a PASS regression — the 150% growth claim is no longer bounded by the probability sentence two claims later.

2. Shares summed per distribution. Rule 6 now groups percentages by their sentence and only sums within a sentence that itself establishes a bounded-share context. The separate-distributions case "A took 60% and B took 40% of traffic. C took 70% and D took 30% of the budget." is a PASS regression (each claim sums to exactly 100%). Mixed growth/share text still warns on the share sentence only: "The power increased by 150%. A took 60% and B took 70% of traffic." → WARN from the 130% share sum, with no contribution from the growth claim.

Regressions added: mixed growth/probability, separate distributions, and mixed growth/share (34/34 parametrized cases pass locally).

[PHM (PHM-Audit)]

@microsoft-github-policy-service agree

[Richard Lundeen (richlundeen)]

Follow-up to the original review, the Copilot review, and the author's first and second fix updates. Specific responses are recorded in the six original inline threads; this is the single consolidated summary:
I verified the author's energy, signed-c and malformed-fall fixes first, then repaired the remaining percentage claim ownership/distribution, decimal/exponent parsing and signed-g context issues. I also restored the six validation workflows to the selected upstream versions. The repaired candidate passes 356 scoped tests, Ruff and the scoped type check; remote CI still needs maintainer workflow approval, which is not claimed as passed.

Verified on published commit bd2176e8e5f58250bef902e0ff7905a5d53cef4b.

[Richard Lundeen (richlundeen)]

Automated review completed. No blocking concerns remain. This review records source coverage; it does not authorize merge.

[Richard Lundeen (richlundeen)]

The physics scorer has a useful place here: its local numeric conversions and arithmetic are not equivalent to existing regex patterns or a target-backed judge. The remaining inclusion issue at 8a163d9840f18dea799f3372877c8a5f7beaf475 is unrelated scope, not a request to rewrite that scorer.

Please remove the changes to these four files, retaining their current main versions:

• .github/PULL_REQUEST_TEMPLATE.md
• .github/instructions/frontend-test.instructions.md
• .github/instructions/scorers.instructions.md
• .github/instructions/test.instructions.md

The current PR diff restores an older PR template and deletes existing frontend usability, typed-scorer/condition, and scenario dataset-isolation guidance. Those rollbacks add no physics-scoring capability. They came from 7d4e1e41a2636efabf3cb4831a0d49b5b749d12b (the PAT-scope maintenance commit), but these four files are not workflows. Keeping the scorer, export, tests, and paired documentation while excluding these unrelated changes preserves one coherent unit of value, consistent with the framework guidance in #4175.

I have preserved the existing automatic merge enrollment; this comment is not an approval or confirmation that the PR has entered the merge queue or merged. The automated readiness gate remains needs-inclusion until the scope correction is accepted and checked.

Update: the scope correction is verified at published commit b386647564347bae46aaf34447370cb194210640. The four files listed above are absent from the current PR diff, and that commit changes only those four files; the scorer, export, tests, and paired documentation are unchanged by it. This scope concern is addressed. This update is not a full re-review, approval, or merge confirmation.

[PHM (PHM-Audit)]

Done — commit b386647 removes the four unrelated files (PULL_REQUEST_TEMPLATE.md and the three instructions files), restoring their current main versions exactly. The scorer, export, tests, and paired documentation are unchanged from 8a163d9. Ready for the readiness gate to re-check.