FEAT: ASSERT headline metrics (-3) by richlundeen · Pull Request #4131 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:28:32Z

Part -3 of a 10-layer ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics	this PR
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	Results grouped by generated category
-10	Planned	Background dataset loading

Each PR targets main and includes the earlier layers, so its commit list isn't a review boundary. Review this layer's source changes in Files changed against -2. -8 still carries prototype commits in its ancestry, so reviewers of later layers should use source comparisons like this one rather than commit lists.

Description

This PR's own commits (the diff against #4130's head is this PR's change):

a24ce22 FEAT: ASSERT headline metrics for foundry.assert_policy runs
166de36 FIX: Address ASSERT metrics review feedback
265a2e6 DOC: Refresh ASSERT metrics notebook output for assert_package
b70cb51 FIX: Read ASSERT judgment metadata through the typed models
b5e5931 Merge published ASSERT metrics layer onto the rebuilt trace evidence layer
cee992a FIX: Identify ASSERT runs by registry name and keep supplied scorers distinct
880a61f FIX: Use only proven ASSERT provenance for metrics policies and dimensions
145b788 FIX: Require built-in ASSERT judge evidence for metrics policies
d0b7c2a FIX: Stop attributing ASSERT violations to the mutable assigned category

foundry.assert_policy reports one success rate per technique, which doesn't separate harm from overrefusal. This PR adds ASSERT's two headline metrics, Impermissible behavior violated and Permissible behavior violated, split by the policy categories the judge found relevant, plus per-dimension and per-category violation rates, in the SDK, the REST API, and on the GUI run page. This was a follow-up listed in #4033.

compute_assert_policy_metrics_async(scenario_result) in pyrit.analytics returns AssertPolicyMetrics (in pyrit.models). For each technique (by name, with the baseline separate, never pooled) it reports:
- The permissible/impermissible split. Each judged conversation counts once per stance it was judged relevant to, from the policy_violation judge's per-category judgments, as in ASSERT.
- A violation rate for each judge dimension, such as policy_violation, overrefusal, or a custom one like budget_compliance. Not-applicable answers and judge replies without a verdict are counted outside the denominator; graded dimensions also report grade counts in declared order and the median, as in ASSERT.
- A per-category violation rate over the conversations that judged each category relevant (ASSERT's per-node findings), kept separate for each policy the judge used, so policies that share a behavior or category name aren't pooled.
- Conversations left out: not-judged counts by reason, plus judged conversations left out of the split because they have no matching stored policy or no per-category judgments.
Each rate is an AssertBinaryRate with ASSERT's 90% Wilson interval and a flag for fewer than 5 applicable conversations. Each row is the latest attempt for one execution unit, as in compute_scenario_statistics.
Provenance. Results are read only from evidence the run itself recorded:
- Judge scores are found by the dimension in AssertDimensionResult (written by [BREAKING] FEAT: ASSERT judge dimensions and presets (-1) #4132's judges). Older unmarked scores count only under the scenario's built-in judge wrapper; a supplied objective_scorer is reported as the objective dimension, whatever its score category.
- A conversation's taxonomy comes from the stored seed group with exactly the seeds the result recorded, and only when it matches the taxonomy the built-in policy judge recorded in its saved scorer identifier. A replaced or edited dataset, or a policy score without that record, leaves the conversation without a matching stored policy instead of using today's seed metadata.
- Difference from ASSERT: no violation is attributed to the category a conversation was generated for. PyRIT's judge recomputes policy_violation from its category judgments, so every violation names a violated category; a stored verdict that disagrees with its category judgments is rejected as malformed.
A result counts as ASSERT when its saved registry name is foundry.assert_policy; older results without one must come from PyRIT's own AssertPolicy class and module.
GET /api/scenarios/runs/{id}/assert-metrics: other scenarios return 400 with problem type /errors/not-assert-policy-run, unfinished runs 409, and unreadable or malformed saved results a sanitized 500.
Docs: the Foundry AssertPolicy section and the analytics guide.

Cross-cutting changes outside the metrics files: one ownership bullet in doc/code/framework.md; ScenarioRunNotCompletedError in scenario_run_service.py (same 409); the assert-metrics route in routes/scenarios.py; lazy exports in pyrit.models and pyrit.analytics; a three-line panel mount in ScenarioRunPage.tsx.

Demo (isolated synthetic run with the unit-test harness's scripted target and judge: four generated groups, baseline and prompt_sending):

assert_metrics = await compute_assert_policy_metrics_async(assert_scenario_result)

baseline: Impermissible behavior violated: 100% (1/1)
baseline: Permissible behavior violated: 33% (1/3)
prompt_sending: Impermissible behavior violated: 100% (1/1)
prompt_sending: Permissible behavior violated: 33% (1/3)

The Foundry notebook's metrics cell is synced from foundry.py but has no output here; it wasn't re-run against live models for this revision.

GUI behavior and scope

Own delta: completed ASSERT run pages only. Adds ASSERT policy metrics before the atomic attack hierarchy. The panel loads from a dedicated endpoint and has loading, error with Retry, and empty-run states. It is absent from ordinary non-ASSERT runs and from running/incomplete ASSERT runs. A run whose saved registry name is another scanner's never shows it; an older run without a registry name shows it only until the route answers with /errors/not-assert-policy-run, and other errors (compatibility, validation, 409, 500) still appear.

Rows separate each technique and the baseline rather than pooling them. Columns show judged counts, permissible/impermissible violation rates, dynamic judge-dimension rates, violated/applicable counts, visible 90% confidence intervals, and low-support badges for fewer than five applicable conversations. Graded dimensions show grade counts and a median. N/A/no-verdict exclusions accompany available rates; a warning identifies conversations left out, including those without a matching stored policy. Rates by policy category is collapsible; when one behavior has several judged policies, its rows add a short policy ID.

Current limitation, not fixed here: when a rate is null, its cell renders a dash before N/A/no-verdict details. An all-N/A dimension can therefore look like no data.

Screenshots are from the exact sources named below, with isolated SQLite and synthetic data: four generated groups, the baseline and red_teaming, scripted targets and judges, and no model or provider calls. Before and after are independent runs with different scripted judge outputs, so their numbers aren't comparable. The empty, incomplete and ordinary runs are seeded records; loading is held by a test fixture, and the error is a controlled one-time 500.

Before (#4130 head 06ccdf2a2), a completed ASSERT run has no metrics panel:

After (d0b7c2ac2):

Loading	Error with Retry	Empty run

Ordinary non-ASSERT run (no panel)	Incomplete ASSERT run (no panel)

Historical screenshots (superseded, from the earlier combined head c58b389, before this revision's provenance changes)

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Critical aggregation and identity-resolution defects can exclude valid judgments and conversations from reported rates, while saved-result validation failures incorrectly return HTTP 409 instead of 500.

3 open issues

What changed in this PR

Adds exact ASSERT headline metrics for foundry.assert_policy, replacing assigned-category approximations with splits based on judge relevance. The metrics integrate with PyRIT’s analytics API, REST endpoint, and completed-run UI.

Changes:

Adds typed metrics, Wilson confidence intervals, macro averages, and per-technique breakdowns.
Exposes ASSERT metrics through the backend and run page, including unavailable results and low-support indicators.
Includes supporting judge presets, trace evidence infrastructure, tests, and documentation from the stacked changes.

File	Description
`third_party/assert-provenance.json`	Records ASSERT adaptation provenance.
`THIRD_PARTY_NOTICES.txt`	Updates attribution for judges and presets.
`tests/unit/score/test_otlp_json_trace_client.py`	Covers OTLP file parsing and completeness.
`tests/unit/score/test_fallback_scorer.py`	Covers evidence-based fallback behavior.
`tests/unit/score/test_assert_judge_presets.py`	Tests preset loading and dimension resolution.
`tests/unit/models/test_execution_trace.py`	Tests execution snapshot validation.
`tests/unit/models/test_assert_judge.py`	Tests judge configuration models.
`tests/unit/common/test_lazy_package_imports.py`	Updates analytics export expectations.
`tests/unit/backend/test_assert_policy_resume.py`	Tests restoration of judge and trace settings.
`tests/unit/backend/test_assert_metrics_routes.py`	Covers metrics endpoint responses and validation.
`tests/unit/analytics/test_rate_statistics.py`	Tests Wilson intervals and macro averages.
`pyrit/score/true_false/true_false_fallback_scorer.py`	Exposes configurable fallback triggers.
`pyrit/score/scorer.py`	Identifies missing-evidence results.
`pyrit/score/observation/trace_client.py`	Adds per-trace failures and completed-trace eviction.
`pyrit/score/observation/otlp_json_trace_client.py`	Reads exported OTLP trace files.
`pyrit/score/observation/otel_trace_source.py`	Shares trace querying and scope validation.
`pyrit/score/observation/otel_span_exporter.py`	Captures events and per-trace export failures.
`pyrit/score/observation/message_trace_resolver.py`	Resolves conversation request trace links.
`pyrit/score/observation/execution.py`	Shares acquisitions and supports execution snapshots.
`pyrit/score/observation/__init__.py`	Exports trace observation utilities lazily.
`pyrit/score/float_scale/float_scale_fallback_scorer.py`	Exposes configurable float-score fallback triggers.
`pyrit/score/fallback_scorer.py`	Implements evidence-triggered fallback and propagation.
`pyrit/score/assert_judge_presets.py`	Loads and merges judge presets.
`pyrit/score/__init__.py`	Exports ASSERT and tracing APIs lazily.
`pyrit/scenario/core/scenario.py`	Extracts scenario identity parameter selection.
`pyrit/prompt_target/common/target_trace_config.py`	Adds conversation identifiers to request spans.
`pyrit/prompt_target/common/prompt_target.py`	Exposes target tracing configuration.
`pyrit/models/score/trace.py`	Defines execution steps, content, and summaries.
`pyrit/models/score/observation.py`	Adds validated execution-trace observations.
`pyrit/models/score/assert_judge.py`	Defines dimensions, ordinal scales, and presets.
`pyrit/models/score/__init__.py`	Exports execution-trace models.
`pyrit/models/results/assert_policy_metrics.py`	Defines ASSERT metric response contracts.
`pyrit/models/analytics.py`	Adds binary-rate and macro-rate models.
`pyrit/models/__init__.py`	Exports metrics, judge, and trace models.
`pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml`	Guides judges on trace evidence and citations.
`pyrit/datasets/score/assert/judge_presets/tool-use.yaml`	Adds tool-use judging dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-extended.yaml`	Adds harm and privacy dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-core.yaml`	References built-in safety dimensions.
`pyrit/datasets/score/assert/judge_presets/robustness.yaml`	Adds injection and boundary dimensions.
`pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml`	Adds policy-adherence judging.
`pyrit/datasets/score/assert/judge_presets/operational.yaml`	Adds workflow and escalation dimensions.
`pyrit/datasets/score/assert/judge_presets/multi-turn.yaml`	Adds context-retention and goal dimensions.
`pyrit/datasets/score/assert/judge_presets/instruction-following.yaml`	Adds constraint and completion dimensions.
`pyrit/datasets/score/assert/judge_presets/grounding.yaml`	Adds grounding and attribution dimensions.
`pyrit/datasets/score/assert/judge_presets/communication.yaml`	Adds communication-quality judging.
`pyrit/datasets/score/assert/judge_presets/alignment.yaml`	Adds honesty and verification dimensions.
`pyrit/datasets/score/assert/dimension_system_prompt.yaml`	Defines the custom-dimension judge prompt.
`pyrit/backend/routes/scenarios.py`	Adds the ASSERT metrics endpoint.
`pyrit/backend/README.md`	Documents endpoint settings and responses.
`pyrit/analytics/rate_statistics.py`	Computes Wilson intervals and macro averages.
`pyrit/analytics/__init__.py`	Exports ASSERT metrics and rate helpers.
`frontend/src/types/index.ts`	Adds typed metrics API contracts.
`frontend/src/services/api.ts`	Adds the metrics request method.
`frontend/src/hooks/useAssertPolicyMetrics.ts`	Loads metrics with cancellation and retry.
`frontend/src/hooks/useAssertPolicyMetrics.test.tsx`	Tests loading, retry, and stale-request handling.
`frontend/src/components/Scenarios/ScenarioRunPage.tsx`	Shows metrics for completed ASSERT runs.
`frontend/src/components/Scenarios/ScenarioRunPage.test.tsx`	Tests conditional panel visibility.
`frontend/src/components/Scenarios/assertPolicyRun.ts`	Identifies ASSERT scenario runs.
`frontend/src/components/Scenarios/AssertPolicyMetricsPanel.test.tsx`	Tests metric presentation and breakdowns.
`frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts`	Styles metric tables and support indicators.
`doc/scanner/foundry.py`	Explains metrics, judge presets, and traces.
`doc/myst.yml`	Adds trace-scoring documentation navigation.
`doc/code/scoring/1_true_false_scorers.py`	Documents custom dimensions and repeated judgments.
`doc/code/scoring/1_true_false_scorers.ipynb`	Updates matching scorer notebook documentation.
`doc/code/scoring/0_scoring.py`	Links to execution-trace scoring guidance.
`doc/code/scoring/0_scoring.ipynb`	Adds the matching notebook link.
`doc/code/framework.md`	Clarifies trace and metrics responsibilities.
`doc/code/analytics/0_attack_results.md`	Documents rate helpers and ASSERT metrics.

_{Review effort: Balanced}

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Review of the ASSERT headline metrics commit (incremental over #4130). No correctness bugs found; 90/90 unit tests pass locally and the semantics match ASSERT's split and Wilson intervals. Comments below are polish/design.

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Re-review of fc44aeb (graded dimensions + not-applicable accounting). The n/a accounting resolves my earlier dimension-rate concern; the grade additions are well tested. Two new code comments below, plus one notebook note: doc/scanner/foundry.ipynb's compute_assert_policy_metrics_async demo cell now has empty outputs (the previous revision had the printed rates) — needs a rerun before ready-for-review. 29/29 metrics/route tests pass locally on this head.

Roman Lutz (romanlutz) · 2026-11-10T21:57:04Z

+        if values and len(numeric) == len(values):
+            median = statistics.median(numeric)
+        elif values:
+            median = sorted(values, key=scale.index)[(len(values) - 1) // 2]


Confirmed crash: scale comes from an arbitrary first row (line 542), so if another row's grade for the same dimension was stored with a different declared scale (e.g. the preset's scale was edited between attempts or before a rescore — _dimension_scores picks the newest marked score per dimension, so old and new attempts can mix), sorted(values, key=scale.index) raises a bare ValueError, failing the whole metrics computation (route returns 500). Repro: two _Rows with _Grade(value='bad', scale=('good','bad')) and _Grade(value='fine', scale=('fine','awful')) for one dimension. The all-numeric variant doesn't crash but silently miscounts (off-scale values enter the median yet not counts). Cheap fix: skip grades whose scale differs from the chosen one, or key distributions by (name, scale).

Roman Lutz (romanlutz) · 2026-11-10T21:57:05Z

+
+function RateCell({ rate, grades }: { rate: BinaryRate | null; grades?: AssertGradeDistribution }) {
+  const styles = useAssertPolicyMetricsPanelStyles()
+  if (!rate || rate.rate === null) return <Text className={styles.hint}>—</Text>


When every answer for a dimension is not-applicable, rate.rate is null, so this early return swallows not_applicable_count — the cell shows "—", indistinguishable from a dimension with no data at all. Showing something like "0 / 0 · N n/a" would keep the new n/a accounting visible exactly in the case where it matters most.

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Match results to their stored group by full seed identity, so groups sharing an objective but not their prompts resolve; keep the objective seed ID suffix only for exact duplicates and match it without parsing technique names. - Read judge dimensions independently: an undetermined objective no longer drops the policy split or other verdicts; count blocked/failed judges per dimension as no verdict, separate from not applicable. - Order tied score timestamps by score ID. - Return 409 only for unfinished runs and a sanitized 500 for unreadable saved results. - Rename BinaryRate to AssertBinaryRate and show 90% intervals in the run-page panel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ASSERT metrics now read the dimension, skip reason, not-applicable answer, grade and scale through AssertDimensionResult, and per-category judgments through AssertCategoryJudgment, instead of decoding the metadata keys locally. Missing judgment metadata stays unset, as on older scores, and malformed metadata now raises instead of being dropped. The scenario's judge wrapper in the layer below writes the dimension marker on judged and skipped scores, so this layer no longer edits it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…layer Bring the published layer-3 metrics (4848f91) onto the rebuilt layer 2 (06ccdf2). The merge applied without conflicts. The Foundry notebook keeps layer 2's outputs and is synced from foundry.py; the new ASSERT metrics cell has no output because it hasn't been run against this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:28

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:29

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread pyrit/analytics/assert_policy_metrics.py Resolved

Comment thread pyrit/analytics/assert_policy_metrics.py Resolved

Comment thread pyrit/backend/routes/scenarios.py Resolved

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from fc96974 to d531d93 Compare November 10, 2026 08:22

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from d531d93 to 1c727cc Compare November 10, 2026 09:11

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from 1c727cc to fc44aeb Compare November 10, 2026 10:14

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from fc44aeb to 5762dcd Compare November 10, 2026 22:33

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) and others added 6 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 4 commits November 10, 2026 16:20

DOC: Refresh ASSERT metrics notebook output for assert_package

265a2e6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-headline-metrics-plan branch from 5762dcd to b70cb51 Compare November 11, 2026 00:29

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Scenario-aware dataset compatibility (-8) #4180

Draft

Richard Lundeen (richlundeen) and others added 17 commits November 10, 2026 17:42

MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5c) i…

9e697fd

…nto trace evidence

Merge commit '9e697fddfef02c1c17fb1b851cb6285c5bbc704d' into richlund…

78f2477

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…

4848f91

…een-assert-headline-metrics-plan

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

FEAT: ASSERT headline metrics (-3) - #4131

FEAT: ASSERT headline metrics (-3)#4131
Richard Lundeen (richlundeen) wants to merge 34 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-headline-metrics-plan

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI behavior and scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading