FEAT: Staged ASSERT generation and the ASSERT policy library (-4) by richlundeen · Pull Request #4133 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:30:28Z

Part -4 of a 10-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)	this PR
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	ASSERT results grouped by generated category
-10	Planned	Background dataset loading (all scanners)

Each PR targets main and includes the earlier layers, so its Files changed tab also shows them. Review this PR's own changes in the -3 to -4 source comparison. Use source comparisons rather than commit lists: -8 keeps prototype commits in its ancestry.

Description

ASSERT generation can now pause after the taxonomy for a person to review and edit it, resume after an interruption without repeating model calls, and estimate its cost before calling a model. It can start from ASSERT's bundled behavior library.

AssertGenerator.advance_async runs the remaining stages of an AssertGenerationCheckpoint and can stop after any stage. with_reviewed_policy applies an edited taxonomy and dimension levels, and each seed group's provenance records which stages were edited. on_checkpoint reports progress after each stage and seed-group batch. execute_async keeps its signature and runs every stage through the same staged path; advance_async returns progress, so it runs setup and teardown but not the result-level strategy events.
Checkpoints carry a format version and record each role's target identifier. Resuming with a target whose identifier hash differs is rejected before any model call, build_result attributes the work to the recorded targets, and stored seed groups must match the planned coverage batches.
AssertGenerator.estimate and TargetObjectiveGenerator.estimate return a provider-free PromptGenerationEstimate: model calls by stage and an approximate token range. Seed-group calls range from one per five groups to one per group until the taxonomy exists, then are exact; token ranges include the dimensions and target system prompt sent with each call.
ASSERT v0.3.0's behavior (52) and scenario (3) library presets are vendored into pyrit/datasets/assert_behaviors and pyrit/datasets/assert_suites, with loaders in pyrit.datasets, attribution headers, provenance entries, and third-party notices.
StorageIO.list_files_async for disk and Azure Blob storage; listed blob URLs percent-encode names so they round-trip. Concurrent Azure Blob operations, including write_stream_async, share one reference-counted container client, so they no longer close each other's client.
TargetObjectiveProvider.package_result stores the request that generated an objective dataset, so it can be regenerated, and the TargetObjectiveGeneratorResult export, used by the backend layer.
The prompt generator notebook shows reviewing the policy before seed groups are generated.

draft = await generator.advance_async(
    checkpoint=AssertGenerationCheckpoint(config=config), stop_after=AssertGenerationStage.DIMENSIONS
)
reviewed = draft.with_reviewed_policy(taxonomy=edited_taxonomy, dimensions=draft.dimensions)
result = generator.build_result(checkpoint=await generator.advance_async(checkpoint=reviewed))

GUI and scope

Own GUI changes: none. This layer changes no frontend files and adds no backend routes; it has no screenshot. The GUI that uses this API arrives in #4135, through the jobs in #4134.

Shared / non-ASSERT impact (SDK only):

TargetObjectiveGenerator.estimate and the request stored by TargetObjectiveProvider.package_result apply to ordinary objective datasets, not only ASSERT.
StorageIO.list_files_async is a new method on the shared disk and Azure Blob storage classes. The Azure client reference counting changes how every concurrent Blob operation shares its container client; single operations behave as before.

Unchanged / deferred:

AssertGenerator.execute_async and the existing generation results keep their signatures.
No dataset versioning, archiving or retention: regenerating writes a new dataset under a new name.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved issues can misattribute checkpoint generation, understate token cost estimates, and cause listed blob URLs to read or delete the wrong blob.

3 open issues

What changed in this PR

This PR extends PyRIT’s ASSERT integration with staged, reviewable generation and bundled policy presets. It also adds supporting scoring, trace-evidence, and analytics interfaces.

Changes:

Supports checkpoint-based generation, policy review, cost estimates, and packaging generated objectives.
Bundles ASSERT behavior, suite, and judge presets with attribution.
Extends execution-trace scoring and exposes ASSERT metrics through the backend and run page.

File	Description
THIRD_PARTY_NOTICES.txt	Updates ASSERT redistribution and adaptation notices.
tests/unit/executor/promptgen/test_target_objective_generator.py	Tests generation estimates and input validation.
tests/unit/datasets/test_target_objective_provider.py	Tests generated-objective packaging.
tests/unit/common/test_lazy_package_imports.py	Updates analytics export expectations.
tests/unit/assert_mocks.py	Adds checkpoint and generated-group fixtures.
tests/unit/analytics/test_rate_statistics.py	Tests Wilson intervals and macro rates.
pyrit/score/true_false/true_false_fallback_scorer.py	Exposes configurable fallback triggers.
pyrit/score/scorer.py	Identifies missing-evidence results.
pyrit/score/observation/otel_span_exporter.py	Captures span events and trace-specific failures.
pyrit/score/observation/message_trace_resolver.py	Resolves conversation request trace links.
pyrit/score/observation/execution.py	Shares evidence acquisition within scoring operations.
pyrit/score/observation/__init__.py	Exports execution-trace observation APIs.
pyrit/score/float_scale/float_scale_fallback_scorer.py	Exposes configurable fallback triggers.
pyrit/score/__init__.py	Exports ASSERT judging and trace APIs.
pyrit/scenario/core/scenario.py	Extracts scenario identity parameter handling.
pyrit/prompt_target/common/target_trace_config.py	Adds conversation identifiers to target spans.
pyrit/prompt_target/common/prompt_target.py	Exposes target tracing configuration.
pyrit/models/score/__init__.py	Exports execution-trace data models.
pyrit/models/prompt_generation_estimate.py	Defines generation-call and token estimates.
pyrit/models/analytics.py	Adds binary and macro rate models.
pyrit/executor/promptgen/target_objective_generator.py	Adds provider-free generation estimates.
pyrit/executor/promptgen/__init__.py	Exports generated-objective results.
pyrit/datasets/seed_datasets/generated/target_objective_provider.py	Adds reusable result packaging.
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml	Defines trace-evidence judging instructions.
pyrit/datasets/score/assert/judge_presets/tool-use.yaml	Adds tool-use judging dimensions.
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml	Adds harm and privacy judging dimensions.
pyrit/datasets/score/assert/judge_presets/safety-core.yaml	References built-in safety dimensions.
pyrit/datasets/score/assert/judge_presets/robustness.yaml	Adds robustness judging dimensions.
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml	Adds policy-adherence judging criteria.
pyrit/datasets/score/assert/judge_presets/operational.yaml	Adds workflow and escalation criteria.
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml	Adds conversation continuity criteria.
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml	Adds constraint and completion criteria.
pyrit/datasets/score/assert/judge_presets/grounding.yaml	Adds grounding and attribution criteria.
pyrit/datasets/score/assert/judge_presets/communication.yaml	Adds communication-quality criteria.
pyrit/datasets/score/assert/judge_presets/alignment.yaml	Adds honesty and verification criteria.
pyrit/datasets/score/assert/dimension_system_prompt.yaml	Adds custom-dimension judging template.
pyrit/datasets/assert_suites/travel_planner.yaml	Bundles travel-planning evaluation context.
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml	Bundles quality-focused travel benchmark context.
pyrit/datasets/assert_suites/telecom_customer_service.yaml	Bundles telecom service evaluation context.
pyrit/datasets/assert_library.py	Loads and validates bundled presets.
pyrit/datasets/assert_behaviors/violent_content.yaml	Bundles violent-content policy.
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml	Bundles unsupported-conclusion behavior.
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml	Bundles unit-conversion behavior.
pyrit/datasets/assert_behaviors/underused_context_failures.yaml	Bundles retrieved-context usage behavior.
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml	Bundles action-authorization behavior.
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml	Bundles tool-argument behavior.
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml	Bundles tool-output interpretation behavior.
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml	Bundles tool-orchestration behavior.
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml	Bundles tool-turn protocol behavior.
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml	Bundles tool-error recovery behavior.
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml	Bundles indentation preference-bias behavior.
pyrit/datasets/assert_behaviors/sycophancy.yaml	Bundles sycophancy behavior.
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml	Bundles self-harm safety policy.
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml	Bundles task-completion criteria behavior.
pyrit/datasets/assert_behaviors/stereotyping.yaml	Bundles stereotyping behavior.
pyrit/datasets/assert_behaviors/stale_state_failures.yaml	Bundles stale-state behavior.
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml	Bundles bicycle-component preference-bias behavior.
pyrit/datasets/assert_behaviors/sexual_content.yaml	Bundles sexual-content safety policy.
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml	Bundles response-completeness behavior.
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml	Bundles repeated-action behavior.
pyrit/datasets/assert_behaviors/prompt_injection.yaml	Bundles prompt-injection resistance behavior.
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml	Bundles procedure-adherence behavior.
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml	Bundles premature-termination behavior.
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml	Bundles retrieval-quality behavior.
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml	Bundles output-consistency behavior.
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml	Bundles scope-compliance behavior.
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml	Bundles observation-handling behavior.
pyrit/datasets/assert_behaviors/malicious_cyber_activity.yaml	Bundles cyber-safety policy.
pyrit/datasets/assert_behaviors/ios_vs_android.yaml	Bundles mobile-platform preference-bias behavior.
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml	Bundles agent-handoff behavior.
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml	Bundles intent-interpretation behavior.
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml	Bundles verification behavior.
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml	Bundles agent-team communication behavior.
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml	Bundles tool-selection behavior.
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml	Bundles answer-synthesis behavior.
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml	Bundles identity-verification behavior.
pyrit/datasets/assert_behaviors/hate_speech_harassment.yaml	Bundles hate and harassment safety policy.
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml	Bundles medical-advice safety behavior.
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml	Bundles grounding and attribution behavior.
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml	Bundles goal-drift behavior.
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml	Bundles planning-quality behavior.
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml	Bundles explicit-constraint behavior.
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml	Bundles escalation-decision behavior.
pyrit/datasets/assert_behaviors/doxxing.yaml	Bundles personal-information protection behavior.
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml	Bundles resort preference-bias behavior.
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml	Bundles conversation-coherence behavior.
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml	Bundles cross-agent constraint behavior.
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml	Bundles instruction-conflict behavior.
pyrit/datasets/assert_behaviors/actionability_failures.yaml	Bundles answer-actionability behavior.
pyrit/datasets/__init__.py	Exports behavior and suite loaders.
pyrit/common/path.py	Defines bundled library paths.
pyrit/backend/routes/scenarios.py	Adds the ASSERT metrics endpoint.
pyrit/backend/README.md	Documents ASSERT metrics responses.
pyrit/analytics/rate_statistics.py	Implements Wilson intervals and macro averages.
pyrit/analytics/__init__.py	Exports ASSERT metrics and rate helpers.
frontend/src/types/index.ts	Defines ASSERT metrics response types.
frontend/src/services/api.ts	Adds the ASSERT metrics client call.
frontend/src/hooks/useAssertPolicyMetrics.ts	Loads metrics with cancellation and retry.
frontend/src/hooks/useAssertPolicyMetrics.test.tsx	Tests metrics loading and runtime changes.
frontend/src/components/Scenarios/ScenarioRunPage.tsx	Displays metrics for completed ASSERT runs.
frontend/src/components/Scenarios/ScenarioRunPage.test.tsx	Tests conditional metrics display.
frontend/src/components/Scenarios/assertPolicyRun.ts	Identifies ASSERT policy runs.
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts	Styles metrics presentation.
doc/myst.yml	Adds trace-scoring documentation navigation.
doc/code/scoring/1_true_false_scorers.py	Documents custom dimensions and repeated judgments.
doc/code/scoring/1_true_false_scorers.ipynb	Mirrors updated ASSERT scoring guidance.
doc/code/scoring/0_scoring.py	Links execution-trace scoring guidance.
doc/code/scoring/0_scoring.ipynb	Mirrors the trace-scoring documentation link.
doc/code/framework.md	Documents trace and analytics responsibilities.
doc/code/executor/7_promptgen.py	Demonstrates staged policy review and generation.
doc/code/executor/7_promptgen.ipynb	Adds the staged-generation notebook example.
doc/code/analytics/0_attack_results.md	Documents rates and ASSERT policy metrics.

_{Review effort: Balanced}

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Match results to their stored group by full seed identity, so groups sharing an objective but not their prompts resolve; keep the objective seed ID suffix only for exact duplicates and match it without parsing technique names. - Read judge dimensions independently: an undetermined objective no longer drops the policy split or other verdicts; count blocked/failed judges per dimension as no verdict, separate from not applicable. - Order tied score timestamps by score ID. - Return 409 only for unfinished runs and a sanitized 500 for unreadable saved results. - Rename BinaryRate to AssertBinaryRate and show 90% intervals in the run-page panel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ASSERT metrics now read the dimension, skip reason, not-applicable answer, grade and scale through AssertDimensionResult, and per-category judgments through AssertCategoryJudgment, instead of decoding the metadata keys locally. Missing judgment metadata stays unset, as on older scores, and malformed metadata now raises instead of being dropped. The scenario's judge wrapper in the layer below writes the dimension marker on judged and skipped scores, so this layer no longer edits it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoints carry a format version and the identifier of the target that produced each role's work. Resuming with a different target is rejected before any model call, build_result attributes work to the recorded producers, and stored seed-group assignments must match the planned batches. Estimates count the dimensions and target system prompt sent with every seed-group call. Objective datasets record the request that generated them, the objective estimate reads its prompt once, and Azure Blob listings percent-encode names so they round-trip. AssertGenerationStage is a StrEnum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

…(-4) Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the reference-counted container client so it can't close a client other operations are using. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…layer Bring the published layer-3 metrics (4848f91) onto the rebuilt layer 2 (06ccdf2). The merge applied without conflicts. The Foundry notebook keeps layer 2's outputs and is synced from foundry.py; the new ASSERT metrics cell has no output because it hasn't been run against this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:30

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:31

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread pyrit/executor/promptgen/assert_generator.py Resolved

Comment thread pyrit/executor/promptgen/assert_generator.py Resolved

Comment thread pyrit/memory/storage/storage.py Resolved

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-sdk branch from 6146e1b to ac96336 Compare November 10, 2026 09:58

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-sdk branch from ac96336 to f760299 Compare November 10, 2026 18:31

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-sdk branch from f760299 to 3b9cb58 Compare November 10, 2026 23:18

Richard Lundeen (richlundeen) and others added 8 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 4 commits November 10, 2026 16:20

DOC: Refresh ASSERT metrics notebook output for assert_package

265a2e6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-sdk branch from 3b9cb58 to d199cba Compare November 11, 2026 01:04

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Scenario-aware dataset compatibility (-8) #4180

Draft

Richard Lundeen (richlundeen) and others added 20 commits November 10, 2026 17:42

MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5c) i…

9e697fd

…nto trace evidence

Merge commit '9e697fddfef02c1c17fb1b851cb6285c5bbc704d' into richlund…

78f2477

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…

4848f91

…een-assert-headline-metrics-plan

Merge refreshed ASSERT metrics (-3) with main into staged generation …

da0f38c

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt ASSERT metrics (-3) into staged generation (-4)

1c0add0

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

FEAT: Staged ASSERT generation and the ASSERT policy library (-4) - #4133

FEAT: Staged ASSERT generation and the ASSERT policy library (-4)#4133
Richard Lundeen (richlundeen) wants to merge 39 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-generation-sdk

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI and scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading