DOC: In-process agents as custom targets (-7) by richlundeen · Pull Request #4144 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T10:03:52Z

Part -7 of a 10-layer ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)	this PR
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	Generated-category results
-10	Planned	Background dataset loading

Each PR targets main and includes the layers above it in this list, so its own diff on GitHub also contains those earlier layers. To review only this layer, use this layer's Files changed: 8f878a58e...5431c3842. Commit lists are not a reliable review boundary: -8 retains prototype commits in its ancestry, so later layers should be reviewed with source comparisons rather than commit lists.

Description

ASSERT tests in-process agents through a Python module:function target. This PR documents how to do the same in PyRIT with a short custom PromptTarget, with the agent's execution traces available to scorers. It adds no new target class, because the base class already provides trace correlation, capabilities, and registration.

doc/code/targets/6_custom_targets: New "Example: an in-process agent" section.
- A deterministic travel agent with an ASSERT-style (message, history) signature calls a traced search_flights tool. It needs no model or credentials.
- A target of about 15 lines passes the history to the agent and declares multi-turn plus editable history.
- With LocalTraceCapture, a two-turn MultiPromptSendingAttack scored by OtelToolCallScorer succeeds.
- Covers capture.instrument() for OpenInference frameworks and propagating trace context across threads. Also shows registering the target and its trace source from a custom initializer for pyrit_scan foundry.assert_policy --target travel_agent --trace-source travel_agent_traces.
- Notes that the scan also needs registered adversarial_chat and objective_scorer_chat targets, or targets named with --adversarial-target and --scoring-target.
doc/code/scoring/5_tool_call_scorer: The attack example uses a local custom target instead of the HTTPTarget + httpx.MockTransport workaround, with the same success, failure and undetermined outcomes. "Configure tracing" separates in-process agents from remote HTTP agents and points to LocalTraceCapture.

GUI changes

Own GUI changes: None. This layer changes only documentation and notebooks, with no GUI, frontend or backend source, and no runtime configuration or secrets. Any GUI you see in this PR's diff comes from the layers below.
Shared and non-ASSERT GUI impact: None.
Unchanged or deferred: Targets, scorers, trace capture and foundry.assert_policy behave the same; the examples use existing APIs. The GUI still has no trace configuration or trace viewer, and in-process targets are registered in Python (a custom initializer), not created from the GUI. ASSERT category-first result groups and nonblocking background dataset loading are Planned in separate follow-up PRs and are not part of this layer.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Metadata refresh unmounts the generator form and discards draft inputs, while synchronous template loading and YAML parsing during estimate requests block the event loop.

2 open issues

What changed in this PR

This PR expands PyRIT’s ASSERT integration with dataset-generation workflows, reusable evaluation presets, and trace-aware scoring support. The supplied changes extend beyond the documentation focus described in the title.

Changes:

Add bundled ASSERT behaviors, suites, judge presets, and public exports for tracing and scoring.
Connect dataset generation and ASSERT metrics to backend services and the GUI.
Add generation estimates, supporting tests, browser-check configuration, and documentation.

File	Description
THIRD_PARTY_NOTICES.txt	Updates ASSERT redistribution attribution.
tests/unit/executor/promptgen/test_target_objective_generator.py	Tests objective-generation estimates.
tests/unit/backend/test_prompt_generation_fixture.py	Checks offline generation fixtures.
tests/unit/assert_mocks.py	Adds checkpoint and generated-group helpers.
pyrit/score/observation/__init__.py	Exports local capture and execution-trace sources.
pyrit/score/__init__.py	Exports ASSERT judging and tracing APIs.
pyrit/scenario/scenarios/foundry/__init__.py	Removes the AssertTechnique export.
pyrit/scenario/core/scenario.py	Extracts scenario identity parameter selection.
pyrit/prompt_target/common/target_trace_config.py	Adds conversation identifiers to trace spans.
pyrit/prompt_target/common/prompt_target.py	Exposes target tracing configuration.
pyrit/models/score/__init__.py	Exports execution-trace data types.
pyrit/executor/promptgen/target_objective_generator.py	Adds call and token estimates.
pyrit/executor/promptgen/__init__.py	Exports objective-generation results.
pyrit/datasets/seed_datasets/generated/target_objective_provider.py	Extracts reusable result packaging.
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml	Defines trace-evidence judging instructions.
pyrit/datasets/score/assert/judge_presets/tool-use.yaml	Adds tool-use judging dimensions.
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml	Adds harm and privacy dimensions.
pyrit/datasets/score/assert/judge_presets/safety-core.yaml	References built-in safety dimensions.
pyrit/datasets/score/assert/judge_presets/robustness.yaml	Adds robustness judging dimensions.
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml	Adds policy-adherence judging.
pyrit/datasets/score/assert/judge_presets/operational.yaml	Adds workflow and escalation dimensions.
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml	Adds conversational-context dimensions.
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml	Adds constraint and completion dimensions.
pyrit/datasets/score/assert/judge_presets/grounding.yaml	Adds grounding and attribution dimensions.
pyrit/datasets/score/assert/judge_presets/communication.yaml	Adds communication-quality judging.
pyrit/datasets/score/assert/judge_presets/alignment.yaml	Adds honesty and verification dimensions.
pyrit/datasets/score/assert/dimension_system_prompt.yaml	Adds the custom-dimension judge template.
pyrit/datasets/assert_suites/travel_planner.yaml	Bundles a travel-planner evaluation suite.
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml	Bundles a quality-only travel benchmark.
pyrit/datasets/assert_suites/telecom_customer_service.yaml	Bundles a telecom service suite.
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml	Adds unsupported-conclusion behavior.
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml	Adds unit-conversion behavior.
pyrit/datasets/assert_behaviors/underused_context_failures.yaml	Adds context-underuse behavior.
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml	Adds unauthorized-action behavior.
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml	Adds tool-argument formatting behavior.
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml	Adds tool-output interpretation behavior.
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml	Adds tool-orchestration behavior.
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml	Adds tool-turn protocol behavior.
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml	Adds tool-error recovery behavior.
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml	Adds indentation-preference bias behavior.
pyrit/datasets/assert_behaviors/sycophancy.yaml	Adds sycophancy behavior.
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml	Adds self-harm safety behavior.
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml	Adds ambiguous completion-criteria behavior.
pyrit/datasets/assert_behaviors/stereotyping.yaml	Adds stereotyping behavior.
pyrit/datasets/assert_behaviors/stale_state_failures.yaml	Adds stale-state behavior.
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml	Adds cycling-brand preference behavior.
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml	Adds response-completeness behavior.
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml	Adds repeated-action loop behavior.
pyrit/datasets/assert_behaviors/prompt_injection.yaml	Adds prompt-injection resistance behavior.
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml	Adds procedure-adherence behavior.
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml	Adds premature-termination behavior.
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml	Adds retrieval-quality behavior.
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml	Adds output-consistency behavior.
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml	Adds scope-boundary behavior.
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml	Adds observation-neglect behavior.
pyrit/datasets/assert_behaviors/ios_vs_android.yaml	Adds mobile-platform preference behavior.
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml	Adds agent-handoff behavior.
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml	Adds intent-interpretation behavior.
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml	Adds insufficient-verification behavior.
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml	Adds inter-agent communication behavior.
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml	Adds tool-selection behavior.
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml	Adds incomplete-synthesis behavior.
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml	Adds identity-verification behavior.
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml	Adds medical-advice safety behavior.
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml	Adds grounding-attribution behavior.
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml	Adds goal-drift behavior.
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml	Adds action-planning behavior.
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml	Adds explicit-constraint behavior.
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml	Adds escalation-decision behavior.
pyrit/datasets/assert_behaviors/doxxing.yaml	Adds personal-information exposure behavior.
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml	Adds travel-recommendation preference behavior.
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml	Adds conversational-coherence behavior.
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml	Adds cross-agent constraint behavior.
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml	Adds instruction-conflict behavior.
pyrit/datasets/assert_behaviors/actionability_failures.yaml	Adds answer-actionability behavior.
pyrit/datasets/__init__.py	Exports ASSERT library lookup APIs.
pyrit/common/path.py	Defines ASSERT library directories.
pyrit/backend/services/service_lifecycle.py	Resets generation services during cleanup.
pyrit/backend/services/runtime_lifecycle.py	Includes generation work in runtime lifecycle checks.
pyrit/backend/services/dataset_service.py	Exposes dataset selection-key parsing.
pyrit/backend/routes/scenarios.py	Adds the ASSERT metrics endpoint.
pyrit/backend/main.py	Registers prompt-generation routes.
pyrit/analytics/__init__.py	Exports ASSERT metrics APIs.
frontend/src/utils/routeParams.ts	Adds dataset-preselected launch links.
frontend/src/utils/routeParams.test.ts	Tests dataset link encoding and parsing.
frontend/src/test-utils/replaceText.ts	Adds a paste-based input helper.
frontend/src/hooks/useRequestedDatasets.ts	Validates requested dataset selections.
frontend/src/hooks/usePromptGenerators.ts	Loads generator metadata by runtime generation.
frontend/src/hooks/usePromptGenerationSubmit.ts	Submits individual and batch generations.
frontend/src/hooks/usePromptGenerationPreflight.ts	Debounces generation validation and estimates.
frontend/src/hooks/usePromptGeneration.ts	Loads and polls generation records.
frontend/src/hooks/useAssertPolicyMetrics.ts	Loads completed-run ASSERT metrics.
frontend/src/hooks/useAssertPolicyMetrics.test.tsx	Tests metrics loading, retries, and cancellation.
frontend/src/hooks/useAssertLibrary.ts	Loads bundled behaviors and suites.
frontend/src/components/Scenarios/ScenarioRunPage.tsx	Displays completed ASSERT run metrics.
frontend/src/components/Scenarios/ScenarioRunPage.test.tsx	Tests conditional metrics display.
frontend/src/components/Scenarios/scenarioLaunchForm.ts	Seeds dataset choices from launch links.
frontend/src/components/Scenarios/assertPolicyRun.ts	Identifies ASSERT policy runs.
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts	Styles metrics presentation.
frontend/src/components/PromptGeneration/TargetObjectiveSummary.tsx	Displays generated objective counts.
frontend/src/components/PromptGeneration/RegenerateDatasetLink.tsx	Links generated datasets to prefilled forms.
frontend/src/components/PromptGeneration/RegenerateDatasetLink.test.tsx	Tests regeneration-link availability.
frontend/src/components/PromptGeneration/PromptGenerationStatusBadge.tsx	Displays generation status badges.
frontend/src/components/PromptGeneration/promptGenerationRoutes.ts	Defines generation routes and query parameters.
frontend/src/components/PromptGeneration/promptGenerationRoutes.test.ts	Tests generation route helpers.
frontend/src/components/PromptGeneration/generatorUi.ts	Defines generator-view contracts.
frontend/src/components/PromptGeneration/generatorRegistry.tsx	Selects generator-specific views.
frontend/src/components/PromptGeneration/GenerationTargetFields.test.tsx	Tests generation target selection.
frontend/src/components/PromptGeneration/GenerationSummaryFacts.tsx	Renders reusable result summaries.
frontend/src/components/PromptGeneration/generationFormHelpers.test.ts	Tests form parsing and validation helpers.
frontend/src/components/PromptGeneration/GenerateDatasetLinks.tsx	Links scenarios to matching generators.
frontend/src/components/PromptGeneration/AssertGenerationSummary.tsx	Displays ASSERT coverage and category counts.
frontend/src/components/Datasets/DatasetSelection.tsx	Adds dataset regeneration navigation.
frontend/src/components/Datasets/DatasetSelection.test.tsx	Tests regeneration navigation.
frontend/src/components/Datasets/DatasetCatalog.tsx	Adds the generation entry point.
frontend/src/components/Datasets/DatasetCatalog.test.tsx	Tests catalog-to-generation navigation.
frontend/src/App.tsx	Registers generation pages.
frontend/README.md	Documents isolated browser checks.
frontend/playwright.prompt-generation.config.ts	Configures offline generation browser checks.
frontend/playwright.config.ts	Excludes the dedicated generation spec.
frontend/package.json	Adds the generation browser-check command.
doc/myst.yml	Adds trace-evidence scoring navigation.
doc/gui/0_gui.md	Documents GUI dataset generation.
doc/code/scoring/1_true_false_scorers.py	Documents custom dimensions and repeated judgments.
doc/code/scoring/1_true_false_scorers.ipynb	Mirrors updated ASSERT scoring guidance.
doc/code/scoring/0_scoring.py	Links execution-trace scoring guidance.
doc/code/scoring/0_scoring.ipynb	Mirrors the trace-scoring documentation link.
doc/code/executor/7_promptgen.py	Demonstrates checkpoint review and continuation.
doc/code/analytics/0_attack_results.md	Explains ASSERT metric semantics.
.github/workflows/frontend_tests.yml	Adds generation browser checks to CI.

_{Review effort: Balanced}

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- Match results to their stored group by full seed identity, so groups sharing an objective but not their prompts resolve; keep the objective seed ID suffix only for exact duplicates and match it without parsing technique names. - Read judge dimensions independently: an undetermined objective no longer drops the policy split or other verdicts; count blocked/failed judges per dimension as no verdict, separate from not applicable. - Order tied score timestamps by score ID. - Return 409 only for unfinished runs and a sanitized 500 for unreadable saved results. - Rename BinaryRate to AssertBinaryRate and show 90% intervals in the run-page panel. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

ASSERT metrics now read the dimension, skip reason, not-applicable answer, grade and scale through AssertDimensionResult, and per-category judgments through AssertCategoryJudgment, instead of decoding the metadata keys locally. Missing judgment metadata stays unset, as on older scores, and malformed metadata now raises instead of being dropped. The scenario's judge wrapper in the layer below writes the dimension marker on judged and skipped scores, so this layer no longer edits it. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoints carry a format version and the identifier of the target that produced each role's work. Resuming with a different target is rejected before any model call, build_result attributes work to the recorded producers, and stored seed-group assignments must match the planned batches. Estimates count the dimensions and target system prompt sent with every seed-group call. Objective datasets record the request that generated them, the objective estimate reads its prompt once, and Azure Blob listings percent-encode names so they round-trip. AssertGenerationStage is a StrEnum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…(-4) Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the reference-counted container client so it can't close a client other operations are using. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

main's dataset editing and deletion read selection keys through the shared parse_dataset_selection_key, which replaced DatasetService._parse_selection_key in this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

main now requires explicitly configured scenario target roles. The offline prompt-generation fixture registers offline adversarial_chat and objective_scorer_chat targets, and the journey chooses its adversarial target explicitly before launching foundry.assert_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… agent docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in main c3cff18 through -5 0ebcd76. The frontend CI build step, npm scripts, and Playwright ignores keep both the prompt-generation and main's scan-launch journeys; ScenarioDetail keeps dataset link preselection with main's target imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

… agent docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…layer Bring the published layer-3 metrics (4848f91) onto the rebuilt layer 2 (06ccdf2). The merge applied without conflicts. The Foundry notebook keeps layer 2's outputs and is synced from foundry.py; the new ASSERT metrics cell has no output because it hasn't been run against this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replaces the per-generator links on scanner launch pages with one Browse datasets / Generate datasets group in the Datasets header. Generate waits for the current runtime's generators, then opens the scenario's only follow-up generator (ASSERT for foundry.assert_policy), otherwise the generator chooser. Linked ?dataset= names stay explicit, exact selections for the server to validate instead of being checked against a second catalog fetch; only identical names are deduplicated. A blank name, or any name on a scanner that takes no datasets, blocks the estimate and launch with an error and a link that removes them. Regenerate links read the stored request by the dataset's exact selection key. foundry.assert_policy marks its datasets as required and asks for one at launch, using the rebuilt -3's ASSERT scenario check. Scenario.get_dataset_options reports inline seeds as fixed sources, so inline-only scenarios offer no dataset picker while their limits and filters stay editable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The rebuilt -3 launch-request tests build foundry.assert_policy requests without a dataset, which this layer now rejects with 'Select an ASSERT dataset.' They select one so they keep checking the default-scoring rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 5 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 10:03

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 10:04

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread frontend/src/hooks/usePromptGenerators.ts Resolved

Comment thread pyrit/executor/promptgen/target_objective_generator.py Resolved

Richard Lundeen (richlundeen) force-pushed the richlundeen-callable-agent-target-plan branch from 7718e62 to a9c24a5 Compare November 10, 2026 19:42

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-callable-agent-target-plan branch from a9c24a5 to e0e31d9 Compare November 10, 2026 23:27

Richard Lundeen (richlundeen) and others added 12 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: Refresh ASSERT metrics notebook output for assert_package

265a2e6

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 29 commits November 10, 2026 17:58

MAINT: Merge refreshed GUI generation layer with main into in-process…

ffa29f3

… agent docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: Note the scenario target roles an in-process agent scan needs

fd4e83a

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…

4848f91

…een-assert-headline-metrics-plan

Merge refreshed ASSERT metrics (-3) with main into staged generation …

da0f38c

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge staged generation (-4) with main into generation jobs (-5)

0ebcd76

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed GUI generation layer with main into in-process…

e1bf06e

… agent docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt ASSERT metrics (-3) into staged generation (-4)

1c0add0

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt staged generation (-4) into generation jobs (-5)

22c7a75

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt generation jobs (-5) into the GUI generation layer (-6)

a296da4

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge rebuilt GUI generation layer (-6) into in-process agent …

5431c38

…docs Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

DOC: In-process agents as custom targets (-7) - #4144

DOC: In-process agents as custom targets (-7)#4144
Richard Lundeen (richlundeen) wants to merge 58 commits into
microsoft:mainfrom
richlundeen:richlundeen-callable-agent-target-plan

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI changes

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading