FEAT: Generate datasets from a policy in the GUI (-6) by richlundeen · Pull Request #4135 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:30:45Z

Part -6 of a 10-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI	this PR
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	ASSERT results grouped by generated category
-10	Planned	Background dataset loading (all scanners)

Each PR targets main and includes the earlier layers, so its Files changed tab also shows them. Review this PR's own changes in the -5 to -6 source comparison. Use source comparisons rather than commit lists: -8 keeps prototype commits in its ancestry.

Description

In CoPyRIT, you can write a policy such as "Explain published return policies. Do not disclose another customer's private order details." or start from a bundled ASSERT behavior. You can then generate and optionally review an ASSERT dataset, and open foundry.assert_policy with it selected. The same pages also generate ordinary objective datasets, and every scanner's Datasets section links to browsing and generating datasets.

Datasets > Generate dataset:
- The generations list groups suite members, with a cancel per member and a link to run the scenario on all of them once they finish.
- The job page shows stage progress and offers cancel, retry, delete, and Edit & regenerate.
- Each generator plugs in its own review step and result summary.
ASSERT form: bundled behavior and suite templates, coverage dimensions, and advanced sampling. Every target role, including the research target, starts with the default adversarial target, also when the target registry loads after the form; choices from the user or an earlier generation are kept. The research target is limited to web-search targets when web search is on, and a default that lacks web search is flagged rather than replaced. Pause-for-review is optional. A debounced preflight shows validation errors, name availability, and an estimate card; a failed check can be run again. Suite mode creates one dataset per behavior in a single batch.
Review editor: edit the taxonomy's categories (definition, examples, permissible, add or remove), terms, and dimension levels, then continue generation.
Objective datasets: a form for the target-objective generator. A generator the GUI doesn't know shows a frontend/backend version mismatch message.
Catalog and dataset pages: the catalog gains Generate dataset and an Origins facet, so you can list only Generated datasets. Dataset pages for generated datasets gain Edit policy & regenerate.
Scanner launch pages:
- The Datasets section header has Browse datasets and Generate datasets. Generate opens the scenario's own generator when exactly one generator follows it up (ASSERT for foundry.assert_policy), otherwise the generator chooser.
- A scenario without default datasets, such as foundry.assert_policy, says "No dataset selected" and lets you choose its first dataset. foundry.assert_policy marks its datasets as required and asks for one when you launch without it.
- Repeatable ?dataset= parameters preselect datasets exactly as named; completed ASSERT generations link to foundry.assert_policy this way. Blank names, or names on a scanner that takes no datasets, block the launch instead of being dropped.
An isolated Playwright journey runs over an offline generation target and is wired into CI. The GUI and scanner docs describe the flow.

Paste a policy, here the example above, and start an ASSERT generation:

Review the generated policy before seed groups are written:

The completed generation, with coverage, per-category counts, and the link to run foundry.assert_policy:

foundry.assert_policy's Datasets section before this PR (-5 22c7a75e0): it has no default datasets and offers no way to choose the first one.

After this PR (-6 8f878a58e): opened from the completed generation with its dataset selected; opened directly, with the required marker and picker; and launching without a dataset.

GUI behavior and scope

Own GUI changes (ASSERT-specific):

The ASSERT single-policy and suite forms, the taxonomy review step, and the ASSERT result summary.
On foundry.assert_policy's launch page, the Datasets heading and the first dataset picker are marked required. Launching without a dataset shows "Select an ASSERT dataset." There's no requirement for a second dataset. The Objective Target marker already existed.
Generate datasets on that page opens the ASSERT form directly.

Shared / non-ASSERT impact. These are this PR's own changes, and users of other scanners and ordinary datasets see them:

Routes: /datasets/generate, /datasets/generate/new and /datasets/generate/:generationId, with a generator chooser but no new sidebar section.
Objective generation: the ordinary target-objective form, the generations list and the job pages, shared by both generators.
Catalog: Generate dataset and an Origins facet, persisted as repeated origin URL parameters and combined with existing filters.
Dataset pages: Edit policy & regenerate appears for any named dataset a known generator made, including objective datasets. A failed lookup other than "not found" shows a retryable error instead of hiding the action.
Dataset actions on every scanner launch page that has a Datasets section:
- Browse datasets and Generate datasets sit in the section header, replacing the table's own Browse link there; the preset editor keeps its link.
- Generate opens the generator chooser unless exactly one generator follows up the scenario.
- It stays temporarily unavailable until the runtime is ready and the current runtime's generator list loads, so it never routes to a generator from the previous runtime. If the list fails to load, it opens the chooser, which shows the error.
First dataset: any editable scenario with no default datasets can choose its first one, while fixed scenarios keep their restrictions.
Inline-only scenarios (SDK/backend change in this PR): Scenario.get_dataset_options() now reports inline seeds / seed_groups as fixed sources, because named sources can't replace them; their limits and filters keep the declared capabilities. This changes the catalog metadata and CLI output for every inline-default scenario; among registered scenarios that is only technique.execute, which doesn't take dataset input. The launch page therefore offers no dataset picker for them.
Linked datasets (?dataset=), on any scanner that takes named datasets:
- Linked names replace saved or preset sources while keeping their limits and filters. A different link starts a new form.
- Names round-trip exactly, including leading or trailing whitespace, and only identical names are deduplicated.
- Every nonblank linked name stays selected, including names the catalog doesn't list. The table marks them as not stored, the estimate reports itself unavailable, and launch rejects any that are neither stored nor available from a provider ("Datasets [...] are missing and have no provider. Import them first.").
- A blank name blocks estimates and launch with an error and a link that drops only the blank names. The form doesn't fall back to the scenario's default datasets.
- On a scanner that takes no datasets, linked names block the launch with an error that names them, plus a link to open the same page without them.
Edit policy & regenerate reads the stored request by the dataset's exact selection key, so whitespace in a name never selects a different dataset.

Unchanged / deferred:

Existing dataset row and import controls, the Load datasets button, the loading-policy selector, Reset datasets, and dataset limits and filters are unchanged. Background loading that replaces Load is -10.
Scorer, judge, trace and operation controls, and numeric limits, are unchanged.
Include an adversarial prompt (tester instructions) starts off here; FEAT: Scenario-aware dataset compatibility (-8) #4180 turns it on by default in the GUI. FEAT: Scenario-aware dataset compatibility (-8) #4180 also adds the scenario-compatible ASSERT dataset listing.
Generation inputs are local form state. There are no saved drafts, and regenerating creates a new dataset under a new name rather than editing or versioning the old one.

Known limitations:

One backend owns generation records (see FEAT: Prompt generation jobs in the backend (-5) #4134). Several backends sharing one results storage aren't supported.
Concurrent review edits are last-write-wins. Review saves carry no version check, so if the same review is open in two tabs, the last successful save replaces the other tab's edits without a warning. Earlier revisions of this stack returned a conflict and reloaded; this PR doesn't restore that check.
A failed generator-list refresh can lose unsaved form input. The form is replaced by an error page, and Retry remounts it from its original values. This isn't fixed here.
Dataset uploads and generations reserve names separately (see FEAT: Prompt generation jobs in the backend (-5) #4134), so an upload can take a running generation's name and make its save fail.

Complete own-layer generation UI inventory

List/detail and actions (both generators). The list shows dataset, generator, status, progress, creation time, and actions; batch members are grouped in request order. States are Queued, Running, Review needed, Saving, Completed, Failed, Cancelled, and Interrupted. Active list and detail pages poll every two seconds; a failed refresh retries once, then shows an error or a stale-record warning. Polling runs only on these generation pages. Records are scoped to the runtime and job ID and wait for runtime readiness. Detail shows stages, timestamps, creator and ID, persisted errors, and generator-specific results. Cancel has explicit errors and a detail-page confirmation. Failed, cancelled and interrupted jobs offer Retry from the last checkpoint. Edit & regenerate pre-fills a new form with a distinct _vN name. Inactive records can be deleted after a confirmation that says saved datasets are kept. Completed jobs offer Browse dataset and the declared Run scenario link; a completed batch can send all member datasets to one launch form.

Shared forms and targets. Changing the generator, source or template URL starts a fresh form. A request opened from an earlier generation keeps its targets; a request read from a dataset uses the configured defaults. Target roles, including ASSERT research, follow the configured default adversarial target until the user chooses; explicit choices survive late registry or default updates. Required fields, constraints and target-capability errors appear before generation. Valid, settled inputs trigger a preflight after 500 ms for configuration and name checks and an estimate, without model calls. Estimates show call and token ranges, stages and exclusions; Check again retries a failed check, and a transport failure still lets you try Generate. While the generator list reloads after a runtime change, the form stays mounted but can't submit. A generator without a frontend form shows an explicit version-mismatch message.

ASSERT form. Single-policy and suite modes provide bundled behavior and suite templates, policy name and text, and application context. Output names follow the behavior or suite until edited; edited suite member names persist. Controls include seed-group and category counts, user-prompt and adversarial-prompt inclusion, described or generated coverage dimensions or explicit levels, with add, remove and edit validation. Advanced controls include sampling method, random seed, stratification, factorial replication, replacement, web search, and default, custom or target-owned instructions. Defaults: 25 groups, 25 suggested categories, three generated levels, pairwise sampling, seed 0, web search off, user prompts on, adversarial prompts off. Generation and research target pickers and an optional pause after coverage-dimension resolution are available.

ASSERT review. Shows the research context and edits the behavior, categories, permissible flags, examples, terms and coverage levels. Save persists edits; Continue saves pending edits before resuming. Reload, save and continue failures are explicit. Saving refreshes the estimate; a failed review estimate falls back to no estimate. Saves are last-write-wins (see Known limitations).

Target-objective form (non-ASSERT). Shares the naming, target, estimate and submission machinery, with instructions, an objective count starting at 20 (1 to 1,000), and comma-separated harm labels. Name and generation target are required. There's no review pause because that generator has no reviewable stage.

Screenshots: the -6 screenshots use synthetic offline fixture data on 8f878a58e. The -5 "before" screenshots use a separate synthetic fixture on 22c7a75e0, so their data and targets differ. Neither made provider or model calls. They show this PR's own UI together with earlier layers'; the scanner role, target and warning controls on launch pages aren't from this PR. In the full ASSERT form screenshots, Include an adversarial prompt was checked by hand; its default in this PR is off, as the template screenshot shows.

All screenshots

Catalog (shared): before (-5), then after with Generate dataset and the Origins facet.

Generator chooser (shared):

Objective dataset (non-ASSERT): form with its estimate, then the completed generation.

ASSERT form: with a coverage dimension, roles, review pause and estimate, at 1440 px and 390 px; Include an adversarial prompt checked by hand.

ASSERT template, advanced settings and suite: a bundled behavior fills the name and policy (adversarial prompt at its default, off); expanded advanced settings; suite mode from a bundled suite, one dataset per behavior.

The suite's dataset-name fields are clipped at this width; each holds a distinct telecom_customer_service_<behavior> name, such as telecom_customer_service_grounding_attribution_errors.

Generations list and regenerate (shared):

foundry.assert_policy launch at 390 px:

Ordinary scanner (shared): foundry.red_team_agent before (-5), with the table's Browse link and the existing Load/Reset/loading controls; after, with Browse datasets and Generate datasets in the header opening the generator chooser, and no required marker.

Linked datasets (shared): a blank ?dataset= blocks launch without falling back to defaults; technique.execute, which takes no datasets, opened with a linked dataset at 1440 px and 390 px; and technique.execute after removing the bad link. Its lack of a Datasets section comes from its metadata (no dataset input) and the capture's assertion, not from this top-of-page image alone.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Unresolved issues can leave draft submissions stuck after a failed start, prevent generation data from refreshing when runtime availability changes, and block regeneration for maximum-length dataset names.

3 open issues

What changed in this PR

This PR connects CoPyRIT’s dataset catalog to prompt generation, allowing users to create policy-based datasets, review generated policy details, and launch scenarios with those datasets selected.

Changes:

Adds generation navigation, reusable generator views, preflight checks, and dataset regeneration links.
Bundles ASSERT behavior and suite presets, with supporting generation, scoring, tracing, and analytics interfaces.
Adds frontend coverage, isolated browser-test configuration, CI wiring, and user documentation.

File	Description
THIRD_PARTY_NOTICES.txt	Updates ASSERT redistribution attribution.
tests/unit/executor/promptgen/test_target_objective_generator.py	Tests generation estimates and validation.
tests/unit/common/test_lazy_package_imports.py	Updates analytics export expectations.
tests/unit/backend/test_prompt_generation_fixture.py	Checks offline generator responses.
tests/unit/assert_mocks.py	Adds checkpoint and generated-group fixtures.
pyrit/score/true_false/true_false_fallback_scorer.py	Exposes configurable fallback triggers.
pyrit/score/scorer.py	Identifies missing-evidence results.
pyrit/score/observation/__init__.py	Exports execution-trace observation helpers.
pyrit/score/float_scale/float_scale_fallback_scorer.py	Exposes configurable fallback triggers.
pyrit/scenario/core/scenario.py	Extracts scenario identity parameter handling.
pyrit/prompt_target/common/target_trace_config.py	Attaches conversation identifiers to spans.
pyrit/prompt_target/common/prompt_target.py	Exposes target tracing configuration.
pyrit/models/score/__init__.py	Exports execution-trace models.
pyrit/models/analytics.py	Adds binary and macro rate models.
pyrit/executor/promptgen/target_objective_generator.py	Adds call and token estimates.
pyrit/executor/promptgen/__init__.py	Exports objective generation results.
pyrit/datasets/seed_datasets/generated/target_objective_provider.py	Extracts reusable result packaging.
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml	Defines trace-evidence judging instructions.
pyrit/datasets/score/assert/judge_presets/tool-use.yaml	Adds tool-use judging dimensions.
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml	Adds harm and privacy dimensions.
pyrit/datasets/score/assert/judge_presets/safety-core.yaml	References core safety dimensions.
pyrit/datasets/score/assert/judge_presets/robustness.yaml	Adds robustness judging dimensions.
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml	Adds policy-adherence judging.
pyrit/datasets/score/assert/judge_presets/operational.yaml	Adds workflow and escalation judging.
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml	Adds conversation continuity judging.
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml	Adds instruction-following judging.
pyrit/datasets/score/assert/judge_presets/grounding.yaml	Adds grounding and attribution judging.
pyrit/datasets/score/assert/judge_presets/communication.yaml	Adds communication-quality judging.
pyrit/datasets/score/assert/judge_presets/alignment.yaml	Adds honesty and verification judging.
pyrit/datasets/score/assert/dimension_system_prompt.yaml	Adds custom-dimension judge template.
pyrit/datasets/assert_suites/travel_planner.yaml	Bundles travel-planner suite.
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml	Bundles quality-only travel suite.
pyrit/datasets/assert_suites/telecom_customer_service.yaml	Bundles telecom support suite.
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml	Bundles unsupported-conclusion behavior.
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml	Bundles unit-conversion behavior.
pyrit/datasets/assert_behaviors/underused_context_failures.yaml	Bundles context-underuse behavior.
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml	Bundles unauthorized-action behavior.
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml	Bundles tool-argument behavior.
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml	Bundles tool-output interpretation behavior.
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml	Bundles tool-orchestration behavior.
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml	Bundles tool-turn protocol behavior.
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml	Bundles tool-error recovery behavior.
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml	Bundles indentation preference-bias behavior.
pyrit/datasets/assert_behaviors/sycophancy.yaml	Bundles sycophancy behavior.
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml	Bundles self-harm safety behavior.
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml	Bundles success-criteria ambiguity behavior.
pyrit/datasets/assert_behaviors/stereotyping.yaml	Bundles stereotyping behavior.
pyrit/datasets/assert_behaviors/stale_state_failures.yaml	Bundles stale-state behavior.
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml	Bundles cycling-brand preference behavior.
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml	Bundles response-completeness behavior.
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml	Bundles repeated-action behavior.
pyrit/datasets/assert_behaviors/prompt_injection.yaml	Bundles prompt-injection behavior.
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml	Bundles procedure-adherence behavior.
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml	Bundles premature-termination behavior.
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml	Bundles retrieval-quality behavior.
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml	Bundles output-consistency behavior.
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml	Bundles scope-boundary behavior.
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml	Bundles observation-neglect behavior.
pyrit/datasets/assert_behaviors/ios_vs_android.yaml	Bundles mobile-platform preference behavior.
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml	Bundles agent-handoff behavior.
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml	Bundles intent-interpretation behavior.
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml	Bundles insufficient-verification behavior.
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml	Bundles agent-communication behavior.
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml	Bundles tool-selection behavior.
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml	Bundles answer-synthesis behavior.
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml	Bundles identity-verification behavior.
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml	Bundles medical-safety behavior.
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml	Bundles grounding-error behavior.
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml	Bundles goal-drift behavior.
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml	Bundles planning-failure behavior.
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml	Bundles constraint-violation behavior.
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml	Bundles escalation-decision behavior.
pyrit/datasets/assert_behaviors/doxxing.yaml	Bundles private-information protection behavior.
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml	Bundles resort preference-bias behavior.
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml	Bundles conversation-coherence behavior.
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml	Bundles cross-agent constraint behavior.
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml	Bundles instruction-conflict behavior.
pyrit/datasets/assert_behaviors/actionability_failures.yaml	Bundles answer-actionability behavior.
pyrit/datasets/__init__.py	Exports ASSERT preset accessors.
pyrit/common/path.py	Defines bundled preset locations.
pyrit/backend/services/service_lifecycle.py	Resets generation service during cleanup.
pyrit/backend/services/runtime_lifecycle.py	Includes generation work in runtime lifecycle.
pyrit/backend/main.py	Registers prompt-generation routes.
pyrit/analytics/__init__.py	Exports ASSERT metrics and rate helpers.
frontend/src/utils/routeParams.ts	Adds dataset-preselected launch URLs.
frontend/src/utils/routeParams.test.ts	Tests dataset URL encoding and parsing.
frontend/src/test-utils/replaceText.ts	Adds efficient text replacement helper.
frontend/src/hooks/useRequestedDatasets.ts	Validates requested dataset selections.
frontend/src/hooks/usePromptGenerators.ts	Loads runtime-specific generator catalog.
frontend/src/hooks/usePromptGenerationPreflight.ts	Debounces validation and estimate requests.
frontend/src/hooks/usePromptGeneration.ts	Loads and polls generation records.
frontend/src/hooks/useAssertPolicyMetrics.ts	Loads completed-run ASSERT metrics.
frontend/src/hooks/useAssertPolicyMetrics.test.tsx	Tests metrics loading and retry behavior.
frontend/src/hooks/useAssertLibrary.ts	Loads bundled behavior and suite templates.
frontend/src/components/Scenarios/ScenarioRunPage.tsx	Displays completed ASSERT run metrics.
frontend/src/components/Scenarios/ScenarioRunPage.test.tsx	Tests conditional metrics display.
frontend/src/components/Scenarios/scenarioLaunchForm.ts	Seeds dataset selections from links.
frontend/src/components/Scenarios/assertPolicyRun.ts	Identifies ASSERT policy runs.
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts	Styles metrics presentation.
frontend/src/components/PromptGeneration/TargetObjectiveSummary.tsx	Displays generated objective counts.
frontend/src/components/PromptGeneration/RegenerateDatasetLink.tsx	Links generated datasets to regeneration.
frontend/src/components/PromptGeneration/RegenerateDatasetLink.test.tsx	Tests regeneration link availability.
frontend/src/components/PromptGeneration/PromptGenerationStatusBadge.tsx	Displays generation status badges.
frontend/src/components/PromptGeneration/promptGenerationRoutes.ts	Defines generation routes and query parameters.
frontend/src/components/PromptGeneration/promptGenerationRoutes.test.ts	Tests generation route helpers.
frontend/src/components/PromptGeneration/GenericReviewStep.tsx	Provides a generic review continuation view.
frontend/src/components/PromptGeneration/GenericGenerationSummary.tsx	Displays generic scalar result summaries.
frontend/src/components/PromptGeneration/generatorUi.ts	Defines generator UI extension interfaces.
frontend/src/components/PromptGeneration/generatorRegistry.tsx	Selects specialized or generic generator views.
frontend/src/components/PromptGeneration/GenerationTargetFields.test.tsx	Tests target selection and errors.
frontend/src/components/PromptGeneration/GenerationSummaryFacts.tsx	Renders reusable result facts.
frontend/src/components/PromptGeneration/generationFormHelpers.test.ts	Tests form validation helpers.
frontend/src/components/PromptGeneration/GenerateDatasetLinks.tsx	Links scenarios to matching generators.
frontend/src/components/Datasets/DatasetSelection.tsx	Adds dataset regeneration entry point.
frontend/src/components/Datasets/DatasetSelection.test.tsx	Tests dataset regeneration integration.
frontend/src/components/Datasets/DatasetCatalog.tsx	Adds generation navigation button.
frontend/src/components/Datasets/DatasetCatalog.test.tsx	Tests generation navigation.
frontend/src/App.tsx	Registers generation pages.
frontend/README.md	Documents isolated browser checks.
frontend/playwright.prompt-generation.config.ts	Configures offline generation browser tests.
frontend/playwright.config.ts	Excludes the dedicated generation spec.
frontend/package.json	Adds generation browser-test command.
doc/myst.yml	Adds trace-scoring documentation navigation.
doc/gui/0_gui.md	Documents GUI dataset generation.
doc/code/scoring/1_true_false_scorers.py	Documents custom ASSERT judging.
doc/code/scoring/1_true_false_scorers.ipynb	Mirrors custom judging documentation.
doc/code/scoring/0_scoring.py	Links execution-trace scoring guidance.
doc/code/scoring/0_scoring.ipynb	Mirrors trace-scoring guidance link.
doc/code/executor/7_promptgen.py	Demonstrates checkpoint review and continuation.
doc/code/analytics/0_attack_results.md	Documents rates and ASSERT policy metrics.
.github/workflows/frontend_tests.yml	Adds isolated generation browser-test CI job.

_{Review effort: Balanced}

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Reviewed as the incremental diff vs richlundeen-assert-generation-backend (PR #4134), ~9k added lines. Overall this is high-quality frontend work: the TS types mirror the backend pydantic models exactly, the API client matches every route, validation replicates the backend validators, and the e2e setup (real FastAPI app over an offline target, plus a contract-pinning unit test) is notably better than a hand-rolled fake. Inline comments cover the bugs; a few broader notes:

Design

The three generator forms triplicate an identical ~40-line shell (state, submit, preflight, footer plumbing); a shared hook would keep the preflight request shape in one place.
AssertGenerationFormContent is ~530 lines with 15 useStates; the suite section (4 parallel states + table) wants to be its own component.
Polling stops on awaiting_review, so a 'Review needed' page/list never refreshes if the review is continued elsewhere; Continue then 409s with no reload path.
Web-search capability is handled asymmetrically: the research-role filter requires supports_web_search === true (targets without capabilities are hidden), while the generation-target fallback only rejects === false.

Test gaps

The preflight transport-failure path (warning + still-enabled Generate + estimate card state) has zero coverage.
Suite-mode validation (duplicate dataset names, 100-dataset cap, label length, cleared name) is untested.
usePolledRecord's subtlest branches are untested: 404 arriving mid-poll, unmount mid-fetch, a changing recordKey.
assertDraftFromRequest is never tested with a stored stratified request.
The e2e journey never exercises cancel/retry/delete/draft/group against the real backend; given the fixture serves the real routes, a short cancel-then-retry journey would be cheap insurance.

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Re-review of d50c7d0 (incremental vs the rebased #4134 head). The rework addressed two of my earlier bugs by removing their feature surface: drafts (and the PATCH/start two-step with its stale expected_version) are gone, and server-side groups (and the divergent group-status aggregation) were replaced by a batch create with list-page grouping — both simplifications look clean, and the hard-coded supported-generator registry with an explicit "unsupported generator" notice is a reasonable trade. The other earlier comments (default target dropped on deep links, estimate card stuck on "Calculating..." after a preflight failure, suite dataset names not following label edits, 128-char regenerate no-op, usePolledRecord remount contract, dataset:named: duplication, "Pause to review" doc wording, the "21 permissible" e2e assertion) still apply at this head.

Roman Lutz (romanlutz) · 2026-11-10T21:56:44Z

+  }
+
+  /** Saves edits; returns false when the save failed. */
+  const saveAsync = async (): Promise<boolean> => {


With expected_version gone from the review PUT (and the version field dropped server-side), concurrent edits to the same review now silently last-write-win — the previous 409-reload path caught the two-tab case. If dropping optimistic concurrency for reviews is intentional (simpler, single-operator assumption), fine, but worth saying so in the PR description since the earlier design explicitly handled it.

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

…(-4) Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the reference-counted container client so it can't close a client other operations are using. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

main's dataset editing and deletion read selection keys through the shared parse_dataset_selection_key, which replaced DatasetService._parse_selection_key in this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

main now requires explicitly configured scenario target roles. The offline prompt-generation fixture registers offline adversarial_chat and objective_scorer_chat targets, and the journey chooses its adversarial target explicitly before launching foundry.assert_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in main c3cff18 through -5 0ebcd76. The frontend CI build step, npm scripts, and Playwright ignores keep both the prompt-generation and main's scan-launch journeys; ScenarioDetail keeps dataset link preselection with main's target imports. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…layer Bring the published layer-3 metrics (4848f91) onto the rebuilt layer 2 (06ccdf2). The merge applied without conflicts. The Foundry notebook keeps layer 2's outputs and is synced from foundry.py; the new ASSERT metrics cell has no output because it hasn't been run against this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Replaces the per-generator links on scanner launch pages with one Browse datasets / Generate datasets group in the Datasets header. Generate waits for the current runtime's generators, then opens the scenario's only follow-up generator (ASSERT for foundry.assert_policy), otherwise the generator chooser. Linked ?dataset= names stay explicit, exact selections for the server to validate instead of being checked against a second catalog fetch; only identical names are deduplicated. A blank name, or any name on a scanner that takes no datasets, blocks the estimate and launch with an error and a link that removes them. Regenerate links read the stored request by the dataset's exact selection key. foundry.assert_policy marks its datasets as required and asks for one at launch, using the rebuilt -3's ASSERT scenario check. Scenario.get_dataset_options reports inline seeds as fixed sources, so inline-only scenarios offer no dataset picker while their limits and filters stay editable. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The rebuilt -3 launch-request tests build foundry.assert_policy requests without a dataset, which this layer now rejects with 'Select an ASSERT dataset.' They select one so they keep checking the default-scoring rules. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:30

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:31

View session

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread frontend/src/components/PromptGeneration/promptGenerationPresentation.ts Resolved

Comment thread frontend/src/hooks/usePolledRecord.ts Resolved

Comment thread frontend/src/hooks/usePromptGenerationSubmit.ts Resolved

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-gui-generation-plan branch from 6feb697 to b3c7848 Compare November 10, 2026 09:58

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-gui-generation-plan branch 2 times, most recently from 6cd2cf7 to d50c7d0 Compare November 10, 2026 19:38

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-gui-generation-plan branch from d50c7d0 to 9f523be Compare November 10, 2026 23:18

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) and others added 6 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Scenario-aware dataset compatibility (-8) #4180

Draft

Richard Lundeen (richlundeen) and others added 28 commits November 10, 2026 17:42

MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5c) i…

9e697fd

…nto trace evidence

Merge commit '9e697fddfef02c1c17fb1b851cb6285c5bbc704d' into richlund…

78f2477

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…

4848f91

…een-assert-headline-metrics-plan

Merge refreshed ASSERT metrics (-3) with main into staged generation …

da0f38c

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge staged generation (-4) with main into generation jobs (-5)

0ebcd76

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt ASSERT metrics (-3) into staged generation (-4)

1c0add0

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt staged generation (-4) into generation jobs (-5)

22c7a75

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt generation jobs (-5) into the GUI generation layer (-6)

a296da4

Integrates -5 22c7a75 (rebuilt -3 d0b7c2a) under the published -6 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

FEAT: Generate datasets from a policy in the GUI (-6) - #4135

FEAT: Generate datasets from a policy in the GUI (-6)#4135
Richard Lundeen (richlundeen) wants to merge 52 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-gui-generation-plan

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI behavior and scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading