FEAT: Prompt generation jobs in the backend (-5) by richlundeen · Pull Request #4134 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:30:32Z

Part -5 of a 10-PR ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)	this PR
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	ASSERT results grouped by generated category
-10	Planned	Background dataset loading (all scanners)

Each PR targets main and includes the earlier layers, so its Files changed tab also shows them. Review this PR's own changes in the -4 to -5 source comparison. Use source comparisons rather than commit lists: -8 keeps prototype commits in its ancestry.

Description

The backend runs prompt generators as background jobs, so the GUI can estimate, run, review, resume, and save generated datasets. It isn't tied to ASSERT: the target-objective generator uses the same jobs.

PromptGenerationService owns the job lifecycle: admission (one generation at a time by default), checkpoint persistence, review pauses, cancel, retry from the last checkpoint, batch creation for suites, dataset-name reservation, and saving a new named dataset. Shutdown lets a dataset save finish. Retrying after an interruption completes a save that already landed instead of failing on the name. pause_for_review is rejected for generators without a review step.
Adapters translate the ASSERT and target-objective generators into the service's generic records. The generators own model calls and checkpoints, and the dataset providers package the results.
Each record is a JSON document at <memory results_path>/prompt_generations/<id>.json, next to media output. No database migration is needed.
Routes:
- /api/prompt-generators: generator descriptors and the bundled ASSERT behaviors and suites.
- /api/prompt-generations: preflight with estimate, create, review, continue/retry/cancel, delete a record, and the request recorded in an ASSERT or objective dataset.
- /api/prompt-generations/batch: queues generations that share settings, such as an ASSERT suite, as ordinary records with a shared group_id, listed in request order.
DatasetService and the generation service share one parse_dataset_selection_key, including the dataset delete route, without changing which selection keys are accepted.
The backend README and framework.md describe the service's responsibility.

Storage and concurrency. One backend instance owns the generation records. It reads them once into an in-process cache and every write updates both; local files are replaced atomically, including when a save is cancelled. Writes are last-write-wins, including review edits. Earlier revisions of this PR checked a record version on edits and periodically refreshed the cache; both were removed. Those checks only worked inside one process and never coordinated separate backends. Several backends sharing one results storage, such as one blob container, aren't supported: record writes, cancellation and dataset-name reservations aren't coordinated across processes, so two backends could accept the same dataset name or cancel each other's work.

GUI and scope

Own GUI changes: none. This layer changes no frontend files; its routes back the GUI added in #4135.

Shared / non-ASSERT impact (backend only):

The same job lifecycle, routes and records serve ordinary target-objective generation, not only ASSERT.
Generations save ordinary named datasets in memory. Deleting a generation record doesn't delete its saved dataset.
Dataset reads and deletes now go through the shared selection-key parser.

Unchanged / deferred:

Generation records name each role's target by registry name. A retry resolves the name again, and the checkpoint rejects a target whose identifier differs from the one that did the earlier work.
Dataset uploads reserve names in the database, while generations reserve them in their records. An upload can therefore take a name that a running generation reserved; that generation then stops at its save with a name conflict, and Retry can't rename the checkpoint's output. A shared reservation is deferred.
Edit & regenerate creates a new generation and a new dataset; there's no in-place dataset edit, versioning, archiving or retention.

The job page below is the later #4135 frontend that uses these routes, captured with synthetic offline fixture data on -6 8f878a58e; it is not UI added by this PR:

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Critical storage and cross-backend coordination races can overwrite terminal state, hide jobs, and release dataset-name reservations while original workers remain active, allowing duplicate generation.

5 open issues

What changed in this PR

This PR adds backend-managed prompt generation jobs so clients can draft, estimate, review, resume, and save generated datasets using PyRIT’s existing generators and dataset providers.

Changes:

Integrates generation jobs with backend routing, runtime lifecycle, estimates, and dataset packaging.
Bundles ASSERT behavior and suite presets with attribution.
Adds supporting ASSERT scoring, trace evidence, analytics, frontend integration, and documentation.

File	Description
THIRD_PARTY_NOTICES.txt	Updates ASSERT attribution.
tests/unit/executor/promptgen/test_target_objective_generator.py	Tests generation estimates.
tests/unit/datasets/test_target_objective_provider.py	Tests result packaging.
tests/unit/common/test_lazy_package_imports.py	Updates analytics export count.
tests/unit/assert_mocks.py	Adds checkpoint fixtures.
tests/unit/analytics/test_rate_statistics.py	Tests binary and macro rates.
pyrit/score/true_false/true_false_fallback_scorer.py	Adds configurable fallback triggers.
pyrit/score/scorer.py	Identifies missing-evidence results.
pyrit/score/observation/otel_span_exporter.py	Exports richer trace evidence.
pyrit/score/observation/message_trace_resolver.py	Resolves conversation trace links.
pyrit/score/observation/execution.py	Shares evidence acquisition across scorers.
pyrit/score/observation/__init__.py	Exports trace observation APIs.
pyrit/score/float_scale/float_scale_fallback_scorer.py	Adds configurable fallback triggers.
pyrit/score/__init__.py	Exports ASSERT and tracing APIs.
pyrit/scenario/core/scenario.py	Extracts scenario identity parameters.
pyrit/prompt_target/common/target_trace_config.py	Adds conversation identifiers to spans.
pyrit/prompt_target/common/prompt_target.py	Exposes target tracing configuration.
pyrit/models/score/__init__.py	Exports execution-trace models.
pyrit/models/prompt_generation_estimate.py	Defines generation estimate contracts.
pyrit/models/analytics.py	Adds rate result models.
pyrit/executor/promptgen/target_objective_generator.py	Estimates objective-generation work.
pyrit/executor/promptgen/__init__.py	Exports objective-generation results.
pyrit/datasets/seed_datasets/generated/target_objective_provider.py	Packages existing generation results.
pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml	Guides execution-evidence judgments.
pyrit/datasets/score/assert/judge_presets/tool-use.yaml	Adds tool-use judging criteria.
pyrit/datasets/score/assert/judge_presets/safety-extended.yaml	Adds extended safety criteria.
pyrit/datasets/score/assert/judge_presets/safety-core.yaml	References core safety dimensions.
pyrit/datasets/score/assert/judge_presets/robustness.yaml	Adds robustness judging criteria.
pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml	Adds policy-adherence criteria.
pyrit/datasets/score/assert/judge_presets/operational.yaml	Adds operational judging criteria.
pyrit/datasets/score/assert/judge_presets/multi-turn.yaml	Adds multi-turn judging criteria.
pyrit/datasets/score/assert/judge_presets/instruction-following.yaml	Adds instruction-following criteria.
pyrit/datasets/score/assert/judge_presets/grounding.yaml	Adds grounding judging criteria.
pyrit/datasets/score/assert/judge_presets/communication.yaml	Adds communication judging criteria.
pyrit/datasets/score/assert/judge_presets/alignment.yaml	Adds alignment judging criteria.
pyrit/datasets/score/assert/dimension_system_prompt.yaml	Adds custom-dimension judge template.
pyrit/datasets/assert_suites/travel_planner.yaml	Bundles travel-planner suite.
pyrit/datasets/assert_suites/travel_planner_benchmark.yaml	Bundles travel-quality benchmark suite.
pyrit/datasets/assert_suites/telecom_customer_service.yaml	Bundles telecom support suite.
pyrit/datasets/assert_library.py	Loads and validates bundled presets.
pyrit/datasets/assert_behaviors/violent_content.yaml	Bundles violence behavior preset.
pyrit/datasets/assert_behaviors/unsupported_conclusion_failures.yaml	Bundles unsupported-conclusion preset.
pyrit/datasets/assert_behaviors/unit_conversion_failures.yaml	Bundles unit-conversion preset.
pyrit/datasets/assert_behaviors/underused_context_failures.yaml	Bundles context-underuse preset.
pyrit/datasets/assert_behaviors/unauthorized_action_failures.yaml	Bundles unauthorized-action preset.
pyrit/datasets/assert_behaviors/tool_parameter_formatting_failures.yaml	Bundles tool-argument preset.
pyrit/datasets/assert_behaviors/tool_output_misinterpretation_failures.yaml	Bundles tool-output interpretation preset.
pyrit/datasets/assert_behaviors/tool_orchestration_errors.yaml	Bundles tool-orchestration preset.
pyrit/datasets/assert_behaviors/tool_call_turn_protocol_failures.yaml	Bundles tool-turn protocol preset.
pyrit/datasets/assert_behaviors/tool_call_error_recovery_failures.yaml	Bundles tool-error recovery preset.
pyrit/datasets/assert_behaviors/tabs_vs_spaces.yaml	Bundles indentation-bias preset.
pyrit/datasets/assert_behaviors/sycophancy.yaml	Bundles sycophancy preset.
pyrit/datasets/assert_behaviors/suicide_self_harm.yaml	Bundles self-harm safety preset.
pyrit/datasets/assert_behaviors/success_criteria_ambiguity_failures.yaml	Bundles success-criteria preset.
pyrit/datasets/assert_behaviors/stereotyping.yaml	Bundles stereotyping preset.
pyrit/datasets/assert_behaviors/stale_state_failures.yaml	Bundles stale-state preset.
pyrit/datasets/assert_behaviors/shimano_vs_sram.yaml	Bundles cycling-brand bias preset.
pyrit/datasets/assert_behaviors/sexual_content.yaml	Bundles sexual-content safety preset.
pyrit/datasets/assert_behaviors/response_completeness_failures.yaml	Bundles response-completeness preset.
pyrit/datasets/assert_behaviors/repeated_action_loop_failures.yaml	Bundles repeated-action preset.
pyrit/datasets/assert_behaviors/prompt_injection.yaml	Bundles prompt-injection preset.
pyrit/datasets/assert_behaviors/procedure_adherence_failures.yaml	Bundles procedure-adherence preset.
pyrit/datasets/assert_behaviors/premature_termination_failures.yaml	Bundles premature-termination preset.
pyrit/datasets/assert_behaviors/poor_retrieval_failures.yaml	Bundles retrieval-quality preset.
pyrit/datasets/assert_behaviors/output_internal_consistency_failures.yaml	Bundles output-consistency preset.
pyrit/datasets/assert_behaviors/out_of_scope_request_failures.yaml	Bundles scope-compliance preset.
pyrit/datasets/assert_behaviors/observation_neglect_failures.yaml	Bundles observation-neglect preset.
pyrit/datasets/assert_behaviors/malicious_cyber_activity.yaml	Bundles cyber-safety preset.
pyrit/datasets/assert_behaviors/ios_vs_android.yaml	Bundles platform-bias preset.
pyrit/datasets/assert_behaviors/inter_agent_handoff_failures.yaml	Bundles agent-handoff preset.
pyrit/datasets/assert_behaviors/intent_misinterpretation_failures.yaml	Bundles intent-interpretation preset.
pyrit/datasets/assert_behaviors/insufficient_verification_failures.yaml	Bundles verification-failure preset.
pyrit/datasets/assert_behaviors/ineffective_team_communication_failures.yaml	Bundles agent-communication preset.
pyrit/datasets/assert_behaviors/incorrect_tool_selection_failures.yaml	Bundles tool-selection preset.
pyrit/datasets/assert_behaviors/incomplete_answer_synthesis_failures.yaml	Bundles answer-synthesis preset.
pyrit/datasets/assert_behaviors/identity_verification_failures.yaml	Bundles identity-verification preset.
pyrit/datasets/assert_behaviors/hate_speech_harassment.yaml	Bundles harassment-safety preset.
pyrit/datasets/assert_behaviors/harmful_medical_advice.yaml	Bundles medical-safety preset.
pyrit/datasets/assert_behaviors/grounding_attribution_errors.yaml	Bundles grounding-error preset.
pyrit/datasets/assert_behaviors/goal_drift_failures.yaml	Bundles goal-drift preset.
pyrit/datasets/assert_behaviors/flawed_action_plan_failures.yaml	Bundles action-planning preset.
pyrit/datasets/assert_behaviors/explicit_constraint_violation_failures.yaml	Bundles constraint-compliance preset.
pyrit/datasets/assert_behaviors/escalation_judgment_failures.yaml	Bundles escalation-decision preset.
pyrit/datasets/assert_behaviors/doxxing.yaml	Bundles privacy-safety preset.
pyrit/datasets/assert_behaviors/crystal_mountain_vs_snoqualmie.yaml	Bundles destination-bias preset.
pyrit/datasets/assert_behaviors/conversation_coherence_breakdown.yaml	Bundles conversation-coherence preset.
pyrit/datasets/assert_behaviors/constraint_propagation_failures.yaml	Bundles constraint-propagation preset.
pyrit/datasets/assert_behaviors/conflicting_instruction_resolution_failures.yaml	Bundles instruction-conflict preset.
pyrit/datasets/assert_behaviors/actionability_failures.yaml	Bundles actionability preset.
pyrit/datasets/__init__.py	Exports preset lookup APIs.
pyrit/common/path.py	Defines bundled preset paths.
pyrit/backend/services/service_lifecycle.py	Resets generation service during cleanup.
pyrit/backend/services/runtime_lifecycle.py	Includes generation jobs in lifecycle checks.
pyrit/backend/routes/scenarios.py	Exposes ASSERT metrics endpoint.
pyrit/backend/README.md	Documents jobs and metrics APIs.
pyrit/backend/main.py	Registers generation routes.
pyrit/analytics/__init__.py	Exports ASSERT analytics helpers.
frontend/src/types/index.ts	Defines ASSERT metrics contracts.
frontend/src/services/api.ts	Adds metrics API client.
frontend/src/hooks/useAssertPolicyMetrics.ts	Loads metrics with request cancellation.
frontend/src/hooks/useAssertPolicyMetrics.test.tsx	Tests metrics loading and retries.
frontend/src/components/Scenarios/ScenarioRunPage.tsx	Displays metrics for completed ASSERT runs.
frontend/src/components/Scenarios/ScenarioRunPage.test.tsx	Tests conditional metrics display.
frontend/src/components/Scenarios/assertPolicyRun.ts	Identifies ASSERT policy runs.
frontend/src/components/Scenarios/AssertPolicyMetricsPanel.styles.ts	Styles metrics presentation.
doc/myst.yml	Adds trace-scoring navigation.
doc/code/scoring/1_true_false_scorers.py	Documents dimensions and sampled judgments.
doc/code/scoring/1_true_false_scorers.ipynb	Updates paired scoring documentation.
doc/code/scoring/0_scoring.py	Links execution-trace scoring guidance.
doc/code/scoring/0_scoring.ipynb	Updates paired scoring overview.
doc/code/framework.md	Defines job, trace, and analytics responsibilities.
doc/code/executor/7_promptgen.py	Demonstrates checkpoint review and continuation.
doc/code/executor/7_promptgen.ipynb	Adds paired generation example.
doc/code/analytics/0_attack_results.md	Documents rate helpers and ASSERT metrics.

_{Review effort: Balanced}

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Reviewed the incremental diff vs the -4 branch (richlundeen-assert-generation-sdk). Solid job layer — the lifecycle edge cases (shielded cancel updates, shutdown waiting for saves, idempotent retry-after-save) are handled with care, and the tests exercise real generators. One small 500-level bug and a handful of polish items below. All 37 new tests pass for me locally.

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Re-review of ea4dbc1. The simplification pass (drafts, versioning, and the groups API removed; single-backend ownership made explicit) resolves my round-1 points 1, 4, 5, and 6 — point 1's 500 bug went away with the PATCH endpoint itself — and the new cancel-before-start handling fixes a latent bug I'd missed. Still open from round 1: pause_for_review accepted for non-reviewable generators, and the exact preset counts in the route test. Three small new items inline. All 35 tests pass locally on the new head.

Roman Lutz (romanlutz) · 2026-11-10T21:55:02Z

+        async with self._lock:
+            taken = await self._get_taken_dataset_names_async(exclude_id=generation_id)
+            if record.dataset_name in taken and not await self._dataset_came_from_async(record=record):
+                raise PromptGenerationConflictError(_name_taken_message(record.dataset_name))


With the rename path (PATCH) gone, a retryable generation whose dataset name has since been taken can never be retried — and cancellation releases the reservation, so this is reachable: cancel, someone reuses the name, retry 409s forever, and the checkpoint's paid-for model work is stranded (from-dataset regeneration starts over). Was dropping rename-before-retry a deliberate tradeoff? If so, fine — but maybe the conflict message shouldn't point at "Edit & regenerate" as if the checkpoint were recoverable.

Roman Lutz (romanlutz) · 2026-11-10T21:55:02Z

+            list[PromptGenerationRecord]: Stored records.
+        """
+        records = (await self._load_cache_async()).values()
+        return sorted(records, key=lambda record: record.created_at, reverse=True)


Records created in one batch share a single created_at, so this sort leaves within-batch order undefined — in-process it happens to be insertion order, but after a restart it becomes file-listing order (random UUID filenames). A stable secondary key (e.g. record.id, or a per-item index carried from the batch request) would keep listings deterministic for the GUI.

Roman Lutz (romanlutz) · 2026-11-10T21:55:03Z

+            record = await self.get_async(generation_id)
+            # A task cancelled before it started never ran its handler, so its record is still queued.
+            if record.status != PromptGenerationStatus.QUEUED:
+                return record


Nice catch on the never-started-task case. It doesn't look covered by a test though — with the default concurrency of 1, queueing two generations and cancelling the second while it waits on the semaphore… actually that one does run its handler; the truly-never-started case is cancelling immediately after create, before the event loop first schedules the task. Worth a test either way, since this branch is what keeps the record from being returned as QUEUED while claiming cancellation succeeded.

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Report ASSERT's permissible/impermissible split, a rate per boolean judge dimension, and judged-category violation rates per technique, with 90% Wilson intervals and ASSERT's minimum support. Expose them through a backend route and the GUI run page. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…RT behavior library AssertGenerator gains a checkpointed staged API (advance_async, build_result, estimate, validate_config): stop after a stage, apply a reviewed taxonomy and dimensions (recorded in each group's provenance), and resume after interruption without repeating model calls. TargetObjectiveGenerator.estimate and the provider-free PromptGenerationEstimate report remaining calls and a token range. The ASSERT v0.3.0 behavior (52) and suite (3) presets are vendored with loaders, and StorageIO gains list_files_async. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Checkpoints carry a format version and the identifier of the target that produced each role's work. Resuming with a different target is rejected before any model call, build_result attributes work to the recorded producers, and stored seed-group assignments must match the planned batches. Estimates count the dimensions and target system prompt sent with every seed-group call. Objective datasets record the request that generated them, the objective estimate reads its prompt once, and Azure Blob listings percent-encode names so they round-trip. AssertGenerationStage is a StrEnum. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

PromptGenerationService runs generators in the background: preflight and estimates, review pauses, cancel, retry from the last checkpoint, batch creation for suites, and saving a new named dataset. One backend instance owns the records, stored as JSON documents in the memory's results storage. Adapters translate the ASSERT and target-objective generators; routes live under /api/prompt-generators and /api/prompt-generations. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A cancelled local save finishes its file replacement before the cancellation handler writes again, and the first cache load is serialized with saves and deletes. pause_for_review is rejected for generators without a review step, objective datasets can be edited and regenerated, and batch members keep their request order. The README states that several backends sharing one results storage aren't supported. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

…(-4) Brings in main 1dc1d5c (including dataset uploads, microsoft#3920) through -3 78f2477. Storage keeps both list_files_async and main's write_stream_async/canonical_path; Azure write_stream_async now shares the reference-counted container client so it can't close a client other operations are using. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

main's dataset editing and deletion read selection keys through the shared parse_dataset_selection_key, which replaced DatasetService._parse_selection_key in this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…een-assert-headline-metrics-plan

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…layer Bring the published layer-3 metrics (4848f91) onto the rebuilt layer 2 (06ccdf2). The merge applied without conflicts. The Foundry notebook keeps layer 2's outputs and is synced from foundry.py; the new ASSERT metrics cell has no output because it hasn't been run against this layer. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…distinct ASSERT metrics now let a saved registry name decide whether foundry.assert_policy produced a result, so another scanner's AssertPolicy class isn't mistaken for ASSERT. Older results without a registry name must come from PyRIT's AssertPolicy module. The metrics route answers other scenarios with a 400 problem of type /errors/not-assert-policy-run. The run page mounts the panel on a matching registry name, or tentatively on the class name for older runs, and hides it only for that problem type; compatibility, validation and other errors still show. A supplied objective scorer's uncategorized verdicts are reported as the objective dimension, not as ASSERT's policy judge. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…sions A result's policy now comes only from the stored seed group with exactly the seeds it recorded, and only when that group's taxonomy matches the taxonomy the policy judge recorded in its saved scorer identifier, which survives sampling and trace wrappers. A replaced or edited dataset leaves the conversation without a matching stored policy instead of reporting another group's categories. Unmarked legacy scores count as ASSERT dimensions only with that judge provenance, so a supplied scorer that uses the policy_violation category is reported as the objective dimension. Category rates are kept per policy, with a policy_id, so policies that share a behavior or category name aren't pooled, and the panel names them apart. Results without a scenario module are no longer accepted as AssertPolicy by class name alone. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

A stored seed group's policy is used only when the policy score's saved identifier comes from the scenario's built-in judge wrapper and records the same taxonomy; a policy score without that evidence leaves the conversation without a matching stored policy, while its dimension rates still count it. Only questions under that wrapper count as judge evidence, so a supplied SelfAskTrueFalseScorer whose question carries ASSERT-shaped metadata and the policy_violation category is reported as the objective dimension. Category rates are keyed by the judged policy (taxonomy, application context and target instructions), so policies that share a taxonomy but differ in context or instructions aren't pooled. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

The metrics no longer read a stored group's assigned category. ASSERT's judges recompute policy_violation from their category judgments, so every violation verdict already names a violated category; ASSERT's attribution of category-less violations to the generated category isn't needed and would depend on seed metadata that can be edited after a run. A stored policy verdict that disagrees with its category judgments is now rejected as malformed instead of being attributed, and AssertCategoryMetrics drops attributed_by_fallback. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:30

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:31

View session

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-backend branch from 38e7c2a to 588e3b9 Compare November 10, 2026 09:58

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-backend branch from 588e3b9 to ea4dbc1 Compare November 10, 2026 18:31

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-backend branch from ea4dbc1 to 6f66fd2 Compare November 10, 2026 23:18

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) and others added 7 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 4 commits November 10, 2026 16:31

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-generation-backend branch from 6f66fd2 to 7875121 Compare November 11, 2026 01:04

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Scenario-aware dataset compatibility (-8) #4180

Draft

Richard Lundeen (richlundeen) and others added 23 commits November 10, 2026 17:42

MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5c) i…

9e697fd

…nto trace evidence

Merge commit '9e697fddfef02c1c17fb1b851cb6285c5bbc704d' into richlund…

78f2477

…een-assert-headline-metrics-plan # Conflicts: # pyrit/backend/routes/scenarios.py

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

Merge commit '7bf65f3969fb6cd8b2c14be28456a848baf9c370' into richlund…

4848f91

…een-assert-headline-metrics-plan

Merge refreshed ASSERT metrics (-3) with main into staged generation …

da0f38c

…(-4) Brings in main c3cff18 through -3 4848f91. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge staged generation (-4) with main into generation jobs (-5)

0ebcd76

Brings in main c3cff18 through -4 da0f38c. No conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt ASSERT metrics (-3) into staged generation (-4)

1c0add0

Integrates -3 d0b7c2a (typed judgments, provenance and verdict corrections) under the published -4 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merge rebuilt staged generation (-4) into generation jobs (-5)

22c7a75

Integrates -4 1c0add0 (rebuilt -3 d0b7c2a) under the published -5 head; no conflicts. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

FEAT: Prompt generation jobs in the backend (-5) - #4134

FEAT: Prompt generation jobs in the backend (-5)#4134
Richard Lundeen (richlundeen) wants to merge 44 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-generation-backend

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI and scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

Roman Lutz (romanlutz) Nov 10, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading