Repository navigation
FEAT: Save converters from the Registry (part 13) - #4052
Open
varunj-msft wants to merge 357 commits into
Open
varunj-msft wants to merge 357 commits into
varunj-msft wants to merge 357 commits into
Conversation
The dialog asks for a registry name and suggests one, saves the target by default, and creates a Temporary target only when it can't be saved, saying why before it is created. The Targets table labels each target Saved, From configuration or Temporary.
# Conflicts: # pyrit/registry/registry_recipe_restore.py
# Conflicts: # pyrit/registry/registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_instance_registry.py
…entials The Azure Speech converters and the Azure Content Safety scorer read this server's key when theirs is empty, or else sign in with its Azure identity, and no value turns that off. They now declare that key as a server fallback, so a saved one is refused when it's saved or restored, as saved targets already are. The audio scorers transcribe with this server's Speech settings, and the local refusal classifier scorer can load its model from a folder on this server or download it with this server's Hugging Face sign-in, so those can't be saved. A class that declares no auth modes now supports only api_key in the saved-entry check, as the saveability check already assumed, so a recipe can't pass by adding an auth_mode the class doesn't take.
Only targets can be saved through the API so far. The steps that turn off the server values an entry would read, refuse what it still can't hold or a Temporary entry it names, write its recipe, register it, and delete the recipe again if registering fails move from the target service into a shared module, along with the route helpers for the owner and the not-saveable problem. The origin enum becomes RegistryEntryOrigin, read by registry_entry_origin, so converters and scorers can use the same pieces. Targets behave as before.
POST /api/converters and POST /api/scorers take "persist": true, as POST /api/targets does: the entry is built the way restore builds it, saved under registry_recipes_source with the signed-in user, and restored at every start. An entry that would use a Temporary target is refused, as is one that would use this server's credentials. A converter that would keep an uploaded file, or a copy made with source, can't be saved yet. Each listed converter and scorer carries its origin, and the converters and scorers the startup scripts register are marked as coming from configuration. A saved converter can't be removed through the API yet, since it would come back from its file at the next start.
The converter dialog only made Temporary converters, wherever it opened: the Converters tab, the converter panel in chat, and New converter in the technique editor. It now saves the converter by default under a suggested name such as base64_converter, which keeps clear of the names the startup initializers use. A converter type that can't be saved, a converter with an uploaded file, one that uses a Temporary target, or a save the server refused is offered as Temporary instead, with the reason. A taken name shows on the name field. GET /api/converters/types says why a type can't be saved. The Converters and Scorers tables label each entry Saved, From configuration, or Temporary, and a saved converter's remove button is disabled with the reason. Save to registry on a converter's changed settings says the copy is Temporary. The origin badge and its labels move to shared names.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Caught duplicate-name collisions can let launch proceed with an unintended saved component, while synchronous restoration can block health and management requests.
2 open issues
What changed in this PR
This PR adds persistent registry entries to PyRIT’s converter-creation workflow and distinguishes saved entries from configuration-provided and temporary instances. It also includes supporting persistence, restoration, and target-management changes from the preceding work.
Changes:
- Add saved-entry storage, validation, restoration, and protection against launch-script replacement.
- Expose persistence and origin information through the API and registry UI.
- Extend regression coverage, isolate test storage, and document saving and removal behavior.
| File | Description |
|---|---|
| tests/unit/setup/test_targets_initializer.py | Tests saved-target collision protection. |
| tests/unit/setup/test_scorer_initializer.py | Tests saved-scorer tagging protection. |
| tests/unit/setup/test_converter_initializer.py | Tests saved-converter replacement protection. |
| tests/unit/setup/test_configuration_loader.py | Tests recipe-storage configuration. |
| tests/unit/registry/test_instance_registry.py | Tests origins and recorded replacement refusals. |
| tests/unit/registry/test_attack_technique_registry.py | Tests saved-technique collision handling. |
| tests/unit/backend/test_scorer_routes.py | Tests scorer persistence and API errors. |
| tests/unit/backend/test_scenario_suite_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_scenario_suite_execution_service.py | Tests saved-entry launch-conflict reporting. |
| tests/unit/backend/test_scenario_run_service.py | Tests launch-script replacement safeguards. |
| tests/unit/backend/test_scenario_preset_routes.py | Tests sanitized storage failures. |
| tests/unit/backend/test_saved_entries.py | Tests temporary-reference rejection. |
| tests/unit/backend/test_mappers.py | Tests target origin and ownership mapping. |
| tests/unit/backend/test_initializer_service.py | Tests local-storage failure responses. |
| tests/unit/backend/test_auth_middleware.py | Tests current-user ID lookup. |
| tests/unit/backend/test_analytics_benchmark.py | Tests benchmark recipe-storage isolation. |
| tests/unit/backend/conftest.py | Isolates saved entries for backend tests. |
| tests/end_to_end/conftest.py | Gives test backends isolated recipe storage. |
| test_support/analytics_benchmark.py | Configures isolated benchmark recipe storage. |
| pyrit/setup/initializers/tools.py | Rejects saved-entry name collisions. |
| pyrit/setup/initializers/targets.py | Protects saved targets during grouping. |
| pyrit/setup/initializers/scorers.py | Prevents saved scorers replacing initializer defaults. |
| pyrit/setup/configuration_loader.py | Adds recipe-storage configuration and validation. |
| pyrit/score/true_false/local_refusal_classifier_scorer.py | Declares model-related persistence restrictions. |
| pyrit/score/true_false/audio_true_false_scorer.py | Declares transcription-related persistence restrictions. |
| pyrit/score/scorer.py | Adds scorer saveability metadata. |
| pyrit/score/float_scale/azure_content_filter_scorer.py | Declares server-credential fallback behavior. |
| pyrit/score/float_scale/audio_float_scale_scorer.py | Declares transcription-related persistence restrictions. |
| pyrit/registry/registry_recipe_storage.py | Implements per-registry JSON recipe storage. |
| pyrit/registry/instance_registry.py | Tracks origins, ownership, and replacement refusals. |
| pyrit/registry/components/scorer_registry.py | Reserves the scorer types route name. |
| pyrit/registry/components/attack_technique_registry.py | Supports saved metadata and collision protection. |
| pyrit/prompt_target/websocket_copilot_target.py | Declares account-related persistence restrictions. |
| pyrit/prompt_target/prompt_shield_target.py | Declares credential and destination rules. |
| pyrit/prompt_target/openai/openai_target.py | Declares credential, header, and destination rules. |
| pyrit/prompt_target/litellm_chat_target.py | Declares API-key fallback behavior. |
| pyrit/prompt_target/hugging_face/hugging_face_chat_target.py | Declares token and model persistence restrictions. |
| pyrit/prompt_target/http_target/http_target.py | Disallows saving credential-bearing raw requests. |
| pyrit/prompt_target/hack_a_prompt_target.py | Declares session and cookie fallback behavior. |
| pyrit/prompt_target/github_copilot_target.py | Declares token fallback behavior. |
| pyrit/prompt_target/common/prompt_target.py | Defines target persistence-policy metadata. |
| pyrit/prompt_target/azure_ml_chat_target.py | Declares credential and endpoint rules. |
| pyrit/prompt_target/azure_blob_storage_target.py | Declares credential and container-address rules. |
| pyrit/prompt_target/a2a_target.py | Declares endpoint and agent-card destination rules. |
| pyrit/models/catalog/target.py | Adds target origin and saver fields. |
| pyrit/models/catalog/scorer.py | Adds scorer origin. |
| pyrit/models/catalog/registry_recipe.py | Defines saved recipes and restoration reports. |
| pyrit/models/catalog/__init__.py | Lazily exports recipe models. |
| pyrit/converter/converter.py | Adds credential-fallback metadata. |
| pyrit/converter/azure_speech_text_to_audio_converter.py | Declares speech credential fallback. |
| pyrit/converter/azure_speech_audio_to_text_converter.py | Declares speech credential fallback. |
| pyrit/backend/services/scorer_service.py | Saves scorers and reports their origins. |
| pyrit/backend/services/scenario_suite_execution_service.py | Exposes actionable saved-entry launch conflicts. |
| pyrit/backend/services/scenario_run_service.py | Detects launch-script replacement refusals. |
| pyrit/backend/services/saved_entries.py | Shares save validation and registration handling. |
| pyrit/backend/services/runtime_lifecycle.py | Restores saved entries and reports failures. |
| pyrit/backend/services/converter_service.py | Saves converters and exposes saveability rules. |
| pyrit/backend/routes/targets.py | Adds persistence errors and caller-aware responses. |
| pyrit/backend/routes/scorers.py | Handles scorer persistence requests and errors. |
| pyrit/backend/routes/scenarios.py | Documents saved-entry launch conflicts. |
| pyrit/backend/routes/scenario_suites.py | Sanitizes storage-error logging and responses. |
| pyrit/backend/routes/scenario_presets.py | Sanitizes storage-error logging and responses. |
| pyrit/backend/routes/initializers.py | Handles local-storage failures safely. |
| pyrit/backend/routes/converters.py | Handles persistence and saved-removal restrictions. |
| pyrit/backend/routes/common.py | Shares ownership and save-refusal response helpers. |
| pyrit/backend/README.md | Documents persistence APIs and restrictions. |
| pyrit/backend/models/targets.py | Adds persistence validation and saveability fields. |
| pyrit/backend/models/scorers.py | Adds scorer persistence validation. |
| pyrit/backend/models/converters.py | Adds persistence, origin, and saveability fields. |
| pyrit/backend/middleware/auth.py | Exposes the authenticated user’s object ID. |
| pyrit/backend/mappers/target_mappers.py | Maps target origin and saver information. |
| pyrit/backend/mappers/converter_mappers.py | Maps converter origin. |
| frontend/src/utils/registryOrigin.ts | Defines shared origin labels. |
| frontend/src/utils/registryName.ts | Generates valid, unused registry names. |
| frontend/src/utils/registryName.test.ts | Tests name validation and suggestions. |
| frontend/src/types/index.ts | Adds persistence and origin API types. |
| frontend/src/test-utils/targetFixtures.ts | Supports origin and saver test data. |
| frontend/src/services/errors.ts | Recognizes typed save refusals. |
| frontend/src/services/errors.test.ts | Tests save-refusal classification. |
| frontend/src/components/Techniques/TechniqueEditor.test.tsx | Updates converter save-button expectations. |
| frontend/src/components/Registry/ScorerRegistry.tsx | Displays scorer origin badges. |
| frontend/src/components/Registry/ScorerRegistry.test.tsx | Tests scorer origin labels. |
| frontend/src/components/Registry/OriginBadge.tsx | Provides shared origin badges. |
| frontend/src/components/Registry/ConverterRegistry.tsx | Shows origins and disables saved-entry removal. |
| frontend/src/components/Registry/ConverterRegistry.test.tsx | Tests origins and saved-removal guidance. |
| frontend/src/components/Converters/SaveConverterDialog.tsx | Clarifies that converter copies remain temporary. |
| frontend/src/components/Converters/SaveConverterDialog.test.tsx | Tests temporary-copy wording. |
| frontend/src/components/Config/TargetTable.tsx | Shows origins, savers, and an ownership filter. |
| frontend/src/components/Config/TargetTable.test.tsx | Tests target origins and ownership filtering. |
| frontend/src/components/Config/TargetTable.styles.ts | Styles saver text and visibility controls. |
| frontend/playwright.config.ts | Isolates seeded backend recipe storage. |
| frontend/e2e/fixtures/manual_send_backend.py | Isolates offline fixture recipe storage. |
| frontend/e2e/config.spec.ts | Tests target saving and ownership presentation. |
| frontend/e2e/_targets.ts | Adds origin and ownership mock fields. |
| doc/gui/0_gui.md | Explains saved entries and manual removal. |
| doc/getting_started/pyrit_conf.md | Documents recipe storage and restoration behavior. |
| .pyrit_conf_example | Adds recipe-storage configuration guidance. |
Review effort: Balanced
Its text can carry a SAS URL. The 503 handler no longer logs it as a debug traceback, and the scenario suite service no longer attaches it to its warning; both log the storage message, which has no credentials.
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. It builds on #4051, which saves converters and scorers created through the API. Until the earlier PRs merge, the diff here also shows their changes.
The New Converter dialog only made converters that are gone after the backend restarts, wherever it opens: the Converters tab, the converter panel in chat, and New converter in the technique editor. It now saves them by default, and the Converters and Scorers tables say which entries come back.
base64_converter. Keeping the_converterending stays clear of the names the startup scripts register (base64,rot13), which would stop a saved converter of that name from restoring. The name field checks the backend's rule for saved names. A name the list already has, or one the backend says is taken (409), shows on the field, and focus moves there.role="status"so screen readers read it out, and target options mark Temporary targets.GET /api/converters/typesaddsnot_saveable_reasonto each type, from the checks a save runs, including the one for types that need an uploaded file. The dialog keeps no copy of those rules.Config/TargetOriginBadge.tsxandutils/targetOrigin.tstoRegistry/OriginBadge.tsxandutils/registryOrigin.ts. The target files change imports only.Screenshots: a new converter saves by default, a converter that uses a Temporary target is offered as Temporary with the reason, and the Converters table before and after a restart.
Not covered: