Repository navigation
FEAT: Show who saved each converter, scorer and technique (part 15) - #4067
Open
varunj-msft wants to merge 376 commits into
Open
varunj-msft wants to merge 376 commits into
varunj-msft wants to merge 376 commits into
Conversation
…entials The Azure Speech converters and the Azure Content Safety scorer read this server's key when theirs is empty, or else sign in with its Azure identity, and no value turns that off. They now declare that key as a server fallback, so a saved one is refused when it's saved or restored, as saved targets already are. The audio scorers transcribe with this server's Speech settings, and the local refusal classifier scorer can load its model from a folder on this server or download it with this server's Hugging Face sign-in, so those can't be saved. A class that declares no auth modes now supports only api_key in the saved-entry check, as the saveability check already assumed, so a recipe can't pass by adding an auth_mode the class doesn't take.
Only targets can be saved through the API so far. The steps that turn off the server values an entry would read, refuse what it still can't hold or a Temporary entry it names, write its recipe, register it, and delete the recipe again if registering fails move from the target service into a shared module, along with the route helpers for the owner and the not-saveable problem. The origin enum becomes RegistryEntryOrigin, read by registry_entry_origin, so converters and scorers can use the same pieces. Targets behave as before.
POST /api/converters and POST /api/scorers take "persist": true, as POST /api/targets does: the entry is built the way restore builds it, saved under registry_recipes_source with the signed-in user, and restored at every start. An entry that would use a Temporary target is refused, as is one that would use this server's credentials. A converter that would keep an uploaded file, or a copy made with source, can't be saved yet. Each listed converter and scorer carries its origin, and the converters and scorers the startup scripts register are marked as coming from configuration. A saved converter can't be removed through the API yet, since it would come back from its file at the next start.
The converter dialog only made Temporary converters, wherever it opened: the Converters tab, the converter panel in chat, and New converter in the technique editor. It now saves the converter by default under a suggested name such as base64_converter, which keeps clear of the names the startup initializers use. A converter type that can't be saved, a converter with an uploaded file, one that uses a Temporary target, or a save the server refused is offered as Temporary instead, with the reason. A taken name shows on the name field. GET /api/converters/types says why a type can't be saved. The Converters and Scorers tables label each entry Saved, From configuration, or Temporary, and a saved converter's remove button is disabled with the reason. Save to registry on a converter's changed settings says the copy is Temporary. The origin badge and its labels move to shared names.
# Conflicts: # pyrit/backend/services/runtime_lifecycle.py # tests/unit/backend/test_runtime_lifecycle.py
# Conflicts: # pyrit/setup/initializers/tools.py # tests/unit/backend/test_scenario_run_service.py # tests/unit/backend/test_scenario_suite_execution_service.py # tests/unit/registry/test_attack_technique_registry.py # tests/unit/registry/test_instance_registry.py # tests/unit/registry/test_registry_recipe_restore.py
create_named_instance checks the name before it builds, and the technique registry checks names case-insensitively before it registers. Both raised the duplicate-name error without recording the clash, so a launch script that caught either one could still leave a saved entry in place of its own. The name check now records the clash itself, as register already did, and the technique check records it too.
# Conflicts: # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/targets.py # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
# Conflicts: # frontend/src/components/Config/CreateTargetDialog.tsx # pyrit/backend/routes/targets.py # pyrit/backend/services/target_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/routes/targets.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_mappers.py # tests/unit/registry/test_registry_recipe_restore.py
# Conflicts: # tests/unit/registry/test_registry_recipe_rules.py
# Conflicts: # pyrit/backend/mappers/target_mappers.py # pyrit/backend/models/scorers.py # pyrit/backend/routes/converters.py # pyrit/backend/routes/targets.py # pyrit/backend/services/converter_service.py # pyrit/backend/services/scorer_service.py # pyrit/backend/services/target_service.py # pyrit/registry/instance_registry.py # tests/unit/backend/test_api_routes.py # tests/unit/backend/test_converter_service.py # tests/unit/backend/test_mappers.py # tests/unit/backend/test_runtime_lifecycle.py # tests/unit/backend/test_scorer_service.py # tests/unit/backend/test_target_service.py # tests/unit/registry/test_instance_registry.py
main now declares that TextTarget takes no credentials, with an empty supported_auth_modes. The saved-entry check read that as supporting no mode at all, so it refused saved TextTargets and left the saveability check for the target types with no mode to report. A class that declares no auth modes is now checked as the default api_key mode, which it has no key for.
…d to OpenAIChatTarget
The runtime status keeps its short summary. The restore log now also lists each field's full location and message, escaped, without the stored values, and with URL credentials removed. Credentials are removed before a location is shortened, so the summary can't keep part of one either.
# Conflicts: # pyrit/registry/registry_recipe_storage.py
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This is one of the changes for saving Registry entries across restarts. Until the earlier PRs merge, the diff here also shows their changes.
The Targets table already says who saved each target (#4028). Converters, scorers and techniques didn't, so in a shared deployment you couldn't tell your saved entries from someone else's.
ConverterInstance,ScorerInstanceandTechniqueInstanceaddsaved_by, the name of the person who saved the entry when sign-in was on, andsaved_by_me. The server works outsaved_by_mefrom the signed-in caller, so no object ID leaves the server. The list and get routes for the three kinds pass the caller in.What stays the same: with sign-in off, nobody is recorded, so no captions or checkbox appear. Who saved an entry doesn't change who can use it.
Screenshots: the Converters tab with "Saved by you" and "Saved by" captions, and with Only converters I saved checked.
Not covered: