Skip to content

FEAT: Save targets created through the API (part 8) - #4026

Open
varunj-msft wants to merge 233 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Targets
Open

varunj-msft wants to merge 233 commits into
microsoft:mainfrom
varunj-msft:varunj-msft/13178-Save-Targets

Conversation

@varunj-msft

@varunj-msft varunj-msft commented Nov 7, 2026 •

Copy link
Copy Markdown
Contributor

Description

Part 8 of saving Registry entries across restarts. Stacked on part 7 (#4025); part 9 (#4027) builds on it. This PR's own changes: varunj-msft/PyRIT@varunj-msft/13178-Refuse-Saved-Target-Keys...varunj-msft/13178-Save-Targets

Design: 4.1 What gets saved, 1. The contract. Where every part fits: PR map.

This is one of the changes for saving Registry entries across restarts. It builds on #4025, which refuses saved targets that would authenticate with a key. Until the earlier PRs merge, the diff here also shows their changes.

Problem. A target created through POST /api/targets is gone when the backend restarts. Restore (#3909) brings saved entries back at every start, but nothing saves one yet, and the target list doesn't say which targets will come back.

Change.

  • POST /api/targets takes "persist": true, which needs register left on and a name that starts with a lowercase letter and has at most 64 lowercase letters, digits and underscores (otherwise 422). The target is built as before, saved as a recipe under registry_recipes_source (by default ~/.pyrit/registry_recipes, so no Azure is needed) with the signed-in user recorded as the one who saved it, and then registered. If registering fails, the file is deleted again; if that delete fails too, the request gets 503 and the server log names the target and its folder, so the file can be removed.
  • Before saving, an OpenAI target's empty headers is set to "{}", so a saved one doesn't send this server's OPENAI_ADDITIONAL_REQUEST_HEADERS. Then the request goes through the same check restore uses, so a target that would hold or use a key, or leave an address to this server's settings, gets 400, and the message says it can still be created without saving it. A saved target can use saved targets and targets from configuration, which keep their own sign-in, but not a Temporary target, such as a round-robin member that won't come back after a restart.
  • A name that a registered target or a saved file already has gets 409. The file is written create-only, so two saves of one name save one target. Storage that can't be written gets 503 and the target isn't registered. The server log names the target, its folder and the error type, in case the write reached storage anyway.
  • A target type that a scan's launch scripts added, or put in place of another class under an existing name, can't be saved (400): restore couldn't rebuild it after a restart, and with the backend staying not ready until every saved entry restores, that file would keep it down.
  • Each listed target has an origin: saved, configuration (registered by the server's startup scripts), or temporary (gone after a restart or reinitialization, such as a target created without persist or registered by a scan's launch scripts).
  • The seeded browser end-to-end runs, and the backend end-to-end runs when they start their own backend, point registry_recipes_source at a new temp folder, so they don't read or write the targets saved on the machine running them.

What stays the same. Creating a target without persist works as before, and permission checks are unchanged and come first, so a non-admin whose target would use a server credential still gets 403. A save runs inside the create request, and apply already reports busy while a request is in flight, so no new lock is needed. Restore is unchanged, except that it first marks the targets the startup scripts registered as configuration.

Not covered.

Tests and Documentation

  • tests/unit/backend/test_target_service.py: an Entra ID target is saved and restore rebuilds it; a no-key target is saved without an owner when sign-in is off; a target that would hold or use a key isn't saved or registered; a taken name is a conflict only when saving; a saved file with the name is a conflict and stays as it was; the file is deleted when the target can't be registered; storage failures are reported as unavailable, and the log names the target, its folder and the error type without the error's text; two saves of one name save one target; a round-robin is saved only over targets that come back after a restart; a class registered under another name is saved so restore finds it; a failed save that can't be undone is reported as storage trouble; a reserved name is refused as invalid, not as taken; the list shows where each target comes from.
  • tests/unit/backend/test_api_routes.py: a save conflict maps to 409 and a storage failure to 503; the saved-entries source and the signed-in owner (or none when sign-in is off) reach the service; persist needs register and a valid name.
  • tests/unit/backend/test_mappers.py: the origin of each registered target, and on the instance.
  • tests/unit/backend/test_runtime_lifecycle.py: startup marks the targets its scripts register but not saved ones; a save in flight makes apply report busy.
  • tests/unit/registry/test_target_registry.py: a class a launch script adds, or puts in place of another under an existing name, is temporary.
  • tests/unit/registry/test_instance_registry.py and test_registry_recipe_storage.py: startup entries are marked and keep the mark when replaced; a recipe is created once, lists back with its owner and version, and is deleted once.
  • Removing each guard in turn fails at least one of these tests.
  • uv run --frozen --extra all python -m pytest -n 8 --dist=loadfile tests/unit/registry tests/unit/backend tests/unit/prompt_target tests/unit/models tests/unit/setup tests/unit/converter tests/unit/score: 19,488 passed, 9 skipped, 4 xfailed
  • Frontend npm run lint and npm run type-check (for playwright.config.ts): passed.
  • E2E_SEEDED_MODE=true npx playwright test --project=seeded --workers=1: 108 passed, with the backend started from the generated temp config.
  • pre-commit (ruff, ty) on the changed files: passed.
  • Run against a real backend with no environment files, so no server keys: an Entra ID target on an Azure OpenAI endpoint saves (201, origin saved); the same name gets 409; a typed key, an empty key and an endpoint with ?sig= get 400 without echoing the key; exactly one recipe file is written, with no key. After a restart the saved target is back and answers a chat message, and the Temporary one is gone.
  • pyrit/backend/README.md and doc/getting_started/pyrit_conf.md: how to save a target, the name rule, what it can hold, the headers default, the OpenAI SDK settings that still apply, and the status codes.

No JupyText changes.

…API input

External construction now rejects a parameter whose type includes Path unless the
calling service passes the name in allowed_path_params after making the value safe.
The converter service allows its upload parameters once they are stored, and the
target service allows the server-set upload folder and the GitHubCopilotTarget
working directory. The scorer-only check is removed, and the "Valid parameters"
list leaves out path parameters callers cannot set.
The target catalog now applies the path rule to required parameters too, so a
target that needs a disallowed path is left out instead of listed without the
field, matching the scorer catalog. allowed_path_params is typed frozenset[str]
so a bare string cannot turn the allow-list into a substring match.
The target catalog still lists a target whose required path parameter is its
server-set upload folder when the server has one, and leaves it out otherwise.
# Conflicts:
#	tests/unit/backend/test_converter_service.py
# Conflicts:
#	pyrit/prompt_target/azure_blob_storage_target.py
The runtime status keeps its short summary. The restore log now also lists each field's full location
and message, escaped, without the stored values, and with URL credentials removed. Credentials are
removed before a location is shortened, so the summary can't keep part of one either.
# Conflicts:
#	pyrit/registry/registry_recipe_storage.py
Its text can carry a SAS URL. The 503 handler no longer logs it as a debug traceback, and the scenario
suite service no longer attaches it to its warning; both log the storage message, which has no credentials.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants