Skip to content

DOC: Make framework.md concise, prescriptive contributor guidance - #4175

Open
Richard Lundeen (richlundeen) wants to merge 12 commits into
microsoft:mainfrom
richlundeen:richlundeen-framework-guidelines
Open

Richard Lundeen (richlundeen) wants to merge 12 commits into
microsoft:mainfrom
richlundeen:richlundeen-framework-guidelines

Conversation

@richlundeen

@richlundeen Richard Lundeen (richlundeen) commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Description

Contributors and coding agents use framework.md to decide where code belongs. This rewrite turns it into a concise, prescriptive north star built on principles and core responsibilities, instead of feature-level implementation detail.

  • Add principles for code, security, docs, tests, and pull requests, including no over-engineering, failing loudly rather than running partially, trusting operators with test data while protecting credentials and server-level powers, and one unit of value per PR
  • Replace the two architecture diagrams with one that shows surfaces, run inputs (objectives, objective target, and scorers), technique configuration, and the shared library
  • Rewrite each component as its responsibility, a few rules, and what it does not own; leave implementation detail to the component guides
  • Keep the backend, frontend, and scenario subclasses thin: new behavior and types go in the library and pyrit.models first, and scenarios let the Scenario base class run them
  • Update Framework Plans to the current code, including moving domain logic out of ChatScoringService, the backend Word finding export, and the frontend conversation export
  • State that the page changes only when maintainers agree to change a principle or responsibility, or when a plan is done
  • Add a Conventions section for patterns that cut across components, such as requirements and capabilities instead of type checks, complete identifiers, metadata value objects, and prompts kept as YAML seeds
  • Add short rule lists to the unit test and notebook guides that link back to the principles, and align the style guide's comment guidance with them

🤖 Generated with Claude Code

Contributors and coding agents use framework.md to decide where code
belongs. Rewrite it around principles and core responsibilities instead
of feature-level detail.

- Add principles for code, docs, and tests, including failing loudly
  rather than running partially
- Replace the two architecture diagrams with one that shows surfaces,
  run inputs, technique configuration, and the shared library
- Keep the backend and frontend thin: new behavior and types belong in
  the library and pyrit.models
- Update the framework plans to match the current code
- Link short docs and unit test rules back to the principles

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Copilot AI balanced review requested due to automatic review settings November 10, 2026 21:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified in the architecture and contributor-guidance documentation changes.

No open issues 🎉

What changed in this PR

Recasts PyRIT’s architecture documentation as concise contributor guidance for deciding where code belongs, with explicit component responsibilities and thin application surfaces.

Changes:

  • Adds shared principles for code, documentation, and tests, and updates component responsibilities and planned refactoring.
  • Consolidates the architecture diagrams into one showing surfaces, run inputs, technique configuration, and shared services.
  • Aligns the notebook, unit-test, and style guides with those principles.
File Description
doc/​contributing/​7_notebooks.md Adds concise documentation-writing rules linked to the shared principles.
doc/​contributing/​5_unit_tests.md Clarifies deterministic, isolated, behavior-focused testing guidance.
doc/​contributing/​3_style_guide.md Prefers self-explanatory code and comments that explain why.
doc/​code/​framework.md Defines contributor principles, component ownership, and architectural plans.
doc/​assets/​pyrit-components.svg Illustrates application surfaces, configuration boundaries, and shared services.

Review effort: Balanced

Mark the page as a north star and limit updates to agreed principle or
responsibility changes and completed plans, so feature work stays in the
component guides.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…runner

Scenario subclasses often rebuild run logic that the Scenario base
class already owns. State that a scenario declares its techniques,
datasets, and scorer, implements _build_atomic_attacks_async, and
leaves the rest of the run to the base class.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@richlundeen
Richard Lundeen (richlundeen) marked this pull request as ready for review November 10, 2026 22:02
State the security model: operators are trusted with test data, so
don't redact what they need, while credentials and server-level powers
stay protected. Make not over-engineering a code principle, and ask for
pull requests that each deliver one unit of value end to end.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Richard Lundeen (richlundeen) added a commit to varunj-msft/PyRIT that referenced this pull request Nov 10, 2026
Treating everything before the last "@" as a user name and password
rewrote a URL such as https://graph.example/users/alex@contoso.com/mail
into one that names the wrong host. Operators need the real URL (microsoft#4175:
don't redact what operators need), and the case it covered, an unescaped
"/", "?" or "#" in a password, is a malformed URL. The linear-time match,
which also finds a URL right after "_", stays.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Ask contributors to match existing patterns, but to do what's best when
a better way exists and converge on it instead of adding variants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
List the patterns components should share instead of local variants:
requirements and capabilities, identifiers and references, defaults,
typed fields and metadata value objects, shared-instance state, prompt
seeds, structured output, media storage, and UTC timestamps. Fold the
Models bullets they replace into the new section.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Drop the conventions bullet about referring to registered components by
name and hash. Memory refers to components by identifier hash while the
registry resolves them by name, and the registry guide already covers
pinned references and recreate_instance.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scope rules that a literal reader could misapply: identifiers leave out
runtime settings and credentials, typed fields are for queried data,
environment reads and file writes are scoped to components and results,
attacks may forward replies, tags select technique subsets, invalid
arguments stay ValueError, and declared skip and adapt policies aren't
fallbacks. Also name the optional scenario hooks and the shared
compatibility checks.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Add a convention that attacks, scorers, converters, and targets receive
the components they use as objects, and leave construction to callers,
initializers, the registry, technique factories, and scenario defaults.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Reframe the pass-in-components convention: a component may build its own
default collaborator, as long as callers can pass an object that
replaces it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Rebuilding has legitimate uses, such as duplicating a component or
building a variant through the registry, and the identifiers convention
already records what actually ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolve the framework.md Datasets conflict with the objective
decomposition work from microsoft#4110: keep the concise guidance, add an
ownership rule (the dataset defines the tree; attacks execute its
paths), and add a Framework Plan for executing decomposition paths.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants