FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) by richlundeen · Pull Request #4130 · microsoft/PyRIT

Richard Lundeen (richlundeen) · 2026-11-10T06:28:32Z

Part -2 of a 10-layer ASSERT stack. Design and how to use it: ASSERT in PyRIT.

	PR
-1	#4132	Judge dimensions, presets and shared techniques
-2	#4130	Trace evidence (OpenTelemetry)	this PR
-3	#4131	Headline metrics
-4	#4133	Staged generation (SDK)
-5	#4134	Generation jobs (backend)
-6	#4135	Generate from the GUI
-7	#4144	In-process agents (docs)
-8	#4180	Scenario-aware dataset compatibility
-9	Planned	Generated-category result groups
-10	Planned	Background dataset loading

Each PR targets main and includes the earlier layers, so its commit list and full diff also contain them. Review this layer's own changes in Files changed from -1 to -2. -8 still has prototype commits in its history, so reviewers of later layers should also compare source between layers rather than reading commit lists.

Description

ASSERT judges can now read what the application did, not only what it said. Tool calls with their arguments and results, model calls, and agent steps are captured with OpenTelemetry and shown to the judge after the turns they produced. Each step has an ID the judge can cite, as ASSERT does with traced targets.

Setup: three pieces turn it on for an in-process agent:

capture = LocalTraceCapture()      # capture.instrument() enables installed OpenInference instrumentors
agent = MyAgentTarget(...)         # a PromptTarget subclass passing trace_config=capture.trace_config()
AssertPolicy(trace_source=capture.source())   # or capture.register(name="my_agent") + trace_source="my_agent"

Evidence and capture:

Evidence model: a new execution_trace observation payload stores normalized ExecutionSteps with bounded content. Each field is cut to 4,000 characters and stored with the full value's SHA-256 digest; a redactor hook or retain_content=False is available. It is persisted with the score, so no migration is needed.
Normalization: OtelExecutionTraceSource reads OpenInference spans first and falls back to GenAI semantic conventions.
Local capture: LocalTraceCapture bundles a provider with no SDK limits and an in-memory store. It keeps only target-request traces, completes each trace once all its spans end, and withdraws completion if a span starts later. When full, it evicts the oldest complete trace, or the oldest other trace, which then reports trace_evicted. Capture failures stay scoped to their trace.

Judging:

Rendering: given a trace_source, a conversation scorer renders fenced, indented EXECUTION TRACE blocks after the turns they produced, with step IDs like [T3]. Coverage is shown in a header, and citations are recorded in score metadata. Trace content cannot forge transcript fences.
One judge per dimension: AssertTraceEvidence(source, requirement=...), passed through build_assert_judge_scorers(..., trace_evidence=...) from [BREAKING] FEAT: ASSERT judge dimensions and presets (-1) #4132, gives each ASSERT dimension judge the trace and execution-evidence instructions, including custom true/false, graded, and not-applicable dimensions, and inside the samples vote. Samples share one trace acquisition per scoring call.
Evidence requirement: TraceEvidenceRequirement is none by default, which always judges with whatever trace exists, as ASSERT does. available and complete return an undetermined score without calling the judge when that evidence is missing. The score's trace_coverage metadata records what the judge saw.
Scenario: foundry.assert_policy accepts trace_source and trace_requirement. Non-default values are kept in the scenario identity, so saved launches resume with them. A target that records no traces gets a warning and is not modified, mirroring ASSERT. trace_requirement other than none without a trace source is rejected.
Custom scorer ownership: with [BREAKING] FEAT: ASSERT judge dimensions and presets (-1) #4132's AssertPolicy(objective_scorer=...), the supplied scorer owns its judge target and evidence configuration. Combining it with a constructor trace_source, a trace_source parameter, or a non-default trace_requirement raises an error instead of silently ignoring the setting. To judge with traces through a custom scorer, configure the trace source on that scorer, for example with create_conversation_scorer(..., trace_source=...).

Demo: from doc/code/scoring/8_trace_evidence_scoring, with a live judge. The agent replies "Done! I booked flight PY450 to Seattle, within your $500 budget." Its trace shows book_flight charged $900.

Judge input	Policy violation	Rationale
Transcript only	False	—
Transcript + execution trace	True	"...in the execution trace, the assistant booked flight PY900, which cost $900, surpassing the user's stated budget without their consent. This violates the 'budget_overrun' category..."

Follow-ups:

Phoenix trace client.
Deduplicating stored trace copies across dimensions.
A client for traces exported by remote agents (e.g., an OTLP collector).

GUI behavior and scope

Own GUI changes: no trace controls are added. #4132's ASSERT launch form uses default scoring and hides scorer-owned settings; this PR adds trace_source and trace_requirement to that hidden set, so the form never renders or sends them. One visible change is new: opening a saved ASSERT configuration with a non-default trace_source or trace_requirement now shows #4132's warning, "This saved ASSERT configuration has custom judge settings. Use the SDK to preserve them; this form uses default scoring.", and no estimate or preview is requested with those settings. The saved configuration itself is left unchanged. Default values (trace_source unset, trace_requirement none) launch normally. The scenario description shown on the form gains a sentence about SDK/CLI trace sources.

Shared/non-ASSERT impact: none. The hidden-setting rule applies only to foundry.assert_policy. Other scanners are unchanged, including the ordinary objective-scorer control on technique.execute. The trace-aware conversation scorer is reusable from the SDK but adds no controls elsewhere.

Unchanged/deferred: trace configuration stays SDK-, CLI- and API-owned, including the trace_source and trace_requirement scenario parameters, their scenario identity, and backend resume of saved trace settings. There is no trace viewer, capture or instrumentation control panel, or automatic target instrumentation in the GUI. A GUI trace-source picker is deferred.

Screenshots: isolated synthetic fixture (local targets, generated ASSERT groups, disposable storage, non-local requests blocked), 1440×1050. Predecessor is #4132 at f696a86f7881740f648d6c83b10787990bc09cf7; candidate is this PR at 06ccdf2a211340d4b1a5b8b7c8be2f5db3dcdf95.

State	Predecessor `f696a86f7`	This PR `06ccdf2a2`
ASSERT launch parameters, default preset	Same image (SHA-256 `d0235561…a48f9`); no trace fields	Same image; no trace fields
Saved preset with `trace_source` and `trace_requirement: available`	—	Warning shown; no estimate requested
Ordinary `technique.execute` launch form	Objective scorer control present	Unchanged

This PR at 06ccdf2a211340d4b1a5b8b7c8be2f5db3dcdf95: a saved preset with trace_source: synthetic_trace_source and trace_requirement: available shows the custom judge settings warning. The preset is kept as saved.

This PR at 06ccdf2a211340d4b1a5b8b7c8be2f5db3dcdf95: ASSERT launch parameters with the default preset; no trace fields. Byte-identical to the predecessor capture at f696a86f7881740f648d6c83b10787990bc09cf7.

This PR at 06ccdf2a211340d4b1a5b8b7c8be2f5db3dcdf95: the ordinary technique.execute form still offers its objective scorer control.

Earlier screenshot (historical): the previous version of this PR showed trace_source and trace_requirement fields on the launch form of published combined c58b389fd8aa67880acd2dec7b77b20df726a3ce. That no longer describes this PR: archived screenshot.

Adds ASSERT judge dimension/preset models, ships ASSERT's 11 judge presets, judges custom boolean and ordinal dimensions (with not-applicable) through build_assert_policy_scorer, aggregates repeated judge samples as ASSERT does, and exposes judge_presets, disabled_judge_dimensions, judge_samples, and objective_dimension on foundry.assert_policy. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ge presets to built-in names - objective_dimension and other non-default judge settings are recorded in scenario identity params, so resume detects changes and saved GUI/API launches restore them. Default identities are unchanged. - Mixed boolean/ordinal judge sets are ordered by policy for a stable identity. - judge_presets accepts only shipped preset names; preset files move to the Python-only judge_preset_files (list[Path]) parameter, which the API guard rejects. Preset load errors no longer echo file contents. - Adds a Scenario._get_identity_params hook. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot

🟡 Changes recommended

Critical synchronization and OTLP parsing flaws can send incomplete evidence to the strict judge, while failed traces can exhaust storage and block subsequent captures.

4 open issues

What changed in this PR

This second ASSERT-stack PR adds OpenTelemetry execution evidence to PyRIT’s conversation judging, allowing judges to evaluate recorded actions alongside assistant replies.

Changes:

Adds local trace capture, OTLP/JSON ingestion, and bounded execution-evidence models.
Introduces evidence-dependent fallback behavior and shared trace acquisition across judge samples.
Documents trace-enabled scoring and adds coverage for capture, normalization, fallback, and saved-launch settings.

File	Description
`third_party/assert-provenance.json`	Records provenance for ASSERT judge adaptations.
`THIRD_PARTY_NOTICES.txt`	Extends attribution to judge presets.
`tests/unit/score/test_otlp_json_trace_client.py`	Tests OTLP parsing and completeness reporting.
`tests/unit/score/test_otel_execution_trace_source.py`	Tests normalization, redaction, and content limits.
`tests/unit/score/test_local_trace_capture.py`	Tests capture lifecycle, instrumentation, and eviction.
`tests/unit/score/test_fallback_scorer.py`	Tests evidence-triggered fallback and policy propagation.
`tests/unit/score/test_assert_judge_presets.py`	Tests preset loading and dimension resolution.
`tests/unit/models/test_execution_trace.py`	Tests execution-evidence validation and serialization.
`tests/unit/models/test_assert_judge.py`	Tests judge configuration and ordinal models.
`tests/unit/backend/test_assert_policy_resume.py`	Tests restoration of judge and trace settings.
`pyrit/score/true_false/true_false_fallback_scorer.py`	Exposes configurable fallback triggers.
`pyrit/score/scorer.py`	Adds the unmet-evidence metadata contract.
`pyrit/score/observation/trace_client.py`	Adds per-trace failures and completed-trace eviction.
`pyrit/score/observation/otlp_json_trace_client.py`	Reads exported traces with settle-window coverage.
`pyrit/score/observation/otel_trace_source.py`	Shares trace querying and scope validation helpers.
`pyrit/score/observation/otel_span_exporter.py`	Captures events, names, and status details.
`pyrit/score/observation/message_trace_resolver.py`	Resolves conversation requests to trace links.
`pyrit/score/observation/local_trace_capture.py`	Bundles local tracing, instrumentation, and completion tracking.
`pyrit/score/observation/execution.py`	Shares acquisitions and resolves execution snapshots.
`pyrit/score/observation/__init__.py`	Exports trace capture and observation APIs.
`pyrit/score/float_scale/float_scale_fallback_scorer.py`	Exposes fallback triggers for float scores.
`pyrit/score/fallback_scorer.py`	Implements evidence-aware fallback and metadata propagation.
`pyrit/score/assert_sampled_scorer.py`	Aggregates repeated ASSERT judgments.
`pyrit/score/assert_judge_presets.py`	Loads presets and resolves enabled dimensions.
`pyrit/score/__init__.py`	Exports judge and execution-trace APIs.
`pyrit/scenario/core/scenario.py`	Adds an overridable identity-parameter hook.
`pyrit/prompt_target/common/target_trace_config.py`	Adds conversation identifiers to request spans.
`pyrit/prompt_target/common/prompt_target.py`	Exposes target tracing configuration.
`pyrit/models/score/trace.py`	Defines execution steps, content, events, and summaries.
`pyrit/models/score/observation.py`	Adds validated execution-trace observation payloads.
`pyrit/models/score/assert_judge.py`	Defines typed judge dimensions, scales, and presets.
`pyrit/models/score/__init__.py`	Exports execution-evidence models.
`pyrit/models/__init__.py`	Exports judge and trace models publicly.
`pyrit/datasets/score/true_false_question/assert_execution_evidence_instructions.yaml`	Defines trace interpretation and citation instructions.
`pyrit/datasets/score/assert/judge_presets/tool-use.yaml`	Adds tool-use judging dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-extended.yaml`	Adds harm and privacy dimensions.
`pyrit/datasets/score/assert/judge_presets/safety-core.yaml`	References built-in safety dimensions.
`pyrit/datasets/score/assert/judge_presets/robustness.yaml`	Adds injection and operational-boundary dimensions.
`pyrit/datasets/score/assert/judge_presets/policy-adherence.yaml`	Adds policy-grounding evaluation.
`pyrit/datasets/score/assert/judge_presets/operational.yaml`	Adds workflow and escalation dimensions.
`pyrit/datasets/score/assert/judge_presets/multi-turn.yaml`	Adds context-retention and goal-drift dimensions.
`pyrit/datasets/score/assert/judge_presets/instruction-following.yaml`	Adds constraint and completion dimensions.
`pyrit/datasets/score/assert/judge_presets/grounding.yaml`	Adds hallucination and attribution dimensions.
`pyrit/datasets/score/assert/judge_presets/communication.yaml`	Adds communication-quality evaluation.
`pyrit/datasets/score/assert/judge_presets/alignment.yaml`	Adds sycophancy and verification dimensions.
`pyrit/datasets/score/assert/dimension_system_prompt.yaml`	Defines the custom-dimension judge prompt.
`doc/scanner/foundry.py`	Documents judge settings and trace-enabled scenarios.
`doc/scanner/foundry.ipynb`	Mirrors scenario documentation in notebook form.
`doc/myst.yml`	Adds trace scoring to documentation navigation.
`doc/code/scoring/8_trace_evidence_scoring.py`	Demonstrates trace-grounded judging and capture setup.
`doc/code/scoring/1_true_false_scorers.py`	Documents custom dimensions and repeated judgments.
`doc/code/scoring/1_true_false_scorers.ipynb`	Mirrors updated ASSERT scoring guidance.
`doc/code/scoring/0_scoring.py`	Links to execution-trace scoring guidance.
`doc/code/scoring/0_scoring.ipynb`	Adds the corresponding notebook documentation link.
`doc/code/framework.md`	Describes trace evidence responsibilities and integration.

_{Review effort: Balanced}

…converters - Custom judge dimensions are boolean bad-event flags judged by SelfAskTrueFalseScorer with an ASSERT dimension template; ASSERT's ordinal scale and allow_not_applicable options are rejected. - Repeated samples use TrueFalseCompositeScorer over the repeated judge with an ASSERT vote (ties count as failures, undetermined samples excluded, category re-vote with override warning), replacing the bespoke sampled scorers. - Public exports trimmed to AssertJudgeDimension, AssertJudgePreset, AssertSampleMetadata, and the preset list/load functions. - foundry.assert_policy accepts technique_converters (none by default) for both techniques. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- AssertTechnique is removed. The technique class is built from the AttackTechniqueRegistry, with red_teaming as the default (ASSERT's scenario case). - The baseline, now on by default, sends each group's stored system and user prompts (ASSERT's prompt case). - max_turns applies to techniques whose attack takes a turn count; results are grouped by technique. - AssertPolicy.VERSION is 2, so runs from version 1 do not resume. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

- allow_not_applicable: a not-applicable answer is undetermined with assert_not_applicable metadata; repeated samples exclude it and keep the marker only when every sample is not applicable. - ASSERT's ordinal scale is accepted with a PyRIT violation_at grade. The judge's grade position is thresholded by FloatScaleThresholdScorer, so every dimension still yields a true/false verdict, and the grade is kept in assert_grade metadata. Graded dimensions can be the objective. - AssertDimensionMetadata is exported from pyrit.score. Default identities are unchanged. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Roman Lutz (romanlutz)

Reviewed the incremental diff vs #4132 (richlundeen-assert-custom-judge-dimensions...richlundeen-assert-otel-judging-plan, +3,969/−122). Overall this is a well-designed slice: the fence-forging defenses are real and tested, redaction runs before hashing/truncation, and the per-trace failure/eviction semantics in InMemoryTraceClient are a clear improvement. All 61 new unit tests pass locally.

One real gap: AssertPolicy(trace_requirement=...) without any trace source is silently ignored (while still entering the scenario identity), unlike create_conversation_scorer, which raises for the same misconfiguration. Details inline, plus a question about score reuse after a process restart and a few smaller notes.

Adds a scenario-local assert_package technique to foundry.assert_policy: PyRIT's RedTeamingAttack whose tester uses ASSERT's own tester system prompt (pinned 04b713fe) filled with each group's stored tester instructions. It is the default technique alongside the baseline; red_teaming stays selectable. Groups without tester instructions are skipped or rejected per incompatible_technique_policy. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Resolve judge presets in a worker thread instead of the synchronous validator. Drop a judge sample whose judgment stays invalid after retries (scoring fails only when every sample does), as ASSERT does. Require native multi-turn targets for turn-taking techniques in foundry.assert_policy, keep turn settings a registered technique sets itself, pass the adversarial chat only to techniques that use one, and warn once per even sample count. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Add AssertDimensionResult, AssertCategoryJudgment (with AssertNodeJudgment) and AssertSampleVotes in pyrit.models as the only readers and writers of the ASSERT score metadata keys, keeping the stored key format. Missing keys are unset and malformed present data raises ValueError. The scenario judge wrapper now records the assert_dimension marker on judged and skipped scores. The custom-dimension judge renders its request from a YAML SeedPrompt, and both ASSERT response handlers validate and convert the decoded response through a new JsonSchemaResponseHandler._convert_response hook instead of parsing twice. AssertDimensionMetadata and AssertSampleMetadata are removed. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ence ASSERT judges can read what the application did, not only what it said: tool calls with arguments and results, model calls, and agent steps captured with OpenTelemetry, rendered after the turns they produced with citable step IDs. - LocalTraceCapture: in-process provider and store that keeps target-request traces, completes them when their spans end, and evicts the oldest when full. - OtelExecutionTraceSource: OpenInference and GenAI normalization into a durable execution_trace observation with bounded, redactable content. - ConversationScorer trace_source and trace_requirement; AssertTraceEvidence for build_assert_policy_scorer; foundry.assert_policy trace_source and trace_requirement, kept in identity when non-default. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Withdraw completion when a span starts after it; export and completion share the capture lock; evict any trace when full with explicit loss markers and bounded bookkeeping; render steps in parent cycles; remove the unreachable unattributed block; require a trace source for trace_requirement; document strict snapshot matching and citation behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…mplate additional_instructions is an explicit dimension_system_prompt.yaml parameter rendered in the sandbox instead of appended after rendering; the prompt is unchanged when it is empty. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Brings in microsoft#3920. No textual conflicts. Updates the assert_package resume test for main's earlier rejection of changed scenario role references, which now reports that the run does not match the current configuration. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…nto trace evidence

…ng in the launch form AssertPolicy(objective_scorer=...) uses a supplied true/false scorer unchanged and rejects mixing it with judge_target, a different scoring_target, or non-default legacy judge settings. Default per-policy judge construction moves into the scorer package as build_assert_judge_scorers. The ASSERT launch form hides the objective-scorer picker and ASSERT judge settings in the form, request and preview, and rejects saved custom scoring instead of replaying it as defaults. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…ts readiness Default ASSERT judges resolve their target through Scenario._resolve_scoring_target, so the explicit scoring_target run parameter takes precedence over judge_target, and AssertPolicy.get_readiness binds that target or reports it missing. A supplied objective scorer needs no default judge target. The ASSERT launch form also treats a carried scoring_target as scorer-owned, and compares saved judge settings to their display defaults by type: an empty list is the default only for list settings, and an empty or list value for target_system_prompt is rejected. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Merges the published trace evidence layer (7bf65f3) into the reshaped judge dimensions layer. Trace evidence now flows through build_assert_judge_scorers, a supplied objective scorer also owns trace configuration, and the launch form hides and rejects non-default trace settings with the other ASSERT scoring settings. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 18:30

Copilot AI balanced review requested due to automatic review settings November 10, 2026 06:28

Copilot started reviewing on behalf of Richard Lundeen (richlundeen) November 10, 2026 06:29

View session

Copilot AI reviewed Nov 10, 2026

View reviewed changes

Comment thread pyrit/score/observation/local_trace_capture.py Resolved

Comment thread pyrit/score/observation/local_trace_capture.py Resolved

Comment thread pyrit/score/observation/otlp_json_trace_client.py Resolved

Comment thread pyrit/score/observation/trace_client.py Resolved

Richard Lundeen (richlundeen) marked this pull request as draft November 10, 2026 06:38

Richard Lundeen (richlundeen) and others added 2 commits November 9, 2026 23:04

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 94a3b56 to 018f41c Compare November 10, 2026 07:56

Roman Lutz (romanlutz) self-assigned this Nov 10, 2026

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 018f41c to d04ae3c Compare November 10, 2026 08:50

Roman Lutz (romanlutz) reviewed Nov 10, 2026

View reviewed changes

Richard Lundeen (richlundeen) mentioned this pull request Nov 10, 2026

DOC: In-process agents as custom targets (-7) #4144

Draft

Richard Lundeen (richlundeen) and others added 2 commits November 10, 2026 13:10

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 5e3676b to 5851ead Compare November 10, 2026 22:02

Richard Lundeen (richlundeen) and others added 6 commits November 10, 2026 16:03

FIX: Order ASSERT trace steps deterministically when start times tie

053dd5c

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

FIX: Version ASSERT trace rendering for cyclic steps

e2c965b

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) force-pushed the richlundeen-assert-otel-judging-plan branch from 9b9900e to 4bb2163 Compare November 11, 2026 00:19

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Scenario-aware dataset compatibility (-8) #4180

Draft

Richard Lundeen (richlundeen) and others added 10 commits November 10, 2026 17:42

MAINT: Merge refreshed ASSERT judge dimensions layer (main 1dc1d5c) i…

9e697fd

…nto trace evidence

MAINT: merge main c3cff18 into ASSERT judge dimensions (-1)

0012d16

Brings in microsoft#4187. No conflicts or integration changes. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Merge refreshed ASSERT judge dimensions layer (main c3cff18) i…

7bf65f3

…nto trace evidence

DOC: note default ASSERT scoring in the launch form and the supplied …

2804b41

…objective scorer Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: make ASSERT judgment metadata helpers keyword-only

f696a86

Satisfies main's check-keyword-only-params gate. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

MAINT: Exempt the OpenTelemetry span processor callback from keyword-…

06ccdf2

…only parameters Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Richard Lundeen (richlundeen) mentioned this pull request Nov 11, 2026

FEAT: Group ASSERT results by generated category #4226

Draft

Navigation Menu

Repository navigation

FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2) - #4130

FEAT: Trace-grounded ASSERT judging with OpenTelemetry (-2)#4130
Richard Lundeen (richlundeen) wants to merge 23 commits into
microsoft:mainfrom
richlundeen:richlundeen-assert-otel-judging-plan

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

Richard Lundeen (richlundeen) commented Nov 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

GUI behavior and scope

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Roman Lutz (romanlutz) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Richard Lundeen (richlundeen) commented Nov 10, 2026 •

edited

Loading