Skip to content

feat: add structured reporting, verification, and benchmarking workflows for agents - #4920

Open
zewenli98 wants to merge 1 commit into
mainfrom
evanli/agentic-torch-trt
Open

zewenli98 wants to merge 1 commit into
mainfrom
evanli/agentic-torch-trt

Conversation

@zewenli98

Copy link
Copy Markdown
Collaborator

Description

This PR adds structured, machine-readable reporting and verification capabilities for Dynamo workflows:

  • Introduces analyze() for preflight converter support, partitioning, coverage, and configuration checks without building engines. (dryrun=True is the existing human-oriented debugging mode that skips TensorRT engine conversion and prints or saves a textual partitioning summary, while analyze() is a programmatic preflight API that returns a typed, JSON-serializable AnalysisReport with node eligibility, assignments, fallback reasons, partitions, coverage, checks, diagnostics, and status. analyze() also predicts behavior using the requested compilation settings, isolates graph and weight changes, preserves RNG state, supports structured events, and reports failures without building engines.)
  • Introduces compile_with_report() to return the compiled module together with detailed analysis, engine, diagnostic, and timing evidence.
  • Introduces validate() to deterministically compare reference and compiled outputs using explicit structure, dtype, shape, tolerance, and nonfinite-value rules while preserving model, input, aliasing, and RNG state.
  • Introduces benchmark() to produce reproducible wall-clock and CUDA-event latency, percentile, throughput, and speedup measurements under explicit synchronization and timing semantics.
  • Defines versioned JSON contracts and schemas for analysis, compilation, validation, benchmarking, events, environments, and artifact manifests.
  • Emits typed, job-scoped events across analysis, partitioning, conversion, and TensorRT engine builds.
  • Preserves caller graphs, model/input state, RNG state, and existing compiler behavior.
  • Adds CPU/GPU tests, golden schema fixtures, documentation, and CI compatibility checks.

Existing compile() and dryrun APIs remain available with unchanged return types and behavior.

This provides the dependency-free Core API layer for the upcoming MCP PR, which will expose these capabilities as policy-controlled MCP tools and add artifact persistence, job orchestration, and lineage tracking outside the core package.

Fixes #4843

Type of change

  • New feature (non-breaking change which adds functionality)
  • This change requires a documentation update

Checklist:

  • My code follows the style guidelines of this project (You can use the linters)
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas and hacks
  • I have made corresponding changes to the documentation
  • I have added tests to verify my fix or my feature
  • New and existing unit tests pass locally with my changes
  • I have added the relevant labels to my PR in so that relevant reviewers are notified

@zewenli98
zewenli98 requested a review from narendasan October 22, 2026 21:49
@zewenli98 zewenli98 self-assigned this Oct 22, 2026
@meta-cla meta-cla Bot added the cla signed label Oct 22, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation component: tests Issues re: Tests component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: build system Issues re: Build system component: api [Python] Issues re: Python API component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths labels Oct 22, 2026
@github-actions
github-actions Bot requested a review from apbose October 22, 2026 21:49
@zewenli98 zewenli98 changed the title feat: add structured reporting and verification workflows for agents feat: add structured reporting, verification, and benchmarking workflows for agents Oct 22, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

cla signed component: api [Python] Issues re: Python API component: build system Issues re: Build system component: conversion Issues re: Conversion stage component: core Issues re: The core compiler component: dynamo Issues relating to the `torch.compile` or `torch._dynamo.export` paths component: tests Issues re: Tests documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant