[Pfannkuchensack]

Stacked on #9894 (family E2E tool), #9897 (calibration runs) and #9898 (VAE coefficients per dtype). Until those merge, the diff also shows their commits. This PR's own changes are 7a75080b44 and 9267c3e166; the rest are merges of the three branches.

Summary

Shipped working-memory coefficients can now come from probe runs through the API, without one-off calibration scripts.

• start --calibrate in the family E2E tool
  • Each compiled case goes through POST /api/v2/models/i/{key}/working_memory/calibrate instead of enqueue_batch.
  • The tool follows the items and saves the store export and status per server lifetime under the run's calibration/. After a crash or stop it copies the store file instead.
  • Time and VRAM are marked as calibration runs and are not rated.
  • A 404/422 for one case becomes that case's error record and the run continues; any other error response stops the run.
• scripts/working_memory/build_shipped.py
  • Turns exports from one or more cards into probe_demand points. It takes valid, fit-eligible windows, keeps the largest per card, estimator, platform and features, and keeps the larger value when it meets an existing point.
  • Platform keys come from the recorded environment through platform_for_environment, which shares its rules with platform_for.
  • It refits the data, shows each line change and the spread across repetitions, and flags fragmentation: reserved grows ≥10 % while allocated stays flat.
  • --write refuses any result that the shared bound rule rejects. A raise driven by a fragmentation-flagged point needs --accept-fragmentation.
• Probe points are lower bounds only. They are warm windows in a running server. The estimate must cover them and they may raise a line, but they never set its cap. The fit rates candidate lines only over fresh-process points, and the upper bound (1.3×, listed exceptions, max_ratio) stays with those points, because a cold first run after a server start is the worst case.
• One bound rule: invokeai/backend/working_memory/bounds.py is used by both the data test and the builder.

Related Issues / Discussions

Builds on #9894, #9897 and #9898.

QA Instructions

• Tests: the affected suites (working-memory backend, builder, tests/family_e2e, package data, invocations, the calibration service/runner/router) pass: 3438 passed, 8 skipped, 5 xfailed. Each new or changed file also passes alone.
  • Mutation checks are each caught:
    • the calibrate path falling back to enqueue_batch;
    • a missing availability check;
    • the OS taken from the build machine instead of the recording;
    • merging that lets the newer point win, or keeps both;
    • the bound refusal disabled;
    • first instead of largest window;
    • probe points choosing a line;
    • case refusals treated as fatal or as per-case.
• Hardware, two real calibration runs (SDXL VAE manifests, fp16/fp32, 512–1280², 25 steps) on an RTX 4090 (CUDA, cudaMallocAsync) and an RX 9060 XT (ROCm, Windows). Both completed and exported their stores.
  • The builder on both exports reads 18 points (3 windows not fit-eligible because of partial residency).
  • It flags the RX 9060 XT SDXL fp32 1280² decode as fragmentation: across repetitions in one server, reserved rose from 8020 to 13000 MiB while allocated stayed at 8020 MiB.
  • It refuses the resulting refit (k_decode_fp32 5988 → 7661 B/px), which would push fresh-process points over 1.3×. The shipped data is unchanged.

Review

Resolved findings

• --write could write data that failed the data test. The upper-bound check only looked at new points, and every new point is a probe point. Now one shared bound rule runs over all points after a refit.
• The builder skipped the lower bound where an analytic term decides; that check now runs.
• Lower-bound-only was enforced after the fit instead of in it; the fit now enforces it.
• A rebuild could replace a shipped point with a lower one. Merging now keeps the larger value.
• A single case refusal ended the whole run, which resume could not get past.
• An idempotency key the tool never reused has been removed.
• The os hard fact moved into feat(working-memory): measure working memory in admin calibration runs #9897, so the environment-id scheme ships once.
• Tests were added for the availability check, the recorded OS, report history, stop, and the probe-wiring regression.

Remaining limits

• Calibration runs work on Windows and on Linux with NVIDIA GPUs (NVML), like the tool; a --calibrate run on 2× RTX PRO 6000 Blackwell under Linux gave 24 valid NVML windows and build_shipped.py read the export. Other machines' exports come from GET /api/v2/models/working_memory/export.
• Warm-up windows count toward the maximum, so a cold first window can dominate.
• fingerprint.os_family and platform._host_os repeat the same sys.platform mapping; a test catches drift between them.

Compatibility / Rollout

• No shipped values change in this PR.
• The new point method probe_demand holds lower bounds only.
• MeasuredPoint gains holds_upper_bound; a max_ratio on a probe point is rejected.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[joshistoast]

The shared bound rule and reuse of existing graph compilers are sound choices, but these defects undermine calibration output, recovery safety, and regression evidence. The full diff also affects ordinary inference through cache restoration and VAE reservations; no actionable backend regression was confirmed.

[Pfannkuchensack]

@joshistoast The fixes for your seven points are in f343ae1 (fitter and build) and 4e6f6b7 (family E2E). Replies are on each thread.

Main has since squash-merged #9919, which restructures fit.py and bounds.py into per-estimator lines. cb36b65 merges main and ports the fitter fix onto those lines. The search and its bounds now work in each line's own units (pixels, tokens for Z-Image, a batch's samples, a tile's 1.25× overlap), and legacy entries stay legacy when their trial fit conflicts with the bounds. The shipped data is unchanged.

One test from #9919 changed. Its 2 GiB token window above 16384 tokens cannot be covered within the 1.3× bound of the fresh 4096-token forward. The old fit returned a line that broke that bound, and the test never checked bounds. It now uses a 512 MiB window, and a new test covers the refusal of the 2 GiB case.

[joshistoast]

found a regression

[joshistoast]

started from the bottom now we here