[Pfannkuchensack]

Stacked on #9927 (and through it #9919, #9910, #9899, #9898, #9897, #9894). Until those merge, the diff also shows their commits. This PR's own change is ca2425839e.

Summary

Tiled VAE operations reserved the untiled line at the tile area × 1.25, and vae.flux added analytic full-image terms on top. Fresh-process measurements on an RTX 4090 and an RX 9060 XT show the formula is wrong in both directions:

• Decodes up to 4096 px over-reserve up to 1.6×.
• Above 4096 px, decodes and encodes under-reserve, down to 0.38×. The decoded tiles plus the concatenated image (the assembly) outgrow the tile's own peak.

The tiled estimates now come from measured lines:

• Tiled encode: the untiled line at the tile, clamped per axis, plus a fitted tiled line in the image's pixels.
• Tiled decode: the larger of that and a fitted assembly line.
• Fit: the tiled line is fitted on what tiling adds over the untiled demand measured at the tile, so it carries over to tiles that were not measured. The tiled and assembly lines are fitted together, by a deterministic search over the upper envelopes.
• SD estimate: it now clamps a covering tile to a single pass and resolves tile_size <= 0 to the VAE's own tile, as the nodes do.
• User calibration: a tiled operation prices its tile on the larger of the user's and the shipped untiled line. What tiling adds stays built-in.

Default allocators, measured demand in GiB, and the reservation as a multiple of it (main → this PR):

Operation (tile)	Card	4096 px	8192 px	12288 px
SDXL fp32 decode (512)	RTX 4090	1.63: 1.44 → 1.43	4.94: 0.48 → 1.34	10.10: 1.37
SDXL fp16 decode (1024)	RTX 4090	4.31: 1.32 → 1.10	4.32: 1.32 → 1.25	6.17: 1.05
FLUX.1 decode (1024)	RTX 4090	4.28: 1.40 → 1.11	4.34: 1.67 → 1.26	6.19: 1.07
SDXL fp32 encode (512)	RTX 4090	1.06: 0.97 → 1.09	1.66: 0.62 → 1.12	2.66: 1.14
FLUX.1 decode (1024)	RX 9060 XT	6.66: 1.35 → 1.07	6.98: 1.47 → 1.09	7.78: 1.09
SDXL fp32 decode (512)	RX 9060 XT	1.87: 1.02 → 1.26	3.07: 0.62 → 1.55	6.58: 1.33
SDXL fp32 encode (512)	RX 9060 XT	1.29: 1.01 → 1.08	1.92: 0.68 → 1.09	2.92: 1.12

Related Issues / Discussions

• Builds on feat(working-memory): reserve FLUX.2 denoise working memory from measured lines #9927 and the stack below it.
• Supersedes the tiled pricing of feat(flux2): tile the FLUX.2 VAE encode and decode #9890.

QA Instructions

Measured points

• 737 tiled points and 118 untiled reference points ship in working_memory/data/vae.json. They come from the new scripts/calibrate_vae_tiling_working_memory.py, which runs each operation the way its node runs it, inside the model_on_device window.
• Cards: RTX 4090 (torch 2.13+cu130), native and cudaMallocAsync; RX 9060 XT (torch 2.13+rocm10.0, Windows), expandable segments and native.
• Grid: vae.flux, vae.flux2 and vae.sd (fp16/fp32), decode and encode.
  • Tiles 512 and 1024 at 1024–12288 px.
  • Tiles 256 and 768 at 2048–8192 px.
  • Tile 1536 at 2048–12288 px, RTX 4090 only.
  • Untiled references at every tile side.
• Tile 1536 was first priced as a holdout, outside the fit: 0.98–1.33×. In the fit it reaches 1.06–1.31×.

Bounds

• Every point is covered (the lowest estimate/measured is 1.044).
• Capping points stay ≤ 1.3× or are listed in UPPER_BOUND_EXCEPTIONS with their cause and exact ratio: 16 on the default allocators, 90 under cudaMallocAsync and MIOpen native.
• Points below 512 px (a tile or image the app never sets) bound from below only.

Tests

• uv run --no-sync pytest -n 4 over the working-memory, calibration-service, invocation, builder and calibration-script suites: 4785 passed.
• Ruff is clean.
• Denoise data and reservations are unchanged.
• Mutation checks are each caught. They cover:
  • the old 1.25 margin;
  • adding the assembly line instead of taking the max;
  • the offset taken from the shipped line;
  • a user line ignored above shipped;
  • the small-size rules;
  • the tie-breaks;
  • the margin on a negative intercept.

Review

Resolved findings

• The tiled line did not carry over to unmeasured tiles. The holdout tiles 256 and 768 measured 0.61–0.99×. The tiled line is now fitted over the measured untiled reference, and those tiles are in the fit.
• Extrapolation past 8192 px under-reserved, down to 0.63× at 12288 px. 12288 px is now measured.
• User calibration no longer reached tiled operations. Their tile is now priced on max(user, shipped).
• Smaller fixes:
  • the per-sample batch base of the assembly line;
  • negative SD tile fields;
  • FLUX fp16/fp32 stand-ins that copied SDXL's lines;
  • a builder path that skipped tiled lines silently;
  • duplicate formula tests.

Remaining limits

• One tiled line serves every tile. Under cudaMallocAsync and MIOpen native it over-reserves by up to about 2.35× and 1.72× at some tile and size combinations. That is extra offloading, never a failure.
• Measured to 12288 px.
  • At app tiles the estimates stay above the trend of the last measured segment to at least 15,800 px.
  • Four exceptions reach only about 12,800–15,100 px: two under cudaMallocAsync, one MIOpen-native encode, and FLUX.2 decode at a 1024 px tile on cuDNN native.
  • The upscale tab's 16× scale reaches past that range.
• Unmeasured platforms floor at the per-value maximum of the measured ones. On CPU/XPU/MPS that is up to 1.29×; on Linux ROCm up to 2.5× for encodes, mostly from the untiled stand-in. It is conservative by design until those platforms are measured.
• Tile 1536 is measured on the RTX 4090 only.

Compatibility / Rollout

• vae.sd, vae.flux and vae.flux2 move to estimator version 3. Stored user calibrations of those estimators from an earlier version are shown as unusable and need a new calibration. Neither version has been released yet.
• No API change. vae.json grows to about 1.3 MB (32 KB gzipped).

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need.}

[Pfannkuchensack]

@JPPhoto, on your review of 2026-10-26:

The planner, calibration service and low-VRAM docs belong to #9910; the fixes below reached this PR through the merge 365899b.

3. JPPhoto (review 2026-10-26), blocker: cached non-canceled verdict outlives a cleared retry

Already fixed: 3022a69 caches outcomes per item (a retry is a new item), and f1080e2 records each retry on its run as the queue commits it. A cleared retry is looked up, not found, counted as canceled, and the attempt becomes aborted_attempt. Test (270dc24): test_a_retry_of_a_judged_failed_step_cleared_before_the_next_refresh_aborts_its_attempt, your exact sequence.

4. JPPhoto: low-vram-mode.mdx discard statement and API runs

Agreed. 116e505 limits the sentence to Model Manager calibrations and says an API run without an attempt_id keeps its sizes if stopped. 47168bf names how to remove them (DELETE /api/v2/models/working_memory/runs/{run_id} or a reset in the Model Manager).

5. JPPhoto: _verdicts grows across reset cycles

_verdicts went away in 3022a69; its successor _item_outcomes and _run_files were still not evicted on reset. Fixed in dcf8ced: the three resets drop the removed runs' entries under the fits' lock. Test: test_each_reset_forgets_what_the_service_kept_of_the_runs_it_removed.

Follow-ups in 7787606:

• Outcomes are keyed by run and item id (a recreated database can reuse item ids). Tests: test_a_new_attempt_whose_item_id_an_old_run_recorded_counts_when_judged_after_the_old_run, test_an_old_run_whose_item_id_a_new_attempt_took_is_not_judged_by_the_new_item.
• Every judgement runs under the removal lock. Test: test_every_judgement_of_item_outcomes_holds_the_lock_removals_hold. The write-back race itself is not deterministically testable, since the lock rules out the interleaving.

6. JPPhoto, "probably not fix": PID reuse between communicate() (scripts/family_e2e/graphs.py:119) and kill_group (scripts/family_e2e/server.py:470)

Agreed, no change: the code belongs to the family E2E PRs (#9894/#9899), and reuse needs every group member gone and the PID counter wrapping inside the gap before terminate().

Related, on disk rather than in memory: the removed_run_ids tombstones of keyed runs also grow with every reset; that predates this stack and is filed as #10056.