[joshistoast]

Summary

Two legacy capabilities were missing from webv2.

Generate PBR Maps. The Gallery context menu offers "Generate PBR Maps" for a single selected image. It queues one utility graph with a pbr_maps node (tile size 512, no border, non-intermediate output on the gallery's auto-add board), at the front of the queue as the legacy action did, and reports enqueue, completion (refreshing the gallery) and failure. A canceled job is reported as information, not an error.

Image storage administration. Settings → Server gains an admin-only "Image storage" editor: the folder layout for new images (flat, date, type or hash, saved as image_subfolder_strategy alone), the state of the current move or recovery job with remaining count and errors, and Start Move / Recover with confirmation. Each action is enabled only when it has work (images outside the layout, an interrupted job) and states why it is disabled; both lock while a layout save or a status re-read is pending. Status polls only while a job runs and only while the editor is open; 403/409/422 responses surface inline. Non-admins never see the section.

One utility-graph watch serves both the existing canvas utility path and the new submission. It settles on item status events and re-reads the accepted items after a reconnect, a queue clear, or a bulk cancel/delete (queue_items_canceled); an item that disappears from the queue is reported as removed. The PBR submission also takes an abort signal, so an account change detaches its listeners and timer, and re-reads at a fixed interval while the item is still open.

During a move the server answers thumbnail and image-file reads with 409. Gallery thumbnails and the Preview image that fail to load report to one probe per account, which re-requests a single failed URL while the server answers 409 or is unreachable (every 30 s while the tab is visible) and reloads every failed image once it is served again, so every tab, admin or not, recovers once the move ends, however long it runs.

Related Issues / Discussions

Follows up the legacy parity audit behind #9803.

QA Instructions

From invokeai/frontend/webv2 (Node 22.14, pnpm 10.12): pnpm check:release passes. Performance baselines were re-recorded for byte growth only (new locale strings; the PBR graph builder joins the image-actions source set); request counts and chunk sets are unchanged. Two unrelated browser suites each failed once under heavy machine load and passed on rerun.

Follow-up at 29ac7cd: pnpm check:release passes (lint, 10145 unit and 2913 browser tests, fixtures, architecture performance, project-file journeys, accessibility). Verified against the mock backend in Chromium: Start → confirm shows the pending spinner while the request runs and the layout Select is disabled behind the dialog; a 409 from a move another admin started re-reads status and locks the controls; thumbnails that answered 409 during a move load on their own once it ends; a PBR item cleared from the queue is reported as removed.

Follow-up at 6533b77: pnpm check:release passes (lint, 11246 unit and 3427 browser tests, fixtures, architecture performance, project-file journeys, accessibility). Request counts are unchanged; the baselines gain only the two new gallery source owners. Verified against the mock backend in Chromium: idle and has-work enablement with their reasons, Escape returning focus to Start Move, a refused start shown inline with focus on the named section, the running state, and 44 thumbnails broken by a move reloading within 32 s of it ending without navigation.

Mock backend (scripts/mock-backend.mjs, representative profile; /__faults?imageMoves=needs-recovery|instant|failed seeds job states). Like the server, the mock refuses Start Move while any queue item is pending or in progress; the representative profile seeds such items, so clear the queue first (PUT /api/v1/queue/default/clear on the mock origin, or the Queue widget).

1. Launchpad → Preferences → Server → Image storage: with the default layout both actions are disabled with a reason. Change the layout; the status reads that 1000 images sit outside the current layout. Start Move → confirm before clearing the queue: the 409 detail shows inline. Clear the queue, Start Move → confirm: controls disable, after a few polls "All images are stored in the current layout" with the last job listed. With needs-recovery: interrupted-job message, Start Move disabled, Recover enabled and works. As a non-admin the Server section is hidden.
2. Open a project, right-click a gallery image → Generate PBR Maps: a success notification appears and POST /api/v1/queue/default/enqueue_batch is sent with prepend: true. The mock never completes items, so the "ready" notice and gallery refresh need a real backend (first run downloads the PBR models).

Review

Material findings resolved: a PBR submission whose account changed, or whose item failed, during the enqueue left its settlement promise unhandled (now marked handled on creation); the watcher's socket listener and timer survived an account change (abort signal); a pending PBR item that a queue clear deleted left the watch hanging until a timeout (the watcher reconciles on queue_cleared, treats a missing item as removed, and re-arms while the item is still open instead of timing out); the image storage editor let the layout change while a move ran when its status cache was stale (status is re-read after a refusal, the Select locks during a request and until status has loaded, and an accepted move cannot be overwritten by an older status read); a move that finished within the start request never refreshed the gallery; the confirm dialog's pending state never showed; enqueue failures showed raw JSON; the browser timing baselines had been re-recorded (restored to the merge-base values; only bytes and the source set changed). Thumbnails broken by a move did not recover outside the admin's open Settings editor; a shared per-account probe now reloads every failed thumbnail and the Preview image once images are served again.

Second review round: thumbnails gave up after about 75 s, so a longer move left the gallery broken (replaced by the shared probe above, which also covers the Preview image and survives a server restart mid-move); a bulk cancel or delete of the pending PBR item went unnoticed until the 30-minute re-read; the early-settlement tests could not catch a regression of the unhandled-rejection fix (they now cross a macrotask); a PBR run failing after enqueue showed a bare "Error" notice; Start Move and Recover stayed enabled during a layout save and unlocked before the status re-read after a refusal; both were enabled with nothing to do; the confirm dialog returned focus to an unnamed container; the retry revision leaked to the next item in a reused slot; the canvas path still showed raw error JSON; Queue exported a snake_case DTO type; prepend had no wire-level test; the mock did not refuse moves during queue work or guard enqueue and image reads while a job awaited recovery.

Remaining limitations: PATCH /app/runtime_config has no server-side maintenance guard, so another client can still change the layout while a move runs; the editor's lock is client-side. Filmstrip, picker and swipe-neighbour images do not take part in move recovery. A proxy error confined to one image URL keeps the probe asking only for that URL until it changes. A paused queue keeps a PBR watch open (one listener, one timer and one read per interval) until the item finishes or the account changes.

Compatibility / Rollout

No persisted-state or API changes. The mock backend gained the image-move and runtime-config routes for fixtures.

Checklist

• The PR has a short but descriptive title, suitable for a changelog
• Meaningful regression coverage added / updated where needed; obsolete tests/code removed
• Persisted-state and API changes include required migrations / compatibility validation
• Relevant performance/efficiency opportunities considered; material claims have evidence
• Material review findings resolved and relevant checks rerun
• Documentation added / updated (if applicable)
• Updated What's New copy (if doing a release after this PR)

---

^{Need help on this PR? Tag @codesmith-bot with what you need. Autofix is disabled.}

[lstein]

Reviewed at 233e61f. The wire contract matches the backend: field names, enums and nullability against image_moves.py, the partial PATCH /runtime_config (exclude_unset, so sending only image_subfolder_strategy can't overwrite other fields), the pbr_maps node fields, and admin gating (no admin-only request is issued for non-admins). The two blockers below are about lifecycle, not the contract.

Blockers

1. The "unhandled settlement promise" fix is incomplete (features/queue/runtime/utilityGraph.ts, submitUtilityGraph)

settled is marked handled only inside fail(). Two sequences reject it while await enqueue(...) is still pending, before the caller has received the promise:

• The account changes during the enqueue POST. onAbort → finish() → settleReject(aborted).
• A fast failed/canceled queue_item_status_changed event for the origin arrives before the enqueue response. The listener is attached first so that a fast completion is not missed, and that also lets a fast failure through.

In both cases, the enqueue succeeds, so fail() never runs. The microtask checkpoint ends with settled rejected and no handler, unhandledrejection fires, and loggingRuntime logs app.unhandled-rejection. useImageActions attaches .then only later. The existing tests miss this because the runtime test aborts only after submitUtilityGraph resolves, and the useImageActions test mocks submitUtilityGraph.

Fix: call settled.catch(() => {}) right after creating the promise; callers still see the rejection through their own .then. Add a test that aborts during a deferred enqueue, and one that delivers a failed event before the enqueue resolves. Also, the !isSettled && guard on the timer is currently untested; the abort-during-enqueue test would cover it.

2. The layout can be changed while a move is running (ImageStorageSettings.tsx, imageStorage.ts)

PATCH /runtime_config has no maintenance guard. _get_new_subfolder reads the live config for each batch, so a layout change mid-move leaves the images split between two layouts. The editor is the only thing preventing that, and it relies on isRunning from a status cache that can be stale:

• 409 on Start/Recover: if another admin or tab already started a move, the error path does not refetch status. The cache still says idle and polling is off, so the Select and buttons stay enabled during the move.
• Request in flight: the Select is disabled by layoutQuery.isPending || isSavingLayout || isRunning, not by isActing. POST /start calls count_images_needing_move() (a full scan) before replying, so on a large library you can confirm Start and then change the layout during that window. The reverse also works: change the layout, then Start before the PATCH lands. The two sync routes run concurrently in the thread pool.
• Initial load: while status is loading, isRunning defaults to false, so the Select is enabled before the editor knows whether a move is running.
• Stale read overwrites the POST result: postImageMove calls setQueryData without cancelling in-flight fetches. A status GET that started earlier (for example from the layout save's invalidateQueries, or a focus refetch) can resolve after the POST and overwrite "running" with "idle". Polling then stops mid-move and Start re-enables.

Fix: refetch or invalidate status in the action error path, disable the Select on isActing and until status has loaded, and call cancelQueries before setQueryData. A maintenance guard on the backend PATCH would be the sturdier fix, but it is out of scope here.

Non-blocking

3. Gallery refresh after a move only happens while the editor is mounted. During a move, is_maintenance_active() makes the image, board and board-image routes answer 409, so the gallery ends up in an error state. The running→idle transition inside the status queryFn is the only thing that recovers it, and there is no socket event when a move ends. If an admin closes Settings mid-move, polling stops. After gcTime, reopening has no previous data, so the refresh never fires. Other tabs and non-admins are never told the move ended. The PR description calls this a limitation, but it leaves the gallery broken, not just stale.
4. A fast move can skip the refresh. _start_background_operation computes is_running after executor.submit, so a tiny move can already be finished when the 202 goes out. postImageMove stores isRunning: false, polling never starts, no running→idle transition is ever seen, and any gallery request that got a 409 during the move is never retried. postImageMove could invalidate the gallery when the accepted status is already not running.
5. A queue clear leaves the PBR watch hanging. clear() emits status events only for items that are running, then deletes the rest. A pending, prepended PBR item disappears without a terminal event. Nothing settles the watch until the 30-minute timeout, where the 404 read is swallowed and the watch ends with a silent timeout. Likewise, an item still pending or running at 30 minutes (paused queue, first model download) that finishes later gets no "ready" notice and no gallery refresh. Consider treating a 404 from the reconcile read as a removal and telling the user.
6. The Start/Recover loading spinner never shows. confirmAction returns void, so ConfirmDialog awaits nothing and immediately runs onClose → setPendingAction(null). isActing && pendingAction === 'start' is therefore never true, and the dialog's own pending state is skipped too. Returning the runAction promise from confirmAction would fix both.
7. The enqueue-failure toast shows raw JSON. ApiError.message is the raw response body (assertOk), and submitUtilityGraph wraps it unchanged, so users see {"detail":"…"}. Use getApiErrorMessage(error.cause ?? error, …). The useImageActions test rejects with new Error('queue paused'), which hides this.
8. Re-recorded timing baselines. In browser-baseline.json, editor longestTaskMaxMs went 180→343, 259→365 and 170→308, routeReadyMedianMs rose 20–27%, and Launchpad (barely touched here) regressed 8–15%. timingPolicy.enforce is false today, but these numbers become the reference once it is enforced. Please keep the byte and source-set deltas and restore the timing fields. The PR's other claims hold: request counts and chunk sets are unchanged, and the only source-set change is pbrMapsGraph.ts.
9. Smaller items:
   • An inconsistent enqueue response reports failure but does not cancel the accepted item; runUtilityGraph does cancel in the same case.
   • Several new @features/gallery/storage exports have no consumers: isTerminalImageMoveJobState, the ImageMoveJob* / ImageMoveOperation types, and getImageMoveStatus / imageStorageKeys outside tests.
   • Removing the abort listener (signal.removeEventListener) is untested. Without it, each PBR click leaves a closure on the account-lifetime signal.
   • The mock backend diverges from the real one in ways that hide items 2–4:
     • It treats an error job as needing recovery; the real _FINISHED_JOB counts it as finished.
     • It never makes image routes return 409 during a move.
     • It always returns is_running: true from the POST.

[Pfannkuchensack]

Reviewed at 9710608. Paths are relative to invokeai/frontend/webv2/ unless they start with invokeai/ or docs/.

From lstein's earlier review: blocker 1 is resolved (settled.catch at src/features/queue/runtime/utilityGraph.ts:330). Blocker 2 is resolved except two short windows (minor 1 and 2). Items 4, 6, 7, 8 and the first bullet of 9 are resolved. Items 3 and 5 are only partly resolved (M1, M2).

Should fix before merge

M1. Thumbnails stop retrying after about 75 s, so a longer move leaves the gallery broken (src/features/gallery/data/queryCache.ts:69, :116; src/features/gallery/ui/GalleryThumbnail.tsx)

• The delays are 3+6+12+24+30 s; after that a tile never retries while it stays mounted. <img onError> carries no status, so a 409 during a move is treated like a missing file.
• A move journals and moves every image and thumbnail. On a library of thousands of images it plausibly runs for minutes (not timed).
• Scenario: a non-admin tab, or an admin with Settings closed, has the gallery open when a 3-minute move starts. Tiles that fail in the first ~75 s exhaust their retries and stay broken after the move ends, until they remount. Full-size/viewer images (get_image_full is guarded too) have no retry at all.
• This contradicts "every tab, admin or not, recovers once the move ends" and leaves item 3 open for longer moves. GalleryLoadFailures.browser.test.tsx only covers one 409 followed by success at 3 s.
• Fix: replace the per-tile cap with one shared, de-duplicated probe. When tiles fail, a module-level poller re-requests one failed URL with fetch (so the status is visible), keeps going while it answers 409 (e.g. every 15–30 s, uncapped while the document is visible) and calls refreshGalleryThumbnails on success; a 404 stops it. That also removes the synchronized per-tile bursts (minor 4). At minimum, narrow the claim in the PR and the docs.

M2. A bulk cancel or delete of the pending PBR item goes unnoticed for 30 minutes (src/features/queue/runtime/utilityGraph.ts:418-430)

• The watch listens to queue_item_status_changed, queue_cleared and reconnects. The bulk operations (cancel_by_destination, cancel_all_except_current, cancel_by_queue_id, delete_by_destination, delete_all_except_current) cancel waiting items in one statement and emit only queue_items_canceled (invokeai/app/services/session_queue/session_queue_default.py:594-602, 663-735). Per-item status events go out only for in-progress items.
• Scenario: a batch is generating, the user clicks Generate PBR Maps (prepended, so pending), then "Cancel all except current" in the queue. The PBR item is canceled without an event the watch hears; about 30 minutes later the interval read finds it, and a "canceled" (or "removed") notice appears out of nowhere.
• This is the same defect class as item 5.
• Fix: also subscribe to queue_items_canceled and call reconcileAcceptedItems(), or settle directly when canceled_item_ids contains an accepted id. Add a test that emits it.

M3. The tests for item 1 don't catch a regression of the fix (src/features/queue/runtime/utilityGraph.test.ts:1341-1384)

• Both "settles quietly…" tests yield only await Promise.resolve() between the early rejection and attaching the handler, so the rejection and the handler land in the same microtask drain and nothing is reported as unhandled.
• Moving settled.catch(() => {}) back into fail() (the pre-fix code) passes all 68 tests. With await new Promise((r) => setTimeout(r, 0)) instead, the pre-fix code fails with an unhandled failed rejection and the fix passes.
• In production the account change or the fast failure happens while the enqueue POST is pending, which crosses macrotasks, so that is the path the guard is for.
• Fix: yield a real macrotask in both tests. The abort test uses fake timers, so it needs real timers there or an explicit unhandledrejection spy.

M4. A PBR run that fails after enqueue shows a bare "Error" notice (src/workbench/image-actions/useImageActions.ts:983)

• Settlement failures go through recordError, whose title is the generic 'Error', with only the backend message: on a first run typically a model download or out-of-memory error, possibly minutes after the click. The user can't tell which action failed.
• pbrMaps.failed ("Could not generate PBR maps") exists but is used only for enqueue failures, and the hook test asserts the generic path.
• Fix: notifications.add({ kind: 'error', title: t('…pbrMaps.failed'), message }), keeping the logging.

Minor

1. Start Move and Recover stay enabled while a layout PATCH is in flight (src/workbench/settings/ImageStorageSettings.tsx:121, :172, :182): isBusy omits isSavingLayout. This is the "reverse" race from item 2; one token fixes it.
2. After a 409 refusal the controls unlock before the status refetch lands (src/features/gallery/data/imageStorage.ts:165-168). The invalidation is fire-and-forget, so the Select is usable against the stale "idle" cache until the GET returns, and that GET runs the full count_images_needing_move scan. Await the refetch before rethrowing.
3. The retry nonce carries over to the next item shown in the same slot (GalleryThumbnail.tsx:142, queryCache.ts:109-115). Sparse slots are keyed by index, so after any failure series later items in that slot load with ?gallery_thumbnail_revision=N and miss the long-lived HTTP cache. Reset the nonce when the URL changes.
4. Tiles that fail together retry together, with no jitter (queryCache.ts:116-125).
5. Recover is always enabled (ImageStorageSettings.tsx:182). At idle it is a guaranteed no-op behind a confirmation that takes the server-wide maintenance lock, and it gives no feedback. Start Move is likewise enabled when nothing needs moving. "Last job 1: completed" shows the raw job id.
6. @features/queue/utility exports UtilityItemStatusReader, a snake_case DTO shape (error_message, error_type) with no consumer outside Queue, against the Queue README's DTO mapping rule. SubmitUtilityGraphOptions has no consumer either, and createGalleryThumbnailRetryController on @features/gallery/queries is only used by Gallery's own UI.
7. The canvas path (runUtilityGraph) still maps enqueue failures with error.message, so raw {"detail":…} can still reach users there (utilityGraph.ts:773); only the new path uses getApiErrorMessage. "Now shared between the existing canvas utility path and the new submission" is broader than what is shared (watchUtilityOrigin); the canvas path also gets no queue-clear or reconnect reconciliation.
8. enqueueUtility's prepend has no wire-level test (src/features/queue/data/submissionApi.ts:149): reverting it to false passes every test. The enqueueGenerate/enqueueWorkflow tests show the pattern.
9. Mock fidelity (scripts/mock-backend.mjs): Start Move is refused only under the queue-active fault, while the real server refuses whenever any item is pending or in progress, so in the mock Start Move succeeds right after a PBR click. The mock also doesn't guard enqueue_batch during a move, and its image routes answer 409 only while a job runs, not while one awaits recovery.
10. Focus after the confirm dialog always goes to the section container (finalFocusEl), also on Cancel/Escape, and that container has no role or accessible name.
11. Docs: image-storage-maintenance.mdx says the status refreshes every few seconds, which only happens while Settings is open. The Gallery README documents neither the storage entry nor the 409/retry behavior during a move.
12. Backend follow-up, not for this PR: guard PATCH /app/runtime_config with assert_image_move_maintenance_inactive() when image_subfolder_strategy changes. A mid-move change is recoverable (a second Start Move moves the rest), so it shouldn't block this PR.

Checked in the browser against the mock backend: layout change, Start Move with the controls locked until completion, Recover, and Generate PBR Maps with its notification.

[Pfannkuchensack]

Follow-up at 6533b77. Approving: every finding from my previous review is resolved.

• The shared recovery probe replaces the capped per-tile retry, has no cap while the server answers 409/502–504, and also covers the Preview full image.
• queue_items_canceled settles bulk-canceled or deleted utility items.
• The item-1 tests now wait a real macrotask; moving settled.catch back into fail() now fails them.
• PBR failures carry the "Could not generate PBR maps" title.
• Start Move and Recover follow the job state and say why they are disabled.
• Status is re-read before unlocking.
• The canvas path maps errors with getApiErrorMessage and shares the watch.
• The mock now refuses Start Move on active queue work and guards enqueue_batch during maintenance.

Checked again in the browser against the mock backend.

Remaining notes, none blocking:

• PreviewNavigation.browser.test.tsx "preview image recovery" lets the probe's fetch reach the real Playwright route. A 409 that arrives after the final advanceTimersByTimeAsync(30_000) schedules the next read 15–30 s later, past the 1 s expect.poll. It passed 8/8 here; answering the probe in-page, as GalleryLoadFailures does, would remove the race.
• After a 409 "already running", the red alert stays after that move has ended.
• ImageContextMenu.tsx:889: the "Generate PBR Maps" label is hard-coded English.
• gallery-reference.mdx: the maps also fall back to Uncategorized when the user can't add to the auto-add board.
• reconcileIntervalMs is only set by its own test.
• The Queue README doesn't describe how a submission watch ends: removed, bulk-cancel re-read, an abort that leaves the item alone.
• dependencyPolicy.test.ts has no case for the storage entry.
• The PR description's Review section still says tiles "retry a failed load on a bounded backoff".
• The backend guard on PATCH /app/runtime_config during a move is still a worthwhile follow-up.

Pfannkuchensack has taken over code review of this PR