Skip to content

fix(models): install and start on drives without hard links - #10007

Merged
joshistoast merged 6 commits into
invoke-ai:mainfrom
Pfannkuchensack:bugfix/install-claim-without-hardlinks
Oct 25, 2026
Merged

joshistoast merged 6 commits into
invoke-ai:mainfrom
Pfannkuchensack:bugfix/install-claim-without-hardlinks

Conversation

@Pfannkuchensack

@Pfannkuchensack Pfannkuchensack commented Oct 24, 2026 •

Copy link
Copy Markdown
Member

Summary

On drives without hard links, such as exFAT/FAT32 (common for external drives on Windows) and many network shares, v7.0.0-alpha.3 cannot install models. Every remote install returns HTTP 500 ("Model install failed to start / Internal server error") because the install staging claim is published with os.link. That call fails on those filesystems: [WinError 1] on Windows, EPERM on Linux. The failed install also leaves a models/tmpinstall_* directory behind. Startup cleanup then claims that directory the same way, so the server stops starting at all (Application startup failed).

Thumbnail repair, gallery archive and restore, and invokeai-db-copy publish files the same way, so they break on those filesystems too.

All of these now go through one helper, invokeai/app/util/no_replace.py. publish_noreplace never replaces an existing file, and it tries three methods in order:

  1. Hard link. Used wherever hard links work, so behavior there is unchanged.
  2. Atomic no-replace rename. renameat2(RENAME_NOREPLACE) on Linux, renamex_np(RENAME_EXCL) on macOS, os.rename on Windows. This is what fixes exFAT and FAT32.
  3. Exclusive create (O_EXCL) plus copy. For mounts that support neither, such as macOS exFAT, NFS, 9p and FUSE.

rename_noreplace is the existing ModelInstallService._rename_noreplace, moved into the helper. invokeai-db-copy loses its racy exists-check plus os.replace fallback.

Related Issues / Discussions

Reported on Discord: "Alpha 3 errors out when I try to install new models." Install requests return 500, and the launcher shows Exception in ASGI application with a traceback through model_install_default.py.

QA Instructions

Reproduced and verified on a real exFAT volume on Linux, with the Invoke root on that volume:

  • v7.0.0-alpha.3:
    • POST /api/v2/models/install?source=stabilityai/sd-vae-ft-mse returns 500, ending in PermissionError: [Errno 1] Operation not permitted from os.link in _claim_install_tmpdir.
    • With a leftover models/tmpinstall_*, startup exits with code 3 (Application startup failed).
  • This branch, same root:
    • The server starts and removes the leftover directory.
    • Both an HF repo install and a URL install return 201.
    • No claim files are left behind.
  • Filesystem behavior on the exFAT volume: os.link fails with EPERM; renameat2(RENAME_NOREPLACE) succeeds and returns EEXIST when the target exists.
  • Focused tests:
    • Ran test_no_replace.py, test_install_staging_ledger.py, test_model_install.py, test_image_files_disk.py, test_gallery_maintenance_service.py and test_database_copy.py; all pass.
    • Each new no-hard-link test fails when its call site is reverted to os.link.
    • The tests simulate the missing filesystem support by patching os.link (EPERM) and rename_noreplace (ENOTSUP). The fallback therefore runs against each CI runner's real rename.
  • Not verified: Windows and macOS were not tested on a real FAT/exFAT volume. invokeai-db-copy is covered only by the helper's tests, because its end-to-end tests need INVOKEAI_TEST_DB_URL.
  • ruff check and ruff format --check pass on the changed files. I left the full suite to CI.

Review

The review led to two changes:

  • The copy fallback was added because some mounts support neither hard links nor no-replace rename. This also removed a regression in invokeai-db-copy on those mounts.
  • The model-manager architecture doc was updated.

Remaining limitations:

  • Copy fallback only: readers can briefly see a partially written file, and a hard kill mid-copy can leave a partial file. Nothing is ever replaced or lost. A partial install claim still counts as held: it is kept as malformed and reported at startup.
  • Gallery archive on Windows FAT/exFAT: it probably still refuses to archive, because its existing identity check includes the file ID and FAT changes that when a file moves to another directory. The source is restored, so nothing is lost. This is unchanged by this PR.

Checklist

  • The PR has a short but descriptive title, suitable for a changelog
  • Meaningful regression coverage added / updated where needed; obsolete tests/code removed
  • Persisted-state and API changes include required migrations / compatibility validation
  • Relevant performance/efficiency opportunities considered; material claims have evidence
  • Material review findings resolved and relevant checks rerun
  • Documentation added / updated (if applicable)
  • Updated What's New copy (if doing a release after this PR)

View with [code]smith Autofix with [code]smith
Need help on this PR? Tag @codesmith-bot with what you need.

Publish install claims, repaired thumbnails, gallery archives and database copies through one helper that never replaces an existing file.
It hard-links where possible, else renames atomically without replacing, else creates exclusively and copies, so exFAT/FAT32 and network drives work.
Before, model installs failed with HTTP 500 there and a leftover staging directory stopped the server from starting.
@github-actions github-actions Bot added python PRs that change python files services PRs that change app services python-tests PRs that change python tests docs PRs that change docs labels Oct 24, 2026
Comment thread invokeai/app/util/no_replace.py Outdated

@joshistoast joshistoast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A few actionable defects in the copy fallback affecting concurrent replacements, file permissions, and thumbnail rollback.

Comment thread invokeai/app/util/no_replace.py Outdated
Comment thread invokeai/app/services/image_files/image_files_disk.py Outdated
… installs

Two tests wrote a resumable staging marker while the installer's startup restore could still scan it; the restore then claimed or installed the directory first, failing them on slow runners.
…copy fallback

A failed copy removes its destination only while it is still the copy it created, and the copy gets the source's permission bits.
Publishing returns the published file's signature, so the thumbnail rollback also removes a copied thumbnail and keeps one rewritten in place.

@joshistoast joshistoast left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

myes great success

@joshistoast
joshistoast merged commit 6af2057 into invoke-ai:main Oct 25, 2026
19 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

docs PRs that change docs python PRs that change python files python-tests PRs that change python tests services PRs that change app services

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants