Skip to content

Close Nebius SDKs and lease cached backends - #4460

Merged
r4victor merged 19 commits into
masterfrom
issue_4454_nebius_sdk_leak
Nov 4, 2026
Merged

r4victor merged 19 commits into
masterfrom
issue_4454_nebius_sdk_leak

Conversation

@r4victor

@r4victor r4victor commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Closes #4454. Nebius SDK clients were never closed, so every backend instance leaked an event loop thread, an executor and the objects they kept alive.

  • Use the Nebius SDK's sync interface and its own runtime instead of dstack's DaemonEventLoop, which is removed.
  • Add Compute.close() and Backend.close(): Nebius closes its SDK with sync_close(), and the Nebius configurator closes the SDK it creates for validation.
  • Use cached backends through leases (lease_project_backends(), lease_project_backend_by_type(), lease_project_backend_by_id()): a backend whose config changes, that is deleted, or that is not leased for 5 minutes is retired, and a periodic close_unused_backends() task closes retired backends once no lease holds them.
  • Split services/backends into cache.py for cached backends and store.py for backend models and configs, and move the offer helpers to services/offers.py.
  • Replace set_get_project_backend_with_model_by_type() with set_select_project_backend_with_model_by_type(), whose selector picks one of the leased project backends.
  • Make get_offers_by_requirements() take backends instead of a project, and return only offers from the run plan functions.
  • Terminate a job with a volume error if its backend is not available when attaching volumes, instead of failing the processing with an unexpected exception.

Why the backend cache had to be re-designed

The SDK cannot be closed from a finalizer: sync_close() raises LoopError when called on an SDK loop thread, and a finalizer runs on whichever thread triggers garbage collection. Finalizers also cannot be relied on to run at all, since the SDK retains exceptions whose tracebacks reference their callers, which keeps backends alive. So backends have to be closed explicitly. But the backends cache handed out backends without knowing who still used them: a backend replaced after a config change or evicted could still be in use by a pipeline or an API request. Leases let the cache know when a backend is no longer used, so that it can be closed safely.

dstack-factory needs a matching update for the new import paths, the selector API and DstackBackend.close().

Since nebius 0.5.0, the SDK runs requests on its own event loop thread and its sync methods are safe to call from sync code, so the separate DaemonEventLoop is no longer needed.
A Nebius SDK runs an event loop thread that keeps it alive, so an SDK that is not closed leaks a thread and file descriptors. Part of #4454.
Callers now borrow backends with lease_project_backend*() context managers. The cache retires a backend if its config changes, if it is deleted from the project, or if it is not leased for 5 minutes, and closes it once its last lease is released.
Leases no longer close backends, so closing does not delay callers and cannot be skipped if a lease is cancelled. Retired backends stay in the cache until close_unused_backends() closes them.
Closing backends on shutdown could close a backend still used by an executor thread of a cancelled worker. The process exits anyway, so leave them open.
@r4victor
r4victor merged commit 4178800 into master Nov 4, 2026
29 checks passed
@r4victor
r4victor deleted the issue_4454_nebius_sdk_leak branch November 4, 2026 07:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Nebius backend leaks a thread and file descriptors on every backend cache rebuild

1 participant