Repository navigation
Close Nebius SDKs and lease cached backends - #4460
Merged
Merged
Conversation
Since nebius 0.5.0, the SDK runs requests on its own event loop thread and its sync methods are safe to call from sync code, so the separate DaemonEventLoop is no longer needed.
A Nebius SDK runs an event loop thread that keeps it alive, so an SDK that is not closed leaks a thread and file descriptors. Part of #4454.
Callers now borrow backends with lease_project_backend*() context managers. The cache retires a backend if its config changes, if it is deleted from the project, or if it is not leased for 5 minutes, and closes it once its last lease is released.
Leases no longer close backends, so closing does not delay callers and cannot be skipped if a lease is cancelled. Retired backends stay in the cache until close_unused_backends() closes them.
Closing backends on shutdown could close a backend still used by an executor thread of a cancelled worker. The process exits anyway, so leave them open.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4454. Nebius SDK clients were never closed, so every backend instance leaked an event loop thread, an executor and the objects they kept alive.
DaemonEventLoop, which is removed.Compute.close()andBackend.close(): Nebius closes its SDK withsync_close(), and the Nebius configurator closes the SDK it creates for validation.lease_project_backends(),lease_project_backend_by_type(),lease_project_backend_by_id()): a backend whose config changes, that is deleted, or that is not leased for 5 minutes is retired, and a periodicclose_unused_backends()task closes retired backends once no lease holds them.services/backendsintocache.pyfor cached backends andstore.pyfor backend models and configs, and move the offer helpers toservices/offers.py.set_get_project_backend_with_model_by_type()withset_select_project_backend_with_model_by_type(), whose selector picks one of the leased project backends.get_offers_by_requirements()take backends instead of a project, and return only offers from the run plan functions.Why the backend cache had to be re-designed
The SDK cannot be closed from a finalizer:
sync_close()raisesLoopErrorwhen called on an SDK loop thread, and a finalizer runs on whichever thread triggers garbage collection. Finalizers also cannot be relied on to run at all, since the SDK retains exceptions whose tracebacks reference their callers, which keeps backends alive. So backends have to be closed explicitly. But the backends cache handed out backends without knowing who still used them: a backend replaced after a config change or evicted could still be in use by a pipeline or an API request. Leases let the cache know when a backend is no longer used, so that it can be closed safely.dstack-factory needs a matching update for the new import paths, the selector API and
DstackBackend.close().