Repository navigation
Send heartbeats for open DMS deployment versions - #7342
Open
Sankalp-Mittal wants to merge 5 commits into
Open
Sankalp-Mittal wants to merge 5 commits into
Sankalp-Mittal wants to merge 5 commits into
Conversation
DMS marks a version in progress at CreateVersion. If the CLI dies before CompleteVersion, nothing refreshes the version and the UI shows it as deploying forever. The service will expose a per-version last update time, refreshed by UpdateOperation and Heartbeat, and the UI will show a version as interrupted once that time is over two minutes old. The CLI now keeps that time fresh while a version is open. A heartbeat is sent only after 30s with no successful UpdateOperation, so a fast deploy sends none. Heartbeats are best effort: a failed one is retried quickly, then reported as a warning, then tried again at the next interval. It never fails or stops the deploy. Co-authored-by: Isaac <no-reply@databricks.com>
Stop is a context cancel instead of a stop channel, a done channel and a sync.OnceFunc guard. The quick retries after a failed heartbeat are gone: the SDK already retries transient errors inside each call, and the next attempt is at most 30s away against a 2-minute UI threshold. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Sankalp-Mittal
marked this pull request as ready for review
November 5, 2026 15:38
Sankalp-Mittal
requested review from
anton-107,
denik and
shreyas-goenka
and removed request for
anton-107
November 5, 2026 15:38
Collaborator
✅ Integration test reportCommit: 27a6b92
Top 3 slowest tests (at least 2 minutes):
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
DMS marks a version
IN_PROGRESSatCreateVersion. If the CLI dies beforeCompleteVersion, nothing refreshes the version and the UI shows it as deploying forever.Design (meeting notes: Nov 2, Nov 3):
UpdateOperationand byHeartbeat, and exposes it asVersion.update_time.CreateVersionstill rejects while the previous version's 2-minute lease (last write or heartbeat + 2 min) is live and closes an expired one asLEASE_EXPIRED, so writes and heartbeats both extend that lease. No auto-close;deploy.lockstays for now and will be removed in a later update.update_timeis over 2 minutes old. The state is recoverable.What
libs/dms/heartbeat.go: a goroutine per open version. It callsPOST .../versions/{v}/heartbeatonly after 30s with no successfulUpdateOperation, so a fast deploy sends none. A failed heartbeat is reported once as a warning (the deployment may show as interrupted) and tried again at the next interval; the SDK already retries transient errors inside each call. It never fails or stops the deploy.libs/dms/operation_buffer.go: a successful write resets the heartbeat timer.bundle/direct/dstate/state.go: the heartbeat starts with the operation buffer (right afterCreateVersion, so deploy and destroy are both covered) and stops inCompleteVersionbefore the RPC.HeartbeatResponse.expire_timeis ignored.Dependencies
Version.update_timeis exposed. Until it lands, a deploy that writes more often than every 30s sends no heartbeat. Nothing reads the field yet, so this is not a regression.Tests
libs/dms/heartbeat_test.go:testing/synctestwith the real 30s constants: beats while idle, a write skips the beat, a failure warns and tries again next interval,Stop().acceptance/bundle/dmsis unchanged: fixtures finish in under 30s, so no heartbeat is recorded. The fake server already serves the endpoint..nextchangesfragment: DMS is experimental and the change is invisible until the UX PR lands.This pull request and its description were written by Isaac.