Skip to content

Send heartbeats for open DMS deployment versions - #7342

Open
Sankalp-Mittal wants to merge 5 commits into
mainfrom
sankalp-mittal/heartbeating-for-dms
Open

Sankalp-Mittal wants to merge 5 commits into
mainfrom
sankalp-mittal/heartbeating-for-dms

Conversation

@Sankalp-Mittal

@Sankalp-Mittal Sankalp-Mittal commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Why

DMS marks a version IN_PROGRESS at CreateVersion. If the CLI dies before CompleteVersion, nothing refreshes the version and the UI shows it as deploying forever.

Design (meeting notes: Nov 2, Nov 3):

  • The service keeps one last-update time per version, refreshed by every UpdateOperation and by Heartbeat, and exposes it as Version.update_time. CreateVersion still rejects while the previous version's 2-minute lease (last write or heartbeat + 2 min) is live and closes an expired one as LEASE_EXPIRED, so writes and heartbeats both extend that lease. No auto-close; deploy.lock stays for now and will be removed in a later update.
  • The UI shows a version as interrupted once update_time is over 2 minutes old. The state is recoverable.
  • The CLI's job is to keep that time fresh while a version is open. Without heartbeats, a resource that takes over 2 minutes with no state write lets a second deploy take the version and the first one's writes then fail.

What

  • libs/dms/heartbeat.go: a goroutine per open version. It calls POST .../versions/{v}/heartbeat only after 30s with no successful UpdateOperation, so a fast deploy sends none. A failed heartbeat is reported once as a warning (the deployment may show as interrupted) and tried again at the next interval; the SDK already retries transient errors inside each call. It never fails or stops the deploy.
  • libs/dms/operation_buffer.go: a successful write resets the heartbeat timer.
  • bundle/direct/dstate/state.go: the heartbeat starts with the operation buffer (right after CreateVersion, so deploy and destroy are both covered) and stops in CompleteVersion before the RPC.
  • No request shapes change; HeartbeatResponse.expire_time is ignored.

Dependencies

  • Server PR databricks-eng/universe#2983937: writes refresh the version timestamp and Version.update_time is exposed. Until it lands, a deploy that writes more often than every 30s sends no heartbeat. Nothing reads the field yet, so this is not a regression.
  • UX PR (separate): show "interrupted" after 2 minutes. Must land after both.

Tests

  • libs/dms/heartbeat_test.go: testing/synctest with the real 30s constants: beats while idle, a write skips the beat, a failure warns and tries again next interval, Stop().
  • acceptance/bundle/dms is unchanged: fixtures finish in under 30s, so no heartbeat is recorded. The fake server already serves the endpoint.
  • No .nextchanges fragment: DMS is experimental and the change is invisible until the UX PR lands.

This pull request and its description were written by Isaac.

DMS marks a version in progress at CreateVersion. If the CLI dies before
CompleteVersion, nothing refreshes the version and the UI shows it as
deploying forever. The service will expose a per-version last update time,
refreshed by UpdateOperation and Heartbeat, and the UI will show a version
as interrupted once that time is over two minutes old.

The CLI now keeps that time fresh while a version is open. A heartbeat is
sent only after 30s with no successful UpdateOperation, so a fast deploy
sends none. Heartbeats are best effort: a failed one is retried quickly,
then reported as a warning, then tried again at the next interval. It
never fails or stops the deploy.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the DABs DABs related issues label Nov 4, 2026
Sankalp-Mittal and others added 3 commits November 4, 2026 15:24
Stop is a context cancel instead of a stop channel, a done channel and a
sync.OnceFunc guard. The quick retries after a failed heartbeat are gone:
the SDK already retries transient errors inside each call, and the next
attempt is at most 30s away against a 2-minute UI threshold.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@Sankalp-Mittal
Sankalp-Mittal marked this pull request as ready for review November 5, 2026 15:38
@Sankalp-Mittal
Sankalp-Mittal requested review from a team as code owners November 5, 2026 15:38
@Sankalp-Mittal
Sankalp-Mittal requested review from anton-107, denik and shreyas-goenka and removed request for anton-107 November 5, 2026 15:38
Resolves conflicts with #7151 and #7169: StartOperationBuffer takes both
BufferOptions and the heartbeat, and InitializeOperationBuffer starts the
heartbeat alongside the buffer in every history mode.

Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 27a6b92

Run: 42206315238

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 4:43
✅​ aws-windows-latest-4core-16gb 273 13 3:13
✅​ azure linux-2core-8gb 270 15 3:41
✅​ azure-windows-latest-4core-16gb 272 13 3:13
✅​ gcp linux-2core-8gb 271 15 5:15
✅​ gcp-windows-latest-4core-16gb 273 13 3:04
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:08 azure-windows-latest-4core-16gb TestAccept
3:07 aws-windows-latest-4core-16gb TestAccept
2:58 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

DABs DABs related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants