Skip to content

Fix cancellation during Thrift status polling - #533

Open
yb-yu wants to merge 1 commit into
databricks:mainfrom
yb-yu:fix/cancel-inflight-status-poll
Open

yb-yu wants to merge 1 commit into
databricks:mainfrom
yb-yu:fix/cancel-inflight-status-poll

Conversation

@yb-yu

@yb-yu yb-yu commented Nov 6, 2026 •

Copy link
Copy Markdown

Summary

Cancelling the caller context does not interrupt an in-flight Thrift status request because polling uses a detached context. The sentinel invokes that request synchronously, so remote cancellation cannot proceed until it returns. If the client exits first, the warehouse query can continue running until its statement timeout.

This patch uses the caller context for status polling while retaining the detached context for remote cancellation. It also lets the sentinel run its cancellation callback when a status request is interrupted.

Impact

This investigation started with a report that cancelling a dbt job left a warehouse query running until its two-hour statement timeout. I have not confirmed the exact cancellation path in that original run, but reproduced the polling failure described below in controlled tests.

Reproduction

I reproduced this with dbt Fusion on Databricks Serverless using dbt's v1.14.2 driver fork. Cancellation was triggered immediately after a status request started.

  • Before the fix, Fusion exited approximately 3 seconds after SIGINT without sending remote cancellation. The warehouse query continued until its configured 120-second statement timeout.
  • With interruptible polling, remote cancellation succeeded approximately 117 ms after SIGINT, and query history reported CANCELED.

The runtime comparison used the dbt fork. On current upstream main, interrupting the status request alone is insufficient: the resulting error returns before the sentinel's cancellation callback runs. This patch also routes that case through the existing cancellation path.

Verification

Regression tests cover explicit cancellation and deadlines during a blocked status request. They verify that remote cancellation receives a live context, preserves context values and the operation handle, and occurs exactly once. The explicit-cancellation test fails on the unmodified upstream baseline and passes with this patch.

Passed locally:

  • CGO_ENABLED=0 go test ./... -count=1 -timeout=5m
  • go test -race ./internal/backend/thrift -count=1 -timeout=2m
  • make lint — 0 issues
  • CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build ./...

Signed-off-by: Youngbin Yu <contact@ybyu.dev>
@yb-yu
yb-yu force-pushed the fix/cancel-inflight-status-poll branch from 4236e1e to 6a162f4 Compare November 6, 2026 12:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant