Skip to content

bundle: add deployment history buffer telemetry - #7232

Open
shreyas-goenka wants to merge 15 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/deployment-history-buffer-telemetry
Open

bundle: add deployment history buffer telemetry#7232
shreyas-goenka wants to merge 15 commits into
mainfrom
shreyas-goenka_data/shreyas-goenka/deployment-history-buffer-telemetry

Conversation

@shreyas-goenka

@shreyas-goenka shreyas-goenka commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Changes

Adds telemetry to measure performance regression and failure rates for uploading operations.

Why

Helps us understand the perf impact of uploading operations and draining the buffer.

@github-actions github-actions Bot added the DABs DABs related issues label Oct 28, 2026
@github-actions

github-actions Bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Commit: 406655c69 | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (119ba53) latest (v1.24.0)
TestAccept/bundle/dms/multiple-resources/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/operation-upload-fails-once/DMS=true ✅ ❌ ❌
TestAccept/bundle/dms/multiple-resources/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌

main (119ba53):

acceptance_test.go:1345: Diff:
        --- bundle/dms/multiple-resources/output.txt
        +++ /tmp/TestAcceptbundledmsmultiple-resourcesDMS=true523605689/002/output.txt
        @@ -13,14 +13,14 @@
         === History telemetry survives version completion and counts all five staged operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 5,
        -  "operation_recording_success_count": 5,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "queued_update_count_at_timeout": 0,
        -  "recording_attempt_failure_count": 0,
        -  "recording_attempt_success_count": 5,
        -  "queue_size_in_range": true
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "queued_update_count_at_timeout": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queue_size_in_range": false
         }
         
         >>> print_requests.py --dms //versions/1/operations --sort --del-body state --oneline

latest (v1.24.0): same output as main (119ba53)

TestAccept/bundle/dms/operation-upload-fails-once/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌

main (119ba53):

acceptance_test.go:1345: Diff:
        --- bundle/dms/operation-upload-fails-once/output.txt
        +++ /tmp/TestAcceptbundledmsoperation-upload-fails-onceDMS=true184547329/002/output.txt
        @@ -71,13 +71,13 @@
         === A failed history upload is counted separately from the queued operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 1,
        -  "operation_recording_success_count": 1,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "recording_attempt_failure_count": 1,
        -  "recording_attempt_success_count": 1,
        -  "queued_update_count_at_timeout": 0
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queued_update_count_at_timeout": null
         }
         
         === The service lists the schema with the failure
        
    acceptance_test.go:1123: 
        LOG.catalog: {
        LOG.catalog:   "catalog_type": "MANAGED_CATALOG",
        LOG.catalog:   "created_at": 1793962991848,
        LOG.catalog:   "created_by": "tester@databricks.com",
        LOG.catalog:   "full_name": "dms_other_dxnj7npjgzffvdnjysfwhcqs5e",
        LOG.catalog:   "metastore_id": "7a0a6ebf-a90d-2564-7a0a-6ebfa90d267b",
        LOG.catalog:   "name": "dms_other_dxnj7npjgzffvdnjysfwhcqs5e",
        LOG.catalog:   "owner": "tester@databricks.com",
        LOG.catalog:   "updated_at": 1793962991848,
        LOG.catalog:   "updated_by": "tester@databricks.com"
        LOG.catalog: }
FAIL

latest (v1.24.0):

acceptance_test.go:1345: Diff:
        --- bundle/dms/operation-upload-fails-once/output.txt
        +++ /tmp/TestAcceptbundledmsoperation-upload-fails-onceDMS=true2576284571/002/output.txt
        @@ -71,13 +71,13 @@
         === A failed history upload is counted separately from the queued operations
         >>> cat [OUTPUT_DIR]/out.requests.txt
         {
        -  "staged_operation_count": 1,
        -  "operation_recording_success_count": 1,
        -  "operation_recording_failure_count": 0,
        -  "operation_recording_pending_count": 0,
        -  "recording_attempt_failure_count": 1,
        -  "recording_attempt_success_count": 1,
        -  "queued_update_count_at_timeout": 0
        +  "staged_operation_count": null,
        +  "operation_recording_success_count": null,
        +  "operation_recording_failure_count": null,
        +  "operation_recording_pending_count": null,
        +  "recording_attempt_failure_count": null,
        +  "recording_attempt_success_count": null,
        +  "queued_update_count_at_timeout": null
         }
         
         === The service lists the schema with the failure
        
    acceptance_test.go:1123: 
        LOG.catalog: {
        LOG.catalog:   "catalog_type": "MANAGED_CATALOG",
        LOG.catalog:   "created_at": 1793962996796,
        LOG.catalog:   "created_by": "tester@databricks.com",
        LOG.catalog:   "full_name": "dms_other_udmfqktf7ndsjg7bfuolkqucym",
        LOG.catalog:   "metastore_id": "7a0a6ec0-cffd-7685-7a0a-6ec0cffd77be",
        LOG.catalog:   "name": "dms_other_udmfqktf7ndsjg7bfuolkqucym",
        LOG.catalog:   "owner": "tester@databricks.com",
        LOG.catalog:   "updated_at": 1793962996796,
        LOG.catalog:   "updated_by": "tester@databricks.com"
        LOG.catalog: }
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Oct 28, 2026 •
edited
Loading

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 406655c

Run: 42206601601

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 273 16 4:34
✅​ aws-windows-latest-4core-16gb 275 14 3:37
✅​ azure linux-2core-8gb 272 16 9:33
✅​ azure-windows-latest-4core-16gb 274 14 3:38
✅​ gcp linux-2core-8gb 273 16 4:19
✅​ gcp-windows-latest-4core-16gb 275 14 3:37
Top 4 slowest tests (at least 2 minutes):
duration env testname
7:42 azure linux-2core-8gb TestFsCpFileToFileFileNotOverwritten/uc-volumes_to_uc-volumes
3:11 aws-windows-latest-4core-16gb TestAccept
3:11 gcp-windows-latest-4core-16gb TestAccept
3:10 azure-windows-latest-4core-16gb TestAccept

@shreyas-goenka
shreyas-goenka marked this pull request as ready for review November 6, 2026 09:49
@shreyas-goenka
shreyas-goenka requested review from a team as code owners November 6, 2026 09:49
@shreyas-goenka
bundle: add deployment history buffer telemetry
285c7d3
@shreyas-goenka
bundle: inline deployment history drain measurements
1facd4b
@shreyas-goenka
bundle: simplify deployment history backlog telemetry
fab76c2
@shreyas-goenka
bundle: record undrained operations only when drain is interrupted
bb27402
@shreyas-goenka
bundle: directly record queue length on drain timeout
e9836b9
@shreyas-goenka
bundle: drop operation buffer telemetry unit tests
28cd255
@shreyas-goenka
bundle: drop deployment state telemetry unit test
ff4a0b1
@shreyas-goenka
bundle: track deployment history recording errors
917de93
@shreyas-goenka
bundle: count history upload successes and errors at the API call
272da3c
@shreyas-goenka
bundle: include total operation count in failed-upload telemetry output
a9daad7
@shreyas-goenka
bundle: report per-operation recording outcomes alongside upload atte…
a02db74 
…mpts
@shreyas-goenka
bundle: use explicit operation, attempt, and queue telemetry names
81ae3a7
@shreyas-goenka
bundle: attach recording state to each buffered update
87dad8f
@shreyas-goenka
bundle: clarify latest operation recording outcome tracking
136fefb
@shreyas-goenka
shreyas-goenka force-pushed the shreyas-goenka_data/shreyas-goenka/deployment-history-buffer-telemetry branch from 7af90d6 to 136fefb Compare November 6, 2026 09:54
pietern
pietern approved these changes Nov 6, 2026
View reviewed changes
Comment thread libs/dms/operation_buffer.go
RecordingAttemptSuccessCount int
OperationRecordingSuccessCount int
OperationRecordingFailureCount int
OperationRecordingPendingCount int

@pietern pietern Nov 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please include a brief comment about each of these.

Comment thread libs/dms/operation_buffer.go
bestEffort: options.BestEffort,
drainTimeout: time.Duration(options.OperationCount) * historyDrainTimeoutPerOperation,
pending: make(map[string]OperationUpdate),
drainTimeout: time.Duration(options.StagedOperationCount) * historyDrainTimeoutPerOperation,

@pietern pietern Nov 6, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unchanged in this PR. Question: how is this timeout used? It is dependent on the total number of operations, so do you assume sequential execution?

@shreyas-goenka shreyas-goenka Nov 6, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and yes.

The operation upload is a single threaded worker. The plan is to add a batch operation API to make this much faster.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pietern pietern pietern approved these changes
@anton-107 anton-107 Awaiting requested review from anton-107 anton-107 is a code owner automatically assigned from databricks/eng-deco-cli
@hectorcast-db hectorcast-db Awaiting requested review from hectorcast-db hectorcast-db is a code owner automatically assigned from databricks/eng-deco-platform

Assignees

No one assigned

Labels

DABs DABs related issues

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@shreyas-goenka @eng-dev-ecosystem-bot @pietern