Repository navigation
Add lifecycle.bind to adopt existing workspace resources at deploy time - #7366
Draft
janniklasrose wants to merge 21 commits into
Draft
janniklasrose wants to merge 21 commits into
janniklasrose wants to merge 21 commits into
Conversation
Squash of the commits on PR #4630 (declarative-import), adapted to the current main: the terraform engine is gone, so bind runs on the only engine and the terraform gating is dropped; bind actions are wired into the per-resource deploy output, plan.CountActions and the deployment history operation types. Config shape at this point: targets.<name>.bind.<type>.<name>.id. The following commits move it under the resource lifecycle block. Co-authored-by: Shreyas Goenka <shreyas.goenka@databricks.com> Co-authored-by: Isaac <no-reply@databricks.com>
A bind block whose resource is not in the resources section was only caught at plan time, by looking the key up in the plan map. State-only keys are in that map as delete entries, so removing a bound resource from config while keeping its bind block passed validation and 'bundle deploy' deleted the bound workspace resource. The agreed behaviour is that delete protection applies for as long as the bind block remains. Move the check into ValidateBindResources (initialize phase): it needs only config, so 'bundle validate' now reports it, and a delete can no longer be planned for a key that still has a bind block. Co-authored-by: Isaac <no-reply@databricks.com>
When a bound resource (in state, bind block present) no longer exists in the workspace, the in-state plan path mapped the 404 to a create. The deploy then created a replacement under a new ID, and every following plan failed with 'already bound to a different ID' until the user edited the bind block. A bind block pins the resource to one ID, so report the deletion and let the user decide: remove the block to create a new resource. Co-authored-by: Isaac <no-reply@databricks.com>
A bind block outside targets was dropped as an unknown field with only a warning, and the deploy then created the resources it named instead of binding the existing ones. Fail config loading with a pointer to targets.<target>.bind instead. Co-authored-by: Isaac <no-reply@databricks.com>
SelectTarget keeps the selected target as a typed snapshot and drops the
targets section from the config tree, so ${var.*} in a bind ID was never
interpolated: the plan tried to read a job with the literal ID
'${var.job_id}'. The generated schema already advertises the variable
pattern for this field.
Add a mutator that resolves bundle, workspace and variable references in
the target's bind IDs after the regular variable resolution, sharing the
${var.x} shorthand rewrite with it.
Co-authored-by: Isaac <no-reply@databricks.com>
- Scope the state ID collision check to the bind's own resource type: IDs
are only unique per type, so a job and an experiment may share one.
- Resolve ${resources.<key>.id} references to a resource being bound from
the bind block instead of deferring them to after apply; the ID is known.
- Reuse the remote state read at plan time in Bind, so dependents such as
permissions no longer trigger a second read of the bound resource.
- Reword the persistent recreate guard for resources already in state: they
are bound, not being bound, and 'make this a bind-only operation' does not
apply. Drop the trailing newline from the error text.
- Mirror the in-state DoUpdate check in first-time bind planning so a
resource without DoUpdate fails at plan time, not at apply time.
- Return a status from handleBindPlan instead of reading the shared error
flag, which is racy under parallel planning.
Co-authored-by: Isaac <no-reply@databricks.com>
Bind is state-less: nothing records that a resource was just unbound, so a bind block still in config re-adopts it on the next deploy. Say so when 'bundle deployment unbind' runs for a key that has a bind block in the selected target. The new pure-bind test also pins the re-bind round trip itself, the pure 'bind' action (the first deploy of a job is always a bind_and_update because the bundle sets deployment metadata on it), and the single remote read of a pure bind. Co-authored-by: Isaac <no-reply@databricks.com>
The command only converts an existing terraform state into a direct state now that the terraform engine is gone; it no longer plans anything, and 'bundle deploy' already migrates automatically with bind blocks present. Rejecting bind blocks here only forced users to remove and re-add them. Also note at the remote-sync call site why it plans without bind blocks. Co-authored-by: Isaac <no-reply@databricks.com>
- int-id: an unquoted (integer) ID, as written in the design doc, binds. - dev-mode: development mode transforms (rename, paused schedule, tags) apply to the adopted resource and are visible in the plan. - block-removed: removing the bind block is the opt-out; the same immutable change then plans a recreate. - rebind-different-id: a bind block for a resource already in state must carry the state ID. Co-authored-by: Isaac <no-reply@databricks.com>
Spell out in the schema descriptions what a bind block does on the first deploy (adopt, and update if the config differs, including target mode transforms), what it guards while it remains (delete, recreate), and what it does not (bundle destroy; no marker in state). The regenerated schema also picks up main's current variable pattern for the bind definitions. Co-authored-by: Isaac <no-reply@databricks.com>
…inter Co-authored-by: Isaac <no-reply@databricks.com>
Bind becomes a lifecycle setting of the resource it adopts:
resources:
jobs:
my_job:
lifecycle:
bind:
id: "12345"
with per-target IDs set through the usual target override of the resource.
This replaces the target-level 'bind' block (design decision DD2 reversed).
The block now lives on the resource, so a whole class of misplacements
cannot happen: a bind for an undefined resource, a root-level or target-level
misplacement that silently created a duplicate, and a stale block whose
resource was removed. Variable references, diagnostic locations and target
overrides come from the regular config pipeline, which removes the dedicated
resolver and validator and the typed target snapshot they worked around.
CalculatePlan derives the bind IDs from the config root again.
Semantics follow prevent_destroy: the guard against recreating the bound
resource holds while the block is in config; removing the resource removes
the block with it and the deploy deletes the resource, which the
removed-from-config test pins. Duplicate-ID detection is now scoped per
resource type, matching the state collision check.
Tests: configs moved to the new shape; orphaned-bind, invalid-resource-type,
bind-permissions and top-level-bind are gone (not expressible); new
target-override test; errors now point at resources.<type>.<name>.lifecycle.bind.id
with file locations.
Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
… test isolation - Regenerate the PyDABs models: Lifecycle gains bind (validate-generated). - List lifecycle.bind as a non-state field for the resources whose state keeps a lifecycle sub-struct (apps, clusters, sql_warehouses, job_runs), like prevent_destroy and depends_on (TestInputSubset). - Give every deploying bind test a per-run bundle name. The deployment-history variants and the OS runners run concurrently against one workspace, so a fixed name shared the root path and failed with 'Failed to acquire deployment lock' or with the other variant's state. - bind-and-resize: create the cluster with the instance pool only; the cloud API rejects node_type_id together with instance_pool_id. - dev-mode: assert only the planned new values; a real workspace reports schedule.pause_status: UNPAUSED where the test server omits it. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Git Bash on Windows rewrites a leading-slash argument as a Windows path
('C:/Program Files/Git/old/storage/path'), so update_file.py did not find the
value and the test failed on every Windows cloud runner. Match the path
without its leading slash instead.
Co-authored-by: Isaac <no-reply@databricks.com>
data_quality_monitors and materialization_sets pick up lifecycle.bind in the reference schema, the required-field validation and the PyDABs models. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Contributor
|
Commit: Regression test reportSelected 20 changed tests (limit=20, 10 not selected) 40 tests: regression=40 unreleased=0 coverage=0 failing=0
TestAccept/bundle/deploy/bind/basic/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): TestAccept/bundle/deploy/bind/basic/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): TestAccept/bundle/deploy/bind/bind-and-resize/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): TestAccept/bundle/deploy/bind/bind-and-resize/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): TestAccept/bundle/deploy/bind/bind-and-update/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/bind-and-update/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): TestAccept/bundle/deploy/bind/block-removed/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/block-removed/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/delete-and-bind-conflict/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/delete-and-bind-conflict/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/destroy-allowed/DMS=: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) TestAccept/bundle/deploy/bind/destroy-allowed/DMS=true: branch ✅ | main (119ba53) ❌ | latest (v1.24.0) ❌main (119ba53): latest (v1.24.0): same output as main (119ba53) 28 more failure details omitted, see the full report in the run. |
Collaborator
✅ Integration test reportCommit: 1af55cb
Top 10 slowest tests (at least 2 minutes):
|
janniklasrose
commented
Nov 5, 2026
Member
Author
|
Pushed the review fixes in 6edb41a6ea.
From a Codex review. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Adds
lifecycle.bindto adopt existing workspace resources through two deployment phases: commit the bindings to remote state, then apply ordinary resource changes. For example:bundle planpreviews both phases without committing state. JSON plan format 3 has separatebindandplanmaps, andbundle deploy --planexecutes both. DMS records binding and resource changes in separate versions.The bind block guards against missing resources, duplicate/conflicting IDs, creation, recreation, and ID-changing updates. Apply rechecks these constraints for saved plans. Removing the block preserves ownership; removing the resource or running
bundle destroycan delete it. Imperative bind/unbind remain available, with notices explaining their interaction with the block.Why
Adoption changes deployment state and must survive a later planning or apply failure. Separating it from resource actions gives retries durable ownership and keeps ordinary updates/resizes in the existing deployment engine.
Tests
1424db5d125049547a3ab3fd8417252292074b14: completion, MLOps Stacks, Git metadata snapshots, and Python source-location snapshots. An intermittent migration upload-count mismatch passes on rerun.Original implementation by Isaac; review, fixes, and two-phase implementation by Codex.