Re-key grants when their parent's ID changes

.nextchanges/bundles/grants-rekey.md

@@ -0,0 +1 @@
+* Fix grants of a renamed or recreated catalog, schema, volume or other UC resource staying recorded under the old name, which made the next `bundle plan` propose to apply them again. ([#7345](https://github.com/databricks/cli/pull/7345))

acceptance/bundle/resources/grants/schemas/change_name/databricks.yml.tmpl

@@ -0,0 +1,12 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  schemas:
+    schema1:
+      catalog_name: main
+      name: test-schema-$UNIQUE_NAME
+      grants:
+        - principal: deco-test-user@databricks.com
+          privileges:
+            - USE_SCHEMA

acceptance/bundle/resources/grants/schemas/change_name/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = true
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resources/grants/schemas/change_name/output.txt

@@ -0,0 +1,123 @@
+
+=== Deploy a schema with grants
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
+Created schemas.schema1
+Created schemas.schema1.grants
+Files: 2 uploaded, 0 deleted
+Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> print_requests.py //unity-catalog
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/schemas",
+  "body": {
+    "catalog_name": "main",
+    "name": "test-schema-[UNIQUE_NAME]"
+  }
+}
+{
+  "method": "PATCH",
+  "path": "/api/2.1/unity-catalog/permissions/schema/main.test-schema-[UNIQUE_NAME]",
+  "body": {
+    "changes": [
+      {
+        "add": [
+          "USE_SCHEMA"
+        ],
+        "principal": "deco-test-user@databricks.com",
+        "remove": [
+          "ALL_PRIVILEGES"
+        ]
+      }
+    ]
+  }
+}
+
+>>> read_state.py schemas schema1.grants id
+schemas schema1.grants id='schema/main.test-schema-[UNIQUE_NAME]'
+
+=== Rename the schema: its ID changes, and so does the ID of its grants
+>>> update_file.py databricks.yml test-schema- test-schema-renamed-
+
+>>> [CLI] bundle plan --output json
+{
+  "full_name": {
+    "action": "update_id",
+    "reason": "id_changes",
+    "old": "main.test-schema-[UNIQUE_NAME]",
+    "new": "",
+    "remote": "main.test-schema-[UNIQUE_NAME]"
+  }
+}
+
+=== Deploy moves the schema to the new name and applies the grants there
+>>> [CLI] bundle deploy --auto-approve
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
+
+This action will result in the deletion or recreation of the following UC schemas. Any underlying data may be lost:
+  recreate resources.schemas.schema1
+Recreated schemas.schema1
+Updated schemas.schema1.grants
+Files: 1 uploaded, 0 deleted
+Resources: 1 created, 1 changed, 1 deleted, 0 unchanged
+
+>>> print_requests.py //unity-catalog
+{
+  "method": "DELETE",
+  "path": "/api/2.1/unity-catalog/schemas/main.test-schema-[UNIQUE_NAME]",
+  "q": {
+    "force": "true"
+  }
+}
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/schemas",
+  "body": {
+    "catalog_name": "main",
+    "name": "test-schema-renamed-[UNIQUE_NAME]"
+  }
+}
+{
+  "method": "PATCH",
+  "path": "/api/2.1/unity-catalog/permissions/schema/main.test-schema-renamed-[UNIQUE_NAME]",
+  "body": {
+    "changes": [
+      {
+        "add": [
+          "USE_SCHEMA"
+        ],
+        "principal": "deco-test-user@databricks.com",
+        "remove": [
+          "ALL_PRIVILEGES"
+        ]
+      }
+    ]
+  }
+}
+
+>>> [CLI] grants get schema main.test-schema-renamed-[UNIQUE_NAME]
+{
+  "principal": "deco-test-user@databricks.com",
+  "privileges": [
+    "USE_SCHEMA"
+  ]
+}
+
+=== The grants are recorded under the new name, so the next plan is a no-op
+>>> read_state.py schemas schema1.grants id
+schemas schema1.grants id='schema/main.test-schema-renamed-[UNIQUE_NAME]'
+
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 2 unchanged
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.schemas.schema1
+
+This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
+  delete resources.schemas.schema1
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
+
+Destroy: 1 deleted

acceptance/bundle/resources/grants/schemas/change_name/script

@@ -0,0 +1,29 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+NEW_SCHEMA="main.test-schema-renamed-${UNIQUE_NAME}"
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve
+
+    # Consume the destroy's requests so none are left recorded.
+    print_requests.py //unity-catalog &> LOG.destroy_requests
+}
+trap cleanup EXIT
+
+title "Deploy a schema with grants"
+trace $CLI bundle deploy
+trace print_requests.py //unity-catalog
+trace read_state.py schemas schema1.grants id
+
+title "Rename the schema: its ID changes, and so does the ID of its grants"
+trace update_file.py databricks.yml test-schema- test-schema-renamed-
+trace $CLI bundle plan --output json | jq '.plan["resources.schemas.schema1.grants"].changes'
+
+title "Deploy moves the schema to the new name and applies the grants there"
+trace $CLI bundle deploy --auto-approve
+trace print_requests.py //unity-catalog
+trace $CLI grants get schema "$NEW_SCHEMA" | jq '.privilege_assignments[] | select(.principal == "deco-test-user@databricks.com")'
+
+title "The grants are recorded under the new name, so the next plan is a no-op"
+trace read_state.py schemas schema1.grants id
+trace $CLI bundle plan

acceptance/bundle/resources/grants/volume_change_name/databricks.yml.tmpl

@@ -0,0 +1,17 @@
+bundle:
+  name: test-bundle-$UNIQUE_NAME
+
+resources:
+  schemas:
+    schema1:
+      catalog_name: main
+      name: test-schema-$UNIQUE_NAME
+  volumes:
+    volume1:
+      catalog_name: main
+      schema_name: ${resources.schemas.schema1.name}
+      name: myvolume
+      grants:
+        - principal: deco-test-user@databricks.com
+          privileges:
+            - READ_VOLUME

acceptance/bundle/resources/grants/volume_change_name/out.test.toml

@@ -0,0 +1,2 @@
+Cloud = true
+EnvMatrix.DMS = ["", "true"]

acceptance/bundle/resources/grants/volume_change_name/output.txt

@@ -0,0 +1,131 @@
+
+=== Deploy a volume with grants
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
+Created schemas.schema1
+Created volumes.volume1
+Created volumes.volume1.grants
+Files: 2 uploaded, 0 deleted
+Resources: 3 created, 0 changed, 0 deleted, 0 unchanged
+
+>>> print_requests.py //unity-catalog
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/schemas",
+  "body": {
+    "catalog_name": "main",
+    "name": "test-schema-[UNIQUE_NAME]"
+  }
+}
+{
+  "method": "POST",
+  "path": "/api/2.1/unity-catalog/volumes",
+  "body": {
+    "catalog_name": "main",
+    "name": "myvolume",
+    "schema_name": "test-schema-[UNIQUE_NAME]",
+    "volume_type": "MANAGED"
+  }
+}
+{
+  "method": "PATCH",
+  "path": "/api/2.1/unity-catalog/permissions/volume/main.test-schema-[UNIQUE_NAME].myvolume",
+  "body": {
+    "changes": [
+      {
+        "add": [
+          "READ_VOLUME"
+        ],
+        "principal": "deco-test-user@databricks.com",
+        "remove": [
+          "ALL_PRIVILEGES"
+        ]
+      }
+    ]
+  }
+}
+
+>>> read_state.py volumes volume1.grants id
+volumes volume1.grants id='volume/main.test-schema-[UNIQUE_NAME].myvolume'
+
+=== Rename the volume: its ID changes, and so does the ID of its grants
+>>> update_file.py databricks.yml myvolume myvolume_renamed
+
+>>> [CLI] bundle plan --output json
+{
+  "full_name": {
+    "action": "update_id",
+    "reason": "id_changes",
+    "old": "main.test-schema-[UNIQUE_NAME].myvolume",
+    "new": "",
+    "remote": "main.test-schema-[UNIQUE_NAME].myvolume"
+  }
+}
+
+=== Deploy renames the volume and applies the grants under the new name
+>>> [CLI] bundle deploy
+Uploading bundle files to /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default/files...
+Updated volumes.volume1
+Updated volumes.volume1.grants
+Files: 1 uploaded, 0 deleted
+Resources: 0 created, 2 changed, 0 deleted, 1 unchanged
+
+>>> print_requests.py //unity-catalog
+{
+  "method": "PATCH",
+  "path": "/api/2.1/unity-catalog/volumes/main.test-schema-[UNIQUE_NAME].myvolume",
+  "body": {
+    "comment": "",
+    "new_name": "myvolume_renamed"
+  }
+}
+{
+  "method": "PATCH",
+  "path": "/api/2.1/unity-catalog/permissions/volume/main.test-schema-[UNIQUE_NAME].myvolume_renamed",
+  "body": {
+    "changes": [
+      {
+        "add": [
+          "READ_VOLUME"
+        ],
+        "principal": "deco-test-user@databricks.com",
+        "remove": [
+          "ALL_PRIVILEGES"
+        ]
+      }
+    ]
+  }
+}
+
+>>> [CLI] grants get volume main.test-schema-[UNIQUE_NAME].myvolume_renamed
+{
+  "principal": "deco-test-user@databricks.com",
+  "privileges": [
+    "READ_VOLUME"
+  ]
+}
+
+=== The grants are recorded under the new name, so the next plan is a no-op
+>>> read_state.py volumes volume1.grants id
+volumes volume1.grants id='volume/main.test-schema-[UNIQUE_NAME].myvolume_renamed'
+
+>>> [CLI] bundle plan
+Plan: 0 to add, 0 to change, 0 to delete, 3 unchanged
+
+>>> [CLI] bundle destroy --auto-approve
+The following resources will be deleted:
+  delete resources.schemas.schema1
+  delete resources.volumes.volume1
+
+This action will result in the deletion of the following UC schemas. Any underlying data may be lost:
+  delete resources.schemas.schema1
+
+This action will result in the deletion of the following volumes.
+For managed volumes, the files stored in the volume are also deleted from your
+cloud tenant within 30 days. For external volumes, the metadata about the volume
+is removed from the catalog, but the underlying files are not deleted:
+  delete resources.volumes.volume1
+
+All files and directories at the following location will be deleted: /Workspace/Users/[USERNAME]/.bundle/test-bundle-[UNIQUE_NAME]/default
+
+Destroy: 2 deleted

acceptance/bundle/resources/grants/volume_change_name/script

@@ -0,0 +1,29 @@
+envsubst < databricks.yml.tmpl > databricks.yml
+
+NEW_VOLUME="main.test-schema-${UNIQUE_NAME}.myvolume_renamed"
+
+cleanup() {
+    trace $CLI bundle destroy --auto-approve
+
+    # Consume the destroy's requests so none are left recorded.
+    print_requests.py //unity-catalog &> LOG.destroy_requests
+}
+trap cleanup EXIT
+
+title "Deploy a volume with grants"
+trace $CLI bundle deploy
+trace print_requests.py //unity-catalog
+trace read_state.py volumes volume1.grants id
+
+title "Rename the volume: its ID changes, and so does the ID of its grants"
+trace update_file.py databricks.yml myvolume myvolume_renamed
+trace $CLI bundle plan --output json | jq '.plan["resources.volumes.volume1.grants"].changes'
+
+title "Deploy renames the volume and applies the grants under the new name"
+trace $CLI bundle deploy
+trace print_requests.py //unity-catalog
+trace $CLI grants get volume "$NEW_VOLUME" | jq '.privilege_assignments[] | select(.principal == "deco-test-user@databricks.com")'
+
+title "The grants are recorded under the new name, so the next plan is a no-op"
+trace read_state.py volumes volume1.grants id
+trace $CLI bundle plan

bundle/direct/dresources/README.md

@@ -17,7 +17,7 @@
 
 Each field with special plan/deploy behavior must be declared in the YAML file of its resource type, under `configs/`. Both files of a resource type are optional and are omitted when they would be empty:
 
- - `configs/<resource_type>.yml` (e.g. `configs/jobs.yml`) — hand-written rules.
+ - `configs/<resource_type>.yml` (e.g. `configs/jobs.yml`) — hand-written rules. A file named after a sub-resource (`configs/grants.yml`) applies to that sub-resource of every parent.
  - `configs/<resource_type>.generated.yml` — API field behaviors from the OpenAPI schema. Generated by `./task generate-direct-resources`, do not edit.
 
 Choose the right category: