Skip to content

Remove the experimental gate from databricks pipelines test - #7715

Open
tharunbhuk-db wants to merge 2 commits into
mainfrom
tharun/pipelines-test-remove-gate
Open

tharunbhuk-db wants to merge 2 commits into
mainfrom
tharun/pipelines-test-remove-gate

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Changes

  • databricks pipelines test runs without an environment variable. The command is no longer labeled experimental.
  • A run of all tests sends test_options as an empty object, so a workspace that does not support it returns a clear error.

Tests

  • Unit: TestNewStartTestUpdateRequest pins the request JSON.
  • Acceptance: new pipelines/test/not-enabled; the pipelines/test goldens drop the gate cases.

This pull request and its description were written by Isaac.

The command no longer needs DATABRICKS_CLI_PIPELINES_TEST=true. Whether a
workspace can run pipeline tests is decided by the service, so a run of all
tests now always sends test_options, empty when nothing is selected. Where the
testing API is off, the service rejects such a request with a clear error,
instead of running its older test-only update.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db requested review from a team as code owners November 17, 2026 10:03
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
The gate is removed in #7715.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

github-actions Bot commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 68cfd82a9 | Workflow run

Regression test report

Selected 6 changed tests (limit=20, 0 not selected)

6 tests: regression=6 unreleased=0 coverage=0 failing=0
test branch main (2f5d1a6) latest (v1.26.0)
TestAccept/pipelines/databricks-cli-help ✅ ❌ ❌
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/immutable ✅ ❌ ❌
TestAccept/pipelines/test/not-enabled ✅ ❌ ❌
TestAccept/pipelines/test/selection ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/pipelines/databricks-cli-help: branch ✅ | main (2f5d1a6) ❌ | latest (v1.26.0) ❌

main (2f5d1a6):

acceptance_test.go:1380: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help3242348508/002/output.txt
        @@ -30,7 +30,7 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  Run the tests of a pipeline
        +  test                  *Experimental* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- pipelines/databricks-cli-help/output.txt
        +++ /tmp/TestAcceptpipelinesdatabricks-cli-help4009735205/002/output.txt
        @@ -30,7 +30,7 @@
           run                   Run a pipeline
           show                  Preview a table's columns and sample rows
           stop                  Stop a pipeline
        -  test                  Run the tests of a pipeline
        +  test                  *Preview* Run the tests of a pipeline
         
         Management Commands
           apply-environment     *Public Preview* Apply the latest environment to the pipeline.
        @@ -52,13 +52,13 @@
           update-permissions    Update pipeline permissions.
         
         Flags:
        -  -h, --help[=true|false]   help for pipelines
        -      --var strings         set values for variables defined in project config. Example: --var="foo=bar"
        +  -h, --help          help for pipelines
        +      --var strings   set values for variables defined in project config. Example: --var="foo=bar"
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         Use "databricks pipelines [command] --help" for more information about a command.
TestAccept/pipelines/test/arguments: branch ✅ | main (2f5d1a6) ❌ | latest (v1.26.0) ❌

main (2f5d1a6):

acceptance_test.go:1380: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments3847458836/002/output.txt
        @@ -2,6 +2,8 @@
         >>> [CLI] pipelines test --help
         Run the tests of a pipeline in a test-only update.
         
        +This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +
         KEY is the unique name of the pipeline, as defined in its YAML file.
         It is required when test selections are given. If there is only one pipeline
         in the project and no arguments are given, it is selected automatically.
        @@ -62,52 +64,52 @@
         Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
        -Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Invalid --pytest-arg values fail before any request
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=
        -Error: --pytest-arg requires a non-empty value
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === The pipeline ID must be a pipeline ID
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: --pipeline-id "1234" is not a pipeline ID
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === JSON output is not supported
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
        -Error: --output json is not supported by pipelines test
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Incompatible run-control flags
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
        -Error: --resume cannot be used with test selections
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
        -Error: --resume cannot be used with --pytest-arg
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
        -Error: --resume cannot be used with --no-wait
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
        -Error: --timeout cannot be used with --no-wait
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
        -Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 
        -Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === --junit-xml needs the results
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --junit-xml out.no-wait.xml
        -Error: --junit-xml cannot be used with --no-wait because no results are available when the command returns
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === An invalid flag replaces an older report
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x --junit-xml out.flag-error.xml
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1133
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"unsupported pytest argument &#34;-x&#34;: only -k and -m are supported\">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="unsupported pytest argument &#34;-x&#34;: only -k and -m are supported">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.flag-error.xml vs /tmp/TestAcceptpipelinestestarguments3847458836/001/out.flag-error.xml
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1133
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"--junit-xml cannot be used with --no-wait because no results are available when the command returns\">--junit-xml cannot be used with --no-wait because no results are available when the command returns</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="--junit-xml cannot be used with --no-wait because no results are available when the command returns">--junit-xml cannot be used with --no-wait because no results are available when the command returns</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.no-wait.xml vs /tmp/TestAcceptpipelinestestarguments3847458836/001/out.no-wait.xml

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments1370273841/002/output.txt
        @@ -2,6 +2,8 @@
         >>> [CLI] pipelines test --help
         Run the tests of a pipeline in a test-only update.
         
        +This command is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it.
        +
         KEY is the unique name of the pipeline, as defined in its YAML file.
         It is required when test selections are given. If there is only one pipeline
         in the project and no arguments are given, it is selected automatically.
        @@ -21,40 +23,22 @@
         
           databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
         
        -The command waits for the results. It exits with code 0 only if at least one
        -test was collected and none failed or errored, so a run whose tests were all
        -skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
        -update starts.
        -
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        -
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. The report replaces the file on every exit except a flag that
        -cannot be parsed. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        -
         Usage:
           databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
         Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait[=true|false]     Don't wait for the tests to complete.
               --pipeline-id string       ID of the pipeline to test, instead of a KEY.
               --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
         
         Flags:
        -  -h, --help[=true|false]   help for test
        +  -h, --help   help for test
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
         === pytest arguments must be passed with --pytest-arg
         
        @@ -62,52 +36,196 @@
         Error: -k is a pytest argument: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -- -k orders
        -Error: the -- separator is not supported: pass pytest arguments with --pytest-arg, for example --pytest-arg=-k --pytest-arg='orders and not slow'
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Invalid --pytest-arg values fail before any request
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=
        -Error: --pytest-arg requires a non-empty value
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === The pipeline ID must be a pipeline ID
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234
        -Error: --pipeline-id "1234" is not a pipeline ID
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === JSON output is not supported
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
        -Error: --output json is not supported by pipelines test
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
         === Incompatible run-control flags
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
        -Error: --resume cannot be used with test selections
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
        -Error: --resume cannot be used with --pytest-arg
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
        -Error: --resume cannot be used with --no-wait
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
        -Error: --timeout cannot be used with --no-wait
        +Error: unknown flag: --no-wait
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
        -Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
... (97 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/pipelines/test/immutable: branch ✅ | main (2f5d1a6) ❌ | latest (v1.26.0) ❌

main (2f5d1a6):

acceptance_test.go:1380: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable1183011596/002/output.txt
        @@ -4,53 +4,16 @@
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Relative test paths are rejected with workspace.sync_mode immutable
         
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path
        -
         === Absolute workspace paths and running all tests still work
         
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        -
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- pipelines/test/immutable/output.txt
        +++ /tmp/TestAcceptpipelinestestimmutable3262756109/002/output.txt
        @@ -4,53 +4,34 @@
         Created pipelines.my_pipeline
         Files: 0 uploaded, 0 deleted
         Resources: 2 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Relative test paths are rejected with workspace.sync_mode immutable
         
         >>> musterr [CLI] pipelines test my_pipeline tests/test_orders.py::test_backfill
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Error: cannot resolve test selection "tests/test_orders.py::test_backfill": relative test paths need synced local files, but workspace.sync_mode is immutable; pass the /Workspace path
        -
         === Absolute workspace paths and running all tests still work
         
         >>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline /Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
         
        ->>> [CLI] pipelines test my_pipeline --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Flags:
        +  -h, --help   help for test
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/someone@example.com/tests/test_orders.py::test_backfill"
        -      ]
        -    }
        -  }
        -}
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        +Exit code: 1
TestAccept/pipelines/test/not-enabled: branch ✅ | main (2f5d1a6) ❌ | latest (v1.26.0) ❌

main (2f5d1a6):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\n\n>>> print_requests.py //updates\n{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines/[UUID]/updates\",\n  \"body\": {\n    \"test_only\": true,\n    \"test_options\": {}\n  }\n}\n"
        	            	actual  : "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\n\n>>> print_requests.py //updates\nFile [OUTPUT_DIR]/out.requests.txt not found\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,13 +4,8 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml
        	            	-Error: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.
        	            	+Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        	            	 
        	            	 >>> print_requests.py //updates
        	            	-{
        	            	-  "method": "POST",
        	            	-  "path": "/api/2.0/pipelines/[UUID]/updates",
        	            	-  "body": {
        	            	-    "test_only": true,
        	            	-    "test_options": {}
        	            	-  }
        	            	-}
        	            	+File [OUTPUT_DIR]/out.requests.txt not found
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/output.txt vs /tmp/TestAcceptpipelinestestnot-enabled2269925624/002/output.txt
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1133
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID]\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\">Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"[UUID]\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,3 @@
        	            	     <testcase name="pipelines test">
        	            	-      <error message="Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.">Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.</error>
        	            	+      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	     </testcase>
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/out.not-enabled.xml vs /tmp/TestAcceptpipelinestestnot-enabled2269925624/001/out.not-enabled.xml

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.\n\n>>> print_requests.py //updates\n{\n  \"method\": \"POST\",\n  \"path\": \"/api/2.0/pipelines/[UUID]/updates\",\n  \"body\": {\n    \"test_only\": true,\n    \"test_options\": {}\n  }\n}\n"
        	            	actual  : "\n=== Where the testing API is off, a run of all tests is rejected, not run as an older test-only update\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n\n>>> print_requests.py //updates\nFile [OUTPUT_DIR]/out.requests.txt not found\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,13 +4,26 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.not-enabled.xml
        	            	-Error: Pipeline testing is not enabled for this pipeline. Contact your Databricks account team to join the preview.
        	            	+Error: unknown flag: --junit-xml
        	            	+
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	+
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	 
        	            	 >>> print_requests.py //updates
        	            	-{
        	            	-  "method": "POST",
        	            	-  "path": "/api/2.0/pipelines/[UUID]/updates",
        	            	-  "body": {
        	            	-    "test_only": true,
        	            	-    "test_options": {}
        	            	-  }
        	            	-}
        	            	+File [OUTPUT_DIR]/out.requests.txt not found
        	            	 
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/pipelines/test/not-enabled
        	Messages:   	pipelines/test/not-enabled/output.txt vs /tmp/TestAcceptpipelinestestnot-enabled3196051760/002/output.txt
    acceptance_test.go:1351: Missing output file: out.not-enabled.xml
TestAccept/pipelines/test/selection: branch ✅ | main (2f5d1a6) ❌ | latest (v1.26.0) ❌

main (2f5d1a6):

acceptance_test.go:1380: Diff:
        --- pipelines/test/selection/output.txt
        +++ /tmp/TestAcceptpipelinestestselection4283115004/002/output.txt
        @@ -4,85 +4,11 @@
         Created pipelines.my_pipeline
         Files: 3 uploaded, 0 deleted
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Run all tests of the only pipeline
         
         >>> [CLI] pipelines test --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        -
        -=== Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms
        -
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "pytest_args": [
        -        "-k",
        -        "orders and not slow",
        -        "-m",
        -        "smoke,fast"
        -      ],
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests",
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/tests/test_other.py"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Relative paths resolve from the working directory
        -
        ->>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][3]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_valid_order"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Test selections require the pipeline KEY first
        -
        ->>> musterr [CLI] pipelines test tests/test_orders.py
        -Error: "tests/test_orders.py" is not a pipeline in this bundle (pipelines: my_pipeline). Usage: databricks pipelines test my_pipeline tests/test_orders.py
        +Exit code: 1
        
FAIL

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- pipelines/test/selection/output.txt
        +++ /tmp/TestAcceptpipelinestestselection3565640637/002/output.txt
        @@ -4,85 +4,29 @@
         Created pipelines.my_pipeline
         Files: 3 uploaded, 0 deleted
         Resources: 1 created, 0 changed, 0 deleted, 0 unchanged
        -View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID][0]?w=[WORKSPACE_ID]
        +View your pipeline my_pipeline here: [DATABRICKS_URL]/pipelines/[UUID]?w=[WORKSPACE_ID]
         
         === Run all tests of the only pipeline
         
         >>> [CLI] pipelines test --no-wait
        -Recommendation: This command runs the last deployed version of the code
        +Error: unknown flag: --no-wait
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][1]
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
         
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {}
        -  }
        -}
        +Flags:
        +  -h, --help   help for test
         
        -=== Select tests by relative path and test ID, with pytest arguments in both --pytest-arg forms
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
        ->>> MSYS_NO_PATHCONV=1 [CLI] pipelines test my_pipeline tests tests/test_orders.py::test_backfill /Workspace/Users/someone@example.com/tests/test_other.py --pytest-arg=-k --pytest-arg=orders and not slow --pytest-arg -m --pytest-arg smoke,fast --no-wait
        -Recommendation: This command runs the last deployed version of the code
         
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][2]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "pytest_args": [
        -        "-k",
        -        "orders and not slow",
        -        "-m",
        -        "smoke,fast"
        -      ],
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests",
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_backfill",
        -        "/Workspace/Users/someone@example.com/tests/test_other.py"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Relative paths resolve from the working directory
        -
        ->>> [CLI] pipelines test my_pipeline test_orders.py::test_valid_order --no-wait
        -Recommendation: This command runs the last deployed version of the code
        -
        -If you've made local changes, run 'databricks pipelines deploy' first to ensure they are included.
        -
        -Pipeline: my_pipeline ([UUID][0])
        -Update: [UUID][3]
        -
        ->>> print_requests.py //updates
        -{
        -  "method": "POST",
        -  "path": "/api/2.0/pipelines/[UUID][0]/updates",
        -  "body": {
        -    "test_only": true,
        -    "test_options": {
        -      "test_selection": [
        -        "/Workspace/Users/[USERNAME]/.bundle/test-pipeline-test-selection/default/files/tests/test_orders.py::test_valid_order"
        -      ]
        -    }
        -  }
        -}
        -
        -=== Test selections require the pipeline KEY first
        -
        ->>> musterr [CLI] pipelines test tests/test_orders.py
        -Error: "tests/test_orders.py" is not a pipeline in this bundle (pipelines: my_pipeline). Usage: databricks pipelines test my_pipeline tests/test_orders.py
        +Exit code: 1

1 more failure details omitted, see the full report in the run.

@tharunbhuk-db
tharunbhuk-db requested review from pranshupand-db and removed request for pranshupand-db and rclarey November 17, 2026 10:15
The command no longer needs an environment variable to run.

Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
The gate is removed in #7715.

Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
The gate is removed in #7715.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant