Skip to content

Write a JUnit XML report from databricks pipelines test - #7434

Merged
tharunbhuk-db merged 3 commits into
mainfrom
tharun/pipelines-test-junit-flag
Nov 17, 2026
Merged

tharunbhuk-db merged 3 commits into
mainfrom
tharun/pipelines-test-junit-flag

Conversation

@tharunbhuk-db

@tharunbhuk-db tharunbhuk-db commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

🥞 Stacked PR


  • Add --junit-xml <path>, which writes the report on every exit after the flags parse.
  • The JUnit report: line prints before the summary line.
  • The help has examples: all tests, one file, one test by ID, CI with --junit-xml, --timeout then --resume, and --pipeline-id.
  • The changelog entry for --junit-xml is in this PR.

Tests: acceptance reports (out.*.xml) in acceptance/pipelines/test/*.

This pull request and its description were written by Isaac.

@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Commit: ef045d01b | Workflow run

Regression test report

Selected 8 changed tests (limit=20, 0 not selected)

8 tests: regression=8 unreleased=0 coverage=0 failing=0
test branch main (a6da538) latest (v1.26.0)
TestAccept/pipelines/test/arguments ✅ ❌ ❌
TestAccept/pipelines/test/invalid-project ✅ ❌ ❌
TestAccept/pipelines/test/results/failed ✅ ❌ ❌
TestAccept/pipelines/test/results/incomplete ✅ ❌ ❌
TestAccept/pipelines/test/results/passed ✅ ❌ ❌
TestAccept/pipelines/test/run-control/canceled ✅ ❌ ❌
TestAccept/pipelines/test/run-control/timeout ✅ ❌ ❌
TestAccept/pipelines/test/submit-error ✅ ❌ ❌
TestAccept/pipelines/test/arguments: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments1465261183/002/output.txt
        @@ -31,17 +31,10 @@
         --timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
         to read the results later. To stop the update, run 'pipelines stop'.
         
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. The report replaces the file on every exit except a flag that
        -cannot be parsed. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        -
         Usage:
           databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
         Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
               --no-wait[=true|false]     Don't wait for the tests to complete.
               --pipeline-id string       ID of the pipeline to test, instead of a KEY.
               --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        @@ -66,8 +55,29 @@
         === With the command off, an older report is still replaced
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --junit-xml out.gate-off.xml
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Error: unknown flag: --junit-xml
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         === pytest arguments must be passed with --pytest-arg
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        @@ -117,9 +126,51 @@
         === --junit-xml needs the results
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --junit-xml out.no-wait.xml
        -Error: --junit-xml cannot be used with --no-wait because no results are available when the command returns
        +Error: unknown flag: --junit-xml
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         === An invalid flag replaces an older report
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --pytest-arg=-x --junit-xml out.flag-error.xml
        -Error: unsupported pytest argument "-x": only -k and -m are supported
        +Error: unknown flag: --junit-xml
        +
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1133
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"unsupported pytest argument &#34;-x&#34;: only -k and -m are supported\">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<testsuites><testsuite name=\"old\" tests=\"1\"><testcase name=\"test_old\"></testcase></testsuite></testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,9 +1,2 @@
        	            	-<?xml version="1.0" encoding="UTF-8"?>
        	            	-<testsuites>
        	            	-  <testsuite name="1234" tests="1" failures="0" errors="1" skipped="0">
        	            	-    <testcase name="pipelines test">
        	            	-      <error message="unsupported pytest argument &#34;-x&#34;: only -k and -m are supported">unsupported pytest argument &#34;-x&#34;: only -k and -m are supported</error>
        	            	-    </testcase>
        	            	-  </testsuite>
        	            	-</testsuites>
        	            	+<testsuites><testsuite name="old" tests="1"><testcase name="test_old"></testcase></testsuite></testsuites>
        	            	 
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.flag-error.xml vs /tmp/TestAcceptpipelinestestarguments1465261183/001/out.flag-error.xml
    acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1133
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<testsuites>\n  <testsuite name=\"1234\" tests=\"1\" failures=\"0\" errors=\"1\" skipped=\"0\">\n    <testcase name=\"pipelines test\">\n      <error message=\"pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it\">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>\n    </testcase>\n  </testsuite>\n</testsuites>\n"
        	            	actual  : "<testsuites><testsuite name=\"old\" tests=\"1\"><testcase name=\"test_old\"></testcase></testsuite></testsuites>\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1,9 +1,2 @@
        	            	-<?xml version="1.0" encoding="UTF-8"?>
        	            	-<testsuites>
        	            	-  <testsuite name="1234" tests="1" failures="0" errors="1" skipped="0">
        	            	-    <testcase name="pipelines test">
        	            	-      <error message="pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it">pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it</error>
        	            	-    </testcase>
        	            	-  </testsuite>
        	            	-</testsuites>
        	            	+<testsuites><testsuite name="old" tests="1"><testcase name="test_old"></testcase></testsuite></testsuites>
        	            	 
        	Test:       	TestAccept/pipelines/test/arguments
        	Messages:   	pipelines/test/arguments/out.gate-off.xml vs /tmp/TestAcceptpipelinestestarguments1465261183/001/out.gate-off.xml
    acceptance_test.go:1351: Missing output file: out.no-wait.xml

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- pipelines/test/arguments/output.txt
        +++ /tmp/TestAcceptpipelinestestarguments1151247066/002/output.txt
        @@ -23,40 +23,22 @@
         
           databricks pipelines test my_pipeline --pytest-arg=-k --pytest-arg='orders and not slow'
         
        -The command waits for the results. It exits with code 0 only if at least one
        -test was collected and none failed or errored, so a run whose tests were all
        -skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
        -update starts.
        -
        ---timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
        -to read the results later. To stop the update, run 'pipelines stop'.
        -
        ---junit-xml writes one entry per test with its outcome and duration. An
        -incomplete run, a run without tests, or a timeout adds a failing entry that
        -names the cause. The report replaces the file on every exit except a flag that
        -cannot be parsed. With workspace.sync_mode immutable, test names keep the path
        -of the deployed snapshot, which changes with each deployment.
        -
         Usage:
           databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
         Pipeline Test Flags:
        -      --junit-xml string         Write the results as a JUnit XML report to this file, replacing it.
        -      --no-wait[=true|false]     Don't wait for the tests to complete.
               --pipeline-id string       ID of the pipeline to test, instead of a KEY.
               --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        -      --resume string            Read the results of an existing test update instead of starting one.
        -      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
         
         Flags:
        -  -h, --help[=true|false]   help for test
        +  -h, --help   help for test
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
         
         === The command is experimental and off by default
         
        @@ -66,8 +38,26 @@
         === With the command off, an older report is still replaced
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --junit-xml out.gate-off.xml
        -Error: pipelines test is experimental. Set DATABRICKS_CLI_PIPELINES_TEST=true to enable it
        +Error: unknown flag: --junit-xml
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         === pytest arguments must be passed with --pytest-arg
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 -k orders
        @@ -97,29 +86,173 @@
         === Incompatible run-control flags
         
         >>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
        -Error: --resume cannot be used with test selections
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
        -Error: --resume cannot be used with --pytest-arg
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
        -Error: --resume cannot be used with --no-wait
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
        -Error: --timeout cannot be used with --no-wait
        +Error: unknown flag: --no-wait
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
        -Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +
        +Flags:
        +  -h, --help   help for test
        +
        +Global Flags:
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
        +      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
         >>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 
        -Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
        +Error: unknown flag: --resume
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
... (112 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/pipelines/test/invalid-project: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/invalid-project/output.txt
        +++ /tmp/TestAcceptpipelinestestinvalid-project202956405/002/output.txt
        @@ -2,5 +2,25 @@
         === A project with errors names its first error in the report
         
         >>> musterr [CLI] pipelines test my_pipeline --junit-xml out.invalid.xml
        -Error: reference does not exist: ${var.missing}
        +Error: unknown flag: --junit-xml
         
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        
    acceptance_test.go:1351: Missing output file: out.invalid.xml
FAIL

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== A project with errors names its first error in the report\n\n>>> musterr [CLI] pipelines test my_pipeline --junit-xml out.invalid.xml\nError: reference does not exist: ${var.missing}\n\n"
        	            	actual  : "\n=== A project with errors names its first error in the report\n\n>>> musterr [CLI] pipelines test my_pipeline --junit-xml out.invalid.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,3 +4,20 @@
        	            	 >>> musterr [CLI] pipelines test my_pipeline --junit-xml out.invalid.xml
        	            	-Error: reference does not exist: ${var.missing}
        	            	+Error: unknown flag: --junit-xml
        	            	+
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	+
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	 
        	Test:       	TestAccept/pipelines/test/invalid-project
        	Messages:   	pipelines/test/invalid-project/output.txt vs /tmp/TestAcceptpipelinestestinvalid-project3538450437/002/output.txt
    acceptance_test.go:1351: Missing output file: out.invalid.xml
TestAccept/pipelines/test/results/failed: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/results/failed/output.txt
        +++ /tmp/TestAcceptpipelinestestresultsfailed2883344120/002/output.txt
        @@ -1,22 +1,26 @@
         
         === Failed, errored and skipped tests
         
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 4 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED]
        -2 of 4 FAILED tests/test_orders.py::test_backfill .............................. [FAILED in 2.14s]
        -3 of 4 SKIPPED tests/test_orders.py::test_legacy_format ........................ [SKIPPED]
        -4 of 4 ERROR tests/test_orders.py::test_fixture ................................ [ERROR]
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        +Error: unknown flag: --junit-xml
         
        -Completed with 1 failed and 1 errored tests:
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        -Failure in tests/test_orders.py::test_backfill
        -  AssertionError: assert 100 == 150
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
         
        -Error in tests/test_orders.py::test_fixture
        -  RuntimeError: fixture orders setup failed
        +Flags:
        +  -h, --help[=true|false]   help for test
         
        -JUnit report: out.report.xml
        -Done. PASSED=1 FAILED=1 SKIPPED=1 ERROR=1 TOTAL=4
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        
    acceptance_test.go:1351: Missing output file: out.report.xml

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== Failed, errored and skipped tests\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\nRunning tests...\n1 of 4 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED]\n2 of 4 FAILED tests/test_orders.py::test_backfill .............................. [FAILED in 2.14s]\n3 of 4 SKIPPED tests/test_orders.py::test_legacy_format ........................ [SKIPPED]\n4 of 4 ERROR tests/test_orders.py::test_fixture ................................ [ERROR]\n\nCompleted with 1 failed and 1 errored tests:\n\nFailure in tests/test_orders.py::test_backfill\n  AssertionError: assert 100 == 150\n\nError in tests/test_orders.py::test_fixture\n  RuntimeError: fixture orders setup failed\n\nJUnit report: out.report.xml\nDone. PASSED=1 FAILED=1 SKIPPED=1 ERROR=1 TOTAL=4\n"
        	            	actual  : "\n=== Failed, errored and skipped tests\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,21 +3,22 @@
        	            	 
        	            	->>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	-Running tests...
        	            	-1 of 4 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED]
        	            	-2 of 4 FAILED tests/test_orders.py::test_backfill .............................. [FAILED in 2.14s]
        	            	-3 of 4 SKIPPED tests/test_orders.py::test_legacy_format ........................ [SKIPPED]
        	            	-4 of 4 ERROR tests/test_orders.py::test_fixture ................................ [ERROR]
        	            	+>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        	            	+Error: unknown flag: --junit-xml
        	            	 
        	            	-Completed with 1 failed and 1 errored tests:
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	 
        	            	-Failure in tests/test_orders.py::test_backfill
        	            	-  AssertionError: assert 100 == 150
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	 
        	            	-Error in tests/test_orders.py::test_fixture
        	            	-  RuntimeError: fixture orders setup failed
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	 
        	            	-JUnit report: out.report.xml
        	            	-Done. PASSED=1 FAILED=1 SKIPPED=1 ERROR=1 TOTAL=4
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	 
        	            	+
        	Test:       	TestAccept/pipelines/test/results/failed
        	Messages:   	pipelines/test/results/failed/output.txt vs /tmp/TestAcceptpipelinestestresultsfailed2945997700/002/output.txt
    acceptance_test.go:1351: Missing output file: out.report.xml
TestAccept/pipelines/test/results/incomplete: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/results/incomplete/output.txt
        +++ /tmp/TestAcceptpipelinestestresultsincomplete3328195179/002/output.txt
        @@ -1,14 +1,26 @@
         
         === Compute ends before all results are received
         
        ->>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 2 PASSED tests/test_compute.py::test_small_order .......................... [PASSED]
        +>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        +Error: unknown flag: --junit-xml
         
        -RUN INCOMPLETE: the update failed before all test results were received.
        -To read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        -JUnit report: out.report.xml
        -Done. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        -Error: update [UUID][1] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        
    acceptance_test.go:1351: Missing output file: out.report.xml

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== Compute ends before all results are received\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\nRunning tests...\n1 of 2 PASSED tests/test_compute.py::test_small_order .......................... [PASSED]\n\nRUN INCOMPLETE: the update failed before all test results were received.\nTo read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]\nJUnit report: out.report.xml\nDone. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2\nError: update [UUID][1] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.\n"
        	            	actual  : "\n=== Compute ends before all results are received\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,13 +3,22 @@
        	            	 
        	            	->>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	-Running tests...
        	            	-1 of 2 PASSED tests/test_compute.py::test_small_order .......................... [PASSED]
        	            	+>>> musterr [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        	            	+Error: unknown flag: --junit-xml
        	            	 
        	            	-RUN INCOMPLETE: the update failed before all test results were received.
        	            	-To read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        	            	-JUnit report: out.report.xml
        	            	-Done. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        	            	-Error: update [UUID][1] is FAILED: ComputeTerminatedException: Compute terminated unexpectedly.
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	 
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	+
        	Test:       	TestAccept/pipelines/test/results/incomplete
        	Messages:   	pipelines/test/results/incomplete/output.txt vs /tmp/TestAcceptpipelinestestresultsincomplete1625591935/002/output.txt
    acceptance_test.go:1351: Missing output file: out.report.xml
TestAccept/pipelines/test/results/passed: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/results/passed/output.txt
        +++ /tmp/TestAcceptpipelinestestresultspassed2606806925/002/output.txt
        @@ -1,32 +1,28 @@
         
         === All tests pass
         
        ->>> [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        -Running tests...
        -1 of 2 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.84s]
        -2 of 2 PASSED tests/test_orders.py::test_backfill .............................. [PASSED in 1.00s]
        +>>> [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        +Error: unknown flag: --junit-xml
         
        -JUnit report: out.report.xml
        -Done. PASSED=2 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
         
        ->>> print_requests.py --get //events
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID][0]/events",
        -  "q": {
        -    "filter": "update_id = '[UUID][1]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/pipelines/[UUID][0]/events",
        -  "q": {
        -    "filter": "update_id = '[UUID][1]'",
        -    "max_results": "250",
        -    "order_by": "timestamp asc"
        -  }
        -}
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        +
        +Exit code: 1
        
    acceptance_test.go:1351: Missing output file: out.report.xml

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== All tests pass\n\n>>> [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\nRunning tests...\n1 of 2 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.84s]\n2 of 2 PASSED tests/test_orders.py::test_backfill .............................. [PASSED in 1.00s]\n\nJUnit report: out.report.xml\nDone. PASSED=2 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2\n\n>>> print_requests.py --get //events\n{\n  \"method\": \"GET\",\n  \"path\": \"/api/2.0/pipelines/[UUID][0]/events\",\n  \"q\": {\n    \"filter\": \"update_id = '[UUID][1]'\",\n    \"max_results\": \"250\",\n    \"order_by\": \"timestamp asc\"\n  }\n}\n{\n  \"method\": \"GET\",\n  \"path\": \"/api/2.0/pipelines/[UUID][0]/events\",\n  \"q\": {\n    \"filter\": \"update_id = '[UUID][1]'\",\n    \"max_results\": \"250\",\n    \"order_by\": \"timestamp asc\"\n  }\n}\n"
        	            	actual  : "\n=== All tests pass\n\n>>> [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml\nError: unknown flag: --junit-xml\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n\nExit code: 1\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -3,31 +3,24 @@
        	            	 
        	            	->>> [CLI] pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	-Running tests...
        	            	-1 of 2 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.84s]
        	            	-2 of 2 PASSED tests/test_orders.py::test_backfill .............................. [PASSED in 1.00s]
        	            	+>>> [CLI] pipelines test --pipeline-id [UUID] --junit-xml out.report.xml
        	            	+Error: unknown flag: --junit-xml
        	            	 
        	            	-JUnit report: out.report.xml
        	            	-Done. PASSED=2 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=2
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	 
        	            	->>> print_requests.py --get //events
        	            	-{
        	            	-  "method": "GET",
        	            	-  "path": "/api/2.0/pipelines/[UUID][0]/events",
        	            	-  "q": {
        	            	-    "filter": "update_id = '[UUID][1]'",
        	            	-    "max_results": "250",
        	            	-    "order_by": "timestamp asc"
        	            	-  }
        	            	-}
        	            	-{
        	            	-  "method": "GET",
        	            	-  "path": "/api/2.0/pipelines/[UUID][0]/events",
        	            	-  "q": {
        	            	-    "filter": "update_id = '[UUID][1]'",
        	            	-    "max_results": "250",
        	            	-    "order_by": "timestamp asc"
        	            	-  }
        	            	-}
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	 
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	+
        	            	+Exit code: 1
        	            	+
        	Test:       	TestAccept/pipelines/test/results/passed
        	Messages:   	pipelines/test/results/passed/output.txt vs /tmp/TestAcceptpipelinestestresultspassed442300306/002/output.txt
    acceptance_test.go:1351: Missing output file: out.report.xml
TestAccept/pipelines/test/run-control/canceled: branch ✅ | main (a6da538) ❌ | latest (v1.26.0) ❌

main (a6da538):

acceptance_test.go:1380: Diff:
        --- pipelines/test/run-control/canceled/output.txt
        +++ /tmp/TestAcceptpipelinestestrun-controlcanceled3838139578/002/output.txt
        @@ -2,11 +2,25 @@
         === A canceled update is an incomplete run
         
         >>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --junit-xml out.report.xml
        -Pipeline: [UUID][0]
        -Update: [UUID][1]
        +Error: unknown flag: --junit-xml
         
        -RUN INCOMPLETE: the update was canceled before all test results were received.
        -To read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        -JUnit report: out.report.xml
        -Done. PASSED=0 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=1
        -Error: update [UUID][1] is CANCELED
        +Usage:
        +  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        +
        +Pipeline Test Flags:
        +      --no-wait[=true|false]     Don't wait for the tests to complete.
        +      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        +      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        +      --resume string            Read the results of an existing test update instead of starting one.
        +      --timeout duration         Stop waiting for results after this duration, for example 30m. The update keeps running.
        +
        +Flags:
        +  -h, --help[=true|false]   help for test
        +
        +Global Flags:
        +      --debug[=true|false]   enable debug logging
        +  -o, --output type          output type: text or json (default text)
        +  -p, --profile string       ~/.databrickscfg profile
        +  -t, --target string        bundle target to use (if applicable)
        +      --var strings          set values for variables defined in project config. Example: --var="foo=bar"
        +
        
    acceptance_test.go:1351: Missing output file: out.report.xml

latest (v1.26.0):

acceptance_test.go:1380: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1380
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1121
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:666
        	Error:      	Not equal: 
        	            	expected: "\n=== A canceled update is an incomplete run\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --junit-xml out.report.xml\nPipeline: [UUID][0]\nUpdate: [UUID][1]\n\nRUN INCOMPLETE: the update was canceled before all test results were received.\nTo read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]\nJUnit report: out.report.xml\nDone. PASSED=0 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=1\nError: update [UUID][1] is CANCELED\n"
        	            	actual  : "\n=== A canceled update is an incomplete run\n\n>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --junit-xml out.report.xml\nError: unknown flag: --resume\n\nUsage:\n  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]\n\nPipeline Test Flags:\n      --pipeline-id string       ID of the pipeline to test, instead of a KEY.\n      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.\n\nFlags:\n  -h, --help   help for test\n\nGlobal Flags:\n      --debug            enable debug logging\n  -o, --output type      output type: text or json (default text)\n  -p, --profile string   ~/.databrickscfg profile\n  -t, --target string    bundle target to use (if applicable)\n      --var strings      set values for variables defined in project config. Example: --var=\"foo=bar\"\n\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -4,10 +4,21 @@
        	            	 >>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --junit-xml out.report.xml
        	            	-Pipeline: [UUID][0]
        	            	-Update: [UUID][1]
        	            	+Error: unknown flag: --resume
        	            	 
        	            	-RUN INCOMPLETE: the update was canceled before all test results were received.
        	            	-To read the results again, run: databricks pipelines test --pipeline-id [UUID][0] --junit-xml out.report.xml --resume [UUID][1]
        	            	-JUnit report: out.report.xml
        	            	-Done. PASSED=0 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=1
        	            	-Error: update [UUID][1] is CANCELED
        	            	+Usage:
        	            	+  databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
        	            	 
        	            	+Pipeline Test Flags:
        	            	+      --pipeline-id string       ID of the pipeline to test, instead of a KEY.
        	            	+      --pytest-arg stringArray   Argument to pass to pytest (repeatable). Only -k and -m are supported.
        	            	+
        	            	+Flags:
        	            	+  -h, --help   help for test
        	            	+
        	            	+Global Flags:
        	            	+      --debug            enable debug logging
        	            	+  -o, --output type      output type: text or json (default text)
        	            	+  -p, --profile string   ~/.databrickscfg profile
        	            	+  -t, --target string    bundle target to use (if applicable)
        	            	+      --var strings      set values for variables defined in project config. Example: --var="foo=bar"
        	            	+
        	            	+
        	Test:       	TestAccept/pipelines/test/run-control/canceled
        	Messages:   	pipelines/test/run-control/canceled/output.txt vs /tmp/TestAcceptpipelinestestrun-controlcanceled2084991272/002/output.txt
    acceptance_test.go:1351: Missing output file: out.report.xml
FAIL

2 more failure details omitted, see the full report in the run.

@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from 98b0082 to c9c5f8c Compare November 10, 2026 07:11
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 58df287 to 9d83e17 Compare November 10, 2026 07:11
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from c9c5f8c to e358a5d Compare November 10, 2026 09:08
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 9d83e17 to 062047b Compare November 10, 2026 09:08
@tharunbhuk-db
tharunbhuk-db marked this pull request as ready for review November 10, 2026 09:21
@tharunbhuk-db
tharunbhuk-db requested review from a team as code owners November 10, 2026 09:21
@tharunbhuk-db
tharunbhuk-db requested review from anton-107 and removed request for a team and anton-107 November 10, 2026 09:21
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 062047b to c7628c9 Compare November 10, 2026 17:14
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from e358a5d to 8b690b3 Compare November 10, 2026 17:14
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 10, 2026
Co-authored-by: Isaac <no-reply@databricks.com>

@pranshupand-db pranshupand-db left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Follow-ups for the stack are in the #7297 review: #7297

tharunbhuk-db added a commit that referenced this pull request Nov 11, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
## 🥞 Stacked PR
Use this [link](https://github.com/databricks/cli/pull/7297/files) to
review incremental changes.
-
[**stack/pipelines-test-pr1-command**](databricks#7297)
[[Files changed](https://github.com/databricks/cli/pull/7297/files)] ←
_this PR_
-
[stack/pipelines-test-bundle-key](databricks#7431)
[[Files changed](https://github.com/databricks/cli/pull/7431/files)]
-
[stack/pipelines-test-read-events](databricks#7432)
[[Files changed](https://github.com/databricks/cli/pull/7432/files)]
-
[stack/pipelines-test-print-results](databricks#7433)
[[Files changed](https://github.com/databricks/cli/pull/7433/files)]
-
[stack/pipelines-test-pr2-output](databricks#7298)
[[Files changed](https://github.com/databricks/cli/pull/7298/files)]
-
[stack/pipelines-test-a4-run-control](databricks#7426)
[[Files changed](https://github.com/databricks/cli/pull/7426/files)]
-
[stack/pipelines-test-pr3-junit](databricks#7299)
[[Files changed](https://github.com/databricks/cli/pull/7299/files)]
-
[stack/pipelines-test-junit-flag](databricks#7434)
[[Files changed](https://github.com/databricks/cli/pull/7434/files)]

---------
- Add `databricks pipelines test --pipeline-id <id>
[TEST_SELECTION...]`: starts a test-only update and prints its IDs.
- Experimental: runs only with `DATABRICKS_CLI_PIPELINES_TEST=true`.
- Selections are workspace paths or paths relative to `root_path`, sent
as typed. `--pytest-arg` accepts only `-k` and `-m`.

Tests: unit tests; acceptance in
`acceptance/pipelines/test/{arguments,pipeline-id,submit-error}`.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
idelafuente-oc pushed a commit to idelafuente-oc/cli that referenced this pull request Nov 11, 2026
## 🥞 Stacked PR
-
[stack/pipelines-test-pr1-command](databricks#7297)
[[Files changed](https://github.com/databricks/cli/pull/7297/files)]
-
[**stack/pipelines-test-bundle-key**](databricks#7431)
[[Files changed](https://github.com/databricks/cli/pull/7431/files)] ←
_this PR_
-
[stack/pipelines-test-read-events](databricks#7432)
[[Files changed](https://github.com/databricks/cli/pull/7432/files)]
-
[stack/pipelines-test-print-results](databricks#7433)
[[Files changed](https://github.com/databricks/cli/pull/7433/files)]
-
[stack/pipelines-test-pr2-output](databricks#7298)
[[Files changed](https://github.com/databricks/cli/pull/7298/files)]
-
[stack/pipelines-test-a4-run-control](databricks#7426)
[[Files changed](https://github.com/databricks/cli/pull/7426/files)]
-
[stack/pipelines-test-pr3-junit](databricks#7299)
[[Files changed](https://github.com/databricks/cli/pull/7299/files)]
-
[stack/pipelines-test-junit-flag](databricks#7434)
[[Files changed](https://github.com/databricks/cli/pull/7434/files)]

---------
- In a bundle, identify the pipeline by KEY: `pipelines test [flags]
[KEY] [TEST_SELECTION...]`.
- Relative selections are local paths, mapped to their deployed paths.
- A first argument that is not a KEY gets an error that names the
bundle's pipelines and the working command.

Tests: unit tests; acceptance in
`acceptance/pipelines/test/{selection,immutable}`.

This pull request and its description were written by Isaac.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from c7628c9 to 0302118 Compare November 16, 2026 22:19
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from 8b690b3 to 29d2e9c Compare November 16, 2026 22:20
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 0302118 to b7f7b0a Compare November 17, 2026 00:18
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from 29d2e9c to cbb83bc Compare November 17, 2026 00:18
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
The changelog fragments of #7298, #7426 and #7434 are left out of this
merge: they belong to those PRs, and this merge goes away when they land.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from b7f7b0a to 04bb909 Compare November 17, 2026 07:01
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from cbb83bc to 40e2046 Compare November 17, 2026 07:01
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 04bb909 to 2eec88e Compare November 17, 2026 08:19
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from 40e2046 to bc59e20 Compare November 17, 2026 08:19
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
…and JUnit PRs

They move here from #7298, #7426 and #7434, so that all pipeline testing
entries are in one PR.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-pr3-junit branch from 2eec88e to 7a5f44f Compare November 17, 2026 09:12
Base automatically changed from tharun/pipelines-test-pr3-junit to main November 17, 2026 09:34
tharunbhuk-db and others added 3 commits November 17, 2026 09:34
--junit-xml writes one testcase per collected test case, with its outcome and
duration; an incomplete run, a run without tests, or a timeout adds a failing
entry that names the cause. The report replaces the file on every exit after
the flags parse, including the disabled-command error and project errors, so
an older report never survives; a flag that cannot be parsed ends the command
before it runs. --junit-xml cannot be combined with the no-wait flag, and the
printed resume command keeps --junit-xml so that it replaces a partial report.
The report line prints before the summary, so the summary stays the last line.

Co-authored-by: Isaac <no-reply@databricks.com>
The help shows the common flows: all tests, one file, one test by ID, CI
with --junit-xml, --timeout then --resume, and --pipeline-id. The entry for
an error before the update starts uses the same name as the other entries
that the command adds. The changelog entry for --junit-xml moves to this PR.

Co-authored-by: Isaac <no-reply@databricks.com>
The entry is added with the other pipeline testing entries in #7439.

Co-authored-by: Isaac <no-reply@databricks.com>
@tharunbhuk-db
tharunbhuk-db force-pushed the tharun/pipelines-test-junit-flag branch from bc59e20 to ef045d0 Compare November 17, 2026 09:35
@tharunbhuk-db
tharunbhuk-db added this pull request to the merge queue Nov 17, 2026
Merged via the queue into main with commit aeed647 Nov 17, 2026
24 checks passed
@tharunbhuk-db
tharunbhuk-db deleted the tharun/pipelines-test-junit-flag branch November 17, 2026 09:59
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
…and JUnit PRs

They move here from #7298, #7426 and #7434, so that all pipeline testing
entries are in one PR.

Co-authored-by: Isaac <no-reply@databricks.com>
tharunbhuk-db added a commit that referenced this pull request Nov 17, 2026
…and JUnit PRs

They move here from #7298, #7426 and #7434, so that all pipeline testing
entries are in one PR.

Co-authored-by: Isaac <no-reply@databricks.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants