Skip to content

Support accelerator_pool_id in AIR workload configuration - #7698

Open
ben-hansen-db wants to merge 5 commits into
mainfrom
ben/accelerator-pool-id
Open

ben-hansen-db wants to merge 5 commits into
mainfrom
ben/accelerator-pool-id

Conversation

@ben-hansen-db

@ben-hansen-db ben-hansen-db commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Changes

Merge blocked pending backend deployment and staging E2E validation.

Rename compute.pool_id to compute.accelerator_pool_id for AIR GPU Pool selection. The old YAML field is rejected as an unknown field, including when supplied alongside the new selector. Preflight and submission send the new API field; bundle conversion retains the legacy field supported by its current schema.

Support HIGH priority and document NORMAL and HIGH; existing priority inputs remain accepted for compatibility.

Why

Align AIR workload configuration with the task-scoped GPU Pool API. Backend support must be deployed before this CLI change merges or releases.

Tests

  • Priority parsing, submission, and config-help checks pass after the priority update.
  • Targeted AIR unit tests pass, covering selector validation, rejection of the removed pool_id YAML field, preflight and submission payloads, bundle conversion, and help output.
  • Full AIR acceptance suite passes.
  • Repository formatting and lint checks pass.
  • Full AIR unit suite passed before the alias removal; affected unit tests were rerun after it.
  • Staging E2E validation is pending backend deployment.

This PR was written with Codex.

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 16, 2026
@ben-hansen-db
ben-hansen-db marked this pull request as ready for review November 16, 2026 19:37
@ben-hansen-db
ben-hansen-db requested review from a team as code owners November 16, 2026 19:37
Comment thread .nextchanges/air/accelerator-pool-id.md Outdated
@@ -0,0 +1 @@
* Add `compute.accelerator_pool_id` for GPU Pool selection in AIR workload YAML, retaining `compute.pool_id` as a compatibility alias. ([#7698](https://github.com/databricks/cli/pull/7698))

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking on Tuesday deployment

@github-actions

github-actions Bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 1744d1fcb | Workflow run

Regression test report

Selected 4 changed tests (limit=20, 0 not selected)

4 tests: regression=4 unreleased=0 coverage=0 failing=0
test branch main (629a5ed) latest (v1.26.0)
TestAccept/air/config-help ✅ ❌ ❌
TestAccept/air/convert-to-dabs ✅ ❌ ❌
TestAccept/air/help ✅ ❌ ❌
TestAccept/air/run-submit-deps ✅ ❌ ❌
TestAccept/air/config-help: branch ✅ | main (629a5ed) ❌ | latest (v1.26.0) ❌

main (629a5ed):

acceptance_test.go:1380: Diff:
        --- air/config-help/output.txt
        +++ /tmp/TestAcceptairconfig-help1673026329/002/output.txt
        @@ -113,12 +113,12 @@
             GPU_1xRTX6000 - 1 accelerator per node
         
           Fields:
        -    num_accelerators     (required) Total number of GPUs to allocate.
        -    accelerator_type     (required) Which accelerator to run on, e.g. GPU_1xA10.
        +    num_accelerators  (required) Total number of GPUs to allocate.
        +    accelerator_type  (required) Which accelerator to run on, e.g. GPU_1xA10.
         
        -  GPU Pools (optional):
        -    accelerator_pool_id  GPU Pool to run on, by ID.
        -    priority_class       Scheduling priority within the pool: NORMAL or HIGH.
        +  Provisioned capacity (optional):
        +    pool_id           GPU pool to run on, by id.
        +    priority_class    Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL.
         
         For details on a field, run "databricks air run -h config.compute.<field>".
         
        @@ -132,15 +131,13 @@
           Required: no
         
         >>> [CLI] air run -h config.compute.accelerator_pool_id
        -config.compute.accelerator_pool_id
        -  GPU Pool to run on, by ID. Must be 1-255 characters. List available pools with 'databricks air pools list'. An invalid or inaccessible pool is rejected. When omitted, workspace capacity selection applies.
        +Error: unknown config field "config.compute.accelerator_pool_id"
         
        -  Type:     string
        -  Required: no
        +fields under "config.compute" are: accelerator_type, num_accelerators, pool_id, priority_class
         
         >>> [CLI] air run -h config.compute.priority_class
         config.compute.priority_class
        -  Scheduling priority within the pool: NORMAL or HIGH. Requires accelerator_pool_id.
        +  Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires pool_id.
         
           Type:     string
           Required: no
        @@ -231,7 +224,7 @@
         >>> [CLI] air run -h config.compute.acclerator_type
         Error: unknown config field "config.compute.acclerator_type"; did you mean "accelerator_type"?
         
        -fields under "config.compute" are: accelerator_pool_id, accelerator_type, num_accelerators, priority_class
        +fields under "config.compute" are: accelerator_type, num_accelerators, pool_id, priority_class
         
         === free-form map keys are not schema fields
         
        
FAIL

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- air/config-help/output.txt
        +++ /tmp/TestAcceptairconfig-help631576086/002/output.txt
        @@ -2,105 +2,75 @@
         === command help
         
         >>> [CLI] air run --help
        -Submit a workload to Databricks serverless GPU compute.
        +Submit a training workload to Databricks serverless GPU compute.
         
        -The workload is described by a YAML file (see --file). To look up a YAML
        -field, pass its path to -h as a separate argument:
        +The workload is described by a YAML config file (see --file).
         
        -  databricks air run -h config                            all fields
        -  databricks air run -h config.compute                    one section
        -  databricks air run -h config.compute.accelerator_type   one field
        +To look up a config field, pass its path to -h:
         
        -To have a coding agent write the YAML and follow the run, install the
        -databricks-ai-runtime skill:
        +  databricks air run -h config
        +  databricks air run -h config.compute
        +  databricks air run -h config.compute.accelerator_type
         
        -  databricks aitools install --skills-only --skills databricks-ai-runtime
        +The path must be a separate argument: cobra reserves -h as a boolean, so
        +-h=config.compute and -hconfig.compute are not accepted.
         
         Usage:
           databricks air run [flags]
         
        -Examples:
        -  # Submit and stream logs until the run finishes
        -  databricks air run -f workload.yaml --watch
        -
        -  # Validate the config against the workspace without submitting
        -  databricks air run -f workload.yaml --dry-run
        -
        -  # Try a different GPU without editing the file
        -  databricks air run -f workload.yaml --override compute.accelerator_type=GPU_1xH100
        -
         Flags:
        +      --dry-run                  Validate the config against the workspace without submitting a run
           -f, --file string              Path to the workload YAML config
        -      --watch[=true|false]       Stream logs until the run completes
        -      --override stringArray     Override a YAML field, e.g. compute.num_accelerators=8 (repeatable)
        -      --dry-run[=true|false]     Validate the config against the workspace without submitting a run
        +  -h, --help                     help for run
               --idempotency-key string   Return the existing run if this key was already used
        -  -h, --help[=true|false]        help for run
        +      --override stringArray     Override a YAML field, e.g. compute.num_accelerators=8 (repeatable)
        +      --watch                    Stream logs until the run completes
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         === schema overview
         
         >>> [CLI] air run -h config
         config
        -  Workload YAML schema. For details on a field, pass its path to -h, for example:
        +  The run YAML schema. Pass a field path for details, e.g. config.compute.accelerator_type.
         
        -    databricks air run -h config.compute.accelerator_type
        +  AI coding agents can use the databricks-ai-runtime skill to write workload
        +  YAML, submit runs, and monitor progress.
         
        -  Minimal example:
        -    experiment_name: my-run
        -    compute:
        -      num_accelerators: 1
        -      accelerator_type: GPU_1xA10
        -    code_source:
        -      root_path: .
        -    command: python "$CODE_SOURCE_PATH/train.py"
        +  Install the skill locally for your coding agent:
         
        -  Workload:
        +    databricks aitools install --skills-only --skills databricks-ai-runtime
        +
        +  Follow the prompts to choose your coding agent and installation scope.
        +  Use --scope project to install into the current project.
        +  See: https://github.com/databricks/databricks-agent-skills/tree/main/skills/databricks-ai-runtime
        +
        +  Fields:
             experiment_name              (required) Name of the experiment.
             compute                      (required) Which accelerators to run on and how many.
        -    command                      (required unless containers is set) Shell command that starts the workload.
             environment                  Python dependencies, runtime version, or a Unity Catalog image for a single-command run.
        +    command                      Shell command that starts the workload.
             containers                   Rank-partitioned containers for heterogeneous workloads.
             env_variables                Plain environment variables, as NAME: value.
             secrets                      Environment variables sourced from secrets, as NAME: scope/key.
             code_source                  Local code to upload and make available to the run.
        -    parameters                   Free-form values passed through to the workload.
        -
        -  Execution:
             max_retries                  How many times to retry a failed run.
             timeout_minutes              Wall-clock limit for the run in minutes.
             idempotency_token            Reuse token: a repeat submission with the same token returns the existing run instead of starting another.
        -
        -  MLflow tracking:
        +    parameters                   Free-form values passed through to the workload.
             mlflow_run_name              Name for the MLflow run.
             mlflow_experiment_directory  Workspace directory holding the MLflow experiment.
             mlflow_artifact_location     DBFS location where MLflow artifacts are written.
        -
        -  Access and cost attribution:
             permissions                  Who may view or manage the run, as a list of principal plus level grants.
        -    usage_policy_name            Usage policy to attribute the run's usage to, by name.
        -    usage_policy_id              Usage policy to attribute the run's usage to, by id.
        +    usage_policy_name            Usage policy to bill the run to, by name.
        +    usage_policy_id              Usage policy to bill the run to, by id.
         
        -For details on a field, run "databricks air run -h config.<field>".
        +Use "-h config.<field>" for details on a field.
         
        -Set on every node (use in command and training code):
        -  CODE_SOURCE_PATH              Extracted code_source directory
        -  MLFLOW_RUN_ID                 MLflow run created for this attempt; log metrics and artifacts to it
        -  NUM_NODES, NODE_RANK          Node count and this node's rank (0 to NUM_NODES-1)
        -  WORLD_SIZE, LOCAL_WORLD_SIZE  GPUs across all nodes, and on this node
        -  MASTER_ADDR, MASTER_PORT      Rank-0 address for distributed launchers such as torchrun
        -  HYPERPARAMETERS_PATH          The parameters YAML, when parameters is set
        -
        -Let a coding agent write this YAML for you with the databricks-ai-runtime skill:
        -  databricks aitools install --skills-only --skills databricks-ai-runtime
        -  (add --scope project to install into the current project)
        -  https://github.com/databricks/databricks-agent-skills/tree/main/skills/databricks-ai-runtime
        -
         === nested object lists its fields
         
         >>> [CLI] air run -h config.compute
        @@ -113,15 +46,13 @@
             GPU_1xRTX6000 - 1 accelerator per node
         
           Fields:
        -    num_accelerators     (required) Total number of GPUs to allocate.
        -    accelerator_type     (required) Which accelerator to run on, e.g. GPU_1xA10.
        +    num_accelerators  Total number of GPUs to allocate.
        +    accelerator_type  Which accelerator to run on, e.g. GPU_1xA10.
        +    pool_id           GPU pool to run on, by id.
        +    priority_class    Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL.
         
        -  GPU Pools (optional):
        -    accelerator_pool_id  GPU Pool to run on, by ID.
        -    priority_class       Scheduling priority within the pool: NORMAL or HIGH.
        +Use "-h config.compute.<field>" for details on a field.
         
        -For details on a field, run "databricks air run -h config.compute.<field>".
        -
         === new submission fields are documented
         
         >>> [CLI] air run -h config.mlflow_artifact_location
        @@ -132,15 +61,13 @@
           Required: no
         
         >>> [CLI] air run -h config.compute.accelerator_pool_id
        -config.compute.accelerator_pool_id
        -  GPU Pool to run on, by ID. Must be 1-255 characters. List available pools with 'databricks air pools list'. An invalid or inaccessible pool is rejected. When omitted, workspace capacity selection applies.
        +Error: unknown config field "config.compute.accelerator_pool_id"
         
        -  Type:     string
        -  Required: no
        +fields under "config.compute" are: accelerator_type, num_accelerators, pool_id, priority_class
         
         >>> [CLI] air run -h config.compute.priority_class
         config.compute.priority_class
        -  Scheduling priority within the pool: NORMAL or HIGH. Requires accelerator_pool_id.
        +  Scheduling priority within the pool: BEST_EFFORT (lowest, preemptable) or NORMAL. Requires pool_id.
         
           Type:     string
           Required: no
        @@ -154,12 +77,8 @@
         
         >>> [CLI] air run -h config.environment.version
         config.environment.version
        -  Managed environment to start from, e.g. 6 for the Standard environment or databricks_ai_v6 for the Databricks AI environment (default: 6). Standard includes CUDA but, from version 5, not PyTorch; add it with dependencies. The Databricks AI environment adds PyTorch, Transformers, and other ML libraries. See Docs below for supported versions and package lists.
        +  Client image version to pin (default: 6). See environment version release notes: https://docs.databricks.com/aws/en/release-notes/serverless/environment-version/
         
        -  Docs:
        -    AWS:   https://docs.databricks.com/aws/en/machine-learning/ai-runtime/develop-workloads/environment
        -    Azure: https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/develop-workloads/environment
        -
           Type:     string or int
           Required: no
         
        @@ -167,19 +85,15 @@
         
         >>> [CLI] air run -h config.compute.accelerator_type
         config.compute.accelerator_type
        -  Which accelerator to run on, e.g. GPU_1xA10. Matched case-sensitively.
... (56 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/air/convert-to-dabs: branch ✅ | main (629a5ed) ❌ | latest (v1.26.0) ❌

main (629a5ed):

acceptance_test.go:1380: Diff:
        --- air/convert-to-dabs/output.txt
        +++ /tmp/TestAcceptairconvert-to-dabs3282055603/002/output.txt
        @@ -2,224 +2,6 @@
         === convert an AIR run YAML into a DABs bundle (in place, next to the source)
         
         >>> [CLI] air convert-to-dabs train.yaml
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        +Error: invalid config train.yaml: line 8: unknown field "compute.accelerator_pool_id"; run "databricks air run -h config.compute" for the list of fields
         
        -To deploy and run this workload as a bundle:
        -  1. [CLI] bundle validate
        -  2. [CLI] bundle deploy
        -  3. [CLI] bundle run torchrun-a10-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== emitted databricks.yml (code_source packaged as a tgz artifact; no code is copied)
        -
        ->>> cat databricks.yml
        -bundle:
        -  name: torchrun-a10-smoke-test
        -sync:
        -  paths:
        -    - generated_artifacts
        -artifacts:
        -  code_source:
        -    type: tgz
        -    path: .
        -    include:
        -      - src
        -    files:
        -      - source: ./dist/code_source.tgz
        -targets:
        -  dev:
        -    mode: development
        -    default: true
        -resources:
        -  jobs:
        -    torchrun-a10-smoke-test:
        -      name: torchrun-a10-smoke-test
        -      tasks:
        -        - task_key: torchrun-a10-smoke-test
        -          environment_key: default
        -          max_retries: 3
        -          retry_on_timeout: true
        -          ai_runtime_task:
        -            experiment: torchrun-a10-smoke-test
        -            deployments:
        -              - command_path: ./generated_artifacts/command.sh
        -                compute:
        -                  accelerator_type: GPU_1xA10
        -                  accelerator_count: 1
        -                  provisioned_capacity_id: capacity-1
        -            code_source_path: ./dist/code_source.tgz
        -            mlflow_artifact_location: dbfs:/Volumes/main/default/artifacts
        -            priority_class: NORMAL
        -            unity_catalog_image_path: main.air.training:prod
        -      environments:
        -        - environment_key: default
        -          spec:
        -            environment_version: "6"
        -
        -=== the generated command.sh carries the run command
        -
        ->>> cat generated_artifacts/command.sh
        -cd $CODE_SOURCE_PATH
        -torchrun --nproc_per_node=1 train.py
        -
        -=== the emitted bundle validates
        -
        ->>> [CLI] bundle validate
        -Name: torchrun-a10-smoke-test
        -Target: dev
        -Workspace:
        -  User: [USERNAME]
        -  Path: /Workspace/Users/[USERNAME]/.bundle/torchrun-a10-smoke-test/dev
        -
        -Validation OK!
        -
        -=== re-converting refuses to clobber the generated bundle
        -
        ->>> [CLI] air convert-to-dabs train.yaml
        -Error: databricks.yml already exists in .; pass --force to overwrite or remove it
        -
         Exit code: 1
        -
        -=== --force overwrites it
        -
        ->>> [CLI] air convert-to-dabs train.yaml --force
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        -
        -To deploy and run this workload as a bundle:
        -  1. [CLI] bundle validate
        -  2. [CLI] bundle deploy
        -  3. [CLI] bundle run torchrun-a10-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== convert a B300 run using the current SDK
        -
        ->>> [CLI] air convert-to-dabs b300.yaml --output-dir generated-b300
        -Wrote a Databricks Asset Bundle to generated-b300:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        -
        -To deploy and run this workload as a bundle:
        -  1. cd generated-b300
        -  2. [CLI] bundle validate
        -  3. [CLI] bundle deploy
        -  4. [CLI] bundle run b300-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== the B300 bundle preserves its accelerator and AI Runtime version
        -
        ->>> cat generated-b300/databricks.yml
        -bundle:
        -  name: b300-smoke-test
        -sync:
        -  paths:
        -    - generated_artifacts
        -targets:
        -  dev:
        -    mode: development
        -    default: true
        -resources:
        -  jobs:
        -    b300-smoke-test:
        -      name: b300-smoke-test
        -      tasks:
        -        - task_key: b300-smoke-test
        -          environment_key: default
        -          max_retries: 3
        -          retry_on_timeout: true
        -          ai_runtime_task:
        -            experiment: b300-smoke-test
        -            deployments:
        -              - command_path: ./generated_artifacts/command.sh
        -                compute:
        -                  accelerator_type: GPU_8xB300
        -                  accelerator_count: 8
        -      environments:
        -        - environment_key: default
        -          spec:
        -            environment_version: "6"
        -
        -=== the current SDK recognizes B300 so bundle validation is clean
        -
        ->>> [CLI] bundle validate
        -Name: b300-smoke-test
        -Target: dev
        -Workspace:
        -  User: [USERNAME]
        -  Path: /Workspace/Users/[USERNAME]/.bundle/b300-smoke-test/dev
        -
        -Validation OK!
        -
        -=== databricks_ai versions use a base environment
        -
        ->>> [CLI] air convert-to-dabs databricks-ai.yaml --force
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        -
... (29 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- air/convert-to-dabs/output.txt
        +++ /tmp/TestAcceptairconvert-to-dabs352107392/002/output.txt
        @@ -2,224 +2,7 @@
         === convert an AIR run YAML into a DABs bundle (in place, next to the source)
         
         >>> [CLI] air convert-to-dabs train.yaml
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        +Error: invalid config train.yaml: yaml: unmarshal errors:
        +  line 8: field accelerator_pool_id not found in type aircmd.computeConfig
         
        -To deploy and run this workload as a bundle:
        -  1. [CLI] bundle validate
        -  2. [CLI] bundle deploy
        -  3. [CLI] bundle run torchrun-a10-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== emitted databricks.yml (code_source packaged as a tgz artifact; no code is copied)
        -
        ->>> cat databricks.yml
        -bundle:
        -  name: torchrun-a10-smoke-test
        -sync:
        -  paths:
        -    - generated_artifacts
        -artifacts:
        -  code_source:
        -    type: tgz
        -    path: .
        -    include:
        -      - src
        -    files:
        -      - source: ./dist/code_source.tgz
        -targets:
        -  dev:
        -    mode: development
        -    default: true
        -resources:
        -  jobs:
        -    torchrun-a10-smoke-test:
        -      name: torchrun-a10-smoke-test
        -      tasks:
        -        - task_key: torchrun-a10-smoke-test
        -          environment_key: default
        -          max_retries: 3
        -          retry_on_timeout: true
        -          ai_runtime_task:
        -            experiment: torchrun-a10-smoke-test
        -            deployments:
        -              - command_path: ./generated_artifacts/command.sh
        -                compute:
        -                  accelerator_type: GPU_1xA10
        -                  accelerator_count: 1
        -                  provisioned_capacity_id: capacity-1
        -            code_source_path: ./dist/code_source.tgz
        -            mlflow_artifact_location: dbfs:/Volumes/main/default/artifacts
        -            priority_class: NORMAL
        -            unity_catalog_image_path: main.air.training:prod
        -      environments:
        -        - environment_key: default
        -          spec:
        -            environment_version: "6"
        -
        -=== the generated command.sh carries the run command
        -
        ->>> cat generated_artifacts/command.sh
        -cd $CODE_SOURCE_PATH
        -torchrun --nproc_per_node=1 train.py
        -
        -=== the emitted bundle validates
        -
        ->>> [CLI] bundle validate
        -Name: torchrun-a10-smoke-test
        -Target: dev
        -Workspace:
        -  User: [USERNAME]
        -  Path: /Workspace/Users/[USERNAME]/.bundle/torchrun-a10-smoke-test/dev
        -
        -Validation OK!
        -
        -=== re-converting refuses to clobber the generated bundle
        -
        ->>> [CLI] air convert-to-dabs train.yaml
        -Error: databricks.yml already exists in .; pass --force to overwrite or remove it
        -
         Exit code: 1
        -
        -=== --force overwrites it
        -
        ->>> [CLI] air convert-to-dabs train.yaml --force
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        -
        -To deploy and run this workload as a bundle:
        -  1. [CLI] bundle validate
        -  2. [CLI] bundle deploy
        -  3. [CLI] bundle run torchrun-a10-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== convert a B300 run using the current SDK
        -
        ->>> [CLI] air convert-to-dabs b300.yaml --output-dir generated-b300
        -Wrote a Databricks Asset Bundle to generated-b300:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
        -
        -To deploy and run this workload as a bundle:
        -  1. cd generated-b300
        -  2. [CLI] bundle validate
        -  3. [CLI] bundle deploy
        -  4. [CLI] bundle run b300-smoke-test --no-wait
        -
        -bundle deploy uploads the code source and launch scripts automatically.
        -To see what it deployed and where: [CLI] bundle summary
        -
        -Unlike `air run` (which submits an ephemeral run), bundle deploy creates a
        -persistent job that is not garbage-collected. When you are done, remove the
        -job and its uploaded files with:
        -  [CLI] bundle destroy
        -
        -Note: the YAML does not set max_retries, so the job uses the `air run` default
        -(max_retries: 3, retry_on_timeout: true). Edit databricks.yml to change it.
        -
        -=== the B300 bundle preserves its accelerator and AI Runtime version
        -
        ->>> cat generated-b300/databricks.yml
        -bundle:
        -  name: b300-smoke-test
        -sync:
        -  paths:
        -    - generated_artifacts
        -targets:
        -  dev:
        -    mode: development
        -    default: true
        -resources:
        -  jobs:
        -    b300-smoke-test:
        -      name: b300-smoke-test
        -      tasks:
        -        - task_key: b300-smoke-test
        -          environment_key: default
        -          max_retries: 3
        -          retry_on_timeout: true
        -          ai_runtime_task:
        -            experiment: b300-smoke-test
        -            deployments:
        -              - command_path: ./generated_artifacts/command.sh
        -                compute:
        -                  accelerator_type: GPU_8xB300
        -                  accelerator_count: 8
        -      environments:
        -        - environment_key: default
        -          spec:
        -            environment_version: "6"
        -
        -=== the current SDK recognizes B300 so bundle validation is clean
        -
        ->>> [CLI] bundle validate
        -Name: b300-smoke-test
        -Target: dev
        -Workspace:
        -  User: [USERNAME]
        -  Path: /Workspace/Users/[USERNAME]/.bundle/b300-smoke-test/dev
        -
        -Validation OK!
        -
        -=== databricks_ai versions use a base environment
        -
        ->>> [CLI] air convert-to-dabs databricks-ai.yaml --force
        -Wrote a Databricks Asset Bundle to .:
        -  databricks.yml
        -  generated_artifacts/training_config.yaml
        -  generated_artifacts/command.sh
... (30 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/air/help: branch ✅ | main (629a5ed) ❌ | latest (v1.26.0) ❌

main (629a5ed):

acceptance_test.go:1380: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp147506417/002/output.txt
        @@ -112,7 +112,7 @@
         
         A GPU pool is dedicated accelerator capacity that your Databricks account team
         provisions. Pools are optional: a run uses one only when its YAML sets
        -compute.accelerator_pool_id.
        +compute.pool_id.
         
         Usage:
           databricks air pools [flags]

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- air/help/output.txt
        +++ /tmp/TestAcceptairhelp238424936/002/output.txt
        @@ -2,48 +2,34 @@
         === help
         
         >>> [CLI] air --help
        -Run GPU workloads on Databricks serverless compute.
        +Run and manage AI Runtime training workloads on Databricks serverless GPU compute.
         
        -Describe a workload in a YAML file and submit it with "databricks air run".
        -Follow a run with "databricks air logs" and "databricks air get", or see your
        -active runs with "databricks air list".
        +These commands are under active development.
         
         Usage:
           databricks air [command]
         
        -Run workloads:
        -  run             Submit a workload from a YAML config
        -
        -Manage runs:
        +Available Commands:
           cancel          Cancel one or more runs
        -  get             Show a run's status, configuration, and timing
        -  list            List your active runs
        -  logs            Stream or fetch a run's logs
        +  convert-to-dabs Convert an AIR run YAML into a Databricks Asset Bundle
        +  get             Show status, configuration, and timing details for a specified run
        +  images          Manage container images for AI Runtime
        +  list            List active runs for the current user
        +  logs            Stream or fetch logs for a run
        +  pools           View provisioned GPU pools
        +  run             Submit a training workload from a YAML config
         
        -Package and deploy:
        -  convert-to-dabs Convert a workload YAML into a Databricks Asset Bundle
        -  images          Push container images for custom environments
        -
        -Provisioned capacity:
        -  pools           View provisioned GPU pools, if any (not required to run)
        -
         Flags:
        -  -h, --help[=true|false]   help for air
        +  -h, --help   help for air
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         Use "databricks air [command] --help" for more information about a command.
         
        -Learn more:
        -  YAML fields and example  databricks air run -h config
        -  Quickstart (AWS)         https://docs.databricks.com/aws/en/machine-learning/ai-runtime/quickstart
        -  Quickstart (Azure)       https://learn.microsoft.com/azure/databricks/machine-learning/ai-runtime/quickstart
        -  Coding agent skill       databricks aitools install --skills-only --skills databricks-ai-runtime
        -
         === list help
         
         >>> [CLI] air list --help
        @@ -52,9 +25,7 @@
         Filters:
           Use --filter KEY=VALUE per condition. A run must match every condition.
         
        -  accelerator_type  Accelerator family or full YAML type. Examples:
        -                    A10, H100, GPU_1xA10, GPU_8xH100.
        -                    Workspace types: databricks air run -h config.compute.accelerator_type
        +  accelerator_type  Accelerator type, such as A10 or H100.
           experiment        Case-insensitive glob matched against the experiment name.
           num_accelerators  Total GPU count.
           user              Run creator identifier.
        @@ -65,78 +36,71 @@
         Examples:
           databricks air list --filter accelerator_type=H100
         
        -  databricks air list --filter accelerator_type=GPU_8xH100
        -
           databricks air list --all-status \
             --filter 'experiment=qwen*' \
             --filter num_accelerators=8 \
             --filter user=person@example.com
         
         Flags:
        -      --all-status[=true|false]   Show runs in all states (default: active only)
        -      --all-users[=true|false]    Show runs from all users
        -      --filter stringArray        Filter runs by KEY=VALUE (repeatable)
        -  -h, --help[=true|false]         help for list
        -      --limit int                 Maximum number of runs to show (default 20)
        +      --all-status           Show runs in all states (default: active only)
        +      --all-users            Show runs from all users
        +      --filter stringArray   Filter runs by KEY=VALUE (repeatable)
        +  -h, --help                 help for list
        +      --limit int            Maximum number of runs to show (default 20)
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         === get help
         
         >>> [CLI] air get --help
        -Show a run's status, configuration, and timing.
        +Show status, configuration, and timing details for a specified run.
         
        -JOB_RUN_ID is the job run ID returned by "databricks air run".
        +JOB_RUN_ID is the job run ID returned by "air run".
         
         Usage:
           databricks air get JOB_RUN_ID [flags]
         
         Flags:
        -  -h, --help[=true|false]             help for get
        -      --include-config[=true|false]   Include stored config in JSON (downloads over 1 MiB are omitted; may contain plaintext env_variables and parameters)
        +  -h, --help   help for get
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         === pools help
         
         >>> [CLI] air pools --help
        -View GPU pools provisioned for your workspace.
        +View provisioned GPU pools
         
        -A GPU pool is dedicated accelerator capacity that your Databricks account team
        -provisions. Pools are optional: a run uses one only when its YAML sets
        -compute.accelerator_pool_id.
        -
         Usage:
           databricks air pools [flags]
           databricks air pools [command]
         
         Available Commands:
        -  get         Show a GPU pool and its accelerator usage
        +  get         Show a GPU pool, including its accelerator usage (POOL_ID may be omitted when the workspace has exactly one pool)
           list        List the GPU pools available to the current workspace
         
         Flags:
        -  -h, --help[=true|false]   help for pools
        +  -h, --help   help for pools
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         Use "databricks air pools [command] --help" for more information about a command.
         
         === images help
         
         >>> [CLI] air images --help
        -Push container images for custom environments
        +Manage container images for AI Runtime
         
         Usage:
           databricks air images [flags]
        @@ -146,13 +69,13 @@
           push        *Preview* Push a container image to Databricks Artifact Registry
         
         Flags:
        -  -h, --help[=true|false]   help for images
        +  -h, --help   help for images
         
         Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        +      --debug            enable debug logging
        +  -o, --output type      output type: text or json (default text)
        +  -p, --profile string   ~/.databrickscfg profile
        +  -t, --target string    bundle target to use (if applicable)
         
         Use "databricks air images [command] --help" for more information about a command.
         
... (92 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)
TestAccept/air/run-submit-deps: branch ✅ | main (629a5ed) ❌ | latest (v1.26.0) ❌

main (629a5ed):

acceptance_test.go:1380: Diff:
        --- air/run-submit-deps/output.txt
        +++ /tmp/TestAcceptairrun-submit-deps3642830875/002/output.txt
        @@ -2,71 +2,9 @@
         === submit with inline dependencies
         
         >>> [CLI] air run -f run.yaml
        -Submitting experiment: deps-smoke
        -Submitted workload with Job Run ID: 555
        -View job run at: [DATABRICKS_URL]/jobs/runs/555?o=[WORKSPACE_ID]
        +Error: invalid config run.yaml: line 6: unknown field "compute.accelerator_pool_id"; run "databricks air run -h config.compute" for the list of fields
         
        -Tip: use --watch when submitting a run to stream logs to your terminal.
        -Stream logs after submission using:
        -  databricks air logs 555
        -
        -=== only config + command are uploaded; no requirements.yaml
        -
        ->>> print_requests.py //api/2.0/workspace-files/import-file --oneline --sort --unique --keep
        -{"method": "POST", "path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/command.sh", "q": {"overwrite": "true"}, "raw_body": "python train.py"}
        -{"method": "POST", "path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/training_config.yaml", "q": {"overwrite": "true"}, "raw_body": "experiment_name: deps-smoke\ncommand: python train.py\ncompute:\n    accelerator_type: GPU_1xH100\n    num_accelerators: 1\n    accelerator_pool_id: capacity-123\nenvironment:\n    version: 5\n    dependencies:\n        - numpy\n        - torch==2.3.0\nmlflow_artifact_location: /Volumes/main/default/air-artifacts\n"}
        -
        -=== inline deps, artifact location, and capacity id reach the submit payload
        -
        ->>> print_requests.py //api/2.2/jobs/runs/submit
        -{
        -  "method": "POST",
        -  "path": "/api/2.2/jobs/runs/submit",
        -  "body": {
        -    "environments": [
        -      {
        -        "environment_key": "default",
        -        "spec": {
        -          "dependencies": [
        -            "numpy",
        -            "torch==2.3.0"
        -          ],
        -          "environment_version": "5"
        -        }
        -      }
        -    ],
        -    "idempotency_token": "[UUID]",
        -    "run_name": "deps-smoke",
        -    "tasks": [
        -      {
        -        "ai_runtime_task": {
        -          "deployments": [
        -            {
        -              "command_path": "/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/command.sh",
        -              "compute": {
        -                "accelerator_count": 1,
        -                "accelerator_pool_id": "capacity-123",
        -                "accelerator_type": "GPU_1xH100"
        -              }
        -            }
        -          ],
        -          "experiment": "deps-smoke",
        -          "mlflow_artifact_location": "dbfs:/Volumes/main/default/air-artifacts"
        -        },
        -        "environment_key": "default",
        -        "max_retries": 3,
        -        "retry_on_timeout": true,
        -        "run_if": "ALL_SUCCESS",
        -        "task_key": "deps-smoke"
        -      }
        -    ]
        -  }
        -}
        -
        -=== a requirements.yaml file path is rejected; deps must be inline
        -
        ->>> [CLI] air run -f run-file.yaml
        -Error: invalid config run-file.yaml: environment.dependencies must be a list of packages or reference a requirements.txt (see https://docs.databricks.com/aws/en/machine-learning/ai-runtime/connecting/cli/yaml-config#reference). A direct file reference is not supported
        -
         Tip: use the --debug flag to see more details and a trace of this error:
           databricks --debug air run …
        +
        +Exit code: 1

latest (v1.26.0):

acceptance_test.go:1380: Diff:
        --- air/run-submit-deps/output.txt
        +++ /tmp/TestAcceptairrun-submit-deps2028251802/002/output.txt
        @@ -2,71 +2,10 @@
         === submit with inline dependencies
         
         >>> [CLI] air run -f run.yaml
        -Submitting experiment: deps-smoke
        -Submitted workload with Job Run ID: 555
        -View job run at: [DATABRICKS_URL]/jobs/runs/555?o=[WORKSPACE_ID]
        +Error: invalid config run.yaml: yaml: unmarshal errors:
        +  line 6: field accelerator_pool_id not found in type aircmd.computeConfig
         
        -Tip: use --watch when submitting a run to stream logs to your terminal.
        -Stream logs after submission using:
        -  databricks air logs 555
        -
        -=== only config + command are uploaded; no requirements.yaml
        -
        ->>> print_requests.py //api/2.0/workspace-files/import-file --oneline --sort --unique --keep
        -{"method": "POST", "path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/command.sh", "q": {"overwrite": "true"}, "raw_body": "python train.py"}
        -{"method": "POST", "path": "/api/2.0/workspace-files/import-file/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/training_config.yaml", "q": {"overwrite": "true"}, "raw_body": "experiment_name: deps-smoke\ncommand: python train.py\ncompute:\n    accelerator_type: GPU_1xH100\n    num_accelerators: 1\n    accelerator_pool_id: capacity-123\nenvironment:\n    version: 5\n    dependencies:\n        - numpy\n        - torch==2.3.0\nmlflow_artifact_location: /Volumes/main/default/air-artifacts\n"}
        -
        -=== inline deps, artifact location, and capacity id reach the submit payload
        -
        ->>> print_requests.py //api/2.2/jobs/runs/submit
        -{
        -  "method": "POST",
        -  "path": "/api/2.2/jobs/runs/submit",
        -  "body": {
        -    "environments": [
        -      {
        -        "environment_key": "default",
        -        "spec": {
        -          "dependencies": [
        -            "numpy",
        -            "torch==2.3.0"
        -          ],
        -          "environment_version": "5"
        -        }
        -      }
        -    ],
        -    "idempotency_token": "[UUID]",
        -    "run_name": "deps-smoke",
        -    "tasks": [
        -      {
        -        "ai_runtime_task": {
        -          "deployments": [
        -            {
        -              "command_path": "/Workspace/Users/[USERNAME]/.air/cli_launch/deps-smoke/deps-smoke_[RUN_ID]/command.sh",
        -              "compute": {
        -                "accelerator_count": 1,
        -                "accelerator_pool_id": "capacity-123",
        -                "accelerator_type": "GPU_1xH100"
        -              }
        -            }
        -          ],
        -          "experiment": "deps-smoke",
        -          "mlflow_artifact_location": "dbfs:/Volumes/main/default/air-artifacts"
        -        },
        -        "environment_key": "default",
        -        "max_retries": 3,
        -        "retry_on_timeout": true,
        -        "run_if": "ALL_SUCCESS",
        -        "task_key": "deps-smoke"
        -      }
        -    ]
        -  }
        -}
        -
        -=== a requirements.yaml file path is rejected; deps must be inline
        -
        ->>> [CLI] air run -f run-file.yaml
        -Error: invalid config run-file.yaml: environment.dependencies must be a list of packages or reference a requirements.txt (see https://docs.databricks.com/aws/en/machine-learning/ai-runtime/connecting/cli/yaml-config#reference). A direct file reference is not supported
        -
         Tip: use the --debug flag to see more details and a trace of this error:
           databricks --debug air run …
        +
        +Exit code: 1
        
FAIL

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 1744d1f

Run: 44132385100

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 270 19 14:32
💚​ aws-windows-latest-4core-16gb 1 272 17 11:22
💚​ azure linux-2core-8gb 1 269 19 15:05
💚​ azure-windows-latest-4core-16gb 1 271 17 9:24
💚​ gcp linux-2core-8gb 1 270 19 13:05
💚​ gcp-windows-latest-4core-16gb 1 272 17 8:40
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 27 slowest tests (at least 2 minutes):
duration env testname
4:40 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsStat
4:01 azure linux-2core-8gb TestFilerWorkspaceFilesExtensionsRead
3:50 aws-windows-latest-4core-16gb TestAccept
3:47 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
3:47 azure-windows-latest-4core-16gb TestAccept
3:39 gcp-windows-latest-4core-16gb TestAccept
3:29 azure linux-2core-8gb TestFilerRecursiveDelete/workspace_files_extensions
3:29 aws-windows-latest-4core-16gb TestLockUnlockWithoutAllowsLockFileNotExist
3:24 azure-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
3:10 aws-windows-latest-4core-16gb TestLock
2:46 azure linux-2core-8gb TestImportDir
2:38 gcp-windows-latest-4core-16gb TestSyncIncrementalSyncPythonNotebookDelete
2:36 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files_extensions
2:31 gcp-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsReadDir
2:31 aws-windows-latest-4core-16gb TestFilerWorkspaceFilesExtensionsRead
2:30 azure linux-2core-8gb TestSyncFullFileSync
2:29 aws-windows-latest-4core-16gb TestFilerWorkspaceNotebook/pythonJupyterNb.ipynb
2:28 gcp linux-2core-8gb TestFilerWorkspaceFilesExtensionsReadDir
2:23 aws-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:23 aws linux-2core-8gb TestSyncFullFileSync
2:20 aws-windows-latest-4core-16gb TestImportDirWithOverwriteFlag
2:13 aws-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:11 gcp-windows-latest-4core-16gb TestImportDir
2:09 aws linux-2core-8gb TestSyncIncrementalFileOverwritesFolder
2:08 azure-windows-latest-4core-16gb TestImportDirDoesNotOverwrite
2:05 azure-windows-latest-4core-16gb TestFilerRecursiveDelete/workspace_files
2:01 azure linux-2core-8gb TestImportDirDoesNotOverwrite

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants