Skip to content

aitools: classify install errors by standard library error type - #7380

Open
anton-107 wants to merge 2 commits into
mainfrom
aitools-error-categories
Open

anton-107 wants to merge 2 commits into
mainfrom
aitools-error-categories

Conversation

@anton-107

Copy link
Copy Markdown
Contributor

Adds 16 AitoolsErrorCategory values and classifies wrapped standard library errors by type, never by error text.

Summary

 classifyInstallError(err)
   nil                          -> UNSPECIFIED
   *installer.SkillError        -> skillErrorCategory
-  anything else                -> UNCATEGORIZED
+  stdlibErrorCategory(err)
+    huh.ErrUserAborted                     -> USER_ABORTED
+    context.DeadlineExceeded               -> TIMEOUT
+    context.Canceled                       -> COMMAND_INTERRUPTED
+    net.Error (but not a bare syscall.Errno)
+      Timeout()                            -> TIMEOUT
+      otherwise                            -> NETWORK
+    clicompat.ErrNotFound                  -> MANIFEST_NOT_FOUND
+    fs.ErrPermission                       -> PERMISSION_DENIED
+    *fs.PathError / *os.LinkError / *os.SyscallError -> FILESYSTEM
+    JSON decode errors, anything else      -> UNCATEGORIZED

The order matters in two places:

  • syscall.Errno implements net.Error. Without the bare-Errno check, every OS file error would be reported as NETWORK. The check is a direct type assertion because errors.As would also match the Errno inside a *net.OpError.
  • The network check runs before the permission check, so a sandbox-blocked socket (connect: EACCES) is still NETWORK.

JSON decode errors stay UNCATEGORIZED. The manifest, the release payload and .state.json all decode JSON, so the error type alone can't tell MANIFEST_INVALID from STATE_INVALID. A follow-up PR adds typed fetch and state errors that set those.

New values (libs/telemetry/protos/aitools_install.go). Only some are emitted by this PR:

Emitted here Emitted later
NETWORK, TIMEOUT, MANIFEST_NOT_FOUND, PERMISSION_DENIED, FILESYSTEM INVALID_FLAGS, UNKNOWN_AGENT, PROMPT_FAILED, USER_ABORTED (follow-up: install flow telemetry)
RELEASE_LOOKUP_FAILED, RATE_LIMITED, SKILLS_FETCH_FAILED, MANIFEST_INVALID, STATE_INVALID, PLUGIN_CLI_TIMEOUT (follow-up: typed installer errors)
COMMAND_INTERRUPTED (nothing on the install path cancels the context yet)

Changelog fragment: .nextchanges/cli/aitools-install-error-categories.md.

Evidence

TestClassifyInstallErrorGenericCauses: one case per branch, including a refused connection, a sandbox-blocked socket, EBUSY on rename, and a real missing-file error from os.ReadFile.

  • Before (main's categorize.go with this PR's tests):
    --- FAIL: TestClassifyInstallErrorGenericCauses/network
    --- FAIL: .../network_errno, network_permission_denied, network_timeout
    --- FAIL: .../deadline, cancelled, cancelled_request, manifest_missing
    --- FAIL: .../permission, permission_errno, permission_on_rename
    --- FAIL: .../filesystem, filesystem_from_os, filesystem_on_rename, filesystem_syscall
    --- FAIL: .../user_abort
    FAIL  github.com/databricks/cli/cmd/aitools
    
    All 16 resolve to UNCATEGORIZED.
  • After (this branch):
    ok  github.com/databricks/cli/cmd/aitools
    
    TestClassifyInstallErrorCorruptStateIsUncategorized runs the real installer.LoadState on a corrupt file and gets UNCATEGORIZED, not MANIFEST_INVALID.

Full run on the same change before rebasing onto current main: unit tests 8633 passed / 0 failed; acceptance tests (every dir except bundle/) 350 passed / 0 failed; lint, fmt, whitespace, deadcode and .nextchanges checks clean. GOOS=windows go vet ./cmd/aitools/ passes. The tests were not run on Windows.

Merge Danger

Door: Two-way for the code, one-way for the enum names once a CLI release ships them.

Reverting the classifier is trivial. But once released binaries send these strings, the server must keep accepting them. The telemetry backend schema must add these values before or together with the CLI release. If the CLI ships first, the backend records unknown names as a NULL error_category; the event is still recorded.

Blast Radius: Telemetry.

Only the error_category in aitools install telemetry and in aitools install --output json changes. Install behaviour and exit codes don't change. Known gap: a malformed DATABRICKS_SKILLS_BASE_URL is reported as NETWORK.

Open item: add this PR's link to the changelog fragment (task check-changelog requires it).

This pull request and its description were written by Isaac.

Add error categories for aitools install failures and classify wrapped
standard library errors (network, timeout, interrupt, missing manifest,
permission and filesystem errors) by type, without reading error text.
JSON decode errors stay UNCATEGORIZED.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: a6fd339

Run: 42052103731

Env ✅​pass 🙈​skip Time
✅​ aws linux-2core-8gb 271 15 4:35
✅​ aws-windows-latest-4core-16gb 273 13 3:28
✅​ azure linux-2core-8gb 270 15 4:54
✅​ azure-windows-latest-4core-16gb 272 13 3:23
✅​ gcp linux-2core-8gb 271 15 4:43
✅​ gcp-windows-latest-4core-16gb 273 13 3:21
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:14 gcp-windows-latest-4core-16gb TestAccept
3:12 aws-windows-latest-4core-16gb TestAccept
2:54 azure-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants