Repository navigation
[bug fix] aitools: validate skill names and file paths in the fetched skills manifest - #7722
Conversation
… skills manifest
## Summary
`databricks aitools install` and `update` join every skill name and file path from the fetched skills manifest onto a local directory. Before this change none of them were checked, so a manifest entry could write anywhere the user can (for example `~/.bashrc`) and the command still reported success. `FetchManifest` now validates the manifest before any install or update work starts:
```diff
GitHubManifestSource.FetchManifest
read + json.Unmarshal manifest
+ validateManifestPaths(manifest)
+ skill name: Agent Skills name format, at most 64 chars ^[a-z0-9]+(-[a-z0-9]+)*$
+ ("../x" "a/b" "." ".state.json" "Foo" "foo." "foo " rejected)
+ skill name: not a Windows device name, on every OS (con prn aux nul com1-9 lpt1-9)
+ each file: filepath.IsLocal ("/etc/x" "../../.bashrc" "a/../../x" rejected)
normalizeManifest
```
Why the name rule is stricter than "a local path":
- Skill directories sit next to the installer's `.state.json`. A skill with that name would replace the state file with a directory.
- On case-insensitive filesystems, and through Windows trailing dots and spaces, two different names can resolve to the same directory. Only lowercase letters, digits and single hyphens are allowed, so that can't happen.
- On Windows, device names like `nul` resolve to a device in any directory. They are rejected on every OS, so a given manifest passes or fails the same way on every platform.
A failure returns a `FetchError` with stage `manifest`, e.g. `invalid skills manifest: skill "databricks-core" lists file "../../../.bashrc" outside its directory`.
The manifest comes from `raw.githubusercontent.com/databricks/databricks-agent-skills/<ref>/manifest.json` (or `DATABRICKS_SKILLS_BASE_URL`). Exploiting this needs control of what is served there, so the change is defense in depth for paths that arrive over the network, not a fix for a remote hole on its own.
## Review order
1. `libs/aitools/installer/source.go`: `validateManifestPaths`, the two name patterns, and the call in `FetchManifest`.
2. `libs/aitools/installer/source_test.go`:
- `TestFetchManifestRejectsPathsOutsideSkillDir`: table of accepted and rejected names and files.
- `TestInstallRejectsStateFileSkillNameWithoutTouchingState`: install, then install again from a manifest naming `.state.json`. Asserts that the install fails and the state file is byte-for-byte unchanged.
## Evidence
- **Before** (the new tests run against `main`'s `source.go`): 18 of the 25 `TestFetchManifestRejectsPathsOutsideSkillDir` cases fail (every case that should be rejected) and so does `TestInstallRejectsStateFileSkillNameWithoutTouchingState`. A manifest listing `../../../../escaped-file.txt` makes `InstallSkillsForAgents` return `nil` and write `$HOME/escaped-file.txt`.
**After:** all cases pass. Those manifests now fail in `FetchManifest` with the errors above, before any file is written.
- Accepted cases: `SKILL.md`, `references/auth.md`, names with digits, exactly 64 characters, and names that only contain a device name, such as `console` and `nul-tools`.
- The published manifests `v0.2.32`, `v0.2.49` and `main` (35–38 skills, 366–420 files) contain no entry the check rejects.
- `go build ./...`, `go test ./libs/aitools/... ./cmd/aitools/...`, the `aitools` acceptance tests and `./task lint-q` pass.
## Merge Danger
**Door:** two-way. Rollback: revert the PR.
**Blast Radius:** narrow.
Only `aitools install`/`update` are affected, and only for manifests that contain a rejected name or file. Those commands now fail instead of writing. No published manifest is affected. A future manifest with a skill name outside the Agent Skills format would be rejected as a whole.
Co-authored-by: Isaac <no-reply@databricks.com>
6a2c8ce to
f5bed94
Compare
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at f5bed94. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking].
This PR validates skill names and file paths in the fetched skills manifest before aitools install/update use them. The core check is sound: it blocks ../ and absolute-path writes, and the published manifest passes it. One blocking issue remains. A manifest that fails validation is classified as SKILLS_FETCH_FAILED, so users and --output json are told to check their network or proxy even though the download worked. The rest are smaller points (lazy regexps per the style guide, deterministic error order, canonical file paths, changelog scope) plus two non-blocking questions on whole-manifest rejection and where the check lives.
Questions
- One entry that breaks the name format now fails
install,update,listandinstall --pathfor every skill. That includes a name that is still a safe local path (e.g.databricks_sql) and an entry in an experimental ormin_cli_version-gated skill the user would never install. Released CLIs can't be patched, so one such upstream entry would break all of them until the skills release is reverted. The description says the manifest would be rejected as a whole, but not why that beats the alternatives. Would you fail hard only on real escapes (separators,..,.,.state.json) and skip format violations with a warning, the wayresolveSkillsskipsmin_cli_versionmismatches (installer.go:552-566)? Or keep it fail-closed and enforce the same name format in the databricks-agent-skills manifest generator or CI, so such a manifest is never published? (non-blocking) - The check runs only inside
GitHubManifestSource.FetchManifest, so every otherManifestSourceskips it, including the test mock most installer tests use. Every caller goes throughFetchSkillsManifestWithFallback(installer.go:264), and every file write goes throughinstallSkillToDir. Would you move the check to that choke point, or add a write-side guard such as writing throughos.OpenRoot(tmpDir), so an escape fails whatever the source and OS? (non-blocking)
| } | ||
|
|
||
| if err := validateManifestPaths(&manifest); err != nil { | ||
| return nil, &FetchError{Stage: FetchStageManifest, StatusCode: resp.StatusCode, Err: fmt.Errorf("invalid skills manifest: %w", err)} |
There was a problem hiding this comment.
[blocking] The validation error is wrapped as a FetchError with stage manifest and status 200. fetchErrorCategory (cmd/aitools/categorize.go:149-177) maps only JSON decode errors and ErrManifestTooLarge to MANIFEST_INVALID, so this error falls through to SKILLS_FETCH_FAILED. downloadRecovery then sends a 200 to unreachableRecovery.
For example, a manifest with skill Databricks_Core or file ../../.bashrc makes aitools install print invalid skills manifest: … followed by "Could not download Databricks skills from raw.githubusercontent.com. Check that this machine can reach it (set HTTPS_PROXY if you use a proxy)". I confirmed that category and recovery with a scratch test through classifyInstallError/installRecovery. The download worked, but --output json reports the wrong error_category and recovery, and telemetry files a possibly tampered manifest under download failures.
Suggest:
- add a sentinel next to
ErrManifestTooLargeand wrap it with%w; - map it to
MANIFEST_INVALIDinfetchErrorCategory; - give it a recovery step that doesn't point at the network (pin
DATABRICKS_SKILLS_REFto an earlier release, or report it); - add a row to
TestClassifyRealManifestFetchErrors.
There was a problem hiding this comment.
Fixed in 51002ef. An escape now wraps a new sentinel installer.ErrUnsafeManifestPath (next to ErrManifestTooLarge), which classifyInstallError maps to MANIFEST_INVALID. installRecovery gives it its own step instead of the captive-portal one: "No files were written. Set DATABRICKS_SKILLS_REF to an earlier release tag from …/releases and re-run …, and report the manifest from at …/cli/issues." The check moved to FetchSkillsManifestWithFallback (see the Q2 reply in the description), so the error is no longer a FetchError and TestClassifyRealManifestFetchErrors never sees it. I added TestClassifyUnsafeManifestPath, which serves a manifest listing ../../.bashrc through GitHubManifestSource and asserts MANIFEST_INVALID, plus a row in TestInstallRecovery.
| var skillNamePattern = regexp.MustCompile(`^[a-z0-9]+(-[a-z0-9]+)*$`) | ||
|
|
||
| // windowsDeviceNamePattern matches the names Windows resolves to a device in | ||
| // any directory. They are rejected on every OS so a manifest is valid or | ||
| // invalid the same way everywhere. | ||
| var windowsDeviceNamePattern = regexp.MustCompile(`^(con|prn|aux|nul|com[1-9]|lpt[1-9])$`) |
There was a problem hiding this comment.
[nice to have] Both patterns compile in package-level var initializers, which .agents/rules/style-guide-go.md:176 forbids. cmd/root imports this package, so they compile on every CLI invocation, including commands that never touch aitools. The same package already uses the approved form at plugin.go:348-350. Wrap them in sync.OnceValue(func() *regexp.Regexp { … }) and call skillNamePattern().MatchString(name). The fixed device-name list could also be a plain switch with no regexp.
There was a problem hiding this comment.
Fixed: skillNamePattern is now a sync.OnceValue, and the device-name check is a plain switch in isWindowsDeviceName, no regexp.
| // outside the skill's own directory. The manifest comes from the network, so it | ||
| // must not be able to name ../../.bashrc. | ||
| func validateManifestPaths(m *Manifest) error { | ||
| for name, meta := range m.Skills { |
There was a problem hiding this comment.
[nit] validateManifestPaths ranges over the m.Skills map and returns the first failure. With two bad entries (say ../a and B), the reported one changes from run to run. The rest of the package iterates in sorted order (dump.go:39). Use for _, name := range slices.Sorted(maps.Keys(m.Skills)).
There was a problem hiding this comment.
Fixed: both passes range over slices.Sorted(maps.Keys(m.Skills)). TestCheckManifestPathsReportsTheFirstSkillInOrder checks it.
| // windowsDeviceNamePattern matches the names Windows resolves to a device in | ||
| // any directory. They are rejected on every OS so a manifest is valid or | ||
| // invalid the same way everywhere. |
There was a problem hiding this comment.
[nit] The comment says device names are rejected on every OS "so a manifest is valid or invalid the same way everywhere". File entries, though, go through the OS-specific filepath.IsLocal. On Windows it also rejects colons, backslash .. and reserved names in any element. So references/aux.md, notes:v2.md or a\..\..\x pass on Linux and macOS, but reject the whole manifest on Windows. Either narrow the comment to skill names, or check file elements with OS-independent rules too: reject \, : and device-name elements on every OS.
There was a problem hiding this comment.
Fixed with OS-independent file rules instead of filepath.IsLocal. Escape detection treats / and \ both as separators and also catches a leading \ and a drive (C:x), so a\..\..\x now fails the manifest on every OS. A colon, a backslash, or a device-name element (references/aux.md, CON/SKILL.md, matched on the stem before the first dot) skips that skill on every OS. The comment now says both rules are the same on every OS.
| return fmt.Errorf("skill name %q is a reserved Windows device name", name) | ||
| } | ||
| for _, file := range meta.Files { | ||
| if !filepath.IsLocal(file) { |
There was a problem hiding this comment.
[nit] filepath.IsLocal also accepts entries that are not a clean file path inside the skill dir:
.andreferences/..resolve to the skill's temp dir itself, soos.WriteFilefails with EISDIR. By then the skills that sort earlier are already installed, and they stay out of the state file becauseSaveStateis never reached../SKILL.mdandreferences//auth.mdare recorded under their raw keys (installer.go:819). Those keys never match the cleaned paths thatdirMatchesRecordswalks (update.go:426), so the installer never recognizes those files as its own.
Also require path.Clean(file) == file && file != ".".
There was a problem hiding this comment.
Fixed: a file has to satisfy path.Clean(file) == file && file != ".", or its skill is skipped. That covers ., references/.., ./SKILL.md, references//auth.md, references/../SKILL.md and "". They are skipped rather than rejected because none of them leaves the skill dir. To get that right I made the escape check depth-aware: references/.. is not an escape, references/../../x is.
| // must not be able to name ../../.bashrc. | ||
| func validateManifestPaths(m *Manifest) error { | ||
| for name, meta := range m.Skills { | ||
| if len(name) > 64 || !skillNamePattern.MatchString(name) { |
There was a problem hiding this comment.
[nit] 64 appears in both the check and the message. Per the named-constants rule in .agents/rules/style-guide-go.md:40, define a constant such as maxSkillNameLength and use it in both places so they can't drift apart.
There was a problem hiding this comment.
Fixed: maxSkillNameLength is used in both the check and the message.
| @@ -0,0 +1 @@ | |||
| * `databricks aitools install` and `databricks aitools update` now reject a skills manifest whose skill names or file paths would write outside the skills directory, and fail before writing any files. ([#7722](https://github.com/databricks/cli/pull/7722)) | |||
There was a problem hiding this comment.
[nit] The fragment names only install and update. aitools list (cmd/aitools/list.go:155-159), install --path (DumpSkillsToPath) and the skills step of apps init (InstallAllSkills) fetch through the same FetchManifest, so they now fail on such a manifest too. Name at least aitools list in the fragment, and in the description's Blast Radius.
There was a problem hiding this comment.
Fixed: the fragment now names install, update and list. The description's Blast Radius also lists install --path and the skills step of apps init.
…ify as MANIFEST_INVALID Addresses review on #7722: - An escaping name or file now wraps ErrUnsafeManifestPath, classified as MANIFEST_INVALID with its own recovery step instead of SKILLS_FETCH_FAILED. - Only real escapes (absolute, drive, too many "..", separators in a name, ".", "..", "", the state file name) fail the manifest. A name outside the Agent Skills format, a device name, or a non-clean file path drops only that skill with a warning, like a min_cli_version mismatch. - The check moved to FetchSkillsManifestWithFallback, so every ManifestSource goes through it. - File rules are OS-independent (both slashes, colons, device-name elements). - Lazy regexp, sorted iteration, maxSkillNameLength constant, wider changelog. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the review, @simonfaltum. Every inline finding is addressed in 51002ef, with replies on each thread. On the two questions: Q1 (whole-manifest rejection): I went with your first option. Only real escapes fail the manifest: a name that is Q2 (where the check lives): I moved it to |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 51002ef. Each finding was verified against the code; simonfaltum may follow up.
Not approving yet: 1 blocking issue(s), tagged [blocking].
This PR checks skill names and file paths in the fetched skills manifest in FetchSkillsManifestWithFallback. An escaping path fails the manifest as MANIFEST_INVALID, and an unportable entry skips its skill. The escape check holds up, the published manifests pass, and the earlier comments are all addressed. One blocking issue remains: a skipped skill counts as removed from the release, so aitools update deletes the user's installed copy, unlike the min_cli_version skip it is meant to match. The rest are smaller points: Windows naming gaps, the warning's name formatting, the recovery wording, and a doc comment.
Accepted answers
- Fail the whole manifest, or skip format violations?: #7722 (comment)
- Where should the check live, and should writes also go through os.OpenRoot?: #7722 (comment)
Comments outside the changed lines
- [nit]
libs/aitools/installer/installer.go:260: The function now validates the manifest. It can returnErrUnsafeManifestPath, and it removes unportable skills from the returned manifest, which every caller (list,update,install --path,apps init) then sees. The doc comment still describes only the fetch and the fallback. Add a sentence on both.
| for _, name := range names { | ||
| if reason := unportableSkill(name, m.Skills[name].Files); reason != "" { | ||
| log.Warnf(ctx, "Skipping skill %s: %s", name, reason) | ||
| delete(m.Skills, name) |
There was a problem hiding this comment.
[blocking] checkManifestPaths deletes an unportable skill from the manifest, so UpdateSkills sees an installed skill that is "not in the manifest". It adds it to Removed and, if the user hasn't changed it, removes it from disk and every agent (update.go:130-143, then removeSkillExposures).
Example: databricks-core is installed, and the next release adds references/aux.md (or ./x.md, or notes:v2.md) to it. aitools update warns and then uninstalls the working copy on every OS, including Linux and macOS, where the file is fine. aitools list also hides the skill, and install --skills databricks-core reports it as "not found".
The reply on the PR says the skip works like the min_cli_version skip. That's true for install, but for that skip update keeps the installed copy and reports it in result.Skipped (update.go:155-159).
Suggest:
- keep only the escape check in
FetchSkillsManifestWithFallback; - apply
unportableSkillwheremin_cli_versionis applied: inresolveSkills, with aSkillErrorfor--skills, and in theUpdateSkillsloop, asSkipped; - add an update test like
TestUpdateSkipsMinCLIVersionSkills. (in reply to [bug fix] aitools: validate skill names and file paths in the fetched skills manifest #7722 (comment))
There was a problem hiding this comment.
Fixed in 02fa672, the way you suggested. FetchSkillsManifestWithFallback now keeps only the escape check. unportableSkill runs where min_cli_version is applied:
resolveSkillsskips the skill with a warning. For--skills, it returns aSkillErrorwith the newReasonUnportableSkill, classifiedMANIFEST_INVALID.- The
UpdateSkillsloop reports the skill inresult.Skippedand keeps the installed copy.
TestUpdateKeepsSkillThatBecomesUnportable installs databricks-sql, then updates to a release that adds references/aux.md. It asserts that the skill is Skipped, not Removed, that its files are still on disk, and that the state still records 0.1.0. TestInstallSkipsUnportableSkill covers the skip and the --skills error. list shows such a skill again, as it does a min_cli_version one.
| return "the name is a reserved Windows device name" | ||
| } | ||
| for _, file := range files { | ||
| if file == "." || path.Clean(file) != file || strings.ContainsAny(file, `\:`) { |
There was a problem hiding this comment.
[nice to have] unportableSkill rejects only unclean paths, \, : and device-name stems, and isWindowsDeviceName (line 168) covers less than Go's own isReservedName. These all pass on every OS:
<>"|?*and control characters (references/what?.md);COM¹²³/LPT¹²³,CONIN$/CONOUT$, and a stem with a trailing space (aux .md);- trailing dots or spaces in an element;
- two files that differ only in case (
SKILL.mdandskill.md).
Linux and macOS install them. On Windows the write fails partway through, before state is saved, or the files alias so that the recorded state stops matching the disk. So "Both rules are the same on every OS" doesn't fully hold, and one such file still breaks the whole install instead of skipping the skill.
Extend the rule, for example with a per-element allowlist like [A-Za-z0-9._-] and no trailing dot plus a check for duplicate names after lowercasing, or narrow the comment and the description.
There was a problem hiding this comment.
Fixed. Each path element must now match [A-Za-z0-9._-]+ and must not end in a dot, and two files that differ only in case skip the skill. That excludes <>"|?*, control characters, COM¹, CONIN$, aux .md and trailing dots and spaces. Each of them has a row in TestCheckManifestPaths, and the case collision has its own test. The published manifests v0.2.32, v0.2.49 and main (1,206 file entries in total) still pass with no skill skipped.
| } | ||
| for _, name := range names { | ||
| if reason := unportableSkill(name, m.Skills[name].Files); reason != "" { | ||
| log.Warnf(ctx, "Skipping skill %s: %s", name, reason) |
There was a problem hiding this comment.
[nit] log.Warnf(ctx, "Skipping skill %s: %s", name, reason) prints a name that just failed validation. For "databricks-core " the warning reads Skipping skill databricks-core : ..., and the trailing space can't be seen. Control or ANSI characters from the manifest also go straight to the terminal. The escape errors above already use %q. Use %q here too.
There was a problem hiding this comment.
Fixed: both skip warnings (install and update) use %q.
| case protos.AitoolsErrorCategoryManifestInvalid: | ||
| host := skillsHost(ctx) | ||
| if errors.Is(err, installer.ErrUnsafeManifestPath) { | ||
| return "No files were written. Set DATABRICKS_SKILLS_REF to an earlier release tag from " + skillsReleasesURL + " and re-run " + rerun + ", and report the manifest from " + host + " at " + cliIssuesURL + "." |
There was a problem hiding this comment.
[nit] In a run that mixes plugin and skills agents, executePlan removes legacy raw skills for kept agents before the skills step. After that step fails, it still installs plugins and records them in .state.json (install.go:836-921). The output then shows "Installed the plugin for N agents." next to "No files were written." Say "No skill files were written." here, and in the changelog fragment.
There was a problem hiding this comment.
Fixed: the step now reads "No skill files were written.", and so does the changelog fragment. It also covers the --skills case for an unportable skill.
…he fetched manifest Addresses the second review on #7722: - FetchSkillsManifestWithFallback keeps only the escape check. unportableSkill now runs where min_cli_version is applied: resolveSkills skips the skill (a SkillError with ReasonUnportableSkill for --skills, MANIFEST_INVALID), and UpdateSkills reports it as Skipped, keeping the installed copy instead of pruning it. - File path elements are limited to [A-Za-z0-9._-] with no trailing dot, and files that differ only in case skip the skill. - Warnings quote the skill name; the recovery step and changelog say "No skill files were written"; the FetchSkillsManifestWithFallback doc comment names the check. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks, @simonfaltum. Everything from the second review is addressed in 02fa672, with replies on each thread. The outside-the-diff nit is done too: the |
simonfaltum
left a comment
There was a problem hiding this comment.
🤖 Automated review from simonfaltum's review agent (pr-review skill) at 02fa672. Each finding was verified against the code; simonfaltum may follow up.
No blocking issues found, approving. 9 non-blocking comment(s) and 2 question(s) below.
This PR fails a fetched skills manifest that names a path outside the skills directory, before anything is written. It skips only a skill whose name or files aren't portable, and update keeps that skill's installed copy. The escape check holds on every write path, every earlier comment is addressed, and CI is green. What remains is non-blocking: the case check misses folder prefixes and file/folder clashes, the new output has no acceptance test, and there are a few smaller wording, ordering and test-layout nits. There are also two questions, on how strict the file-name allowlist is and on the duplicated skip rules.
Questions
- The per-element allowlist
^[A-Za-z0-9._-]+$is stricter than what is portable. A space (not trailing),+,@,(,),{,},,and~are valid file-name characters on Windows, macOS and Linux. A future release withreferences/c++.md,assets/logo@2x.pngor a template folder{{.project_name}}/would make install drop the whole skill on every OS, andupdatewould keep the old copy. Was the narrower set chosen on purpose? Or would you deny only what is actually unportable (<>:"|?*\, control characters, trailing dot or space, device names)? (non-blocking) - The skip rules (portability, experimental,
min_cli_version) now live in two copies:resolveSkillsand theUpdateSkillsloop. Each copy has its own log text, level and order. Would a shared helper that returns the skip reason keep install and update from drifting apart? Or do you prefer to keep them inline? (non-blocking)
Comments outside the changed lines
- [nit]
libs/aitools/installer/update.go:41:Skipped // skills skipped (experimental, version constraint)here, and theresolveSkillsdoc atinstaller.go:535-536, don't mention the new portability skip. Add it to both.
| folded := strings.ToLower(file) | ||
| if other, ok := seen[folded]; ok { | ||
| return fmt.Sprintf("files %q and %q differ only in case", other, file) | ||
| } | ||
| seen[folded] = file |
There was a problem hiding this comment.
[nice to have] unportableSkill lowercases only whole file paths. Folders that differ only in case (refs/a.md and REFS/b.md) pass the check, and so does a name used as both a file and a folder (scripts and scripts/run.py); both return "" from a scratch copy of the helper.
- Folders that differ only in case: on macOS or Windows they merge into one folder.
dirMatchesRecords(update.go:434) then finds a disk path that is not in the state, so the skill is never pruned. - A name used as both a file and a folder:
installSkillToDirrunsWriteFileandMkdirAllon the same path, and the whole install fails. WithRefsandrefs/a.mdthat happens only on case-insensitive filesystems. The skills that sort earlier are then already replaced, butSaveStatenever runs.
Fold the case of every folder prefix too, skip a skill that uses a path as both a file and a folder, and add rows for both cases to TestCheckManifestPaths.
There was a problem hiding this comment.
Fixed in eb44fea. unportableSkill now folds the case of every folder prefix as well as the whole path, and records whether each path is a file or a folder. refs/a.md with REFS/b.md, and Refs with refs/a.md, now skip as "differ only in case". scripts with scripts/run.py, in either order, skips as "is both a file and a folder". TestUnportableSkillPathClashes has rows for each.
| // The manifest arrived and decoded, but lists a path outside the | ||
| // skills directory or a skill in a form it cannot be installed in. | ||
| if _, ok := errors.AsType[*installer.SkillError](err); ok || errors.Is(err, installer.ErrUnsafeManifestPath) { | ||
| return "No skill files were written. Set DATABRICKS_SKILLS_REF to an earlier release tag from " + skillsReleasesURL + " and re-run " + rerun + ", and report the manifest from " + host + " at " + cliIssuesURL + "." |
There was a problem hiding this comment.
[nice to have] .agents/rules/testing.md asks for acceptance tests first for user-visible CLI output. The PR changes no acceptance test. The output a user sees is checked only piecewise in unit tests:
- the new error and its "No skill files were written…" step;
error_category: MANIFEST_INVALIDunder--output json;- the
--skills <unportable>error.
Add a section with a ../../.bashrc manifest, and one with an unportable --skills name, to acceptance/cmd/aitools/telemetry/install-failure or acceptance/experimental/aitools/skills/install-output-json-error. Both directories already stub manifest.json.
There was a problem hiding this comment.
Added acceptance/experimental/aitools/skills/install-unsafe-manifest. It covers:
- the
../../.bashrcmanifest in text and--output json(MANIFEST_INVALIDplus the recovery step), and checks that no skill file was written; - an unportable skill name, which is skipped with the warning while the other skill installs;
--skills=<unportable>with--output json.
|
|
||
| result := make(map[string]SkillMeta, len(candidates)) | ||
| for name, meta := range candidates { | ||
| if reason := unportableSkill(name, meta.Files); reason != "" { |
There was a problem hiding this comment.
[nit] The portability check runs before the experimental filter, here and in update.go:149, and warnings print at the default log level. So an experimental skill with an unportable name or file prints Skipping skill "…" on every install, update and apps init, even for users who never pass --experimental. The min_cli_version skip runs after the experimental filter. Move the portability check after the experimental filter in both places.
| // ErrUnsafeManifestPath reports a skills manifest that names a skill or file | ||
| // outside the skills directory. The command layer classifies it like a | ||
| // manifest that does not parse. | ||
| var ErrUnsafeManifestPath = errors.New("skills manifest names a path outside the skills directory") |
There was a problem hiding this comment.
[nit] The error reads "names a path outside the skills directory", but it also fires for .state.json (a file inside the directory), . and "". A user sees skills manifest names a path outside the skills directory: skill name ".state.json". A wording such as "names an unsafe path" fits every case. Update the matching comment in cmd/aitools/categorize.go.
| // The manifest arrived and decoded, but lists a path outside the | ||
| // skills directory or a skill in a form it cannot be installed in. | ||
| if _, ok := errors.AsType[*installer.SkillError](err); ok || errors.Is(err, installer.ErrUnsafeManifestPath) { | ||
| return "No skill files were written. Set DATABRICKS_SKILLS_REF to an earlier release tag from " + skillsReleasesURL + " and re-run " + rerun + ", and report the manifest from " + host + " at " + cliIssuesURL + "." |
There was a problem hiding this comment.
[nit] For --skills <unportable>, the step says to pin an earlier release and re-run the same command. If the skill always had that name (say databricks_sql), every release fails the same way. If the skill didn't exist earlier, the re-run fails with SKILL_NOT_FOUND. For the SkillError case, suggest leaving the skill out of --skills, and report it against the skills repo.
| if other, ok := seen[folded]; ok { | ||
| return fmt.Sprintf("files %q and %q differ only in case", other, file) |
There was a problem hiding this comment.
[nit] A skill that lists the same path twice (["SKILL.md", "SKILL.md"]) is now skipped with files "SKILL.md" and "SKILL.md" differ only in case. That reason is wrong, and before this change such an entry installed fine. Skip exact duplicates before the case comparison, or give them their own reason.
There was a problem hiding this comment.
Fixed with the same change: a file listed twice is the same path of the same kind, so it no longer counts as a clash and installs as before (row "same file listed twice").
| assert.Equal(t, protos.AitoolsErrorCategoryManifestInvalid, classifyInstallError(err)) | ||
| } | ||
|
|
||
| func TestClassifyUnportableSkill(t *testing.T) { |
There was a problem hiding this comment.
[nit] TestClassifyInstallError (line 29) already has a row for every other SkillError reason, and .agents/rules/testing.md asks for table-driven tests for similar cases. Move this case into that table.
| if isSpecific { | ||
| return nil, &SkillError{Skill: name, Reason: ReasonUnportableSkill, Detail: "cannot be installed: " + reason} | ||
| } | ||
| log.Warnf(ctx, "Skipping skill %q: %s", name, reason) |
There was a problem hiding this comment.
[nit] resolveSkills ranges over the candidates map, so with two unportable skills the Skipping skill warnings print in a different order on each run. .agents/rules/style-guide-go.md asks for sorted order when it reaches logs. The existing min_cli_version warning has the same problem. Iterate slices.Sorted(maps.Keys(candidates)).
… clashes, add acceptance test Addresses the first three comments of the approving review on #7722: - Replace the [A-Za-z0-9._-] allowlist with a denylist of what is actually unportable: control characters, <>:"\|?*, a trailing dot or space, and Windows device names (now including COM1-3 superscripts, CONIN$, CONOUT$, and stems with trailing spaces). c++.md, logo@2x.png and {{.project_name}}/ install again. - Fold the case of every folder prefix, not just whole paths, and skip a skill that uses one path as both a file and a folder. A file listed twice is no longer reported as a case clash. - Add acceptance/experimental/aitools/skills/install-unsafe-manifest: the escaping manifest's text and --output json error (MANIFEST_INVALID) with no file written, the unportable-skill skip warning, and the --skills error. Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the approval, @simonfaltum. eb44fea addresses the first three items: the case and file/folder clashes, the duplicate-file nit that came with them, and the acceptance test. On question 1 (the allowlist): it wasn't a deliberate product choice, and you're right that it was too strict. It's now a denylist of only what is actually unportable: control characters, I'll handle the remaining nits and question 2 (a shared skip helper) in a follow-up so this one can merge. |
|
Commit: Regression test reportSelected 1 changed tests (limit=20, 0 not selected) 1 tests: regression=1 unreleased=0 coverage=0 failing=0
TestAccept/experimental/aitools/skills/install-unsafe-manifest: branch ✅ | main (629a5ed) ❌ | latest (v1.26.0) ❌main (629a5ed): latest (v1.26.0): |
Summary
databricks aitools install,updateandlistjoin every skill name and file path from the fetched skills manifest onto a local directory. Before this change none of them were checked, so a manifest entry could write anywhere the user can (for example~/.bashrc) and the command still reported success.FetchSkillsManifestWithFallback, which every command and everyManifestSourcegoes through, now checks the manifest before any install or update work starts:Both rules are the same on every OS, so a manifest passes, fails or skips the same way everywhere.
Why escapes fail but portability problems only skip:
databricks_sql, or a newreferences/aux.md) can come from a normal upstream release, and released CLIs can't be patched. Skipping that one skill keeps every other skill installable, andupdatekeeps the copy the user already has, exactly as for a skill that needs a newer CLI. A skipped skill writes nothing, so the reasons for the stricter rules (case-insensitive aliasing, Windows trailing dots, device names, state paths thatupdatecan't match) still hold.A failure looks like
skills manifest names a path outside the skills directory: skill "databricks-core" lists file "../../../.bashrc", with the recovery step "No skill files were written. Set DATABRICKS_SKILLS_REF to an earlier release tag … and report the manifest …".--output jsonreportserror_category: MANIFEST_INVALID.The manifest comes from
raw.githubusercontent.com/databricks/databricks-agent-skills/<ref>/manifest.json(orDATABRICKS_SKILLS_BASE_URL). Exploiting this needs control of what is served there, so this is defense in depth for paths that arrive over the network, not a fix for a remote hole on its own.Review order
libs/aitools/installer/source.go:checkManifestPaths,escapesDir,unportableSkill,isWindowsDeviceName.libs/aitools/installer/installer.go: the escape check inFetchSkillsManifestWithFallbackand the skip inresolveSkills.update.go: the skip in theUpdateSkillsloop.errors.go:ErrUnsafeManifestPath,ReasonUnportableSkill.cmd/aitools/categorize.goandrecovery.go:MANIFEST_INVALIDand its recovery step.TestCheckManifestPaths: a table of accepted, skipped and rejected names and files.TestUnportableSkillPathClashes,TestCheckManifestPathsReportsTheFirstSkillInOrder,TestFetchSkillsManifestChecksEverySource(mock source).TestInstallSkipsUnportableSkill(skip, and the--skillserror) andTestUpdateKeepsSkillThatBecomesUnportable(Skipped, not Removed, files and state kept).TestInstallRejectsStateFileSkillNameWithoutTouchingState: the state file is byte-for-byte unchanged after a rejected install.TestClassifyUnsafeManifestPath,TestClassifyUnportableSkill, and twoTestInstallRecoveryrows.acceptance/experimental/aitools/skills/install-unsafe-manifest: the user-visible output. It shows the escape error in text and--output json, with no file written, the skip warning for an unportable skill, and the--skillserror.Evidence
main): a manifest listing../../../../escaped-file.txtmakesInstallSkillsForAgentsreturnniland write$HOME/escaped-file.txt.After: it fails in
FetchSkillsManifestWithFallbackbefore any file is written, classifiedMANIFEST_INVALID.v0.2.32,v0.2.49andmain(35, 38 and 38 skills; 366, 420 and 420 files) pass with no error and no skipped skill.go test ./libs/aitools/... ./cmd/aitools/..., the 27aitoolsacceptance tests,GOOS=windows go veton both packages, and./task lint-qall pass.Merge Danger
Door: two-way. Rollback: revert the PR.
Blast Radius: narrow.
Affects
aitools install,update,list,install --pathand the skills step ofapps init, all of which fetch throughFetchSkillsManifestWithFallback. A manifest with an escaping entry now fails these commands instead of writing. A skill whose name or files are not portable is skipped with a warning,--skillsnaming it fails, andupdatekeeps its installed copy. No published manifest is affected.This pull request and its description were written by Isaac.