Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
43 commits
Select commit Hold shift + click to select a range
c826639
Log AitoolsMcpEvent from aitools mcp and the serve proxy
simonfaltum Nov 12, 2026
dd69f36
Create the agent config dirs in the telemetry acceptance cases
simonfaltum Nov 12, 2026
be6a80c
Record the sign-in prompt only when a page URL was shown
simonfaltum Nov 13, 2026
bd5b8a2
Keep one set of login steps on an MCP authentication failure
simonfaltum Nov 13, 2026
b421db0
Resolve the remove upload's auth after the local edit
simonfaltum Nov 13, 2026
80fe25b
Log serve's start after its checks, and keep the failed start's upload
simonfaltum Nov 13, 2026
e97355c
Log only events with a reached server, and count profiles by config file
simonfaltum Nov 13, 2026
a5eca7c
Derive the telemetry service enum from the known system.ai table
simonfaltum Nov 13, 2026
73ce890
Tidy the telemetry acceptance scripts and the fake telemetry endpoint
simonfaltum Nov 13, 2026
de8a655
Document the error categories of the server result and tidy its test
simonfaltum Nov 13, 2026
3c2b655
Authenticate telemetry uploads without refreshing a token
simonfaltum Nov 13, 2026
5bd3941
Resolve remove's telemetry auth only after it logs an event
simonfaltum Nov 13, 2026
0f0e1a0
Resolve add, login and logout telemetry auth as install does
simonfaltum Nov 13, 2026
dd1c115
Log PROXY_STARTED only for a serving start
simonfaltum Nov 13, 2026
9b947c7
Tidy the MCP telemetry: sign-in and record-only states, service enum,…
simonfaltum Nov 13, 2026
942533a
Upload telemetry with a config the command sets
simonfaltum Nov 13, 2026
0cf8065
Authenticate MCP telemetry uploads with the profile each command binds
simonfaltum Nov 13, 2026
a5aafa2
Record sign-in prompts of waits on a cancelled page, and test the ref…
simonfaltum Nov 13, 2026
7e155fe
Tidy the MCP telemetry: one SIGN_IN builder, rejected-start test, doc…
simonfaltum Nov 13, 2026
68debe4
Refuse the MCP telemetry upload after the service rejects the credential
simonfaltum Nov 13, 2026
380ca19
Keep the telemetry upload's host metadata failures out of the command…
simonfaltum Nov 13, 2026
f016d6e
Group the MCP telemetry profile count as mcp list does, and tidy the …
simonfaltum Nov 13, 2026
4016b82
Read the removed servers through jsonServers in the telemetry tests
simonfaltum Nov 16, 2026
8548dbf
Regenerate the serve telemetry golden for the Error prefix
simonfaltum Nov 16, 2026
f2b7617
Adapt telemetry to the failure helper from card 04
simonfaltum Nov 17, 2026
d32c9a2
Adapt the name-conflict check to the telemetry error enum
simonfaltum Nov 17, 2026
26ab9b2
Regenerate goldens for the GovCloud GitHub app in the service note
simonfaltum Nov 17, 2026
20eb03a
Move the debug-only log handler to libs/log/handler
simonfaltum Nov 17, 2026
17fc972
Demote the registered host metadata warning of the telemetry upload
simonfaltum Nov 17, 2026
8d4081a
Guard the telemetry logger's refusal flag and events with a mutex
simonfaltum Nov 17, 2026
0998aa5
Correct the serve and add test comments on the upload config
simonfaltum Nov 17, 2026
628dabc
Lock the telemetry events read by BufferedEvents and Upload
simonfaltum Nov 17, 2026
4455ed3
Test the debug-only log handler
simonfaltum Nov 17, 2026
f9a7f6b
Shorten the host metadata comment and fix the offline remove test com…
simonfaltum Nov 17, 2026
2afef51
Use the test context in the debug-only handler test
simonfaltum Nov 17, 2026
547699e
Adapt card 10a to the --names refusal from card 05
simonfaltum Nov 17, 2026
1b5371e
aitools mcp add --login: log a failed sign-in with a PENDING state wh…
simonfaltum Nov 17, 2026
4b31731
aitools mcp telemetry: an entry already gone is CHANGED, not SKIPPED
simonfaltum Nov 17, 2026
44b16f3
aitools mcp remove telemetry: group the upload by profile and config …
simonfaltum Nov 17, 2026
a79f3d7
aitools mcp serve: set the upload config at start, so the upload uses…
simonfaltum Nov 17, 2026
62401bc
Adapt card 10a to the remove changes from card 05
simonfaltum Nov 17, 2026
7df6abe
aitools mcp: upload telemetry after a rejected credential
simonfaltum Nov 17, 2026
9910702
aitools mcp serve: drop the exit upload when the profile was re-pointed
simonfaltum Nov 17, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
not = [valid
1 change: 1 addition & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
94 changes: 94 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,94 @@

=== A new install for two agents: CHANGED per (service, agent); github needs sign-in, so its sign-in state is PENDING

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GITHUB",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "PENDING"
},
{
"service": "GITHUB",
"agent": "CODEX",
"outcome": "CHANGED",
"sign_in_state": "PENDING"
}
],
"bound_profile_count": 1
}

=== The same servers again are re-asserted: UNCHANGED

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GITHUB",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "PENDING"
},
{
"service": "GITHUB",
"agent": "CODEX",
"outcome": "UNCHANGED",
"sign_in_state": "PENDING"
}
],
"bound_profile_count": 1
}

=== An agent config that does not parse fails that agent only: FAILED with AGENT_CONFIG_INVALID

>>> musterr [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code,codex --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "UNCHANGED",
"sign_in_state": "NOT_REQUIRED"
},
{
"service": "GENIE_ONE",
"agent": "CODEX",
"outcome": "FAILED",
"error_category": "AGENT_CONFIG_INVALID",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 1
}
22 changes: 22 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-add/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
sethome home
mkdir -p home/.claude home/.codex
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

# Seeding the fake is not an MCP command, so its uploads (if any) are dropped here.
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=github" --profile logfood --json '{"comment":"Repositories, issues, pull requests","config":{"source_connection":{"name":"connections/github","options":{"oauth_provider":"github"}}}}' > /dev/null
rm -f "$OUT_REQUESTS"

title "A new install for two agents: CHANGED per (service, agent); github needs sign-in, so its sign-in state is PENDING"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json > /dev/null
mcp_events

title "The same servers again are re-asserted: UNCHANGED"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp,system.ai.github --agents claude-code,codex --output json > /dev/null
mcp_events

title "An agent config that does not parse fails that agent only: FAILED with AGENT_CONFIG_INVALID"
cp codex-config.toml home/.codex/config.toml
trace musterr $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code,codex --output json > /dev/null
mcp_events
7 changes: 7 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood

[dogfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
95 changes: 95 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@

=== Installed for two profiles: the bound profile count is 2 after the second add

>>> [CLI] aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 1
}

>>> [CLI] aitools mcp add --profile dogfood --names system.ai.genie_one_mcp --agents claude-code --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "ADD",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED",
"sign_in_state": "NOT_REQUIRED"
}
],
"bound_profile_count": 2
}

=== remove --profile dogfood: CHANGED for its agent entry, and one profile stays bound

>>> [CLI] aitools mcp remove --profile dogfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED"
}
],
"bound_profile_count": 1
}

=== An entry changed outside this CLI is left in place: SKIPPED

>>> [CLI] aitools mcp remove --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "SKIPPED"
}
],
"bound_profile_count": 1
}

=== An entry already gone from the agent config is CHANGED, and the state record goes

>>> [CLI] aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "GENIE_ONE",
"agent": "CLAUDE_CODE",
"outcome": "CHANGED"
}
],
"bound_profile_count": 0
}

=== A record with no agent left goes with a REMOVE of its own: CHANGED with no agent, and no profile bound

>>> [CLI] aitools mcp remove --names dbsql-logfood --profile logfood --output json
{
"calling_agent": "TYPE_UNSPECIFIED",
"operation": "REMOVE",
"servers": [
{
"service": "DBSQL",
"outcome": "CHANGED"
}
],
"bound_profile_count": 0
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"schema_version": 3,
"mcp": {
"servers": [
{
"name": "dbsql-logfood",
"service": "system.ai.dbsql",
"profile": "logfood",
"host": "$DATABRICKS_HOST",
"workspace_id": "",
"sign_in_required": false,
"command": "",
"config_file": "$PWD/.databrickscfg",
"agents": {},
"installed_at": "2026-01-01T00:00:00Z",
"cli_version": "0.0.0"
}
]
}
}
33 changes: 33 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-remove/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
sethome home
mkdir -p home/.claude
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
rm -f "$OUT_REQUESTS"

title "Installed for two profiles: the bound profile count is 2 after the second add"
trace $CLI aitools mcp add --profile logfood --names system.ai.genie_one_mcp --agents claude-code --output json > /dev/null
mcp_events
trace $CLI aitools mcp add --profile dogfood --names system.ai.genie_one_mcp --agents claude-code --output json > /dev/null
mcp_events

title "remove --profile dogfood: CHANGED for its agent entry, and one profile stays bound"
trace $CLI aitools mcp remove --profile dogfood --output json > /dev/null
mcp_events

title "An entry changed outside this CLI is left in place: SKIPPED"
jq '.mcpServers["genie_one_mcp-logfood"].command = "/usr/bin/other"' home/.claude.json > home/claude.tmp && mv home/claude.tmp home/.claude.json
trace $CLI aitools mcp remove --profile logfood --output json > /dev/null
mcp_events

title "An entry already gone from the agent config is CHANGED, and the state record goes"
jq 'del(.mcpServers["genie_one_mcp-logfood"])' home/.claude.json > home/claude.tmp && mv home/claude.tmp home/.claude.json
trace $CLI aitools mcp remove --names genie_one_mcp-logfood --profile logfood --output json > /dev/null
mcp_events

title "A record with no agent left goes with a REMOVE of its own: CHANGED with no agent, and no profile bound"
mkdir -p home/.databricks/aitools/skills
envsubst < record-only.state.json > home/.databricks/aitools/skills/.state.json
trace $CLI aitools mcp remove --names dbsql-logfood --profile logfood --output json > /dev/null
mcp_events
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
1 change: 1 addition & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/out.test.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
44 changes: 44 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/output.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@

=== serve logs PROXY_STARTED after the binding checks; the upload is sent when stdin closes

>>> [CLI] aitools mcp serve --profile logfood --service system.ai.genie_one_mcp --agent claude-code
No install record for system.ai.genie_one_mcp on profile "logfood"; using its current host [DATABRICKS_URL]
{
"calling_agent": "CLAUDE_CODE",
"operation": "PROXY_STARTED",
"servers": [
{
"service": "GENIE_ONE"
}
],
"bound_profile_count": 0
}

=== A binding that fails logs no PROXY_STARTED, so no upload follows

>>> musterr [CLI] aitools mcp serve --profile gone --service system.ai.genie_one_mcp --agent codex
Error: MCP server system.ai.genie_one_mcp needs Databricks profile "gone", which does not exist in [TEST_TMP_DIR]/.databrickscfg. Set it up with: databricks auth login --profile gone

=== authenticate opens the sign-in page: PROXY_STARTED, then SIGN_IN_PROMPTED with the sign-in state after the wait
{
"calling_agent": "CLAUDE_CODE",
"operation": "PROXY_STARTED",
"servers": [
{
"service": "GITHUB"
}
],
"bound_profile_count": 0
}
{
"calling_agent": "CLAUDE_CODE",
"operation": "SIGN_IN_PROMPTED",
"servers": [
{
"service": "GITHUB",
"error_category": "SIGN_IN_REQUIRED",
"sign_in_state": "ACTIVE"
}
],
"bound_profile_count": 0
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
{"jsonrpc":"2.0","id":4,"method":"tools/list"}
{"jsonrpc":"2.0","id":5,"method":"tools/call","params":{"name":"echo","arguments":{"text":"hello"}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{"jsonrpc":"2.0","id":3,"method":"tools/call","params":{"name":"authenticate","arguments":{}}}
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"test-client","version":"0.0.1"}}}
{"jsonrpc":"2.0","method":"notifications/initialized"}
{"jsonrpc":"2.0","id":2,"method":"tools/list"}
32 changes: 32 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-serve/script
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
sethome home
envsubst < .databrickscfg > out && mv out .databrickscfg
export DATABRICKS_CONFIG_FILE=.databrickscfg

# No browser: the proxy prints the sign-in URL and does not open anything.
export BROWSER=none
export RETRY_MAX_ATTEMPTS=60
# Seeding the fake is not an MCP command, so its uploads (if any) are dropped here.
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=genie_one_mcp" --profile logfood --json '{"comment":"Ask questions about your data"}' > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post "/api/2.1/unity-catalog/mcp-services?parent=schemas/system.ai&mcp_service_id=github" --profile logfood --json '{"comment":"Repositories, issues, pull requests","config":{"source_connection":{"name":"connections/github","options":{"oauth_provider":"github"}}}}' > /dev/null
rm -f "$OUT_REQUESTS"

title "serve logs PROXY_STARTED after the binding checks; the upload is sent when stdin closes"
trace $CLI aitools mcp serve --profile logfood --service system.ai.genie_one_mcp --agent claude-code < requests-list.ndjson > /dev/null
mcp_events

title "A binding that fails logs no PROXY_STARTED, so no upload follows"
trace musterr $CLI aitools mcp serve --profile gone --service system.ai.genie_one_mcp --agent codex < /dev/null > /dev/null
mcp_events

title "authenticate opens the sign-in page: PROXY_STARTED, then SIGN_IN_PROMPTED with the sign-in state after the wait"
{
cat requests-list.ndjson
retry --until '"id":2' cat serve.stdout.txt > /dev/null
cat requests-authenticate.ndjson
retry --until "Waiting for sign-in" cat serve.stderr.txt > /dev/null
MSYS_NO_PATHCONV=1 $CLI api post /api/2.1/unity-catalog/mcp-services/system.ai.github/user-credentials --profile logfood --json '{}' > /dev/null
retry --until '"id":3' cat serve.stdout.txt > /dev/null
cat requests-after.ndjson
} | $CLI aitools mcp serve --profile logfood --service system.ai.github --agent claude-code > serve.stdout.txt 2> serve.stderr.txt
mcp_events
rm serve.stdout.txt serve.stderr.txt
3 changes: 3 additions & 0 deletions acceptance/cmd/aitools/telemetry/mcp-sign-in/.databrickscfg
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[logfood]
host = $DATABRICKS_HOST
token = test-token-logfood
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
Loading
Loading