Skip to content

feat(simulate): say when a run starts that Agent Observability is off - #1134

Open
u9g wants to merge 2 commits into
mainfrom
jason/sims-observability-notice
Open

u9g wants to merge 2 commits into
mainfrom
jason/sims-observability-notice

Conversation

@u9g

@u9g u9g commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

lk agent simulate now says when a run starts that the project's Agent Observability is off. Until now, users found out only at the end, when the run had no summary and the message called it "user data recording".

Context: a new Simulations customer left Observability off because of that wording, then asked for metrics the dashboard already has (Slack: #sims feedback, Linear OCD-3768).

What changes

  • TUI: a ⚠ notice with the project's settings link sits under the job list while the run is going. When the run ends without a summary, the reason (wrapped now, since it's longer) is followed by Turn on Agent Observability: <link>.
  • CI / plain output: one ::warning:: annotation with the link at setup, so it shows in the GitHub run summary and isn't hidden in the collapsed Setup group. The end-of-run "no summary" line is now a ::warning:: too.
  • The link uses the project's own dashboard, so staging projects link to staging. Without a project ID, the notice points to the project's Observability settings instead.

How the CLI knows: the new observability_disabled field on SimulationRun.Create.Response, set by agent-service. The CLI doesn't parse error text, and it can't read project settings itself under API-key auth, which is the default.

Blocked on livekit/protocol#2002 (new field) merging and a github.com/livekit/protocol bump here. This branch doesn't compile against the currently pinned protocol. It was built and tested with a local replace pointing at the protocol branch. go test ./cmd/lk/ passes except TestDeployPrebuiltImageUnderUserAuth, which also fails on main on this machine (no docker-credential-osxkeychain).

Commits: a pure refactor (createSimulationRun returns the whole response), then the feature.

Related: livekit/protocol#2002 (field), livekit/agents-private#1010 (sets the field), livekit/agents-private#1009 (rewrites the no-summary message).

@u9g

u9g commented Nov 19, 2026

Copy link
Copy Markdown
Contributor Author

The PR-branch CLI against a local fake of the simulation API that returns observability_disabled and a finished run with agent-service's new no-summary message. First CI output (::warning:: with the settings link at setup and at the end), then the TUI (notice under the jobs while running, link under the no-summary message at the end).

sims-observability-notice.mp4

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant