Skip to content

Add the Session API crate - #1593

Open
MaxHeimbrock wants to merge 52 commits into
mainfrom
max/session-api
Open

MaxHeimbrock wants to merge 52 commits into
mainfrom
max/session-api

Conversation

@MaxHeimbrock

@MaxHeimbrock MaxHeimbrock commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

New livekit-session crate with Session. Similar to the other session APIs in https://docs.livekit.io/frontends/build/sessions/.

Look at the example or my PR in the Rust dev client for usage.

Disclaimer: I mostly used Claude to write this. I am still not a Rust expert. I tried to find wholes as much as possible, look at the developer facing APIs and test it myself through the Rust dev client.

Decisions I took:

The platform audio is activated for the user by default, opt out via the session options (with_platform_audio = false).

If you pass a cached token source, you will run into issues reusing the same token. This is a footgun we accept, since invalidating tokens after use is not easy to expose in the current design. Also fixed token sources are allowed, which cause the same problem.

We don't plan to expose this Rust session via FFI for now.

Using the actor pattern according to AGENT.md.

Public API

Every enum is #[non_exhaustive], and so are SessionState and Message. SessionOptions and Agent have private fields. A Message for restore_messages is built with Message::new.

Item Kind Purpose
Session struct, Clone, Debug A conversation with an agent. Handles share one session; dropping the last one closes the room
Session::new(token_source, options) constructor Credentials from a TokenSourceFixed, e.g. literal
Session::with_fetch_options(token_source, fetch_options, options) constructor Credentials from a TokenSourceConfigurable with full TokenSourceFetchOptions
Session::with_agent(token_source, agent_name, options) constructor Shorthand for with_fetch_options that dispatches agent_name
Session::start(&self) -> Result<(), SessionStartError> async Fetches credentials, connects, and publishes the platform microphone; resolves at room connect, not agent connect
Session::end(&self) async Closes the room, releasing the audio devices, or cancels a pending start; idempotent
Session::state(&self) -> watch::Receiver<SessionState> method Current state (borrow) and changes (changed, wait_for). The agent and its tracks are read from here
Session::room(&self) -> Option<Arc<Room>> method The room while connected or reconnecting, for what the session does not do itself, e.g. publishing a camera
Session::send_text(&self, text: &str) -> Result<Message, MessageSendError> async Sends chat on lk.chat and adds it to the feed once sent
Session::messages(&self) -> Vec<Message> method The feed, ordered by seq
Session::restore_messages(&self, messages: impl IntoIterator<Item = Message>) method Replaces the feed in the given order, renumbered from zero. Takes the messages by value
Session::subscribe(&self) -> mpsc::UnboundedReceiver<SessionEvent> method Changes to the message feed
SessionOptions struct, Default Private fields, set through the builders with_agent_connect_timeout (20 s default), with_room_options, and with_platform_audio (true by default: the microphone is published and the agent's audio played)
SessionState { connection, agent } struct, Clone, Default Snapshot published through state(). Not Copy: it holds the agent's tracks, so a kept value keeps them alive
SessionConnectionState enum, Copy Disconnected, Connecting, Connected, Reconnecting; is_connected()
AgentState enum, Clone Disconnected, Connecting, Joined(Agent), Failed(AgentFailure); is_connected(), is_pending(), is_finished(), and identity(), activity(), audio_track(), avatar_video_track() to read the agent whichever variant holds it
Agent struct, Clone Snapshot of the agent in the room: identity(), activity(), audio_track() (its microphone, once subscribed), avatar_video_track() (the avatar worker's camera, once subscribed). Equality compares identity, activity and track sids
AgentActivity enum, Copy Initializing, Idle, Listening, Thinking, Speaking, Unknown; from the lk.agent.state attribute
AgentFailure enum Timeout { agent: Option<Agent> } (the slow agent, if one is in the room), Left
SessionEvent enum MessageUpserted(Message), MessagesRestored
Message { id, seq, timestamp, origin, participant_identity, text, is_final } struct One feed entry; identified by id + participant_identity. Message::new(id, origin, participant_identity, text) creates a final message timestamped now; builders with_timestamp, with_is_final
MessageOrigin enum, Copy AgentTranscript, UserTranscript, UserInput, Chat
SessionStartError error AlreadyStarted, Cancelled, Token(TokenSourceError), Connect(RoomError), PlatformAudio(AudioError) (no audio devices), Microphone(RoomError) (the microphone failed to publish; the room is closed again)
MessageSendError error NotConnected, Stream(StreamError)

Commits

Group 1. Crate scaffolding

  1. feat(session): add the livekit-session crate manifest and module layout
  2. chore(workspace): register livekit-session with cargo, knope and CODEOWNERS
  3. IMPORTANT(session): add the agent interop constants
  4. feat(session): add the start and send error types

Group 2. Message feed
5. feat(session): add the Message model and the MessageStore feed
6. test(session): cover MessageStore ordering, upsert and restore
7. IMPORTANT(session): aggregate lk.transcription streams into transcript messages
8. test(session): cover transcript aggregation

Group 3. Agent projection
9. feat(session): add AgentState and AgentFailure
10. feat(session): find the agent, its avatar worker and their tracks
11. IMPORTANT(session): project AgentState with timeout and leave detection
12. test(session): cover agent discovery and state parsing
13. test(session): cover agent projection across reconnects, leaves and timeouts

Group 4. Session runtime
14. feat(session): add the Dispatcher fan-out
15. test(session): cover Dispatcher pruning
16. feat(session): add the state shared between a session and its actor
17. feat(session): add SessionOptions, SessionState and SessionEvent
18. feat(session): add the Session handle and its constructors
19. IMPORTANT(session): add the session actor and its agent-state projection
20. IMPORTANT(session): read chat and transcript streams in the actor
21. IMPORTANT(session): start and end a session
22. feat(session): expose state, room, agent tracks and the message feed
23. test(session): assert with_agent fetches with the agent name

Group 5. End-to-end tests
24. test(session): add the e2e harness with a stand-in agent
25. test(session): e2e agent state follows joins, leaves and reconnects
26. test(session): e2e agent connect timeout
27. test(session): e2e start publishes the platform microphone
28. test(session): e2e the feed merges chat and transcripts
29. test(session): e2e agent audio and avatar video tracks
30. test(session): e2e restart after a disconnect and a rejected token
31. test(session): e2e end and drop cancel a pending start
32. test(session): e2e a start cancelled during the handshake leaves nothing behind
33. test(session): e2e memory lifecycle

Group 6. Docs, example and changeset
34. docs(session): add the crate README
35. feat(example): add the session voice and text chat example
36. chore(session): add the changeset

@MaxHeimbrock MaxHeimbrock changed the title livekit-session: add the Session API crate Add the Session API crate Oct 28, 2026
@MaxHeimbrock
MaxHeimbrock added this pull request to stack #1598 October 29, 2026 14:57
@github-actions

github-actions Bot commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Changeset ✓

This PR includes a changeset covering all affected packages:

Package Bump
livekit-session patch

@xianshijing-lk
xianshijing-lk requested review from 1egoman and removed request for 1egoman October 30, 2026 23:18
@xianshijing-lk

This comment was marked as resolved.

@MaxHeimbrock
MaxHeimbrock removed this pull request from stack #1598 November 2, 2026 10:15
@MaxHeimbrock
MaxHeimbrock changed the base branch from max/token-source-invalidate-dispatch to max/room-restart-event-order November 2, 2026 10:15
@MaxHeimbrock
MaxHeimbrock added this pull request to stack #1621 November 2, 2026 10:15
@MaxHeimbrock

This comment was marked as resolved.

@MaxHeimbrock
MaxHeimbrock force-pushed the max/session-api branch 2 times, most recently from 99c83dc to 5f9d744 Compare November 3, 2026 17:08
@MaxHeimbrock
MaxHeimbrock marked this pull request as ready for review November 3, 2026 17:09
@MaxHeimbrock
MaxHeimbrock requested a review from ladvoc as a code owner November 3, 2026 17:09
devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread livekit-session/src/agent.rs Resolved
Comment thread livekit-session/src/session.rs Resolved
Comment on lines +239 to +244
/// options, such as agent metadata.
pub fn with_agent(
token_source: impl TokenSourceConfigurable + Send + Sync + 'static,
agent_name: &str,
session_options: SessionOptions,
) -> Self {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: It could be interesting to expose a builder type api here maybe? Session::new_configurable(configurable token source).with_agent("agent name").with_options(...) or something...

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree, I am of the opinion functions with more than two arguments should be avoided in Rust since there are no argument labels which creates ambiguity at the call site.

Comment on lines +436 to +443
/// A receiver of changes to the message feed.
///
/// Unbounded, like [`Room::subscribe`]: the session must never block on a
/// slow subscriber, and the feed's rate is bounded by the conversation.
/// Drop the receiver to unsubscribe; it is removed on the next event.
pub fn subscribe(&self) -> mpsc::UnboundedReceiver<SessionEvent> {
self.inner.shared.subscribe()
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: very interesting, this could play well into some stuff blaze has been talking about to make uniffi events more stream based...

Comment thread livekit-session/src/session.rs Outdated
Comment on lines +504 to +506
/// What the agent is doing.
pub agent: AgentState,
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thought: We can talk about this in our 1:1, but some original thinking was done back in the day to try to make session able to support multiple agents in the future with minimal updates. It would be nice to try to carry over those decisions in here as well.

Comment thread AGENTS.md Resolved

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

}
// Dropped while the actor was publishing the microphone. Dropping its
// `stop` sender stops it.
drop(lifecycle.take_running(self.generation));

@devin-ai-integration devin-ai-integration Bot Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Cancelled starts detach room tasks

Dropping start() during microphone publication drops the Running task handle without joining it. The actor closes in the background, but no owner observes its completion or failure.

Learn more

A running session keeps the actor's JoinHandle in Running, and Running::join normally drops the stop sender and awaits that handle. Here take_running removes the only handle, then drops it without awaiting. Tokio detaches the actor, so no owner can observe a failed shutdown or guarantee a join. The same unowned task path occurs when the lifecycle changes after spawning the actor.

Example: A caller times out session.start() while the microphone publication is pending. Its guard signals stop but discards the task handle; the actor closes the room asynchronously without a joiner.

Recommended fix: Arrange for both cancellation paths to transfer Running to a named shutdown owner that awaits Running::join, while preserving background teardown when start() is dropped. Add a lifecycle test that verifies the shutdown owner joins the task.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@MaxHeimbrock

This comment was marked as outdated.

devin-ai-integration[bot]

This comment was marked as resolved.

This commit adds the first end-to-end tests of the agent projection.

The tests show that a started session reports the agent once it
joins, that `end` disconnects and is idempotent, that an agent that
leaves is reported as `Left`, and that a full reconnect is not
reported as `Left`.
This commit adds the end-to-end tests of the agent-connect timeout.

The tests show that the timeout fires without an agent, that a late
agent recovers the state, that the timeout is armed without an agent
dispatch, and that the timeout fires for an agent stuck in
`initializing`.
This commit adds the end-to-end test of the default platform audio.

The test shares the platform-audio handle with the session. It shows
that `start` publishes the microphone track and that `end` releases
the audio devices. The test skips on a machine without a microphone.
This commit adds the end-to-end tests of the message feed.

The first test shows that user input reaches the agent on `lk.chat`,
that an agent transcript grows in place and is finalized by the
trailer, that chat from the agent joins the feed, and that a failed
send adds nothing.

The second test shows that the session leaves streams on other topics
to the app.
This commit adds the end-to-end tests of the agent media accessors.

The helpers publish a silent audio track or a blank video track from
the agent room.

The tests show that only the microphone is the agent's audio, that
only the avatar worker's camera is the avatar video, and that an
avatar worker of kind agent is not taken for the agent.
This commit adds two end-to-end tests of the start path.

The first test removes the user from the room through the server API.
It shows that the session reports `Disconnected`, that a new `start`
releases the old room, and that `end` after a disconnect is a no-op.

The second test shows that a rejected token fails `start` with
`Connect` and leaves the session idle.
This commit adds the end-to-end tests of a start that waits on the
token source.

`PendingSource` never answers. The tests show that `end` cancels the
start with `Cancelled`, and that a dropped `start` future resets the
session.
…ing behind

This commit adds the end-to-end tests of a start that waits in
`Room::connect`.

`SilentServer` accepts one connection and never answers. The tests
cancel the start with `end` or by a drop of the future. They show
that the client hung up, that no actor was spawned, and that the task
count returns to the baseline.
This commit adds the end-to-end lifecycle tests that AGENTS.md asks
for.

The tests show that dropping the last `Session` handle closes the
room, that repeated start and end cycles leave no task or subscriber
behind, and that failed starts spawn nothing.

Each test checks observable destruction: a `Weak` that no longer
upgrades, a strong count of one, or the task count of the runtime.
This commit adds the README of the crate.

The README gives a usage example and explains the credentials, the
lifecycle, the state table, the message feed, and the things the
session owns.
This commit adds `examples/session` and registers it in the
workspace.

The example creates a session from a pre-generated token or from a
development token server. It prints the session state and the
completed messages. Lines typed on stdin go to the agent.

By default, the example uses the platform audio. The `--text` flag
turns it off.

`Cargo.lock` adds the lock entry of the example.
This commit adds the knope changeset for the new crate.

The summary is one line of prose, as the "Documenting changes"
section of AGENTS.md requires.
…d avatar track as well as the participant identity
Fix test after debug session reorder

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 3 new potential issues.

4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment on lines +339 to +342
let connection = tokio::select! {
connection = self.connect() => connection,
_ = cancel_rx => Err(SessionStartError::Cancelled),
};

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 Cancelled startup leaks signaling connections

After the WebSocket opens, cancelling start drops the connection future while it can still await the join response. SignalStream has no drop cleanup; its reader and writer retain each other through their channel. The session reports disconnection without closing those tasks or their socket.

Learn more

Cancellation must clean up transports created before a complete room exists. SignalInner::connect opens the WebSocket and then awaits the join response through close_on_err. That helper closes the stream when its cancellation token fires, but dropping the helper bypasses that cleanup. SignalStream::spawn starts a reader that retains the writer's channel sender, while both tasks retain the connection. Dropping their join handles detaches them rather than terminating them. The existing silent-server tests cancel before the WebSocket handshake completes, before these tasks exist.

Example: A server completes the WebSocket upgrade, and the user ends the session before its join response arrives. The session returns Cancelled, but the detached transport tasks continue until the peer closes the connection.

Recommended fix: Make the room connection path cancellation-safe across transport establishment and ICE negotiation. Propagate cancellation to the connection owner and await its cleanup instead of simply dropping its future. Also cover dropping the public start future with cleanup ownership that survives cancellation. Add regression tests that complete the WebSocket handshake before cancellation and assert transport closure and task counts returning to baseline.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +207 to +212
if !self.partials.contains_key(&key) {
// A new segment means the participant's previous turn is over.
self.partials.retain(|(segment, participant), _| {
participant != &stream.participant_identity || segment == &segment_id
});
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Buffered transcripts discard unfinished segments

on_chunk discards an unfinished segment when another buffered segment from the same participant starts. SessionActor interleaves readers, so the earlier segment can still have unread chunks and a pending final event. Its transcript loses text or never becomes final.

Learn more

A new segment does not imply that the previous segment's reader has been drained. SessionActor::on_text_stream inserts independent readers into SelectAll, which yields their events independently rather than draining streams in opening order. Consequently, even streams sent sequentially can interleave when both are buffered. Removing an earlier partial makes its remaining chunks start a new partial, and on_stream_end cannot finalize a partial that another segment removed.

Example: One agent sends segment A with chunks Hel and lo, followed by segment B with World. If the actor consumes A's first chunk, B's first chunk, then A's second chunk, the feed changes A from Hel to lo instead of Hello. B's partial also disappears before its final event.

Recommended fix: Retain partials for every active segment until its reader completes. Track reader completion separately from interim-segment retention so abandoned interim results can still be cleaned up without deleting active readers' state. Add coverage with multiple fully buffered transcript readers, checking complete text and final events for every segment.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Comment on lines +429 to +432
let end = stream::once(future::ready(ReaderEvent::TranscriptEnd(transcript.clone())));
let chunks =
reader.map(move |chunk| ReaderEvent::TranscriptChunk(transcript.clone(), chunk));
self.readers.push(chunks.chain(end).boxed());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Failed transcripts marked complete

When a transcript fails without lk.transcription_final=false, on_reader_event later finalizes its partial text. TranscriptEnd follows reader errors too, and the error handler retains the partial. Consumers receive truncated transcripts as complete messages.

Learn more

The completion event does not distinguish clean exhaustion from a failed reader. on_reader_event logs errors without invalidating the accumulated partial, then processes the chained completion event. TranscriptAggregator::on_stream_end finalizes that partial unless the attributes explicitly say false. The incoming stream manager can terminate a reader with errors such as an abnormal end or missing bytes through close_stream_with_error.

Example: A transcript with no final attribute delivers Please transfer, then fails before delivering the rest. The session emits that prefix with is_final=true, although the transfer never completed.

Recommended fix: Carry a clean-completion result into TranscriptEnd, or track failed reader identities and prevent their end events from finalizing partials. Preserve incomplete text without claiming successful completion. Test a successful chunk followed by a terminal read error with the final attribute absent and with it set to true.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants