Skip to content

fix(voice): create room IO media streams off the event loop - #7502

Open
muntazir868 wants to merge 6 commits into
livekit:mainfrom
muntazir868:fix/7441-offloop-stream-create
Open

muntazir868 wants to merge 6 commits into
livekit:mainfrom
muntazir868:fix/7441-offloop-stream-create

Conversation

@muntazir868

@muntazir868 muntazir868 commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes #7441 (RoomIO half).

Rebased onto current main. The inference STT log-level change that used to be bundled here is now split out into #7918, so this PR only touches the RoomIO stream-creation path.

Problem

AudioStream.from_track / VideoStream.from_track run a synchronous FFI request, and _on_track_available called it directly from the track_subscribed handler. Initializing noise cancellation (e.g. BVC()) blocked the loop for 106–119ms and tripped the loop monitor.

Changes

  • _create_stream is now async; the audio/video implementations run from_track via asyncio.to_thread and pass loop= explicitly so the stream still dispatches frames onto the job loop.
  • Creation runs in a task serialized by a lock, with a generation counter: a track event that supersedes an in-flight creation discards the pending stream instead of installing it, and _close_stream defers processor cleanup to that task while it settles.
  • In-flight creations are shielded so a teardown-time cancellation cannot leave a created stream (and its processor) orphaned.
  • A transient creation failure now retries the current subscription directly. Retries bypass the duplicate-subscription guard but retain closed/generation/track checks, and are bounded to three attempts per generation; exhaustion clears the subscription and logs once.
  • Creation failures release a selector-owned processor even when superseded, and teardown is shielded through stream/forward-task cleanup so cancellation cannot leave listeners or processors behind.

Testing

  • uv run pytest tests/test_room_io.py -q (80 passed)
  • uv run ruff format --check and ruff check for changed files
  • uv run --group typing mypy --untyped-calls-exclude=smithy_aws_core --platform linux -p livekit.agents (238 source files)
  • rebased onto current upstream/main (d78f39990)

@muntazir868
muntazir868 requested a review from a team as a code owner September 27, 2026 14:26
@CLAassistant

CLAassistant commented Sep 27, 2026 •

Copy link
Copy Markdown

CLA assistant check
All committers have signed the CLA.

devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

@muntazir868

Copy link
Copy Markdown
Contributor Author

Rebased onto current main β€” this now merges cleanly and CI is re-running.

Also split this PR: the inference STT log-level change is unrelated to the RoomIO stream-creation fix and made this hard to review, so it's now on its own PR: #7918. This PR is now only the RoomIO change.

One test needed updating: test_warns_on_double_noise_cancellation patched AudioStream.from_track and called _on_track_available synchronously. Since creation now happens in a task, the patch context could exit before creation ran. The test now awaits the in-flight creation task inside the patch context β€” no behavior change to the test's assertion.

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 1 new potential issue.

⚠️ 1 issue in files not directly in the diff

⚠️ Handled STT rejects emit duplicate errors

When the gateway rejects a session before input ends, recv_task logs the retryable error at ERROR. Its exception decorator logs another ERROR, even when the stream retries successfully.

2 flags not posted on this PR by your GitHub settings β€” view them in Devin Review. (Configure)

Devin Review

@muntazir868
muntazir868 force-pushed the fix/7441-offloop-stream-create branch from fd66a12 to 5ca0db1 Compare November 14, 2026 13:07
devin-ai-integration[bot]

This comment was marked as resolved.

devin-ai-integration[bot]

This comment was marked as resolved.

AudioStream/VideoStream.from_track performs a synchronous FFI request
(initializing noise cancellation can block for >100ms) and was called
directly from the track_subscribed handler, tripping the loop monitor.
Create the stream in a task via asyncio.to_thread, pass the running
loop explicitly so frames still dispatch onto it, and guard against
track events that supersede an in-flight creation with a generation
counter plus a lock around processor ownership.
- Fix aclose() to clean up resources even when cancelled during settle
- Add retry mechanism for failed stream creation (don't clear track on failure)
- Increment generation counter to allow retry
@muntazir868 muntazir868 changed the title fix: create room IO media streams off the event loop and demote retryable STT errors fix(voice): create room IO media streams off the event loop Nov 21, 2026
@muntazir868
muntazir868 force-pushed the fix/7441-offloop-stream-create branch from 37bfccf to 193b990 Compare November 21, 2026 20:47

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 new potential issue.

1 flag not posted on this PR by your GitHub settings β€” view it in Devin Review. (Configure)

Devin Review

Comment on lines +449 to +453
else:
# The subscription is dead from RoomIO's perspective. Clear
# it so a later track/publication event can start a fresh
# generation; don't keep retrying a permanently broken track.
self._close_stream()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

πŸ”΄ Failed screen share blocks camera fallback

When screen share creation exhausts its retries, _close_stream forgets the selected publication. If the participant stops sharing, the track removal handlers skip the fallback scan. The agent keeps receiving no video even though the camera is still subscribed.

Learn more

RoomIO accepts both camera and screen share for the linked participant. When a new publication takes over, _on_track_available closes the old local stream without unsubscribing from its publication. The track removal handlers scan other subscribed tracks only when the event matches the selected publication. At retry exhaustion, _close_stream clears that publication, so its eventual removal never triggers the scan.

Example: The agent receives camera frames. The same participant starts sharing their screen, but all three local attempts to open that stream fail. They stop sharing without turning off their camera. The agent remains without video because neither removal event selects the camera.

Recommended fix: Keep the failed track and publication associated with the exhausted generation without scheduling more retries. A real removal can then run the existing fallback scan, and a new subscription or replacement track can still start a fresh generation. Add a test that exhausts retries for one publication, removes it, and verifies that another already-subscribed publication starts forwarding.

Devin Review


Was this helpful? React with πŸ‘ or πŸ‘Ž to provide feedback.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Session start blocks the loop in AudioStream.from_track, and xai/stt-1 init fails with Streaming ASR unavailable (2006)

2 participants