Skip to content

fix(inference): detect a silently dropped Inference STT socket instead of hanging - #8023

Merged
davidzhao merged 1 commit into
livekit:mainfrom
regalmoix:fix/inference-stt-heartbeat
Oct 25, 2026
Merged

davidzhao merged 1 commit into
livekit:mainfrom
regalmoix:fix/inference-stt-heartbeat

Conversation

@regalmoix

@regalmoix regalmoix commented Oct 21, 2026 •

Copy link
Copy Markdown
Contributor

Title: fix(inference): detect a silently dropped Inference STT socket instead of hanging

Same gap as #7900, but in LiveKit Inference STT, which isn't a plugin and isn't covered by the open per-plugin PRs.

inference/stt.py opens its websocket without heartbeat. On a half-open client↔gateway connection (no FIN/RST), recv_task waits on ws.receive() forever. Nothing raises, so the stream never retries and FallbackAdapter never switches.

Scope: this covers the client↔gateway socket only. A provider that goes silent behind a healthy gateway is a separate problem (see the review on #5883, and #7826).

Fix

Same as the merged Deepgram fix (#7206):

  • heartbeat=30.0 on ws_connect. This is the fix: aiohttp now closes a socket that stops answering pings.
  • WSMsgType.ERROR raises a retryable APIConnectionError chained from ws.exception(). Before, it was logged as an unexpected type and then surfaced as a bare 1006 with no cause.

Impact

The probe from #7900, adapted to inference.STT: a local server accepts the session, then never sends and never answers pings. Run with max_retry=0, streaming silence.

before after
detected no error after 75s 46.4s
error none retryable

The error now carries its cause:

APIConnectionError: LiveKit Inference STT connection lost (caused by ServerTimeoutError: No PONG received after 15.0 seconds)

The 46s detection time matches the 30s heartbeat every fixed plugin uses. Shortening it is left to maintainers (see the discussion in #7900).

Testing

  • test_half_open_socket_raises_retryable_error runs a real local gateway that never answers pings. It fails on main, and fails if either change is reverted. 70 runs passed, including under CPU load.
  • uv run pytest --unit --audio_eot passes, apart from test_room.py, which needs a livekit-server binary and fails the same way on main. make check passes.

Why no regression

  • aiohttp pings only after 30s with no bytes received, and any incoming frame resets that timer. Healthy sessions, including long, quiet ones, are unaffected.
  • The error is retryable, so a false trip costs one reconnect: the in-flight partial transcript and the audio sent during it. That needs a ~15s event-loop stall.
  • The ERROR branch uses the same input_ended / session.closed guard as the CLOSED branch. Nothing in the repo matches on the old 1006 message.
  • Merges cleanly with fix(inference): log retryable STT recv errors at debug #7918.

Could a maintainer confirm the gateway/LB answers WS pings? Inference TTS and /bargein sockets have the same gap; I'm happy to follow up.

@regalmoix
regalmoix requested a review from a team as a code owner October 21, 2026 10:25

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@regalmoix

Copy link
Copy Markdown
Contributor Author

@tinalenguyen — would you mind taking a look when you have a moment? Same two-change pattern as #7206, #7357, #7359, #7364, #7369, #7315 that you merged for the plugins. CI green.

…d of hanging

Add heartbeat=30.0 to the Inference STT websocket so a half-open
client-to-gateway connection raises instead of parking recv_task forever,
and raise a retryable APIConnectionError carrying ws.exception() on
WSMsgType.ERROR. Mirrors livekit#7206.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@regalmoix
regalmoix force-pushed the fix/inference-stt-heartbeat branch from 8fc9d63 to 4ff9582 Compare October 21, 2026 14:16
@regalmoix

Copy link
Copy Markdown
Contributor Author

Closing to retrigger CI — unit-tests hit the unrelated test_google_long_decode.py leaked-task flake (same error seen on several other branches today). Reopening.

@davidzhao
davidzhao merged commit 1f514fd into livekit:main Oct 25, 2026
36 of 37 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants