Skip to content

Expose data streams over uniffi for room-apis livekit-ffi consumers - #1649

Open
1egoman wants to merge 29 commits into
room-apis-data-track-uniffifrom
room-apis-data-stream-uniffi
Open

1egoman wants to merge 29 commits into
room-apis-data-track-uniffifrom
room-apis-data-stream-uniffi

Conversation

@1egoman

@1egoman 1egoman commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Overview

Today, there are two ways client sdks that implement data streams (using the rust implementation) consume the rust code:

  • core-modules style: swift, android, and flutter all use the existing data streams uniffi interface, built around IncomingDataStreamManager and OutgoingDataStreamManager.
  • room-apis style: node, python, c++, and unity all use the legacy C ABI style FFI interface, squashing all their call semantics into request / response protobufs.

The focus of this pull request is to bring an interface very similar to the current core-modules style to consumers using livekit-ffi via the room-apis style.

Summary of change

The primary implementation goal of this is to reuse as much code between the two livekit-ffi "modes" as possible. Any "record" type structs like ByteStreamInfo / TextStreamInfo should be used by both styles, as well as ByteStreamReader, TextStreamReader, ByteStreamWriter, and TextStreamWriter. This means that all downstream sdks which use rust can now use most of the same primitives!

Also, just like the sox resampler change and the data tracks change, the stream readers and writers now can be created from a ffi handle id, and similarly a uniffi object can have a ffi handle created from it. This allows for a more flexible migration strategy, so it can be done piecemeal if desired.

Usage Example

Here's how previously you would read an incoming text stream in python:

# 1. The reader: a handle owned by the Python side.
reader_handle = FfiHandle(owned.handle.id)
print(participant_identity, owned.info.topic)

# 2. Read incrementally. Chunks arrive as FfiEvents, followed by an EOS that
#    carries any error and the attributes from the stream's trailer.
queue = FfiClient.instance.queue.subscribe(
    filter_fn=lambda e: (
        e.WhichOneof("message") == "text_stream_reader_event"
        and e.text_stream_reader_event.reader_handle == reader_handle.handle
    ),
)
try:
    req = proto_ffi.FfiRequest()
    req.text_read_incremental.reader_handle = reader_handle.handle
    FfiClient.instance.request(req)

    while True:
        event = (await queue.get()).text_stream_reader_event
        if event.WhichOneof("detail") == "eos":
            if event.eos.HasField("error"):
                raise RuntimeError(event.eos.error.description)
            print(dict(event.eos.attributes))
            break

        print(event.chunk_received.content)
finally:
    # 3. Tear down. Reading consumed the reader handle; dropping it is a no-op
    #    now, but releases the reader if the read request never went out.
    FfiClient.instance.queue.unsubscribe(queue)
    reader_handle.dispose()  # livekit_ffi_drop_handle

And, here's how it now works with the new uniffi based api - less verbose, and matches almost 1:1 with how it works in swift / kotlin / flutter today:

# 1. Take up the C ABI handle as a uniffi object. The two sides now co-own the
#    reader, so drop the C ABI handle to leave uniffi as the only owner.
reader = livekit_ffi.TextStreamReader.from_ffi_handle_id(owned.handle.id)
FfiHandle(owned.handle.id).dispose()
print(participant_identity, reader.info().topic)

# 2. Read. Each call awaits the next chunk, and a failure raises here instead
#    of arriving as an EOS event.
try:
    while (chunk := await reader.next()) is not None:  # None once the stream has ended
        print(chunk)
except livekit_ffi.DataStreamError as e:
    print("stream failed:", e)
    return

# Attributes from the stream's trailer are merged in once it ends.
print(reader.info().attributes)

# 3. Tear down. Releasing the last reference frees the Rust object, which ends
#    the stream if it is still open.
del reader

Protobuf-driven e2e tests for data tracks, in the style of the resampler's
migration tests: publish, subscribe and push, unpublish ending the stream,
dropping the local handle unpublishing, duplicate-name errors and handle type
checks. They need a server, so they sit behind a new `__lk-e2e-test` feature
that the workspace-wide e2e job already passes.
… both surfaces

Both now carry their `uniffi` feature unconditionally rather than only under
`core-modules`, so the data track objects can be exported by either surface.
In room-apis these are the crate instances `livekit` already pulls in.
LocalDataTrack, RemoteDataTrack, DataTrackStream and the records they use
(DataTrackFrame, DataTrackInfo, DataTrackSchemaId, DataTrackOptions,
DataTrackSubscribeOptions, RemoteDataTrackPipelineOptions) move verbatim to a
top-level `data_track` module that both surfaces compile. The `Bytes`
remote-type registration they depend on moves to the crate root with them.

The managers, their delegates and the signal-response helpers stay in
core-modules: they exist only for SDKs that run their own room.
…manager

The crate root republishes core-modules with a glob, and the new top-level
`data_track` silently shadowed the core-modules module of the same name,
leaving the managers with no public path. What remains there is the managers,
so name it for them.
With the data track objects exported by room-apis, livekit_ffi.py now imports
the livekit_common and livekit_datatrack modules generated beside it. UniFFI
defaults to relative imports, which only work inside a package, so the
documented `import livekit_ffi` from a flat directory broke. Configure
absolute imports, and have CI import the bindings that way.

Generated bindings for core-modules are unchanged by the data track move apart
from item order, checksums and one doc comment.
LocalDataTrack and RemoteDataTrack become named structs with a
`handle_id: OnceLock<FfiHandleId>` field that only exists in room-apis
builds, the slot `migrate_from_ffi!` expects, as on SoxResampler. A `new`
constructor keeps the cfg out of callers; core-modules' managers use it.
…and uniffi

In room-apis, LocalDataTrack and RemoteDataTrack gain the `migrate_from_ffi!`
methods (`ffi_handle_id`, `from_ffi_handle_id`, `take_ffi_handle_id`), so a
track can cross between the two FFI surfaces like SoxResampler. Core-modules
builds have no handle map and are unaffected.
…ffi objects

The room-apis request handlers now keep each track as an Arc of the shared
LocalDataTrack/RemoteDataTrack, published through `ffi_handle_id()` like
SoxResampler, instead of a separate FfiLocalDataTrack/FfiRemoteDataTrack
wrapper. A handle the room hands out can therefore be picked up over uniffi with
`from_ffi_handle_id`, and the two sides co-own the track: it is unpublished on
drop only once neither holds it.

The protobuf request surface is unchanged; the handlers become `_ffi` methods
on the shared types. Data track streams stay independent per surface.
…rfaces

A local data track unpublishes when its last owner drops it. Once a C ABI
track handle has been taken up over UniFFI, the handle map and the UniFFI side
co-own the track, so dropping the handle alone no longer unpublishes it. Say so
on the proto handles and where the handle map stores the tracks, and add the
livekit-ffi changeset for the data track work.
The Python SDK consumes the bindings from inside a package, where UniFFI's
default relative imports work. Drop the external_packages overrides and the
flat-import smoke test; CI keeps asserting the data track objects are present.
…quest handlers

The handlers in requests.rs, and the call sites that hand out track handles,
now call the shared tracks directly instead of going through `_ffi` methods on
them. Setting up a legacy subscription moves to FfiDataTrackStream::subscribe,
beside the stream it creates.
Call `migrate_from_ffi!` next to each type, as SoxResampler does, so
`handle_id` no longer needs crate visibility and the nested migration module
in room-apis goes away. The request handlers that need the wrapped track read
it through an `inner()` accessor compiled only into room-apis.
@1egoman
1egoman requested a review from ladvoc as a code owner November 9, 2026 01:05

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

Devin Review

Comment thread livekit-ffi/src/core_modules/mod.rs Resolved
let info = reader.info().clone();
let ffi_reader = FfiByteStreamReader { handle_id, inner: reader };
server.store_handle(ffi_reader.handle_id, ffi_reader);
let handle_id = Arc::new(data_stream::ByteStreamReader::new(reader)).ffi_handle_id();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 New stream handles lack lifecycle tests

New reader handles enter the FFI map, but no test repeats initialization, use, disconnect, and shutdown. Failed reads and partial teardown also lack destruction checks required for new FFI handles.

Learn more

Publishing a reader with ffi_handle_id adds an Arc to the server's handle map. The same change creates writer handles in stream_bytes and stream_text. The new tests exercise individual successful operations, but do not repeat the full lifecycle or verify destruction after failure and cancellation. A retained handle or task can therefore escape those tests.

Example: Repeatedly connect two participants, open a stream, abandon its read, disconnect, and dispose; verify the handle map and running tasks return to their baseline on every iteration.

Recommended fix: Add a repeated lifecycle workload covering both reader and writer handles, including successful completion, failed/cancelled operations, and teardown before an operation completes. Assert handle-map cleanup and observable destruction or task completion.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

@1egoman
1egoman added this pull request to stack #1650 November 9, 2026 01:07
The data stream readers spawn onto `crate::runtime::runtime()` when closed, and
are about to be shared by both surfaces. Room-apis returns the FFI server's
runtime; core-modules keeps its own.
A pending `next()` holds the stream, so releasing it from the foreign side
could not end the subscription while a read was in flight. Swift and Dart
cannot cancel that read, which left the subscription running until the next
frame arrived. `close()` wakes the pending read with `None` and drops the core
stream, matching `close()` on the data stream readers.

`tokio-util` is now a dependency of both surfaces, since the shared stream
uses its `CancellationToken`.
Protobuf-driven e2e tests for data streams, alongside the data track ones:
send_text and send_bytes read whole, a text writer read incrementally with its
close attributes, a close with a reason erroring the reader, write_to_file,
handles consumed by their first read or close, and handle type checks. The room
helpers move to a shared `e2e` module.
…oth surfaces

The data stream readers and writers are about to be exported by both
surfaces. In room-apis livekit-data-stream is the instance `livekit` already
uses.
The readers and writers (ByteStreamReader, TextStreamReader, ByteStreamWriter,
TextStreamWriter) and the records, enums and error they use move verbatim to a
top-level `data_stream` module that both surfaces compile. So do the stream
option records and StreamProgressDelegate, as DataTrackOptions did.

The managers, their delegates, the polled adapters, ClientCapability,
PacketDeliveryError and the wire decode helper stay in core-modules.
…m_manager

The crate root republishes core-modules with a glob, so the new top-level
`data_stream` hid the core-modules module of the same name. What remains there
is the managers, so name it for them, as with data_track_manager.
…s handle id

The readers and writers gain a `handle_id: OnceLock<FfiHandleId>` field that
only exists in room-apis builds, the slot `migrate_from_ffi!` expects. The
writers become named structs with a private `inner` and a `new` constructor
the core-modules manager uses.
…dle map and uniffi

In room-apis the readers and writers gain the `migrate_from_ffi!` methods
(`ffi_handle_id`, `from_ffi_handle_id`, `take_ffi_handle_id`), called beside
each type. The request handlers get room-apis-only access to the wrapped
values: writers through `inner()`, and readers through `take_inner()`, which
hands over the core reader the legacy reads consume. The uniffi reader then
reads as closed without ending the stream, so dropping it can't cut the legacy
read short.
…niffi objects

The room-apis handle map now keeps each reader and writer as an Arc of the
shared object, published through `ffi_handle_id()` instead of the separate
FfiByteStreamReader/FfiTextStreamReader/FfiByteStreamWriter/FfiTextStreamWriter
wrappers, so a handle the room hands out can be picked up over uniffi with
`from_ffi_handle_id`.

The request handlers are inlined into requests.rs, and the call sites that hand
out handles build the shared objects directly. The two incremental read loops
stay in room_apis/server/data_stream.rs as free functions. A legacy read of a
reader the uniffi side already closed reports AlreadyClosed. The protobuf
request surface is unchanged.
@1egoman
1egoman force-pushed the room-apis-data-stream-uniffi branch from 1525267 to 5605b28 Compare November 17, 2026 20:40
Comment thread livekit-ffi/src/data_stream/common.rs Resolved
#[derive(uniffi::Error, thiserror::Error, Debug)]
pub enum PacketDeliveryError {
#[error("failed to deliver packets: {reason}")]
Failed { reason: String },

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

question: No strong opinion, but is a tuple case also valid (e.g., Failed(String))?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did it this way to match with other cases like this one, this, etc. I also don't have a strong opinion but IMO we should pick a pattern for this type of thing and use it everywhere.

/// A foreign transport failed to deliver outbound packets; thrown by hosts from
/// [`OutgoingDataStreamManagerDelegate::on_packets_available`](super::outgoing::OutgoingDataStreamManagerDelegate::on_packets_available).
///
/// Morally `struct PacketDeliveryError(String)`, but uniffi error types must be enums, so the

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nitpick: This last part is too much information to expose in a doc comment IMO since this will be visible in generated bindings.

/// Reports how much of the payload was sent. Called on the thread that polls the send, so it
/// must return quickly.
///
/// It must not throw. The method returns no error, so UniFFI turns an exception from a Kotlin

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Maybe we could leverage catch_unwind so we don't have to have this disclaimer?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this would make a difference, since catch_unwind wouldn't trigger because panic = "abort" is being set.

I also will say that I think this "progress delegate" approach is the main thing which I think will get rewritten as we move more to the patterns blaze has been pioneering in the "rust experimental clients" project. I opted to keep it as is because it is how it works today, and I wanted to wait for the patterns to firm up a bit more. I think a follow up here to drop this and make it pull based is likely.

More explicit than ds_api, and the convention data tracks will follow too.
The note on why it is an enum is an implementation detail, and doc comments
show up in the generated bindings.
More explicit than ds, matching the data_stream_api alias.
…tream_ffi

Makes it clear at the call sites that Arc<data_stream_ffi::TextStreamWriter> and
friends wrap the FFI wrapper objects, not the core streams.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants