Skip to content

Expose data tracks over uniffi for room-apis livekit-ffi consumers - #1620

Open
1egoman wants to merge 22 commits into
mainfrom
room-apis-data-track-uniffi
Open

1egoman wants to merge 22 commits into
mainfrom
room-apis-data-track-uniffi

Conversation

@1egoman

@1egoman 1egoman commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Overview

Today, there are two ways client sdks that implement data tracks (using the rust implementation) consume the rust code:

  • core-modules style: swift, android, and flutter all use the existing data tracks uniffi interface which has existed for many months. This is what used to be in livekit-uniffi prior to the livekit-ffi / livekit-uniffi merge.
  • room-apis style: node, python, c++, and unity all use the legacy C ABI style FFI interface, squashing all their call semantics into request / response protobufs.

The focus of this pull request is to bring an interface very similar to the current core-modules style to consumers using livekit-ffi via the room-apis style.

Summary of change

The primary implementation goal of this is to reuse as much code between the two livekit-ffi "modes" as possible. Any "record" type structs like DataTrackInfo should be used by both styles, as well as LocalDataTrack and RemoteDataTrack. This means that all downstream sdks which use rust can now use most of the same primitives!

Also, just like the sox resampler change, LocalDataTrack and RemoteDataTrack now can be created from a ffi handle id, and similarly a uniffi object can have a ffi handle created from it. This allows for a more flexible migration strategy, so it can be done piecemeal if desired.

Usage Example

Here's how previously you would subscribe to a remote data track in python:

# 1. The remote track: a handle owned by the Python side.
track_handle = FfiHandle(owned.handle.id)
print(owned.publisher_identity, owned.info.name)

# 2. Subscribe. The stream handle comes back right away; the subscription
#    itself is set up asynchronously, and a failure arrives as the stream's EOS.
req = proto_ffi.FfiRequest()
req.subscribe_data_track.track_handle = track_handle.handle
req.subscribe_data_track.options.buffer_size = 16
resp = FfiClient.instance.request(req)
stream_handle = FfiHandle(resp.subscribe_data_track.stream.handle.id)

# Frames arrive as FfiEvents, one per read request.
queue = FfiClient.instance.queue.subscribe(
    filter_fn=lambda e: (
        e.WhichOneof("message") == "data_track_stream_event"
        and e.data_track_stream_event.stream_handle == stream_handle.handle
    ),
)
try:
    for _ in range(5):
        read = proto_ffi.FfiRequest()
        read.data_track_stream_read.stream_handle = stream_handle.handle
        FfiClient.instance.request(read)

        event = (await queue.get()).data_track_stream_event
        if event.WhichOneof("detail") == "eos":
            if event.eos.HasField("error"):
                raise RuntimeError(event.eos.error.message)
            break

        frame = event.frame_received.frame
        ts = frame.user_timestamp if frame.HasField("user_t
        print(frame.payload, ts)
finally:
    # 3. Tear down. Dropping the stream handle unsubscribes; dropping the
    #    track handle releases the track.
    FfiClient.instance.queue.unsubscribe(queue)
    stream_handle.dispose()  # livekit_ffi_drop_handle
    track_handle.dispose()   # livekit_ffi_drop_handle

And, here's how it now works with the new uniffi based api - less verbose, and matches almost 1:1 with how it works in swift / kotlin / flutter today:

# 1. Take up the C ABI handle as a uniffi object. The two sides now co-own the
#    track, so drop the C ABI handle to leave uniffi as the only owner.
track = livekit_ffi.RemoteDataTrack.from_ffi_handle_id(owned.handle.id)
FfiHandle(owned.handle.id).dispose()
print(track.publisher_identity(), track.info().name)

# 2. Subscribe. This awaits until the subscription is established, and a
#    failure raises here instead of arriving as an EOS event.
try:
    stream = await track.subscribe_with_options(
        livekit_ffi.DataTrackSubscribeOptions(buffer_size=16)
    )
except livekit_datatrack.DataTrackSubscribeError as e:
    print("subscribe failed:", e)
    return

for _ in range(5):
    frame = await stream.next()  # None once the subscription has ended
    if frame is None:
        break
    print(frame.payload, frame.user_timestamp)  # bytes, Optional[int]

# 3. Tear down. Releasing the last reference frees the Rust object: the
#    stream unsubscribes, then the track is released.
del stream
del track

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Changeset ✓

This PR includes a changeset covering all affected packages:

Package Bump
livekit-ffi minor

//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewers: I took all livekit-ffi/src/core_modules/data_track/xxx.rs‎ files and split each out into two versions:

  • livekit-ffi/src/core_modules/data_track_manager/xxx.rs contains the core-modules-only parts of the change - so the data track managers and associated infrastructure
  • livekit-ffi/src/data_track/xxx.rs contains all the infrastructure that will be used both in room-apis and core-modules, which includes "record" type structs like DataTrackInfo, higher level "object" type structs like LocalDataTrack, and some utilities.

Comment on lines +275 to +288

/// Protobuf-driven tests for data tracks, the next types to be migrated.
///
/// Unlike the resampler, a data track only exists inside a connected room, so
/// these run against a LiveKit server (`livekit-server --dev` with
/// `enable_data_tracks: true`), the same as the `livekit` crate's e2e tests.
#[cfg(feature = "__lk-e2e-test")]
pub(crate) mod data_track {
use super::request;
use crate::{
proto,
server::{requests::handle_request, FfiConfig},
FfiHandleId, FFI_SERVER,
};

@1egoman 1egoman Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewers: I've added some livekit-ffi legacy ffi data track e2e tests here.

Note that these are the majority of the diff. Without these, the diff would be about ~400 lines or so.

I have some e2e tests which exercise the full room apis uniffi interface locally as well, but I'm unsure how thorough those should really be. We already have pretty good test coverage of data tracks as a feature, and the sox resampler tests exercise the shared legacy fii <-> uniffi conversion paths. The uniffi wrapper layer is pretty thin and so I'm not currently convinced that it warrants additional test coverage. If you disagree though, let me know and I can push them up.

}

impl FfiHandle for FfiLocalDataTrack {}
impl FfiHandle for FfiRemoteDataTrack {}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note to reviewers: I think all the changes in this file are the part I am least confident on and could use the most eyes from others.

@1egoman
1egoman marked this pull request as ready for review November 2, 2026 01:02
@1egoman
1egoman requested a review from ladvoc as a code owner November 2, 2026 01:02

@devin-ai-integration devin-ai-integration Bot left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note

Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

///
/// Not named `data_track`: the crate root republishes this module with a glob, and the root's own
/// `data_track` would silently shadow it.
pub mod data_track_manager;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Core-module delegate API paths disappear

Existing Rust consumers of data_track::local::LocalDataTrackManagerDelegate lose that path when the manager moves. The remote delegate and signal-response error paths also disappear, breaking source compatibility.

Learn more

The old public data_track tree included the manager delegate traits and HandleSignalResponseError, in addition to track objects. Renaming that tree removes those paths from the exported Rust API even though the corresponding UniFFI names remain available. The new root-level data_track tree only includes shared track objects, so a previous Rust import of a delegate trait no longer resolves.

Example: A downstream implementation importing livekit_ffi::data_track::remote::RemoteDataTrackManagerDelegate must change its import after this release, despite not adopting any new API.

Recommended fix: Preserve the old public paths with re-exports in data_track::local, data_track::remote, and data_track::common under core-modules, or explicitly document the breaking API change if retaining them is impossible.

Devin Review


Was this helpful? React with 👍 or 👎 to provide feedback.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, this is fine, but I should probably make it a minor release - done in 049dfc0

@MaxHeimbrock

Copy link
Copy Markdown
Contributor

In your code bottom example in the PR description, the higher level python classes track and stream would still have to be written in the higher language (Python, C#, etc) right?

@1egoman

1egoman commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor Author

In your code bottom example in the PR description, the higher level python classes track and stream would still have to be written in the higher language (Python, C#, etc) right?

@MaxHeimbrock To clarify - yes, these high level classes would still exist in each downstream client sdk and written in that client sdk's higher level language. These would then very thinly wrap objects like RemoteDataTrack / DataTrackSubscribeOptions / etc which are generated by the uniffi python bindgen.

@MaxHeimbrock

Copy link
Copy Markdown
Contributor

In your code bottom example in the PR description, the higher level python classes track and stream would still have to be written in the higher language (Python, C#, etc) right?

@MaxHeimbrock To clarify - yes, these high level classes would still exist in each downstream client sdk and written in that client sdk's higher level language. These would then very thinly wrap objects like RemoteDataTrack / DataTrackSubscribeOptions / etc which are generated by the uniffi python bindgen.

Ok, so if I read this correctly, most of the higher logic will be generated by uniffi, but we still need to hand-write a thin wrapper ourselves? Or will that all be working out of the box with the uniffi generated code?

I think it would be best for me to try this myself, using Unity as the example. I would check out your branch, build the ffi library, but can now generate uniffi C# code by pointing the bindgen at this library now. Then I can import that C# code into the LiveKit SDK and write code similar to yours, acquiring the handle via FFI but using it in the uniffi classes after calling from_ffi_handle_id.

Am I missing something?

@MaxHeimbrock

Copy link
Copy Markdown
Contributor

Where is from_ffi_handle_id implemented? I can't find it. Is it generated from uniffi or should it already be available in the FFI language SDKs?

@1egoman

1egoman commented Nov 3, 2026

Copy link
Copy Markdown
Contributor Author

Where is from_ffi_handle_id implemented? I can't find it. Is it generated from uniffi or should it already be available in the FFI language SDKs?

Already available in the FFI language sdks. For example, in the unity sdk, this would be the ffi handle id that FFIHandle wraps.

@MaxHeimbrock

Copy link
Copy Markdown
Contributor

Where is from_ffi_handle_id implemented? I can't find it. Is it generated from uniffi or should it already be available in the FFI language SDKs?

Already available in the FFI language sdks. For example, in the unity sdk, this would be the ffi handle id that FFIHandle wraps.

Got it, then if I find time I will try the flow from the description in C#

devin-ai-integration[bot]

This comment was marked as resolved.

Comment thread livekit-ffi/src/data_track/local.rs Resolved
Comment thread livekit-ffi/src/lib.rs Resolved
let _ = send_event(proto::DataTrackPublished { track }.into());
let info = track.info().clone();
let publisher_identity = track.publisher_identity().to_string();
let track = Arc::new(data_track::RemoteDataTrack::new(track));

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: Need to look into this more closely to see if this can be avoided in a UniFFI context, but data track objects generally shouldn't be wrapped in Arc and rather should just be cloned directly (Arc is already used internally).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is another case where the naming ended up being unintentionally confusing - data_track::RemoteDataTrack is the ffi data track type, not the not the livekit version. So this is needed, yes.

To make this less confusing, I have renamed it to data_track_ffi in 891e976. We keep running into this issue though so I wonder if there's a more structural way you can think to address this. Maybe aim to fully qualify each ffi struct usage with a livekit_ffi::xxx::yyy::MyStruct type of pattern?

Comment thread livekit-ffi/src/data_track/remote.rs Resolved
Protobuf-driven e2e tests for data tracks, in the style of the resampler's
migration tests: publish, subscribe and push, unpublish ending the stream,
dropping the local handle unpublishing, duplicate-name errors and handle type
checks. They need a server, so they sit behind a new `__lk-e2e-test` feature
that the workspace-wide e2e job already passes.
… both surfaces

Both now carry their `uniffi` feature unconditionally rather than only under
`core-modules`, so the data track objects can be exported by either surface.
In room-apis these are the crate instances `livekit` already pulls in.
LocalDataTrack, RemoteDataTrack, DataTrackStream and the records they use
(DataTrackFrame, DataTrackInfo, DataTrackSchemaId, DataTrackOptions,
DataTrackSubscribeOptions, RemoteDataTrackPipelineOptions) move verbatim to a
top-level `data_track` module that both surfaces compile. The `Bytes`
remote-type registration they depend on moves to the crate root with them.

The managers, their delegates and the signal-response helpers stay in
core-modules: they exist only for SDKs that run their own room.
…manager

The crate root republishes core-modules with a glob, and the new top-level
`data_track` silently shadowed the core-modules module of the same name,
leaving the managers with no public path. What remains there is the managers,
so name it for them.
With the data track objects exported by room-apis, livekit_ffi.py now imports
the livekit_common and livekit_datatrack modules generated beside it. UniFFI
defaults to relative imports, which only work inside a package, so the
documented `import livekit_ffi` from a flat directory broke. Configure
absolute imports, and have CI import the bindings that way.

Generated bindings for core-modules are unchanged by the data track move apart
from item order, checksums and one doc comment.
LocalDataTrack and RemoteDataTrack become named structs with a
`handle_id: OnceLock<FfiHandleId>` field that only exists in room-apis
builds, the slot `migrate_from_ffi!` expects, as on SoxResampler. A `new`
constructor keeps the cfg out of callers; core-modules' managers use it.
…and uniffi

In room-apis, LocalDataTrack and RemoteDataTrack gain the `migrate_from_ffi!`
methods (`ffi_handle_id`, `from_ffi_handle_id`, `take_ffi_handle_id`), so a
track can cross between the two FFI surfaces like SoxResampler. Core-modules
builds have no handle map and are unaffected.
…ffi objects

The room-apis request handlers now keep each track as an Arc of the shared
LocalDataTrack/RemoteDataTrack, published through `ffi_handle_id()` like
SoxResampler, instead of a separate FfiLocalDataTrack/FfiRemoteDataTrack
wrapper. A handle the room hands out can therefore be picked up over uniffi with
`from_ffi_handle_id`, and the two sides co-own the track: it is unpublished on
drop only once neither holds it.

The protobuf request surface is unchanged; the handlers become `_ffi` methods
on the shared types. Data track streams stay independent per surface.
…rfaces

A local data track unpublishes when its last owner drops it. Once a C ABI
track handle has been taken up over UniFFI, the handle map and the UniFFI side
co-own the track, so dropping the handle alone no longer unpublishes it. Say so
on the proto handles and where the handle map stores the tracks, and add the
livekit-ffi changeset for the data track work.
The Python SDK consumes the bindings from inside a package, where UniFFI's
default relative imports work. Drop the external_packages overrides and the
flat-import smoke test; CI keeps asserting the data track objects are present.
…quest handlers

The handlers in requests.rs, and the call sites that hand out track handles,
now call the shared tracks directly instead of going through `_ffi` methods on
them. Setting up a legacy subscription moves to FfiDataTrackStream::subscribe,
beside the stream it creates.
Call `migrate_from_ffi!` next to each type, as SoxResampler does, so
`handle_id` no longer needs crate visibility and the nested migration module
in room-apis goes away. The request handlers that need the wrapped track read
it through an `inner()` accessor compiled only into room-apis.
The data stream readers spawn onto `crate::runtime::runtime()` when closed, and
are about to be shared by both surfaces. Room-apis returns the FFI server's
runtime; core-modules keeps its own.
A pending `next()` holds the stream, so releasing it from the foreign side
could not end the subscription while a read was in flight. Swift and Dart
cannot cancel that read, which left the subscription running until the next
frame arrived. `close()` wakes the pending read with `None` and drops the core
stream, matching `close()` on the data stream readers.

`tokio-util` is now a dependency of both surfaces, since the shared stream
uses its `CancellationToken`.
wait_for_unpublish awaits a tokio watch channel, which needs no runtime to poll.
…datatrack

#[uniffi(default)] accepts only a literal, so the FFI record repeated the core
default and could drift from it. The field is now optional, and an unset value
falls back to the core RemoteDataTrackPipelineOptions default.
Its exported close() clashes with the AutoCloseable close() UniFFI generates
for every Kotlin object, so it becomes close_stream there, like the other
stream objects.
…ack_ffi

Makes it clear at the call sites that Arc<data_track_ffi::RemoteDataTrack> wraps
the FFI wrapper object, not the core DataTrack.
@1egoman
1egoman force-pushed the room-apis-data-track-uniffi branch from 049dfc0 to 33246bc Compare November 19, 2026 23:39

@alan-george-lk alan-george-lk left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I integrated this into the C++ SDK as part of my bindgen async work -- all existing data tracks tests passed without test modification. Nice work!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants