Skip to content

C API: add svs_index_load_stream_dynamic_ex - #477

Merged
yuejiaointel merged 2 commits into
mainfrom
dev/yuejiao/capi-load-stream-dynamic-ex
Nov 17, 2026
Merged

yuejiaointel merged 2 commits into
mainfrom
dev/yuejiao/capi-load-stream-dynamic-ex

Conversation

@yuejiaointel

@yuejiaointel yuejiaointel commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds svs_index_load_stream_dynamic_ex, the stream counterpart of svs_index_load_dynamic_ex.

The C API can save the concurrent (FINE_GRAIN) index to a stream, and can load it back as concurrent from a directory, but loading from a stream always returned the non-concurrent index (SVS_SYNC_KIND_NONE). The new function takes svs_dynamic_index_params_t (block params and sync_kind, NULL = defaults), so SVS_SYNC_KIND_FINE_GRAIN loads the concurrent index.

SVS_API svs_index_h svs_index_load_stream_dynamic_ex(
    svs_index_builder_h builder,
    svs_stream_i stream,
    const svs_dynamic_index_params_t* params /*=NULL*/,
    svs_error_h out_err /*=NULL*/
);

Changes

  • svs_c.h / svs_c.cpp: new function, same validation as svs_index_load_dynamic_ex and the stream handling of svs_index_load_stream_dynamic.
  • IndexBuilder::load_stream_dynamic: picks the concurrent or plain loader by sync kind, like load_dynamic.
  • Concurrent dispatcher: adds the stream source and dispatch_concurrent_vamana_index_load_stream, mirroring the directory path.
  • svs_index_load_stream_dynamic is unchanged (still NONE).

Testing

  • New section in c_api_stream.cpp ("Dynamic stream load with sync kind"): save a dynamic index to a stream, load with svs_index_load_stream_dynamic_ex → FINE_GRAIN sync kind and correct size; params = NULL → NONE.
  • C API public build: 26/26.

Split out of #470 as requested in review.

🤖 Generated with Claude Code

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The FINE_GRAIN stream loader needs a committed round-trip regression test covering _ex synchronization behavior, index operations, and NULL/legacy fallback to NONE.

1 open issue
What changed in this PR

Adds svs_index_load_stream_dynamic_ex so C API callers can restore a concurrent dynamic index from a stream, matching the existing directory-loading capability.

Changes:

  • Adds a parameterized stream-loading API accepting block settings and synchronization kind.
  • Routes FINE_GRAIN loads through the concurrent Vamana dispatcher while preserving the legacy loader’s NONE behavior.
  • Documents the new API and its stream-handling contract.
File Description
bindings/​c/​src/​svs_c.cpp Implements the new entry point and argument validation.
bindings/​c/​src/​index_builder.cpp Selects concurrent or plain dynamic stream loading.
bindings/​c/​src/​dispatcher_concurrent_vamana.hpp Declares the concurrent stream-loading dispatcher.
bindings/​c/​src/​dispatcher_concurrent_vamana.cpp Adds concurrent stream assembly and dispatch registration.
bindings/​c/​include/​svs/​c/​svs_c.h Declares and documents the public API.
bindings/​c/​docs/​C_API_Design.md Includes the new function in the API reference.

Review effort: Balanced

Comment thread bindings/c/src/svs_c.cpp
Comment on lines +1265 to +1268
auto index = builder->impl->load_stream_dynamic(
std::make_unique<InputStream>(*stream->ops, stream->self),
dynamic_params.block_params,
dynamic_params.sync_kind

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added: saves to a stream, loads with svs_index_load_stream_dynamic_ex and checks the FINE_GRAIN sync kind and size, and that params = NULL gives NONE.

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yuejiaointel
yuejiaointel merged commit 265a8f8 into main Nov 17, 2026
6 of 7 checks passed
@yuejiaointel
yuejiaointel deleted the dev/yuejiao/capi-load-stream-dynamic-ex branch November 17, 2026 18:54
@eshenayo

eshenayo commented Nov 17, 2026 •

Copy link
Copy Markdown

Tested through valkey-search together with #470 (results and setup: #470 (comment)). With svs_index_load_stream_dynamic_ex and SVS_SYNC_KIND_FINE_GRAIN, indexes restored from an RDB stream report FINE_GRAIN and take concurrent writers and background consolidation like freshly built ones, which removes valkey-search's exclusive-mutation fallback for restored indexes. Covered by unit tests (sync kind after load, concurrent writers on a restored index, a pending SQ8 index restored and then converted), the SVS integration tests (DEBUG RELOAD, BGSAVE + restart) and a clean TSAN run. Thanks for splitting it out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants