Skip to content

Save concurrent dynamic index without consolidate/compact - #470

Open
yuejiaointel wants to merge 16 commits into
mainfrom
dev/yuejiao/svs257-save-without-compact
Open

yuejiaointel wants to merge 16 commits into
mainfrom
dev/yuejiao/svs257-save-without-compact

Conversation

@yuejiaointel

@yuejiaointel yuejiaointel commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

The concurrent (FINE_GRAIN) dynamic index now saves without cleanup. save() no longer runs consolidate() + compact(); it writes the slots as they are, holes and soft-deleted IDs included (format 0.0.1). Compact moves every vector, so saving from a forked child (e.g. a background snapshot) used to copy the whole index.

The same change for the non-concurrent dynamic index will follow in a separate PR. Loading a stream back as the concurrent index is in #477.

Details

  • save_dynamic_index branches on Index::saves_slot_status (like needs_id_translation): the concurrent index skips consolidate/compact and writes one status byte per slot; the non-concurrent index is unchanged (0.0.0).
  • Save takes a new save-only lock exclusive: add/delete/consolidate/compact/replace_external_id wait, searches keep running. The FINE_GRAIN C API wrapper's save takes the shared lock.
  • Why a new lock: save must block writers but not searches. The existing storage lock (compact_mutex_) is held shared by both searches and writers, so it can only block both (exclusive) or neither (shared). save_mutex_ is taken shared by writers only; save takes it exclusive, so writers wait and searches keep running. save_gatekeeper_ keeps new writers from starving a pending save, like compact_gatekeeper_. Saves still run one at a time.
  • Loaders (both index types, directory and stream) read 0.0.0 and 0.0.1. The concurrent loader keeps soft-deleted IDs mapped; the non-concurrent loader drops them (matching its own delete).
  • Old SVS can't read 0.0.1 files. Saved files keep holes, so they are larger.
  • update_entries (Add update_entries method to concurrent Vamana Index #467) must also take the save lock (one line); whichever PR merges second adds it.
  • Saving from a forked child: stop writers before fork(), otherwise the child's save waits for a lock held by a thread that doesn't exist (same on main).

Testing

  • New test: save/load of a consolidated but not compacted index with soft-deleted slots (fails on main, passes here).
  • Full suite at 420cb94 matches main (198 | 197 passed | 1 failed with ~[inverted]; the one failure, tests/svs/core/io/binary.cpp:138, fails the same way on main). C API: 26/26. Full runs sometimes SIGSEGV at tests/svs/index/vamana/index.cpp:95 on both main and this branch: FlatIndex Logging Test leaves a log sink pointing at a destroyed local (tests/svs/index/flat/flat.cpp:33-57).
  • Locally: save while 4 threads add/delete returns in ~0.02 s and searches keep completing during saves; reload into both index types (directory and stream) and old 0.0.0 files; with Add update_entries method to concurrent Vamana Index #467/C API: expose update_entries and replace_external_id for dynamic index #468 and C API: add svs_index_load_stream_dynamic_ex #477 applied locally, update_points on a deleted ID after a FINE_GRAIN stream reload reuses the same slot, and 10 saves during an update_entries loop all return and reload.
  • Forked save (200k × 128 float, 10% deleted): memory copied by the child 666 MB on main vs 0.8 MB.

🤖 Generated with Claude Code

@yuejiaointel yuejiaointel changed the title Save concurrent dynamic index without consolidate/compact Save concurrent dynamic index without consolidate/compact; add svs_index_load_stream_dynamic_ex Nov 13, 2026

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better to split the PR on 2:

  1. svs_index_load_stream_ex
  2. Save without consolidate/compact

@rfsaliev rfsaliev left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At least 1 test has to be added to validate save/load for an index which is consolidated but not compacted.

@yuejiaointel yuejiaointel changed the title Save concurrent dynamic index without consolidate/compact; add svs_index_load_stream_dynamic_ex Save concurrent dynamic index without consolidate/compact Nov 16, 2026
@yuejiaointel

yuejiaointel commented Nov 16, 2026 •

Copy link
Copy Markdown
Contributor Author

Would be better to split the PR on 2:

1. `svs_index_load_stream_ex`

2. Save without consolidate/compact

svs_index_load_stream_ex changes are now in #477

@yuejiaointel
yuejiaointel marked this pull request as ready for review November 17, 2026 01:00
@yuejiaointel
yuejiaointel requested review from razdoburdin and rfsaliev and a balanced review from Copilot November 17, 2026 01:00

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Regression coverage does not verify save-time search progress and writer exclusion, including the FINE_GRAIN wrapper, or deleted-ID handling in non-concurrent directory and stream loaders.

2 open issues
What changed in this PR

This PR lets the concurrent dynamic index save without consolidation or compaction, preserving holes and soft-deleted slots while avoiding full-index copying during forked snapshots.

Changes:

  • Adds format 0.0.1 with slot status and updates both index loaders to read it.
  • Adds save-specific locking that blocks writers while allowing searches to continue.
  • Updates the FINE_GRAIN C wrapper, synchronization documentation, and slot-preservation regression coverage.
File Description
tests/​svs/​concurrent/​dynamic_index.cpp Adds a stream round trip preserving empty and deleted slots.
include/​svs/​orchestrators/​concurrent_dynamic_vamana.h Documents save concurrency guarantees.
include/​svs/​index/​vamana/​dynamic_index.h Adds status serialization and non-concurrent loading support.
include/​svs/​concurrent/​README.md Documents save locks and lock ordering.
include/​svs/​concurrent/​dynamic_index.h Adds writer exclusion during save and status-aware loading.
bindings/​c/​src/​index_vamana.hpp Uses shared wrapper locks for concurrent saves.
bindings/​c/​include/​svs/​c/​svs_c.h Updates public synchronization documentation.
bindings/​c/​docs/​C_API_Design.md Updates the FINE_GRAIN save locking description.

Review effort: Balanced

Comment on lines +1775 to +1776
std::lock_guard gatekeeper_lock{*save_gatekeeper_};
std::lock_guard save_lock{*save_mutex_};

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the tests to what save/load preserves. Writer exclusion and searches during save were checked locally (save returns in ~0.02 s with 4 writer threads; searches complete during saves), but a blocking-stream test depends on thread timing, so not adding it here.

Comment on lines +1709 to +1713
auto to_drop = std::vector<size_t>();
for (auto pair : translator) {
if (pair.second < datasize && status[pair.second] == SlotMetadata::Deleted) {
to_drop.push_back(pair.second);
}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added: the save/load section now also loads the same save into the dynamic index (stream and directory) and checks size, ID visibility and search results.

yuejiaointel added a commit that referenced this pull request Nov 17, 2026
Adds `svs_index_load_stream_dynamic_ex`, the stream counterpart of
`svs_index_load_dynamic_ex`.

The C API can save the concurrent (FINE_GRAIN) index to a stream, and
can load it back as concurrent from a directory, but loading from a
stream always returned the non-concurrent index (`SVS_SYNC_KIND_NONE`).
The new function takes `svs_dynamic_index_params_t` (block params and
`sync_kind`, `NULL` = defaults), so `SVS_SYNC_KIND_FINE_GRAIN` loads the
concurrent index.

```c
SVS_API svs_index_h svs_index_load_stream_dynamic_ex(
    svs_index_builder_h builder,
    svs_stream_i stream,
    const svs_dynamic_index_params_t* params /*=NULL*/,
    svs_error_h out_err /*=NULL*/
);
```

## Changes
- `svs_c.h` / `svs_c.cpp`: new function, same validation as
`svs_index_load_dynamic_ex` and the stream handling of
`svs_index_load_stream_dynamic`.
- `IndexBuilder::load_stream_dynamic`: picks the concurrent or plain
loader by sync kind, like `load_dynamic`.
- Concurrent dispatcher: adds the stream source and
`dispatch_concurrent_vamana_index_load_stream`, mirroring the directory
path.
- `svs_index_load_stream_dynamic` is unchanged (still `NONE`).

## Testing
- New section in `c_api_stream.cpp` ("Dynamic stream load with sync
kind"): save a dynamic index to a stream, load with
`svs_index_load_stream_dynamic_ex` → FINE_GRAIN sync kind and correct
size; `params = NULL` → NONE.
- C API public build: 26/26.

Split out of #470 as requested in review.

🤖 Generated with [Claude Code](https://claude.com/claude-code)
yuejiaointel added a commit that referenced this pull request Nov 17, 2026
@eshenayo

Copy link
Copy Markdown

Thanks for this. Test results from valkey-search, shared as review data.

Setup: valkey-search fork PR eshenayo/valkey-search#77 (FINE_GRAIN indexes, background consolidation) on SVS main 0b4b1c3, against the same with this PR (420cb94) and #477 (9726e98) merged on top; both merge cleanly. Branch: svs-save-load-470-477. On the valkey-search side: restored indexes load as FINE_GRAIN through svs_index_load_stream_dynamic_ex, and since deleted slots now survive a save, valkey-search stores its own list of deleted ids in its RDB header and restores it on load (it uses the count to trigger consolidation).

Benchmark: integration/benchmark_svs_churn.py (50K x 128-d uniform, L2, 8 writer and 8 reader clients, 10 churn rounds), then BGSAVE, SAVE and DEBUG RELOAD with about 5.2K deleted slots present. 3 runs each, median (range):

main with #470 + #477
SAVE 9.3 s (7.9-10.7) 0.86 s (0.85-0.87)
BGSAVE 9.0 s (8.5-10.4) 0.99 s (0.98-1.05)
BGSAVE copy-on-write (rdb_last_cow_size) 133 MB (119-134) 21 MB (21-42)
DEBUG RELOAD 2.95 s (2.47-4.39) 3.32 s (3.24-3.40), ranges overlap
Deleted slots after reload 0 same count as before the save; the next consolidation reclaims them

Churn throughput, worst write, search QPS and recall show no difference outside the run-to-run ranges, as expected since this PR only changes save and load.

Checks: valkey-search SVS unit tests 104/104 and SVS integration tests 123 passed (13 skipped); TSAN unit suite clean (16/16 binaries, no reports), including concurrent writers on a restored index.

On the fork note in the description: valkey-search suspends its writer, reader and utility thread pools before fork() and waits for running tasks, so no thread holds the save lock or the C API wrapper's lock when the child saves. Nothing extra needed on our side.

Correction to my earlier numbers on #467: the SAVE / BGSAVE figures I posted there for the background-consolidation build (for example SAVE 1.3-1.5 s) aren't comparable with the other columns. A save's cost depends on how many deleted slots exist when it starts, and in those runs a consolidation had just reclaimed most of them. The table above has the same deleted-slot count on both sides.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants