Repository navigation
Save concurrent dynamic index without consolidate/compact - #470
yuejiaointel wants to merge 16 commits into
Conversation
rfsaliev
left a comment
There was a problem hiding this comment.
Would be better to split the PR on 2:
svs_index_load_stream_ex- Save without consolidate/compact
rfsaliev
left a comment
There was a problem hiding this comment.
At least 1 test has to be added to validate save/load for an index which is consolidated but not compacted.
svs_index_load_stream_ex changes are now in #477 |
There was a problem hiding this comment.
🟡 Changes recommended
Regression coverage does not verify save-time search progress and writer exclusion, including the FINE_GRAIN wrapper, or deleted-ID handling in non-concurrent directory and stream loaders.
2 open issues
What changed in this PR
This PR lets the concurrent dynamic index save without consolidation or compaction, preserving holes and soft-deleted slots while avoiding full-index copying during forked snapshots.
Changes:
- Adds format
0.0.1with slot status and updates both index loaders to read it. - Adds save-specific locking that blocks writers while allowing searches to continue.
- Updates the FINE_GRAIN C wrapper, synchronization documentation, and slot-preservation regression coverage.
| File | Description |
|---|---|
tests/svs/concurrent/dynamic_index.cpp |
Adds a stream round trip preserving empty and deleted slots. |
include/svs/orchestrators/concurrent_dynamic_vamana.h |
Documents save concurrency guarantees. |
include/svs/index/vamana/dynamic_index.h |
Adds status serialization and non-concurrent loading support. |
include/svs/concurrent/README.md |
Documents save locks and lock ordering. |
include/svs/concurrent/dynamic_index.h |
Adds writer exclusion during save and status-aware loading. |
bindings/c/src/index_vamana.hpp |
Uses shared wrapper locks for concurrent saves. |
bindings/c/include/svs/c/svs_c.h |
Updates public synchronization documentation. |
bindings/c/docs/C_API_Design.md |
Updates the FINE_GRAIN save locking description. |
Review effort: Balanced
| std::lock_guard gatekeeper_lock{*save_gatekeeper_}; | ||
| std::lock_guard save_lock{*save_mutex_}; |
There was a problem hiding this comment.
Keeping the tests to what save/load preserves. Writer exclusion and searches during save were checked locally (save returns in ~0.02 s with 4 writer threads; searches complete during saves), but a blocking-stream test depends on thread timing, so not adding it here.
| auto to_drop = std::vector<size_t>(); | ||
| for (auto pair : translator) { | ||
| if (pair.second < datasize && status[pair.second] == SlotMetadata::Deleted) { | ||
| to_drop.push_back(pair.second); | ||
| } |
There was a problem hiding this comment.
Added: the save/load section now also loads the same save into the dynamic index (stream and directory) and checks size, ID visibility and search results.
Adds `svs_index_load_stream_dynamic_ex`, the stream counterpart of
`svs_index_load_dynamic_ex`.
The C API can save the concurrent (FINE_GRAIN) index to a stream, and
can load it back as concurrent from a directory, but loading from a
stream always returned the non-concurrent index (`SVS_SYNC_KIND_NONE`).
The new function takes `svs_dynamic_index_params_t` (block params and
`sync_kind`, `NULL` = defaults), so `SVS_SYNC_KIND_FINE_GRAIN` loads the
concurrent index.
```c
SVS_API svs_index_h svs_index_load_stream_dynamic_ex(
svs_index_builder_h builder,
svs_stream_i stream,
const svs_dynamic_index_params_t* params /*=NULL*/,
svs_error_h out_err /*=NULL*/
);
```
## Changes
- `svs_c.h` / `svs_c.cpp`: new function, same validation as
`svs_index_load_dynamic_ex` and the stream handling of
`svs_index_load_stream_dynamic`.
- `IndexBuilder::load_stream_dynamic`: picks the concurrent or plain
loader by sync kind, like `load_dynamic`.
- Concurrent dispatcher: adds the stream source and
`dispatch_concurrent_vamana_index_load_stream`, mirroring the directory
path.
- `svs_index_load_stream_dynamic` is unchanged (still `NONE`).
## Testing
- New section in `c_api_stream.cpp` ("Dynamic stream load with sync
kind"): save a dynamic index to a stream, load with
`svs_index_load_stream_dynamic_ex` → FINE_GRAIN sync kind and correct
size; `params = NULL` → NONE.
- C API public build: 26/26.
Split out of #470 as requested in review.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
|
Thanks for this. Test results from valkey-search, shared as review data. Setup: valkey-search fork PR eshenayo/valkey-search#77 (FINE_GRAIN indexes, background consolidation) on SVS Benchmark:
Churn throughput, worst write, search QPS and recall show no difference outside the run-to-run ranges, as expected since this PR only changes save and load. Checks: valkey-search SVS unit tests 104/104 and SVS integration tests 123 passed (13 skipped); TSAN unit suite clean (16/16 binaries, no reports), including concurrent writers on a restored index. On the fork note in the description: valkey-search suspends its writer, reader and utility thread pools before Correction to my earlier numbers on #467: the SAVE / BGSAVE figures I posted there for the background-consolidation build (for example SAVE 1.3-1.5 s) aren't comparable with the other columns. A save's cost depends on how many deleted slots exist when it starts, and in those runs a consolidation had just reclaimed most of them. The table above has the same deleted-slot count on both sides. |

The concurrent (FINE_GRAIN) dynamic index now saves without cleanup.
save()no longer runsconsolidate()+compact(); it writes the slots as they are, holes and soft-deleted IDs included (format0.0.1). Compact moves every vector, so saving from a forked child (e.g. a background snapshot) used to copy the whole index.The same change for the non-concurrent dynamic index will follow in a separate PR. Loading a stream back as the concurrent index is in #477.
Details
save_dynamic_indexbranches onIndex::saves_slot_status(likeneeds_id_translation): the concurrent index skips consolidate/compact and writes one status byte per slot; the non-concurrent index is unchanged (0.0.0).replace_external_idwait, searches keep running. The FINE_GRAIN C API wrapper's save takes the shared lock.compact_mutex_) is held shared by both searches and writers, so it can only block both (exclusive) or neither (shared).save_mutex_is taken shared by writers only; save takes it exclusive, so writers wait and searches keep running.save_gatekeeper_keeps new writers from starving a pending save, likecompact_gatekeeper_. Saves still run one at a time.0.0.0and0.0.1. The concurrent loader keeps soft-deleted IDs mapped; the non-concurrent loader drops them (matching its own delete).0.0.1files. Saved files keep holes, so they are larger.update_entries(Add update_entries method to concurrent Vamana Index #467) must also take the save lock (one line); whichever PR merges second adds it.fork(), otherwise the child's save waits for a lock held by a thread that doesn't exist (same on main).Testing
198 | 197 passed | 1 failedwith~[inverted]; the one failure,tests/svs/core/io/binary.cpp:138, fails the same way on main). C API: 26/26. Full runs sometimes SIGSEGV attests/svs/index/vamana/index.cpp:95on both main and this branch:FlatIndex Logging Testleaves a log sink pointing at a destroyed local (tests/svs/index/flat/flat.cpp:33-57).0.0.0files; with Add update_entries method to concurrent Vamana Index #467/C API: expose update_entries and replace_external_id for dynamic index #468 and C API: add svs_index_load_stream_dynamic_ex #477 applied locally,update_pointson a deleted ID after a FINE_GRAIN stream reload reuses the same slot, and 10 saves during anupdate_entriesloop all return and reload.🤖 Generated with Claude Code