Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
16 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion bindings/c/docs/C_API_Design.md
Original file line number Diff line number Diff line change
Expand Up @@ -278,7 +278,7 @@ Invalid parameters are rejected with `SVS_ERROR_INVALID_ARGUMENT`; unlike the le
|-----------------|----------|
| `SVS_SYNC_KIND_NONE` | No internal synchronization (default); the caller must serialize access |
| `SVS_SYNC_KIND_GLOBAL` | Index-wide reader/writer lock: read-only operations (search, `has_id`, `get_distance`, reconstruct, `get_size`, `get_num_threads`, memory queries, use as a conversion source) run concurrently; mutating operations (add/delete points, consolidate, compact, save, `set_num_threads`) are exclusive |
| `SVS_SYNC_KIND_FINE_GRAIN` | Backed by the concurrent Vamana index, which synchronizes updates itself: read-only operations and add/delete points, consolidate, compact all take a shared lock and run in parallel (compact still waits inside the index). Save, `set_num_threads`, memory queries and use as a conversion source are exclusive |
| `SVS_SYNC_KIND_FINE_GRAIN` | Backed by the concurrent Vamana index, which synchronizes updates itself: read-only operations and add/delete points, consolidate, compact all take a shared lock and run in parallel (compact still waits inside the index). Save takes the shared lock too: it runs alongside searches and makes writers wait. `set_num_threads`, memory queries and use as a conversion source are exclusive |

Synchronization limits:
- `svs_index_free()` must not run concurrently with any other call on the same handle.
Expand Down
3 changes: 2 additions & 1 deletion bindings/c/include/svs/c/svs_c.h
Original file line number Diff line number Diff line change
Expand Up @@ -597,7 +597,8 @@ struct svs_memory_breakdown {
/// With SVS_SYNC_KIND_FINE_GRAIN, the handle is backed by a concurrent index that
/// synchronizes updates itself: add_points, delete_points, consolidate and compact take a
/// shared lock and run alongside searches and each other (compact still waits for
/// in-flight operations inside the index). Only save, set_num_threads, memory queries
/// in-flight operations inside the index). save also takes a shared lock: it runs
/// alongside searches and makes writers wait. Only set_num_threads, memory queries
/// and use as a conversion source take an exclusive lock.
///
/// @remarks Synchronization covers operations on the index only:
Expand Down
15 changes: 13 additions & 2 deletions bindings/c/src/index_vamana.hpp
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,9 @@ struct DynamicIndexVamana : public DynamicIndex {
/// Dynamic index backed by svs::ConcurrentDynamicVamana.
///
/// The index synchronizes add/delete/consolidate/compact itself, so they take the shared
/// lock alongside searches. Operations that are not thread-safe in the index (save,
/// set_num_threads) and whole-index reads (memory breakdown, use as a conversion source)
/// lock alongside searches. save() also takes the shared lock: the index excludes its own
/// writers while saving. Operations that are not thread-safe in the index
/// (set_num_threads) and whole-index reads (memory breakdown, use as a conversion source)
/// take the exclusive lock.
struct ConcurrentIndexVamana : public DynamicIndexVamana {
ConcurrentIndexVamana(
Expand All @@ -179,6 +180,16 @@ struct ConcurrentIndexVamana : public DynamicIndexVamana {

size_t delete_points(std::span<const size_t> ids) override;

void save(const std::filesystem::path& directory) override {
auto lock = read_lock();
index.save(directory / "config", directory / "graph", directory / "data");
}

void save(std::ostream& stream) override {
auto lock = read_lock();
index.save(stream);
}

void consolidate() override {
auto lock = read_lock();
index.consolidate();
Expand Down
11 changes: 7 additions & 4 deletions include/svs/concurrent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,14 @@ share nothing.
You may call `compact()` at any time, but it blocks all other work. It makes the
storage smaller, so it must wait for every reader to finish first. It is the only
caller that takes `compact_mutex_` exclusive. It runs `consolidate_locked()`
before it compacts.
before it compacts. `save()` takes `save_gatekeeper_` and
`save_mutex_` exclusive, so writers wait while searches keep running. It writes the
slots as they are.

### 1d. Not safe at all

Call these only when no other thread uses the index:

- `save()` - it holds no lock while it writes the files. It calls `consolidate()`
and `compact()` first, but it releases both locks before writing.
- `set_alpha`, `set_prune_to`, `set_max_candidates`,
`set_construction_window_size`, `set_threadpool` - plain writes, no lock.

Expand Down Expand Up @@ -142,6 +142,8 @@ There are three groups:

| Lock | Protects | Taken exclusive by |
| --- | --- | --- |
| `save_mutex_` | the index state stays unchanged while `save()` writes it | `save()` only |
| `save_gatekeeper_` | new writers wait behind a pending `save()` | `save()` only |
| `compact_mutex_` | the storage stays alive | `compact()` only |
| `translator_mutex_` | the two id hash maps | `add_points`, `consolidate`, `compact`, `replace_external_id` |
| `pending_insertions_mutex_` | `pending_insertions_` | `add_points` |
Expand All @@ -159,6 +161,7 @@ so it waits for all of them.
deadlock.

```
save_gatekeeper_ -> save_mutex_ -> compact_gatekeeper_ -> compact_mutex_
compact_mutex_ -> slot_alloc_mutex_
compact_mutex_ -> translator_mutex_
l2e_mutex_ -> e2l_mutex_ (multi index)
Expand Down Expand Up @@ -247,7 +250,7 @@ The dataset must use a `SegmentedBlocked` allocator; this is checked at compile
time. `distance` may be a functor or a `svs::DistanceType`. `batch_iterator()`
returns this index's own `BatchIterator`, so it takes the locks described in 1a.

The rules from section 1 apply unchanged: `save()`, `set_threadpool()` and the
The rules from section 1 apply unchanged: `set_threadpool()` and the
build-parameter setters still need exclusive access.

```cpp
Expand Down
66 changes: 47 additions & 19 deletions include/svs/concurrent/dynamic_index.h
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,7 @@ class MutableVamanaIndex {
static constexpr bool supports_deletions = true;
static constexpr bool supports_saving = true;
static constexpr bool needs_id_translation = true;
static constexpr bool saves_slot_status = true;

// Type Aliases
using Idx = typename Graph::index_type;
Expand Down Expand Up @@ -250,7 +251,8 @@ class MutableVamanaIndex {
// compact() holds it exclusive.
//
// Lock acquisition order across the codebase:
// compact_gatekeeper_ -> compact_mutex_ (never reversed)
// save_gatekeeper_ -> save_mutex_ -> compact_gatekeeper_ -> compact_mutex_
// (never reversed)
// compact_mutex_ -> slot_alloc_mutex_ (never reversed)
// compact_mutex_ -> translator_mutex_ (never reversed)
// slot_alloc_mutex_ and translator_mutex_ are never held simultaneously
Expand All @@ -263,6 +265,13 @@ class MutableVamanaIndex {
std::make_unique<std::shared_mutex>()};
// Iterators check this under compact_mutex_ before reusing cached internal IDs.
size_t compaction_epoch_ = 0;
// Outermost lock. Writers (add_points, delete_entries, consolidate, compact,
// replace_external_id) hold it shared; save() holds it exclusive. Searches never take
// it.
std::unique_ptr<std::shared_mutex> save_mutex_{std::make_unique<std::shared_mutex>()};
// Once save() owns this gate, new writers cannot keep it waiting on save_mutex_.
std::unique_ptr<std::shared_mutex> save_gatekeeper_{
std::make_unique<std::shared_mutex>()};
// Writer-only mutex serializing slot allocation in add_points
std::unique_ptr<std::mutex> slot_alloc_mutex_{std::make_unique<std::mutex>()};

Expand Down Expand Up @@ -447,7 +456,7 @@ class MutableVamanaIndex {
///
/// * data.size() == graph.n_nodes() == status.size().
/// * ``status`` contains no ``Pending`` slots.
/// * ``translator`` maps exactly the ``Valid`` slots of ``status``.
/// * ``translator`` maps every ``Valid`` slot and no ``Empty`` slot of ``status``.
template <threads::ThreadPool Pool>
MutableVamanaIndex(
const VamanaIndexParameters& config,
Expand Down Expand Up @@ -513,17 +522,12 @@ class MutableVamanaIndex {
status_[i] = s;
}

if (translator_.size() != num_valid) {
throw ANNEXCEPTION(
"Translator has {} IDs but status has {} valid slots",
translator_.size(),
num_valid
);
}
for (auto pair : translator_) {
if (pair.second >= n || status[pair.second] != SlotMetadata::Valid) {
if (pair.second >= n || (status[pair.second] != SlotMetadata::Valid &&
status[pair.second] != SlotMetadata::Deleted)) {
throw ANNEXCEPTION(
"Translator maps external id {} to non-valid slot {}",
"Translator maps external id {} to slot {} that is not valid or "
"deleted",
pair.first,
pair.second
);
Expand Down Expand Up @@ -826,6 +830,7 @@ class MutableVamanaIndex {
/// @param new_id The external ID to assign. Must not be mapped at all.
///
ReplaceExternalIdResult replace_external_id(size_t old_id, size_t new_id) {
auto save_lock = lock_save_shared();
auto compact_lock = lock_storage_shared();
std::lock_guard lock{*translator_mutex_};

Expand Down Expand Up @@ -882,6 +887,12 @@ class MutableVamanaIndex {
// This is an internal method, mostly used to help implement the batch iterator.
ValidBuilder internal_search_builder() const { return ValidBuilder{status_}; }

/// @brief Shared lock taken first by every writer; waits behind a pending save().
[[nodiscard]] std::shared_lock<std::shared_mutex> lock_save_shared() const {
std::shared_lock gatekeeper_lock{*save_gatekeeper_};
return std::shared_lock<std::shared_mutex>(*save_mutex_);
}

/// @brief RAII shared lock guarding data_/graph_ against compact()'s shrink
/// (which frees segments). Used by searches, iterators and mutations that
/// may overlap each other but must exclude compaction.
Expand Down Expand Up @@ -1108,6 +1119,8 @@ class MutableVamanaIndex {
return {};
}

auto save_lock = lock_save_shared();

// Reserve slot ownership against compact(). Held for the entire call,
// including the lock-free Phase 2-4 below; compact() takes this
// exclusive and so will block until every in-flight add finishes.
Expand Down Expand Up @@ -1315,6 +1328,7 @@ class MutableVamanaIndex {
/// graph.
///
template <typename T> size_t delete_entries(const T& ids) {
auto save_lock = lock_save_shared();
auto compact_lock = lock_storage_shared();
std::shared_lock lock{*translator_mutex_};
size_t deleted = 0;
Expand Down Expand Up @@ -1400,6 +1414,7 @@ class MutableVamanaIndex {
/// improve performance but requires more working memory.
///
void compact(Idx batch_size = 1'000) {
auto save_lock = lock_save_shared();
std::lock_guard gatekeeper_lock{*compact_gatekeeper_};
std::lock_guard compact_lock{*compact_mutex_};

Expand Down Expand Up @@ -1613,6 +1628,7 @@ class MutableVamanaIndex {

///// Mutation
void consolidate() {
auto save_lock = lock_save_shared();
auto compact_lock = lock_storage_shared();
consolidate_locked();
}
Expand Down Expand Up @@ -1702,6 +1718,7 @@ class MutableVamanaIndex {
/// @returns The number of listed IDs that were consolidated.
///
template <typename T> size_t consolidate(const T& ids) {
auto save_lock = lock_save_shared();
auto compact_lock = lock_storage_shared();

// Collect the internal slots of the listed, already-soft-deleted IDs.
Expand Down Expand Up @@ -1741,18 +1758,24 @@ class MutableVamanaIndex {
get_search_parameters()};
}

static constexpr lib::Version save_version = lib::Version(0, 0, 0);
static constexpr lib::Version save_version = lib::Version(0, 0, 1);
void save(
const std::filesystem::path& config_directory,
const std::filesystem::path& graph_directory,
const std::filesystem::path& data_directory
) {
std::lock_guard gatekeeper_lock{*save_gatekeeper_};
std::lock_guard save_lock{*save_mutex_};
vamana::detail::save_dynamic_index(
*this, config_directory, graph_directory, data_directory
);
}

void save(std::ostream& os) { vamana::detail::save_dynamic_index(*this, os); }
void save(std::ostream& os) {
std::lock_guard gatekeeper_lock{*save_gatekeeper_};
std::lock_guard save_lock{*save_mutex_};
Comment on lines +1775 to +1776

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Keeping the tests to what save/load preserves. Writer exclusion and searches during save were checked locally (save returns in ~0.02 s with 4 writer threads; searches complete during saves), but a blocking-stream test depends on thread timing, so not adding it here.

vamana::detail::save_dynamic_index(*this, os);
}

/////
///// Calibrate
Expand Down Expand Up @@ -2323,6 +2346,7 @@ auto auto_dynamic_assemble(
// Read the combined TOML (parameters + translation)
// and the translator binary data.
auto table = lib::detail::read_metadata(is);
auto version = table.template cast<toml::table>().version();

auto parameters = lib::load<VamanaIndexParameters>(
table.template cast<toml::table>().at("parameters").template cast<toml::table>()
Expand All @@ -2333,6 +2357,13 @@ auto auto_dynamic_assemble(

auto translator = IDTranslator::load(translation, is);

auto status = std::vector<SlotMetadata>();
if (version == lib::Version(0, 0, 1)) {
status.resize(lib::load_at<size_t>(table.template cast<toml::table>(), "num_status")
);
lib::read_binary(is, status);
}

auto data = data_loader();
auto graph = graph_loader();

Expand All @@ -2344,11 +2375,8 @@ auto auto_dynamic_assemble(
);
}

auto translator_size = translator.size();
if (translator_size != datasize) {
throw ANNEXCEPTION(
"Translator has {} IDs but should have {}", translator_size, datasize
);
if (version != lib::Version(0, 0, 1)) {
status.assign(datasize, SlotMetadata::Valid);
}

auto threadpool = threads::as_threadpool(std::move(threadpool_proto));
Expand All @@ -2357,7 +2385,7 @@ auto auto_dynamic_assemble(
std::move(data),
std::move(graph),
std::move(distance),
std::vector<SlotMetadata>(datasize, SlotMetadata::Valid),
std::move(status),
std::move(translator),
std::move(threadpool),
std::move(logger)};
Expand Down
Loading
Loading