Integrate tiered storage by kim · Pull Request #6305 · clockworklabs/SpacetimeDB

kim · 2026-10-15T14:44:54Z

Description of Changes

Integrates tiered storage into the rest of the codebase.

This is a big change, because every call site that may access a page or blob
becomes fallible due to the possibility of fault errors.

Much of this is mechanical, but note that transaction commit becomes a two-step
process with this patch: first, all pages and blobs affected by the transaction
are faulted in if necessary, and pinned so they can't be evicted until the
transaction completes. Then, the transaction is actually applied -- this remains
infallible and therefore atomic.

API and ABI breaking changes

Internal only.

Rollback safety impact

Currently rollback safe.

Expected complexity level and risk

5

Testing

Remove nine newly added Deep storage tests (429 lines) at the user's request to reduce the review diff. This removes new coverage for resident/tiered storage, prepared inserts, page reuse, and blob pins. Existing tests from #6305 and all production code remain unchanged. Validation: 112 existing spacetimedb-table library tests pass; clippy for the library and tests passes with -D warnings.

Remove the local macOS benchmarking accommodations from the review diff. Restore the original unconditional Table=328 assertion and keep TxState=144, which accounts for the inline transaction blob map. Non-macOS compiled code is unchanged. Native macOS builds again fail the original size checks, as on #6305; a compatibility patch is saved outside the branch for future local profiling. Validation: rustfmt and git diff --check; exact diff is six deleted lines. Native clippy confirms the expected Table size assertion failure (328 versus 336). Linux checks were not rerun locally.

joshua-spacetime · 2026-10-21T19:39:57Z

-        serialized.write_large_blobs(blob_store)
+        serialized.write_large_blobs(blobs)?


We may need to rollback previous changes to the page now that this is fallible, no?

Ya possibly. I think it is actually infallible because we pinned all the blobs, but that is not expressible locally at the moment.

joshua-spacetime · 2026-10-21T19:52:17Z

+        // All page mutations have finished, so pinned blob frames no longer
+        // need to be pinned by the commit guard.
+        drop(prepared.pinned_blobs);


This releases the blob pins, but where do we decrement the ref count and eventually drop/remove the blob(s)?

The refcounts are in the BlobSet that is owned by the transaction state, while the blob frames are shared via the ResidentObjects. The normal insertion path then updates the refcount in the committed state's BlobSet.

Just dropping the handles will leave dangling weak references in ResidentObjects, unless successful insertion created new strong references in the committed state BlobSet (they are guaranteed to become resident, because we pinned them).

joshua-spacetime · 2026-10-21T19:59:53Z

-        ptrs.extend(self.scan_rows(&NullBlobStore).map(|row| row.pointer()));
-        ptrs
+        for row in self.scan_rows(&BlobSet::new_for_test()) {


Is this equivalent to NullBlobStore?

No it's not, the NullBlobStore panics on any method invocation. I'll have to find a way to emulate this, as BlobSet is not a trait object.

joshua-spacetime · 2026-10-21T20:31:14Z


        // If we removed the last unique index, add a pointer map.
        if index.is_unique() && !self.indexes.values().any(|idx| idx.is_unique()) {
-            self.pointer_map = Some(pointer_map.unwrap_or_else(|| self.rebuild_pointer_map(blob_store)));
+            self.pointer_map = Some(pointer_map.map_or_else(|| self.rebuild_pointer_map(blobs), Ok)?);


When do we record a pending schema change for a deleted index? Before or after this call? If after, we'll rollback to an invalid state if the pointer map rebuild fails.

Normally before. I added rollback_add_index specifically to not rebuild the pointer map on rollback -- we should end up in the same state that we started with as pending schema changes get reverted in reverse order (and rebuilding on intermediate reverts was probably a bug to begin with). The TODO is to remind me that this needs a test.

joshua-spacetime · 2026-10-21T21:20:57Z

@@ -1377,33 +1610,34 @@ impl Table {
        // SAFETY:
        // - `self` trivially has the same `row_layout` as `self`.
        // - We just inserted `temp_ptr`, so it's valid.
-        let (_, existing_row_ptr) = unsafe { Self::find_same_row(self, self, blob_store, temp_ptr, None) };
+        let (_, existing_row_ptr) = unsafe { Self::find_same_row(self, self, blobs, temp_ptr, None) }?;


The temporary insert above needs to be rolled back in the case of a failure here as well as the other fallible calls below.

I don't see at moment how this method could be made atomic, because deleting the temporary row can itself fail. But what should happen then?

joshua-spacetime

More rollback safety concerns. It would be good to add some rollback tests if they're indeed legitimate.

joshua-spacetime · 2026-10-21T22:15:53Z

-                let effects = UpdateEffects::after_migration(res, &tx);
+                let effects = UpdateEffects::after_migration(res, &tx)?;


Rollback safety: If this fails, do we undo the migration?

joshua-spacetime · 2026-10-21T22:19:14Z

Rollback safety: The revert is skipped if get_row_ref fails.

joshua-spacetime · 2026-10-21T22:24:55Z

+                commit_table.make_index_non_unique(index_id, commit_blob_store)?;
+                tx_table.make_index_non_unique(index_id, tx_blob_store)?;


Rollback safety: make_index_non_unique succeeds for commit_table but fails on tx_table. Is this a problem? It would appear so, since we haven't added the pending schema change at this point.

Remove nine newly added Deep storage tests (429 lines) at the user's request to reduce the review diff. This removes new coverage for resident/tiered storage, prepared inserts, page reuse, and blob pins. Existing tests from #6305 and all production code remain unchanged. Validation: 112 existing spacetimedb-table library tests pass; clippy for the library and tests passes with -D warnings.

Remove the local macOS benchmarking accommodations from the review diff. Restore the original unconditional Table=328 assertion and keep TxState=144, which accounts for the inline transaction blob map. Non-macOS compiled code is unchanged. Native macOS builds again fail the original size checks, as on #6305; a compatibility patch is saved outside the branch for future local profiling. Validation: rustfmt and git diff --check; exact diff is six deleted lines. Native clippy confirms the expected Table size assertion failure (328 versus 336). Linux checks were not rerun locally.

Introduces the datastructures for tiered storage / faultable objects.

Integrates tiered storage, replacing `Pages` with `PageSet` and `BlobStore` with `BlobSet`. Handles fallible object acquisition everywhere. Also implements two-phase transaction commit, which preflights the transaction and pins all affected objects. This allows the actual merge into the committed state to be infallible and therefore atomic.

Keep directly owned pages and borrowed access for resident tables while retaining tiered page eviction and the newer RowRef API. Stream eligible resident commits while their transaction state is alive, reuse resident pages and budget permits, and reduce row/blob bookkeeping overhead. Also fix shared blob registration and preserve logical blob entries on eviction, with coverage for both backends, budgets, snapshots and pins. Validation: 239 storage tests pass; Clippy passes with -D warnings. Three alternating 60-second TypeScript keynote runs with pipelining and confirmed reads enabled averaged 231,612 transfers/sec versus 241,902 on master (4.25% lower; paired gaps 3.26%, 3.49%, and 5.99%).

the page to be faulted in.

assert that an existing frame is actually the same allocation.

Commit changes only after potential pointer map rebuild succeeded.

If the pointer map needs to be rebuilt, rebuild it first before committing the changes.

This prevents unnecessarily faulting in blobs. With tiered storage, constructing a RowRef requires to load a BlobReadSet, so calling `scan_rows` with an empty BlobSet is not equivalent to the previous `NullBlobStore`.

`BlobReadSet` fails.

Fallibly collect blob references before mutating anything. Delete rows by marking all pages as absent.

`MutTxId::delete_by_row_value`, too

To make this possible, introduce a `PinnedRow` that works like an owned `RowRef`. When a pinned row is in scope, potentially faulting functions can be unwrapped, as they are guaranteed to not fault the pinned row's page.

instead of `RowPointer`.

# Description of Changes Tiered storage makes loading rows fallible. Some cleanup, result-construction, and duplicate-lookup paths looked up rows again even though they already held the required page and blob pins. This PR reuses those existing pins, making the distinction between acquiring a row and using an already-loaded row clearer: - **Cleanup:** Expose the existing `Table::delete_unchecked` helper to the datastore and use it to delete rows whose pins are already held. Its deletion implementation is unchanged. - **Update results:** Retain the replacement row’s pins through confirmation and construct the returned `RowRef` from them, avoiding another lookup after mutation. - **Duplicate lookup:** Pass a borrowed `RowRef` constructed from existing pins into lookup and comparison helpers to avoid redundant row acquisition. # API and ABI breaking changes N/A # Rollback safety impact N/A # Expected complexity level and risk 3 # Testing Relies on existing coverage

# Description of Changes Tiered storage makes loading rows fallible. When mutation code already holds a `PinnedRow`, it can reuse the row’s page and blob pins instead of performing another storage lookup during cleanup. This PR uses those existing pins more consistently, keeping cleanup free of additional storage reads and making unsafe mutation error behavior easier to reason about: - Route insert/update comparison failures through existing cleanup, using the attempted row’s retained pins. - Restore the old row’s index entries without repeating duplicate-row comparisons. - During insertion, load the committed row being returned before removing its transaction-local deletion marker. The PR also extends commit preparation to pin pages needed for pointer-map hash-collision comparisons. This protects committed-state, since those comparisons must not encounter a storage failure after committed rows have already been changed. # API and ABI breaking changes N/A # Rollback safety impact N/A # Expected complexity level and risk 3 # Testing Existing coverage

## Description of Changes Fix two cases where rolling back a transaction could leave committed tables with different duplicate-detection or uniqueness behavior: - **Preserve the pointer map when undoing a non-unique index addition.** Adding that index leaves the existing pointer map untouched, but rollback previously replaced it with `None`. Subsequent duplicate checks could panic. Restore the saved map only when undoing a unique index addition. - **Preserve indexes that were already unique before adding a constraint.** Rollback previously made every matching index non-unique, including indexes enforcing an existing constraint. Convert and record only previously non-unique indexes, so undo reverses only changes made by this operation. Both fixes use the existing undo records. ## API and ABI breaking changes N/A ## Rollback safety impact N/A ## Expected complexity level and risk 2 ## Testing - Verify that rolling back unique and non-unique index additions restores the pointer map and preserves duplicate detection. - Verify that rolling back an additional unique constraint preserves the original constraint, its index uniqueness, and rejection of conflicting inserts.

kim added this pull request to stack #6306 October 15, 2026 14:44

kim force-pushed the kim/tiered-storage-integration branch from 7625b27 to b840802 Compare October 19, 2026 06:31

joshua-spacetime reviewed Oct 21, 2026

View reviewed changes

kim force-pushed the kim/tiered-storage-base branch from e1cd71a to 63d327f Compare October 22, 2026 16:05

kim force-pushed the kim/tiered-storage-integration branch from 8c3a919 to 4a229fb Compare October 22, 2026 16:10

kim removed this pull request from stack #6306 October 22, 2026 16:12

kim added this pull request to stack #6360 October 22, 2026 16:21

kim force-pushed the kim/tiered-storage-integration branch 5 times, most recently from f9e3421 to 89a9a07 Compare October 23, 2026 22:51

kim force-pushed the kim/tiered-storage-base branch from 63d327f to 490b308 Compare October 23, 2026 23:04

kim force-pushed the kim/tiered-storage-integration branch from 89a9a07 to 002db2f Compare October 23, 2026 23:06

kim force-pushed the kim/tiered-storage-base branch from 490b308 to 397e694 Compare October 23, 2026 23:06

joshua-spacetime force-pushed the kim/tiered-storage-base branch from 397e694 to 6905eab Compare October 29, 2026 18:41

joshua-spacetime force-pushed the kim/tiered-storage-integration branch 2 times, most recently from 1e739db to bb400af Compare October 30, 2026 20:00

kim and others added 5 commits October 30, 2026 17:14

Tiered storage base

e6b6ece

Introduces the datastructures for tiered storage / faultable objects.

Tests + fixes

c8b7a86

Fix blob set races

b6daafa

kim added 23 commits October 30, 2026 17:14

Require the page slot lock to be held when acquiring a frame lock, and

2ac9758

the page to be faulted in.

Use a Mutex for the page slot

c5a8d65

Guard against races in page eviction

fb2fe0e

Put page index back into free_page_slots if allocation fails.

dbe645f

Count only resident blob bytes for memory accounting

aab5d33

Reuse registry frames when reserving already-pinned blobs

b44a60f

Auto-evict pages under budget pressure

5dbd586

Now that ReservedBlob contains an Arc<Frame>, we can use ptr_eq to

0492986

assert that an existing frame is actually the same allocation.

Use arrays instead of slices for object key / hashes

18b2e73

Properly roll back row insertion when blob writing fails

6f247f9

Make index deletion atomic

4c07959

Commit changes only after potential pointer map rebuild succeeded.

Make make_index_non_unique atomic

a0eee5a

If the pointer map needs to be rebuilt, rebuild it first before committing the changes.

Make constraint manipulations atomic

17c26ee

Avoid going through RowRef when collecting row pointers.

5254e8c

This prevents unnecessarily faulting in blobs. With tiered storage, constructing a RowRef requires to load a BlobReadSet, so calling `scan_rows` with an empty BlobSet is not equivalent to the previous `NullBlobStore`.

Roll back migrations when view refresh fails

e9c2540

Make delete_equal_row atomic (used during replay)

686c6bb

Clean up rebase artifacts

ad2e5c3

Undo physical inserts when pinning of the resulting RowRef's

339a675

`BlobReadSet` fails.

Restore the tx index when dropping the committed index fails

db05cf6

Make Table::clear atomic

c772edc

Fallibly collect blob references before mutating anything. Delete rows by marking all pages as absent.

Generalize temporary row handling, and apply

a84e245

`MutTxId::delete_by_row_value`, too

Fix two more rollback safety issues.

f61dea2

To make this possible, introduce a `PinnedRow` that works like an owned `RowRef`. When a pinned row is in scope, potentially faulting functions can be unwrapped, as they are guaranteed to not fault the pinned row's page.

Make a number of operations infallible again by passing PinnedRow

8eb4f30

instead of `RowPointer`.

joshua-spacetime force-pushed the kim/tiered-storage-integration branch from bb400af to 8eb4f30 Compare October 31, 2026 00:14

joshua-spacetime added 3 commits November 2, 2026 11:53

kim removed this pull request from stack #6360 November 3, 2026 07:03

kim changed the base branch from kim/tiered-storage-base to master November 3, 2026 16:33

Merge 'origin/master' into 'kim/tiered-storage-integration'

d6c2152

		serialized.write_large_blobs(blob_store)
		serialized.write_large_blobs(blobs)?

		let effects = UpdateEffects::after_migration(res, &tx);
		let effects = UpdateEffects::after_migration(res, &tx)?;

		commit_table.make_index_non_unique(index_id, commit_blob_store)?;
		tx_table.make_index_non_unique(index_id, tx_blob_store)?;

Conversation

kim commented Oct 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of Changes

API and ABI breaking changes

Rollback safety impact

Expected complexity level and risk

Testing

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

joshua-spacetime left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

kim commented Oct 15, 2026 •

edited

Loading