Repository navigation
Narrow the take-over race of the row lock on SQLite - #3195
Open
pournasserian wants to merge 6 commits into
Open
pournasserian wants to merge 6 commits into
pournasserian wants to merge 6 commits into
Conversation
This was referenced Nov 16, 2026
pournasserian
added a commit
that referenced
this pull request
Nov 16, 2026
Closes #3200 Since #3182 the take of the row lock runs in an explicit transaction. On the first use of a name, takes that start together race on the insert of its row. A loser's insert fails with a unique violation, and in `catch (DbUpdateException)` it called `ExistsAsync`, which reads `cms_lock` on another connection while the loser's own transaction is still open. On the in-memory SQLite database of `FluentCmsTestHost` (shared cache), that open transaction holds the write lock, so the read got `SQLITE_LOCKED` and retried for the 30-second command timeout. Meanwhile other takes overshot their waits and the holder's own write failed with `database table is locked`, which was the 500 of the plugin template's `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` in the `Templates` job. The take now rolls its transaction back before it reads the row, as `SaveChangesAsync` did before #3182. The backend spec's "Multi-instance as built" paragraph gets one sentence on it. The preview release notes do not describe the lock change of #3182, so they are unchanged. What I checked in `DatabaseDistributedLock`, by reading it: - No other path reads or writes on another connection or scope while a take's or a renewal's transaction is open. The `ExecuteUpdateAsync` path of the take and `RenewAsync`'s update run on the transaction's own context. When either throws, `await using` disposes the transaction (a rollback) before the caller's `catch`, and `IsBusy` only inspects the exception. `RenewAsync`'s fallback read runs after its transaction is disposed, on the same context. `ReleaseAsync` opens no explicit transaction, and the handle waits for its renewal loop to end before it releases. `TryAcquireAsync` only waits after a null take and touches no database. - PostgreSQL: a failed statement puts the transaction block in an aborted state. The docs give `ROLLBACK TO` a savepoint, "short of rolling it back completely", as the way out of it ([tutorial: transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html)), so the rollback is allowed and needed, and a read in the same transaction would fail. - SQL Server: with `SET XACT_ABORT OFF`, the default and not changed anywhere in the code, a failed statement can roll back only that statement and leave the transaction open ([SET XACT_ABORT](https://learn.microsoft.com/sql/t-sql/statements/set-xact-abort-transact-sql)), so `Rollback` ends a pending transaction. `SqlTransaction.Rollback` throws `InvalidOperationException` only when the server already rolled the transaction back ([SqlTransaction.Rollback](https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqltransaction.rollback)), which a unique violation does not do with `XACT_ABORT OFF`. Measured by the diagnosing lane (the generated `Acme.ReviewNotes` test, 20 runs per build under load): | Build | Failed (500) | Race | 30 s stall | Most runs took | | --- | --- | --- | --- | --- | | 7b6cd7e, before #3182 | 0 / 20 | 18 | 0 | 10–25 s | | 7a085f3 (#3182) | 9 / 20 | 18 | 18 | 37–41 s | | c0f6d5e (tip of `main`) | 7 / 20 | 19 | 19 | 37–54 s | | tip plus the rollback | 0 / 20 | 20 | 0 | 8–14 s | Measured here: | Run | Result | | --- | --- | | New test `TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout`, without the rollback | 30 / 30 failed, all on the elapsed time (test took 36–66 s) | | The same test with the rollback | 0 / 30 failed (each process 8–12 s, host start included) | | `DistributedLockTests` with the rollback, SQLite | 15 / 15 passed | | `FluentCMS.Data.Tests` `Migrations` namespace with the rollback, PostgreSQL 15 | 143 passed, 46 skipped, 0 failed | | `Acme.ReviewNotes` `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` from packages of this branch, as the `Templates` job makes it | 0 / 20 failed, 6.7–15.5 s per run | The new test runs ten takes with a 500 ms wait on a name never used before, on the in-memory database. Exactly one holds the lock: the lease is 30 seconds on a wall clock that does not move, and nobody releases before the count. ## Dependencies None. #3195 follows this pull request and merges `main` after it. ## Acceptance - [x] The cause is named in this issue, with the host's exception - [ ] The test passes repeated on `main`, in CI's `Templates` job, and the fix has a test in the repository's own tests if the cause is in the product (the regression test is in `DistributedLockTests`, and the generated test passed 20 runs locally; CI's `Templates` job on `main` is not verified yet) - [x] If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now
# Conflicts: # docs/planning/backend-technical-spec.md
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part of #3186
#3182 made a lock holder on SQLite keep its lock through a long write transaction. Its review measured four things that remained. This pull request fixes three of them and narrows the fourth, the take-over race, which it does not close.
The race: narrowed, not closed. The holder's lease runs out while its renewals wait for SQLite's one writer. When the transaction ends, a waiter that found the lease over can get the writer first; before this change it did in 4 of 16 measured rounds.
A take now reads the row again inside its transaction. On a database with one writer, which the new provider member
IDataProvider.HasOneWriternames (true on SQLite, false on PostgreSQL and SQL Server), a take-over is confirmed:ownerandlease_untilover answers "held", and the next try comes a second later.ownerandlease_until, and its BEGIN took at most 50 ms. The BEGIN is timed after the connection is opened, so the open and its pragmas do not count.TimeSpan.Zerocallers (export and import) still take over a holder that stopped.What it closes. A long transaction alone no longer lets a waiter in, because a take that waited for it starts the confirmation again.
What remains. Two hosts on one file, a holder lease of 1.5 s, the lease over inside a long write transaction, and other writers running 5 ms transactions with 10 ms pauses. The table gives the rounds in which the waiter took the live holder's lock. My counts were measured after merging
mainwith #3210. The last row is the holder's own transactions back to back (a table rebuild), where a waiter could see the lease over in one gap and confirm it in another.mainbefore this changeThe cause: a renewal that has waited longer than about 228 ms polls every 100 ms. Every fresh writer, the confirming take included, polls after 1, 2, 5, 10 and 15 ms, so the window gives the waiter several tries against the renewal's one or none. A backlog of requests that queued behind the long transaction is that load.
The holder's
Lostfires 64 to 119 ms after a take-over only with no other writers. Under write load the renewal that would notice the loss is starved by the same writers: the second review sawLostfire 63 ms to 1.73 s after the take-over in 11 of 14 take-overs, and not yet after 1.6 to 2.1 s in the other 3.Hardening tried and dropped. Requiring three tries spread over the second, each with a BEGIN under 2 ms, did not help: 4 of 60 rounds with 16 writers and 3 of 40 with 32, measured before the merge on a file database, which the stall of #3200 on in-memory SQLite does not touch. It is not in this PR. A longer confirmation would not help under sustained load either.
Liveness. A holder that stopped is taken over about 2 s after its lease runs out on a database whose writer is free: up to 1 s of polling to see it, then the 1 s confirmation. Each try whose BEGIN waited adds a second. Of two waiters, the conditional update lets exactly one take the lock.
PostgreSQL and SQL Server. Nothing is confirmed there. They get the read inside the transaction and the update conditional on the owner read, which add one round trip to a take of a free or expired row. One edge changes: a holder whose lease is over and that releases between the take's read and its update leaves the update 0 rows. The take then answers "held" for one poll, and a
TimeSpan.Zerocaller gets null wheremaintook the lock.The other three parts:
fluentcms:schema-synctimeout anddocs/content-migrations.mdsay that this can also be the cause, and that the row may then be free. TheIDistributedLockcontract is unchanged.schema applygaveFCMS3302 ... TaskCanceledException. The lost lock (Lostcanceled, the caller's token not) is now caught before the database's own exceptions. The message says the lock was lost, whatever canceledLost, and names the type of a provider's own exception when the canceled command ended with one. It givesFCMS3339during planning (nothing applied) or during the operations (rolled back), andFCMS3340after they committed and before the history row. Each says to run again.The remarks of
IDistributedLock.TryAcquireAsyncsay that an implementation may answer shortly after the timeout while it confirms a take-over.The specification's paragraph "Multi-instance as built", with these figures, the error codes of schema apply and the event range (9160 to 9166) describe this.
Tests. Core integration tests on a SQLite file:
TimeSpan.Zerocaller takes over a holder that stopped in 30 of 30 rounds;Data tests cover a lost lock in
schema applybefore the plan, during the operations (after the first one ran, which is rolled back) and after them, each finished by a second run, and the schema sync timeout text.Dependencies: none.
schema applythat loses its lock after its change says to run again