Skip to content

Narrow the take-over race of the row lock on SQLite - #3195

Open
pournasserian wants to merge 6 commits into
mainfrom
3186-lock-take-over-race
Open

pournasserian wants to merge 6 commits into
mainfrom
3186-lock-take-over-race

Conversation

@pournasserian

@pournasserian pournasserian commented Nov 16, 2026 •

Copy link
Copy Markdown
Member

Part of #3186

#3182 made a lock holder on SQLite keep its lock through a long write transaction. Its review measured four things that remained. This pull request fixes three of them and narrows the fourth, the take-over race, which it does not close.

The race: narrowed, not closed. The holder's lease runs out while its renewals wait for SQLite's one writer. When the transaction ends, a waiter that found the lease over can get the writer first; before this change it did in 4 of 16 measured rounds.

A take now reads the row again inside its transaction. On a database with one writer, which the new provider member IDataProvider.HasOneWriter names (true on SQLite, false on PostgreSQL and SQL Server), a take-over is confirmed:

  • The first sighting of an owner and lease_until over answers "held", and the next try comes a second later.
  • That try takes the lock over only if it finds the same owner and lease_until, and its BEGIN took at most 50 ms. The BEGIN is timed after the connection is opened, so the open and its pragmas do not count.
  • A try whose BEGIN waited longer starts the confirmation again, and a try that gave up as busy drops it.
  • A confirmation goes on for up to 5 s past the caller's timeout and keeps its full second, so TimeSpan.Zero callers (export and import) still take over a holder that stopped.
  • A free row, a released one and the first use of a name are taken at once.

What it closes. A long transaction alone no longer lets a waiter in, because a take that waited for it starts the confirmation again.

What remains. Two hosts on one file, a holder lease of 1.5 s, the lease over inside a long write transaction, and other writers running 5 ms transactions with 10 ms pauses. The table gives the rounds in which the waiter took the live holder's lock. My counts were measured after merging main with #3210. The last row is the holder's own transactions back to back (a table rebuild), where a waiter could see the lease over in one gap and confirm it in another.

Other writers First review Second review Second review, main before this change Mine
0 0 of 44 0 of 50 22 of 30 0 of 20
2 0 of 24
4 0 of 24
8 3 of 55 3 of 50 17 of 30 0 of 20
16 2 of 23 7 of 50 16 of 30 1 of 20
Back-to-back transactions 0 of 30, twice 12 and 19 of 30

The cause: a renewal that has waited longer than about 228 ms polls every 100 ms. Every fresh writer, the confirming take included, polls after 1, 2, 5, 10 and 15 ms, so the window gives the waiter several tries against the renewal's one or none. A backlog of requests that queued behind the long transaction is that load.

The holder's Lost fires 64 to 119 ms after a take-over only with no other writers. Under write load the renewal that would notice the loss is starved by the same writers: the second review saw Lost fire 63 ms to 1.73 s after the take-over in 11 of 14 take-overs, and not yet after 1.6 to 2.1 s in the other 3.

Hardening tried and dropped. Requiring three tries spread over the second, each with a BEGIN under 2 ms, did not help: 4 of 60 rounds with 16 writers and 3 of 40 with 32, measured before the merge on a file database, which the stall of #3200 on in-memory SQLite does not touch. It is not in this PR. A longer confirmation would not help under sustained load either.

Liveness. A holder that stopped is taken over about 2 s after its lease runs out on a database whose writer is free: up to 1 s of polling to see it, then the 1 s confirmation. Each try whose BEGIN waited adds a second. Of two waiters, the conditional update lets exactly one take the lock.

PostgreSQL and SQL Server. Nothing is confirmed there. They get the read inside the transaction and the update conditional on the owner read, which add one round trip to a take of a free or expired row. One edge changes: a holder whose lease is over and that releases between the take's read and its update leaves the update 0 rows. The take then answers "held" for one poll, and a TimeSpan.Zero caller gets null where main took the lock.

The other three parts:

  • A busy database read as "lock held". A wait that reaches its timeout while its last try gave up as busy logs 9166. The fluentcms:schema-sync timeout and docs/content-migrations.md say that this can also be the cause, and that the row may then be free. The IDistributedLock contract is unchanged.
  • A lost lock in schema apply gave FCMS3302 ... TaskCanceledException. The lost lock (Lost canceled, the caller's token not) is now caught before the database's own exceptions. The message says the lock was lost, whatever canceled Lost, and names the type of a provider's own exception when the canceled command ended with one. It gives FCMS3339 during planning (nothing applied) or during the operations (rolled back), and FCMS3340 after they committed and before the history row. Each says to run again.
  • Event 9160 misreported. It now logs from the row read inside the take's transaction, and the update is conditional on that owner. It names a holder only when that holder's lease was taken over. Its text adds that, on a database with one writer, the holder may not have stopped but could not renew behind other writes.

The remarks of IDistributedLock.TryAcquireAsync say that an implementation may answer shortly after the timeout while it confirms a take-over.

The specification's paragraph "Multi-instance as built", with these figures, the error codes of schema apply and the event range (9160 to 9166) describe this.

Tests. Core integration tests on a SQLite file:

  • a waiter at the end of a long write transaction, 10 rounds: the holder keeps the lock through the waiter's sighting and its confirming tries, and no 9160 is logged;
  • a holder that stopped is taken over at the confirming try, with 9160 naming it;
  • a TimeSpan.Zero caller takes over a holder that stopped in 30 of 30 rounds;
  • of two waiters, exactly one takes the lock, and the loser is canceled;
  • a lock released while the waiter waited is taken with no 9160;
  • a busy timeout on a free row logs 9166.

Data tests cover a lost lock in schema apply before the plan, during the operations (after the first one ran, which is rolled back) and after them, each finished by a second run, and the schema sync timeout text.

Dependencies: none.

  • With a waiter present at the end of a holder's long transaction, the holder keeps the lock, in a test that fails today (met with no other writers; not met under concurrent writes, as measured above)
  • A wait that times out on a busy database says so in the log and in the schema sync error
  • A schema apply that loses its lock after its change says to run again
  • Event 9160 names a holder only when that holder's lease was taken over

@pournasserian pournasserian added type:bug Defect area:storage EF Core model, providers and migrations labels Nov 16, 2026
pournasserian added a commit that referenced this pull request Nov 16, 2026
Closes #3200

Since #3182 the take of the row lock runs in an explicit transaction. On the first use of a name, takes that start together race on the insert of its row. A loser's insert fails with a unique violation, and in `catch (DbUpdateException)` it called `ExistsAsync`, which reads `cms_lock` on another connection while the loser's own transaction is still open. On the in-memory SQLite database of `FluentCmsTestHost` (shared cache), that open transaction holds the write lock, so the read got `SQLITE_LOCKED` and retried for the 30-second command timeout. Meanwhile other takes overshot their waits and the holder's own write failed with `database table is locked`, which was the 500 of the plugin template's `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` in the `Templates` job. The take now rolls its transaction back before it reads the row, as `SaveChangesAsync` did before #3182. The backend spec's "Multi-instance as built" paragraph gets one sentence on it. The preview release notes do not describe the lock change of #3182, so they are unchanged.

What I checked in `DatabaseDistributedLock`, by reading it:

- No other path reads or writes on another connection or scope while a take's or a renewal's transaction is open. The `ExecuteUpdateAsync` path of the take and `RenewAsync`'s update run on the transaction's own context. When either throws, `await using` disposes the transaction (a rollback) before the caller's `catch`, and `IsBusy` only inspects the exception. `RenewAsync`'s fallback read runs after its transaction is disposed, on the same context. `ReleaseAsync` opens no explicit transaction, and the handle waits for its renewal loop to end before it releases. `TryAcquireAsync` only waits after a null take and touches no database.
- PostgreSQL: a failed statement puts the transaction block in an aborted state. The docs give `ROLLBACK TO` a savepoint, "short of rolling it back completely", as the way out of it ([tutorial: transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html)), so the rollback is allowed and needed, and a read in the same transaction would fail.
- SQL Server: with `SET XACT_ABORT OFF`, the default and not changed anywhere in the code, a failed statement can roll back only that statement and leave the transaction open ([SET XACT_ABORT](https://learn.microsoft.com/sql/t-sql/statements/set-xact-abort-transact-sql)), so `Rollback` ends a pending transaction. `SqlTransaction.Rollback` throws `InvalidOperationException` only when the server already rolled the transaction back ([SqlTransaction.Rollback](https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqltransaction.rollback)), which a unique violation does not do with `XACT_ABORT OFF`.

Measured by the diagnosing lane (the generated `Acme.ReviewNotes` test, 20 runs per build under load):

| Build | Failed (500) | Race | 30 s stall | Most runs took |
| --- | --- | --- | --- | --- |
| 7b6cd7e, before #3182 | 0 / 20 | 18 | 0 | 10–25 s |
| 7a085f3 (#3182) | 9 / 20 | 18 | 18 | 37–41 s |
| c0f6d5e (tip of `main`) | 7 / 20 | 19 | 19 | 37–54 s |
| tip plus the rollback | 0 / 20 | 20 | 0 | 8–14 s |

Measured here:

| Run | Result |
| --- | --- |
| New test `TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout`, without the rollback | 30 / 30 failed, all on the elapsed time (test took 36–66 s) |
| The same test with the rollback | 0 / 30 failed (each process 8–12 s, host start included) |
| `DistributedLockTests` with the rollback, SQLite | 15 / 15 passed |
| `FluentCMS.Data.Tests` `Migrations` namespace with the rollback, PostgreSQL 15 | 143 passed, 46 skipped, 0 failed |
| `Acme.ReviewNotes` `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` from packages of this branch, as the `Templates` job makes it | 0 / 20 failed, 6.7–15.5 s per run |

The new test runs ten takes with a 500 ms wait on a name never used before, on the in-memory database. Exactly one holds the lock: the lease is 30 seconds on a wall clock that does not move, and nobody releases before the count.

## Dependencies

None. #3195 follows this pull request and merges `main` after it.

## Acceptance

- [x] The cause is named in this issue, with the host's exception
- [ ] The test passes repeated on `main`, in CI's `Templates` job, and the fix has a test in the repository's own tests if the cause is in the product (the regression test is in `DistributedLockTests`, and the generated test passed 20 runs locally; CI's `Templates` job on `main` is not verified yet)
- [x] If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now
@pournasserian pournasserian changed the title Close the take-over race of the row lock on SQLite Narrow the take-over race of the row lock on SQLite Nov 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:storage EF Core model, providers and migrations type:bug Defect

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant