You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
The plugin template's concurrent posts test fails on main with a 500 after 31 seconds #3200
Acme.ReviewNotes.Tests.ReviewNotesPluginTests.PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow, a test of the project that the plugin template generates, fails in the Templates job of CI on main. It sends ten posts together and accepts 201, 409 or 503 for each; one of the ten answers 500, and the test takes about 31 seconds.
It also failed once on the pull request of #3199 (run 43837904406) and passed on the same commit in a second run. The four runs on main before 7a085f3 passed. The job's log holds nothing from the host about the 500: no error code, no exception.
Not known: the cause. What the timing suggests, and nothing more than that: the first failure is on the first push after #3182, which changed how the database row lock behaves on a busy SQLite file (a take that meets a busy writer now waits and answers "held" where it used to throw at once; the review of #3182 noted that a caller can then wait about 35 seconds); #3188 between them changed test code of another project only. If a busy database used to end in the 503 the test accepts and now ends in a long wait and a 500, that would fit the 31 seconds. It has not been checked.
Proposal
Get the host's exception for the 500: generate the project as the Templates job does, run the test repeated, with the host's log captured.
Fix the cause. If it is in the product, a busy database must keep answering what the specification says a write answers under contention, quickly, and never a 500. If it is in the template's test, fix the template.
Acceptance
The cause is named in this issue, with the host's exception
The test passes repeated on main, in CI's Templates job, and the fix has a test in the repository's own tests if the cause is in the product
If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now
Size: S. Goes to main, as bugs do. It keeps main red, so it comes before the tag.
The 500 comes from the row lock, src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs, on the first use of a lock name: TakeAsync keeps its write transaction open on a failed insert while ExistsAsync reads cms_lock on another connection. #3182 (7a085f3) introduced it.
What the ten posts do: each one takes the plugin's lock review-notes.notes (10 second wait), counts the site's notes and inserts one while it holds the lock. A null lock answers 503, a full count answers 409. The test host is a fresh in-memory SQLite database with a shared cache, so cms_lock has no row yet, and several posts read "no row" and insert it at the same time.
One post wins the insert. A loser's SaveChangesAsync fails with UNIQUE constraint failed: cms_lock.name.
Since Keep a SQLite lock holder's lease while a write transaction blocks it #3182, that insert runs inside await using var transaction = await context.Database.BeginTransactionAsync(...). In the catch (DbUpdateException) the transaction is still open and holds the shared cache's write lock, and ExistsAsync opens a second connection to read cms_lock. In shared-cache mode that read gets SQLITE_LOCKED, and Microsoft.Data.Sqlite retries it until the command timeout. The take is waiting on itself: Failed executing DbCommand (30,148ms) ... SELECT EXISTS (SELECT 1 FROM "cms_lock" ...) and then SqliteException: SQLite Error 6: 'database table is locked: cms_lock'.
For those 30 seconds every other take blocks reading cms_lock, so they overshoot their 10 second wait and answer 503. The post that holds the lock blocks too. Its SaveChangesAsync of the note cannot begin a transaction while the stuck transaction is open, and after 30 seconds it throws. That is the 500 (FCMS3199, logged by ProblemDetailsExceptionHandler, event 9110):
Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 6: 'database table is locked'.
at Microsoft.Data.Sqlite.SqliteConnection.BeginTransaction(IsolationLevel isolationLevel, Boolean deferred)
...
at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(...)
at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(...)
at Acme.ReviewNotes.ReviewNotesEndpoints.<>c.<<MapReviewNotesEndpoints>b__2_1>d.MoveNext() in .../ReviewNotesEndpoints.cs:line 69
Response: {"type":"urn:fluentcms:error:FCMS3199","title":"Internal error","status":500,"code":"FCMS3199",...}. The 30 second command timeout plus the test's own work is the 31 seconds seen in CI.
Before #3182 there was no explicit transaction. SaveChangesAsync rolled back its own transaction when the insert failed, so ExistsAsync read a free table, and the race cost a few milliseconds. On a file database in WAL mode the read does not wait for the writer, so this mostly affects the in-memory database of FluentCmsTestHost, which is what plugin authors' tests use. In any case, a take should not hold a write transaction while it waits on another connection.
Evidence
The generated Acme.ReviewNotes project, as templates/test-templates.sh makes it, ran 20 times at each commit, with the host's log captured in a local copy of the test. "Race" means a run whose log shows the losing insert. "Stall" means a command that failed after about 30 seconds.
tip plus the rollback below (local, not committed)
0 / 20
20
0
8–14 s
At 7a085f3 and at the tip, every run with the race stalls for 30 seconds, and six to eight of the ten posts answer 503. A run fails only when the stall overlaps the write of the post that holds the lock. That is why CI sometimes passes: a passing run there still took 30 extra seconds. The test asks for nothing the product does not promise: 201, 409 and a 503 from a null lock are what the plugin returns.
Fix (in DatabaseDistributedLock.cs, so it goes to the lane of #3195)
Roll back the take's transaction before checking whether the row exists:
catch(DbUpdateException){// Another holder made the row first. Only the first use of a name inserts, so this is rare.awaittransaction.RollbackAsync(CancellationToken.None);if(awaitExistsAsync(name))
A read inside the same transaction would not work on PostgreSQL, where the failed insert aborts the transaction. The rollback works on all three providers.
Regression test for tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs: on a new FluentCmsTestHost() (in-memory, shared cache), about ten concurrent TryAcquireAsync calls on a name never used before, each with a short timeout and each releasing at once. All of them end within a few seconds, well under the 30 second command timeout, and none throws. Without the rollback, the losers of the insert block for 30 seconds.
The spec (backend spec, "Multi-instance as built") already says a take runs in a transaction of its own. It could add that a take which loses the first insert rolls back before it reads the row. #3195 does not change this path at present: its diff touches the catch (DbUpdateException) block only as context.
Coordinator: the cause above matches main (the catch block in DatabaseDistributedLock.TakeAsync reads through ExistsAsync while the take's transaction is open). It is fixed in a PR of its own, ahead of #3195, since main is red on it; #3195 merges main afterwards.
Activity
pournasserian commentedon Nov 16, 2026
Cause
The 500 comes from the row lock,
src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs, on the first use of a lock name:TakeAsynckeeps its write transaction open on a failed insert whileExistsAsyncreadscms_lockon another connection. #3182 (7a085f3) introduced it.What the ten posts do: each one takes the plugin's lock
review-notes.notes(10 second wait), counts the site's notes and inserts one while it holds the lock. A null lock answers 503, a full count answers 409. The test host is a fresh in-memory SQLite database with a shared cache, socms_lockhas no row yet, and several posts read "no row" and insert it at the same time.SaveChangesAsyncfails withUNIQUE constraint failed: cms_lock.name.await using var transaction = await context.Database.BeginTransactionAsync(...). In thecatch (DbUpdateException)the transaction is still open and holds the shared cache's write lock, andExistsAsyncopens a second connection to readcms_lock. In shared-cache mode that read getsSQLITE_LOCKED, and Microsoft.Data.Sqlite retries it until the command timeout. The take is waiting on itself:Failed executing DbCommand (30,148ms) ... SELECT EXISTS (SELECT 1 FROM "cms_lock" ...)and thenSqliteException: SQLite Error 6: 'database table is locked: cms_lock'.cms_lock, so they overshoot their 10 second wait and answer 503. The post that holds the lock blocks too. ItsSaveChangesAsyncof the note cannot begin a transaction while the stuck transaction is open, and after 30 seconds it throws. That is the 500 (FCMS3199, logged byProblemDetailsExceptionHandler, event 9110):Response:
{"type":"urn:fluentcms:error:FCMS3199","title":"Internal error","status":500,"code":"FCMS3199",...}. The 30 second command timeout plus the test's own work is the 31 seconds seen in CI.Before #3182 there was no explicit transaction.
SaveChangesAsyncrolled back its own transaction when the insert failed, soExistsAsyncread a free table, and the race cost a few milliseconds. On a file database in WAL mode the read does not wait for the writer, so this mostly affects the in-memory database ofFluentCmsTestHost, which is what plugin authors' tests use. In any case, a take should not hold a write transaction while it waits on another connection.Evidence
The generated
Acme.ReviewNotesproject, astemplates/test-templates.shmakes it, ran 20 times at each commit, with the host's log captured in a local copy of the test. "Race" means a run whose log shows the losing insert. "Stall" means a command that failed after about 30 seconds.main)At 7a085f3 and at the tip, every run with the race stalls for 30 seconds, and six to eight of the ten posts answer 503. A run fails only when the stall overlaps the write of the post that holds the lock. That is why CI sometimes passes: a passing run there still took 30 extra seconds. The test asks for nothing the product does not promise: 201, 409 and a 503 from a null lock are what the plugin returns.
Fix (in
DatabaseDistributedLock.cs, so it goes to the lane of #3195)Roll back the take's transaction before checking whether the row exists:
A read inside the same transaction would not work on PostgreSQL, where the failed insert aborts the transaction. The rollback works on all three providers.
Regression test for
tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs: on anew FluentCmsTestHost()(in-memory, shared cache), about ten concurrentTryAcquireAsynccalls on a name never used before, each with a short timeout and each releasing at once. All of them end within a few seconds, well under the 30 second command timeout, and none throws. Without the rollback, the losers of the insert block for 30 seconds.The spec (backend spec, "Multi-instance as built") already says a take runs in a transaction of its own. It could add that a take which loses the first insert rolls back before it reads the row. #3195 does not change this path at present: its diff touches the
catch (DbUpdateException)block only as context.pournasserian commentedon Nov 16, 2026
Coordinator: the cause above matches
main(the catch block inDatabaseDistributedLock.TakeAsyncreads throughExistsAsyncwhile the take's transaction is open). It is fixed in a PR of its own, ahead of #3195, sincemainis red on it; #3195 mergesmainafterwards.Bring main into feature/a1 and record what a confirmed logout ends (#…