Skip to content

The plugin template's concurrent posts test fails on main with a 500 after 31 seconds #3200

Closed

Description

@pournasserian

Problem

Acme.ReviewNotes.Tests.ReviewNotesPluginTests.PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow, a test of the project that the plugin template generates, fails in the Templates job of CI on main. It sends ten posts together and accepts 201, 409 or 503 for each; one of the ten answers 500, and the test takes about 31 seconds.

Commit on main Run Templates
7a085f3 (#3182) 43811756498 passed
af45a9d (#3188) 43813905252 failed
3cf43ea (#3190) 43821796595 failed
c0f6d5e (#3194) 43827584585 passed

It also failed once on the pull request of #3199 (run 43837904406) and passed on the same commit in a second run. The four runs on main before 7a085f3 passed. The job's log holds nothing from the host about the 500: no error code, no exception.

Not known: the cause. What the timing suggests, and nothing more than that: the first failure is on the first push after #3182, which changed how the database row lock behaves on a busy SQLite file (a take that meets a busy writer now waits and answers "held" where it used to throw at once; the review of #3182 noted that a caller can then wait about 35 seconds); #3188 between them changed test code of another project only. If a busy database used to end in the 503 the test accepts and now ends in a long wait and a 500, that would fit the 31 seconds. It has not been checked.

Proposal

  • Get the host's exception for the 500: generate the project as the Templates job does, run the test repeated, with the host's log captured.
  • Find which change introduced it: run it on 7b6cd7e (before Keep a SQLite lock holder's lease while a write transaction blocks it #3182), on 7a085f3 and on the tip, enough times each to tell.
  • Fix the cause. If it is in the product, a busy database must keep answering what the specification says a write answers under contention, quickly, and never a 500. If it is in the template's test, fix the template.

Acceptance

  • The cause is named in this issue, with the host's exception
  • The test passes repeated on main, in CI's Templates job, and the fix has a test in the repository's own tests if the cause is in the product
  • If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now

Size: S. Goes to main, as bugs do. It keeps main red, so it comes before the tag.

Activity

  1. pournasserian commented on Nov 16, 2026

    @pournasserian
    MemberAuthor

    Cause

    The 500 comes from the row lock, src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs, on the first use of a lock name: TakeAsync keeps its write transaction open on a failed insert while ExistsAsync reads cms_lock on another connection. #3182 (7a085f3) introduced it.

    What the ten posts do: each one takes the plugin's lock review-notes.notes (10 second wait), counts the site's notes and inserts one while it holds the lock. A null lock answers 503, a full count answers 409. The test host is a fresh in-memory SQLite database with a shared cache, so cms_lock has no row yet, and several posts read "no row" and insert it at the same time.

    1. One post wins the insert. A loser's SaveChangesAsync fails with UNIQUE constraint failed: cms_lock.name.
    2. Since Keep a SQLite lock holder's lease while a write transaction blocks it #3182, that insert runs inside await using var transaction = await context.Database.BeginTransactionAsync(...). In the catch (DbUpdateException) the transaction is still open and holds the shared cache's write lock, and ExistsAsync opens a second connection to read cms_lock. In shared-cache mode that read gets SQLITE_LOCKED, and Microsoft.Data.Sqlite retries it until the command timeout. The take is waiting on itself:
      Failed executing DbCommand (30,148ms) ... SELECT EXISTS (SELECT 1 FROM "cms_lock" ...) and then SqliteException: SQLite Error 6: 'database table is locked: cms_lock'.
    3. For those 30 seconds every other take blocks reading cms_lock, so they overshoot their 10 second wait and answer 503. The post that holds the lock blocks too. Its SaveChangesAsync of the note cannot begin a transaction while the stuck transaction is open, and after 30 seconds it throws. That is the 500 (FCMS3199, logged by ProblemDetailsExceptionHandler, event 9110):
    Microsoft.Data.Sqlite.SqliteException (0x80004005): SQLite Error 6: 'database table is locked'.
       at Microsoft.Data.Sqlite.SqliteConnection.BeginTransaction(IsolationLevel isolationLevel, Boolean deferred)
       ...
       at Microsoft.EntityFrameworkCore.Update.Internal.BatchExecutor.ExecuteAsync(...)
       at Microsoft.EntityFrameworkCore.DbContext.SaveChangesAsync(...)
       at Acme.ReviewNotes.ReviewNotesEndpoints.<>c.<<MapReviewNotesEndpoints>b__2_1>d.MoveNext() in .../ReviewNotesEndpoints.cs:line 69
    

    Response: {"type":"urn:fluentcms:error:FCMS3199","title":"Internal error","status":500,"code":"FCMS3199",...}. The 30 second command timeout plus the test's own work is the 31 seconds seen in CI.

    Before #3182 there was no explicit transaction. SaveChangesAsync rolled back its own transaction when the insert failed, so ExistsAsync read a free table, and the race cost a few milliseconds. On a file database in WAL mode the read does not wait for the writer, so this mostly affects the in-memory database of FluentCmsTestHost, which is what plugin authors' tests use. In any case, a take should not hold a write transaction while it waits on another connection.

    Evidence

    The generated Acme.ReviewNotes project, as templates/test-templates.sh makes it, ran 20 times at each commit, with the host's log captured in a local copy of the test. "Race" means a run whose log shows the losing insert. "Stall" means a command that failed after about 30 seconds.

    Build Failed (500) Race 30 s stall Most runs took
    7b6cd7e, before #3182 0 / 20 18 0 10–25 s
    7a085f3 (#3182) 9 / 20 18 18 37–41 s
    c0f6d5e (tip of main) 7 / 20 19 19 37–54 s
    tip plus the rollback below (local, not committed) 0 / 20 20 0 8–14 s

    At 7a085f3 and at the tip, every run with the race stalls for 30 seconds, and six to eight of the ten posts answer 503. A run fails only when the stall overlaps the write of the post that holds the lock. That is why CI sometimes passes: a passing run there still took 30 extra seconds. The test asks for nothing the product does not promise: 201, 409 and a 503 from a null lock are what the plugin returns.

    Fix (in DatabaseDistributedLock.cs, so it goes to the lane of #3195)

    Roll back the take's transaction before checking whether the row exists:

                catch (DbUpdateException)
                {
                    // Another holder made the row first. Only the first use of a name inserts, so this is rare.
                    await transaction.RollbackAsync(CancellationToken.None);
                    if (await ExistsAsync(name))

    A read inside the same transaction would not work on PostgreSQL, where the failed insert aborts the transaction. The rollback works on all three providers.

    Regression test for tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs: on a new FluentCmsTestHost() (in-memory, shared cache), about ten concurrent TryAcquireAsync calls on a name never used before, each with a short timeout and each releasing at once. All of them end within a few seconds, well under the 30 second command timeout, and none throws. Without the rollback, the losers of the insert block for 30 seconds.

    The spec (backend spec, "Multi-instance as built") already says a take runs in a transaction of its own. It could add that a take which loses the first insert rolls back before it reads the row. #3195 does not change this path at present: its diff touches the catch (DbUpdateException) block only as context.

  2. pournasserian commented on Nov 16, 2026

    @pournasserian
    MemberAuthor

    Coordinator: the cause above matches main (the catch block in DatabaseDistributedLock.TakeAsync reads through ExistsAsync while the take's transaction is open). It is fixed in a PR of its own, ahead of #3195, since main is red on it; #3195 merges main afterwards.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions