Skip to content

Bring main into feature/a1 and record what a confirmed logout ends - #3216

Merged
pournasserian merged 9 commits into
feature/a1from
sync-main-into-a1-1116
Nov 16, 2026
Merged

pournasserian merged 9 commits into
feature/a1from
sync-main-into-a1-1116

Conversation

@pournasserian

Copy link
Copy Markdown
Member

Brings main at f8ff094 into feature/a1, as a merge commit, so that the feature branch is tested with what main gained since the last sync:

One conflict, in the "Multi-instance as built" paragraph of docs/planning/backend-technical-spec.md: main's text is taken, and the feature branch's own change to the paragraph on the commands in the reference host is kept.

With #3199 in, docs/decisions/0007-admin-sign-in-flow.md says what a logout ends: an unconfirmed one leaves the server's session and its tokens; a confirmed one ends the session and revokes every authorization issued in it; a sign-in from before #3198 and a third-party client's consent are not ended.

Dependencies

None.

Part of #3152.

…#3182)

Closes #3176

On SQLite, the default lock renewed its lease on its own connection, and SQLite has one writer at a time. While the holder's own write transaction was open, the renewal waited for it and gave up after about 35 seconds (the busy timeout, then the retries of Microsoft.Data.Sqlite). The lease ran out, `Lost` was canceled at about 45 seconds, and that canceled the rest of the holder's work.

This was reproduced through FluentCMS:

- `schema apply` with two added columns failed with `FCMS3302`, and neither column was added.
- Schema sync of the same change rolled back.
- A waiter threw `SqliteException` "database is locked" out of `TryAcquireAsync`. That happened on a take-over, on the first use of a lock name, and, in the rollback journal mode, already on the read of the row.

`IDataProvider.IsBusy` recognises a command that gave up waiting for the database's writer. On SQLite that is every error whose primary code is busy or locked; on PostgreSQL and SQL Server it is always false. Such a command changed nothing, and the lock now handles it as follows:

- **A waiter** whose read of the row, take-over or first insert gives up this way counts the lock as held. It ends with the lock or with its timeout, and never with `SqliteException`. It can overshoot its timeout by one such wait: a `TimeSpan.Zero` caller on a busy file now waits about 35 seconds and gets null, where before an exception came after the same wait.
- **A take and a renewal** each run in a transaction of their own and read the clock once it has begun. On SQLite, beginning it waits for the writer, so the lease starts when the writer let them in, and a write that waited never writes a lease that is short or already over. On PostgreSQL and SQL Server, the clock is still read before a statement that can wait on the row, as before. Before, a renewal that waited 25 seconds wrote a lease that ran out 5 seconds later, and a waiter could take a live holder's lock.
- **A holder whose renewal gives up** reads the row; in WAL mode, reads do not wait for the writer.
  - If the row still names the holder, it keeps the lock, renews again at once, and logs once that it waits (9165, Information). No other holder can take the lock while the transaction lasts, because taking it is a write too.
  - If the row names another owner, it cancels `Lost`. A take-over can commit between two such transactions.
  - There is no time bound, since a bound would bring the original bug back.

On PostgreSQL and SQL Server, the take and the renewal now run in an explicit transaction around the same statements. That adds a round trip and does not change what they do.

**The race that remains.** A waiter that found the lease over during a long transaction waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other owner and cancels `Lost`. But a holder with more than one unit of work under the lock may still have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. As measured in review: with a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 rounds, and the holder's `Lost` fired 64 to 119 ms after the commit. When the holder began its next write at once, the waiter took it in none of 16.

Known, and left to #3186:

- A `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again. Running it again records the history.
- The waiter's event 9160 then says that a holder stopped without releasing the lock, while that holder is alive.

The operator hint of the `fluentcms:schema-sync` timeout and `docs/content-migrations.md` now add the missing state: on SQLite, a `lease_until` standing still behind the clock can mean the holder is working inside a long transaction. The specification describes all of this and lists the lock's event ids, 9160 to 9169.

**Tests:**

- Core integration tests, on a write transaction held on a second connection:
  - a holder keeps the lock past a 300 ms lease;
  - a waiter ends with its timeout, on a take-over and on a first use;
  - a renewal and a take that waited for the writer write a lease that is still running;
  - a holder is told when another owner took the row while its renewals keep waiting.
- Data tests on SQLite:
  - `schema apply` and schema sync each finish while their first `ALTER TABLE` is held until a renewal gave up;
  - a waiter on a rollback-journal database whose read of the row waits ends with its timeout.

All of them fail on `main`'s lock except `AHolderIsToldWhenAnotherTookTheLockWhileAWriteTransactionKeepsItsRenewalsWaiting`. That test passes on `main`, where `Lost` fires through lease expiry, and fails on the first commit of this PR, which is what it guards.

Dependencies: none.

- [x] A test holds the lock on SQLite inside a write transaction for longer than the lease, and the work finishes
- [x] A waiter on SQLite ends with the lock's timeout or with the lock, never with `SqliteException`
- [x] PostgreSQL and SQL Server behave as before
Closes #3187

A test bug, not a product bug. The list of received events is a `ConcurrentQueue`, so it never changed under an enumeration. What changed was when the event arrived. `ShouldHaveSingleItem` counts the list once and lists it again for the message, and the queue only grows after `Clear()`. So the count found no event, and by the time the message was built, the event was there.

The event arrived after `PublishAsync` returned, which the spec (In-process handlers, as built) allows. A write waits for each in-process handler at most `FluentCms:Events:Handlers:Timeout`, 5 seconds by default. After that it goes on and leaves the handler running. `EventDispatcher.RunAsync` starts the handler with `Task.Run` and waits on `WaitAsync(options.Timeout, time)`. The failing run took 15 s on a loaded Windows runner, so the handler, which was not even scheduled yet, went past the bound. The tests of the data project read what their handlers saw as soon as the write returns, so all of them depended on that bound never being reached.

- `TestDatabase.CreateServices` gives in-process handlers a 1-minute bound, as `EventLimitsTests` already does where the bound under test is another one. While the bound is not reached, a write returns only after its handlers ran, so no handler can still write to a list once the write has returned. Ordered assertions such as `Calls.ShouldBe([...])` and the database a handler saw stay exact, which a wait for the event would not give. The tests of the bound set their own value in `configure`, which runs after this.
- `Seen.Of<T>()` returns a copy, so an assertion and its message read the same events.

The other event test classes read `Seen` after a write, a dispatch or an awaited drain. The one test that starts a background runner (`OutboxTests`) already waits for the delivery with a bound. The integration tests that record invalidations already take a snapshot after an awaited drain, or wait on a channel with a bound.

## Dependencies

None.

## Acceptance

- [x] The test asserts on a list that can no longer change, and passes repeated under load
- [x] No other test of the event test classes enumerates a list that a handler can still write to
On SQLite, EF Core's migrations lock is a row in `__EFMigrationsLock`. A process killed while it migrates leaves the row behind, and EF then retries every second with no end. I reproduced it: a start with `MigrateOnStartup` and a pending migration, and `fluentcms migrate`, waited until the test cancelled them, and nothing in the log named the row. A start with nothing pending is not held up, because the runner calls `MigrateAsync` only for a stream that has something to apply.

Before each stream that has a pending migration, the runner now reads the row through a new `IDataProvider.ReadMigrationsLockAsync`. On SQLite that is the row's time, or null for a database that does not exist yet, so EF's creator still makes it in WAL mode. On PostgreSQL and SQL Server it is null, because their locks end with their transaction or session.

While the row is there, the runner waits, before it logs "Applying". After 30 seconds it logs a warning once (event 9305) and then looks every 5 seconds. After 15 minutes it fails with a `TimeoutException`. A start stops, and `migrate` exits with 1 and `FCMS3302`. Both the warning and the error:

- name the row and its time, and say that the time is when the lock was taken and is never renewed
- give the `DELETE` that clears the row once no other host and no `fluentcms migrate` runs on the database
- say that if the next run fails on a table or column that already exists, an interrupted table rebuild was partly applied and the backup is restored

The row is never cleared automatically. Its time cannot tell a dead holder from another host's start or a `fluentcms migrate` still rebuilding large tables, and EF has no supported way to bound or clear the lock. One gap remains: two processes that both read no row before either inserts leave the second waiting inside EF, with no warning and no bound, as before.

The tests take the lock through EF's own `IHistoryRepository.AcquireDatabaseLockAsync`, so a change of EF's lock fails them. Another test checks that a first start still creates the database in WAL mode with `writeAheadLog: false`.

On SQL Server, EF's `sp_getapplock` has no lock timeout. A wait for another process's migrations fails at the command timeout (30 seconds) with `Execution Timeout Expired`. That is left to #3192.

The docs say what an operator sees and does:

- `content-migrations.md` covers the partial migration and the remaining gap.
- `docker.md` shows how to delete the row in a volume from a throwaway container, since the image has no shell and no `sqlite3`.
- The backend spec records the design.

Closes #3177

## Dependencies

None.

## Acceptance

- [x] A start on SQLite with an abandoned row in `__EFMigrationsLock` ends within a bound, with an error that names the cause or with the migration applied
- [x] A test holds it
- [x] The operations documentation says what an operator sees and does
Closes #3193

## Cause

`Results.Json(value, options)` builds a `JsonHttpResult<T>`, and its constructor runs `if (!options.IsReadOnly) options.TypeInfoResolver ??= new DefaultJsonTypeInfoResolver();`. That is a check, a read and a set on the shared static options, with no lock.

The options become read-only on first serialization: when a result executes, or on any `JsonSerializer` call with them. If one request reads a null resolver and another request then sets the resolver and locks the options before the first runs the setter, the setter throws `InvalidOperationException` and the request answers 500.

One host is enough to hit this: two concurrent first requests share the static. The race is open once per process, until the options are locked. Reproduced in a test: on options made the same way, without the fix, 5,000 rounds of 8 threads failed 1, 0 and 4 times in three runs.

## Change

The shared options handed to `Results.Json` are now locked with `MakeReadOnly(populateMissingResolver: true)` in a static constructor, so they are complete and read-only before any request has them:

- `AdminJson.Options`
- `SchemaEndpoints.Options`
- `StartupReportJson.Options`
- `MediaEndpoints.PresetJson`

The repository has no source-generated JSON contexts, and `Results.Json` already gave these options the reflection resolver, so the JSON the API writes does not change. `AdminBodiesTests` pass, and a new test compares the text written with the locked options against the text written with options built as before.

The other static options in `src` were already safe and are unchanged:

- They go only to `JsonSerializer`, `JsonNode.ToJsonString`, `JsonElement.Deserialize` or `JsonDocument.Parse`. The serializer fills in a missing resolver through a field and never throws, so their first use is thread-safe. These are `TransferPackage.Json`, `BackupArchive.Json`, `JsonDepth.Serializer`, `PermissionConstraints`, `McpClientState`, `WebhookDispatcher`, `MediaTransfer`, `MediaJson`, `McpArguments`, `McpSiteResources`, `FluentCmsClient` and the CLI's `ManagementClient.Json`.
- `SettingsJson` was already locked.
- `JobRegistry` and `OutboxRegistration` use `JsonSerializerOptions.Web`, which is read-only.

## Dependencies

None.

## Acceptance

- [x] The options cannot be changed after the type is initialized, and a test that uses them from many threads at first use passes repeated
- [x] No other static `JsonSerializerOptions` in `src` is completed lazily by a request

## Tests

- `AdminJsonTests` asserts that the shared options are read-only and complete. It also runs the first-use race (5,000 rounds of 8 threads) on options locked the same way, and checks the written text. Ran 5 times.
- `MediaPresetTests` asserts the same for `PresetJson`.
- Also ran: the fast checks, `dotnet build -c Release -warnaserror`, the host's `ExitCriteriaTests` 3 times, and the integration tests for content, roles, identity, schema, the startup report and OpenAPI.
Closes #3201

It was the test, not the product.

`SharingAsync` starts three hosts on one output cache store. The first adds a grant to the default site's Public role. `RoleVersionInterceptor` stages a `RolePermissionsChangedEvent` for it and kicks the outbox. These hosts kept their outbox worker, so `OutputCacheEviction` evicted `site:{default site}` from the shared store whenever the worker got to it. Every host has the same default site id, so that tag reaches the copies of all three. Usually the eviction landed while the hosts started. On a slow runner it landed after `firstClient` stored its copy and before `sameClient` read it. `same` then answered from its own database, where the Public role has no `content.note.read`, and gave 404. That is `shared.Status.ShouldBe(OK)`. The release build reports it at lines 366/371.

Evidence: with the worker off and the first host's outbox drained right after its read, the test fails every time with `shared.Status should be HttpStatusCode.OK but was NotFound`, from the same frames as the CI failure. The cache served no copy across schemas, and no host answered 404 for a copy it had. #3194 is unrelated.

- The three hosts of `AStoreSharedWithAnotherSchemaServesItNoCopyOfThisOne` run without an outbox worker, as `DeliveryCacheFixture` does. The first host drains its outbox after the grant, so the eviction is done before the reads. The test still shows that the host of the other schema is served nothing.
- `DeliveryHostSiteTests` had the same exposure: a seeded grant with a live worker, then a read that must come from the cache. Its host now runs without an outbox worker too. The other Delivery hosts that assert cache hits already have no worker or have the output cache off.

## Dependencies

None.

## Acceptance

- [x] The cause is named in this issue
- [x] The test passes repeated, alone and under load, and a product defect, if there is one, has its own test (no product defect)
Closes #3200

Since #3182 the take of the row lock runs in an explicit transaction. On the first use of a name, takes that start together race on the insert of its row. A loser's insert fails with a unique violation, and in `catch (DbUpdateException)` it called `ExistsAsync`, which reads `cms_lock` on another connection while the loser's own transaction is still open. On the in-memory SQLite database of `FluentCmsTestHost` (shared cache), that open transaction holds the write lock, so the read got `SQLITE_LOCKED` and retried for the 30-second command timeout. Meanwhile other takes overshot their waits and the holder's own write failed with `database table is locked`, which was the 500 of the plugin template's `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` in the `Templates` job. The take now rolls its transaction back before it reads the row, as `SaveChangesAsync` did before #3182. The backend spec's "Multi-instance as built" paragraph gets one sentence on it. The preview release notes do not describe the lock change of #3182, so they are unchanged.

What I checked in `DatabaseDistributedLock`, by reading it:

- No other path reads or writes on another connection or scope while a take's or a renewal's transaction is open. The `ExecuteUpdateAsync` path of the take and `RenewAsync`'s update run on the transaction's own context. When either throws, `await using` disposes the transaction (a rollback) before the caller's `catch`, and `IsBusy` only inspects the exception. `RenewAsync`'s fallback read runs after its transaction is disposed, on the same context. `ReleaseAsync` opens no explicit transaction, and the handle waits for its renewal loop to end before it releases. `TryAcquireAsync` only waits after a null take and touches no database.
- PostgreSQL: a failed statement puts the transaction block in an aborted state. The docs give `ROLLBACK TO` a savepoint, "short of rolling it back completely", as the way out of it ([tutorial: transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html)), so the rollback is allowed and needed, and a read in the same transaction would fail.
- SQL Server: with `SET XACT_ABORT OFF`, the default and not changed anywhere in the code, a failed statement can roll back only that statement and leave the transaction open ([SET XACT_ABORT](https://learn.microsoft.com/sql/t-sql/statements/set-xact-abort-transact-sql)), so `Rollback` ends a pending transaction. `SqlTransaction.Rollback` throws `InvalidOperationException` only when the server already rolled the transaction back ([SqlTransaction.Rollback](https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqltransaction.rollback)), which a unique violation does not do with `XACT_ABORT OFF`.

Measured by the diagnosing lane (the generated `Acme.ReviewNotes` test, 20 runs per build under load):

| Build | Failed (500) | Race | 30 s stall | Most runs took |
| --- | --- | --- | --- | --- |
| 7b6cd7e, before #3182 | 0 / 20 | 18 | 0 | 10–25 s |
| 7a085f3 (#3182) | 9 / 20 | 18 | 18 | 37–41 s |
| c0f6d5e (tip of `main`) | 7 / 20 | 19 | 19 | 37–54 s |
| tip plus the rollback | 0 / 20 | 20 | 0 | 8–14 s |

Measured here:

| Run | Result |
| --- | --- |
| New test `TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout`, without the rollback | 30 / 30 failed, all on the elapsed time (test took 36–66 s) |
| The same test with the rollback | 0 / 30 failed (each process 8–12 s, host start included) |
| `DistributedLockTests` with the rollback, SQLite | 15 / 15 passed |
| `FluentCMS.Data.Tests` `Migrations` namespace with the rollback, PostgreSQL 15 | 143 passed, 46 skipped, 0 failed |
| `Acme.ReviewNotes` `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` from packages of this branch, as the `Templates` job makes it | 0 / 20 failed, 6.7–15.5 s per run |

The new test runs ten takes with a 500 ms wait on a name never used before, on the in-memory database. Exactly one holds the lock: the lease is 30 seconds on a wall clock that does not move, and nobody releases before the count.

## Dependencies

None. #3195 follows this pull request and merges `main` after it.

## Acceptance

- [x] The cause is named in this issue, with the host's exception
- [ ] The test passes repeated on `main`, in CI's `Templates` job, and the fix has a test in the repository's own tests if the cause is in the product (the regression test is in `DistributedLockTests`, and the generated test passed 20 runs locally; CI's `Templates` job on `main` is not verified yet)
- [x] If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now
Closes #3198

A logout cleared only the sign-in cookie, so a refresh token issued in that session could still be redeemed until it expired. A test reproduced this: after a confirmed logout the token endpoint still answered 200.

With this change:

- Every new sign-in cookie, of either pool, carries a server session id in its properties (`.fcms.server_session`), which its renewals keep. The same user signing in again on the login page over a live cookie keeps it.
- The authorize endpoint issues the code of a client without a consent step under an ad-hoc authorization marked with that id (`fcms_server_session` in the authorization's properties, so no migration). The identity token carries the id as `sid`.
- `UserSessions.EndAsync` revokes every valid authorization of the user marked with the session, with its tokens. The refresh tokens are refused at once. The access tokens are refused from the next request on, since the bearer scheme checks the token and authorization entries on every request.

Where a session's authorizations now end:

- The confirmed logout, or a logout with a hint of the session's own `sid`, whichever clients of the pool signed in through that session.
- The member logout page.
- Another user's sign-in over the cookie, which ends the first user's session in that browser.

What ends the cookie alone and revokes nothing:

- The sign-outs that one cross-site GET can cause: `prompt=login`, `max_age`, `AbsoluteSessionLifetime` and a second factor the user's role now requires, at the authorize endpoint, whose request carries nothing secret, and the SSO provider's remote sign-out, which nothing authenticates. These are open in #3203, #3204 and #3205. A refresh token without `amr=mfa` is still refused by the token endpoint once the role requires two-factor.
- A hint without the cookie's `sid` goes to the logout page, which asks. This covers an identity token of the same user from another session, another device or an earlier sign-in, a third-party client's token (a consent belongs to no session), and one from a sign-in before this change.
- A logout without `id_token_hint` goes to the confirmation page as before, and nothing is revoked until the confirmed POST with its antiforgery token.
- A cancelled or unconfirmed logout changes nothing.
- A request without a live cookie revokes nothing, a valid hint included. Such a GET returns to the client without asking only when its hint is an identity token signed by this server with its algorithm (an expired one, or one a first logout revoked, still counts), issued to the client its `client_id` names, and the `post_logout_redirect_uri` it asks for is registered for that client. Any other hint goes to the logout page.
- The user's other sessions (another browser, another device) stay.
- A third-party client's consent stays, with its tokens.
- A cookie from before this change has no id, so its logout ends the cookie alone, as before.

A browser logout therefore ends every client without a consent step that signed in through that browser, whatever `offline_access` it holds. A client that must outlive a browser logout is registered with `RequireConsent`. The specification and the release notes say this.

The response is unchanged, so it does not show whether a session existed. Sign-outs write no activity row, and the revocation adds none (#3207). `FluentCMS.Testing` builds its cookies through `CmsSignInManager.MarkSignIn`, the same path as the product.

Dependencies: none. The check of a hint's signature uses `Microsoft.IdentityModel.JsonWebTokens`, which OpenIddict already brings.

- [x] After `/connect/logout` for a session, a refresh token issued under that session's authorization is refused, in a test
- [x] The specification says what end-session revokes and what it leaves
- [ ] The decision record of the admin's sign-in (0007) no longer names an unrevoked token after a completed logout as a known gap, or says what remains (the record is on `feature/a1`, not on `main`; it is updated there after this merges)
# Conflicts:
#	docs/planning/backend-technical-spec.md
@pournasserian pournasserian added this to the A1 Shell and sign-in milestone Nov 16, 2026
@pournasserian pournasserian added type:chore Repository, tooling or housekeeping area:admin The Blazor WebAssembly admin and the admin halves of plugins labels Nov 16, 2026
@pournasserian
pournasserian merged commit 37dd98f into feature/a1 Nov 16, 2026
22 checks passed
@pournasserian
pournasserian deleted the sync-main-into-a1-1116 branch November 16, 2026 14:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:admin The Blazor WebAssembly admin and the admin halves of plugins type:chore Repository, tooling or housekeeping

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant