From 7a085f3a78f24f280f03772e825669634d9a9ed7 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 00:40:16 -0500 Subject: [PATCH 1/8] Keep a SQLite lock holder's lease while a write transaction blocks it (#3182) Closes #3176 On SQLite, the default lock renewed its lease on its own connection, and SQLite has one writer at a time. While the holder's own write transaction was open, the renewal waited for it and gave up after about 35 seconds (the busy timeout, then the retries of Microsoft.Data.Sqlite). The lease ran out, `Lost` was canceled at about 45 seconds, and that canceled the rest of the holder's work. This was reproduced through FluentCMS: - `schema apply` with two added columns failed with `FCMS3302`, and neither column was added. - Schema sync of the same change rolled back. - A waiter threw `SqliteException` "database is locked" out of `TryAcquireAsync`. That happened on a take-over, on the first use of a lock name, and, in the rollback journal mode, already on the read of the row. `IDataProvider.IsBusy` recognises a command that gave up waiting for the database's writer. On SQLite that is every error whose primary code is busy or locked; on PostgreSQL and SQL Server it is always false. Such a command changed nothing, and the lock now handles it as follows: - **A waiter** whose read of the row, take-over or first insert gives up this way counts the lock as held. It ends with the lock or with its timeout, and never with `SqliteException`. It can overshoot its timeout by one such wait: a `TimeSpan.Zero` caller on a busy file now waits about 35 seconds and gets null, where before an exception came after the same wait. - **A take and a renewal** each run in a transaction of their own and read the clock once it has begun. On SQLite, beginning it waits for the writer, so the lease starts when the writer let them in, and a write that waited never writes a lease that is short or already over. On PostgreSQL and SQL Server, the clock is still read before a statement that can wait on the row, as before. Before, a renewal that waited 25 seconds wrote a lease that ran out 5 seconds later, and a waiter could take a live holder's lock. - **A holder whose renewal gives up** reads the row; in WAL mode, reads do not wait for the writer. - If the row still names the holder, it keeps the lock, renews again at once, and logs once that it waits (9165, Information). No other holder can take the lock while the transaction lasts, because taking it is a write too. - If the row names another owner, it cancels `Lost`. A take-over can commit between two such transactions. - There is no time bound, since a bound would bring the original bug back. On PostgreSQL and SQL Server, the take and the renewal now run in an explicit transaction around the same statements. That adds a round trip and does not change what they do. **The race that remains.** A waiter that found the lease over during a long transaction waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other owner and cancels `Lost`. But a holder with more than one unit of work under the lock may still have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. As measured in review: with a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 rounds, and the holder's `Lost` fired 64 to 119 ms after the commit. When the holder began its next write at once, the waiter took it in none of 16. Known, and left to #3186: - A `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again. Running it again records the history. - The waiter's event 9160 then says that a holder stopped without releasing the lock, while that holder is alive. The operator hint of the `fluentcms:schema-sync` timeout and `docs/content-migrations.md` now add the missing state: on SQLite, a `lease_until` standing still behind the clock can mean the holder is working inside a long transaction. The specification describes all of this and lists the lock's event ids, 9160 to 9169. **Tests:** - Core integration tests, on a write transaction held on a second connection: - a holder keeps the lock past a 300 ms lease; - a waiter ends with its timeout, on a take-over and on a first use; - a renewal and a take that waited for the writer write a lease that is still running; - a holder is told when another owner took the row while its renewals keep waiting. - Data tests on SQLite: - `schema apply` and schema sync each finish while their first `ALTER TABLE` is held until a renewal gave up; - a waiter on a rollback-journal database whose read of the row waits ends with its timeout. All of them fail on `main`'s lock except `AHolderIsToldWhenAnotherTookTheLockWhileAWriteTransactionKeepsItsRenewalsWaiting`. That test passes on `main`, where `Lost` fires through lease expiry, and fails on the first commit of this PR, which is what it guards. Dependencies: none. - [x] A test holds the lock on SQLite inside a write transaction for longer than the lease, and the work finishes - [x] A waiter on SQLite ends with the lock's timeout or with the lock, never with `SqliteException` - [x] PostgreSQL and SQL Server behave as before --- docs/content-migrations.md | 2 +- docs/planning/backend-technical-spec.md | 4 +- .../Caching/DatabaseDistributedLock.cs | 99 ++++++++-- src/FluentCMS.Core/Caching/LockLog.cs | 4 + src/FluentCMS.Core/Storage/IDataProvider.cs | 6 + src/FluentCMS.Core/Storage/MigrationRunner.cs | 4 +- .../PostgreSqlDataProvider.cs | 3 + .../SqlServerDataProvider.cs | 3 + .../SqliteDataProvider.cs | 4 + .../ContentTableReaches.cs | 3 +- .../Caching/DistributedLockTests.cs | 173 +++++++++++++++++- .../Migrations/LongTransaction.cs | 64 +++++++ .../Migrations/SchemaApplyTests.cs | 16 ++ .../Migrations/SchemaSyncTests.cs | 16 ++ .../Sqlite/RollbackJournalLockTests.cs | 56 ++++++ .../Sqlite/SqliteFileDatabase.cs | 2 + 16 files changed, 431 insertions(+), 28 deletions(-) create mode 100644 tests/FluentCMS.Data.Tests/Migrations/LongTransaction.cs create mode 100644 tests/FluentCMS.Data.Tests/Sqlite/RollbackJournalLockTests.cs diff --git a/docs/content-migrations.md b/docs/content-migrations.md index 09791bd87..a5d071a68 100644 --- a/docs/content-migrations.md +++ b/docs/content-migrations.md @@ -145,6 +145,6 @@ With `Auto`, startup skips the content stream and applies additive changes by co | Check constraint of a new JSON column | Reported, because SQLite adds one only by rebuilding the table | | Dropped table, column or index; changed column type, nullability or index | Reported | -Reported changes are logged as warnings and never applied. To get them, recreate the development database. Instances that start together on one database apply each change once: an instance with something to apply takes the `fluentcms:schema-sync` lock and compares again before it applies. It waits up to 15 minutes for the lock and then fails to start with an error that names it. Check the `owner` and `lease_until` of the lock's row in `cms_lock`: a `lease_until` that keeps moving means its holder is still working, and one that stands still ahead of the instance's clock means its holder stopped and the clocks of the instances are not in step. Once no instance is running, set the row's `owner` to null. A table under a plugin's prefix (`cms_{plugin}_`) belongs to that plugin's stream, so schema sync never reports it as a dropped table. That holds for a plugin the host no longer adds too, while the database has the plugin's history table `cms_{plugin}__migrations`. For a host without content migrations, schema sync records the schema history only when it applied every change and the history counts none as lost, such as a removed type or a field renamed whose values are kept in JSON; such a change keeps the newest row for `fluentcms schema apply`. A host with content migrations records after every sync that left nothing undone, as before. +Reported changes are logged as warnings and never applied. To get them, recreate the development database. Instances that start together on one database apply each change once: an instance with something to apply takes the `fluentcms:schema-sync` lock and compares again before it applies. It waits up to 15 minutes for the lock and then fails to start with an error that names it. Check the `owner` and `lease_until` of the lock's row in `cms_lock`: a `lease_until` that keeps moving means its holder is still working, and one that stands still ahead of the instance's clock means its holder stopped and the clocks of the instances are not in step. On SQLite, one that stands still behind the clock can also mean that its holder is working inside a long write transaction, which keeps it from renewing until the transaction ends. Once no instance is running, set the row's `owner` to null. A table under a plugin's prefix (`cms_{plugin}_`) belongs to that plugin's stream, so schema sync never reports it as a dropped table. That holds for a plugin the host no longer adds too, while the database has the plugin's history table `cms_{plugin}__migrations`. For a host without content migrations, schema sync records the schema history only when it applied every change and the history counts none as lost, such as a removed type or a field renamed whose values are kept in JSON; such a change keeps the newest row for `fluentcms schema apply`. A host with content migrations records after every sync that left nothing undone, as before. A database that schema sync has changed has no content history, so the first migration of the content stream will not apply to it. Switch to `Off` on a new database. diff --git a/docs/planning/backend-technical-spec.md b/docs/planning/backend-technical-spec.md index a66174f2e..f9ce25a0f 100644 --- a/docs/planning/backend-technical-spec.md +++ b/docs/planning/backend-technical-spec.md @@ -1163,11 +1163,11 @@ Rows 10.3, 10.5 to 10.8, 10.11 to 10.15. **Observability.** One `ActivitySource` (`FluentCMS`) with spans for content operations, hooks, jobs, webhooks and MCP calls; one `Meter` with request counts by surface, query duration, job latency, cache hit ratio, webhook outcomes; structured logs with event ids and `FCMS` codes. Exporters are the host's choice. Nothing is sent to the project. -**Health and observability, as built.** `FluentCmsHealthCheckTags` (`FluentCMS.Diagnostics`, in the SDK, so plugins tag their checks) has two tags, `live` and `ready`, and `/health/live` and `/health/ready` run the checks tagged with each through `Microsoft.Extensions.Diagnostics.HealthChecks`, with no authentication and no rate limit policy of the endpoints. They stay reachable without credentials, because the image's `HEALTHCHECK` and an orchestrator's probes call them, so they are kept cheap and bounded instead: the FluentCMS authentication step skips the `health` surface, so a key sent there is never looked up, and the check before authentication counts them per client address in a partition of their own, apart from the coarse limit, with `FluentCms:RateLimits:Health:PermitLimit` (1200 a minute, twenty a second from one address, far above what an orchestrator sends one instance). A path under `/health` that no endpoint answers counts there too. A refusal is the same 429 problem, and the probes carry no `RateLimit-*` headers. Where the network allows, probes should still be restricted to the orchestrator and the load balancer. This answers decision 16 of the review of 7 Nov 2026 ([#2847](https://github.com/fluentcms/FluentCMS/issues/2847), [#2855](https://github.com/fluentcms/FluentCMS/issues/2855)). They answer 200 for Healthy and Degraded and 503 for Unhealthy, with the status as the only body, never a description or an exception. Core registers `process` (live), `database` (ready, one row of a system table, so it also fails when the system migrations did not run) and `job-runner` (ready); `services.AddHealthChecks().AddFluentCmsReadyCheck("name")` adds a ready check with a five second timeout (it is in Core, because the SDK has no reference to the health check abstractions; a plugin that references only the SDK calls `AddCheck` with the tag). The job runner beats in memory at every turn of its maintenance loop (`JobRunnerOptions.MaintenanceInterval`, 15 seconds), and ready is Unhealthy when the last beat is older than `FluentCms:Health:JobRunnerHeartbeatTimeout` (default 2 minutes; `Timeout.InfiniteTimeSpan` leaves it out for an instance that runs no jobs) or when the runner has not started, and Degraded when the oldest pending job of a queue has waited longer than `FluentCms:Health:PendingJobAgeLimit` (4.12). `GET /api/management/v1/system/health` needs `system.read`, an instance-level permission: it is in the catalogue and checked with `InstanceAccess`, so a role of one site or a grant limited to sites does not reach it, nor does a key bound to sites or a Delivery or Preview key. It runs every check and answers 200 with `{ status, totalDurationMs, checks: [{ name, status, durationMs, tags, description, error }] }`, whatever the status; descriptions and exception text pass through `SecretScrubber`. The one `ActivitySource` and the one `Meter` are both named `FluentCMS` (`FluentCmsTelemetry.Name`), and nothing is exported: a host subscribes with `AddSource` and `AddMeter` from its own OpenTelemetry setup, and Core takes no dependency on it. Spans are named `content {operation}` (create, update, patch, delete, restore, purge, publish, unpublish, discard-draft, schedule, cancel-schedule, versions, version, restore-version, diff, get, query, query-trash, count, bulk), `hook {kind}` (one per run of the hooks of a kind, with `fluentcms.hooks` their number) and `job {type}` (with `fluentcms.queue`). The MCP plugin adds `mcp {tool}` for a tool call (5.4). They carry `fluentcms.operation`, `fluentcms.type` (the name the schema resolves, and not on jobs) and `fluentcms.site`, and never a value of content; a job's span has the Error status when the attempt throws. The instruments are `fluentcms.requests` (counter, by `fluentcms.surface` of delivery, management, connect (with `/.well-known`), account, health or docs (the OpenAPI document and Scalar), and `fluentcms.status_class`, counted with the status the client got, including a refusal an endpoint throws), `fluentcms.query.duration` (histogram, seconds, by the `fluentcms.type` the schema resolved, and not recorded for a query refused as invalid), `fluentcms.job.latency` (histogram, seconds, the length of an attempt by `fluentcms.operation` the job type and `fluentcms.outcome`), `fluentcms.job.oldest_pending_age` (observable gauge, seconds, by `fluentcms.queue`, 4.12), `fluentcms.event.handler.duration` (histogram, seconds, the length of a run of an in-process event handler by `fluentcms.operation` the event type, `fluentcms.handler` and `fluentcms.outcome`, 4.11), and `fluentcms.cache.requests` (counter by `fluentcms.cache` and `fluentcms.result`) with `fluentcms.cache.hit_ratio` (observable gauge by `fluentcms.cache`); the Delivery output cache reports its lookups as `delivery`. Every `[LoggerMessage]` in Core has an event id, in a range per area that follows the `FCMS` families: 90xx identity (the authorization server 9001 to 9003 and 9005, the hosted pages 9004, admin SSO 9040 to 9044, the member pool 9070 to 9076, member providers 9090 to 9094), 91xx host (health 9101 to 9104, the problem handler 9110, email 9120, the startup report 9130 to 9132, the command runner 9170 to 9172 of the range 9170 to 9179), 92xx plugins (per-site enablement 9230 to 9234), 93xx storage (migrations 9301, schema history 9311), 94xx content (9401, scheduled jobs 9411, trash purge 9421), 95xx jobs, 96xx events, 97xx activity, 98xx settings and 99xx seeding. The event name is the method name, and a message carries the `FCMS` code where one applies. +**Health and observability, as built.** `FluentCmsHealthCheckTags` (`FluentCMS.Diagnostics`, in the SDK, so plugins tag their checks) has two tags, `live` and `ready`, and `/health/live` and `/health/ready` run the checks tagged with each through `Microsoft.Extensions.Diagnostics.HealthChecks`, with no authentication and no rate limit policy of the endpoints. They stay reachable without credentials, because the image's `HEALTHCHECK` and an orchestrator's probes call them, so they are kept cheap and bounded instead: the FluentCMS authentication step skips the `health` surface, so a key sent there is never looked up, and the check before authentication counts them per client address in a partition of their own, apart from the coarse limit, with `FluentCms:RateLimits:Health:PermitLimit` (1200 a minute, twenty a second from one address, far above what an orchestrator sends one instance). A path under `/health` that no endpoint answers counts there too. A refusal is the same 429 problem, and the probes carry no `RateLimit-*` headers. Where the network allows, probes should still be restricted to the orchestrator and the load balancer. This answers decision 16 of the review of 7 Nov 2026 ([#2847](https://github.com/fluentcms/FluentCMS/issues/2847), [#2855](https://github.com/fluentcms/FluentCMS/issues/2855)). They answer 200 for Healthy and Degraded and 503 for Unhealthy, with the status as the only body, never a description or an exception. Core registers `process` (live), `database` (ready, one row of a system table, so it also fails when the system migrations did not run) and `job-runner` (ready); `services.AddHealthChecks().AddFluentCmsReadyCheck("name")` adds a ready check with a five second timeout (it is in Core, because the SDK has no reference to the health check abstractions; a plugin that references only the SDK calls `AddCheck` with the tag). The job runner beats in memory at every turn of its maintenance loop (`JobRunnerOptions.MaintenanceInterval`, 15 seconds), and ready is Unhealthy when the last beat is older than `FluentCms:Health:JobRunnerHeartbeatTimeout` (default 2 minutes; `Timeout.InfiniteTimeSpan` leaves it out for an instance that runs no jobs) or when the runner has not started, and Degraded when the oldest pending job of a queue has waited longer than `FluentCms:Health:PendingJobAgeLimit` (4.12). `GET /api/management/v1/system/health` needs `system.read`, an instance-level permission: it is in the catalogue and checked with `InstanceAccess`, so a role of one site or a grant limited to sites does not reach it, nor does a key bound to sites or a Delivery or Preview key. It runs every check and answers 200 with `{ status, totalDurationMs, checks: [{ name, status, durationMs, tags, description, error }] }`, whatever the status; descriptions and exception text pass through `SecretScrubber`. The one `ActivitySource` and the one `Meter` are both named `FluentCMS` (`FluentCmsTelemetry.Name`), and nothing is exported: a host subscribes with `AddSource` and `AddMeter` from its own OpenTelemetry setup, and Core takes no dependency on it. Spans are named `content {operation}` (create, update, patch, delete, restore, purge, publish, unpublish, discard-draft, schedule, cancel-schedule, versions, version, restore-version, diff, get, query, query-trash, count, bulk), `hook {kind}` (one per run of the hooks of a kind, with `fluentcms.hooks` their number) and `job {type}` (with `fluentcms.queue`). The MCP plugin adds `mcp {tool}` for a tool call (5.4). They carry `fluentcms.operation`, `fluentcms.type` (the name the schema resolves, and not on jobs) and `fluentcms.site`, and never a value of content; a job's span has the Error status when the attempt throws. The instruments are `fluentcms.requests` (counter, by `fluentcms.surface` of delivery, management, connect (with `/.well-known`), account, health or docs (the OpenAPI document and Scalar), and `fluentcms.status_class`, counted with the status the client got, including a refusal an endpoint throws), `fluentcms.query.duration` (histogram, seconds, by the `fluentcms.type` the schema resolved, and not recorded for a query refused as invalid), `fluentcms.job.latency` (histogram, seconds, the length of an attempt by `fluentcms.operation` the job type and `fluentcms.outcome`), `fluentcms.job.oldest_pending_age` (observable gauge, seconds, by `fluentcms.queue`, 4.12), `fluentcms.event.handler.duration` (histogram, seconds, the length of a run of an in-process event handler by `fluentcms.operation` the event type, `fluentcms.handler` and `fluentcms.outcome`, 4.11), and `fluentcms.cache.requests` (counter by `fluentcms.cache` and `fluentcms.result`) with `fluentcms.cache.hit_ratio` (observable gauge by `fluentcms.cache`); the Delivery output cache reports its lookups as `delivery`. Every `[LoggerMessage]` in Core has an event id, in a range per area that follows the `FCMS` families: 90xx identity (the authorization server 9001 to 9003 and 9005, the hosted pages 9004, admin SSO 9040 to 9044, the member pool 9070 to 9076, member providers 9090 to 9094), 91xx host (health 9101 to 9104, the problem handler 9110, email 9120, the startup report 9130 to 9132, the distributed lock 9160 to 9165 of the range 9160 to 9169, the command runner 9170 to 9172 of the range 9170 to 9179), 92xx plugins (per-site enablement 9230 to 9234), 93xx storage (migrations 9301, schema history 9311), 94xx content (9401, scheduled jobs 9411, trash purge 9421), 95xx jobs, 96xx events, 97xx activity, 98xx settings and 99xx seeding. The event name is the method name, and a message carries the `FCMS` code where one applies. **Multi-instance.** Stateless API; Data Protection and signing keys persisted to the database by default; jobs lease through the database; `IDistributedLock` defaults to a `cms_lock` table and gets a Redis provider in `FluentCMS.Caching.Redis`; output cache in memory by default or Redis; media on shared storage. -**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. +**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. SQLite has one writer, so while another connection keeps a write transaction open, such as the schema apply or schema sync that holds the lock, a renewal waits for it and gives up after the busy timeout and the retries of Microsoft.Data.Sqlite, about 35 seconds, and so does a take; in the rollback journal mode a read of the row can wait too. Such a command, which `IDataProvider.IsBusy` recognises (SQLite's busy and locked errors by their primary code, never on PostgreSQL or SQL Server), changed nothing: a waiter counts the lock as held and ends with the lock or its timeout, which it can overshoot by one such wait, and the holder reads the row, which in WAL mode does not wait, and keeps the lock while the row still names it, renewing again at once and logging once that it waits (9165). A take and a renewal each run in a transaction of their own and read the clock once it has begun, so on SQLite, where beginning it waits for the writer, a lease starts when the database let the write in, not when it was asked for; on PostgreSQL and SQL Server the clock is still read before a statement that can wait on the row, as before. Before, a holder whose transaction lasted longer than about 45 seconds lost the lock, which canceled the rest of its work and rolled the transaction back, and a waiter failed with `SqliteException` ([#3176](https://github.com/fluentcms/FluentCMS/issues/3176)). One race remains: a waiter that found the lease over while such a transaction lasted waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other holder and cancels `Lost`, but a holder with more than one unit of work under the lock may have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. With a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 measured rounds, and the holder's `Lost` fired 64 to 119 milliseconds after the commit; when the holder began its next write at once, the waiter took it in none of 16. Known and left to [#3186](https://github.com/fluentcms/FluentCMS/issues/3186): a `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again, which records the history; and the waiter's event 9160 then says that a holder stopped without releasing the lock while that holder is alive. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. **Reverse proxy.** Forwarded headers, `PathBase` option for sub-path hosting, HTTPS redirection and HSTS options; documented for Nginx, Caddy, YARP, Azure Front Door and Cloudflare. diff --git a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs index 5e5248933..5726db84d 100644 --- a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs +++ b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs @@ -50,24 +50,44 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP } } - // The end of the new lease, or null while another holder has the lock. + // The end of the new lease, or null while another holder has the lock. On a database with one writer, reading the row or + // beginning the write can give up waiting for another connection's transaction: nothing was taken, and the lock counts + // as held, since that transaction may be the holder's own work, which keeps it from renewing. private async Task TryTakeAsync(string name, string holder, CancellationToken cancellationToken) { await using var scope = scopes.CreateAsyncScope(); var context = scope.ServiceProvider.GetRequiredService(); + try + { + return await TakeAsync(context, name, holder, cancellationToken); + } + catch (Exception exception) when (scope.ServiceProvider.GetRequiredService().IsBusy(exception)) + { + return null; + } + } + + private async Task TakeAsync(SystemDbContext context, string name, string holder, CancellationToken cancellationToken) + { + var current = await context.Set().AsNoTracking().SingleOrDefaultAsync(row => row.Name == name, cancellationToken); + if (current is { Owner: not null } && current.LeaseUntil > time.GetUtcNow()) + { + return null; + } + // The writes are not canceled, so a take that commits always comes back as a handle that renews and releases it. On + // SQLite the transaction begins once the database's one writer is free, so the lease starts from the clock then. + await using var transaction = await context.Database.BeginTransactionAsync(CancellationToken.None); var now = time.GetUtcNow(); var until = StorageConventions.ToMicroseconds(now + Lease); - var current = await context.Set().AsNoTracking().SingleOrDefaultAsync(row => row.Name == name, cancellationToken); - - // The writes are not canceled, so a take that commits always comes back as a handle that renews and releases it. if (current is null) { context.Add(new LockRow { Name = name, Owner = holder, AcquiredAt = now, LeaseUntil = until }); try { await context.SaveChangesAsync(CancellationToken.None); + await transaction.CommitAsync(CancellationToken.None); return until; } catch (DbUpdateException) @@ -82,11 +102,6 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP } } - if (current.Owner is not null && current.LeaseUntil > now) - { - return null; - } - var taken = await context.Set() .Where(row => row.Name == name && (row.Owner == null || row.LeaseUntil == null || row.LeaseUntil <= now)) .ExecuteUpdateAsync( @@ -95,6 +110,7 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP .SetProperty(column => column.AcquiredAt, now) .SetProperty(column => column.LeaseUntil, until), CancellationToken.None); + await transaction.CommitAsync(CancellationToken.None); if (taken == 0) { @@ -115,18 +131,40 @@ private async Task ExistsAsync(string name) return await scope.ServiceProvider.GetRequiredService().Set().AnyAsync(row => row.Name == name); } - // The end of the renewed lease, or null when the holder no longer has the lock. - private async Task RenewAsync(string name, string holder, CancellationToken cancellationToken) + // The end of the renewed lease, or null when the holder no longer has the lock; Busy when the database has one writer, + // the renewal gave up waiting for another connection's transaction, and the row may still name the holder. + private async Task<(DateTimeOffset? Until, bool Busy)> RenewAsync(string name, string holder, CancellationToken cancellationToken) { await using var scope = scopes.CreateAsyncScope(); var context = scope.ServiceProvider.GetRequiredService(); + var provider = scope.ServiceProvider.GetRequiredService(); + try + { + // As for a take, the lease starts from the clock once the transaction has begun. + await using var transaction = await context.Database.BeginTransactionAsync(cancellationToken); + var until = StorageConventions.ToMicroseconds(time.GetUtcNow() + Lease); + var renewed = await context.Set() + .Where(row => row.Name == name && row.Owner == holder) + .ExecuteUpdateAsync(row => row.SetProperty(column => column.LeaseUntil, until), cancellationToken); + await transaction.CommitAsync(CancellationToken.None); + + return (renewed == 1 ? until : null, false); + } + catch (Exception exception) when (provider.IsBusy(exception)) + { + } - var until = StorageConventions.ToMicroseconds(time.GetUtcNow() + Lease); - var renewed = await context.Set() - .Where(row => row.Name == name && row.Owner == holder) - .ExecuteUpdateAsync(row => row.SetProperty(column => column.LeaseUntil, until), cancellationToken); - - return renewed == 1 ? until : null; + // A take-over can commit between two transactions that keep the renewal waiting. Reads do not wait for the writer + // in WAL mode, so the row tells whether the lock is still the holder's; a read that waits as well tells nothing. + try + { + var owner = await context.Set().AsNoTracking().Where(row => row.Name == name).Select(row => row.Owner).SingleOrDefaultAsync(cancellationToken); + return (null, owner == holder); + } + catch (Exception exception) when (provider.IsBusy(exception)) + { + return (null, true); + } } private async Task ReleaseAsync(string name, string holder) @@ -203,18 +241,39 @@ public async ValueTask DisposeAsync() private async Task RenewAsync() { var interval = _owner.Lease / 3; + var delay = interval; + var waiting = false; while (true) { try { - await _owner.DelayAsync(interval, _stopping.Token); - if (await _owner.RenewAsync(Name, _holder, _stopping.Token) is not { } until) + await _owner.DelayAsync(delay, _stopping.Token); + var (until, busy) = await _owner.RenewAsync(Name, _holder, _stopping.Token); + + // While another connection holds the one writer of the database, such as the holder's own work in a + // long transaction, no other holder can take the lock either, since taking it is a write. So the lease + // does not count as run out, and the renewal waits again at once, to be waiting when that write ends. + if (busy) + { + if (!waiting) + { + LockLog.RenewWaiting(_owner.Logger, Name); + waiting = true; + } + + delay = TimeSpan.Zero; + continue; + } + + waiting = false; + delay = interval; + if (until is null) { await LoseAsync("another holder took it"); return; } - _leaseUntil = until; + _leaseUntil = until.Value; } catch (OperationCanceledException) when (_stopping.IsCancellationRequested) { diff --git a/src/FluentCMS.Core/Caching/LockLog.cs b/src/FluentCMS.Core/Caching/LockLog.cs index e3eaa4812..bd6f00975 100644 --- a/src/FluentCMS.Core/Caching/LockLog.cs +++ b/src/FluentCMS.Core/Caching/LockLog.cs @@ -10,6 +10,7 @@ internal static partial class LockLog public const int RenewFailedId = 9162; public const int ReleaseFailedId = 9163; public const int WentOnWithoutLockId = 9164; + public const int RenewWaitingId = 9165; [LoggerMessage(EventId = LeaseTakenOverId, EventName = "LockLeaseTakenOver", Level = LogLevel.Warning, Message = "Took the lock {Name} after the lease of {Holder} ran out; that holder stopped without releasing it.")] public static partial void LeaseTakenOver(ILogger logger, string name, string holder); @@ -25,4 +26,7 @@ internal static partial class LockLog [LoggerMessage(EventId = WentOnWithoutLockId, EventName = "LockWentOnWithout", Level = LogLevel.Warning, Message = "The lock {Name} was not free within {Timeout}, so {Work} went on without it.")] public static partial void WentOnWithoutLock(ILogger logger, string name, TimeSpan timeout, string work); + + [LoggerMessage(EventId = RenewWaitingId, EventName = "LockRenewWaiting", Level = LogLevel.Information, Message = "The lease of the lock {Name} is renewed once another connection's write transaction ends; the database has one writer, so no other holder can take the lock meanwhile.")] + public static partial void RenewWaiting(ILogger logger, string name); } diff --git a/src/FluentCMS.Core/Storage/IDataProvider.cs b/src/FluentCMS.Core/Storage/IDataProvider.cs index fd70e8c2d..bb75603ff 100644 --- a/src/FluentCMS.Core/Storage/IDataProvider.cs +++ b/src/FluentCMS.Core/Storage/IDataProvider.cs @@ -30,6 +30,12 @@ public interface IDataProvider // Whether the database ended the transaction as the victim of a deadlock, which it asks to be run again. bool IsDeadlock(Exception exception); + // Whether a command gave up waiting for a database that has one writer at a time, while another connection's transaction + // held it: a write, or a read in a journal mode where a commit keeps readers out. The command changed nothing. On SQLite + // that is every error whose primary code is SQLITE_BUSY or SQLITE_LOCKED, extended codes such as BUSY_SNAPSHOT and + // LOCKED_SHAREDCACHE included. False where writers do not wait for the whole database. + bool IsBusy(Exception exception); + // A condition of a LINQ query: the text holds the fragment, as it is (LikePattern), without regard to case as the database // compares text. A match on a part of the text uses no index. Expression> ContainsIgnoringCase(string fragment); diff --git a/src/FluentCMS.Core/Storage/MigrationRunner.cs b/src/FluentCMS.Core/Storage/MigrationRunner.cs index 1415f98e3..780f94008 100644 --- a/src/FluentCMS.Core/Storage/MigrationRunner.cs +++ b/src/FluentCMS.Core/Storage/MigrationRunner.cs @@ -203,7 +203,9 @@ internal async Task LockContentSchemaAsync(CancellationT $"The lock {SchemaSyncLock} was not free within {SchemaSyncWait.TotalMinutes:0} minutes, so the content tables were not changed; " + $"check the owner and lease_until of its row in {StorageNames.Lock}: a lease_until that keeps moving means another instance or a " + "'fluentcms schema apply' still holds it and is working, and one that stands still ahead of this instance's clock means its holder " + - "stopped and the clocks of the instances are not in step; once no instance is running, set the row's owner to null."); + "stopped and the clocks of the instances are not in step; on SQLite one that stands still behind the clock can also mean that its " + + "holder is working inside a long write transaction, which keeps it from renewing until the transaction ends; once no instance " + + "is running, set the row's owner to null."); // The steps that bring the content tables up to the host's schema: schema sync's comparison, with renames and the // changes sync leaves undone. The applied snapshot tells which tables under a plugin's prefix are content tables. diff --git a/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs b/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs index 99db8f22d..1e3f97b00 100644 --- a/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs +++ b/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs @@ -40,6 +40,9 @@ public bool IsDeadlock(Exception exception) return false; } + // Writers wait for the rows they write, not for the whole database. + public bool IsBusy(Exception exception) => false; + public Expression> ContainsIgnoringCase(string fragment) { var pattern = LikePattern.Containing(fragment); diff --git a/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs b/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs index 79d35dfb1..0fb04f60e 100644 --- a/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs +++ b/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs @@ -41,6 +41,9 @@ public bool IsDeadlock(Exception exception) return false; } + // Writers wait for the rows they write, not for the whole database. + public bool IsBusy(Exception exception) => false; + // Under a case-insensitive collation of its own, so that a database whose default compares case does not change it. public Expression> ContainsIgnoringCase(string fragment) { diff --git a/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs b/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs index eed524658..318fe5953 100644 --- a/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs +++ b/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs @@ -42,6 +42,10 @@ public bool IsUniqueViolation(Exception exception) => // A writer takes the lock at BEGIN IMMEDIATE and holds it to the end, so writers queue and never deadlock. public bool IsDeadlock(Exception exception) => false; + // What Microsoft.Data.Sqlite throws once the busy timeout and its own retries are over, matched by the primary code. + public bool IsBusy(Exception exception) => + (exception as SqliteException ?? exception.InnerException as SqliteException) is { SqliteErrorCode: raw.SQLITE_BUSY or raw.SQLITE_LOCKED }; + // SQLite's LIKE folds the case of ASCII letters only. public Expression> ContainsIgnoringCase(string fragment) { diff --git a/tests/FluentCMS.Architecture.Tests/ContentTableReaches.cs b/tests/FluentCMS.Architecture.Tests/ContentTableReaches.cs index dd08880ea..f27b5977b 100644 --- a/tests/FluentCMS.Architecture.Tests/ContentTableReaches.cs +++ b/tests/FluentCMS.Architecture.Tests/ContentTableReaches.cs @@ -10,7 +10,7 @@ internal static class ContentTableReaches { private static readonly string[] Allowed = ["FluentCMS.Core.Content", "FluentCMS.Core.Storage", "FluentCMS.Data", "FluentCMS.Cli"]; - // These open transactions on SystemDbContext for the job and identity tables, or on a plugin's context for its own + // These open transactions on SystemDbContext for the job, identity and lock tables, or on a plugin's context for its own // tables, or ask whether one is open on a core or a plugin context, and reach no content. The search plugin's store // matches its own index with the full-text SQL of the provider, and reads content only through the content service. private static readonly string[] AllowedTypes = @@ -18,6 +18,7 @@ internal static class ContentTableReaches "FluentCMS.Core.Jobs.JobStore", "FluentCMS.Core.Jobs.RecurringJobs", "FluentCMS.Core.Jobs.JobRetentionJob", + "FluentCMS.Core.Caching.DatabaseDistributedLock", "FluentCMS.Core.Identity.IdentityChanges", "FluentCMS.Core.Authorization.RoleVersionInterceptor", "FluentCMS.Core.Plugins.PluginTransactions", diff --git a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs index e013944cd..7e1e3cc46 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs @@ -1,3 +1,4 @@ +using System.Diagnostics; using FluentCMS.Core.Caching; using FluentCMS.Core.IntegrationTests.Testing; using FluentCMS.Core.Storage; @@ -22,15 +23,16 @@ public sealed class DistributedLockTests : IAsyncDisposable private static readonly TimeSpan Patience = TimeSpan.FromSeconds(30); private readonly DirectoryInfo _directory = Directory.CreateTempSubdirectory("fluentcms-lock-"); + private readonly string _file; private readonly CommandSignals _commands = CommandSignals.OfLock(Name); private readonly FluentCmsTestHost _first; private readonly FluentCmsTestHost _second; public DistributedLockTests() { - var file = Path.Combine(_directory.FullName, "cms.db"); - _first = Watched(new FluentCmsTestHost().UseSqlite(file), _commands); - _second = Watched(new FluentCmsTestHost().UseSqlite(file), _commands); + _file = Path.Combine(_directory.FullName, "cms.db"); + _first = Watched(new FluentCmsTestHost().UseSqlite(_file), _commands); + _second = Watched(new FluentCmsTestHost().UseSqlite(_file), _commands); } public async ValueTask DisposeAsync() @@ -199,6 +201,127 @@ await scope.ServiceProvider.GetRequiredService().Set() (await RowAsync(host)).Owner.ShouldBe(Crashed); } + // SQLite has one writer, so a renewal waits for another connection's write transaction, such as the holder's own work, + // and gives up before it ends: after about 5 seconds here, where a host waits about 35. It then reads the row. + [Fact] + public async Task AHolderKeepsTheLockWhileAWriteTransactionOutlastsItsLease() + { + var commands = CommandSignals.OfLock(Name); + await using var host = Impatient(Watched(new FluentCmsTestHost().UseSqlite(_file), commands)); + await using var held = await RealClockShortLease(host).TryAcquireAsync(Name, TimeSpan.Zero, Token); + held.ShouldNotBeNull(); + + await using (var writer = await WriteTransactionAsync()) + { + commands.Reads.Forget(); + await commands.Reads.NextAsync(1, Patience); + held.Lost.IsCancellationRequested.ShouldBeFalse(); + } + + // The renewal after the transaction finds the lock still the holder's. + commands.Writes.Forget(); + await commands.Writes.NextAsync(1, Patience); + held.Lost.IsCancellationRequested.ShouldBeFalse(); + } + + [Fact] + public async Task AWaiterOnADatabaseThatAWriteTransactionHoldsEndsWithItsTimeout() + { + await using var host = Impatient(new FluentCmsTestHost().UseSqlite(_file)); + var waiter = RealClockShortLease(host); + var held = await RealClockShortLease(_first).TryAcquireAsync(Name, TimeSpan.Zero, Token); + held.ShouldNotBeNull(); + + // The holder cannot renew while the transaction lasts, so the waiter finds the lease over and tries to take it. + await using (var writer = await WriteTransactionAsync()) + { + (await waiter.TryAcquireAsync(Name, TimeSpan.FromMilliseconds(500), Token)).ShouldBeNull(); + } + + await held.DisposeAsync(); + await using var next = await waiter.TryAcquireAsync(Name, Patience, Token); + next.ShouldNotBeNull(); + } + + // A renewal that waited for the writer longer than the lease of 3 seconds would have written one that was already over. + [Fact] + public async Task ARenewalThatWaitedForTheWriterStartsItsLeaseWhenItIsWritten() + { + await using var host = new FluentCmsTestHost().UseSqlite(_file); + await using var held = await RealClockLease(host, TimeSpan.FromSeconds(3)).TryAcquireAsync(Name, TimeSpan.Zero, Token); + held.ShouldNotBeNull(); + var taken = (await RowAsync(host)).LeaseUntil; + + DateTimeOffset released; + await using (var writer = await WriteTransactionAsync()) + { + await Task.Delay(TimeSpan.FromSeconds(5), Token); + released = DateTimeOffset.UtcNow; + } + + // The lease the waited renewal committed, which the next renewal a second later would replace. + var renewed = await NextLeaseAsync(host, taken); + renewed.ShouldBeGreaterThan(released); + held.Lost.IsCancellationRequested.ShouldBeFalse(); + } + + [Fact] + public async Task ATakeThatWaitedForTheWriterStartsItsLeaseWhenItIsWritten() + { + var locks = RealClockLease(_first, TimeSpan.FromSeconds(3)); + await (await locks.TryAcquireAsync(Name, TimeSpan.Zero, Token)).ShouldNotBeNull().DisposeAsync(); + + Task waiting; + await using (var writer = await WriteTransactionAsync()) + { + // Microsoft.Data.Sqlite runs its asynchronous calls synchronously, so the take that waits needs a thread. + waiting = Task.Run(() => locks.TryAcquireAsync(Name, Patience, Token), Token); + await Task.Delay(TimeSpan.FromSeconds(4), Token); + } + + await using var held = await waiting; + held.ShouldNotBeNull(); + (await RowAsync(_first)).LeaseUntil.ShouldNotBeNull().ShouldBeGreaterThan(DateTimeOffset.UtcNow); + } + + [Fact] + public async Task AFirstUseOnADatabaseThatAWriteTransactionHoldsEndsWithItsTimeout() + { + await using var host = Impatient(new FluentCmsTestHost().UseSqlite(_file)); + var locks = RealClockShortLease(host); + + await using (var writer = await WriteTransactionAsync()) + { + (await locks.TryAcquireAsync(Name, TimeSpan.FromMilliseconds(500), Token)).ShouldBeNull(); + } + + await using var held = await locks.TryAcquireAsync(Name, Patience, Token); + held.ShouldNotBeNull(); + } + + // What another instance does in the moment between two write transactions that keep the holder's renewals waiting. + [Fact] + public async Task AHolderIsToldWhenAnotherTookTheLockWhileAWriteTransactionKeepsItsRenewalsWaiting() + { + await using var host = Impatient(new FluentCmsTestHost().UseSqlite(_file)); + await using var held = await RealClockShortLease(host).TryAcquireAsync(Name, TimeSpan.Zero, Token); + held.ShouldNotBeNull(); + + await using var writer = new SqliteConnection($"Data Source={_file};Pooling=False"); + await writer.OpenAsync(Token); + await using (var command = writer.CreateCommand()) + { + command.CommandText = $"UPDATE cms_lock SET owner = '{Crashed}' WHERE name = '{Name}'; BEGIN IMMEDIATE"; + await command.ExecuteNonQueryAsync(Token); + } + + var lost = new TaskCompletionSource(TaskCreationOptions.RunContinuationsAsynchronously); + using (held.Lost.Register(() => lost.TrySetResult())) + { + await lost.Task.WaitAsync(Patience, Token); + } + } + [Fact] public async Task ABadNameOrTimeoutIsRefused() { @@ -226,6 +349,50 @@ public async Task ABadNameOrTimeoutIsRefused() Lease = TimeSpan.FromMilliseconds(300), }; + // A lease of 300 milliseconds on the real clock, renewed every 100, for a test whose time passes in SQLite's waits. + private static DatabaseDistributedLock RealClockShortLease(FluentCmsTestHost host) => RealClockLease(host, TimeSpan.FromMilliseconds(300)); + + private static DatabaseDistributedLock RealClockLease(FluentCmsTestHost host, TimeSpan lease) => new( + host.Services.GetRequiredService(), + TimeProvider.System, + NullLogger.Instance) + { + Lease = lease, + }; + + // A write transaction on a connection of its own, as the schema apply of another process holds one; disposing it ends it. + private async Task WriteTransactionAsync() + { + var connection = new SqliteConnection($"Data Source={_file};Pooling=False"); + await connection.OpenAsync(Token); + await using var command = connection.CreateCommand(); + command.CommandText = "BEGIN IMMEDIATE"; + await command.ExecuteNonQueryAsync(Token); + return connection; + } + + // Writes of the lock that give up waiting for SQLite's write lock at the first busy timeout, of 5 seconds, rather than + // retrying it until the command timeout of 30 seconds. + private static FluentCmsTestHost Impatient(FluentCmsTestHost host) => + host.Configure(cms => cms.Services.ConfigureDbContext(options => options.UseSqlite(sqlite => sqlite.CommandTimeout(1)))); + + // The first lease_until other than the one given, read every 10 milliseconds, since a renewal signals its UPDATE before it + // commits. + private static async Task NextLeaseAsync(FluentCmsTestHost host, DateTimeOffset? before) + { + var watch = Stopwatch.StartNew(); + while (true) + { + if ((await RowAsync(host)).LeaseUntil is { } lease && lease != before) + { + return lease; + } + + watch.Elapsed.ShouldBeLessThan(Patience); + await Task.Delay(TimeSpan.FromMilliseconds(10), Token); + } + } + private static async Task RowAsync(FluentCmsTestHost host) { await using var scope = host.Services.CreateAsyncScope(); diff --git a/tests/FluentCMS.Data.Tests/Migrations/LongTransaction.cs b/tests/FluentCMS.Data.Tests/Migrations/LongTransaction.cs new file mode 100644 index 000000000..b6f5288ed --- /dev/null +++ b/tests/FluentCMS.Data.Tests/Migrations/LongTransaction.cs @@ -0,0 +1,64 @@ +using System.Data.Common; +using FluentCMS.Core.Caching; +using FluentCMS.Core.DependencyInjection; +using FluentCMS.Core.Storage; +using FluentCMS.Providers; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Logging; +using Xunit; + +namespace FluentCMS.Data.Tests.Migrations; + +// Work on the content tables that keeps SQLite's write transaction for longer than the lease of the lock it holds: the +// first command that adds a column waits, inside the transaction, until a renewal of the lock gave up waiting for it and +// read the row, and the commands after it run as they would. The lock has a lease of 300 milliseconds on the real clock, +// and its writes give up after the first busy timeout, about 5 seconds, where a host's wait about 35. +internal sealed class LongTransaction : DbCommandInterceptor +{ + private readonly TaskCompletionSource _renewalGaveUp = new(TaskCreationOptions.RunContinuationsAsynchronously); + private int _held; + + // Registered before AddFluentCms, so that core's default lock gives way to it. + public static void Lock(IServiceCollection services) => + services.AddSingleton(provider => new DatabaseDistributedLock( + provider.GetRequiredService(), + TimeProvider.System, + provider.GetRequiredService>()) + { + Lease = TimeSpan.FromMilliseconds(300), + }); + + public void Configure(FluentCmsBuilder cms) + { + cms.Services.ConfigureDbContext(options => options + .UseSqlite(sqlite => sqlite.CommandTimeout(1)) + .AddInterceptors(new Renewals(this))); + cms.Services.ConfigureDbContext(options => options.AddInterceptors(this)); + } + + public override async ValueTask NonQueryExecutedAsync(DbCommand command, CommandExecutedEventData eventData, int result, CancellationToken cancellationToken = default) + { + if (command.CommandText.StartsWith("ALTER TABLE", StringComparison.OrdinalIgnoreCase) && Interlocked.Exchange(ref _held, 1) == 0) + { + await _renewalGaveUp.Task.WaitAsync(TimeSpan.FromSeconds(30), TestContext.Current.CancellationToken); + } + + return result; + } + + // A renewal that gave up reads the row; nothing else reads it while the command waits. + private sealed class Renewals(LongTransaction work) : DbCommandInterceptor + { + public override ValueTask ReaderExecutedAsync(DbCommand command, CommandExecutedEventData eventData, DbDataReader result, CancellationToken cancellationToken = default) + { + if (Volatile.Read(ref work._held) == 1 && command.CommandText.Contains("\"cms_lock\"", StringComparison.Ordinal)) + { + work._renewalGaveUp.TrySetResult(); + } + + return ValueTask.FromResult(result); + } + } +} diff --git a/tests/FluentCMS.Data.Tests/Migrations/SchemaApplyTests.cs b/tests/FluentCMS.Data.Tests/Migrations/SchemaApplyTests.cs index 96c454fdb..52e8d17ad 100644 --- a/tests/FluentCMS.Data.Tests/Migrations/SchemaApplyTests.cs +++ b/tests/FluentCMS.Data.Tests/Migrations/SchemaApplyTests.cs @@ -612,6 +612,22 @@ public async Task ARunGivesUpOnALockThatIsNeverReleased(bool frozenWallClock) exception.Message.ShouldStartWith($"The lock {MigrationRunner.SchemaSyncLock} was not free within 15 minutes, so the content tables were not changed; check the owner and lease_until of its row in cms_lock"); } + // SQLite has one writer, so the lock cannot renew its lease while the transaction of the run lasts. + [SqliteFact("SQLite's one writer")] + public async Task ARunWhoseTransactionOutlastsTheLeaseOfItsLockFinishes() + { + await MigrateAndApplyAsync(ArticleSchemas.Titled); + var work = new LongTransaction(); + using var host = _database.CreateHost(ArticleSchemas.Article("title", "subtitle", "summary"), Production, settings: NoMigrationOnStartup, services: LongTransaction.Lock, configure: work.Configure); + + var (exitCode, output, error) = await RunAsync(host, "schema", "apply"); + + exitCode.ShouldBe(0, error); + output.ShouldContain("Applied 2 changes; the schema history records schema "); + _database.Columns("cms_article").ShouldContain("subtitle"); + _database.Columns("cms_article").ShouldContain("summary"); + } + [Theory] [InlineData(new[] { "--output", "schema.sql" }, "FCMS3300: --output names the file of --script, so it needs --script")] [InlineData(new[] { "--script", "--dry-run" }, "FCMS3300: --script and --dry-run both apply nothing; give one")] diff --git a/tests/FluentCMS.Data.Tests/Migrations/SchemaSyncTests.cs b/tests/FluentCMS.Data.Tests/Migrations/SchemaSyncTests.cs index 331775817..1ea7f2f91 100644 --- a/tests/FluentCMS.Data.Tests/Migrations/SchemaSyncTests.cs +++ b/tests/FluentCMS.Data.Tests/Migrations/SchemaSyncTests.cs @@ -402,6 +402,22 @@ public async Task ASyncGivesUpOnALockThatIsNeverReleased(bool frozenWallClock) _database.Columns("cms_article").ShouldNotContain("subtitle"); } + // SQLite has one writer, so the lock cannot renew its lease while the transaction of the sync lasts. + [SqliteFact("schema sync, which only SQLite supports")] + public async Task ASyncWhoseTransactionOutlastsTheLeaseOfItsLockFinishes() + { + var work = new LongTransaction(); + using var host = _database.CreateHost(ArticleSchemas.Titled, Environments.Development, services: LongTransaction.Lock, configure: work.Configure); + await host.StartAsync(Token); + + var report = await Runner(host).SyncContentSchemaAsync(ArticleSchemas.Article("title", "subtitle", "summary"), Token); + + report.Applied.Count.ShouldBe(2); + _database.Columns("cms_article").ShouldContain("subtitle"); + _database.Columns("cms_article").ShouldContain("summary"); + await host.StopAsync(Token); + } + [SqliteFact("schema sync, which only SQLite supports")] public async Task SchemaSyncOffInDevelopmentUsesTheContentStream() { diff --git a/tests/FluentCMS.Data.Tests/Sqlite/RollbackJournalLockTests.cs b/tests/FluentCMS.Data.Tests/Sqlite/RollbackJournalLockTests.cs new file mode 100644 index 000000000..11db7ec4a --- /dev/null +++ b/tests/FluentCMS.Data.Tests/Sqlite/RollbackJournalLockTests.cs @@ -0,0 +1,56 @@ +using FluentCMS.Core.Storage; +using FluentCMS.Data.Tests.Providers; +using FluentCMS.Providers; +using Microsoft.Data.Sqlite; +using Microsoft.EntityFrameworkCore; +using Microsoft.Extensions.DependencyInjection; +using Shouldly; +using Xunit; + +namespace FluentCMS.Data.Tests.Sqlite; + +// The rollback journal of a host that opts out of WAL, where a writer that commits or spills its cache keeps readers out, +// so the read of a lock's row waits for it as a write does. Its commands give up after a second. +public sealed class RollbackJournalLockTests : IDisposable +{ + private const string Name = "tests:lock"; + + private readonly SqliteFileDatabase _database = new(writeAheadLog: false); + private readonly ServiceProvider _services; + + public RollbackJournalLockTests() + { + _services = _database.CreateServices(configure: cms => cms.Services.ConfigureDbContext(options => options.UseSqlite(sqlite => sqlite.CommandTimeout(1)))); + using var scope = _services.CreateScope(); + scope.ServiceProvider.GetRequiredService().Database.Migrate(); + + // EF creates a new database file in WAL mode. + _database.Execute("PRAGMA journal_mode = delete"); + } + + public void Dispose() + { + _services.Dispose(); + _database.Dispose(); + } + + [SqliteFact("SQLite's file locking")] + public async Task AWaiterWhoseReadOfTheRowWaitsEndsWithItsTimeout() + { + var token = TestContext.Current.CancellationToken; + var locks = _services.GetRequiredService(); + + await using (var writer = new SqliteConnection(_database.ConnectionString)) + { + await writer.OpenAsync(token); + await using var command = writer.CreateCommand(); + command.CommandText = "BEGIN EXCLUSIVE"; + await command.ExecuteNonQueryAsync(token); + + (await locks.TryAcquireAsync(Name, TimeSpan.FromMilliseconds(500), token)).ShouldBeNull(); + } + + await using var held = await locks.TryAcquireAsync(Name, TimeSpan.Zero, token); + held.ShouldNotBeNull(); + } +} diff --git a/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs b/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs index 982715d6b..2de511b2b 100644 --- a/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs +++ b/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs @@ -62,6 +62,8 @@ private sealed class UnflushedSqliteProvider(SqliteDataProvider inner) : IDataPr public bool IsDeadlock(Exception exception) => inner.IsDeadlock(exception); + public bool IsBusy(Exception exception) => inner.IsBusy(exception); + public Expression> ContainsIgnoringCase(string fragment) => inner.ContainsIgnoringCase(fragment); public string? FilePath => ((IDatabaseFile)inner).FilePath; From af45a9d79f07015539c6f6a6b528be419d623367 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 01:00:42 -0500 Subject: [PATCH 2/8] Wait for in-process handlers past any load in the data tests (#3188) Closes #3187 A test bug, not a product bug. The list of received events is a `ConcurrentQueue`, so it never changed under an enumeration. What changed was when the event arrived. `ShouldHaveSingleItem` counts the list once and lists it again for the message, and the queue only grows after `Clear()`. So the count found no event, and by the time the message was built, the event was there. The event arrived after `PublishAsync` returned, which the spec (In-process handlers, as built) allows. A write waits for each in-process handler at most `FluentCms:Events:Handlers:Timeout`, 5 seconds by default. After that it goes on and leaves the handler running. `EventDispatcher.RunAsync` starts the handler with `Task.Run` and waits on `WaitAsync(options.Timeout, time)`. The failing run took 15 s on a loaded Windows runner, so the handler, which was not even scheduled yet, went past the bound. The tests of the data project read what their handlers saw as soon as the write returns, so all of them depended on that bound never being reached. - `TestDatabase.CreateServices` gives in-process handlers a 1-minute bound, as `EventLimitsTests` already does where the bound under test is another one. While the bound is not reached, a write returns only after its handlers ran, so no handler can still write to a list once the write has returned. Ordered assertions such as `Calls.ShouldBe([...])` and the database a handler saw stay exact, which a wait for the event would not give. The tests of the bound set their own value in `configure`, which runs after this. - `Seen.Of()` returns a copy, so an assertion and its message read the same events. The other event test classes read `Seen` after a write, a dispatch or an awaited drain. The one test that starts a background runner (`OutboxTests`) already waits for the delivery with a bound. The integration tests that record invalidations already take a snapshot after an awaited drain, or wait on a channel with a bound. ## Dependencies None. ## Acceptance - [x] The test asserts on a list that can no longer change, and passes repeated under load - [x] No other test of the event test classes enumerates a list that a handler can still write to --- tests/FluentCMS.Data.Tests/Events/EventFixture.cs | 3 ++- tests/FluentCMS.Data.Tests/Providers/TestDatabase.cs | 3 +++ 2 files changed, 5 insertions(+), 1 deletion(-) diff --git a/tests/FluentCMS.Data.Tests/Events/EventFixture.cs b/tests/FluentCMS.Data.Tests/Events/EventFixture.cs index f4b168cfa..b49ecef70 100644 --- a/tests/FluentCMS.Data.Tests/Events/EventFixture.cs +++ b/tests/FluentCMS.Data.Tests/Events/EventFixture.cs @@ -25,8 +25,9 @@ internal sealed class Seen // Runs inside the handler, so a test can look at the database at the moment the event arrives or make the handler fail. public Func Behavior { get; set; } = (_, _) => Task.CompletedTask; + // A copy, so that an assertion and its message read the same events. public IEnumerable Of() - where T : CmsEvent => Events.OfType(); + where T : CmsEvent => [.. Events.OfType()]; public async ValueTask HeardAsync(string name, CmsEvent @event, EventDelivery? delivery, CancellationToken token) { diff --git a/tests/FluentCMS.Data.Tests/Providers/TestDatabase.cs b/tests/FluentCMS.Data.Tests/Providers/TestDatabase.cs index 319389584..d0b766a61 100644 --- a/tests/FluentCMS.Data.Tests/Providers/TestDatabase.cs +++ b/tests/FluentCMS.Data.Tests/Providers/TestDatabase.cs @@ -54,6 +54,9 @@ public ServiceProvider CreateServices(SchemaModel? schema = null, Action Date: Mon, 16 Nov 2026 02:09:23 -0500 Subject: [PATCH 3/8] Bound the wait for an abandoned migrations lock row on SQLite (#3190) On SQLite, EF Core's migrations lock is a row in `__EFMigrationsLock`. A process killed while it migrates leaves the row behind, and EF then retries every second with no end. I reproduced it: a start with `MigrateOnStartup` and a pending migration, and `fluentcms migrate`, waited until the test cancelled them, and nothing in the log named the row. A start with nothing pending is not held up, because the runner calls `MigrateAsync` only for a stream that has something to apply. Before each stream that has a pending migration, the runner now reads the row through a new `IDataProvider.ReadMigrationsLockAsync`. On SQLite that is the row's time, or null for a database that does not exist yet, so EF's creator still makes it in WAL mode. On PostgreSQL and SQL Server it is null, because their locks end with their transaction or session. While the row is there, the runner waits, before it logs "Applying". After 30 seconds it logs a warning once (event 9305) and then looks every 5 seconds. After 15 minutes it fails with a `TimeoutException`. A start stops, and `migrate` exits with 1 and `FCMS3302`. Both the warning and the error: - name the row and its time, and say that the time is when the lock was taken and is never renewed - give the `DELETE` that clears the row once no other host and no `fluentcms migrate` runs on the database - say that if the next run fails on a table or column that already exists, an interrupted table rebuild was partly applied and the backup is restored The row is never cleared automatically. Its time cannot tell a dead holder from another host's start or a `fluentcms migrate` still rebuilding large tables, and EF has no supported way to bound or clear the lock. One gap remains: two processes that both read no row before either inserts leave the second waiting inside EF, with no warning and no bound, as before. The tests take the lock through EF's own `IHistoryRepository.AcquireDatabaseLockAsync`, so a change of EF's lock fails them. Another test checks that a first start still creates the database in WAL mode with `writeAheadLog: false`. On SQL Server, EF's `sp_getapplock` has no lock timeout. A wait for another process's migrations fails at the command timeout (30 seconds) with `Execution Timeout Expired`. That is left to #3192. The docs say what an operator sees and does: - `content-migrations.md` covers the partial migration and the remaining gap. - `docker.md` shows how to delete the row in a volume from a throwaway container, since the image has no shell and no `sqlite3`. - The backend spec records the design. Closes #3177 ## Dependencies None. ## Acceptance - [x] A start on SQLite with an abandoned row in `__EFMigrationsLock` ends within a bound, with an error that names the cause or with the migration applied - [x] A test holds it - [x] The operations documentation says what an operator sees and does --- docs/content-migrations.md | 20 ++ docs/docker.md | 15 +- docs/planning/backend-technical-spec.md | 4 +- src/FluentCMS.Core/Storage/IDataProvider.cs | 5 + src/FluentCMS.Core/Storage/MigrationLog.cs | 3 + src/FluentCMS.Core/Storage/MigrationRunner.cs | 51 +++++ .../PostgreSqlDataProvider.cs | 3 + .../SqlServerDataProvider.cs | 3 + .../SqliteDataProvider.cs | 23 ++ .../Migrations/MigrationLockTests.cs | 211 ++++++++++++++++++ .../Sqlite/SqliteFileDatabase.cs | 2 + 11 files changed, 338 insertions(+), 2 deletions(-) create mode 100644 tests/FluentCMS.Data.Tests/Migrations/MigrationLockTests.cs diff --git a/docs/content-migrations.md b/docs/content-migrations.md index a5d071a68..3350ea169 100644 --- a/docs/content-migrations.md +++ b/docs/content-migrations.md @@ -117,6 +117,26 @@ A command runs before any hosted service starts, so nothing migrates on start, l The tool sends the host's console logs to standard error, so that standard output holds only what the command writes. Neither `migrate` nor the other commands need `MigrateOnStartup`. +### The migrations lock on SQLite + +EF Core holds a lock while it applies migrations, and on SQLite that lock is a row in the table `__EFMigrationsLock`, which EF deletes when it is done. A process killed while it applies migrations leaves the row behind ([#3177](https://github.com/fluentcms/FluentCMS/issues/3177)). A start with `MigrateOnStartup` and `fluentcms migrate` take the lock only for a stream that has a pending migration, so a start with nothing pending is not held up by the row. One that has something to apply waits for the row to go. After 30 seconds it logs a warning once (event id 9305) that names the stream and the time the row was written, and then looks every 5 seconds. After 15 minutes it fails with a `TimeoutException` that says the same, and nothing of that stream is applied. A start stops there, and `fluentcms migrate` exits with 1 and `FCMS3302`: + +```text +FCMS3302: 'fluentcms migrate' failed: TimeoutException: EF Core's migrations lock, the row of __EFMigrationsLock written at 2026-11-15 10:00:00+00:00, was not released within 15 minutes, so the system stream was not migrated; ... +``` + +A healthy holder is another host's start or a `fluentcms migrate` applying migrations to the same database, and the row goes when it is done. The time in the row is when the lock was taken, and nothing renews it, so an old time does not show that the holder stopped. The row is never cleared on its own, because a migration that rebuilds large tables can take long. Check that no other host and no `fluentcms migrate` runs on the database file, then delete the row and start again ([in a container](docker.md#a-migrations-lock-left-behind)): + +```sql +DELETE FROM "__EFMigrationsLock"; +``` + +A migration that only adds tables, columns or indexes runs in one transaction, so a process killed inside it left nothing of it behind. A migration that rebuilds a table on SQLite, such as one that drops or changes a column, commits around the rebuild, and EF writes the migration's history row last. A process killed inside such a migration can leave it partly applied. Then the next run fails on a table or column that already exists, such as `duplicate column name`, or leaves a table named `ef_temp_...`. Restore the backup taken before the upgrade ([backups](backups.md)) and run again. + +One gap remains. Two processes that both read that no row is there before either takes the lock leave the second one waiting inside EF. That wait logs no warning and has no bound, and if the first is killed, the second waits as it did before this check. + +PostgreSQL and SQL Server hold their lock with the session or transaction that took it, so a process that stops releases it. On SQL Server, a process that waits for another's migrations longer than the command timeout (30 seconds by default) fails with `Execution Timeout Expired`, which does not name the lock. It succeeds when run again once the other is done ([#3192](https://github.com/fluentcms/FluentCMS/issues/3192)). + ## Startup validation With `MigrateOnStartup` set to `false`, startup first stops with `FCMS1305` if the system or a plugin stream has a pending migration, which is what an upgrade to a new FluentCMS version can bring: diff --git a/docs/docker.md b/docs/docker.md index 4593fdc67..e3bed23eb 100644 --- a/docs/docker.md +++ b/docs/docker.md @@ -40,7 +40,7 @@ docker run -d --name fluentcms -p 8080:8080 -v fluentcms-data:/data \ `FluentCms__PublicOrigin` is the https address you reach the instance at through the proxy that ends TLS, which sign-in needs (see [before production](#before-production)). With no such address yet, `FluentCms__Security__AllowRequestOrigin=true` in its place builds links from each request. The startup report keeps listing each opt-in, as `generated-signing-key`, `data-protection-keys-unencrypted` or `public-origin-unset`. -It runs on SQLite, in `/data/fluentcms.db`, and applies the migrations on start. Keep `/data` on a volume, or the database goes with the container. The database is in WAL mode, so `/data` also holds `fluentcms.db-wal` and `fluentcms.db-shm` while the host runs: keep the three files together, and stop the container before copying them, since `fluentcms.db` alone misses the commits still in `-wal`, or take a backup while it runs with `fluentcms backup` ([backups](backups.md)). Put the volume on a local disk, not a network share such as NFS, SMB or Azure Files, where SQLite's locks and the shared memory WAL needs do not work (see [SQLite](host-configuration.md#sqlite)). `GET http://localhost:8080/health/ready` answers `Healthy` once the database and the job runner are up, and `docker ps` shows the container as healthy: the image's `HEALTHCHECK` runs the host itself with `--probe`, because a chiseled image has no `curl`. +It runs on SQLite, in `/data/fluentcms.db`, and applies the migrations on start; a container killed while it applied them can leave EF's lock row behind, which the next start names and waits for ([a migrations lock left behind](#a-migrations-lock-left-behind)). Keep `/data` on a volume, or the database goes with the container. The database is in WAL mode, so `/data` also holds `fluentcms.db-wal` and `fluentcms.db-shm` while the host runs: keep the three files together, and stop the container before copying them, since `fluentcms.db` alone misses the commits still in `-wal`, or take a backup while it runs with `fluentcms backup` ([backups](backups.md)). Put the volume on a local disk, not a network share such as NFS, SMB or Azure Files, where SQLite's locks and the shared memory WAL needs do not work (see [SQLite](host-configuration.md#sqlite)). `GET http://localhost:8080/health/ready` answers `Healthy` once the database and the job runner are up, and `docker ps` shows the container as healthy: the image's `HEALTHCHECK` runs the host itself with `--probe`, because a chiseled image has no `curl`. ## Content types @@ -202,6 +202,19 @@ initContainers: name: fluentcms-database ``` +### A migrations lock left behind + +On SQLite, a container killed while it applied migrations can leave EF's lock row in `__EFMigrationsLock`. The next start or `fluentcms migrate` then logs a warning (event id 9305) after 30 seconds and fails after 15 minutes, naming the row ([the migrations lock on SQLite](content-migrations.md#the-migrations-lock-on-sqlite)). The image has no shell and no `sqlite3`, so stop every container that uses the volume, check that no `fluentcms migrate` runs on it, and delete the row from a throwaway container that mounts the volume: + +```bash +docker stop fluentcms +docker run --rm -v fluentcms-data:/data alpine:3.20 \ + sh -c 'apk add --no-cache sqlite && sqlite3 /data/fluentcms.db "DELETE FROM __EFMigrationsLock;"' +docker start fluentcms +``` + +If the next start then fails on a table or column that already exists, the interrupted migration was partly applied: restore the backup taken before the upgrade ([backups](backups.md)). + ## Administer an instance The same image runs the commands that administer the instance, as the system, with the origin `Cli` in the activity log: diff --git a/docs/planning/backend-technical-spec.md b/docs/planning/backend-technical-spec.md index f9ce25a0f..77e5ff5eb 100644 --- a/docs/planning/backend-technical-spec.md +++ b/docs/planning/backend-technical-spec.md @@ -552,6 +552,8 @@ At runtime one `CmsDbContext` holds the union model for querying, with the syste **The commands of the runner as built (B-90, [#3013](https://github.com/fluentcms/FluentCMS/issues/3013)).** `fluentcms migrate` calls `MigrateAsync`, which applies every stream whatever `MigrateOnStartup` says, refuses a foreign set first (`FCMS1306`) and, for a host with content migrations, records the schema history when no column is missing, and prints what it applied under each stream. What a call applied is what EF names, in the `MigrationApplying` event of its diagnostic listener, for that call's own migrator (`AppliedMigrations`), not what was pending before: two processes that migrate one database at once both find a migration pending, one of them applies it, and the other reports nothing for it. EF's migrations lock is enough on SQLite and SQL Server; Npgsql takes it inside each migration's transaction, after the migrator has read what is applied, so two processes both applied the next migration. `IDataProvider.LockMigrationsAsync`, a member with a default that takes nothing, lets a provider hold the database while the runner applies its streams: `PostgreSqlDataProvider` takes a session advisory lock keyed by `hashtext(current_database() || ':fluentcms:migrations')`, since advisory keys are shared by the server, and releases it explicitly, since a pooled connection keeps its session. The runner takes it before it reads what is pending, at start and in `migrate` alike. On SQL Server, the first step of the system baseline, which turns `READ_COMMITTED_SNAPSHOT` on, disconnects the database's other sessions (ADR 0005), so on a new database whose option is off, a second run that waits for EF's lock meanwhile fails and succeeds when run again; the data tests run with the option on, and `docs/docker.md` says to run one migrate step or let it restart. `migrate --check` lists `GetPendingMigrationsAsync` and exits 1 when anything is pending (`FCMS3303`). `MigrationRunner.ListAsync` gives each stream's applied and pending migrations (`migrations list`), `MigrationsOf` the set of a stream, and `ScriptAsync` the SQL of the host's provider from one migration of a stream to another, through EF's `IMigrator.GenerateScript`, without opening the database; a plugin or content set scaffolded for another provider is refused as `MigrateAsync` refuses it. `migrations script` without `--stream` writes every stream that has migrations in the runner's order, each after a `-- stream {name}` line; `--from` and `--to` name a migration of one stream, so they need `--stream`; an unknown stream or migration is `FCMS3304` or `FCMS3305`. EF writes no idempotent script for SQLite, so `--idempotent` there is `FCMS3306`. A script records no schema history. `migrations remove` (6.1) deletes the newest content migration through EF's scaffolder with a history repository that reports nothing applied and a migrator that refuses any use, because EF's own check opens the factory's connection, and with `--force` reverts the migration in that database; the tool asks the host instead. +**The migrations lock on SQLite as built ([#3177](https://github.com/fluentcms/FluentCMS/issues/3177)).** EF Core 10 takes its migrations lock on SQLite by inserting the row with id 1 and the time into `__EFMigrationsLock` and releases it by deleting the row, so a process killed inside `MigrateAsync` leaves the row, and `AcquireDatabaseLockAsync` then tries again every second with no end, its cancellation token being the only way out; the type is internal and has no option that bounds or clears the lock. Before each stream that has a pending migration, the runner therefore reads `IDataProvider.ReadMigrationsLockAsync`, the row's time on SQLite and null on PostgreSQL and SQL Server, whose locks end with their transaction or session, and waits while it is there, before it logs 9301: it logs once after 30 seconds (9305), then looks every 5 seconds, since EF logs each read, and gives up after `MigrationsLockWait`, 15 minutes, with a `TimeoutException`, at start and in `migrate` alike. The warning and the error name the row and its time, say that the time is when the lock was taken and is never renewed, give the `DELETE` that clears it once no other host and no `fluentcms migrate` runs on the database, and say that a migration that rebuilds a table may have been partly applied, since EF commits around the rebuild and writes the history row last, so that a next run that fails on a table or column that exists needs the backup. On SQLite the read returns null for a database that does not exist yet, so that EF's creator still makes it, in WAL mode. A stream with nothing pending never takes the lock, so a start with nothing to apply is not held up. The row is not cleared on its own: a `fluentcms migrate` beside a host, or another host's start, may be applying a long migration, and only those call `MigrateAsync` (`schema apply` takes `fluentcms:schema-sync`). Two processes that both read no row before either inserts leave the second waiting inside EF, with no warning and no bound, as before. On SQL Server, EF's `sp_getapplock` has no lock timeout, so a wait for another process's migrations ends at the command timeout, 30 seconds by default, with `Execution Timeout Expired`, which names nothing; that run fails and succeeds once run again ([#3192](https://github.com/fluentcms/FluentCMS/issues/3192)). + **Schema sync and plugin tables as built.** Schema sync builds its context from the options of the host's `CmsDbContext` with the target schema swapped in, so its model maps the plugins' tables as the runtime context does. A table under the prefix `cms_{plugin}_` of a plugin of the plugin host or of a stream registered through `IPluginMigrations` is never a stale content table, whether the plugin maps it, migrates it or neither, and neither is any `__migrations` history table. Nor is one under the prefix of a plugin that the host does not register while the database has its history table `cms_{plugin}__migrations`, as after a plugin is taken out of the host ([#3072](https://github.com/fluentcms/FluentCMS/issues/3072)); the system's `cms__migrations` and the content stream's `cms_content__migrations` name no plugin. A host that runs schema sync with a plugin that owns tables therefore records its schema history row. The code review of 7 Nov 2026 asked whether the plugin-less context was deliberate; it was not (#2872). **The provider of a migration set as built.** A content or plugin set carries the store types of the provider it was scaffolded for, and startup validation compares names only, so EF would apply a SQLite set to PostgreSQL unnoticed. A provider switch on an existing database is not supported (#2875). Startup and `MigrateAsync` read the provider that EF writes on the model of every snapshot and migration target model: PostgreSQL and SQL Server record their value generation strategy under `Npgsql:` and `SqlServer:`, and SQLite records neither. They compare it with the provider of the context's own model and refuse a set that differs with `FCMS1306`, which names each stream, the set's provider and the host's (`stream content was scaffolded for SQLite; this host runs PostgreSQL, and a migration set applies only to the provider it was scaffolded for`). This happens before anything is applied and without opening the database. The migration files need no marker. A hand-written migration with no target model names no provider and is not refused. The system stream ships with its provider package and is not checked, and schema sync skips the content stream, so a host can sync on SQLite in Development while its content set is for its server provider. A plugin's set is its provider's own (below). Startup still compares names only, although the information schema reader now reports whole store types (#2879). @@ -1163,7 +1165,7 @@ Rows 10.3, 10.5 to 10.8, 10.11 to 10.15. **Observability.** One `ActivitySource` (`FluentCMS`) with spans for content operations, hooks, jobs, webhooks and MCP calls; one `Meter` with request counts by surface, query duration, job latency, cache hit ratio, webhook outcomes; structured logs with event ids and `FCMS` codes. Exporters are the host's choice. Nothing is sent to the project. -**Health and observability, as built.** `FluentCmsHealthCheckTags` (`FluentCMS.Diagnostics`, in the SDK, so plugins tag their checks) has two tags, `live` and `ready`, and `/health/live` and `/health/ready` run the checks tagged with each through `Microsoft.Extensions.Diagnostics.HealthChecks`, with no authentication and no rate limit policy of the endpoints. They stay reachable without credentials, because the image's `HEALTHCHECK` and an orchestrator's probes call them, so they are kept cheap and bounded instead: the FluentCMS authentication step skips the `health` surface, so a key sent there is never looked up, and the check before authentication counts them per client address in a partition of their own, apart from the coarse limit, with `FluentCms:RateLimits:Health:PermitLimit` (1200 a minute, twenty a second from one address, far above what an orchestrator sends one instance). A path under `/health` that no endpoint answers counts there too. A refusal is the same 429 problem, and the probes carry no `RateLimit-*` headers. Where the network allows, probes should still be restricted to the orchestrator and the load balancer. This answers decision 16 of the review of 7 Nov 2026 ([#2847](https://github.com/fluentcms/FluentCMS/issues/2847), [#2855](https://github.com/fluentcms/FluentCMS/issues/2855)). They answer 200 for Healthy and Degraded and 503 for Unhealthy, with the status as the only body, never a description or an exception. Core registers `process` (live), `database` (ready, one row of a system table, so it also fails when the system migrations did not run) and `job-runner` (ready); `services.AddHealthChecks().AddFluentCmsReadyCheck("name")` adds a ready check with a five second timeout (it is in Core, because the SDK has no reference to the health check abstractions; a plugin that references only the SDK calls `AddCheck` with the tag). The job runner beats in memory at every turn of its maintenance loop (`JobRunnerOptions.MaintenanceInterval`, 15 seconds), and ready is Unhealthy when the last beat is older than `FluentCms:Health:JobRunnerHeartbeatTimeout` (default 2 minutes; `Timeout.InfiniteTimeSpan` leaves it out for an instance that runs no jobs) or when the runner has not started, and Degraded when the oldest pending job of a queue has waited longer than `FluentCms:Health:PendingJobAgeLimit` (4.12). `GET /api/management/v1/system/health` needs `system.read`, an instance-level permission: it is in the catalogue and checked with `InstanceAccess`, so a role of one site or a grant limited to sites does not reach it, nor does a key bound to sites or a Delivery or Preview key. It runs every check and answers 200 with `{ status, totalDurationMs, checks: [{ name, status, durationMs, tags, description, error }] }`, whatever the status; descriptions and exception text pass through `SecretScrubber`. The one `ActivitySource` and the one `Meter` are both named `FluentCMS` (`FluentCmsTelemetry.Name`), and nothing is exported: a host subscribes with `AddSource` and `AddMeter` from its own OpenTelemetry setup, and Core takes no dependency on it. Spans are named `content {operation}` (create, update, patch, delete, restore, purge, publish, unpublish, discard-draft, schedule, cancel-schedule, versions, version, restore-version, diff, get, query, query-trash, count, bulk), `hook {kind}` (one per run of the hooks of a kind, with `fluentcms.hooks` their number) and `job {type}` (with `fluentcms.queue`). The MCP plugin adds `mcp {tool}` for a tool call (5.4). They carry `fluentcms.operation`, `fluentcms.type` (the name the schema resolves, and not on jobs) and `fluentcms.site`, and never a value of content; a job's span has the Error status when the attempt throws. The instruments are `fluentcms.requests` (counter, by `fluentcms.surface` of delivery, management, connect (with `/.well-known`), account, health or docs (the OpenAPI document and Scalar), and `fluentcms.status_class`, counted with the status the client got, including a refusal an endpoint throws), `fluentcms.query.duration` (histogram, seconds, by the `fluentcms.type` the schema resolved, and not recorded for a query refused as invalid), `fluentcms.job.latency` (histogram, seconds, the length of an attempt by `fluentcms.operation` the job type and `fluentcms.outcome`), `fluentcms.job.oldest_pending_age` (observable gauge, seconds, by `fluentcms.queue`, 4.12), `fluentcms.event.handler.duration` (histogram, seconds, the length of a run of an in-process event handler by `fluentcms.operation` the event type, `fluentcms.handler` and `fluentcms.outcome`, 4.11), and `fluentcms.cache.requests` (counter by `fluentcms.cache` and `fluentcms.result`) with `fluentcms.cache.hit_ratio` (observable gauge by `fluentcms.cache`); the Delivery output cache reports its lookups as `delivery`. Every `[LoggerMessage]` in Core has an event id, in a range per area that follows the `FCMS` families: 90xx identity (the authorization server 9001 to 9003 and 9005, the hosted pages 9004, admin SSO 9040 to 9044, the member pool 9070 to 9076, member providers 9090 to 9094), 91xx host (health 9101 to 9104, the problem handler 9110, email 9120, the startup report 9130 to 9132, the distributed lock 9160 to 9165 of the range 9160 to 9169, the command runner 9170 to 9172 of the range 9170 to 9179), 92xx plugins (per-site enablement 9230 to 9234), 93xx storage (migrations 9301, schema history 9311), 94xx content (9401, scheduled jobs 9411, trash purge 9421), 95xx jobs, 96xx events, 97xx activity, 98xx settings and 99xx seeding. The event name is the method name, and a message carries the `FCMS` code where one applies. +**Health and observability, as built.** `FluentCmsHealthCheckTags` (`FluentCMS.Diagnostics`, in the SDK, so plugins tag their checks) has two tags, `live` and `ready`, and `/health/live` and `/health/ready` run the checks tagged with each through `Microsoft.Extensions.Diagnostics.HealthChecks`, with no authentication and no rate limit policy of the endpoints. They stay reachable without credentials, because the image's `HEALTHCHECK` and an orchestrator's probes call them, so they are kept cheap and bounded instead: the FluentCMS authentication step skips the `health` surface, so a key sent there is never looked up, and the check before authentication counts them per client address in a partition of their own, apart from the coarse limit, with `FluentCms:RateLimits:Health:PermitLimit` (1200 a minute, twenty a second from one address, far above what an orchestrator sends one instance). A path under `/health` that no endpoint answers counts there too. A refusal is the same 429 problem, and the probes carry no `RateLimit-*` headers. Where the network allows, probes should still be restricted to the orchestrator and the load balancer. This answers decision 16 of the review of 7 Nov 2026 ([#2847](https://github.com/fluentcms/FluentCMS/issues/2847), [#2855](https://github.com/fluentcms/FluentCMS/issues/2855)). They answer 200 for Healthy and Degraded and 503 for Unhealthy, with the status as the only body, never a description or an exception. Core registers `process` (live), `database` (ready, one row of a system table, so it also fails when the system migrations did not run) and `job-runner` (ready); `services.AddHealthChecks().AddFluentCmsReadyCheck("name")` adds a ready check with a five second timeout (it is in Core, because the SDK has no reference to the health check abstractions; a plugin that references only the SDK calls `AddCheck` with the tag). The job runner beats in memory at every turn of its maintenance loop (`JobRunnerOptions.MaintenanceInterval`, 15 seconds), and ready is Unhealthy when the last beat is older than `FluentCms:Health:JobRunnerHeartbeatTimeout` (default 2 minutes; `Timeout.InfiniteTimeSpan` leaves it out for an instance that runs no jobs) or when the runner has not started, and Degraded when the oldest pending job of a queue has waited longer than `FluentCms:Health:PendingJobAgeLimit` (4.12). `GET /api/management/v1/system/health` needs `system.read`, an instance-level permission: it is in the catalogue and checked with `InstanceAccess`, so a role of one site or a grant limited to sites does not reach it, nor does a key bound to sites or a Delivery or Preview key. It runs every check and answers 200 with `{ status, totalDurationMs, checks: [{ name, status, durationMs, tags, description, error }] }`, whatever the status; descriptions and exception text pass through `SecretScrubber`. The one `ActivitySource` and the one `Meter` are both named `FluentCMS` (`FluentCmsTelemetry.Name`), and nothing is exported: a host subscribes with `AddSource` and `AddMeter` from its own OpenTelemetry setup, and Core takes no dependency on it. Spans are named `content {operation}` (create, update, patch, delete, restore, purge, publish, unpublish, discard-draft, schedule, cancel-schedule, versions, version, restore-version, diff, get, query, query-trash, count, bulk), `hook {kind}` (one per run of the hooks of a kind, with `fluentcms.hooks` their number) and `job {type}` (with `fluentcms.queue`). The MCP plugin adds `mcp {tool}` for a tool call (5.4). They carry `fluentcms.operation`, `fluentcms.type` (the name the schema resolves, and not on jobs) and `fluentcms.site`, and never a value of content; a job's span has the Error status when the attempt throws. The instruments are `fluentcms.requests` (counter, by `fluentcms.surface` of delivery, management, connect (with `/.well-known`), account, health or docs (the OpenAPI document and Scalar), and `fluentcms.status_class`, counted with the status the client got, including a refusal an endpoint throws), `fluentcms.query.duration` (histogram, seconds, by the `fluentcms.type` the schema resolved, and not recorded for a query refused as invalid), `fluentcms.job.latency` (histogram, seconds, the length of an attempt by `fluentcms.operation` the job type and `fluentcms.outcome`), `fluentcms.job.oldest_pending_age` (observable gauge, seconds, by `fluentcms.queue`, 4.12), `fluentcms.event.handler.duration` (histogram, seconds, the length of a run of an in-process event handler by `fluentcms.operation` the event type, `fluentcms.handler` and `fluentcms.outcome`, 4.11), and `fluentcms.cache.requests` (counter by `fluentcms.cache` and `fluentcms.result`) with `fluentcms.cache.hit_ratio` (observable gauge by `fluentcms.cache`); the Delivery output cache reports its lookups as `delivery`. Every `[LoggerMessage]` in Core has an event id, in a range per area that follows the `FCMS` families: 90xx identity (the authorization server 9001 to 9003 and 9005, the hosted pages 9004, admin SSO 9040 to 9044, the member pool 9070 to 9076, member providers 9090 to 9094), 91xx host (health 9101 to 9104, the problem handler 9110, email 9120, the startup report 9130 to 9132, the distributed lock 9160 to 9165 of the range 9160 to 9169, the command runner 9170 to 9172 of the range 9170 to 9179), 92xx plugins (per-site enablement 9230 to 9234), 93xx storage (migrations 9301 to 9305, schema history 9311), 94xx content (9401, scheduled jobs 9411, trash purge 9421), 95xx jobs, 96xx events, 97xx activity, 98xx settings and 99xx seeding. The event name is the method name, and a message carries the `FCMS` code where one applies. **Multi-instance.** Stateless API; Data Protection and signing keys persisted to the database by default; jobs lease through the database; `IDistributedLock` defaults to a `cms_lock` table and gets a Redis provider in `FluentCMS.Caching.Redis`; output cache in memory by default or Redis; media on shared storage. diff --git a/src/FluentCMS.Core/Storage/IDataProvider.cs b/src/FluentCMS.Core/Storage/IDataProvider.cs index bb75603ff..d079ca0d8 100644 --- a/src/FluentCMS.Core/Storage/IDataProvider.cs +++ b/src/FluentCMS.Core/Storage/IDataProvider.cs @@ -53,4 +53,9 @@ public interface IDataProvider // Holds the database while one process applies its streams, for a provider whose own migrations lock lets two processes // apply the same migration; disposing it lets the next go. Null where EF's lock is enough. Task LockMigrationsAsync(CancellationToken cancellationToken) => Task.FromResult(null); + + // The time written in the row of __EFMigrationsLock while one holds EF's migrations lock, as the database keeps it, for a + // provider whose lock is that row, which a process that stops while it migrates leaves behind. Null when no row holds + // it, and always where the lock ends with the session or transaction that took it. + Task ReadMigrationsLockAsync(DbContext context, CancellationToken cancellationToken); } diff --git a/src/FluentCMS.Core/Storage/MigrationLog.cs b/src/FluentCMS.Core/Storage/MigrationLog.cs index bcc445083..94ff6949a 100644 --- a/src/FluentCMS.Core/Storage/MigrationLog.cs +++ b/src/FluentCMS.Core/Storage/MigrationLog.cs @@ -15,4 +15,7 @@ internal static partial class MigrationLog [LoggerMessage(EventId = 9304, Level = LogLevel.Warning, Message = "{Key} is Auto, but schema sync runs only in Development on SQLite, or where the host allows it with AllowSchemaSyncOutsideDevelopment; the content stream applies instead.")] public static partial void SchemaSyncRefused(ILogger logger, string key); + + [LoggerMessage(EventId = 9305, Level = LogLevel.Warning, Message = "The {Stream} stream waits for EF Core's migrations lock, the row of __EFMigrationsLock written at {Written}; this is logged once, and the wait fails after {Minutes} minutes; {Fix}.")] + public static partial void WaitingForMigrationsLock(ILogger logger, string stream, string written, double minutes, string fix); } diff --git a/src/FluentCMS.Core/Storage/MigrationRunner.cs b/src/FluentCMS.Core/Storage/MigrationRunner.cs index 780f94008..23ab34cf0 100644 --- a/src/FluentCMS.Core/Storage/MigrationRunner.cs +++ b/src/FluentCMS.Core/Storage/MigrationRunner.cs @@ -48,9 +48,20 @@ public sealed class MigrationRunner( // on a lease that never runs out by this instance's clock. internal static readonly TimeSpan SchemaSyncWait = TimeSpan.FromMinutes(15); + // As far above a healthy wait: another process may be applying a migration that rebuilds large tables. + private static readonly TimeSpan MigrationsLockWait = TimeSpan.FromMinutes(15); + + private static readonly TimeSpan MigrationsLockWaitLogged = TimeSpan.FromSeconds(30); + + private static readonly TimeSpan FirstMigrationsLockPoll = TimeSpan.FromMilliseconds(50); + private static readonly TimeSpan LongestMigrationsLockPoll = TimeSpan.FromSeconds(1); + private static readonly TimeSpan ReportedMigrationsLockPoll = TimeSpan.FromSeconds(5); + private static readonly Stream System = new(SystemStream, services => services.GetRequiredService()); private static readonly Stream Content = new(ContentStream, services => services.GetRequiredService()); + private readonly TimeProvider _time = time ?? TimeProvider.System; + private readonly SchemaHistory _history = new(scopes, time ?? TimeProvider.System, logger); private readonly IPluginMigrations[] _plugins = Checked(plugins); @@ -524,6 +535,7 @@ private async Task> ApplyAsync(IEnumerable()); await context.Database.MigrateAsync(cancellationToken); @@ -533,6 +545,45 @@ private async Task> ApplyAsync(IEnumerable= MigrationsLockWaitLogged) + { + MigrationLog.WaitingForMigrationsLock(logger, stream, written, MigrationsLockWait.TotalMinutes, MigrationsLockFix); + logged = true; + } + + if (waited >= MigrationsLockWait) + { + throw new TimeoutException( + $"EF Core's migrations lock, the row of __EFMigrationsLock written at {written}, was not released within " + + $"{MigrationsLockWait.TotalMinutes:0} minutes, so the {stream} stream was not migrated; {MigrationsLockFix}."); + } + + // Each read is logged by EF, so a wait that has been reported looks less often. + await Task.Delay(poll, _time, cancellationToken); + poll = logged ? ReportedMigrationsLockPoll : poll * 2 < LongestMigrationsLockPoll ? poll * 2 : LongestMigrationsLockPoll; + } + } + + private const string MigrationsLockFix = + "the time is when the lock was taken, and it is never renewed; a host that starts or a 'fluentcms migrate' holds the " + + "lock while it applies migrations, and one that stopped while it migrated left the row behind; check that no other host " + + "and no 'fluentcms migrate' runs on the database, then delete the row with DELETE FROM \"__EFMigrationsLock\" and run " + + "again; a migration that rebuilds a table may have been partly applied, so if the next run fails on a table or column " + + "that already exists, restore the backup taken before the upgrade; a migration without a rebuild rolls back whole"; + // With whether it recorded the schema history, which it does when it applied every change and, for a host that schema // apply serves, the history says none is lost. private async Task<(SchemaSyncReport Report, bool Recorded)> SyncAsync(SchemaModel target, CancellationToken cancellationToken) diff --git a/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs b/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs index 1e3f97b00..11ff0f2dc 100644 --- a/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs +++ b/src/FluentCMS.Data.PostgreSql/PostgreSqlDataProvider.cs @@ -43,6 +43,9 @@ public bool IsDeadlock(Exception exception) // Writers wait for the rows they write, not for the whole database. public bool IsBusy(Exception exception) => false; + // Npgsql's migrations lock is a lock of the history table, which ends with the transaction that took it. + public Task ReadMigrationsLockAsync(DbContext context, CancellationToken cancellationToken) => Task.FromResult(null); + public Expression> ContainsIgnoringCase(string fragment) { var pattern = LikePattern.Containing(fragment); diff --git a/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs b/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs index 0fb04f60e..73fac7dcd 100644 --- a/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs +++ b/src/FluentCMS.Data.SqlServer/SqlServerDataProvider.cs @@ -44,6 +44,9 @@ public bool IsDeadlock(Exception exception) // Writers wait for the rows they write, not for the whole database. public bool IsBusy(Exception exception) => false; + // EF's migrations lock is an application lock of the session, which ends with it. + public Task ReadMigrationsLockAsync(DbContext context, CancellationToken cancellationToken) => Task.FromResult(null); + // Under a case-insensitive collation of its own, so that a database whose default compares case does not change it. public Expression> ContainsIgnoringCase(string fragment) { diff --git a/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs b/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs index 318fe5953..9cea08536 100644 --- a/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs +++ b/src/FluentCMS.Data.Sqlite/SqliteDataProvider.cs @@ -5,6 +5,8 @@ using Microsoft.Data.Sqlite; using Microsoft.EntityFrameworkCore; using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Storage; using SQLitePCL; namespace FluentCMS.Data.Sqlite; @@ -72,6 +74,27 @@ public void ConfigureSystemMigrations(DbContextOptionsBuilder options) => .MigrationsHistoryTable(SystemDbContext.HistoryTable)) .AddInterceptors(_interceptors); + // EF inserts the row with id 1 to take its lock and deletes it to release it. A database that does not exist yet is left + // for EF to create, since opening it here would make an empty file that EF then takes for an existing database. + public async Task ReadMigrationsLockAsync(DbContext context, CancellationToken cancellationToken) + { + if (!await context.GetService().ExistsAsync(cancellationToken)) + { + return null; + } + + var tables = await context.Database + .SqlQueryRaw("SELECT count(*) AS \"Value\" FROM sqlite_master WHERE type = 'table' AND name = '__EFMigrationsLock'") + .ToListAsync(cancellationToken); + if (tables[0] == 0) + { + return null; + } + + var rows = await context.Database.SqlQueryRaw("SELECT \"Timestamp\" AS \"Value\" FROM \"__EFMigrationsLock\"").ToListAsync(cancellationToken); + return rows.FirstOrDefault(); + } + void IDatabaseFile.Snapshot(string destination) => _file.Snapshot(destination); bool IDatabaseFile.IsInUse() => _file.IsInUse(); diff --git a/tests/FluentCMS.Data.Tests/Migrations/MigrationLockTests.cs b/tests/FluentCMS.Data.Tests/Migrations/MigrationLockTests.cs new file mode 100644 index 000000000..029314c1d --- /dev/null +++ b/tests/FluentCMS.Data.Tests/Migrations/MigrationLockTests.cs @@ -0,0 +1,211 @@ +using System.Data.Common; +using System.Threading.Channels; +using FluentCMS.Core.Commands; +using FluentCMS.Core.Storage; +using FluentCMS.Data.Tests.Providers; +using FluentCMS.Data.Tests.Sqlite; +using Microsoft.EntityFrameworkCore; +using Microsoft.EntityFrameworkCore.Diagnostics; +using Microsoft.EntityFrameworkCore.Infrastructure; +using Microsoft.EntityFrameworkCore.Migrations; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Hosting; +using Microsoft.Extensions.Logging; +using Shouldly; +using Xunit; + +namespace FluentCMS.Data.Tests.Migrations; + +// EF locks a SQLite database for its migrations with the row of __EFMigrationsLock, which a process that is killed while it +// migrates leaves behind, and EF then waits for it without end (#3177). +public sealed class MigrationLockTests : IDisposable +{ + private const string Production = "Production"; + + private const string Because = "EF's migrations lock, a row on SQLite only"; + + private readonly SqliteFileDatabase _database = new(); + + private static CancellationToken Token => TestContext.Current.CancellationToken; + + public void Dispose() => _database.Dispose(); + + // Through EF itself and never released, as by a process killed while it migrates; the time EF wrote in the row. + private async Task LeaveLockAsync(IHost host) + { + await using var scope = host.Services.CreateAsyncScope(); + var context = scope.ServiceProvider.GetRequiredService(); + _ = await context.GetService().AcquireDatabaseLockAsync(Token); + return _database.ReadStrings("select \"Timestamp\" from \"__EFMigrationsLock\"").ShouldHaveSingleItem(); + } + + // A bound of its own, so that a wait without end fails the test as a cancellation rather than hanging it. + private static CancellationTokenSource Bound() + { + var bound = CancellationTokenSource.CreateLinkedTokenSource(Token); + bound.CancelAfter(TimeSpan.FromSeconds(60)); + return bound; + } + + // The runner's monotonic clock runs a thousand times fast, so the 15 minutes of its wait pass within a second or two. + private IHost FastHost(Dictionary? settings, Logs logs) + { + var clock = new FastMonotonicClock(null); + return _database.CreateHost(ArticleSchemas.Titled, Production, GeneratedMigrations.CompileFirst(ArticleSchemas.Titled), settings, services => + { + services.AddSingleton(clock); + services.AddSingleton(logs); + }); + } + + [SqliteFact(Because)] + public async Task AStartGivesUpOnALockRowLeftBehindAndNamesIt() + { + var logs = new Logs(); + using var host = FastHost(null, logs); + var written = await LeaveLockAsync(host); + using var bound = Bound(); + + var exception = await Should.ThrowAsync(host.StartAsync(bound.Token)); + + exception.Message.ShouldStartWith($"EF Core's migrations lock, the row of __EFMigrationsLock written at {written}, was not released within 15 minutes, so the system stream was not migrated;"); + exception.Message.ShouldContain("the time is when the lock was taken, and it is never renewed;"); + exception.Message.ShouldContain("check that no other host and no 'fluentcms migrate' runs on the database, then delete the row with DELETE FROM \"__EFMigrationsLock\""); + logs.Messages(9305).ShouldHaveSingleItem().ShouldStartWith($"The system stream waits for EF Core's migrations lock, the row of __EFMigrationsLock written at {written};"); + _database.Tables().ShouldNotContain("cms_role"); + } + + [SqliteFact(Because)] + public async Task MigrateGivesUpOnALockRowLeftBehindAndNamesIt() + { + using var host = FastHost(new() { ["FluentCms:Database:MigrateOnStartup"] = "false" }, new Logs()); + var written = await LeaveLockAsync(host); + using var bound = Bound(); + using var output = new StringWriter(); + using var error = new StringWriter(); + + var exitCode = await CommandRunner.RunAsync(host.Services, [CommandRunner.Prefix, "migrate"], output, error, bound.Token); + + exitCode.ShouldBe(1); + error.ToString().ShouldStartWith("FCMS3302: 'fluentcms migrate' failed: TimeoutException: "); + error.ToString().ShouldContain($"the row of __EFMigrationsLock written at {written}, was not released within 15 minutes"); + _database.Tables().ShouldNotContain("cms_role"); + } + + [SqliteFact(Because)] + public async Task AStartWaitsForTheRowToGoAndThenMigrates() + { + var reads = new LockReads(); + using var host = _database.CreateHost( + ArticleSchemas.Titled, + Production, + GeneratedMigrations.CompileFirst(ArticleSchemas.Titled), + configure: cms => cms.Services.ConfigureDbContext(options => options.AddInterceptors(reads))); + await LeaveLockAsync(host); + using var bound = Bound(); + + var start = host.StartAsync(bound.Token); + + // Two reads of the row: the start found the lock held, waited, and looked again. + await reads.NextAsync(2); + start.IsCompleted.ShouldBeFalse(); + _database.Execute("DELETE FROM \"__EFMigrationsLock\""); + + await start; + + _database.Tables().ShouldContain("cms_role"); + _database.ReadStrings("select count(*) from \"__EFMigrationsLock\"").ShouldBe(["0"]); + await host.StopAsync(Token); + } + + // The runner reads what is pending first, and takes the lock only for a stream that has something to apply. + [SqliteFact(Because)] + public async Task AStartWithNothingPendingLeavesTheRowAlone() + { + var migrations = GeneratedMigrations.CompileFirst(ArticleSchemas.Titled); + using (var first = _database.CreateHost(ArticleSchemas.Titled, Production, migrations)) + { + await first.StartAsync(Token); + await first.StopAsync(Token); + } + + using var host = _database.CreateHost(ArticleSchemas.Titled, Production, migrations); + var written = await LeaveLockAsync(host); + using var bound = Bound(); + + await host.StartAsync(bound.Token); + + _database.ReadStrings("select \"Timestamp\" from \"__EFMigrationsLock\"").ShouldBe([written]); + await host.StopAsync(Token); + } + + // The read of the lock leaves a database that does not exist yet to EF, which creates it in WAL mode, also for a host + // that keeps the rollback journal and so sets no journal mode itself. + [SqliteFact(Because)] + public async Task AFirstStartCreatesTheDatabaseInWalModeAsEfDoes() + { + using var database = new SqliteFileDatabase(writeAheadLog: false); + using var host = database.CreateHost(ArticleSchemas.Titled, Production, GeneratedMigrations.CompileFirst(ArticleSchemas.Titled)); + + await host.StartAsync(Token); + + database.ReadStrings("PRAGMA journal_mode").ShouldBe(["wal"]); + await host.StopAsync(Token); + } + + private sealed class Logs : ILoggerProvider, ILogger + { + private readonly List<(int EventId, string Message)> _messages = []; + + public List Messages(int eventId) + { + lock (_messages) + { + return [.. _messages.Where(message => message.EventId == eventId).Select(message => message.Message)]; + } + } + + public ILogger CreateLogger(string categoryName) => this; + + public void Dispose() + { + } + + public IDisposable? BeginScope(TState state) + where TState : notnull => null; + + public bool IsEnabled(LogLevel logLevel) => true; + + public void Log(LogLevel logLevel, EventId eventId, TState state, Exception? exception, Func formatter) + { + lock (_messages) + { + _messages.Add((eventId.Id, formatter(state, exception))); + } + } + } + + // Signals each read of the lock's row, which a start waiting for the lock repeats while the row is there. + private sealed class LockReads : DbCommandInterceptor + { + private readonly Channel _reads = Channel.CreateUnbounded(); + + public async Task NextAsync(int count) + { + for (var read = 0; read < count; read++) + { + await _reads.Reader.ReadAsync(Token).AsTask().WaitAsync(TimeSpan.FromSeconds(30), Token); + } + } + + public override ValueTask ReaderExecutedAsync(DbCommand command, CommandExecutedEventData eventData, DbDataReader result, CancellationToken cancellationToken = default) + { + if (command.CommandText.Contains("FROM \"__EFMigrationsLock\"", StringComparison.Ordinal)) + { + _reads.Writer.TryWrite(true); + } + + return ValueTask.FromResult(result); + } + } +} diff --git a/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs b/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs index 2de511b2b..b8f7b001d 100644 --- a/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs +++ b/tests/FluentCMS.Data.Tests/Sqlite/SqliteFileDatabase.cs @@ -64,6 +64,8 @@ private sealed class UnflushedSqliteProvider(SqliteDataProvider inner) : IDataPr public bool IsBusy(Exception exception) => inner.IsBusy(exception); + public Task ReadMigrationsLockAsync(DbContext context, CancellationToken cancellationToken) => inner.ReadMigrationsLockAsync(context, cancellationToken); + public Expression> ContainsIgnoringCase(string fragment) => inner.ContainsIgnoringCase(fragment); public string? FilePath => ((IDatabaseFile)inner).FilePath; From c0f6d5e61c3cf3caf856cbf4b5c38b92c9787111 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 02:58:19 -0500 Subject: [PATCH 4/8] Lock the shared JSON options of Results.Json before any request (#3194) Closes #3193 ## Cause `Results.Json(value, options)` builds a `JsonHttpResult`, and its constructor runs `if (!options.IsReadOnly) options.TypeInfoResolver ??= new DefaultJsonTypeInfoResolver();`. That is a check, a read and a set on the shared static options, with no lock. The options become read-only on first serialization: when a result executes, or on any `JsonSerializer` call with them. If one request reads a null resolver and another request then sets the resolver and locks the options before the first runs the setter, the setter throws `InvalidOperationException` and the request answers 500. One host is enough to hit this: two concurrent first requests share the static. The race is open once per process, until the options are locked. Reproduced in a test: on options made the same way, without the fix, 5,000 rounds of 8 threads failed 1, 0 and 4 times in three runs. ## Change The shared options handed to `Results.Json` are now locked with `MakeReadOnly(populateMissingResolver: true)` in a static constructor, so they are complete and read-only before any request has them: - `AdminJson.Options` - `SchemaEndpoints.Options` - `StartupReportJson.Options` - `MediaEndpoints.PresetJson` The repository has no source-generated JSON contexts, and `Results.Json` already gave these options the reflection resolver, so the JSON the API writes does not change. `AdminBodiesTests` pass, and a new test compares the text written with the locked options against the text written with options built as before. The other static options in `src` were already safe and are unchanged: - They go only to `JsonSerializer`, `JsonNode.ToJsonString`, `JsonElement.Deserialize` or `JsonDocument.Parse`. The serializer fills in a missing resolver through a field and never throws, so their first use is thread-safe. These are `TransferPackage.Json`, `BackupArchive.Json`, `JsonDepth.Serializer`, `PermissionConstraints`, `McpClientState`, `WebhookDispatcher`, `MediaTransfer`, `MediaJson`, `McpArguments`, `McpSiteResources`, `FluentCmsClient` and the CLI's `ManagementClient.Json`. - `SettingsJson` was already locked. - `JobRegistry` and `OutboxRegistration` use `JsonSerializerOptions.Web`, which is read-only. ## Dependencies None. ## Acceptance - [x] The options cannot be changed after the type is initialized, and a test that uses them from many threads at first use passes repeated - [x] No other static `JsonSerializerOptions` in `src` is completed lazily by a request ## Tests - `AdminJsonTests` asserts that the shared options are read-only and complete. It also runs the first-use race (5,000 rounds of 8 threads) on options locked the same way, and checks the written text. Ran 5 times. - `MediaPresetTests` asserts the same for `PresetJson`. - Also ran: the fast checks, `dotnet build -c Release -warnaserror`, the host's `ExitCriteriaTests` 3 times, and the integration tests for content, roles, identity, schema, the startup report and OpenAPI. --- .../Api/Management/AdminJson.cs | 4 + .../Api/Management/SchemaEndpoints.cs | 3 + .../Diagnostics/StartupReportService.cs | 3 + .../FluentCMS.Plugins.Media/MediaEndpoints.cs | 4 + .../Api/Management/AdminJsonTests.cs | 117 ++++++++++++++++++ .../MediaPresetTests.cs | 12 ++ 6 files changed, 143 insertions(+) create mode 100644 tests/FluentCMS.Core.Tests/Api/Management/AdminJsonTests.cs diff --git a/src/FluentCMS.Core/Api/Management/AdminJson.cs b/src/FluentCMS.Core/Api/Management/AdminJson.cs index d27b82a2c..7a2d24294 100644 --- a/src/FluentCMS.Core/Api/Management/AdminJson.cs +++ b/src/FluentCMS.Core/Api/Management/AdminJson.cs @@ -13,6 +13,10 @@ internal static class AdminJson Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) }, }; + // Results.Json sets the resolver of options that are not read-only yet, which throws if another request locks them + // in between, so they are locked before any request has them. + static AdminJson() => Options.MakeReadOnly(populateMissingResolver: true); + public static IResult Ok(object value, int statusCode = StatusCodes.Status200OK) => Results.Json(value, Options, statusCode: statusCode); public static IResult Items(IEnumerable items) => Results.Json(new ItemList(items), Options); diff --git a/src/FluentCMS.Core/Api/Management/SchemaEndpoints.cs b/src/FluentCMS.Core/Api/Management/SchemaEndpoints.cs index 6e2afc712..55ca8cc92 100644 --- a/src/FluentCMS.Core/Api/Management/SchemaEndpoints.cs +++ b/src/FluentCMS.Core/Api/Management/SchemaEndpoints.cs @@ -23,6 +23,9 @@ internal static class SchemaEndpoints DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, }; + // Locked before any request has them, as AdminJson's are. + static SchemaEndpoints() => Options.MakeReadOnly(populateMissingResolver: true); + public static IServiceCollection AddManagementSchemaApi(this IServiceCollection services) => services.AddFluentCmsEndpoints(FluentCmsEndpointGroup.Management, Map); diff --git a/src/FluentCMS.Core/Diagnostics/StartupReportService.cs b/src/FluentCMS.Core/Diagnostics/StartupReportService.cs index 1fc065750..61cc25843 100644 --- a/src/FluentCMS.Core/Diagnostics/StartupReportService.cs +++ b/src/FluentCMS.Core/Diagnostics/StartupReportService.cs @@ -21,6 +21,9 @@ internal sealed class StartupReportHolder internal static class StartupReportJson { public static readonly JsonSerializerOptions Options = new(JsonSerializerDefaults.Web); + + // Locked before the diagnostics endpoint hands them to Results.Json, as AdminJson's are. + static StartupReportJson() => Options.MakeReadOnly(populateMissingResolver: true); } // Runs after every StartingAsync, which is where the migrations are, so the report sees the database as the host will use it. diff --git a/src/plugins/FluentCMS.Plugins.Media/MediaEndpoints.cs b/src/plugins/FluentCMS.Plugins.Media/MediaEndpoints.cs index c659bea3e..ab9e02385 100644 --- a/src/plugins/FluentCMS.Plugins.Media/MediaEndpoints.cs +++ b/src/plugins/FluentCMS.Plugins.Media/MediaEndpoints.cs @@ -36,6 +36,10 @@ internal static class MediaEndpoints Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) }, }; + // Results.Json sets the resolver of options that are not read-only yet, which throws if another request locks them + // in between, so they are locked before any request has them. + static MediaEndpoints() => PresetJson.MakeReadOnly(populateMissingResolver: true); + public static RouteGroupBuilder MapMediaEndpoints(this RouteGroupBuilder group) { group.MapGet("", async (string? folder, bool? subfolders, string? type, string? tag, string? search, int? page, int? pageSize, MediaService media, CancellationToken cancellationToken) => diff --git a/tests/FluentCMS.Core.Tests/Api/Management/AdminJsonTests.cs b/tests/FluentCMS.Core.Tests/Api/Management/AdminJsonTests.cs new file mode 100644 index 000000000..5c9d69465 --- /dev/null +++ b/tests/FluentCMS.Core.Tests/Api/Management/AdminJsonTests.cs @@ -0,0 +1,117 @@ +using System.Reflection; +using System.Text; +using System.Text.Json; +using System.Text.Json.Serialization; +using FluentCMS.Core.Api.Management; +using FluentCMS.Core.Diagnostics; +using Microsoft.AspNetCore.Http; +using Microsoft.Extensions.DependencyInjection; +using Shouldly; +using Xunit; + +namespace FluentCMS.Core.Tests.Api.Management; + +// Results.Json gives options that are not read-only the reflection resolver when it is handed them, so a request that locks +// shared options between that check and the set answered 500 (#3193). First use happens once a process, so the race is run +// on options made as the shared ones are, a new set each round: half the threads hand them to Results.Json and half +// serialize with them, which locks them as a request that has got further does. +public sealed class AdminJsonTests +{ + private const int Rounds = 5000; + private const int Threads = 8; + + private static readonly ItemList Value = new([new Shape("Square", ShapeKind.FourSided)]); + + public static TheoryData Shared => new() { "admin", "schema", "startup report" }; + + [Theory] + [MemberData(nameof(Shared))] + public void SharedOptionsAreCompleteAndReadOnlyOnceTheTypeIsInitialized(string name) + { + var options = Options(name); + + options.IsReadOnly.ShouldBeTrue(); + options.TypeInfoResolver.ShouldNotBeNull(); + } + + [Fact] + public void OptionsMadeReadOnlyAsTheSharedOnesTakeConcurrentFirstUses() + { + FirstUseFailures(() => + { + var options = AsAdminJsonMakesThem(); + options.MakeReadOnly(populateMissingResolver: true); + return options; + }).ShouldBe(0); + } + + [Fact] + public async Task TheReadOnlyOptionsWriteTheSameText() + { + (await WriteAsync(AdminJson.Options)).ShouldBe(await WriteAsync(AsAdminJsonMakesThem())); + } + + private static JsonSerializerOptions Options(string name) => name switch + { + "admin" => AdminJson.Options, + "schema" => (JsonSerializerOptions)typeof(SchemaEndpoints).GetField("Options", BindingFlags.NonPublic | BindingFlags.Static)!.GetValue(null)!, + _ => StartupReportJson.Options, + }; + + private static JsonSerializerOptions AsAdminJsonMakesThem() => new(JsonSerializerDefaults.Web) + { + Converters = { new JsonStringEnumConverter(JsonNamingPolicy.CamelCase) }, + }; + + private static int FirstUseFailures(Func create) + { + var options = create(); + var failures = 0; + using var round = new Barrier(Threads, _ => options = create()); + + var threads = Enumerable.Range(0, Threads).Select(index => new Thread(() => + { + for (var i = 0; i < Rounds; i++) + { + round.SignalAndWait(); + try + { + if (index % 2 == 0) + { + Results.Json(Value, options); + } + else + { + JsonSerializer.Serialize(Value, options); + } + } + catch (InvalidOperationException) + { + Interlocked.Increment(ref failures); + } + } + })).ToList(); + + threads.ForEach(thread => thread.Start()); + threads.ForEach(thread => thread.Join()); + return failures; + } + + private static async Task WriteAsync(JsonSerializerOptions options) + { + var context = new DefaultHttpContext { RequestServices = new ServiceCollection().AddLogging().BuildServiceProvider() }; + using var body = new MemoryStream(); + context.Response.Body = body; + + await Results.Json(Value, options).ExecuteAsync(context); + + return Encoding.UTF8.GetString(body.ToArray()); + } + + public sealed record Shape(string Name, ShapeKind Kind); + + public enum ShapeKind + { + FourSided, + } +} diff --git a/tests/FluentCMS.Plugins.Media.Tests/MediaPresetTests.cs b/tests/FluentCMS.Plugins.Media.Tests/MediaPresetTests.cs index dec04ef48..d20d28885 100644 --- a/tests/FluentCMS.Plugins.Media.Tests/MediaPresetTests.cs +++ b/tests/FluentCMS.Plugins.Media.Tests/MediaPresetTests.cs @@ -1,5 +1,7 @@ using System.Net; using System.Net.Http.Json; +using System.Reflection; +using System.Text.Json; using FluentCMS.Media; using Shouldly; using Xunit; @@ -12,6 +14,16 @@ public sealed class MediaPresetTests(MediaHost media) : IClassFixture { private const string Presets = $"{Root}/presets"; + // Results.Json would set the resolver of options that are not read-only, racing a request that locks them (#3193). + [Fact] + public void TheJsonOfPresetsIsCompleteAndReadOnlyBeforeAnyRequest() + { + var options = (JsonSerializerOptions)typeof(MediaEndpoints).GetField("PresetJson", BindingFlags.NonPublic | BindingFlags.Static)!.GetValue(null)!; + + options.IsReadOnly.ShouldBeTrue(); + options.TypeInfoResolver.ShouldNotBeNull(); + } + [Fact] public async Task APresetIsStoredListedAppliedByAUrlAndDeleted() { From fe3fbc4cf62fa8570271658ababa0aad6669d0a7 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 05:47:18 -0500 Subject: [PATCH 5/8] Deliver the grant's eviction before the shared-store reads (#3208) Closes #3201 It was the test, not the product. `SharingAsync` starts three hosts on one output cache store. The first adds a grant to the default site's Public role. `RoleVersionInterceptor` stages a `RolePermissionsChangedEvent` for it and kicks the outbox. These hosts kept their outbox worker, so `OutputCacheEviction` evicted `site:{default site}` from the shared store whenever the worker got to it. Every host has the same default site id, so that tag reaches the copies of all three. Usually the eviction landed while the hosts started. On a slow runner it landed after `firstClient` stored its copy and before `sameClient` read it. `same` then answered from its own database, where the Public role has no `content.note.read`, and gave 404. That is `shared.Status.ShouldBe(OK)`. The release build reports it at lines 366/371. Evidence: with the worker off and the first host's outbox drained right after its read, the test fails every time with `shared.Status should be HttpStatusCode.OK but was NotFound`, from the same frames as the CI failure. The cache served no copy across schemas, and no host answered 404 for a copy it had. #3194 is unrelated. - The three hosts of `AStoreSharedWithAnotherSchemaServesItNoCopyOfThisOne` run without an outbox worker, as `DeliveryCacheFixture` does. The first host drains its outbox after the grant, so the eviction is done before the reads. The test still shows that the host of the other schema is served nothing. - `DeliveryHostSiteTests` had the same exposure: a seeded grant with a live worker, then a read that must come from the cache. Its host now runs without an outbox worker too. The other Delivery hosts that assert cache hits already have no worker or have the output cache off. ## Dependencies None. ## Acceptance - [x] The cause is named in this issue - [x] The test passes repeated, alone and under load, and a product defect, if there is one, has its own test (no product defect) --- .../Delivery/DeliveryCacheTests.cs | 20 ++++++++++++++----- .../Delivery/DeliveryHostSiteTests.cs | 3 +++ 2 files changed, 18 insertions(+), 5 deletions(-) diff --git a/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryCacheTests.cs b/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryCacheTests.cs index 8714df524..485324b1b 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryCacheTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryCacheTests.cs @@ -434,14 +434,24 @@ public async Task AnErrorIsNotCachedAndSaysNoStore() private static async Task SharingAsync(IOutputCacheStore store, bool other, bool grant) { - var host = new FluentCmsTestHost().WithServices(services => services.AddSingleton(store)); + // No worker drains the outbox: the grant evicts the default site, whose tag every host of the store shares, so it is + // delivered here rather than between the reads, where it would evict the first host's copy before the second reads it. + var host = new FluentCmsTestHost() + .WithServices(services => services.AddSingleton(store)) + .Configure(cms => cms.Jobs.Options.Queues[OutboxDrainJob.Queue] = 0); host = other ? host.WithContentTypes() : host.WithContentTypes(); if (grant) { - await using var scope = host.Services.CreateAsyncScope(); - var context = scope.ServiceProvider.GetRequiredService(); - context.Set().Add(new RoleGrant { Id = Guid.CreateVersion7(), RoleId = DefaultSite.PublicRole, Permission = "content.note.read" }); - await context.SaveChangesAsync(Token); + await using (var scope = host.Services.CreateAsyncScope()) + { + var context = scope.ServiceProvider.GetRequiredService(); + context.Set().Add(new RoleGrant { Id = Guid.CreateVersion7(), RoleId = DefaultSite.PublicRole, Permission = "content.note.read" }); + await context.SaveChangesAsync(Token); + } + + while (await host.Services.GetRequiredService().DrainAsync(Token) > 0) + { + } } return host; diff --git a/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryHostSiteTests.cs b/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryHostSiteTests.cs index 391170b17..4b00030b9 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryHostSiteTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Delivery/DeliveryHostSiteTests.cs @@ -1,4 +1,5 @@ using System.Net; +using FluentCMS.Core.Events; using FluentCMS.Core.Sites; using FluentCMS.Core.Storage; using FluentCMS.Core.Storage.Entities; @@ -24,6 +25,8 @@ public DeliveryHostSiteTests() _host = new FluentCmsTestHost() .WithContentTypes() .WithServices(services => services.AddSingleton(_hosts)) + // The seeded grant evicts the site once its outbox row is delivered, which a worker could do between the two reads. + .Configure(cms => cms.Jobs.Options.Queues[OutboxDrainJob.Queue] = 0) .Seed(async services => { var context = services.GetRequiredService(); From 60edb85a3574427e83c97361fa74f8b8790ce17b Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 06:29:43 -0500 Subject: [PATCH 6/8] Roll back a take that lost the first insert of a lock name (#3210) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Closes #3200 Since #3182 the take of the row lock runs in an explicit transaction. On the first use of a name, takes that start together race on the insert of its row. A loser's insert fails with a unique violation, and in `catch (DbUpdateException)` it called `ExistsAsync`, which reads `cms_lock` on another connection while the loser's own transaction is still open. On the in-memory SQLite database of `FluentCmsTestHost` (shared cache), that open transaction holds the write lock, so the read got `SQLITE_LOCKED` and retried for the 30-second command timeout. Meanwhile other takes overshot their waits and the holder's own write failed with `database table is locked`, which was the 500 of the plugin template's `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` in the `Templates` job. The take now rolls its transaction back before it reads the row, as `SaveChangesAsync` did before #3182. The backend spec's "Multi-instance as built" paragraph gets one sentence on it. The preview release notes do not describe the lock change of #3182, so they are unchanged. What I checked in `DatabaseDistributedLock`, by reading it: - No other path reads or writes on another connection or scope while a take's or a renewal's transaction is open. The `ExecuteUpdateAsync` path of the take and `RenewAsync`'s update run on the transaction's own context. When either throws, `await using` disposes the transaction (a rollback) before the caller's `catch`, and `IsBusy` only inspects the exception. `RenewAsync`'s fallback read runs after its transaction is disposed, on the same context. `ReleaseAsync` opens no explicit transaction, and the handle waits for its renewal loop to end before it releases. `TryAcquireAsync` only waits after a null take and touches no database. - PostgreSQL: a failed statement puts the transaction block in an aborted state. The docs give `ROLLBACK TO` a savepoint, "short of rolling it back completely", as the way out of it ([tutorial: transactions](https://www.postgresql.org/docs/current/tutorial-transactions.html)), so the rollback is allowed and needed, and a read in the same transaction would fail. - SQL Server: with `SET XACT_ABORT OFF`, the default and not changed anywhere in the code, a failed statement can roll back only that statement and leave the transaction open ([SET XACT_ABORT](https://learn.microsoft.com/sql/t-sql/statements/set-xact-abort-transact-sql)), so `Rollback` ends a pending transaction. `SqlTransaction.Rollback` throws `InvalidOperationException` only when the server already rolled the transaction back ([SqlTransaction.Rollback](https://learn.microsoft.com/dotnet/api/system.data.sqlclient.sqltransaction.rollback)), which a unique violation does not do with `XACT_ABORT OFF`. Measured by the diagnosing lane (the generated `Acme.ReviewNotes` test, 20 runs per build under load): | Build | Failed (500) | Race | 30 s stall | Most runs took | | --- | --- | --- | --- | --- | | 7b6cd7e2, before #3182 | 0 / 20 | 18 | 0 | 10–25 s | | 7a085f3a (#3182) | 9 / 20 | 18 | 18 | 37–41 s | | c0f6d5e6 (tip of `main`) | 7 / 20 | 19 | 19 | 37–54 s | | tip plus the rollback | 0 / 20 | 20 | 0 | 8–14 s | Measured here: | Run | Result | | --- | --- | | New test `TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout`, without the rollback | 30 / 30 failed, all on the elapsed time (test took 36–66 s) | | The same test with the rollback | 0 / 30 failed (each process 8–12 s, host start included) | | `DistributedLockTests` with the rollback, SQLite | 15 / 15 passed | | `FluentCMS.Data.Tests` `Migrations` namespace with the rollback, PostgreSQL 15 | 143 passed, 46 skipped, 0 failed | | `Acme.ReviewNotes` `PostsSentTogetherAddNoMoreNotesThanTheSettingsAllow` from packages of this branch, as the `Templates` job makes it | 0 / 20 failed, 6.7–15.5 s per run | The new test runs ten takes with a 500 ms wait on a name never used before, on the in-memory database. Exactly one holds the lock: the lease is 30 seconds on a wall clock that does not move, and nobody releases before the count. ## Dependencies None. #3195 follows this pull request and merges `main` after it. ## Acceptance - [x] The cause is named in this issue, with the host's exception - [ ] The test passes repeated on `main`, in CI's `Templates` job, and the fix has a test in the repository's own tests if the cause is in the product (the regression test is in `DistributedLockTests`, and the generated test passed 20 runs locally; CI's `Templates` job on `main` is not verified yet) - [x] If the row lock changed what a busy SQLite database answers to a request, the specification says what it answers now --- docs/planning/backend-technical-spec.md | 2 +- .../Caching/DatabaseDistributedLock.cs | 4 +++- .../Caching/DistributedLockTests.cs | 19 +++++++++++++++++++ 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/docs/planning/backend-technical-spec.md b/docs/planning/backend-technical-spec.md index 77e5ff5eb..413c969ad 100644 --- a/docs/planning/backend-technical-spec.md +++ b/docs/planning/backend-technical-spec.md @@ -1169,7 +1169,7 @@ Rows 10.3, 10.5 to 10.8, 10.11 to 10.15. **Multi-instance.** Stateless API; Data Protection and signing keys persisted to the database by default; jobs lease through the database; `IDistributedLock` defaults to a `cms_lock` table and gets a Redis provider in `FluentCMS.Caching.Redis`; output cache in memory by default or Redis; media on shared storage. -**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. SQLite has one writer, so while another connection keeps a write transaction open, such as the schema apply or schema sync that holds the lock, a renewal waits for it and gives up after the busy timeout and the retries of Microsoft.Data.Sqlite, about 35 seconds, and so does a take; in the rollback journal mode a read of the row can wait too. Such a command, which `IDataProvider.IsBusy` recognises (SQLite's busy and locked errors by their primary code, never on PostgreSQL or SQL Server), changed nothing: a waiter counts the lock as held and ends with the lock or its timeout, which it can overshoot by one such wait, and the holder reads the row, which in WAL mode does not wait, and keeps the lock while the row still names it, renewing again at once and logging once that it waits (9165). A take and a renewal each run in a transaction of their own and read the clock once it has begun, so on SQLite, where beginning it waits for the writer, a lease starts when the database let the write in, not when it was asked for; on PostgreSQL and SQL Server the clock is still read before a statement that can wait on the row, as before. Before, a holder whose transaction lasted longer than about 45 seconds lost the lock, which canceled the rest of its work and rolled the transaction back, and a waiter failed with `SqliteException` ([#3176](https://github.com/fluentcms/FluentCMS/issues/3176)). One race remains: a waiter that found the lease over while such a transaction lasted waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other holder and cancels `Lost`, but a holder with more than one unit of work under the lock may have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. With a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 measured rounds, and the holder's `Lost` fired 64 to 119 milliseconds after the commit; when the holder began its next write at once, the waiter took it in none of 16. Known and left to [#3186](https://github.com/fluentcms/FluentCMS/issues/3186): a `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again, which records the history; and the waiter's event 9160 then says that a holder stopped without releasing the lock while that holder is alive. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. +**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. SQLite has one writer, so while another connection keeps a write transaction open, such as the schema apply or schema sync that holds the lock, a renewal waits for it and gives up after the busy timeout and the retries of Microsoft.Data.Sqlite, about 35 seconds, and so does a take; in the rollback journal mode a read of the row can wait too. Such a command, which `IDataProvider.IsBusy` recognises (SQLite's busy and locked errors by their primary code, never on PostgreSQL or SQL Server), changed nothing: a waiter counts the lock as held and ends with the lock or its timeout, which it can overshoot by one such wait, and the holder reads the row, which in WAL mode does not wait, and keeps the lock while the row still names it, renewing again at once and logging once that it waits (9165). A take and a renewal each run in a transaction of their own and read the clock once it has begun, so on SQLite, where beginning it waits for the writer, a lease starts when the database let the write in, not when it was asked for; on PostgreSQL and SQL Server the clock is still read before a statement that can wait on the row, as before. A take that loses the insert of a name's first row rolls its transaction back before it reads the row, since a read on another connection of the in-memory database waits for it ([#3200](https://github.com/fluentcms/FluentCMS/issues/3200)). Before, a holder whose transaction lasted longer than about 45 seconds lost the lock, which canceled the rest of its work and rolled the transaction back, and a waiter failed with `SqliteException` ([#3176](https://github.com/fluentcms/FluentCMS/issues/3176)). One race remains: a waiter that found the lease over while such a transaction lasted waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other holder and cancels `Lost`, but a holder with more than one unit of work under the lock may have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. With a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 measured rounds, and the holder's `Lost` fired 64 to 119 milliseconds after the commit; when the holder began its next write at once, the waiter took it in none of 16. Known and left to [#3186](https://github.com/fluentcms/FluentCMS/issues/3186): a `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again, which records the history; and the waiter's event 9160 then says that a holder stopped without releasing the lock while that holder is alive. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. **Reverse proxy.** Forwarded headers, `PathBase` option for sub-path hosting, HTTPS redirection and HSTS options; documented for Nginx, Caddy, YARP, Azure Front Door and Cloudflare. diff --git a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs index 5726db84d..61b260b15 100644 --- a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs +++ b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs @@ -92,7 +92,9 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP } catch (DbUpdateException) { - // Another holder made the row first. Only the first use of a name inserts, so this is rare. + // Another holder made the row first. Only the first use of a name inserts, so this is rare. The open + // transaction holds the write lock, which a read on another connection of the in-memory database waits for. + await transaction.RollbackAsync(CancellationToken.None); if (await ExistsAsync(name)) { return null; diff --git a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs index 7e1e3cc46..431e0ed72 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs @@ -333,6 +333,25 @@ public async Task ABadNameOrTimeoutIsRefused() await Should.ThrowAsync(() => locks.TryAcquireAsync(Name, TimeSpan.FromSeconds(-2), Token)); } + // The first use of a name inserts its row, so takes that start together race on the insert, and a loser reads the row + // on another connection. On the in-memory database that read waited for the loser's own transaction, for 30 seconds. + [Fact] + public async Task TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout() + { + await using var host = new FluentCmsTestHost(); + var locks = host.Services.GetRequiredService(); + var watch = Stopwatch.StartNew(); + + var attempts = await Task.WhenAll(Enumerable.Range(0, 10).Select(_ => Task.Run(() => locks.TryAcquireAsync(Name, TimeSpan.FromMilliseconds(500), Token), Token))); + + watch.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(10)); + attempts.Count(held => held is not null).ShouldBe(1); + foreach (var held in attempts.OfType()) + { + await held.DisposeAsync(); + } + } + // The second instance starts after the first has migrated the database, as the instances of a deployment do. private (IDistributedLock First, IDistributedLock Second) BothInstances() { From f8ff09409080b7831ae7d62b0e9831aa742f5e04 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 07:53:57 -0500 Subject: [PATCH 7/8] Revoke the session's authorizations at end-session (#3199) Closes #3198 A logout cleared only the sign-in cookie, so a refresh token issued in that session could still be redeemed until it expired. A test reproduced this: after a confirmed logout the token endpoint still answered 200. With this change: - Every new sign-in cookie, of either pool, carries a server session id in its properties (`.fcms.server_session`), which its renewals keep. The same user signing in again on the login page over a live cookie keeps it. - The authorize endpoint issues the code of a client without a consent step under an ad-hoc authorization marked with that id (`fcms_server_session` in the authorization's properties, so no migration). The identity token carries the id as `sid`. - `UserSessions.EndAsync` revokes every valid authorization of the user marked with the session, with its tokens. The refresh tokens are refused at once. The access tokens are refused from the next request on, since the bearer scheme checks the token and authorization entries on every request. Where a session's authorizations now end: - The confirmed logout, or a logout with a hint of the session's own `sid`, whichever clients of the pool signed in through that session. - The member logout page. - Another user's sign-in over the cookie, which ends the first user's session in that browser. What ends the cookie alone and revokes nothing: - The sign-outs that one cross-site GET can cause: `prompt=login`, `max_age`, `AbsoluteSessionLifetime` and a second factor the user's role now requires, at the authorize endpoint, whose request carries nothing secret, and the SSO provider's remote sign-out, which nothing authenticates. These are open in #3203, #3204 and #3205. A refresh token without `amr=mfa` is still refused by the token endpoint once the role requires two-factor. - A hint without the cookie's `sid` goes to the logout page, which asks. This covers an identity token of the same user from another session, another device or an earlier sign-in, a third-party client's token (a consent belongs to no session), and one from a sign-in before this change. - A logout without `id_token_hint` goes to the confirmation page as before, and nothing is revoked until the confirmed POST with its antiforgery token. - A cancelled or unconfirmed logout changes nothing. - A request without a live cookie revokes nothing, a valid hint included. Such a GET returns to the client without asking only when its hint is an identity token signed by this server with its algorithm (an expired one, or one a first logout revoked, still counts), issued to the client its `client_id` names, and the `post_logout_redirect_uri` it asks for is registered for that client. Any other hint goes to the logout page. - The user's other sessions (another browser, another device) stay. - A third-party client's consent stays, with its tokens. - A cookie from before this change has no id, so its logout ends the cookie alone, as before. A browser logout therefore ends every client without a consent step that signed in through that browser, whatever `offline_access` it holds. A client that must outlive a browser logout is registered with `RequireConsent`. The specification and the release notes say this. The response is unchanged, so it does not show whether a session existed. Sign-outs write no activity row, and the revocation adds none (#3207). `FluentCMS.Testing` builds its cookies through `CmsSignInManager.MarkSignIn`, the same path as the product. Dependencies: none. The check of a hint's signature uses `Microsoft.IdentityModel.JsonWebTokens`, which OpenIddict already brings. - [x] After `/connect/logout` for a session, a refresh token issued under that session's authorization is refused, in a test - [x] The specification says what end-session revokes and what it leaves - [ ] The decision record of the admin's sign-in (0007) no longer names an unrevoked token after a completed logout as a known gap, or says what remains (the record is on `feature/a1`, not on `main`; it is updated there after this merges) --- docs/planning/backend-technical-spec.md | 4 +- docs/releases/1.0.0-preview.1.md | 1 + .../Api/Account/MemberPageModels.cs | 13 +- .../Identity/CmsSignInManager.cs | 61 ++- .../Identity/Members/MemberConnect.cs | 9 + .../Identity/Members/MemberSignInManager.cs | 19 +- .../Identity/Oidc/ConnectEndpoints.cs | 94 ++++- .../Identity/Oidc/TokenClaims.cs | 9 +- .../Identity/Oidc/UserSessions.cs | 55 +++ src/FluentCMS.Testing/OAuthFlows.cs | 11 +- .../AuthorizationServerTests.Hardening.cs | 52 ++- .../AuthorizationServerTests.Sessions.cs | 2 +- .../Identity/HostedPagesBrowser.cs | 3 + .../Identity/HostedPagesTests.Logout.cs | 375 ++++++++++++++++++ .../Identity/HostedPagesTests.Sessions.cs | 6 + .../Members/MemberPagesTests.cs | 136 +++++++ 16 files changed, 807 insertions(+), 43 deletions(-) create mode 100644 tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Logout.cs diff --git a/docs/planning/backend-technical-spec.md b/docs/planning/backend-technical-spec.md index 413c969ad..ba29f086a 100644 --- a/docs/planning/backend-technical-spec.md +++ b/docs/planning/backend-technical-spec.md @@ -802,7 +802,7 @@ Rows 4.1, 4.2, 4.9 to 4.11, 4.13, 4.15, 4.16, 2.12, 2.13; decisions 7, 30, 33. **Endpoints.** OpenIddict defaults: `/connect/authorize`, `/connect/token`, `/connect/userinfo`, `/connect/introspect`, `/connect/revoke`, `/connect/logout`, `/.well-known/openid-configuration`. Flows: Authorization Code with PKCE for public clients (admin, MCP clients, CLI), rotating refresh tokens, client credentials for service accounts. Access tokens are JWTs signed with keys from a configured certificate or, when none is configured, a generated key persisted in `cms_signing_key` through Data Protection. Audiences: `fluentcms:management` for admins and keys, `fluentcms:members` for members. -**Authorization server as built.** OpenIddict 7 with its EF Core stores on the system model in `cms_oidc_application`, `cms_oidc_authorization`, `cms_oidc_scope` and `cms_oidc_token` (GUID keys, snake_case columns), and `cms_signing_key`. The endpoints are in the `Connect` route group, which takes the `auth` rate limit, the Management and the Members CORS origins and the security headers; discovery and the key set are at `/.well-known/openid-configuration` and `/.well-known/jwks` in a `WellKnown` group with the same origins. PKCE is required of every client, not only public ones, and only `S256` is offered. A refresh token rotates, and a used one is refused once OpenIddict's reuse leeway of 30 seconds is over, which lets two refreshes that cross, as two tabs of one application make, both succeed; a reuse after it revokes the session's tokens. A zero leeway was the first choice and was given up (#2854), since it signed users out on such a race. The token endpoint reads the user again for a code or a refresh token and refuses one who is deactivated or locked out, or whose security stamp, which codes and refresh tokens carry, has changed, so a password change or reset ends them. A bearer token is read from the `Authorization` header only. Access tokens are signed, not encrypted, JWTs with the audience `fluentcms:management`, and carry `sub`, `name`, `fcms_kind` (`AdminUser`, or `ServiceAccount` for client credentials, whose `sub` is the client's id), `scope`, `client_id` and, for a service account only, `roles`; a user's roles are read from `cms_user_role` on each request as before. `fluentcms:members` is the audience of a member's tokens, and the bearer scheme accepts both (members as built). Scopes are `openid`, `profile`, `email`, `offline_access`, `draft`, `management` and `mcp`, and narrow what roles allow as 4.8 says. The `FluentCms` scheme's bearer scheme is OpenIddict's validation scheme on the local server, which rewrites a valid token's principal to those claims and checks the token and its authorization in the database on each request, so a revoked session stops its access tokens at once. The request bridge sets `OperationContext.ClientId` from `client_id`, and a request from the client named by `cms.Clients.AdminClientId` (`fluentcms-admin` by default) runs with `Origin.Admin`. A refused credential keeps its handler's 401 code over HTTP (`FCMS3016` or `FCMS3019`), and `FCMS3102` otherwise. Tokens are signed with `FluentCms:Security:SigningCertificate` (a PKCS#12 file with an RSA key; `FCMS3022` when it cannot be read) or else with a generated RSA key kept in `cms_signing_key` and protected with Data Protection, which is what `ISigningKeyStatus` reports for `FCMS1102`; the symmetric key that encrypts codes and refresh tokens is always generated and kept there. The first key of each use has a fixed id, so instances that start together on an empty table insert one row and the others read it; the primary key settles that race with no wait, so it does not take `IDistributedLock` (4.15). The Data Protection key ring is in `cms_data_protection_key` under the application name `FluentCMS` unless the host persists it elsewhere. Clients are registered at start from `cms.Clients.Applications`, bound from `FluentCms:Clients:Applications:{clientId}` (`FCMS3021` for a bad one): a client with redirect URIs uses the code flow and refresh tokens, one with a secret is confidential and may use client credentials with the instance-wide roles it names, a listed client is replaced by its options at each start, one the options registered and no longer list is deleted with its sessions and tokens and a warning, unless the options list none at all, and others are left alone. Consent is implicit for these clients unless one sets `RequireConsent` (see the hosted pages as built). The authorize endpoint challenges the Identity application cookie, whose login page is `/account/login`, and answers `login_required` to `prompt=none`; logout ends that cookie on the POST of the logout page, which carries an antiforgery token, or with a valid `id_token_hint` of the user the cookie belongs to; any other request, a cross-site POST or a POST with a hint but without the cookie included, goes to the logout page, which asks, and a request without the cookie only returns to the client, with no expiring cookie (#2854). `UserSessions` lists a user's valid authorizations that still hold a token in use as sessions and revokes one with its tokens, for the Management API (#2786), and the `oidc.prune` job removes unusable tokens and authorizations after 14 days. The Data Protection ring is stored unencrypted unless the host adds an encryptor, which `IDataProtectionStatus` reports, and Production refuses to start without one (`FCMS1106`, `UnencryptedKeyRingCheck`) unless `FluentCms:Security:AllowUnencryptedKeyRing` is true. A host adds it with Data Protection's own builder, `AddDataProtection().ProtectKeysWithCertificate(...)` or `ProtectKeysWithAzureKeyVault(...)` or any other `IXmlEncryptor`, which it reaches from `IServiceCollection` like any host, so Core has no pass-through for it. Export and backup packages must never include `cms_data_protection_key`, `cms_signing_key` or `cms_oidc_*`. Codes `FCMS3021` and `FCMS3022`. +**Authorization server as built.** OpenIddict 7 with its EF Core stores on the system model in `cms_oidc_application`, `cms_oidc_authorization`, `cms_oidc_scope` and `cms_oidc_token` (GUID keys, snake_case columns), and `cms_signing_key`. The endpoints are in the `Connect` route group, which takes the `auth` rate limit, the Management and the Members CORS origins and the security headers; discovery and the key set are at `/.well-known/openid-configuration` and `/.well-known/jwks` in a `WellKnown` group with the same origins. PKCE is required of every client, not only public ones, and only `S256` is offered. A refresh token rotates, and a used one is refused once OpenIddict's reuse leeway of 30 seconds is over, which lets two refreshes that cross, as two tabs of one application make, both succeed; a reuse after it revokes the session's tokens. A zero leeway was the first choice and was given up (#2854), since it signed users out on such a race. The token endpoint reads the user again for a code or a refresh token and refuses one who is deactivated or locked out, or whose security stamp, which codes and refresh tokens carry, has changed, so a password change or reset ends them. A bearer token is read from the `Authorization` header only. Access tokens are signed, not encrypted, JWTs with the audience `fluentcms:management`, and carry `sub`, `name`, `fcms_kind` (`AdminUser`, or `ServiceAccount` for client credentials, whose `sub` is the client's id), `scope`, `client_id` and, for a service account only, `roles`; a user's roles are read from `cms_user_role` on each request as before. `fluentcms:members` is the audience of a member's tokens, and the bearer scheme accepts both (members as built). Scopes are `openid`, `profile`, `email`, `offline_access`, `draft`, `management` and `mcp`, and narrow what roles allow as 4.8 says. The `FluentCms` scheme's bearer scheme is OpenIddict's validation scheme on the local server, which rewrites a valid token's principal to those claims and checks the token and its authorization in the database on each request, so a revoked session stops its access tokens at once. The request bridge sets `OperationContext.ClientId` from `client_id`, and a request from the client named by `cms.Clients.AdminClientId` (`fluentcms-admin` by default) runs with `Origin.Admin`. A refused credential keeps its handler's 401 code over HTTP (`FCMS3016` or `FCMS3019`), and `FCMS3102` otherwise. Tokens are signed with `FluentCms:Security:SigningCertificate` (a PKCS#12 file with an RSA key; `FCMS3022` when it cannot be read) or else with a generated RSA key kept in `cms_signing_key` and protected with Data Protection, which is what `ISigningKeyStatus` reports for `FCMS1102`; the symmetric key that encrypts codes and refresh tokens is always generated and kept there. The first key of each use has a fixed id, so instances that start together on an empty table insert one row and the others read it; the primary key settles that race with no wait, so it does not take `IDistributedLock` (4.15). The Data Protection key ring is in `cms_data_protection_key` under the application name `FluentCMS` unless the host persists it elsewhere. Clients are registered at start from `cms.Clients.Applications`, bound from `FluentCms:Clients:Applications:{clientId}` (`FCMS3021` for a bad one): a client with redirect URIs uses the code flow and refresh tokens, one with a secret is confidential and may use client credentials with the instance-wide roles it names, a listed client is replaced by its options at each start, one the options registered and no longer list is deleted with its sessions and tokens and a warning, unless the options list none at all, and others are left alone. Consent is implicit for these clients unless one sets `RequireConsent` (see the hosted pages as built). The authorize endpoint challenges the Identity application cookie, whose login page is `/account/login`, and answers `login_required` to `prompt=none`; logout ends that cookie on the POST of the logout page, which carries an antiforgery token, or with a valid `id_token_hint` issued in the session of the cookie; any other request, a cross-site POST or a POST with a hint but without the cookie included, goes to the logout page, which asks, and a request without the cookie only returns to the client, with no expiring cookie (#2854). A logout ends what was issued in the session it ends ([#3198](https://github.com/fluentcms/FluentCMS/issues/3198)). Every new cookie carries the id of its server session in its properties (`.fcms.server_session`), which its renewals keep as they keep the time of the sign-in; a sign-in on the login page over a live cookie of the same user keeps that id, so the browser's session goes on. The authorize endpoint issues the code of a client without a consent step under an ad-hoc authorization marked with that id (the property `fcms_server_session`), and the identity token carries it as `sid`, as RP-Initiated Logout names it. `UserSessions` revokes every valid authorization of the user marked with a session, with its tokens, whichever clients of the pool signed in through it, where the user ends the session or where no request of another site can cause it: a confirmed logout or one with the session's own hint, the member logout page, and another user's sign-in over the cookie. Their refresh tokens are refused at once and their access tokens from the next request on, since the bearer scheme checks the token and its authorization on each request. The sign-outs that one cross-site GET can cause end the cookie alone and revoke nothing: `prompt=login`, `max_age`, `AbsoluteSessionLifetime` and a second factor that the user's role now requires and the sign-in did not give, at the authorize endpoint, since the request carries nothing secret, and the provider's remote sign-out (admin SSO as built), since nothing authenticates it; carrying the session across a re-authentication, where the access tokens of a user whose role now requires a second factor also live until they expire (#3203), authenticating the remote sign-out (#3204) and counting the absolute lifetime from the sign-in rather than the authorize request (#3205) are open. A hint skips the logout page only when its `sid` is the cookie's session: an identity token of the same user from another session, another device or an earlier sign-in, or one without `sid`, goes to the page, which asks. Nothing is revoked where no sign-in ends: a logout that is not confirmed, a confirmation the user does not give, and a request without the cookie, a valid hint included; such a GET returns to the client without asking when its hint is an identity token signed by this server, an expired one or one a first logout revoked included, issued to the client its `client_id` names, if it names one, and whose client registered the `post_logout_redirect_uri` it asks for, if it asks for one; any other hint goes to the logout page. The user's other sessions, in another browser or on another device, stay, as does the consent of a third-party client, a permanent authorization of no server session that ends on the account page and gives no `sid`. A browser logout therefore also ends every client without a consent step that signed in through that browser, a CLI, a native app or a first-party MCP client included, whatever `offline_access` it holds; a client that must outlive a browser logout is registered with `RequireConsent`. A cookie from before sessions were marked ends no authorization, and its tokens last as before. A sign-out writes no activity row, and the revocation adds none (#3207). `UserSessions` lists a user's valid authorizations that still hold a token in use as sessions and revokes one with its tokens, for the Management API (#2786), and the `oidc.prune` job removes unusable tokens and authorizations after 14 days. The Data Protection ring is stored unencrypted unless the host adds an encryptor, which `IDataProtectionStatus` reports, and Production refuses to start without one (`FCMS1106`, `UnencryptedKeyRingCheck`) unless `FluentCms:Security:AllowUnencryptedKeyRing` is true. A host adds it with Data Protection's own builder, `AddDataProtection().ProtectKeysWithCertificate(...)` or `ProtectKeysWithAzureKeyVault(...)` or any other `IXmlEncryptor`, which it reaches from `IServiceCollection` like any host, so Core has no pass-through for it. Export and backup packages must never include `cms_data_protection_key`, `cms_signing_key` or `cms_oidc_*`. Codes `FCMS3021` and `FCMS3022`. **Protected resources as built** ([#2977](https://github.com/fluentcms/FluentCMS/issues/2977)). A plugin's group mapped with `MapProtectedResource(prefix, options)` (`FluentCMS.Plugins.Abstractions`) is an OAuth protected resource of its own (RFC 9728), which is how the MCP plugin maps `/mcp` (5.4). The group gets what `MapGroup` gives (4.13), with four differences. Its operations enter with the origin the plugin names, `Api` or `Mcp`. It takes the Management rate limit and CORS policy, since it takes writes. No endpoint of it opts out of the signed-in caller with `AllowAnonymous()`. And a bearer token needs the scope the plugin names, one of the scopes that say which surface a token is for, which is `mcp` alone today. An API key holds no OAuth scope and is taken by its class, which the evaluator reads; a preview token carries the kind of the principal that minted it, a key's included, so it is told apart by its entry claim and refused as a token without the scope. A request without credentials is a 401 (`FCMS3102`) with `WWW-Authenticate: Bearer resource_metadata="{metadata URL}", scope="{scope}"`, one whose credentials are refused carries `error="invalid_token"` as well, and a token without the scope is a 403 (`FCMS3103`) with `Bearer error="insufficient_scope", scope="{scope}", resource_metadata="{metadata URL}"`. Both come before the per-site gate, as on any group of a plugin. Core serves the metadata, so a plugin still maps nothing under `/.well-known` (`FCMS1524`): `GET /.well-known/oauth-protected-resource{prefix}` answers `{ resource, authorization_servers, scopes_supported, bearer_methods_supported, resource_name }` to anyone, where `resource` is the public URL of the group, `authorization_servers` holds the issuer exactly as the discovery document names it, `scopes_supported` the scope of the resource and `bearer_methods_supported` `header`. `GET /.well-known/oauth-protected-resource`, the document a client tries when it does not know the path of the resource, answers the same while the host has one protected resource and 404 (`FCMS3104`) once it has several, since it could describe one of them only. The documents are endpoints of the `WellKnown` group, so they take its CORS origins and count as the `connect` surface, and they are not gated per site, since an anonymous caller learns nothing of where a plugin is off. The URLs are built from the public origin and the path base (4.15), and from the request when no origin is set, as the issuer is. Under a path base the documents are under it, where RFC 9728 would put the well-known segment before it; the challenge names the URL, which is what a client follows. The authorization server takes the `resource` parameter of RFC 8707 on the authorization request and on the token request when every value is the public URL of a protected resource of the host, compared as an absolute URI with the path as written, and refuses any other with `invalid_target`; like every refusal while the authorization request is checked, it is shown on the error page rather than sent to the client. OpenIddict's own check holds the resources in a list fixed in its options, which cannot name a URL that follows the request, so it is off (`DisableResourceValidation`) and a handler of core's stands in its place. The resource permissions of clients are off too (`IgnoreResourcePermissions`), because the parameter grants nothing: the audience of a token stays `fluentcms:management`, and the scope of the resource, which a client does need a permission for, is what holds a token to it. A token with the `mcp` scope alone manages through `Origin.Mcp` only (4.8) and is refused by the Management API (403). The consent page describes `mcp` as reading and changing content as an AI agent or tool, as far as the user's roles allow. A scope that is not one of those, an origin other than `Api` or `Mcp`, and a prefix mapped as a protected resource twice stop the host with `FCMS1527`; a prefix under a group of core or with a route parameter anywhere in it is `FCMS1524`. Code `FCMS1527`. @@ -846,7 +846,7 @@ Rows 4.1, 4.2, 4.9 to 4.11, 4.13, 4.15, 4.16, 2.12, 2.13; decisions 7, 30, 33. *Groups and permissions.* `cms_member_group (id, name, description, role_id, created_at)` and `cms_member_group_member (group_id, member_id)`. A group has a name of 1 to 64 lower-case letters, digits, `-` and `_`, unique in the instance, so that it is one name on every database and one word of the principal's groups, and is bound to one role, of the instance or of a site; a group always has a role, and one that only names its members for a row rule may be bound to a role without grants. The membership of such a group is managed with `members.manage` alone, in both directions (4.10), so a row rule that must hold against those callers, a negated one above all, reads a group whose role holds a grant they lack. A member holds the Public role of the site of the request, the site's own or else the seeded one as for a visitor, the built-in Member role, and the roles of its groups. The evaluator reads them with their stamps, and the names of the groups, in one query on every request, so a change holds from the next request; a member that is deactivated or deleted holds nothing, not even what a visitor reads, and nothing is taken from the token, whose roles or groups claim is never read. The principal a row rule sees carries the group names in the claim `fcms_groups`, and `Principal.IsInGroup(name)` (`FluentCMS.Content`, in the SDK) reads it, as in `ctx => perk => perk.Tier == (ctx.Principal.IsInGroup("gold") ? "gold" : "standard")`. Only that principal carries them, and `IsInGroup` throws on any other (4.8). A member's token has no scope that manages, so its grants give it published reads and nothing else, a group bound to a role with `*` included (4.8). A role is not deleted while a group is bound to it (`FCMS3044`, 4.10); the foreign key cascades, so a site that is deleted takes the groups bound to its roles along, as it does the API keys bound to them. -*Clients and tokens.* A client of `cms.Clients.Applications` names its pool with `Pool`, `Admins` (the default) or `Members`. A client of the member pool asks for `openid`, `profile`, `email` and `offline_access` only, names no roles, has no consent step, needs redirect URIs or `AllowDirectLogin`, and is not the admin application's client (`FCMS3021`); it may have a secret, and is never a service account: it holds neither the client credentials grant nor introspection. The authorize endpoint answers such a client from the member cookie and never from the admin's, challenging to `/account/members/login`, with `prompt`, `max_age` and the absolute session lifetime as for an admin user. Its tokens carry the audience `fluentcms:members`, `fcms_kind` `Member`, `sub`, `name` (the display name, or the address) and the scopes asked for among those four, whatever else was asked; they carry no roles and no groups. The token endpoint reads the member again for a code or a refresh token and refuses one that is deactivated, deleted, locked out or whose security stamp changed (`invalid_grant`). A code or refresh token says whose it is by its kind and is redeemed only through a client of the pool it was issued in, so a client the configuration moves to the other pool redeems nothing. The bearer scheme accepts both audiences, and a member's token is kept out of the Management API, of drafts and of `/mcp` by its scopes: it never holds `management`, `draft` or `mcp`. `/connect/userinfo` answers a member's token from the member. `/connect/logout` ends the sign-in of the pool of the client the request names, by its `client_id` or the audience of its `id_token_hint`, and the admin's when it names none; the two logout pages share an antiforgery token, so the member page marks its form (`fcms_pool`), neither page passes the mark on from a link, and the endpoint takes a confirmation only from the page of the pool it signs out. Deactivating or deleting a member, a reset and a change of password revoke its authorizations and every token of its subject (`MemberSessions`), so an access token stops on the next request, one of a direct login without a session included. `UserSessions` lists and ends a member's sessions for the Management API, since a member's id is the subject of its tokens. +*Clients and tokens.* A client of `cms.Clients.Applications` names its pool with `Pool`, `Admins` (the default) or `Members`. A client of the member pool asks for `openid`, `profile`, `email` and `offline_access` only, names no roles, has no consent step, needs redirect URIs or `AllowDirectLogin`, and is not the admin application's client (`FCMS3021`); it may have a secret, and is never a service account: it holds neither the client credentials grant nor introspection. The authorize endpoint answers such a client from the member cookie and never from the admin's, challenging to `/account/members/login`, with `prompt`, `max_age` and the absolute session lifetime as for an admin user. Its tokens carry the audience `fluentcms:members`, `fcms_kind` `Member`, `sub`, `name` (the display name, or the address) and the scopes asked for among those four, whatever else was asked; they carry no roles and no groups. The token endpoint reads the member again for a code or a refresh token and refuses one that is deactivated, deleted, locked out or whose security stamp changed (`invalid_grant`). A code or refresh token says whose it is by its kind and is redeemed only through a client of the pool it was issued in, so a client the configuration moves to the other pool redeems nothing. The bearer scheme accepts both audiences, and a member's token is kept out of the Management API, of drafts and of `/mcp` by its scopes: it never holds `management`, `draft` or `mcp`. `/connect/userinfo` answers a member's token from the member. `/connect/logout` ends the sign-in of the pool of the client the request names, by its `client_id` or the audience of its `id_token_hint`, and the admin's when it names none; the two logout pages share an antiforgery token, so the member page marks its form (`fcms_pool`), neither page passes the mark on from a link, and the endpoint takes a confirmation only from the page of the pool it signs out. The member logout page, which ends a member's sign-in itself when no client sent the member there, revokes the authorizations of its server session as the endpoint does. Deactivating or deleting a member, a reset and a change of password revoke its authorizations and every token of its subject (`MemberSessions`), so an access token stops on the next request, one of a direct login without a session included. `UserSessions` lists and ends a member's sessions for the Management API, since a member's id is the subject of its tokens. *Direct login* (Q4). `grant_type=member_password` with `username`, `password` and `client_id` at `/connect/token`, a custom grant of OpenIddict, for a client of the member pool with `AllowDirectLogin` while `cms.Members.AllowDirectLogin` is on; both are off by default. Every refusal is `invalid_grant` with one description: the option off, a client without the flag, an unknown address, a wrong password, an address that is not verified, a member that is deactivated or locked out. The reason is in the log (event 9075, with the client and the client address, never the address typed or the password), and an address or a password that was refused is a `member.login_failed` activity row. A wrong password counts towards the lockout, and the hash runs for an unknown address and for a member that may not sign in, as on the login page. Every client of the member pool holds the grant's permission, so that the endpoint answers; a client of the admin pool does not, and OpenIddict refuses it (`unauthorized_client`). The discovery document names the grant only while the option is on, and the startup report lists `member-direct-login` in Production as the less secure way in (4.15). diff --git a/docs/releases/1.0.0-preview.1.md b/docs/releases/1.0.0-preview.1.md index 8e05ca41b..648925381 100644 --- a/docs/releases/1.0.0-preview.1.md +++ b/docs/releases/1.0.0-preview.1.md @@ -70,6 +70,7 @@ A media field read with `populate`, and an image of rich text read with `richTex ## Bugs fixed - A setting that is not text, such as `maxOutputTokens` of AI assist, is no longer taken for a secret by its name (#3079). +- A logout revokes the authorizations issued in the browser's sign-in, with their refresh and access tokens, where it ended the sign-in cookie alone. The confirmed logout, one with an `id_token_hint` of that session and the member logout page revoke them, and so does another user's sign-in in the same browser. The sign-outs another site can cause with one GET, `prompt=login`, `max_age`, `AbsoluteSessionLifetime` and a second factor the role now requires at the authorize endpoint, and a provider's remote sign-out, end the cookie alone (#3203, #3204, #3205). An identity token carries the session as `sid`, and a logout whose hint lacks the session's `sid`, such as a third-party client's token or one from a sign-in before this preview, is asked to confirm, and a logout without the cookie returns to the client without asking only for an identity token of this server issued to that client. The user's other sessions and a third-party client's consent stay; a client without a consent step that must outlive a browser logout, such as a CLI or a native app, is registered with `RequireConsent` (#3198). - Restoring, reading and diffing a version written before a field was renamed with `[FormerNames]` keep the field's value, and a former name that another field of the type holds is refused (#3085). - A media reference read on the Delivery API resolves to its public asset, where it answered null (#3109). - A media reference read on `/graphql` by a caller that may not manage resolves to its public asset, where it answered null (#3112). diff --git a/src/FluentCMS.Core/Api/Account/MemberPageModels.cs b/src/FluentCMS.Core/Api/Account/MemberPageModels.cs index 29f677188..968b47f33 100644 --- a/src/FluentCMS.Core/Api/Account/MemberPageModels.cs +++ b/src/FluentCMS.Core/Api/Account/MemberPageModels.cs @@ -1,6 +1,7 @@ using System.ComponentModel.DataAnnotations; using FluentCMS.Core.Identity; using FluentCMS.Core.Identity.Members; +using FluentCMS.Core.Identity.Oidc; using Microsoft.AspNetCore.Authentication; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Mvc; @@ -121,7 +122,8 @@ public sealed class MemberLoginInput // Asks before signing out. With the request of a client, which /connect/logout sent here, the form posts it back there // with an antiforgery token and the mark of this page, so that the endpoint ends the member's sign-in and never the admin -// user's. Without one, the page ends the member's sign-in itself. There is no sign-out by GET. +// user's. Without one, the page ends the member's sign-in itself, with the authorizations of its server session, as the +// endpoint does. There is no sign-out by GET. public sealed class MemberLogoutModel : MemberPageModel { public string? Email { get; private set; } @@ -142,7 +144,14 @@ public async Task OnGetAsync() public async Task OnPostAsync() { - await Service().SignOutAsync(); + var signIn = Service(); + var cookie = await HttpContext.AuthenticateAsync(MemberAuthentication.CookieScheme); + if (cookie.Succeeded) + { + await Service().EndAsync(signIn.UserManager.GetUserId(cookie.Principal), CmsSignInManager.ServerSession(cookie.Properties), CancellationToken.None); + } + + await signIn.SignOutAsync(); return RedirectToPage(); } } diff --git a/src/FluentCMS.Core/Identity/CmsSignInManager.cs b/src/FluentCMS.Core/Identity/CmsSignInManager.cs index 25a3e773b..4e13d2eed 100644 --- a/src/FluentCMS.Core/Identity/CmsSignInManager.cs +++ b/src/FluentCMS.Core/Identity/CmsSignInManager.cs @@ -5,6 +5,7 @@ using Microsoft.AspNetCore.Authentication; using Microsoft.AspNetCore.Http; using Microsoft.AspNetCore.Identity; +using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using static OpenIddict.Abstractions.OpenIddictConstants; @@ -32,6 +33,10 @@ internal sealed class CmsSignInManager( // issue time: prompt=login, max_age and the absolute session lifetime read it. internal const string SignedInAtItem = ".fcms.signed_in_at"; + // The id of the server session, kept in the cookie's properties in the same way: the authorizations issued in it carry + // it, so that the logout of that session ends them (#3198). A cookie from before has none. + internal const string ServerSessionItem = ".fcms.server_session"; + public static DateTimeOffset? SignedInAt(AuthenticationProperties properties) { ArgumentNullException.ThrowIfNull(properties); @@ -41,6 +46,49 @@ internal sealed class CmsSignInManager( : properties.IssuedUtc; } + public static string? ServerSession(AuthenticationProperties properties) + { + ArgumentNullException.ThrowIfNull(properties); + + return properties.Items.TryGetValue(ServerSessionItem, out var value) && !string.IsNullOrEmpty(value) ? value : null; + } + + // Marks a new sign-in with its time and the id of its server session, unless the properties are a renewal's. + internal static void MarkSignIn(AuthenticationProperties properties, TimeProvider time) + { + ArgumentNullException.ThrowIfNull(properties); + ArgumentNullException.ThrowIfNull(time); + + properties.Items.TryAdd(SignedInAtItem, time.GetUtcNow().ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture)); + properties.Items.TryAdd(ServerSessionItem, Guid.NewGuid().ToString("N")); + } + + // A new sign-in over a live cookie of the pool: the same user's keeps the browser's server session, so that its logout + // still finds what was issued before; another user's ends that session, as a logout would. + internal static async Task TakeOverAsync(HttpContext context, string scheme, string userId, AuthenticationProperties properties, Func userIdOf) + { + if (properties.Items.ContainsKey(ServerSessionItem)) + { + return; + } + + var live = await context.AuthenticateAsync(scheme); + if (!live.Succeeded || ServerSession(live.Properties!) is not { } serverSession) + { + return; + } + + var previous = userIdOf(live.Principal!); + if (previous == userId) + { + properties.Items[ServerSessionItem] = serverSession; + } + else + { + await context.RequestServices.GetRequiredService().EndAsync(previous, serverSession, CancellationToken.None); + } + } + public override async Task CanSignInAsync(CmsUser user) { ArgumentNullException.ThrowIfNull(user); @@ -111,15 +159,14 @@ public override async Task CheckPasswordSignInAsync(CmsUser user, } // Every sign-in that makes a new cookie comes here; a renewal passes the cookie's properties, which keep the time. - public override Task SignInWithClaimsAsync(CmsUser user, AuthenticationProperties? authenticationProperties, IEnumerable additionalClaims) + public override async Task SignInWithClaimsAsync(CmsUser user, AuthenticationProperties? authenticationProperties, IEnumerable additionalClaims) { - var properties = authenticationProperties ?? new AuthenticationProperties(); - if (!properties.Items.ContainsKey(SignedInAtItem)) - { - properties.Items[SignedInAtItem] = time.GetUtcNow().ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture); - } + ArgumentNullException.ThrowIfNull(user); - return base.SignInWithClaimsAsync(user, properties, additionalClaims); + var properties = authenticationProperties ?? new AuthenticationProperties(); + await TakeOverAsync(Context, IdentityConstants.ApplicationScheme, user.Id.ToString("D"), properties, UserManager.GetUserId); + MarkSignIn(properties, time); + await base.SignInWithClaimsAsync(user, properties, additionalClaims); } public override async Task TwoFactorAuthenticatorSignInAsync(string code, bool isPersistent, bool rememberClient) diff --git a/src/FluentCMS.Core/Identity/Members/MemberConnect.cs b/src/FluentCMS.Core/Identity/Members/MemberConnect.cs index 26ba5cff5..b32144571 100644 --- a/src/FluentCMS.Core/Identity/Members/MemberConnect.cs +++ b/src/FluentCMS.Core/Identity/Members/MemberConnect.cs @@ -58,6 +58,7 @@ public static async Task AuthorizeAsync(HttpContext context, OpenIddict return ConnectEndpoints.Refuse(Errors.LoginRequired, "The sign-in is older than the client or the server accepts."); } + // The cookie alone ends: another site can send this request, so it revokes nothing (#3203). await signIn.SignOutAsync(); return Results.Challenge(new AuthenticationProperties { RedirectUri = afterSignIn }, [MemberAuthentication.CookieScheme]); } @@ -67,6 +68,14 @@ public static async Task AuthorizeAsync(HttpContext context, OpenIddict .SetClaim(Claims.AuthenticationTime, signedInAt.ToUnixTimeSeconds()) .SetClaim(TokenClaims.SessionStart, now.ToUnixTimeSeconds()); TokenClaims.ForMember(identity, member, await signIn.UserManager.GetSecurityStampAsync(member), request.GetScopes(), methods); + if (CmsSignInManager.ServerSession(cookie.Properties!) is { } serverSession + && await context.RequestServices.GetRequiredService().FindByClientIdAsync(request.ClientId!, context.RequestAborted) is { } application) + { + TokenClaims.ForServerSession(identity, serverSession); + identity.SetAuthorizationId(await context.RequestServices.GetRequiredService() + .BeginAsync(new ClaimsPrincipal(identity), application, serverSession, context.RequestAborted)); + } + return Results.SignIn(new ClaimsPrincipal(identity), properties: null, OpenIddictServerAspNetCoreDefaults.AuthenticationScheme); } diff --git a/src/FluentCMS.Core/Identity/Members/MemberSignInManager.cs b/src/FluentCMS.Core/Identity/Members/MemberSignInManager.cs index 1125ff210..f08e2ad0d 100644 --- a/src/FluentCMS.Core/Identity/Members/MemberSignInManager.cs +++ b/src/FluentCMS.Core/Identity/Members/MemberSignInManager.cs @@ -1,4 +1,3 @@ -using System.Globalization; using System.Security.Claims; using FluentCMS.Content; using FluentCMS.Events; @@ -122,17 +121,17 @@ public override async Task CheckPasswordSignInAsync(CmsMember user return await base.CheckPasswordSignInAsync(user, password, lockoutOnFailure); } - // Every sign-in that makes a new cookie comes here. The time of the sign-in is kept in the cookie's properties, as - // for an admin user: prompt=login, max_age and the absolute session lifetime read it. - public override Task SignInWithClaimsAsync(CmsMember user, AuthenticationProperties? authenticationProperties, IEnumerable additionalClaims) + // Every sign-in that makes a new cookie comes here. The time of the sign-in and the id of its server session are kept + // in the cookie's properties, as for an admin user: prompt=login, max_age and the absolute session lifetime read the + // time, and the logout ends the authorizations of the session. + public override async Task SignInWithClaimsAsync(CmsMember user, AuthenticationProperties? authenticationProperties, IEnumerable additionalClaims) { - var properties = authenticationProperties ?? new AuthenticationProperties(); - if (!properties.Items.ContainsKey(CmsSignInManager.SignedInAtItem)) - { - properties.Items[CmsSignInManager.SignedInAtItem] = _time.GetUtcNow().ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture); - } + ArgumentNullException.ThrowIfNull(user); - return base.SignInWithClaimsAsync(user, properties, additionalClaims); + var properties = authenticationProperties ?? new AuthenticationProperties(); + await CmsSignInManager.TakeOverAsync(Context, MemberAuthentication.CookieScheme, user.Id.ToString("D"), properties, _members.GetUserId); + CmsSignInManager.MarkSignIn(properties, _time); + await base.SignInWithClaimsAsync(user, properties, additionalClaims); } // The member cookie alone. Identity's sign-out also ends its external and two-factor cookies, which are the admin's. diff --git a/src/FluentCMS.Core/Identity/Oidc/ConnectEndpoints.cs b/src/FluentCMS.Core/Identity/Oidc/ConnectEndpoints.cs index cb2eb9008..6d6ce62c7 100644 --- a/src/FluentCMS.Core/Identity/Oidc/ConnectEndpoints.cs +++ b/src/FluentCMS.Core/Identity/Oidc/ConnectEndpoints.cs @@ -17,7 +17,10 @@ using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Options; using Microsoft.Extensions.Primitives; +using Microsoft.IdentityModel.JsonWebTokens; +using Microsoft.IdentityModel.Tokens; using OpenIddict.Abstractions; +using OpenIddict.Server; using OpenIddict.Server.AspNetCore; using static OpenIddict.Abstractions.OpenIddictConstants; @@ -115,6 +118,7 @@ private static async Task AuthorizeAsync(HttpContext context) return Refuse(Errors.LoginRequired, "The sign-in is older than the client or the server accepts."); } + // The cookie alone ends: another site can send this request, so it revokes nothing (#3203). await context.SignOutAsync(IdentityConstants.ApplicationScheme); return Results.Challenge(new AuthenticationProperties { RedirectUri = afterSignIn }, [IdentityConstants.ApplicationScheme]); } @@ -176,6 +180,12 @@ private static async Task AuthorizeAsync(HttpContext context) identity.SetAuthorizationId(await context.RequestServices.GetRequiredService().GetIdAsync(consent, context.RequestAborted)); } + else if (CmsSignInManager.ServerSession(cookie.Properties!) is { } serverSession) + { + TokenClaims.ForServerSession(identity, serverSession); + identity.SetAuthorizationId(await context.RequestServices.GetRequiredService() + .BeginAsync(new ClaimsPrincipal(identity), application, serverSession, context.RequestAborted)); + } return Results.SignIn(new ClaimsPrincipal(identity), properties: null, OpenIddictServerAspNetCoreDefaults.AuthenticationScheme); } @@ -323,8 +333,10 @@ private static async Task UserInfoAsync(HttpContext context) // Ends the user's sign-in on the hosted pages; the server then returns to the client's post-logout URI when it gave one, // or to the logout page. The user confirms on the logout page, whose form posts back with an antiforgery token, or the - // client sends a valid id_token_hint of the signed-in user. Anything else, a cross-site POST included, goes to the logout - // page, which asks. A request without the sign-in cookie has no sign-in to end, so it sends no expiring cookie either. + // client sends a valid id_token_hint issued in the signed-in user's server session. Anything else, a cross-site POST + // included, goes to the logout page, which asks. A request without the sign-in cookie has no sign-in to end, so it + // sends no expiring cookie either. + // Ending a sign-in revokes the authorizations issued in its server session, with their tokens, and no others. // The client the request names, by its client_id or by the audience of its hint, says whose sign-in ends: a client of // the member pool ends the member's on the member pages, and any other request the admin user's, so that neither // signs the other out. @@ -342,14 +354,14 @@ private static async Task LogoutAsync(HttpContext context) : (IdentityConstants.ApplicationScheme, HostedPagePaths.Logout); var cookie = await context.AuthenticateAsync(scheme); - if (!await ConfirmedAsync(context, members) && !HintsSignedInUser(context, hint, cookie, members)) + if (!await ConfirmedAsync(context, members) && !await HintsSignedInUserAsync(context, hint, cookie, members)) { return Results.Redirect(context.Request.PathBase + page + QueryString.Create(LogoutParameters(context))); } if (cookie.Succeeded) { - await context.SignOutAsync(scheme); + await EndSignInAsync(context, cookie, members); } return Results.SignOut( @@ -397,26 +409,80 @@ private static IResult ErrorPage(HttpContext context, OpenIddictResponse respons })); // OpenIddict passes a hint it cannot read through without a principal, so the hint is checked here. Without the cookie - // a GET, which a browser sends with it, has no one signed in; a POST may be a cross-site one, which never carries it. - // The cookie is the one of the pool the request is for, read by that pool's manager. - private static bool HintsSignedInUser(HttpContext context, AuthenticateResult hint, AuthenticateResult cookie, bool members) + // a GET, which a browser sends with it, has no one signed in, so an identity token of this server for the client the + // request names returns to it, an expired one or a repeated logout's, whose token the first one revoked, included; any + // other hint goes to the logout page. A POST may be a cross-site one, which never carries it. + // With the cookie, the hint must be of the cookie's server session (its sid), not only of its user: an id token from + // another session or device, or from an earlier sign-in, goes to the logout page. The cookie is the one of the pool the + // request is for, read by that pool's manager. + private static async Task HintsSignedInUserAsync(HttpContext context, AuthenticateResult hint, AuthenticateResult cookie, bool members) { - if (hint.Principal?.GetClaim(Claims.Subject) is not { } subject) + if (!cookie.Succeeded) + { + return HttpMethods.IsGet(context.Request.Method) && await IsOwnIdentityTokenAsync(context, ServerRequest(context).IdTokenHint); + } + + return hint.Principal?.GetClaim(Claims.Subject) is { } subject + && SignedInUserId(context, cookie, members) == subject + && CmsSignInManager.ServerSession(cookie.Properties!) is { } serverSession + && hint.Principal.GetClaim(TokenClaims.ServerSession) == serverSession; + } + + // An identity token signed with a key of this server, whatever its lifetime or its entry's status, of the client the + // request names and whose post-logout URI it asks for. OpenIddict checks those only on a hint it can read. + private static async Task IsOwnIdentityTokenAsync(HttpContext context, string? token) + { + if (string.IsNullOrEmpty(token)) { return false; } - if (!cookie.Succeeded) + var options = context.RequestServices.GetRequiredService>().CurrentValue; + var result = await new JsonWebTokenHandler().ValidateTokenAsync(token, new TokenValidationParameters + { + IssuerSigningKeys = [.. options.SigningCredentials.Select(credentials => credentials.Key)], + ValidAlgorithms = [.. options.SigningCredentials.Select(credentials => credentials.Algorithm).Distinct(StringComparer.Ordinal)], + ValidTypes = [JwtConstants.HeaderType], + ValidateAudience = false, + ValidateIssuer = false, + ValidateLifetime = false, + }); + if (!result.IsValid || result.SecurityToken is not JsonWebToken identity) + { + return false; + } + + var audiences = identity.Audiences.ToList(); + var clientId = identity.TryGetPayloadValue(Claims.AuthorizedParty, out var party) ? party : audiences.Count == 1 ? audiences[0] : null; + var request = ServerRequest(context); + if (clientId is null || (request.ClientId is { } named && named != clientId)) + { + return false; + } + + if (string.IsNullOrEmpty(request.PostLogoutRedirectUri)) { - return HttpMethods.IsGet(context.Request.Method); + return true; } - var signedIn = members - ? context.RequestServices.GetRequiredService().GetUserId(cookie.Principal) - : context.RequestServices.GetRequiredService>().GetUserId(cookie.Principal); - return signedIn == subject; + var applications = context.RequestServices.GetRequiredService(); + return await applications.FindByClientIdAsync(clientId, context.RequestAborted) is { } application + && await applications.ValidatePostLogoutRedirectUriAsync(application, request.PostLogoutRedirectUri, context.RequestAborted); + } + + // Ends the cookie's sign-in and, with it, the authorizations of its server session, so that nothing issued in a + // session outlives its cookie; an end the browser stops waiting for still ends them. + private static async Task EndSignInAsync(HttpContext context, AuthenticateResult cookie, bool members) + { + await context.RequestServices.GetRequiredService() + .EndAsync(SignedInUserId(context, cookie, members), CmsSignInManager.ServerSession(cookie.Properties!), CancellationToken.None); + await context.SignOutAsync(members ? MemberAuthentication.CookieScheme : IdentityConstants.ApplicationScheme); } + private static string? SignedInUserId(HttpContext context, AuthenticateResult cookie, bool members) => members + ? context.RequestServices.GetRequiredService().GetUserId(cookie.Principal!) + : context.RequestServices.GetRequiredService>().GetUserId(cookie.Principal!); + // A new sign-in answers prompt=login and max_age, so the request it comes back with no longer asks for one. internal static IEnumerable> WithoutReauthentication(List> parameters) { diff --git a/src/FluentCMS.Core/Identity/Oidc/TokenClaims.cs b/src/FluentCMS.Core/Identity/Oidc/TokenClaims.cs index d70b1846d..3b9c9c3d6 100644 --- a/src/FluentCMS.Core/Identity/Oidc/TokenClaims.cs +++ b/src/FluentCMS.Core/Identity/Oidc/TokenClaims.cs @@ -21,6 +21,10 @@ internal static class TokenClaims // When the session began, in Unix seconds, for its absolute lifetime; in codes and refresh tokens only, too. public const string SessionStart = "fcms_session_start"; + // The server session, RP-Initiated Logout's sid, in the identity token of a code issued under it: a logout whose hint + // carries the sid of the cookie's session needs no confirmation. + public const string ServerSession = "sid"; + // The authentication method reference Identity gives a sign-in with a second factor. public const string MultiFactor = "mfa"; @@ -74,6 +78,9 @@ public static ClaimsIdentity ForMember(ClaimsIdentity identity, CmsMember member return identity; } + public static ClaimsIdentity ForServerSession(ClaimsIdentity identity, string serverSession) => + identity.SetClaim(ServerSession, serverSession).SetDestinations(Destinations); + public static ClaimsIdentity ForServiceAccount(ClaimsIdentity identity, Guid applicationId, string name, IEnumerable roles, IEnumerable scopes) { identity.SetClaim(Claims.Subject, applicationId.ToString("D")) @@ -125,7 +132,7 @@ public static bool SignedInWithSecondFactor(Principal principal) => Claims.Name when claim.Subject?.HasScope(Scopes.Profile) == true => [OpenIddictConstants.Destinations.AccessToken, OpenIddictConstants.Destinations.IdentityToken], Claims.Email when claim.Subject?.HasScope(Scopes.Email) == true => [OpenIddictConstants.Destinations.IdentityToken], Claims.AuthenticationMethodReference => [OpenIddictConstants.Destinations.AccessToken, OpenIddictConstants.Destinations.IdentityToken], - Claims.AuthenticationTime => [OpenIddictConstants.Destinations.IdentityToken], + Claims.AuthenticationTime or ServerSession => [OpenIddictConstants.Destinations.IdentityToken], Claims.Name or CmsClaimTypes.Kind or CmsClaimTypes.Roles or CmsClaimTypes.ClientRole => [OpenIddictConstants.Destinations.AccessToken], _ => [], }; diff --git a/src/FluentCMS.Core/Identity/Oidc/UserSessions.cs b/src/FluentCMS.Core/Identity/Oidc/UserSessions.cs index 30ee023b3..3d846017e 100644 --- a/src/FluentCMS.Core/Identity/Oidc/UserSessions.cs +++ b/src/FluentCMS.Core/Identity/Oidc/UserSessions.cs @@ -1,3 +1,5 @@ +using System.Security.Claims; +using System.Text.Json; using OpenIddict.Abstractions; using static OpenIddict.Abstractions.OpenIddictConstants; @@ -11,6 +13,59 @@ internal sealed record UserSession(Guid Id, string? ClientId, string? ClientName // every token issued under it, so its refresh token is refused at once, and so is an access token on the next request. internal sealed class UserSessions(IOpenIddictAuthorizationManager authorizations, IOpenIddictApplicationManager applications, IOpenIddictTokenManager tokens, TimeProvider time) { + // The property of an authorization that names the server session, the sign-in cookie, it was issued in. + public const string ServerSessionProperty = "fcms_server_session"; + + // The ad-hoc authorization that OpenIddict would make for a code of the client, marked with the server session the + // user signed in with, so that the logout of that session finds it. Returns its id. + public async Task BeginAsync(ClaimsPrincipal principal, object application, string serverSession, CancellationToken cancellationToken = default) + { + var descriptor = new OpenIddictAuthorizationDescriptor + { + ApplicationId = await applications.GetIdAsync(application, cancellationToken), + CreationDate = time.GetUtcNow(), + Principal = principal, + Status = Statuses.Valid, + Subject = principal.GetClaim(Claims.Subject), + Type = AuthorizationTypes.AdHoc, + }; + descriptor.Scopes.UnionWith(principal.GetScopes()); + descriptor.Properties[ServerSessionProperty] = JsonSerializer.SerializeToElement(serverSession); + + var authorization = await authorizations.CreateAsync(descriptor, cancellationToken); + return await authorizations.GetIdAsync(authorization, cancellationToken) + ?? throw new InvalidOperationException("The authorization was stored without an id."); + } + + // The end of a server session of the user: every valid authorization of the user issued in it, with its tokens. The + // user's other sessions, and a consent, which is no session's, stay. Without a session id, as for a sign-in from + // before sessions were marked, nothing ends here. + public async Task EndAsync(string? userId, string? serverSession, CancellationToken cancellationToken = default) + { + if (string.IsNullOrEmpty(serverSession) || !Guid.TryParse(userId, out var user)) + { + return; + } + + List ids = []; + await foreach (var authorization in authorizations.FindBySubjectAsync(user.ToString("D"), cancellationToken)) + { + if (await authorizations.HasStatusAsync(authorization, Statuses.Valid, cancellationToken) + && (await authorizations.GetPropertiesAsync(authorization, cancellationToken)).TryGetValue(ServerSessionProperty, out var value) + && value.ValueKind == JsonValueKind.String + && value.GetString() == serverSession + && await authorizations.GetIdAsync(authorization, cancellationToken) is { } id) + { + ids.Add(id); + } + } + + foreach (var id in ids) + { + await RevokeAsync(user, Guid.Parse(id), cancellationToken); + } + } + public async Task> ListAsync(Guid userId, CancellationToken cancellationToken = default) { List sessions = []; diff --git a/src/FluentCMS.Testing/OAuthFlows.cs b/src/FluentCMS.Testing/OAuthFlows.cs index db2336d0a..63349056c 100644 --- a/src/FluentCMS.Testing/OAuthFlows.cs +++ b/src/FluentCMS.Testing/OAuthFlows.cs @@ -1,5 +1,4 @@ using System.Buffers.Text; -using System.Globalization; using System.Net; using System.Security.Cryptography; using System.Text; @@ -55,9 +54,10 @@ public static async Task MemberCookieAsync(IServiceProvider services, Gu var principal = await signIn.CreateUserPrincipalAsync(member); var options = provider.GetRequiredService>().Get(MemberAuthentication.CookieScheme); - var now = provider.GetRequiredService().GetUtcNow(); + var time = provider.GetRequiredService(); + var now = time.GetUtcNow(); var properties = new AuthenticationProperties { IssuedUtc = now, ExpiresUtc = now + options.ExpireTimeSpan }; - properties.Items[CmsSignInManager.SignedInAtItem] = now.ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture); + CmsSignInManager.MarkSignIn(properties, time); var ticket = new AuthenticationTicket(principal, properties, MemberAuthentication.CookieScheme); return $"{options.Cookie.Name}={options.TicketDataFormat.Protect(ticket)}"; } @@ -71,9 +71,10 @@ public static async Task CookieAsync(IServiceProvider services, Guid use var principal = await provider.GetRequiredService>().CreateUserPrincipalAsync(user); var options = provider.GetRequiredService>().Get(IdentityConstants.ApplicationScheme); - var now = provider.GetRequiredService().GetUtcNow(); + var time = provider.GetRequiredService(); + var now = time.GetUtcNow(); var properties = new AuthenticationProperties { IssuedUtc = now, ExpiresUtc = now + options.ExpireTimeSpan }; - properties.Items[CmsSignInManager.SignedInAtItem] = now.ToUnixTimeSeconds().ToString(CultureInfo.InvariantCulture); + CmsSignInManager.MarkSignIn(properties, time); var ticket = new AuthenticationTicket(principal, properties, IdentityConstants.ApplicationScheme); return $"{options.Cookie.Name}={options.TicketDataFormat.Protect(ticket)}"; } diff --git a/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Hardening.cs b/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Hardening.cs index f164ddd9f..4b70024c7 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Hardening.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Hardening.cs @@ -1,4 +1,7 @@ +using System.Buffers.Text; using System.Net; +using System.Security.Cryptography; +using System.Text; using FluentCMS.Content; using FluentCMS.Core.Authorization; using FluentCMS.Core.Identity; @@ -12,6 +15,7 @@ using Microsoft.AspNetCore.DataProtection.KeyManagement; using Microsoft.AspNetCore.DataProtection.XmlEncryption; using Microsoft.AspNetCore.Identity; +using Microsoft.AspNetCore.WebUtilities; using Microsoft.EntityFrameworkCore; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; @@ -281,10 +285,11 @@ public async Task LogoutByGetNeedsTheSessionsIdTokenHint() { var editor = await _host.CreateUserAsync(BuiltInRoles.EditorName, cancellationToken: Token); var other = await _host.CreateUserAsync(BuiltInRoles.EditorName, cancellationToken: Token); - var tokens = await _host.SignInAsync(editor, "openid", "management"); var othersTokens = await _host.SignInAsync(other, "openid", "management"); + var anotherSession = await _host.SignInAsync(editor, "openid", "offline_access", "management"); var cookie = await _host.SignInCookieAsync(editor); using var client = _host.CreateSecureClient(); + var tokens = await SignInAsync(client, cookie); // Without the hint the logout page asks the user, and posts the request back. using (var bare = await SendAsync(client, HttpMethod.Get, "/connect/logout?state=abc", cookie)) @@ -308,6 +313,16 @@ public async Task LogoutByGetNeedsTheSessionsIdTokenHint() someoneElse.Headers.Contains("Set-Cookie").ShouldBeFalse(); } + // The user's own hint from another session, another browser or an earlier sign-in, is not this session's: the page + // asks, and nothing ends. + using (var otherSession = await SendAsync(client, HttpMethod.Get, $"/connect/logout?id_token_hint={anotherSession.IdentityToken}", cookie)) + { + otherSession.StatusCode.ShouldBe(HttpStatusCode.Redirect); + new Uri(client.BaseAddress!, otherSession.Headers.Location!).AbsolutePath.ShouldBe("/account/logout"); + otherSession.Headers.Contains("Set-Cookie").ShouldBeFalse(); + } + + tokens = await TokenAsync(client, Refresh(tokens.RefreshToken!), Token); using (var hinted = await SendAsync(client, HttpMethod.Get, $"/connect/logout?id_token_hint={tokens.IdentityToken}", cookie)) { hinted.StatusCode.ShouldBe(HttpStatusCode.Redirect); @@ -315,6 +330,19 @@ public async Task LogoutByGetNeedsTheSessionsIdTokenHint() hinted.Headers.GetValues("Set-Cookie").ShouldContain(value => value.StartsWith(".AspNetCore.Identity.Application=;", StringComparison.Ordinal)); } + // The session's tokens end with it, and the other session's stay. + using (var refused = await PostTokenAsync(client, Refresh(tokens.RefreshToken!))) + { + (await ErrorAsync(refused)).ShouldBe("invalid_grant"); + } + + (await TokenAsync(client, Refresh(anotherSession.RefreshToken!), Token)).RefreshToken.ShouldNotBeNull(); + + // A repeated logout with the hint, whose token the first one revoked, finds no cookie and returns without asking. + using var again = await SendAsync(client, HttpMethod.Get, $"/connect/logout?id_token_hint={tokens.IdentityToken}", ""); + again.StatusCode.ShouldBe(HttpStatusCode.Redirect); + again.Headers.Location!.OriginalString.ShouldBe("/account/logout"); + again.Headers.Contains("Set-Cookie").ShouldBeFalse(); } [Fact] @@ -382,6 +410,28 @@ private async Task WithUserAsync(TestUser user, Func, CmsUs private static string AuthorizePath() => $"/connect/authorize?client_id={FluentCmsTestHost.OAuthClientId}&response_type=code&scope=openid&redirect_uri={Uri.EscapeDataString(FluentCmsTestHost.OAuthRedirectUri)}&code_challenge=abcdefghijabcdefghijabcdefghijabcdefghijabc&code_challenge_method=S256"; + // The tokens of the test client from a code flow in the session of the cookie, so that its identity token is of it. + private static async Task SignInAsync(HttpClient client, string cookie, string scope = "openid offline_access management") + { + var verifier = Base64Url.EncodeToString(RandomNumberGenerator.GetBytes(32)); + var challenge = Base64Url.EncodeToString(SHA256.HashData(Encoding.ASCII.GetBytes(verifier))); + var path = $"/connect/authorize?client_id={TestClientId}&response_type=code&scope={Uri.EscapeDataString(scope)}&redirect_uri={Uri.EscapeDataString(FluentCmsTestHost.OAuthRedirectUri)}&code_challenge={challenge}&code_challenge_method=S256"; + using var authorize = await SendAsync(client, HttpMethod.Get, path, cookie); + var code = QueryHelpers.ParseQuery(authorize.Headers.Location!.Query)["code"].ToString(); + code.ShouldNotBeNullOrEmpty(); + return await TokenAsync( + client, + new Dictionary + { + ["grant_type"] = "authorization_code", + ["code"] = code, + ["redirect_uri"] = FluentCmsTestHost.OAuthRedirectUri, + ["client_id"] = TestClientId, + ["code_verifier"] = verifier, + }, + Token); + } + private static async Task SendAsync(HttpClient client, HttpMethod method, string path, string cookie, HttpContent? content = null) { using var request = new HttpRequestMessage(method, path) { Content = content }; diff --git a/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Sessions.cs b/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Sessions.cs index 6312fa05c..c00a205a5 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Sessions.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Identity/AuthorizationServerTests.Sessions.cs @@ -34,9 +34,9 @@ public sealed partial class AuthorizationServerTests public async Task ALogoutPostWithoutTheLogoutPagesTokenSignsNoOneOut() { var editor = await _host.CreateUserAsync(BuiltInRoles.EditorName, cancellationToken: Token); - var tokens = await _host.SignInAsync(editor, "openid", "management"); var cookie = await _host.SignInCookieAsync(editor); using var client = _host.CreateSecureClient(); + var tokens = await SignInAsync(client, cookie, "openid management"); // A form on another site that posts here: the logout page asks, and the cookie stays. using (var forged = await SendAsync(client, HttpMethod.Post, "/connect/logout", cookie, Form(("state", "xyz")))) diff --git a/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesBrowser.cs b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesBrowser.cs index 30e267468..e1d70c0f7 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesBrowser.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesBrowser.cs @@ -30,6 +30,9 @@ public HostedPagesBrowser(FluentCmsTestHost host, string address = "10.1.0.1", U public bool HasCookie(string name) => _cookies.GetCookies(_client.BaseAddress!).Any(cookie => cookie.Name == name && !cookie.Expired); + // A cookie the browser holds without a page having set it, as name=value. + public void SetCookie(string cookie) => _cookies.SetCookies(_client.BaseAddress!, cookie + "; path=/; secure"); + public Task GetAsync(string url) => _client.GetAsync(url, Token); public async Task PageAsync(string url) diff --git a/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Logout.cs b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Logout.cs new file mode 100644 index 000000000..da27491a4 --- /dev/null +++ b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Logout.cs @@ -0,0 +1,375 @@ +using System.Buffers.Text; +using System.Net; +using System.Net.Http.Headers; +using System.Security.Cryptography; +using System.Text.Json; +using FluentCMS.Core.Authorization; +using FluentCMS.Core.Identity; +using FluentCMS.Core.Identity.Oidc; +using FluentCMS.Core.IntegrationTests.Management; +using FluentCMS.Testing; +using Microsoft.AspNetCore.Authentication; +using Microsoft.AspNetCore.Authentication.Cookies; +using Microsoft.AspNetCore.Identity; +using Microsoft.Extensions.DependencyInjection; +using Microsoft.Extensions.Options; +using Microsoft.IdentityModel.JsonWebTokens; +using Microsoft.IdentityModel.Tokens; +using Shouldly; +using Xunit; + +namespace FluentCMS.Core.IntegrationTests.Identity; + +// What a logout ends of the sessions of its user (#3198). +public sealed partial class HostedPagesTests +{ + [Fact] + public async Task AConfirmedLogoutRevokesTheTokensOfItsSessionAndLeavesTheUsersOtherSessions() + { + var editor = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.16"); + using var otherDevice = new HostedPagesBrowser(_host, "10.1.0.9"); + await SignInAsync(browser, editor.Email); + await SignInAsync(otherDevice, editor.Email); + var tokens = await CodeFlowAsync(_host, browser); + var secondTab = await CodeFlowAsync(_host, browser); + var other = await CodeFlowAsync(_host, otherDevice); + + // A logout that is not confirmed, by a GET without a hint or a POST without the page's token, revokes nothing. + using (var asked = await browser.GetAsync("/connect/logout")) + { + asked.Headers.Location!.OriginalString.ShouldBe("/account/logout"); + } + + using (var forged = await browser.PostAsync("/connect/logout", new Dictionary())) + { + forged.Headers.Location!.OriginalString.ShouldBe("/account/logout"); + } + + var refreshToken = await RotatedAsync(_host, Text(tokens, "refresh_token")); + (await MeAsync(_host, Text(tokens, "access_token"))).ShouldBe(HttpStatusCode.OK); + + await SignOutAsync(browser); + + (await RefusedAsync(_host, refreshToken)).ShouldBe("invalid_grant"); + (await RefusedAsync(_host, Text(secondTab, "refresh_token"))).ShouldBe("invalid_grant"); + (await MeAsync(_host, Text(tokens, "access_token"))).ShouldBe(HttpStatusCode.Unauthorized); + (await RotatedAsync(_host, Text(other, "refresh_token"))).ShouldNotBeNullOrEmpty(); + (await MeAsync(_host, Text(other, "access_token"))).ShouldBe(HttpStatusCode.OK); + } + + [Fact] + public async Task ALogoutWithAHintRevokesOnlyWhenItEndsTheSessionOfItsCookie() + { + var editor = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.17"); + using var otherDevice = new HostedPagesBrowser(_host, "10.1.0.11"); + await SignInAsync(browser, editor.Email); + await SignInAsync(otherDevice, editor.Email); + var tokens = await CodeFlowAsync(_host, browser); + var hint = "/connect/logout?id_token_hint=" + Uri.EscapeDataString(Text(tokens, "id_token")); + + // Without the cookie there is no session to end, and the hint alone revokes nothing. + using (var stranger = new HostedPagesBrowser(_host, "10.1.0.10")) + using (var returned = await stranger.GetAsync(hint)) + { + returned.Headers.Location!.OriginalString.ShouldBe("/account/logout"); + } + + // The user's hint is not the other device's session: that device is asked, and its cookie and this session stay. + using (var elsewhere = await otherDevice.GetAsync(hint)) + { + elsewhere.Headers.Location!.OriginalString.ShouldStartWith("/account/logout?"); + } + + otherDevice.HasCookie(SignInCookie).ShouldBeTrue(); + var refreshToken = await RotatedAsync(_host, Text(tokens, "refresh_token")); + + using (var signedOut = await browser.GetAsync(hint)) + { + signedOut.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + browser.HasCookie(SignInCookie).ShouldBeFalse(); + (await RefusedAsync(_host, refreshToken)).ShouldBe("invalid_grant"); + + // Logging out again with the same hint, as a second tab or a double click does, finds nothing to end and asks nothing. + using var again = await browser.GetAsync(hint); + again.Headers.Location!.OriginalString.ShouldBe("/account/logout"); + } + + [Fact] + public async Task ALogoutLeavesTheConsentOfAThirdPartyClient() + { + var editor = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.18"); + await SignInAsync(browser, editor.Email); + + var request = AuthorizeRequest.For(ThirdPartyClientId, ThirdPartyRedirectUri, "openid management"); + using var asked = await browser.GetAsync(request.Url); + var consent = asked.Headers.Location!.OriginalString; + var page = await browser.PageAsync(consent); + using var allowed = await browser.PostAsync(consent, page, new Dictionary { ["decision"] = "allow" }); + using var done = await browser.FollowAsync(allowed, ThirdPartyRedirectUri); + var issued = await request.ExchangeAsync(_host, done); + + // A consent is no session's, so its identity token names none, and its hint is asked about. + Payload(Text(issued, "id_token")).TryGetProperty("sid", out _).ShouldBeFalse(); + using (var hinted = await browser.GetAsync("/connect/logout?id_token_hint=" + Uri.EscapeDataString(Text(issued, "id_token")))) + { + hinted.Headers.Location!.OriginalString.ShouldStartWith("/account/logout?"); + } + + browser.HasCookie(SignInCookie).ShouldBeTrue(); + await SignOutAsync(browser); + + // The client keeps its token, and the next sign-in is not asked again. + (await MeAsync(_host, Text(issued, "access_token"))).ShouldBe(HttpStatusCode.OK); + await SignInAsync(browser, editor.Email); + using var again = await browser.GetAsync(AuthorizeRequest.For(ThirdPartyClientId, ThirdPartyRedirectUri, "openid management").Url); + again.Headers.Location!.AbsoluteUri.ShouldStartWith(ThirdPartyRedirectUri + "?code="); + } + + [Fact] + public async Task ASignInOverALiveOneKeepsTheSessionOfItsUserAndEndsAnothers() + { + var editor = await CreateUserAsync(_host); + var other = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.19"); + await SignInAsync(browser, editor.Email); + var before = await CodeFlowAsync(_host, browser); + + // The same user signs in again on the login page: the browser's session goes on, and its logout ends both. + await SignInAsync(browser, editor.Email); + var after = await CodeFlowAsync(_host, browser); + Sid(after).ShouldBe(Sid(before)); + await SignOutAsync(browser); + (await RefusedAsync(_host, Text(before, "refresh_token"))).ShouldBe("invalid_grant"); + (await RefusedAsync(_host, Text(after, "refresh_token"))).ShouldBe("invalid_grant"); + + // Another user signs in over a live sign-in: the first user's session in this browser ends. + await SignInAsync(browser, editor.Email); + var replaced = await CodeFlowAsync(_host, browser); + await SignInAsync(browser, other.Email); + (await RefusedAsync(_host, Text(replaced, "refresh_token"))).ShouldBe("invalid_grant"); + Sid(await CodeFlowAsync(_host, browser)).ShouldNotBe(Sid(replaced)); + } + + [Fact] + public async Task AReauthenticationAnotherSiteCanAskForEndsTheCookieAndRevokesNothing() + { + await using var host = NewHost(_emails).Configure(cms => cms.Clients.AbsoluteSessionLifetime = TimeSpan.FromHours(8)); + var editor = await CreateUserAsync(host); + using var browser = new HostedPagesBrowser(host); + + // A top-level GET from another site carries the cookie and nothing secret: prompt=login, max_age=0, and any + // authorize request past the absolute lifetime sign the cookie out and leave the session's tokens working. + var authorize = AuthorizeRequest.For(FluentCmsTestHost.OAuthClientId, FluentCmsTestHost.OAuthRedirectUri).Url; + var login = AuthorizeRequest.For(FluentCmsTestHost.OAuthClientId, FluentCmsTestHost.OAuthRedirectUri, prompt: "login").Url; + foreach (var (url, wait) in new[] { (login, TimeSpan.Zero), (authorize + "&max_age=0", TimeSpan.Zero), (authorize, TimeSpan.FromHours(2)) }) + { + await SignInAsync(browser, editor.Email); + host.Time.Advance(TimeSpan.FromHours(7)); + var tokens = await CodeFlowAsync(host, browser); + host.Time.Advance(wait); + + using var foreign = await browser.GetAsync(url); + new Uri(OAuthBase, foreign.Headers.Location!).AbsolutePath.ShouldBe("/account/login", url); + browser.HasCookie(SignInCookie).ShouldBeFalse(); + + var renewed = await RenewedAsync(host, Text(tokens, "refresh_token")); + (await MeAsync(host, Text(renewed, "access_token"))).ShouldBe(HttpStatusCode.OK, url); + } + } + + [Fact] + public async Task ASecondFactorTheRoleNowRequiresEndsTheCookieAtAuthorizeAndRevokesNothing() + { + using var admin = await _host.AsUserAsync(await _host.CreateUserAsync(BuiltInRoles.SuperAdminName, cancellationToken: Token)); + var editor = await CreateUserAsync(_host); + var (key, _) = await EnableTwoFactorAsync(_host, editor); + using var browser = new HostedPagesBrowser(_host, "10.1.0.13"); + + // A sign-in with the second factor on a remembered device, then one with the password alone over it, in one session. + using (var password = await browser.SubmitAsync("/account/login", Credentials(editor.Email, Password))) + using (var remembered = await browser.SubmitAsync(password.Headers.Location!.OriginalString, new Dictionary { ["Input.Code"] = Totp(key), ["Input.RememberMachine"] = "true" })) + { + remembered.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + var withSecondFactor = await CodeFlowAsync(_host, browser); + await SignInAsync(browser, editor.Email); + var withPassword = await CodeFlowAsync(_host, browser); + Sid(withPassword).ShouldBe(Sid(withSecondFactor)); + + // Another site sends the browser to a plain authorize request once the role requires the second factor: the cookie + // ends, and the tokens are left to the token endpoint, which refuses only those without it. + (await AdminFixture.SendAsync(admin, HttpMethod.Put, $"users/{editor.Id:D}/roles/{await GuardedRoleAsync(admin)}")).Status.ShouldBe(HttpStatusCode.OK); + using var challenged = await browser.GetAsync(AuthorizeRequest.For(FluentCmsTestHost.OAuthClientId, FluentCmsTestHost.OAuthRedirectUri).Url); + challenged.Headers.Location!.AbsolutePath.ShouldBe("/account/login"); + browser.HasCookie(SignInCookie).ShouldBeFalse(); + + (await MeAsync(_host, Text(withSecondFactor, "access_token"))).ShouldBe(HttpStatusCode.OK); + (await RotatedAsync(_host, Text(withSecondFactor, "refresh_token"))).ShouldNotBeNullOrEmpty(); + (await RefusedAsync(_host, Text(withPassword, "refresh_token"))).ShouldBe("invalid_grant"); + } + + [Fact] + public async Task AHintWithoutTheCookieReturnsOnlyAnIdentityTokenOfThisServerToItsOwnClient() + { + var editor = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.14"); + using var stranger = new HostedPagesBrowser(_host, "10.1.0.15"); + var signedOut = Uri.EscapeDataString(ThirdPartySignedOutUri); + + // Not an identity token of this server: something else, one signed with another key, and an access token of it. + await SignInAsync(browser, editor.Email); + var tokens = await CodeFlowAsync(_host, browser); + using var otherKey = RSA.Create(2048); + var foreign = new JsonWebTokenHandler().CreateToken(new SecurityTokenDescriptor + { + Issuer = OAuthBase.AbsoluteUri, + Audience = ThirdPartyClientId, + Claims = new Dictionary { ["sub"] = editor.Id.ToString("D"), ["azp"] = ThirdPartyClientId }, + SigningCredentials = new SigningCredentials(new RsaSecurityKey(otherKey), SecurityAlgorithms.RsaSha256), + }); + foreach (var hint in new[] { "not.a.token", foreign, Text(tokens, "access_token") }) + { + using var response = await stranger.GetAsync($"/connect/logout?id_token_hint={Uri.EscapeDataString(hint)}&client_id={ThirdPartyClientId}&post_logout_redirect_uri={signedOut}"); + response.Headers.Location!.OriginalString.ShouldStartWith("/account/logout?"); + } + + // A revoked identity token of the test client is not the third-party client's, whose signed-out URI it asks for. + await SignOutAsync(browser); + var revoked = Uri.EscapeDataString(Text(tokens, "id_token")); + foreach (var query in new[] { $"&post_logout_redirect_uri={signedOut}", $"&client_id={ThirdPartyClientId}&post_logout_redirect_uri={signedOut}", $"&client_id={ThirdPartyClientId}" }) + { + using var response = await stranger.GetAsync($"/connect/logout?id_token_hint={revoked}{query}&state=s1"); + response.Headers.Location!.OriginalString.ShouldStartWith("/account/logout?", Case.Sensitive, query); + } + + // The third-party client's own revoked hint returns to its own signed-out URI without asking. + await SignInAsync(browser, editor.Email); + var request = AuthorizeRequest.For(ThirdPartyClientId, ThirdPartyRedirectUri, "openid management"); + using var asked = await browser.GetAsync(request.Url); + var consent = asked.Headers.Location!.OriginalString; + using var allowed = await browser.PostAsync(consent, await browser.PageAsync(consent), new Dictionary { ["decision"] = "allow" }); + using var done = await browser.FollowAsync(allowed, ThirdPartyRedirectUri); + var own = Uri.EscapeDataString(Text(await request.ExchangeAsync(_host, done), "id_token")); + await using (var scope = _host.Services.CreateAsyncScope()) + { + await scope.ServiceProvider.GetRequiredService().RevokeAllAsync(editor.Id, Token); + } + + using var back = await stranger.GetAsync($"/connect/logout?id_token_hint={own}&client_id={ThirdPartyClientId}&post_logout_redirect_uri={signedOut}&state=s1"); + back.Headers.Location!.AbsoluteUri.ShouldBe(ThirdPartySignedOutUri + "?state=s1"); + } + [Fact] + public async Task ASignInWithTheSecondFactorAndAPasswordChangeKeepTheSessionThatTheLogoutEnds() + { + var editor = await CreateUserAsync(_host); + var (key, _) = await EnableTwoFactorAsync(_host, editor); + using var browser = new HostedPagesBrowser(_host, "10.1.0.12"); + using (var password = await browser.SubmitAsync("/account/login", Credentials(editor.Email, Password))) + using (var challenged = await browser.SubmitAsync(password.Headers.Location!.OriginalString, new Dictionary { ["Input.Code"] = Totp(key) })) + { + challenged.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + var first = await CodeFlowAsync(_host, browser); + + using (var changed = await browser.SubmitAsync("/account/change-password", new Dictionary(NewPasswordFields("A-changed-password-1")) { ["Input.CurrentPassword"] = Password })) + { + changed.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + var second = await CodeFlowAsync(_host, browser); + Sid(second).ShouldBe(Sid(first)); + + await SignOutAsync(browser); + (await RefusedAsync(_host, Text(second, "refresh_token"))).ShouldBe("invalid_grant"); + } + + [Fact] + public async Task TheLogoutOfACookieFromBeforeSessionsWereMarkedEndsTheCookieAlone() + { + var editor = await CreateUserAsync(_host); + using var browser = new HostedPagesBrowser(_host, "10.1.0.20"); + browser.SetCookie(await UnmarkedCookieAsync(editor)); + var tokens = await CodeFlowAsync(_host, browser); + Payload(Text(tokens, "id_token")).TryGetProperty("sid", out _).ShouldBeFalse(); + + await SignOutAsync(browser); + + browser.HasCookie(SignInCookie).ShouldBeFalse(); + (await RotatedAsync(_host, Text(tokens, "refresh_token"))).ShouldNotBeNullOrEmpty(); + } + + // The code flow of the test client in the browser's session. + private static async Task CodeFlowAsync(FluentCmsTestHost host, HostedPagesBrowser browser) + { + var request = AuthorizeRequest.For(FluentCmsTestHost.OAuthClientId, FluentCmsTestHost.OAuthRedirectUri); + using var redirect = await browser.GetAsync(request.Url); + return await request.ExchangeAsync(host, redirect); + } + + // The rotated refresh token. + private static async Task RotatedAsync(FluentCmsTestHost host, string refreshToken) + { + using var response = await RefreshAsync(host, refreshToken); + var body = await response.Content.ReadAsStringAsync(Token); + response.StatusCode.ShouldBe(HttpStatusCode.OK, body); + using var json = JsonDocument.Parse(body); + return json.RootElement.GetProperty("refresh_token").GetString()!; + } + + private static async Task RenewedAsync(FluentCmsTestHost host, string refreshToken) + { + using var response = await RefreshAsync(host, refreshToken); + var body = await response.Content.ReadAsStringAsync(Token); + response.StatusCode.ShouldBe(HttpStatusCode.OK, body); + using var json = JsonDocument.Parse(body); + return json.RootElement.Clone(); + } + + private static async Task RefusedAsync(FluentCmsTestHost host, string refreshToken) + { + using var response = await RefreshAsync(host, refreshToken); + response.StatusCode.ShouldBe(HttpStatusCode.BadRequest); + using var json = JsonDocument.Parse(await response.Content.ReadAsStringAsync(Token)); + return json.RootElement.GetProperty("error").GetString(); + } + + private static async Task MeAsync(FluentCmsTestHost host, string accessToken) + { + using var client = host.CreateSecureClient(); + client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken); + using var response = await client.GetAsync("/api/management/v1/me", Token); + return response.StatusCode; + } + + // A sign-in cookie as one made before sessions were marked: the time of the sign-in and no server session. + private async Task UnmarkedCookieAsync(TestUser user) + { + await using var scope = _host.Services.CreateAsyncScope(); + var provider = scope.ServiceProvider; + var stored = await provider.GetRequiredService().FindByIdAsync(user.Id.ToString()); + var principal = await provider.GetRequiredService>().CreateUserPrincipalAsync(stored!); + var options = provider.GetRequiredService>().Get(IdentityConstants.ApplicationScheme); + var now = _host.Time.GetUtcNow(); + var properties = new AuthenticationProperties { IssuedUtc = now, ExpiresUtc = now + options.ExpireTimeSpan }; + properties.Items[CmsSignInManager.SignedInAtItem] = now.ToUnixTimeSeconds().ToString(System.Globalization.CultureInfo.InvariantCulture); + var ticket = new AuthenticationTicket(principal, properties, IdentityConstants.ApplicationScheme); + return $"{options.Cookie.Name}={options.TicketDataFormat.Protect(ticket)}"; + } + + private static string Text(JsonElement tokens, string name) => tokens.GetProperty(name).GetString()!; + + private static string? Sid(JsonElement tokens) => Payload(Text(tokens, "id_token")).GetProperty("sid").GetString(); + + private static JsonElement Payload(string jwt) + { + using var document = JsonDocument.Parse(Base64Url.DecodeFromChars(jwt.Split('.')[1])); + return document.RootElement.Clone(); + } +} diff --git a/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Sessions.cs b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Sessions.cs index 6780c898c..22574b2c1 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Sessions.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Identity/HostedPagesTests.Sessions.cs @@ -33,6 +33,8 @@ public async Task TheRemoteSignOutOfAProviderEndsTheAdminSessionItMadeAndNoOther var editor = await CreateUserAsync(host); using var password = new HostedPagesBrowser(host); await SignInAsync(password, editor.Email); + var ssoTokens = await CodeFlowAsync(host, sso); + var passwordTokens = await CodeFlowAsync(host, password); // What the provider's front-channel logout loads in the browser. foreach (var (browser, provider) in new[] { (sso, Trusted), (password, Corp) }) @@ -42,9 +44,13 @@ public async Task TheRemoteSignOutOfAProviderEndsTheAdminSessionItMadeAndNoOther browser.HasCookie(SignInCookie).ShouldBeTrue(); } + // Nothing authenticates the provider's request, so it ends the cookie alone and the session's tokens work on (#3204). using var signedOut = await sso.GetAsync($"/account/sso/{Corp}/signout"); signedOut.StatusCode.ShouldBe(HttpStatusCode.OK); sso.HasCookie(SignInCookie).ShouldBeFalse(); + var renewed = await RenewedAsync(host, Text(ssoTokens, "refresh_token")); + (await MeAsync(host, Text(renewed, "access_token"))).ShouldBe(HttpStatusCode.OK); + (await RotatedAsync(host, Text(passwordTokens, "refresh_token"))).ShouldNotBeNullOrEmpty(); } [Fact] diff --git a/tests/FluentCMS.Core.IntegrationTests/Members/MemberPagesTests.cs b/tests/FluentCMS.Core.IntegrationTests/Members/MemberPagesTests.cs index c272a0a44..6d466e366 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Members/MemberPagesTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Members/MemberPagesTests.cs @@ -242,6 +242,142 @@ public async Task AMembersIdTokenAsAHintEndsTheMemberSessionAndLeavesTheAdmins() browser.HasCookie(AdminCookie).ShouldBeTrue(); } + [Fact] + public async Task TheMemberLogoutPageRevokesTheRefreshTokenOfItsSession() + { + var address = await fixture.RegisterVerifiedAsync(); + using var browser = new HostedPagesBrowser(fixture.Host, "10.8.0.20"); + await SignInMemberAsync(browser, address); + var request = AuthorizeRequest.For(FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + using var code = await browser.GetAsync(request.Url); + var refreshToken = (await request.ExchangeAsync(fixture.Host, code)).GetProperty("refresh_token").GetString()!; + + var page = await browser.PageAsync("/account/members/logout"); + using (var signedOut = await browser.PostAsync("/account/members/logout", page, new Dictionary())) + { + signedOut.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, refreshToken)).ShouldBe(HttpStatusCode.BadRequest); + } + + [Fact] + public async Task TheMemberClientsLogoutRevokesTheMembersSessionAndLeavesTheAdminsTokens() + { + var admin = await CreateAdminAsync(); + var address = await fixture.RegisterVerifiedAsync(); + using var browser = new HostedPagesBrowser(fixture.Host, "10.8.0.21"); + await SignInAdminAsync(browser, admin.Email); + await SignInMemberAsync(browser, address); + var adminRequest = AuthorizeRequest.For(FluentCmsTestHost.OAuthClientId, "openid offline_access management"); + using var adminCode = await browser.GetAsync(adminRequest.Url); + var adminRefresh = (await adminRequest.ExchangeAsync(fixture.Host, adminCode)).GetProperty("refresh_token").GetString()!; + var memberRequest = AuthorizeRequest.For(FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + using var memberCode = await browser.GetAsync(memberRequest.Url); + var memberRefresh = (await memberRequest.ExchangeAsync(fixture.Host, memberCode)).GetProperty("refresh_token").GetString()!; + + using (var asked = await browser.GetAsync($"/connect/logout?client_id={FluentCmsTestHost.OAuthMemberClientId}")) + { + var page = await browser.PageAsync(asked.Headers.Location!.OriginalString); + using var confirmed = await browser.PostAsync("/connect/logout", page, new Dictionary + { + ["client_id"] = FluentCmsTestHost.OAuthMemberClientId, + [MemberAuthentication.LogoutPageField] = MemberAuthentication.LogoutPageMark, + }); + confirmed.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, memberRefresh)).ShouldBe(HttpStatusCode.BadRequest); + (await RefreshAsync(FluentCmsTestHost.OAuthClientId, adminRefresh)).ShouldBe(HttpStatusCode.OK); + } + + [Fact] + public async Task AnAdminLogoutLeavesTheMembersTokens() + { + var admin = await CreateAdminAsync(); + var address = await fixture.RegisterVerifiedAsync(); + using var browser = new HostedPagesBrowser(fixture.Host, "10.8.0.22"); + await SignInAdminAsync(browser, admin.Email); + await SignInMemberAsync(browser, address); + var adminTokens = await CodeFlowAsync(browser, FluentCmsTestHost.OAuthClientId, "openid offline_access management"); + var memberTokens = await CodeFlowAsync(browser, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + + var page = await browser.PageAsync("/account/logout"); + using (var signedOut = await browser.PostAsync("/connect/logout", page, new Dictionary())) + { + signedOut.StatusCode.ShouldBe(HttpStatusCode.Redirect); + } + + (await RefreshAsync(FluentCmsTestHost.OAuthClientId, adminTokens.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.BadRequest); + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, memberTokens.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.OK); + } + + [Fact] + public async Task AMemberSignInOverALiveOneKeepsTheSameMembersSessionAndEndsOnlyAnothers() + { + var first = await fixture.RegisterVerifiedAsync(); + var second = await fixture.RegisterVerifiedAsync(); + using var browser = new HostedPagesBrowser(fixture.Host, "10.8.0.23"); + using var otherDevice = new HostedPagesBrowser(fixture.Host, "10.8.0.24"); + await SignInMemberAsync(browser, first); + await SignInMemberAsync(otherDevice, first); + var before = await CodeFlowAsync(browser, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + var elsewhere = await CodeFlowAsync(otherDevice, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + + // The same member signs in again: the browser's session goes on. + await SignInMemberAsync(browser, first); + var after = await CodeFlowAsync(browser, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + Sid(after).ShouldBe(Sid(before)); + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, before.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.OK); + + // Another member signs in over it: the first member's session in this browser ends, and its other device's stays. + await SignInMemberAsync(browser, second); + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, after.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.BadRequest); + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, elsewhere.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.OK); + Sid(await CodeFlowAsync(browser, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access")).ShouldNotBe(Sid(after)); + } + + [Fact] + public async Task AMemberReauthenticationAnotherSiteCanAskForEndsTheCookieAndRevokesNothing() + { + var address = await fixture.RegisterVerifiedAsync(); + using var browser = new HostedPagesBrowser(fixture.Host, "10.8.0.25"); + foreach (var prompt in new[] { "&prompt=login", "&max_age=0" }) + { + await SignInMemberAsync(browser, address); + var tokens = await CodeFlowAsync(browser, FluentCmsTestHost.OAuthMemberClientId, "openid offline_access"); + + using var foreign = await browser.GetAsync("/connect/authorize" + AuthorizeRequest.Query(FluentCmsTestHost.OAuthMemberClientId, "openid") + prompt); + foreign.Headers.Location!.OriginalString.ShouldContain("/account/members/login", Case.Sensitive, prompt); + browser.HasCookie(MemberCookie).ShouldBeFalse(); + (await RefreshAsync(FluentCmsTestHost.OAuthMemberClientId, tokens.GetProperty("refresh_token").GetString()!)).ShouldBe(HttpStatusCode.OK, prompt); + } + } + private async Task CodeFlowAsync(HostedPagesBrowser browser, string clientId, string scope) + { + var request = AuthorizeRequest.For(clientId, scope); + using var code = await browser.GetAsync(request.Url); + return await request.ExchangeAsync(fixture.Host, code); + } + + private static string? Sid(JsonElement tokens) + { + using var payload = JsonDocument.Parse(Base64Url.DecodeFromChars(tokens.GetProperty("id_token").GetString()!.Split('.')[1])); + return payload.RootElement.GetProperty("sid").GetString(); + } + private async Task RefreshAsync(string clientId, string refreshToken) + { + using var client = fixture.Host.CreateSecureClient(); + using var content = new FormUrlEncodedContent(new Dictionary + { + ["grant_type"] = "refresh_token", + ["refresh_token"] = refreshToken, + ["client_id"] = clientId, + }); + using var refreshed = await client.PostAsync("/connect/token", content, Token); + return refreshed.StatusCode; + } + [Theory] [InlineData("https://evil.example/")] [InlineData("//evil.example/")] From 33a69b38e29f0e92b92e2b4f736b75537d5ad885 Mon Sep 17 00:00:00 2001 From: Amir Pournasserian Date: Mon, 16 Nov 2026 08:48:50 -0500 Subject: [PATCH 8/8] Say in the sign-in decision record what a confirmed logout ends --- docs/decisions/0007-admin-sign-in-flow.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/decisions/0007-admin-sign-in-flow.md b/docs/decisions/0007-admin-sign-in-flow.md index 1bc2767b6..3fbe4e888 100644 --- a/docs/decisions/0007-admin-sign-in-flow.md +++ b/docs/decisions/0007-admin-sign-in-flow.md @@ -43,5 +43,5 @@ We will write the sign-in in the admin, in C#, as `AdminSession` in `FluentCMS.A - Drafts are in `sessionStorage` in clear text until they are restored and saved, discarded, removed at sign-out, or the tab closes. - A wrong `authority` in `config.json` sends the code, the PKCE verifier and a stored refresh token to that origin; `config.json` is as trusted as `index.html` (spec 2.5). - Every page of the same origin, the hosted pages and the API reference among them, can read the tab's `sessionStorage`. - - A logout the user does not confirm on the hosted page leaves the server's session alive, so opening the admin again signs in without a prompt. + - A logout the user does not confirm on the hosted page leaves the server's session and its tokens alive, so opening the admin again signs in without a prompt. A confirmed logout ends the server's session and revokes every authorization issued in it, with its refresh and access tokens (#3198), so a revocation at `/connect/revoke` that is cut off or fails at sign-out leaves no refresh token redeemable once the logout is confirmed. A logout does not end a sign-in from before #3198, whose cookie carries no session id, or the consent of a third-party client. - The browser end-to-end tests of A-11 prove the flow in a browser against the reference host; the component and API tests cannot show the whole-page redirects, `sessionStorage` itself, or the Content-Security-Policy.