diff --git a/docs/planning/backend-technical-spec.md b/docs/planning/backend-technical-spec.md index 77e5ff5eb..413c969ad 100644 --- a/docs/planning/backend-technical-spec.md +++ b/docs/planning/backend-technical-spec.md @@ -1169,7 +1169,7 @@ Rows 10.3, 10.5 to 10.8, 10.11 to 10.15. **Multi-instance.** Stateless API; Data Protection and signing keys persisted to the database by default; jobs lease through the database; `IDistributedLock` defaults to a `cms_lock` table and gets a Redis provider in `FluentCMS.Caching.Redis`; output cache in memory by default or Redis; media on shared storage. -**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. SQLite has one writer, so while another connection keeps a write transaction open, such as the schema apply or schema sync that holds the lock, a renewal waits for it and gives up after the busy timeout and the retries of Microsoft.Data.Sqlite, about 35 seconds, and so does a take; in the rollback journal mode a read of the row can wait too. Such a command, which `IDataProvider.IsBusy` recognises (SQLite's busy and locked errors by their primary code, never on PostgreSQL or SQL Server), changed nothing: a waiter counts the lock as held and ends with the lock or its timeout, which it can overshoot by one such wait, and the holder reads the row, which in WAL mode does not wait, and keeps the lock while the row still names it, renewing again at once and logging once that it waits (9165). A take and a renewal each run in a transaction of their own and read the clock once it has begun, so on SQLite, where beginning it waits for the writer, a lease starts when the database let the write in, not when it was asked for; on PostgreSQL and SQL Server the clock is still read before a statement that can wait on the row, as before. Before, a holder whose transaction lasted longer than about 45 seconds lost the lock, which canceled the rest of its work and rolled the transaction back, and a waiter failed with `SqliteException` ([#3176](https://github.com/fluentcms/FluentCMS/issues/3176)). One race remains: a waiter that found the lease over while such a transaction lasted waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other holder and cancels `Lost`, but a holder with more than one unit of work under the lock may have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. With a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 measured rounds, and the holder's `Lost` fired 64 to 119 milliseconds after the commit; when the holder began its next write at once, the waiter took it in none of 16. Known and left to [#3186](https://github.com/fluentcms/FluentCMS/issues/3186): a `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again, which records the history; and the waiter's event 9160 then says that a holder stopped without releasing the lock while that holder is alive. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. +**Multi-instance as built.** `IDistributedLock` (`FluentCMS.Providers`, in the SDK) has one method, `TryAcquireAsync(name, timeout)`, which returns a handle that releases the lock when disposed, or null when the timeout passed first (`TimeSpan.Zero` tries once, `Timeout.InfiniteTimeSpan` waits until the lock is free); the handle's `Lost` token is canceled when the lock is lost while held. Core's default, added with `TryAdd` and with no options, keeps one row per name in `cms_lock (name, owner, acquired_at, lease_until)`. A holder takes a 30 second lease and renews it every 10 seconds while it holds the lock, so a holder that stopped without releasing it blocks the others until its lease runs out, and the next holder takes it over with a warning (event id 9160). Taking the lock is an update conditional on the row being free or its lease over, so of two that try at once one wins, and a waiter only reads the row while the lock is held, polling from every 50 milliseconds up to once a second. A free lock keeps its row with no owner, so only the first use of a name inserts, and lock names are a fixed set, one per task rather than one per item. A renewal that finds another holder, or that fails until the lease has run out, cancels `Lost` (9161); a renewal or a release that fails is logged (9162, 9163), and a release changes nothing once another holder has the lock. SQLite has one writer, so while another connection keeps a write transaction open, such as the schema apply or schema sync that holds the lock, a renewal waits for it and gives up after the busy timeout and the retries of Microsoft.Data.Sqlite, about 35 seconds, and so does a take; in the rollback journal mode a read of the row can wait too. Such a command, which `IDataProvider.IsBusy` recognises (SQLite's busy and locked errors by their primary code, never on PostgreSQL or SQL Server), changed nothing: a waiter counts the lock as held and ends with the lock or its timeout, which it can overshoot by one such wait, and the holder reads the row, which in WAL mode does not wait, and keeps the lock while the row still names it, renewing again at once and logging once that it waits (9165). A take and a renewal each run in a transaction of their own and read the clock once it has begun, so on SQLite, where beginning it waits for the writer, a lease starts when the database let the write in, not when it was asked for; on PostgreSQL and SQL Server the clock is still read before a statement that can wait on the row, as before. A take that loses the insert of a name's first row rolls its transaction back before it reads the row, since a read on another connection of the in-memory database waits for it ([#3200](https://github.com/fluentcms/FluentCMS/issues/3200)). Before, a holder whose transaction lasted longer than about 45 seconds lost the lock, which canceled the rest of its work and rolled the transaction back, and a waiter failed with `SqliteException` ([#3176](https://github.com/fluentcms/FluentCMS/issues/3176)). One race remains: a waiter that found the lease over while such a transaction lasted waits for the writer beside the holder's renewal, and when the transaction ends it can come first. The renewal then finds the other holder and cancels `Lost`, but a holder with more than one unit of work under the lock may have work left: schema apply records the schema history after its change, and EF commits what came before a SQLite table rebuild. With a waiter present at the end of a long transaction, the waiter took the lock in 4 of 16 measured rounds, and the holder's `Lost` fired 64 to 119 milliseconds after the commit; when the holder began its next write at once, the waiter took it in none of 16. Known and left to [#3186](https://github.com/fluentcms/FluentCMS/issues/3186): a `schema apply` that loses the lock this way after its change ends with the generic `FCMS3302 ... TaskCanceledException`, not with a message that says to run it again, which records the history; and the waiter's event 9160 then says that a holder stopped without releasing the lock while that holder is alive. Leases are read against each instance's clock, so instances keep their clocks in step, as the job leases already need. The Data Protection ring is in `cms_data_protection_key` (4.7), so instances on one database read each other's cookies and protected payloads, and the keys survive a restart. Data Protection makes a key when its ring has none it can use, and instances that started together on an empty ring each made one; one that had read only its own kept it for up to a day, refusing the others' cookies once the two minutes in which Data Protection reloads the ring for an unknown key were over. The key manager now makes a key under the lock `fluentcms:data-protection-key`, after reading the ring again, and returns a key that another instance made in the last five minutes (the clock skew Data Protection allows) instead of making a second; when the lock is not free within 10 seconds it makes the key without it (9164). The wait blocks the thread Data Protection asks on, since `IKeyManager` is synchronous, so it covers a live holder, which makes its key within a second, and not one that stopped without releasing the lock, whose 30 second lease it used to wait out. A synchronous `TryAcquire` on `IDistributedLock` is declined: the default lock is a table read through EF Core's asynchronous calls, so it would only move the blocking, and every provider of the slot would have to implement it (item 6 of [#2887](https://github.com/fluentcms/FluentCMS/issues/2887)). The changes that can take Super Admin from its last holder run under `fluentcms:super-admins` (4.7), and fail rather than go on without it. The Delivery output cache is the `IOutputCacheStore` slot, in memory by default, which evicts only the copies of its own instance; a host that registers a shared store, such as `Microsoft.AspNetCore.OutputCaching.StackExchangeRedis`, evicts for every instance, and the key's schema hash keeps instances on different schemas apart (4.10). The member mail limit (4.7) is kept in the `IDistributedCache` slot, the framework's `MemoryDistributedCache` by default, one per instance, so N instances behind a load balancer send an address up to N mails a minute of each slot; a host with several instances registers a shared cache, such as `RedisCache`, and the startup report lists `member-mail-limit-per-instance` in Production while an email sender is set up and the cache is the in-memory one ([#3073](https://github.com/fluentcms/FluentCMS/issues/3073)). Each instance keeps its own copy of the sites and their locales, of plugin enablement and of settings, and sees a change another instance made within 30 seconds (4.6, 4.13); role changes are seen on the next request everywhere, since every request reads the role stamps (4.8). The Redis provider in `FluentCMS.Caching.Redis` is still to come. **Reverse proxy.** Forwarded headers, `PathBase` option for sub-path hosting, HTTPS redirection and HSTS options; documented for Nginx, Caddy, YARP, Azure Front Door and Cloudflare. diff --git a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs index 5726db84d..61b260b15 100644 --- a/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs +++ b/src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs @@ -92,7 +92,9 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP } catch (DbUpdateException) { - // Another holder made the row first. Only the first use of a name inserts, so this is rare. + // Another holder made the row first. Only the first use of a name inserts, so this is rare. The open + // transaction holds the write lock, which a read on another connection of the in-memory database waits for. + await transaction.RollbackAsync(CancellationToken.None); if (await ExistsAsync(name)) { return null; diff --git a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs index 7e1e3cc46..431e0ed72 100644 --- a/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs +++ b/tests/FluentCMS.Core.IntegrationTests/Caching/DistributedLockTests.cs @@ -333,6 +333,25 @@ public async Task ABadNameOrTimeoutIsRefused() await Should.ThrowAsync(() => locks.TryAcquireAsync(Name, TimeSpan.FromSeconds(-2), Token)); } + // The first use of a name inserts its row, so takes that start together race on the insert, and a loser reads the row + // on another connection. On the in-memory database that read waited for the loser's own transaction, for 30 seconds. + [Fact] + public async Task TakesThatRaceOnTheFirstUseOfANameEndWithinTheirTimeout() + { + await using var host = new FluentCmsTestHost(); + var locks = host.Services.GetRequiredService(); + var watch = Stopwatch.StartNew(); + + var attempts = await Task.WhenAll(Enumerable.Range(0, 10).Select(_ => Task.Run(() => locks.TryAcquireAsync(Name, TimeSpan.FromMilliseconds(500), Token), Token))); + + watch.Elapsed.ShouldBeLessThan(TimeSpan.FromSeconds(10)); + attempts.Count(held => held is not null).ShouldBe(1); + foreach (var held in attempts.OfType()) + { + await held.DisposeAsync(); + } + } + // The second instance starts after the first has migrated the database, as the instances of a deployment do. private (IDistributedLock First, IDistributedLock Second) BothInstances() {