Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/content-migrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,6 @@ With `Auto`, startup skips the content stream and applies additive changes by co
| Check constraint of a new JSON column | Reported, because SQLite adds one only by rebuilding the table |
| Dropped table, column or index; changed column type, nullability or index | Reported |

Reported changes are logged as warnings and never applied. To get them, recreate the development database. Instances that start together on one database apply each change once: an instance with something to apply takes the `fluentcms:schema-sync` lock and compares again before it applies. It waits up to 15 minutes for the lock and then fails to start with an error that names it. Check the `owner` and `lease_until` of the lock's row in `cms_lock`: a `lease_until` that keeps moving means its holder is still working, and one that stands still ahead of the instance's clock means its holder stopped and the clocks of the instances are not in step. On SQLite, one that stands still behind the clock can also mean that its holder is working inside a long write transaction, which keeps it from renewing until the transaction ends. Once no instance is running, set the row's `owner` to null. A table under a plugin's prefix (`cms_{plugin}_`) belongs to that plugin's stream, so schema sync never reports it as a dropped table. That holds for a plugin the host no longer adds too, while the database has the plugin's history table `cms_{plugin}__migrations`. For a host without content migrations, schema sync records the schema history only when it applied every change and the history counts none as lost, such as a removed type or a field renamed whose values are kept in JSON; such a change keeps the newest row for `fluentcms schema apply`. A host with content migrations records after every sync that left nothing undone, as before.
Reported changes are logged as warnings and never applied. To get them, recreate the development database. Instances that start together on one database apply each change once: an instance with something to apply takes the `fluentcms:schema-sync` lock and compares again before it applies. It waits up to 15 minutes for the lock and then fails to start with an error that names it. Check the `owner` and `lease_until` of the lock's row in `cms_lock`: a `lease_until` that keeps moving means its holder is still working, and one that stands still ahead of the instance's clock means its holder stopped and the clocks of the instances are not in step. On SQLite, one that stands still behind the clock can also mean that its holder is working inside a long write transaction, which keeps it from renewing until the transaction ends. When the log has event 9166 for the lock, it can also mean that the database was busy: another connection kept SQLite's one writer to the end of the wait, and the row may be free. Once no instance is running, set the row's `owner` to null. A table under a plugin's prefix (`cms_{plugin}_`) belongs to that plugin's stream, so schema sync never reports it as a dropped table. That holds for a plugin the host no longer adds too, while the database has the plugin's history table `cms_{plugin}__migrations`. For a host without content migrations, schema sync records the schema history only when it applied every change and the history counts none as lost, such as a removed type or a field renamed whose values are kept in JSON; such a change keeps the newest row for `fluentcms schema apply`. A host with content migrations records after every sync that left nothing undone, as before.

A database that schema sync has changed has no content history, so the first migration of the content stream will not apply to it. Switch to `Off` on a new database.
6 changes: 3 additions & 3 deletions docs/planning/backend-technical-spec.md

Large diffs are not rendered by default.

105 changes: 85 additions & 20 deletions src/FluentCMS.Core/Caching/DatabaseDistributedLock.cs
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,18 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP
private static readonly TimeSpan FirstPoll = TimeSpan.FromMilliseconds(50);
private static readonly TimeSpan LongestPoll = TimeSpan.FromSeconds(1);

// On a database with one writer: how long after a take saw a lease over its next try comes, which gives the holder's
// renewal, trying for the writer at least every 150 milliseconds while it waits, the time to get it when no other
// connection writes (a median of 80 milliseconds after the transaction, at most 660 in 50 measured rounds); and how long
// the beginning of a write transaction takes at most when it did not wait for another connection's. These are SQLite's
// and its driver's timings, kept here beside the lock that relies on them.
private static readonly TimeSpan Confirmation = TimeSpan.FromSeconds(1);
private static readonly TimeSpan LongBegin = TimeSpan.FromMilliseconds(50);

// How long past its timeout a wait goes on confirming a take-over, so that a caller that tries once still takes the lock
// of a holder that stopped, even when a try of the confirmation waited and started it again.
private static readonly TimeSpan LateConfirmation = 5 * Confirmation;

// Who holds a lock, for whoever reads cms_lock; with a GUID per acquisition it stays within the owner column.
private readonly string _instance = $"{Environment.MachineName[..Math.Min(Environment.MachineName.Length, 64)]}/{Environment.ProcessId}";

Expand All @@ -32,63 +44,105 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP
var holder = $"{_instance}/{Guid.NewGuid():N}";
var started = time.GetTimestamp();
var poll = FirstPoll;
Sighting? sighting = null;
while (true)
{
if (await TryTakeAsync(name, holder, cancellationToken) is { } leaseUntil)
var attempt = await TryTakeAsync(name, holder, sighting, cancellationToken);
if (attempt.Until is { } leaseUntil)
{
return new Handle(this, name, holder, leaseUntil);
}

var remaining = timeout == Timeout.InfiniteTimeSpan ? poll : timeout - time.GetElapsedTime(started);
if (remaining <= TimeSpan.Zero)
sighting = attempt.Sighting;
var next = sighting is null ? poll : Confirmation;
var remaining = timeout == Timeout.InfiniteTimeSpan ? TimeSpan.MaxValue : timeout - time.GetElapsedTime(started);
if (remaining <= TimeSpan.Zero && (sighting is null || -remaining >= LateConfirmation))
{
if (attempt.Busy)
{
LockLog.TimedOutBusy(logger, name, timeout);
}

return null;
}

await Task.Delay(remaining < poll ? remaining : poll, time, cancellationToken);
poll = poll * 2 < LongestPoll ? poll * 2 : LongestPoll;
// A confirmation keeps its length when the timeout comes first.
await Task.Delay(sighting is null && remaining < next ? remaining : next, time, cancellationToken);
if (sighting is null)
{
poll = poll * 2 < LongestPoll ? poll * 2 : LongestPoll;
}
}
}

// The end of the new lease, or null while another holder has the lock. On a database with one writer, reading the row or
// beginning the write can give up waiting for another connection's transaction: nothing was taken, and the lock counts
// as held, since that transaction may be the holder's own work, which keeps it from renewing.
private async Task<DateTimeOffset?> TryTakeAsync(string name, string holder, CancellationToken cancellationToken)
// On a database with one writer, reading the row or beginning the write can give up waiting for another connection's
// transaction: nothing was taken, and the lock counts as held, since that transaction may be the holder's own work, which
// keeps it from renewing.
private async Task<Attempt> TryTakeAsync(string name, string holder, Sighting? sighting, CancellationToken cancellationToken)
{
await using var scope = scopes.CreateAsyncScope();
var context = scope.ServiceProvider.GetRequiredService<SystemDbContext>();
var provider = scope.ServiceProvider.GetRequiredService<IDataProvider>();
try
{
return await TakeAsync(context, name, holder, cancellationToken);
return await TakeAsync(context, provider, name, holder, sighting, cancellationToken);
}
catch (Exception exception) when (scope.ServiceProvider.GetRequiredService<IDataProvider>().IsBusy(exception))
catch (Exception exception) when (provider.IsBusy(exception))
{
return null;
return new(null, null, Busy: true);
}
}

private async Task<DateTimeOffset?> TakeAsync(SystemDbContext context, string name, string holder, CancellationToken cancellationToken)
private async Task<Attempt> TakeAsync(SystemDbContext context, IDataProvider provider, string name, string holder, Sighting? sighting, CancellationToken cancellationToken)
{
var current = await context.Set<LockRow>().AsNoTracking().SingleOrDefaultAsync(row => row.Name == name, cancellationToken);
if (current is { Owner: not null } && current.LeaseUntil > time.GetUtcNow())
{
return null;
return default;
}

// The writes are not canceled, so a take that commits always comes back as a handle that renews and releases it. On
// SQLite the transaction begins once the database's one writer is free, so the lease starts from the clock then.
// Opened first, so that only the beginning of the transaction is timed, and not the connection and its pragmas.
await context.Database.OpenConnectionAsync(cancellationToken);
var beginning = time.GetTimestamp();
await using var transaction = await context.Database.BeginTransactionAsync(CancellationToken.None);
var waited = time.GetElapsedTime(beginning) > LongBegin;
var now = time.GetUtcNow();
var until = StorageConventions.ToMicroseconds(now + Lease);

// Read again inside the transaction, so that what the take writes and logs is the row as it is now: the holder may
// have renewed or released the lock while the transaction waited to begin.
current = await context.Set<LockRow>().AsNoTracking().SingleOrDefaultAsync(row => row.Name == name, cancellationToken);
if (current is { Owner: not null })
{
if (current.LeaseUntil > now)
{
return default;
}

// On a database with one writer, a lease runs out while a write transaction keeps the holder's renewals waiting,
// and when it ends this take can get the writer before the renewal does. So a take-over gives the renewal time to
// get the writer: it takes only a lease that its try before, a confirmation ago, saw over and as it is, in a
// transaction that began at once, and a transaction that waited starts the confirmation again.
if (provider.HasOneWriter)
{
var same = !waited && sighting is not null && sighting.Owner == current.Owner && sighting.LeaseUntil == current.LeaseUntil;
if (!same)
{
return new(null, new Sighting(current.Owner, current.LeaseUntil), Busy: false);
}
}
}

if (current is null)
{
context.Add(new LockRow { Name = name, Owner = holder, AcquiredAt = now, LeaseUntil = until });
try
{
await context.SaveChangesAsync(CancellationToken.None);
await transaction.CommitAsync(CancellationToken.None);
return until;
return new(until, null, Busy: false);
}
catch (DbUpdateException)
{
Expand All @@ -97,15 +151,19 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP
await transaction.RollbackAsync(CancellationToken.None);
if (await ExistsAsync(name))
{
return null;
return default;
}

throw;
}
}

var taken = await context.Set<LockRow>()
.Where(row => row.Name == name && (row.Owner == null || row.LeaseUntil == null || row.LeaseUntil <= now))
// Conditional on the row as read, so that a take-over that writes took over the lease of the holder it logs.
var rows = context.Set<LockRow>().Where(row => row.Name == name);
rows = current.Owner is { } owner
? rows.Where(row => row.Owner == owner && (row.LeaseUntil == null || row.LeaseUntil <= now))
: rows.Where(row => row.Owner == null);
var taken = await rows
.ExecuteUpdateAsync(
row => row
.SetProperty(column => column.Owner, holder)
Expand All @@ -116,17 +174,24 @@ internal sealed class DatabaseDistributedLock(IServiceScopeFactory scopes, TimeP

if (taken == 0)
{
return null;
return default;
}

if (current.Owner is not null)
{
LockLog.LeaseTakenOver(logger, name, current.Owner);
}

return until;
return new(until, null, Busy: false);
}

// What a take found: the end of the lease it took; or, while it did not, a lease over that it saw and waits to confirm, and
// whether the database kept it waiting for another connection's transaction until it gave up.
private readonly record struct Attempt(DateTimeOffset? Until, Sighting? Sighting, bool Busy);

// A lease over, as a take on a database with one writer saw it; the next try comes a confirmation later.
private sealed record Sighting(string Owner, DateTimeOffset? LeaseUntil);

private async Task<bool> ExistsAsync(string name)
{
await using var scope = scopes.CreateAsyncScope();
Expand Down
6 changes: 5 additions & 1 deletion src/FluentCMS.Core/Caching/LockLog.cs
Original file line number Diff line number Diff line change
Expand Up @@ -11,8 +11,9 @@ internal static partial class LockLog
public const int ReleaseFailedId = 9163;
public const int WentOnWithoutLockId = 9164;
public const int RenewWaitingId = 9165;
public const int TimedOutBusyId = 9166;

[LoggerMessage(EventId = LeaseTakenOverId, EventName = "LockLeaseTakenOver", Level = LogLevel.Warning, Message = "Took the lock {Name} after the lease of {Holder} ran out; that holder stopped without releasing it.")]
[LoggerMessage(EventId = LeaseTakenOverId, EventName = "LockLeaseTakenOver", Level = LogLevel.Warning, Message = "Took the lock {Name} after the lease of {Holder} ran out; that holder stopped without releasing it or, on a database with one writer, could not renew it behind other writes.")]
public static partial void LeaseTakenOver(ILogger logger, string name, string holder);

[LoggerMessage(EventId = LostId, EventName = "LockLost", Level = LogLevel.Warning, Message = "Lost the lock {Name} while holding it, because {Reason}.")]
Expand All @@ -29,4 +30,7 @@ internal static partial class LockLog

[LoggerMessage(EventId = RenewWaitingId, EventName = "LockRenewWaiting", Level = LogLevel.Information, Message = "The lease of the lock {Name} is renewed once another connection's write transaction ends; the database has one writer, so no other holder can take the lock meanwhile.")]
public static partial void RenewWaiting(ILogger logger, string name);

[LoggerMessage(EventId = TimedOutBusyId, EventName = "LockTimedOutBusy", Level = LogLevel.Warning, Message = "The lock {Name} was not taken within {Timeout}, and the last try gave up waiting for another connection's write transaction; the database has one writer, so the lock may be free while that transaction lasts.")]
public static partial void TimedOutBusy(ILogger logger, string name, TimeSpan timeout);
}
39 changes: 37 additions & 2 deletions src/FluentCMS.Core/Commands/SchemaApplyCommand.cs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
using FluentCMS.Core.Events;
using FluentCMS.Core.Schema;
using FluentCMS.Core.Storage;
using FluentCMS.Providers;
using FluentCMS.Schema;
using Microsoft.EntityFrameworkCore;
using Microsoft.Extensions.DependencyInjection;
Expand Down Expand Up @@ -55,8 +56,20 @@ public override async Task<int> RunAsync(CommandContext context, CancellationTok
await using var held = applying ? await runner.LockContentSchemaAsync(cancellationToken) : null;
using var guarded = CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, held?.Lost ?? CancellationToken.None);

var applied = await runner.GetAppliedSnapshotAsync(guarded.Token);
var plan = await runner.PlanContentSchemaAsync(applied, guarded.Token);
string? applied;
SchemaApplyPlan plan;
try
{
applied = await runner.GetAppliedSnapshotAsync(guarded.Token);
plan = await runner.PlanContentSchemaAsync(applied, guarded.Token);
}
catch (Exception exception) when (LockLost(held, cancellationToken))
{
throw new CommandException(
CommandErrorCodes.SchemaApplyFailed,
$"{Lost(exception)} while the operations were planned, so nothing was applied; run 'fluentcms schema apply' again");
}

var steps = plan.Steps;
var (report, unreadable) = SchemaHistory.Compare(applied, schema);
if (!script)
Expand Down Expand Up @@ -101,6 +114,13 @@ public override async Task<int> RunAsync(CommandContext context, CancellationTok
{
await runner.ApplyContentSchemaAsync(steps, guarded.Token);
}
// First, since a provider can end a command that the lost lock canceled with an exception of its own.
catch (Exception exception) when (LockLost(held, cancellationToken))
{
throw new CommandException(
CommandErrorCodes.SchemaApplyFailed,
$"{Lost(exception)} while the operations were applied; their transaction was rolled back, except what SQLite committed before a table rebuild, which the next run leaves out, and the schema history keeps its last row; run 'fluentcms schema apply' again");
}
catch (DbException exception)
{
throw new CommandException(
Expand All @@ -114,6 +134,12 @@ public override async Task<int> RunAsync(CommandContext context, CancellationTok
{
await runner.RecordSchemaAsync(guarded.Token);
}
catch (Exception exception) when (LockLost(held, cancellationToken))
{
throw new CommandException(
CommandErrorCodes.SchemaNotRecorded,
$"the content tables match schema {hash}, but {Lost(exception)} before the schema history was written; run 'fluentcms schema apply' again with the same options to record it");
}
catch (Exception exception) when (exception is DbException or DbUpdateException or InvalidOperationException)
{
throw new CommandException(
Expand All @@ -127,6 +153,15 @@ await context.Output.WriteLineAsync(steps.Count == 0
return CommandExitCodes.Success;
}

// The lock's Lost canceled the work, and not the caller.
private static bool LockLost(IDistributedLockHandle? held, CancellationToken cancellationToken) =>
held is { Lost.IsCancellationRequested: true } && !cancellationToken.IsCancellationRequested;

// Lost is canceled for every reason the lock goes: another holder took it, its lease ran out, or the host stopped. A
// provider's own exception after it is named.
private static string Lost(Exception exception) =>
$"the lock {MigrationRunner.SchemaSyncLock} was lost{(exception is OperationCanceledException ? "" : $" ({exception.GetType().Name})")}";

private static async Task RefuseAsync(MigrationRunner runner, CancellationToken cancellationToken)
{
var migrations = await runner.GetContentMigrationsAsync(cancellationToken);
Expand Down
Loading
Loading