Add support for aggregate and group-by expressions to continuous aggregates

.claude/reference/architecture.md

@@ -119,7 +119,7 @@ Drops negative (before EF table drops, reverse dependency order); adds/alters po
 
 ## Continuous Aggregate Notes
 
-- Operation properties: `MaterializedViewName`, `ParentName` (entity name, resolved via EF metadata), `TimeBucketWidth`, `TimeBucketSourceColumn`, `AggregateFunctions` (colon-delimited wire format, see patterns.md), `GroupByColumns`, `WhereClause` (raw SQL, emitted verbatim — quoted identifiers must match resolved column names)
+- Operation properties: `MaterializedViewName`, `ParentName` (entity name, resolved via EF metadata), `TimeBucketWidth`, `TimeBucketSourceColumn`, `AggregateFunctions` (colon-delimited wire format, see patterns.md), `GroupByColumns`, `WhereClause` (raw SQL, emitted verbatim — quoted identifiers must match resolved column names), `AggregateExpressions` / `GroupByExpressions` (`ContinuousAggregateExpression` lists, see patterns.md)
 - `first()`/`last()` take the time-bucket column as second argument: `last("price", "timestamp")`
 - Aggregate column aliases must match property names for EF mapping
 - **Bucket column naming**: `WithTimeBucketProperty` designates the property whose mapped column becomes the view's bucket alias; without it the alias falls back to `time_bucket` (snapshot compatibility). Renaming the bucket column is a structural change (drop + recreate).

.claude/reference/patterns.md

@@ -77,7 +77,9 @@ SqlBuilderHelper.QuoteIdentifier("my_column");                     // "my_column
 
 ## 7. Continuous Aggregate Function Encoding
 
-Typed API: `ContinuousAggregateFunction(Alias, Function, SourceColumn)`. Wire format on the operation: `"alias:Function:sourceColumn"` — always exactly three parts; malformed entries are skipped on parse. `First`/`Last` take no time column in the wire format; the SQL generator supplies the time-bucket column as second argument (`last("price", "timestamp")`).
+Typed API: `ContinuousAggregateFunction(Alias, Function, SourceColumn)`. Wire format on the operation: `"alias:Function:sourceColumn"` — always exactly three parts; malformed entries are skipped on parse and raise the `ContinuousAggregateEntryIgnored` warning at model validation. `First`/`Last` take no time column in the wire format; the SQL generator supplies the time-bucket column as second argument (`last("price", "timestamp")`).
+
+Expressions: `ContinuousAggregateExpression(Alias, Sql, Columns)` for aggregates and grouped expressions the enum cannot express. `Sql` is a template with `{n}` placeholders (`Internals/ColumnTemplate` parses and formats them, skipping quoted text; `{{`/`}}` escape braces). Annotations `AggregateExpressions` / `GroupByExpressions` hold JSON lists with model names (`Internals/ContinuousAggregateExpressionJson`); the extractor resolves aliases against the view and columns against the parent, throwing when a column does not resolve. Group-by expressions are selected with their alias and repeated in GROUP BY.
 
 ## 8. Expression-Based Configuration
 

docs/03-scaffolding.md

@@ -42,4 +42,4 @@ Scaffolding reproduces what the database stores, which is not always the literal
 - **CLR type fidelity is not preserved**: custom mapped types (e.g. NodaTime `LocalDateTime`) scaffold as BCL types with an explicit store type (`[Column(TypeName = "timestamp without time zone")] DateTime`). The column definition is identical; re-apply the custom type mapping manually if needed.
 - **`InitialStart` sub-microsecond precision is lost**: PostgreSQL stores microseconds, so .NET's seventh fractional digit (100 ns ticks) does not survive — `…19.3905112Z` scaffolds as `…19.3905110Z`.
 - **Non-column GROUP BY expressions have no data-annotations representation**: in `--data-annotations` mode they are reported as a warning and must be configured via the Fluent API's `AddGroupByColumn(string)`.
-- **Unparseable view definitions degrade gracefully**: when a continuous aggregate's view SQL cannot be parsed into typed configuration, a warning is reported and the configuration is preserved as `.HasAnnotation(...)` calls; migrations still recreate the view correctly from the raw SQL definition.
+- **Views the builder cannot express are scaffolded as SQL**: when a continuous aggregate's view has a column that is neither the time bucket, a supported aggregate function nor a plain GROUP BY column (for example a toolkit sketch such as `mcv_agg(...)` or a grouped expression), a warning is reported and the entity is scaffolded with `IsContinuousAggregateFromSql(...)` holding the view SQL verbatim. In `--data-annotations` mode the configuration is preserved as `.HasAnnotation(...)` calls instead. Either way migrations recreate the view from the same SQL.

docs/data-annotations/continuous-aggregates.md

@@ -367,7 +367,33 @@ public class TradeHourlyAggregate
 }
 ```
 
-> :warning: **Note:** Raw SQL GROUP BY expressions (e.g. `EXTRACT(HOUR FROM time)`) cannot be expressed as an attribute. Use the [Fluent API](../fluent-api/continuous-aggregates#grouping-data)'s `AddGroupByColumn(string)` overload for those.
+## Aggregate and Group-By Expressions
+
+`[AggregateExpression]` defines a property as a SQL aggregate expression, and `[GroupByExpression]` as a grouped expression selected as a column. Column references in the SQL are `{0}`, `{1}`, … placeholders bound to the source columns that follow, which resolve to mapped column names. Literal braces are written as `{{` and `}}`.
+
+```csharp
+[Keyless]
+[ContinuousAggregate(
+    MaterializedViewName = "trade_daily_stats",
+    ParentName = nameof(Trade))]
+[TimeBucket("1 day", nameof(Trade.Timestamp))]
+public class TradeDailyAggregate
+{
+    [GroupByExpression("{0} >= 1000", nameof(Trade.Size))]
+    public bool IsBlockTrade { get; set; }
+
+    [AggregateExpression("count(*) FILTER (WHERE {0} = 'NASDAQ')", nameof(Trade.Exchange))]
+    public long NasdaqTrades { get; set; }
+
+    // Requires the timescaledb_toolkit extension.
+    [AggregateExpression("mcv_agg(10, {0})", nameof(Trade.Ticker))]
+    public string TopTickers { get; set; } = string.Empty;
+}
+```
+
+See the [Fluent API](../fluent-api/continuous-aggregates#aggregate-expressions) page for validation rules and rollups in hierarchical aggregates.
+
+> :warning: **Note:** A raw `AddGroupByColumn(string)` expression appears only in `GROUP BY` and has no attribute form; use `[GroupByExpression]` to select it as a column instead.
 
 ## Refresh Policies
 

docs/fluent-api/continuous-aggregates.md

@@ -86,6 +86,35 @@ public void Configure(EntityTypeBuilder<TradeAggregate> builder)
 }
 ```
 
+### Aggregate Expressions
+
+For aggregates outside `EAggregateFunction`, such as filtered counts, `bool_or` or [timescaledb_toolkit](https://docs.tigerdata.com/api/latest/hyperfunctions/) sketches, use `.AddAggregateExpression()`. The SQL is a template whose column references are `{0}`, `{1}`, … placeholders bound to the selectors that follow it, so each reference resolves to its mapped column name (including naming conventions such as snake case). Literal braces are written as `{{` and `}}`; braces inside string literals and quoted identifiers are left alone.
+
+```csharp
+public void Configure(EntityTypeBuilder<TradeAggregate> builder)
+{
+    builder.HasNoKey();
+
+    builder.IsContinuousAggregate<TradeAggregate, Trade>(
+            "trade_daily_stats",
+            "1 day",
+            x => x.Timestamp)
+        .AddAggregateExpression(x => x.LargeTrades, "count(*) FILTER (WHERE {0} >= 1000)", x => x.Size)
+        .AddAggregateExpression(x => x.PriceRange, "max({0}) - min({0})", x => x.Price)
+        // Requires the timescaledb_toolkit extension: modelBuilder.HasPostgresExtension("timescaledb_toolkit")
+        .AddAggregateExpression(x => x.TopTickers, "mcv_agg(10, {0})", x => x.Ticker);
+}
+```
+
+The model fails to build when a placeholder has no selector, a selector is never referenced, or a selector does not resolve to a column of the source. A change to an expression's template or columns drops and recreates the aggregate, as with any other structural change.
+
+Hierarchical aggregates combine sketches from their parent with the matching rollup function, referencing the parent aggregate's property:
+
+```csharp
+builder.IsContinuousAggregate<TradeWeekly, TradeAggregate>("trade_weekly_stats", "7 days", x => x.Bucket)
+    .AddAggregateExpression(x => x.TopTickers, "rollup({0})", x => x.TopTickers);
+```
+
 ## Grouping Data
 
 Beyond the time bucket, continuous aggregates can group data by additional columns from the source hypertable. In Data Annotations configuration, the same is expressed with the property-level [`[GroupByColumn]` attribute](../data-annotations/continuous-aggregates#grouping-by-additional-columns); raw SQL expressions are Fluent-API-only.
@@ -126,6 +155,40 @@ public void Configure(EntityTypeBuilder<TradeAggregate> builder)
 }
 ```
 
+A raw `AddGroupByColumn` expression appears only in `GROUP BY`, so no property can read it. To select a grouped expression as a column, use `.AddGroupByExpression()`, which takes a placeholder template like `.AddAggregateExpression()`:
+
+```csharp
+public void Configure(EntityTypeBuilder<TradeAggregate> builder)
+{
+    builder.HasNoKey();
+
+    builder.IsContinuousAggregate<TradeAggregate, Trade>(
+            "trade_hourly_stats_by_size",
+            "1 hour",
+            x => x.Timestamp)
+        .AddAggregateFunction(x => x.AveragePrice, x => x.Price, EAggregateFunction.Avg)
+        // SELECT "size" >= 1000 AS "is_block_trade" ... GROUP BY 1, ("size" >= 1000)
+        .AddGroupByExpression(x => x.IsBlockTrade, "{0} >= 1000", x => x.Size);
+}
+```
+
+## Defining the View in SQL
+
+When a definition cannot be expressed through the builder, `.IsContinuousAggregateFromSql()` takes the SELECT statement verbatim. Identifiers in it are not resolved through the model, so it must use database column names. Options such as `.MaterializedOnly()`, `.WithNoData()`, compression and policies still apply.
+
+```csharp
+builder.IsContinuousAggregateFromSql(
+        "trade_daily_vwap",
+        nameof(Trade),
+        """
+        SELECT time_bucket('1 day', "timestamp") AS "bucket", "ticker",
+               sum("price" * "size") / nullif(sum("size"), 0) AS "vwap"
+        FROM "trades"
+        GROUP BY 1, "ticker"
+        """)
+    .MaterializedOnly(false);
+```
+
 ## Filtering Source Data
 
 Apply filtering conditions to the source hypertable before aggregation using the `.Where()` method.
@@ -201,11 +264,13 @@ The string-based builder used by scaffolded code exposes an equivalent `.WithTim
 
 Structured aggregates (those configured through the builders rather than a raw view definition) are validated at model finalization:
 
-- Duplicate output column names are rejected with an `InvalidOperationException`. The check compares the bucket column, all GROUP BY columns, and every aggregate alias after resolving them to database column names. A source column that collides with the bucket column name is caught at model build.
+- Duplicate output column names are rejected with an `InvalidOperationException`. The check compares the bucket column, all GROUP BY columns, every aggregate alias and every expression alias after resolving them to database column names. A source column that collides with the bucket column name is caught at model build.
+- An expression column that does not resolve to a column of the source, or expressions combined with `IsContinuousAggregateFromSql`, raise an `InvalidOperationException`.
+- An `.AddAggregateFunction(...)` or `[Aggregate]` entry that is malformed, or whose source column does not resolve, is left out of the view and raises the `TimescaleDbEventId.ContinuousAggregateEntryIgnored` warning, which can be escalated with `ConfigureWarnings(w => w.Throw(...))`.
 - A property designated via `.WithTimeBucketProperty(...)` that does not exist on the entity raises an `InvalidOperationException`.
 - An aggregate with no time-bucket designation (`.WithTimeBucketProperty(...)`) and no property mapping to the default bucket column `time_bucket` emits a warning through the configured EF logger. The view still exposes a `time_bucket` column, but it cannot be queried through the entity; previously this surfaced only at query time as an opaque Postgres "column does not exist" error. Remedy by designating a property or mapping one to `time_bucket`. This is a warning rather than an exception because deliberately not exposing the bucket is legal. The warning is raised as `TimescaleDbEventId.TimeBucketColumnUnmapped` and can be suppressed with `ConfigureWarnings(w => w.Ignore(...))` or escalated to an exception with `w.Throw(...)`.
 
-> :warning: **Note:** Entities scaffolded with a raw view definition are exempt from all checks, because the structured projection fields are unused on that path.
+> :warning: **Note:** Aggregates defined with `IsContinuousAggregateFromSql` (including scaffolded ones) are exempt from the other checks, because the structured projection fields are unused on that path.
 
 ## Hierarchical Continuous Aggregates