Repository navigation
fix(alerts): quieter anomaly detection: rolling 60-minute window, production only by default, floors - #2720
Open
Makisuo wants to merge 9 commits into
Open
fix(alerts): quieter anomaly detection: rolling 60-minute window, production only by default, floors#2720Makisuo wants to merge 9 commits into
Makisuo wants to merge 9 commits into
Maple Review Bot / Maple / review
succeeded
Nov 16, 2026 in 3m 35s
Confidence 10/10 · No issues found
🟢 Confidence 10/10 · safe to merge. Window blending, environment gating and both chart live points are pinned by new tests; the migration is a NOT NULL default-'[]' column, so no data risk.
no findings · tests covered · risk medium · 2/2 new units observable
Replaces the in-progress-hour comparison with rolling 60-minute windows blended against hourly baselines, skips non-production environments unless the org opts in, and adds floors plus a flapping damper. The new settings field, migration, table column and API mapping are wired end to end.
- Golden signals read a rolling 60-minute window from
service_overview_minutely, hourly rows as fallback - Baselines are blended 60-minute windows from
service_overview_hourlywith a same-time-of-day guard isWatchedEnvironmentskips non-production names unless listed inincludedEnvironmentsERROR_SPIKE_MIN_COUNTand per-signal floors rise;reopenCountextendshealthyToResolveup to 3x
Before merge
- Migration · Check it is safe on production data and with the code still deployed ·
packages/db/drizzle/20261116160757_anomaly_included_environments/migration.sql
Fixed since the last review
- ✅
F3 · Log chart live point drops the previous hour for a recentstart_time - ✅
F4 · Golden chart's zero live point has no test
Production impact
Production traffic of the changed files (last 7 days)
| File | Calls/day | Busiest operations |
|---|---|---|
packages/backend/src/services/alerts/AnomalyDetectionService.ts |
12k | AnomalyDetectionService.fetchGoldenSeries 12k/day, 0.0% err |
packages/backend/src/services/alerts/anomaly/detection.ts |
92 | preview 53/day, 0.3% errtest 39/day, 16.3% err |
Telemetry this change adds and removes (2)
- ➕ span name
AnomalyDetectionService.fetchGoldenWindow·packages/backend/src/services/alerts/AnomalyDetectionService.ts:1092 - ➕ attribute
query.rollup.fallback·packages/backend/src/services/alerts/AnomalyDetectionService.ts:1109
What was checked
windowBaselineweights the straddled hours byf, matching the [now-60m, now) window at both hour edges (detection.ts:184)- Both the minutely window and the hourly baseline read the same entry-point rollup (
queries/anomaly.ts:192) - The log chart query now starts an hour before the first drawn hour and the recent-
start_timecase is pinned (window.test.ts:353), so F3 and F4 are fixed
Observability coverage: 2 of 2 changes observable
| Change | Kind | Observable | Evidence |
|---|---|---|---|
| anomalyTraceSignalsWindowQuery (service_overview_minutely, current 60-minute window) | outbound warehouse query | yes | Runs through withOutboundPermit(warehouse.compiledQuery(..., { context: "anomalyTraceSignalsCurrent" })) at AnomalyDetectionService.ts:1104; the service reports AnomalyDetectionService.fetchGoldenSeries ~12k/day in Maple |
AnomalyDetectionService.fetchGoldenWindow span, with a query.rollup.fallback attribute on the hourly fallback |
background work / span | yes | Effect.fn("AnomalyDetectionService.fetchGoldenWindow") at AnomalyDetectionService.ts:1093 with Effect.annotateCurrentSpan("query.rollup.fallback", true) at :1109 |
08e4c92 · Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.
Loading