Skip to content

fix(alerts): quieter anomaly detection: rolling 60-minute window, production only by default, floors - #2720

Open
Makisuo wants to merge 9 commits into
mainfrom
fix/anomaly-detector-noise
Open

Makisuo wants to merge 9 commits into
mainfrom
fix/anomaly-detector-noise

fix(alerts): pad the log chart query from the hour so a recent start …

08e4c92
Select commit
Loading
Failed to load commit list.
Maple Review Bot / Maple / review succeeded Nov 16, 2026 in 3m 35s

Confidence 10/10 · No issues found

🟢 Confidence 10/10 · safe to merge. Window blending, environment gating and both chart live points are pinned by new tests; the migration is a NOT NULL default-'[]' column, so no data risk.
no findings · tests covered · risk medium · 2/2 new units observable

Replaces the in-progress-hour comparison with rolling 60-minute windows blended against hourly baselines, skips non-production environments unless the org opts in, and adds floors plus a flapping damper. The new settings field, migration, table column and API mapping are wired end to end.

  • Golden signals read a rolling 60-minute window from service_overview_minutely, hourly rows as fallback
  • Baselines are blended 60-minute windows from service_overview_hourly with a same-time-of-day guard
  • isWatchedEnvironment skips non-production names unless listed in includedEnvironments
  • ERROR_SPIKE_MIN_COUNT and per-signal floors rise; reopenCount extends healthyToResolve up to 3x

Before merge

Fixed since the last review

  • ✅ F3 · Log chart live point drops the previous hour for a recent start_time
  • ✅ F4 · Golden chart's zero live point has no test

Production impact

Production traffic of the changed files (last 7 days)
File Calls/day Busiest operations
packages/backend/src/services/alerts/AnomalyDetectionService.ts 12k AnomalyDetectionService.fetchGoldenSeries 12k/day, 0.0% err
packages/backend/src/services/alerts/anomaly/detection.ts 92 preview 53/day, 0.3% err
test 39/day, 16.3% err
Telemetry this change adds and removes (2)
  • ➕ span name AnomalyDetectionService.fetchGoldenWindow · packages/backend/src/services/alerts/AnomalyDetectionService.ts:1092
  • ➕ attribute query.rollup.fallback · packages/backend/src/services/alerts/AnomalyDetectionService.ts:1109
What was checked
  • windowBaseline weights the straddled hours by f, matching the [now-60m, now) window at both hour edges (detection.ts:184)
  • Both the minutely window and the hourly baseline read the same entry-point rollup (queries/anomaly.ts:192)
  • The log chart query now starts an hour before the first drawn hour and the recent-start_time case is pinned (window.test.ts:353), so F3 and F4 are fixed
Observability coverage: 2 of 2 changes observable
Change Kind Observable Evidence
anomalyTraceSignalsWindowQuery (service_overview_minutely, current 60-minute window) outbound warehouse query yes Runs through withOutboundPermit(warehouse.compiledQuery(..., { context: "anomalyTraceSignalsCurrent" })) at AnomalyDetectionService.ts:1104; the service reports AnomalyDetectionService.fetchGoldenSeries ~12k/day in Maple
AnomalyDetectionService.fetchGoldenWindow span, with a query.rollup.fallback attribute on the hourly fallback background work / span yes Effect.fn("AnomalyDetectionService.fetchGoldenWindow") at AnomalyDetectionService.ts:1093 with Effect.annotateCurrentSpan("query.rollup.fallback", true) at :1109

08e4c92 · Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.