Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions apps/api/src/routes/v2/anomalies.http.ts
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,7 @@ const toV2Settings = (s: AnomalyDetectorSettingsDocument): V2AnomalySettings =>
enabled: s.enabled,
sensitivity: s.sensitivity,
muted_signals: s.mutedSignals,
included_environments: s.includedEnvironments,
updated_at: s.updatedAt,
updated_by: s.updatedBy,
})
Expand Down Expand Up @@ -252,6 +253,9 @@ export const HttpV2AnomaliesLive = HttpApiBuilder.group(MapleApiV2, "anomalies",
mutedSignals: payload.muted_signals,
}
: undefined),
...(payload.included_environments !== undefined
? { includedEnvironments: payload.included_environments }
: undefined),
}),
)

Expand Down
2 changes: 2 additions & 0 deletions apps/api/src/routes/v2/phase1-resources.http.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -252,6 +252,7 @@ const settingsFixture = new AnomalyDetectorSettingsDocument({
enabled: true,
sensitivity: "normal",
mutedSignals: [],
includedEnvironments: [],
updatedAt: decodeIso("2026-07-15T09:12:00.000Z"),
updatedBy: null,
})
Expand Down Expand Up @@ -1243,6 +1244,7 @@ describe("v2 anomalies over HTTP", () => {
expect(settings.body.object).toBe("anomaly_settings")
expect(settings.body.enabled).toBe(true)
expect(settings.body.muted_signals).toEqual([])
expect(settings.body.included_environments).toEqual([])
await harness.dispose()
})

Expand Down
55 changes: 31 additions & 24 deletions apps/landing/src/content/docs/alerting/anomalies.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,51 +5,57 @@ group: "Alerting"
order: 6
---

Maple compares every service against its own recent history and opens an anomaly when a signal moves well outside it. There are no rules to write and nothing to install: detection starts on every service that sends traces or logs, once it has a few days of history.
Maple compares every service against its own recent history and opens an anomaly when a signal moves well outside it. There are no rules to write and nothing to install: detection starts on every production service that sends traces or logs, once it has a few days of history.

Anomalies are for the problems you did not write an alert for. For a threshold you know you care about, use an [alert rule](/docs/alerting/alert-rules).

## What is watched

| Signal | Per | Opens when |
| ---------------- | ----------------------- | --------------------------------------------------------------------------------------- |
| **Error rate** | service and environment | The share of failed requests is well above its baseline and at least 2 points higher. |
| **p95 latency** | service and environment | p95 latency is well above its baseline and at least 50 ms slower. |
| **Traffic loss** | service and environment | A service with steady traffic drops below 5% of its usual request rate. |
| **Error logs** | service and environment | The rate of error-level logs is well above its baseline. |
| **Error spike** | error fingerprint | One error type occurs far more often in 30 minutes than it does in a typical half hour. |
| Signal | Per | Opens when |
| ---------------- | ----------------------- | ---------------------------------------------------------------------------------------------------------- |
| **Error rate** | service and environment | The share of failed requests is well above its baseline, at least 2 points higher, with at least 5 errors. |
| **p95 latency** | service and environment | p95 latency is well above its baseline and at least 100 ms slower. |
| **Traffic loss** | service and environment | A service with steady traffic drops below 5% of its usual request rate. |
| **Error logs** | service and environment | The rate of error-level logs is well above its baseline. |
| **Error spike** | error fingerprint | One error type occurs far more often in 30 minutes than it does in a typical half hour. |

Traffic loss only fires on near-total loss of a stream that is normally busy. A slow day does not open an anomaly. An error spike only applies to errors Maple has seen for more than a day; a brand-new error opens an [issue](/docs/errors/overview) instead.
Traffic loss only fires on near-total loss of a stream that is normally busy, at least one request a minute. A slow day does not open an anomaly. An error spike only applies to errors Maple has seen for more than a day; a brand-new error opens an [issue](/docs/errors/overview) instead.

### Environments

Maple watches production by default. It skips environments whose `deployment.environment` reads as non-production, such as `staging`, `development`, `dev-eu`, `local`, `qa`, `preview` or `pr-123`. An empty or unrecognised environment counts as production. To watch one of the skipped environments, list it in `included_environments` (see [Tuning](#tuning)).

## How detection works

Maple checks every 5 minutes. For the current hour of the day, it builds a baseline from the same hour, and the hours on either side of it, over the last 7 days. A Monday 14:00 is compared with 13:00 to 15:00 on the previous days, so daily traffic patterns are not anomalies.
Maple checks every 5 minutes and always looks at the last 60 minutes. It compares them with the same 60 minutes on each of the previous 7 days, and with that window an hour earlier and an hour later. At 14:20 on a Monday, the window 13:20 to 14:20 is compared with 12:20 to 15:20 on the previous days, so daily traffic patterns are not anomalies.

The baseline is the median of those windows, and the spread around it is measured robustly, so one bad hour last week does not move it. A signal breaches when it is far outside that spread **and** a set multiple above the median **and** past a minimum absolute change. All three must hold, which keeps quiet services from paging on noise.

The baseline is the median of those hours, and the spread around it is measured robustly, so one bad hour last week does not move it. A signal breaches when it is far outside that spread **and** a set multiple above the median **and** past a minimum absolute change. All three must hold, which keeps quiet services from paging on noise.
Maple also checks the same window on its own. When a service is slow or failing at this time on most previous days, for example during a nightly job, that is its normal and does not open an anomaly.

A series is skipped until it has enough data:

- At least 6 hours of baseline history at matching hours.
- At least 50 requests in the current hour for error rate and latency.
- At least 10 minutes into the current hour for traffic and error logs.
- At least 10 occurrences in 30 minutes for an error spike.
- At least 6 baseline windows with traffic.
- At least 50 requests in the last 60 minutes for error rate and latency.
- At least 30 error logs in the last 60 minutes for error logs.
- At least 20 occurrences in 30 minutes for an error spike.

On busy organizations Maple evaluates the 200 busiest services.

## Open, resolve, reopen

An anomaly opens after 2 breaching checks in a row (10 minutes). Traffic loss needs 3 (15 minutes).

It resolves after 3 healthy checks in a row, or 1 for traffic loss once requests come back. An anomaly with no data for an hour resolves on its own. You can also resolve one by hand.
It resolves after 3 healthy checks in a row, or 1 for traffic loss once requests come back. An anomaly with no breach for an hour resolves on its own. You can also resolve one by hand.

An anomaly that breaches again within 6 hours of resolving reopens the same anomaly rather than opening a new one, and its reopen count goes up. Error spikes on one service and environment are grouped into one anomaly with every fingerprint listed.
An anomaly that breaches again within 6 hours of resolving reopens the same anomaly rather than opening a new one, and its reopen count goes up. Each reopen makes it wait longer before resolving again, up to 3 times as long, so a flapping signal stays one open anomaly instead of resolving and reopening every hour. Error spikes on one service and environment are grouped into one anomaly with every fingerprint listed.

### Severity

Every anomaly is a **warning** unless it is far out:

- **Error rate** is **critical** at 3 times its baseline or 10 points above it.
- **p95 latency** is **critical** at 4 times its baseline.
- **p95 latency** is **critical** at 4 times its baseline and at least 500 ms slower.
- **Error spike** is **critical** at 3 times the spike threshold.

Traffic loss and error logs are always warnings.
Expand All @@ -66,17 +72,18 @@ Traffic loss and error logs are always warnings.

Detection is on for every organization with **normal** sensitivity. An organization admin can change it over the [API](/docs/reference/api) with `PATCH /v2/anomalies/settings`:

| Field | Values | What it does |
| -------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `true`, `false` | Turns detection on or off for the whole organization. |
| `sensitivity` | `low`, `normal`, `high` | How far from baseline a signal must move. `low` needs about 1.5 times the deviation of `normal`, `high` about three quarters of it. |
| `mutedSignals` | `error_rate`, `latency_p95`, `throughput`, `error_spike`, `log_volume` | Signals that never open an anomaly. |
| Field | Values | What it does |
| ----------------------- | ---------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- |
| `enabled` | `true`, `false` | Turns detection on or off for the whole organization. |
| `sensitivity` | `low`, `normal`, `high` | How far from baseline a signal must move. `low` needs about 1.5 times the deviation of `normal`, `high` about three quarters of it. |
| `muted_signals` | `error_rate`, `latency_p95`, `throughput`, `error_spike`, `log_volume` | Signals that never open an anomaly. |
| `included_environments` | environment names, e.g. `["staging"]` | Non-production environments to watch anyway. Up to 50. |

```bash
curl -X PATCH https://api.maple.dev/v2/anomalies/settings \
-H "Authorization: Bearer $MAPLE_API_KEY" \
-H "Content-Type: application/json" \
-d '{"sensitivity": "low", "mutedSignals": ["log_volume"]}'
-d '{"sensitivity": "low", "muted_signals": ["log_volume"], "included_environments": ["staging"]}'
```

Every change is recorded in the organization's audit log.
Original file line number Diff line number Diff line change
Expand Up @@ -45,12 +45,12 @@ const goldenRow = (hourMs: number, errorCount: number) => ({
})

/**
* Matched hours over the last 7 days: the four most recent days ran a 30% error rate
* under maintenance, the three before were at 1%. The current hour runs at 20%.
* The hours the baseline windows read over the last 7 days: the four most recent days ran
* a 30% error rate under maintenance, the three before were at 1%. The current window runs at 20%.
*/
const makeLayer = (testDb: TestDb, currentHourStartMs: number) => {
const baseline = Arr.flatMap(Arr.range(1, 7), (day) =>
Arr.map([-1, 0, 1], (offset) =>
Arr.map([-2, -1, 0, 1], (offset) =>
goldenRow(currentHourStartMs - day * DAY_MS + offset * HOUR_MS, day <= 4 ? 300 : 10),
),
)
Expand Down
Loading
Loading