Skip to content

perf(promql): evaluate range functions and aggregations in ClickHouse - #2664

Open
Makisuo wants to merge 24 commits into
promql/2-warehousefrom
promql/3-pushdown
Open

Makisuo wants to merge 24 commits into
promql/2-warehousefrom
promql/3-pushdown

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Native PromQL, part 3: evaluate range functions and aggregations in ClickHouse.

Why

A selector the evaluator reads returns every bucket of every series, about 140 bytes of JSON each. A 24 h panel over 1,000 series at a 60 s step was 193 MB of JSON and 350 MB of heap, far past a Worker's 128 MB.

What changed

  • The common panel shapes run in ClickHouse whole: a range function over a selector (rate, increase, delta, irate, idelta, resets, changes, the *_over_time family) and sum/avg/min/max/count/group by (...) over one or over a bare selector. Each bucket fans out to the steps whose window holds it, and the window is recombined with the evaluator's exact rules, so one value per series or group and step comes back.
  • The source declines a pushdown it cannot make exact (more than one stored metric, an le matcher) and the evaluator reads the selector as before.
  • A selector read past 24 MB fails with a clear execution error instead of exhausting the isolate.

Evidence

apps/cli/scripts/bench-promql.ts, 12.7 M rows on chDB (1,000 counter series and 200 histogram series, 15 s scrape over 24 h):

query JSON heap eval SQL
24 h, 60 s step, sum by (status) (rate(...[4m])) 193 MB → 0.1 MB 351 → 30 MB 1,062 → 2 ms ~1.5 s, unchanged
24 h, 5 m step, sum by (route) (rate(...[5m])) 41 MB → <0.1 MB 94 → 30 MB 250 → 3 ms ~1 s
6 h p95 histogram_quantile 32 → 5 MB 82 → 38 MB 271 → 98 ms ~0.9 s

Every corpus query still equals the reference with pushdown on and off (chDB), and more than 100 corpus subtrees were pushed.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

A selector the evaluator reads returns every bucket of every series, about
140 bytes of JSON each. A 24 h Grafana panel over 1,000 series at a 60 s
step is 193 MB of JSON and 350 MB of heap, far past a Worker's 128 MB.

The common panel shapes now run in ClickHouse whole: a range function
over a selector (rate, increase, delta, irate, idelta, resets, changes,
the *_over_time family) and sum, avg, min, max, count or group by
labels over one or over a bare selector. Each bucket fans out to the
steps whose window holds it and the window is recombined with the
evaluator's exact rules, so one value per series or group and step
comes back. The source declines a pushdown it cannot make exact (more
than one stored metric, an `le` matcher) and the evaluator reads the
selector as before.

On 12.7 M rows in chDB (scripts/bench-promql.ts): the 24 h, 60 s-step
`sum by (status) (rate(...[4m]))` went from 193 MB of JSON, 351 MB of
heap and 1,062 ms of evaluation to 0.1 MB, 30 MB and 2 ms; SQL time is
unchanged at about 1.5 s. Every corpus query still equals the reference
with pushdown on and off (apps/cli/test/promql-engine.test.ts), and a
selector read past 24 MB now fails with a clear execution error instead
of exhausting the isolate.
# Conflicts:
#	lib/promql/src/engine/index.ts
#	lib/promql/src/engine/plan.ts
#	packages/backend/src/services/promql/PromqlQueryService.test.ts
#	packages/backend/src/services/promql/PromqlQueryService.ts
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 9 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: f0f00679-fc9d-42fd-bde1-814313094865
Commits

Reviewing files that changed from the base of the PR and between 5c23298 and 2564734.

Files selected for processing: 13

  • apps/cli/scripts/bench-promql.ts
  • apps/cli/test/promql-engine.test.ts
  • knip.json
  • lib/promql/scripts/compliance/queries.ts
  • lib/promql/src/engine/evaluate.ts
  • lib/promql/src/engine/index.ts
  • lib/promql/src/engine/plan.ts
  • lib/promql/src/engine/types.ts
  • packages/backend/src/services/promql/PromqlQueryService.test.ts
  • packages/backend/src/services/promql/PromqlQueryService.ts
  • packages/query-engine/src/ch/queries/promql.ts
  • packages/query-engine/src/promql/index.ts
  • packages/query-engine/src/promql/source.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@Makisuo
Makisuo added this pull request to stack #2668 November 15, 2026 14:45

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 5 potential issues.

Devin Review

Comment thread lib/promql/src/engine/evaluate.ts Resolved

@maple-review-bot maple-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 inline note from Maple's review. The score and summary are in the review comment above.

# Conflicts:
#	lib/promql/src/engine/evaluate.ts
A pushed-down subtree past the series cap now fails like a selector read
does instead of returning a truncated answer. sum by (__name__) over a
range function no longer carries the metric name, which every pushed
function but last_over_time drops. Grouping by le on data that is not a
histogram bucket is left to the evaluator. Both name cases join the
compliance corpus: 420 of 420 match Prometheus 3.5.
@maple-review-bot

This comment has been minimized.

# Conflicts:
#	packages/backend/src/services/promql/PromqlQueryService.ts
@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

maple-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. Window and bucket math is pinned by the chDB on/off parity corpus and the compliance queries; the open groupArray pairing finding is untouched and the new 24 MB ceiling is tighter than needed.
2 notes · tests covered · risk medium · 1/1 new units observable

Part 3 of native PromQL: planPushdowns offers range-function and by (...) subtrees, ClickHouse evaluates each (series, step) window from fanned-out buckets with the evaluator's own recombination rules, and fromPushed feeds the result back as steps. The window arithmetic is right and matches the reference with pushdown on and off.

  • planPushdowns offers func(selector[range]) and sum|avg|min|max|count|group by subtrees.
  • ClickHouse fans each bucket out to the steps whose window holds it, then recombines.
  • fromPushed drops __name__ except for last_over_time and pushed aggregates.
  • Selector and pushdown reads now go through compiledQueryBounded with a 24 MB ceiling.

Findings

🔵 Note · F4 · MAX_RESPONSE_BYTES rejects selector reads that fit the isolate

correctness · packages/backend/src/services/promql/PromqlQueryService.ts:66

At ~140 bytes per bucket, 24 MB is about 170k samples: a 24 h panel of 15 s scrapes fails past ~30 series, and raw-sample functions (quantile_over_time, stddev_over_time) never push down so they always hit it. The PR's own numbers put 193 MB of JSON at ~350 MB of heap, so ~60 MB reads used to complete.

Raise `MAX_RESPONSE_BYTES` to ~64 MB: at the measured JSON-to-heap ratio that still fits a 128 MB isolate and keeps the larger panels that work today rendering.
🤖 Prompt to fix this finding with an AI agent
Findings from an automated review of commit aca29ab354220a09177f67a7438114466ed96453. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F4 · Note · correctness · packages/backend/src/services/promql/PromqlQueryService.ts:66
`MAX_RESPONSE_BYTES` rejects selector reads that fit the isolate
At ~140 bytes per bucket, 24 MB is about 170k samples: a 24 h panel of 15 s scrapes fails past ~30 series, and raw-sample functions (`quantile_over_time`, `stddev_over_time`) never push down so they always hit it. The PR's own numbers put 193 MB of JSON at ~350 MB of heap, so ~60 MB reads used to complete.
Suggested fix: Raise `MAX_RESPONSE_BYTES` to ~64 MB: at the measured JSON-to-heap ratio that still fits a 128 MB isolate and keeps the larger panels that work today rendering.

Still open from earlier reviews

aca29ab · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@maple-review-bot

Copy link
Copy Markdown

Note

⏳ Maple is reviewing this pull request at 2564734. This comment updates with the review when it finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant