Skip to content

feat(promql): run PromQL on the metric tables - #2663

Open
Makisuo wants to merge 24 commits into
promql/1-enginefrom
promql/2-warehouse
Open

Makisuo wants to merge 24 commits into
promql/1-enginefrom
promql/2-warehouse

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 15, 2026 •

Copy link
Copy Markdown
Member

Native PromQL, part 2: run PromQL on the metric tables.

What changed

  • packages/query-engine/src/promql: a selector's metric name resolves through metric_catalog to stored metrics the way the OTLP-to-Prometheus translation names them (http.server.request.duration in seconds answers to http_server_request_duration_seconds_bucket, a monotonic sum to ..._total, scraped names match as stored). Each (table, component) becomes one warehouse query that pushes label matchers into WHERE, turns delta sums into running counters, expands explicit histograms into cumulative le series, and returns one row per series with every bucket's statistics.
  • Series labels follow the OTLP mapping: datapoint attributes, promoted resource attributes, and job/instance from service.name and service.instance.id. The series key hashes exactly what the labels are built from, so a deploy that changes service.version keeps one series.
  • PromqlQueryService: validate, plan, one catalog read, the selector queries concurrently, evaluate. A selector may resolve to at most 100 stored metrics, and an org with more catalog rows than one read holds gets an error instead of silently missing names.

How it was verified

  • apps/cli/test/promql-engine.test.ts (chDB, added to the CI chDB list) stores the compliance dataset the way Maple stores it and requires every corpus query to equal the in-memory reference, which matches Prometheus 3.5: all pass.
  • apps/cli/test/promql-source.test.ts: every bucket field against a reference in bucketed and raw mode, all four matcher operators, a counter reset, delta sums, histograms, org isolation, the series cap, and one series across a deploy.
  • Service tests for validation, limits and the catalog guard.

Known limit: a delta-temporality counter's raw value is a running sum from the scan start, so only functions of differences (rate, increase, resets) are exact on it.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

Summary by CodeRabbit

  • New Features
    • Added PromQL query support for stored metrics, including instant and range queries, metric selection, and histogram data.
    • Queries support Prometheus-style metric names, labels, matchers, and units.
    • Added safeguards for invalid queries, excessive time ranges or result sizes, and organization data isolation.

The Maple source behind @maple/promql. A selector's metric name resolves
through metric_catalog to stored metrics the way Prometheus' OTLP
translation names them (`http.server.request.duration` answers to
`http_server_request_duration_seconds_bucket`, a monotonic sum to
`..._total`, scraped names match as stored), and each (table, component)
it resolves to becomes one warehouse query.

That query pushes the label matchers into WHERE, rebuilds counters
(delta sums become running sums, explicit histograms expand into
cumulative `le` series) and returns one row per series with every
bucket's statistics, so the evaluator recombines whole buckets instead
of reading raw samples. Series labels follow the OTLP mapping: datapoint
attributes, promoted resource attributes, and job/instance from
service.name and service.instance.id.

PromqlQueryService runs a query: validate, plan, one catalog read, the
selector queries concurrently, evaluate. apps/cli/test/promql-engine
stores the compliance dataset in chDB the way Maple stores it and every
corpus query must equal the in-memory reference, which matches
Prometheus 3.5.
The series key hashed every resource attribute, but the labels drop
service.*, telemetry.* and maple* keys. A deploy that changed
service.version split one counter into two series with the same labels,
and rate() over the deploy failed with a duplicate labelset. The key now
hashes exactly what the labels are built from.

Pushed-down regexes are anchored as ^(?s:...)$, the way Prometheus and
the evaluator's RE2 anchor them. A selector may resolve to at most 100
stored metrics, the catalog read fails instead of truncating past 50,000
metrics, and resolution limits run before anything is read.
# Conflicts:
#	.github/workflows/ci.yml
#	packages/query-engine/package.json
@coderabbitai

coderabbitai Bot commented Nov 15, 2026 •

Copy link
Copy Markdown

Explain Change →

Note

CodeRabbit ASCII loading animation

Currently processing new changes in this PR. This may take a few minutes, please wait...

Review info

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 6879ff81-179b-48ef-b4a3-fa9ee281f086
Commits

Reviewing files that changed from the base of the PR and between 3d68a80 and f9f31fe.

Files ignored due to path filters: 1

  • bun.lock is excluded by !**/*.lock
Files selected for processing: 18

  • .github/workflows/ci.yml
  • apps/cli/package.json
  • apps/cli/test/promql-engine.test.ts
  • apps/cli/test/promql-source.test.ts
  • lib/promql/package.json
  • lib/promql/scripts/compliance/index.ts
  • packages/backend/package.json
  • packages/backend/src/services/promql/PromqlQueryService.test.ts
  • packages/backend/src/services/promql/PromqlQueryService.ts
  • packages/query-engine/package.json
  • packages/query-engine/src/ch/index.ts
  • packages/query-engine/src/ch/queries/promql.ts
  • packages/query-engine/src/promql/index.ts
  • packages/query-engine/src/promql/labels.test.ts
  • packages/query-engine/src/promql/labels.ts
  • packages/query-engine/src/promql/names.test.ts
  • packages/query-engine/src/promql/names.ts
  • packages/query-engine/src/promql/source.ts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@Makisuo
Makisuo added this pull request to stack #2668 November 15, 2026 14:45

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 5 potential issues.

Devin Review

@maple-review-bot

This comment has been minimized.

…ver splits a series

The catalog took one arbitrary unit per metric name, so a name exported
in ms by one service and in s by another answered to one Prometheus name
with mixed values. Names now resolve per (name, unit) and the scan reads
only the resolved pairs: request_duration_seconds and
request_duration_milliseconds are separate metrics. The series key hashes
sorted attribute maps, so an exporter reordering attributes keeps one
series. A selector read that hit its row limit fails instead of
returning what survived the le check after the limit.
@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot maple-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 inline note from Maple's review. The score and summary are in the review comment above.

Matcher values can carry user identifiers; record the query length instead. The selector and
metric attributes already identify the read.
@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot

This comment has been minimized.

@maple-review-bot maple-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 inline note from Maple's review. The score and summary are in the review comment above.

Without it the OrgId filter did not count, every selector read compiled as untenanted and the
warehouse refused it. The chDB parity suite now asserts every compiled read is single-tenant.
@maple-review-bot

This comment has been minimized.

@maple-review-bot

maple-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Maple review

🟡 Confidence 6/10 · needs attention. Held at 6 because a correctness warning is open.
1 warning −1 · 1 note · tests partial −1 · risk medium · 2/2 new units observable

Runs PromQL on the metric tables: Prometheus names resolve through metric_catalog, one ClickHouse selector query per (table, component) pushes label matchers down and returns bucket statistics, and PromqlQueryService reads the catalog once then evaluates. metric_catalog still keeps one unit per service and hour, so the two-unit resolution only half works.

  • resolveSelectorName maps __name__ matchers onto stored metric, unit and component
  • promqlSelectorQuery pushes matchers into WHERE, accumulates delta sums, expands buckets
  • seriesLabels/labelValueExpr build the same label set in TypeScript and in ClickHouse
  • PromqlQueryService validates, caps metrics and series, and evaluates the fetched rows

Before merge

  • Manual · If metric_catalog gains MetricUnit in its sorting key, run bun run tinybird:manifest and bun run local-schema:bump metric-catalog

Findings

🟠 Warning · F4 · metric_catalog holds one unit per metric, so a second unit's series vanish

correctness · packages/query-engine/src/ch/queries/promql.ts:362

The unit comes from metric_catalog, which rolls up one row per (OrgId, MetricType, ServiceName, MetricName, Hour) with anyLast(MetricUnit) (packages/domain/src/tinybird/datasources.ts:1936; MV GROUP BY at materializations.ts:1875). A service writing request.duration in both s and ms in one hour keeps one unit, so the other name returns an empty series.

Resolve units from the raw metric tables, or add `MetricUnit` to `metric_catalog`'s sorting key and to every catalog MV's `GROUP BY`, so both units are kept.
🤖 Prompt to fix this finding with an AI agent
Findings from an automated review of commit f9f31febd157314d6d3e7d3a039d59e6916d5df0. Verify each one against the current code before changing anything, fix only those that still apply, and keep each fix to the lines it names.

---

F4 · Warning · correctness · packages/query-engine/src/ch/queries/promql.ts:362
`metric_catalog` holds one unit per metric, so a second unit's series vanish
The unit comes from `metric_catalog`, which rolls up one row per (OrgId, MetricType, ServiceName, MetricName, Hour) with `anyLast(MetricUnit)` (`packages/domain/src/tinybird/datasources.ts:1936`; MV `GROUP BY` at `materializations.ts:1875`). A service writing `request.duration` in both `s` and `ms` in one hour keeps one unit, so the other name returns an empty series.
Suggested fix: Resolve units from the raw metric tables, or add `MetricUnit` to `metric_catalog`'s sorting key and to every catalog MV's `GROUP BY`, so both units are kept.

Still open from earlier reviews

f9f31fe · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@maple-review-bot maple-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

1 inline note from Maple's review. The score and summary are in the review comment above.

@maple-review-bot

Copy link
Copy Markdown

Note

⏳ Maple is reviewing this pull request at 5c23298. This comment updates with the review when it finishes.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant