Skip to content

fix(integrations): vendor refusals and outages are connection health, not error issues - #2708

Merged
Makisuo merged 3 commits into
mainfrom
triage/integrations-health
Nov 16, 2026
Merged

Makisuo merged 3 commits into
mainfrom
triage/integrations-health

Conversation

@Makisuo

@Makisuo Makisuo commented Nov 16, 2026 •

Copy link
Copy Markdown
Collaborator

What

Integration pollers opened an error issue every time a customer's grant, plan or a vendor outage refused a call. These failures are connection health: the poller already writes them to the connection's state and source health. This stops them from failing spans.

  • Vendor HTTP kit (vendor-http.ts): the VendorHttp.request attempt span now fails only when the answer can't be decoded. Refusals (401/402/403/404/429), vendor 5xx, transport failures and timeouts keep error.type and the status code on an Ok span. The caller still gets the same typed error. Cloudflare retries a 403 "auth blip" up to 9 times per call, and every attempt used to be a separate error event.
  • Cloudflare analytics: authz, revoked, billing and upstream dataset failures log a warning. They still land on the rows' lastError, the poll summary and source health. Only other (an unrecognised GraphQL error, which could be our query) still raises CloudflareAnalyticsPollError. One org whose token can't read workers/DO/queue analytics raised 4 errors every 5 minutes, ~8k events since 11-08.
  • vendorCallSpan: a new helper for client wrapper spans. A failure the caller handles is recorded as error.type instead of failing the span. Applied to the Supabase reads (queryLogs, listLogDrains, getAdvisors, getTopQueries) and to Vercel createDrain, whose plan and permission refusals are already turned into a 400 the user can act on.

Unhandled failures still fail the caller's own span or the HTTP server span, so real bugs stay visible.

Verification

  • bunx vitest run on vendor-http, CloudflareAnalyticsService, GcpMetricsService, GithubAppClient.span, CloudflareApi, Supabase{Connection,Insights,Logs}Service, supabase/logs and VercelIntegrationService: 7 files, 201 tests pass.
  • New tests:
    • fails the attempt span only on an undecodable answer, not on a refusal or outage
    • vendorCallSpan keeps an expected failure off the span and still fails the call
    • a zone authz failure is the connection's health, not an exception event: fails without the fix (9 failed spans)
    • an unrecognised GraphQL error still fails its span
  • tsc --noEmit for packages/backend, oxlint -c .oxlintrc.effect.json on the changed files.

🤖 Generated with Claude Code


Autofix with [code]smith
Open in [code]smith · Need help on this PR? Tag @codesmith-bot with what you need.


Devin Review

… not error issues

A customer's missing permission, plan limit or a vendor 5xx used to fail a span on
every poll and open an error issue. The vendor HTTP attempt span now fails only on an
answer it cannot decode; Cloudflare analytics logs tenant and vendor failures as a
warning on the rows' health; Supabase reads and Vercel drain creation keep handled
refusals off their spans.
@coderabbitai

coderabbitai Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 11 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used all 4 included reviews currently available.

Learn how review limits work.

Review configuration:

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 653e2846-d01e-438d-b968-7fab4eb18249
Commits

Reviewing files that changed from the base of the PR and between fdd5135 and 9fda746.

Files selected for processing: 10

  • packages/backend/src/services/integrations/CloudflareAnalyticsService.test.ts
  • packages/backend/src/services/integrations/CloudflareAnalyticsService.ts
  • packages/backend/src/services/integrations/GcpMetricsService.test.ts
  • packages/backend/src/services/integrations/supabase/api.ts
  • packages/backend/src/services/integrations/supabase/insights.ts
  • packages/backend/src/services/integrations/supabase/logs.test.ts
  • packages/backend/src/services/integrations/supabase/logs.ts
  • packages/backend/src/services/integrations/vendor-http.test.ts
  • packages/backend/src/services/integrations/vendor-http.ts
  • packages/backend/src/services/integrations/vercel/api.ts
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@maple-review-bot

This comment has been minimized.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔍 Devin Review: 1 flag

Not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

Devin Review

@maple-review-bot

This comment has been minimized.

@Makisuo

Makisuo commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

@maple-review-bot won't fix F3: the { error } body from the Supabase logs endpoint is how it rejects the SQL we send, so it can be our bug and should stay a failed span. F2: the doc block was rewritten in 1c197b5; the remaining comment at :1246 sits after the early warning return, so it only describes the other path, which still records the exception event.

@maple-review-bot

Copy link
Copy Markdown

Checked the head (1c197b5) in CloudflareAnalyticsService.ts:1227-1272, supabase/logs.ts:81-107, supabase/api.ts:22-26 and vendor-http.ts:164-186.

F2 — resolved. The doc block and the exception-event comment now match the code: the warning return is at 1247-1249, so the comment at 1251-1253 only describes the other path, which does fail inside the withSpan and record the event.

F3 — dismissed, you're right. logsQuerySql is a fixed string Maple writes (logs.ts:50), so a 200 {error} is the endpoint rejecting our SQL, not a refusal (401/402/403) or an outage — it is not the class of failure this PR moves to health, and it's already on the connection row plus a warning (SupabaseLogsService.ts:358-364). Failing the span is the right signal for it.

One inversion worth a glance while you're here: in the same classifier the genuinely-ours case is now quiet. An undecodable 200 body becomes SupabaseApiError with kind: "upstream" (logs.ts:94-99, insights.ts:125-126), which supabaseCallSpan names as expected (api.ts:24-26), so it reports Ok — while the SQL rejection reports Error. vendor-http.ts:164-165 keeps exactly that distinction (failsAttemptSpan only for invalid_response), but these callers decode after .text(), so they never reach it. If that's not intended, a distinct kind for the decode mismatch would restore it.

Dismissed F3. Marked F2 resolved.

@Makisuo

Makisuo commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator Author

Inversion fixed in 9fda746: Supabase decode mismatches now carry kind invalid_response (logs, insights, paged reads and the vendor mapping), which supabaseCallSpan does not treat as expected, so they fail the span again. Test added in supabase/logs.test.ts.

@maple-review-bot

maple-review-bot Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Maple review

🟢 Confidence 10/10 · safe to merge. The classifier the four wrappers use is pinned by a test through supabaseCallSpan itself; the wiring only moves each span onto that helper.
no findings · tests covered · risk medium · 1/1 new units observable

Adds a supabaseCallSpan/vendorCallSpan wrapper so a refusal or outage stays on the span as error.type while an undecodable answer still fails it, and gives Supabase decode mismatches their own invalid_response kind. Safe to merge.

  • supabaseCallSpan keeps a refusal Ok and fails only invalid_response
  • Supabase decode mismatches now carry kind: "invalid_response"
  • queryLogs, listLogDrains, getAdvisors, getTopQueries spans move to that helper

9fda746 · What was checked, coverage and telemetry details are on the Maple / review check. Reply "won't fix" to dismiss a finding, or mention @maple-review-bot to ask about one.

@Makisuo
Makisuo merged commit a17fff3 into main Nov 16, 2026
40 checks passed
@Makisuo
Makisuo deleted the triage/integrations-health branch November 16, 2026 16:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant