Skip to content

Log orchestrator payment readiness rejections - #92

Merged
j0sh merged 4 commits into
mainfrom
ja-codex/log-chain-readiness-rejections
Nov 18, 2026
Merged

j0sh merged 4 commits into
mainfrom
ja-codex/log-chain-readiness-rejections

Conversation

@j0sh

@j0sh j0sh commented Nov 18, 2026

Copy link
Copy Markdown
Collaborator

A short chain-readiness failure on the staging FFmpeg orchestrator caused its discovery response to lose all 17 app entries while the app health endpoint and heartbeats stayed healthy. The generic readiness failure made it difficult to tell which payment dependency rejected admission.

  • Log the reason for a paid orchestrator's /readyz rejection when the reason changes. A successful check resets the suppression.
  • Distinguish chain-view expiry and invalidation, stale chain heads or L1 progress, failed recipient checks, and confirmed recipient inactivity. Report storage, worker, and price freshness failures separately.
  • Preserve the existing readiness decisions and HTTP response.

Verified with go test -race ./eth ./orchestrator ./internal/observability.

Copilot AI balanced review requested due to automatic review settings November 18, 2026 20:59
@j0sh
j0sh deployed to docker-publish November 18, 2026 21:02 — with GitHub Actions Active

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The logger-enabled test omits recovery-reset and changed-rejection cases, leaving regressions that suppress later outage logs undetected.

1 open issue
What changed in this PR

Adds reason-specific logging for paid orchestrator /readyz rejections, making payment dependency failures easier to diagnose while preserving readiness decisions and HTTP responses.

Changes:

  • Classifies chain, recipient, storage, worker, and price freshness failures.
  • Logs rejection changes and resets duplicate suppression after recovery.
  • Adds assertions for observer rejection reasons and repeated-log suppression.
File Description
orchestrator/​metrics.go Selects readiness rejection reasons and logs changes.
orchestrator/​metrics_test.go Checks repeated rejection-log suppression.
orchestrator/​app.go Passes the logger into paid readiness checks.
eth/​payment_observer.go Exposes specific chain and recipient rejection reasons.
eth/​payment_observer_test.go Verifies rejection reasons across observer states.

Review effort: Balanced

Comment thread orchestrator/metrics_test.go Resolved
@j0sh
j0sh deployed to docker-publish November 18, 2026 21:03 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 18, 2026 21:03 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 18, 2026 21:03 — with GitHub Actions Active
Copilot AI balanced review requested due to automatic review settings November 18, 2026 22:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues remain for the readiness-rejection diagnostic logging changes.

No open issues 🎉

1 resolved since last review

Review effort: Balanced

@j0sh
j0sh deployed to docker-publish November 18, 2026 22:04 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 18, 2026 22:04 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 18, 2026 22:04 — with GitHub Actions Active
@j0sh
j0sh deployed to docker-publish November 18, 2026 22:05 — with GitHub Actions Active
@j0sh
j0sh merged commit 1d060c9 into main Nov 18, 2026
14 checks passed
@j0sh
j0sh deleted the ja-codex/log-chain-readiness-rejections branch November 18, 2026 22:10

This branch was successfully deployed

1 active deployment
docker-publish — 851949d7 Deployed Nov 18, 2026 by j0sh via docker-pr / promote #200
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants