Skip to content

Consumer loop logs no recovery after a failure run (backoff resets silently) #134

Closed

Description

@gregoryfoster

Filed from CannObserv/replicator#154 (Replicator's backlog epic) by its coordinator, with the operator's approval (2026-11-13). Replicator just closed the same gap in its own fetch-policy tail: CannObserv/replicator#188, shipped in PR CannObserv/replicator#193.

The gap [traced at 34d0af0]

src/processor/consumer.py, run():

  • Every failed step logs "consumer loop error; backing off" at :145-149: a WARNING for a transient error, an ERROR with a traceback otherwise.
  • The next good step resets backoff and backing_off at :138-139 without a log.

So after a broker outage the journal shows when the failures stopped, never that the loop came back. The reader is left inferring recovery from silence. liveness.py sees it (backing_off clears), but nothing reaches the journal.

Why it matters

On 2026-11-11 co-broker rebooted (CannObserv/broker#141). Replicator's four consume loops each logged poll cycle recovered with after_failures. Its policy tail didn't, and that was CannObserv/replicator#188. A cohort-wide reading of one broker incident needs each service to say when it came back.

Suggested shape (your call)

Mirror Replicator's loop.py (poll cycle recovered, extra={"after_failures": n}) or policy.py's _recovered() from CannObserv/replicator#193:

  • count failures in a row next to backoff;
  • on the first good step after one or more, log one INFO record with the count, then reset;
  • no record on a good step that follows no failure.

Test-first: failure then success gives exactly one record; two failure runs give two records; success alone gives none.

No contract or wire change. Only a restart is needed to take effect.

Activity

  1. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Harvest for #134 (coordinator, epic #84, 2026-11-16)

    From #138's closing comment (6927047294), under For #134:

    #134 is the epic's last unblocked child. Its brief comes when the operator asks for it.

  2. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Brief: #134, log the consumer loop's recovery (coordinator, epic #84, 2026-11-16)

    Authorities: the issue body is the spec. This brief adds a pre-flight against origin/main 82f35a8, and the harvest comment above (6928583580) carries #138's notes. Where they disagree, stop and ask. Labels: [measured] read from the live system; [traced] followed through the repo or a sibling's; [stated] an issue or a person says so.

    What changed since filed

    Pre-flight [traced at 82f35a8 unless labelled]

    • The template is Replicator's, merged: #188: the fetch-policy tail logs its recovery after failed cycles replicator#193 (762fd1a, src/worker/policy.py:616-634, _recovered()). It logs one INFO with extra={"after_failures": n} when the count is non-zero, then returns 0. Its consume loop (loop.py:1431-1433) does the same inline as poll cycle recovered. Port the shape: a counter beside backoff; increment in the except; log and reset on the next good step.
    • Where the good step ends:
      • run() resets at :139-140, after step() returns. step() does a due reclaim, then one blocking read (read_block_ms, default 5,000), plus at most one command.
      • So the record lands up to 5 s after the broker answers, or after the first command finishes. After an outage the journal can read: WARNINGs, then ack: complete, then recovered.
      • Weigh: keep it at the reset, which is the issue's shape and Replicator's (recommended: the order is honest about when the loop finished a step), or move it earlier. Earlier would mean inside step() after the read, which spreads the counter across two methods.
    • The boot path is covered, and has a test seed.
      • When the broker is down at start, start() (ensure_group) fails inside the same try. The failures count, and the first good step logs consuming, then the recovery. That ordering is right.
      • tests/test_consumer.py:337-378 (test_a_broker_name_that_does_not_resolve_yet_is_retried) already forces two name-resolution failures before the loop starts, and asserts 2 WARNINGs.
      • Its caplog is set at WARNING (:359). Lower it to INFO and assert exactly one recovery with after_failures == 2.
    • A unit-level template: tests/test_consumer.py:881-907 (test_a_backoff_is_progress_and_says_so) scripts a step sequence and asserts backing_off per step ((False, True, True, True, False)). That's the right harness for the body's three cases: fail then succeed gives 1 record; two runs give 2; success alone gives 0. Add a fourth case: stop during a backoff gives no recovery record. stop.wait() ends the loop before any good step.
    • Name the record.
      • The failure record is consumer loop error; backing off. Recommend consumer loop recovered, so one jq filter on consumer loop reads both ends.
      • Replicator's poll cycle recovered names Replicator's loop, not ours; the shared key is after_failures.
      • Don't name it consuming: scripts/deploy.sh:520 selects message == "consuming" as its readiness proof.
    • No formatter change: processor.logging carries extra= fields through, and INFO gives journald PRIORITY=6 (WARNING and ERROR records reach journald at INFO priority #76's mapping).

    Docs that name the loop's records (git grep for the record and backoff)

    Place Says Recommend
    docs/NETWORK.md:123, § Tailnet flaps, signature consumer loop error; backing off … "until the path is back" add that the run ends with one consumer loop recovered, after_failures n
    docs/NETWORK.md, same signature's last bullet "Recovery on its own: NRestarts=0, and ack: complete resumes" name the record as the positive signal
    docs/MONITORING.md:100, § Liveness "To find out which" a broker outage reads as backing_off: true optional: one line saying the journal shows the end of the run
    spec §3 Health (:391-400) "a broker outage backs off and counts as alive" per AGENTS.md, a behaviour change amends the spec. #76, a logging-only change, did (0203bf1, 42abfca). Lean: a short Amended entry plus a clause here

    The plans/ mentions are history: leave them.

    Hazards to weigh

    • Live proof can't be forced. Nothing in production fails the loop on demand. Before the deploy, the integration test above is the proof. Optionally, also run a scratch-bus demonstration: processor run against the local redis-server, stop it, start it, and read one consumer loop recovered.
    • The deploy is a restart (src/), about 20 s of bus absence. That restart has no failure run, so it logs no recovery record: consuming comes on a fresh process. Expected.
    • Logs only: no change to last_progress, backing_off or the liveness check-in. tests/test_liveness.py should not move.
    • The patch window, Tue 2026-11-17 05:00–10:00Z: its in-guest reboot (if needrestart calls for one) restarts processor, and it's not a broker outage either. Ship before or after it; either order works.

    Template

    Replicator's port above for the code. For the shape of a small src/ change with a deploy: #76 (PR #85).

    Out of scope

  3. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Brief check against 82f35a8 (implementer, 2026-11-16)

    Brief holds; building to it: record consumer loop recovered, after_failures n, INFO, logged at the reset; no record on stop during a backoff. Corrections and additions:

    1. step() can run more than one command before the record. A due reclaim walks the whole PEL and runs every entry idle past reclaim_min_idle_ms (consumer.py:156-158, reclaim()), then the read takes at most one new command. So the record trails the broker's return by the reclaim walk plus one read or command. In practice the PEL holds at most the in-flight entry. Doesn't change the recommendation: keep it at the reset.
    2. tests/test_consumer.py is module-marked integration (:49). The shipping gate never runs it; CI does, and so does uv run pytest -m integration here. The new tests go there anyway (tests mirror source; the seeds are there). The pre-deploy proof is CI's integration job, plus a local run.
    3. The flap query hides the new record. docs/NETWORK.md:151 reads the journal with -p warning; an INFO consumer loop recovered (PRIORITY=6) never shows. Adding a line for it to that block, not dropping -p warning.
    4. One more doc place: docs/NETWORK.md:160, "Log a flap … the window (first WARNING to the first ack: complete after it)". On an idle queue there's no ack: complete; the recovery record ends the window either way. Changing it to the record (falling back to ack: complete before this deploy).
    5. consumer.py's module docstring (run bullet, :25-28) describes the loop's records; it gets a clause too.

    Spec: a short Amended 2026-11-16 entry plus the §3 Health clause, as the brief leans. MONITORING.md: one line. Plan: docs/plans/2026-11-16-134-consumer-recovery-log.md (#76 had one).

  4. added 2 commits that reference this issue on Nov 16, 2026
  5. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Post-merge host steps (implementer, 2026-11-16)

    PR #141, final 1b5e306 (CI run 43955801114 green: lint, test). Ship is held for the operator. Once it says go:

    1. Merge from the deploy checkout (/home/exedev/processor, on main): git fetch origin && git merge --ff-only origin/134-consumer-recovery-log && git push origin main. Confirm gh pr view 141 says MERGED.
    2. Deploy: the merge touches src/, so run scripts/deploy.sh. It's a restart, about 20 s of bus absence. No generation bump (processor_version stays 0.38.0+4), no Watcher notice, no lock move, no env or /etc/processor/ change. Deploy within 8 h of the push, or the drift check alerts.
    3. Verify: cat /srv/processor/live/REVISION is the merge SHA; deploy.sh's own proof (starting, child_containment: required, then consuming; the smoke run) passes. No consumer loop recovered on this restart: a fresh process has no failure run. That's expected.
    4. Patch window Tue 2026-11-17 05:00–10:00Z: ship before or after it. Its reboot is a restart too, not a broker outage, so it gives no recovery record either.
    5. Live proof (verification debt on Epic: backlog burn-down 2 — #143 open; #54 and #140 parked on triggers #84): at the next real failure run (a tailnet flap, Redis 7.0 is upstream EOL (2024-07-29): decide the broker's next Redis broker#85's Valkey window), read:
      journalctl -u processor -o cat --since <start> | jq -cR 'fromjson? | select(.message | startswith("consumer loop")) | {timestamp, level, message, after_failures}'
      Expected: the run's consumer loop error; backing off WARNINGs, then one INFO consumer loop recovered with after_failures equal to their count, PRIORITY=6.
  6. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Shipped (#134)

    PR #141, FF-merged 1b5e306 (CI run 43957304130 green on main). Live 1b5e30629b17 since 2026-11-16T21:13:08Z: scripts/deploy.sh exit 0; pid 294004 logged starting (0.38.0+4, unchanged; child_containment: required), then consuming; the smoke run passed; NRestarts=0. No consumer loop recovered on the restart, as expected (no failure run).

    • Consumer.run() counts failed steps in a row. The first good step after one or more logs one INFO consumer loop recovered, extra={"after_failures": n}, then resets (#188: the fetch-policy tail logs its recovery after failed cycles replicator#193's shape). No record on a good step after none, nor when a stop ends a backoff (1e075c6).
    • Tests (integration): 3 failures then good → 1 record (after_failures == 3, after the WARNINGs); two runs → [2, 1]; good only → none; stop during a backoff → none; the boot path (broker.invalid, 2 failures) → 1 record, after consuming.
    • Docs: spec Amended 2026-11-16 + §3 Health; NETWORK.md § Tailnet flaps (signature, a query for the INFO record that -p warning hides, the window ends at the record); MONITORING.md; plan docs/plans/2026-11-16-134-consumer-recovery-log.md.
    • Review: CR 1 (dd7685a, after_failures counts the loop's error records, not every WARNING), CR 2 (1b5e306, test helper simplified); CR 3 stet (red and green in one commit; red run seen in session).

    Range: 82f35a8..1b5e306. Host steps 1–3 done; 4 is moot (shipped before the window); 5 is below (host steps).

    Follow-up:

    • Optional: an outage duration on the record (after_s). The journal's timestamps bracket the run already, and Replicator logs the count only; not filed.

    Known limits:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions