You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Consumer loop logs no recovery after a failure run (backoff resets silently) #134
Every failed step logs "consumer loop error; backing off" at :145-149: a WARNING for a transient error, an ERROR with a traceback otherwise.
The next good step resets backoff and backing_off at :138-139without a log.
So after a broker outage the journal shows when the failures stopped, never that the loop came back. The reader is left inferring recovery from silence. liveness.py sees it (backing_off clears), but nothing reaches the journal.
Why it matters
On 2026-11-11 co-broker rebooted (CannObserv/broker#141). Replicator's four consume loops each logged poll cycle recovered with after_failures. Its policy tail didn't, and that was CannObserv/replicator#188. A cohort-wide reading of one broker incident needs each service to say when it came back.
Suggested shape (your call)
Mirror Replicator's loop.py (poll cycle recovered, extra={"after_failures": n}) or policy.py's _recovered() from CannObserv/replicator#193:
count failures in a row next to backoff;
on the first good step after one or more, log one INFO record with the count, then reset;
no record on a good step that follows no failure.
Test-first: failure then success gives exactly one record; two failure runs give two records; success alone gives none.
No contract or wire change. Only a restart is needed to take effect.
Don't run uv lock --upgrade. The lock now carries [tool.uv] override-dependencies = ["soupsieve>=4.1,<5"], and tests/test_lock_floors.py fails if it's lost.
Authorities: the issue body is the spec. This brief adds a pre-flight against origin/main82f35a8, and the harvest comment above (6928583580) carries #138's notes. Where they disagree, stop and ask. Labels: [measured] read from the live system; [traced] followed through the repo or a sibling's; [stated] an issue or a person says so.
The template is Replicator's, merged:#188: the fetch-policy tail logs its recovery after failed cycles replicator#193 (762fd1a, src/worker/policy.py:616-634, _recovered()). It logs one INFO with extra={"after_failures": n} when the count is non-zero, then returns 0. Its consume loop (loop.py:1431-1433) does the same inline as poll cycle recovered. Port the shape: a counter beside backoff; increment in the except; log and reset on the next good step.
Where the good step ends:
run() resets at :139-140, after step() returns. step() does a due reclaim, then one blocking read (read_block_ms, default 5,000), plus at most one command.
So the record lands up to 5 s after the broker answers, or after the first command finishes. After an outage the journal can read: WARNINGs, then ack: complete, then recovered.
Weigh: keep it at the reset, which is the issue's shape and Replicator's (recommended: the order is honest about when the loop finished a step), or move it earlier. Earlier would mean inside step() after the read, which spreads the counter across two methods.
The boot path is covered, and has a test seed.
When the broker is down at start, start() (ensure_group) fails inside the same try. The failures count, and the first good step logs consuming, then the recovery. That ordering is right.
tests/test_consumer.py:337-378 (test_a_broker_name_that_does_not_resolve_yet_is_retried) already forces two name-resolution failures before the loop starts, and asserts 2 WARNINGs.
Its caplog is set at WARNING (:359). Lower it to INFO and assert exactly one recovery with after_failures == 2.
A unit-level template:tests/test_consumer.py:881-907 (test_a_backoff_is_progress_and_says_so) scripts a step sequence and asserts backing_off per step ((False, True, True, True, False)). That's the right harness for the body's three cases: fail then succeed gives 1 record; two runs give 2; success alone gives 0. Add a fourth case: stop during a backoff gives no recovery record. stop.wait() ends the loop before any good step.
Name the record.
The failure record is consumer loop error; backing off. Recommend consumer loop recovered, so one jq filter on consumer loop reads both ends.
Replicator's poll cycle recovered names Replicator's loop, not ours; the shared key is after_failures.
Don't name it consuming:scripts/deploy.sh:520 selects message == "consuming" as its readiness proof.
Docs that name the loop's records (git grep for the record and backoff)
Place
Says
Recommend
docs/NETWORK.md:123, § Tailnet flaps, signature
consumer loop error; backing off … "until the path is back"
add that the run ends with one consumer loop recovered, after_failures n
docs/NETWORK.md, same signature's last bullet
"Recovery on its own: NRestarts=0, and ack: complete resumes"
name the record as the positive signal
docs/MONITORING.md:100, § Liveness "To find out which"
a broker outage reads as backing_off: true
optional: one line saying the journal shows the end of the run
spec §3 Health (:391-400)
"a broker outage backs off and counts as alive"
per AGENTS.md, a behaviour change amends the spec. #76, a logging-only change, did (0203bf1, 42abfca). Lean: a short Amended entry plus a clause here
The plans/ mentions are history: leave them.
Hazards to weigh
Live proof can't be forced. Nothing in production fails the loop on demand. Before the deploy, the integration test above is the proof. Optionally, also run a scratch-bus demonstration: processor run against the local redis-server, stop it, start it, and read one consumer loop recovered.
Expected: this goes on the epic as verification debt, not as a gap in the child.
The deploy is a restart (src/), about 20 s of bus absence. That restart has no failure run, so it logs no recovery record: consuming comes on a fresh process. Expected.
Logs only: no change to last_progress, backing_off or the liveness check-in. tests/test_liveness.py should not move.
The patch window, Tue 2026-11-17 05:00–10:00Z: its in-guest reboot (if needrestart calls for one) restarts processor, and it's not a broker outage either. Ship before or after it; either order works.
Template
Replicator's port above for the code. For the shape of a small src/ change with a deploy: #76 (PR #85).
adding an outage duration. The journal's timestamps already bracket the run, and Replicator logs the count only. Offer it as a follow-up if you think it's worth it.
Brief check against 82f35a8 (implementer, 2026-11-16)
Brief holds; building to it: record consumer loop recovered, after_failures n, INFO, logged at the reset; no record on stop during a backoff. Corrections and additions:
step() can run more than one command before the record. A due reclaim walks the whole PEL and runs every entry idle past reclaim_min_idle_ms (consumer.py:156-158, reclaim()), then the read takes at most one new command. So the record trails the broker's return by the reclaim walk plus one read or command. In practice the PEL holds at most the in-flight entry. Doesn't change the recommendation: keep it at the reset.
tests/test_consumer.py is module-marked integration (:49). The shipping gate never runs it; CI does, and so does uv run pytest -m integration here. The new tests go there anyway (tests mirror source; the seeds are there). The pre-deploy proof is CI's integration job, plus a local run.
The flap query hides the new record.docs/NETWORK.md:151 reads the journal with -p warning; an INFO consumer loop recovered (PRIORITY=6) never shows. Adding a line for it to that block, not dropping -p warning.
One more doc place:docs/NETWORK.md:160, "Log a flap … the window (first WARNING to the first ack: complete after it)". On an idle queue there's no ack: complete; the recovery record ends the window either way. Changing it to the record (falling back to ack: complete before this deploy).
consumer.py's module docstring (run bullet, :25-28) describes the loop's records; it gets a clause too.
Spec: a short Amended 2026-11-16 entry plus the §3 Health clause, as the brief leans. MONITORING.md: one line. Plan: docs/plans/2026-11-16-134-consumer-recovery-log.md (#76 had one).
PR #141, final 1b5e306 (CI run 43955801114 green: lint, test). Ship is held for the operator. Once it says go:
Merge from the deploy checkout (/home/exedev/processor, on main): git fetch origin && git merge --ff-only origin/134-consumer-recovery-log && git push origin main. Confirm gh pr view 141 says MERGED.
Deploy: the merge touches src/, so run scripts/deploy.sh. It's a restart, about 20 s of bus absence. No generation bump (processor_version stays 0.38.0+4), no Watcher notice, no lock move, no env or /etc/processor/ change. Deploy within 8 h of the push, or the drift check alerts.
Verify:cat /srv/processor/live/REVISION is the merge SHA; deploy.sh's own proof (starting, child_containment: required, then consuming; the smoke run) passes. Noconsumer loop recovered on this restart: a fresh process has no failure run. That's expected.
Patch window Tue 2026-11-17 05:00–10:00Z: ship before or after it. Its reboot is a restart too, not a broker outage, so it gives no recovery record either.
Expected: the run's consumer loop error; backing off WARNINGs, then one INFO consumer loop recovered with after_failures equal to their count, PRIORITY=6.
PR #141, FF-merged 1b5e306 (CI run 43957304130 green on main). Live 1b5e30629b17 since 2026-11-16T21:13:08Z: scripts/deploy.sh exit 0; pid 294004 logged starting (0.38.0+4, unchanged; child_containment: required), then consuming; the smoke run passed; NRestarts=0. No consumer loop recovered on the restart, as expected (no failure run).
Consumer.run() counts failed steps in a row. The first good step after one or more logs one INFO consumer loop recovered, extra={"after_failures": n}, then resets (#188: the fetch-policy tail logs its recovery after failed cycles replicator#193's shape). No record on a good step after none, nor when a stop ends a backoff (1e075c6).
Tests (integration): 3 failures then good → 1 record (after_failures == 3, after the WARNINGs); two runs → [2, 1]; good only → none; stop during a backoff → none; the boot path (broker.invalid, 2 failures) → 1 record, after consuming.
Docs: spec Amended 2026-11-16 + §3 Health; NETWORK.md § Tailnet flaps (signature, a query for the INFO record that -p warning hides, the window ends at the record); MONITORING.md; plan docs/plans/2026-11-16-134-consumer-recovery-log.md.
Review: CR 1 (dd7685a, after_failures counts the loop's error records, not every WARNING), CR 2 (1b5e306, test helper simplified); CR 3 stet (red and green in one commit; red run seen in session).
Range: 82f35a8..1b5e306. Host steps 1–3 done; 4 is moot (shipped before the window); 5 is below (host steps).
Follow-up:
Optional: an outage duration on the record (after_s). The journal's timestamps bracket the run already, and Replicator logs the count only; not filed.
The optional scratch-bus demonstration (processor run against local redis, stop/start) was skipped: the integration tests drive the real loop against the scratch Redis, the boot-path one through real name-resolution failures.
The record trails the broker's return by up to one step: a due reclaim walk, then one read (5 s on an idle queue) or one command. Accepted (brief, check comment 6928995289).
Activity
gregoryfoster commentedon Nov 16, 2026
Harvest for #134 (coordinator, epic #84, 2026-11-16)
From #138's closing comment (6927047294), under For #134:
processor_versionis0.38.0+4from Lock advisories: soupsieve 4.1 (extraction child, override past co-core 0.38.0's cap; LOCAL_GENERATION +4) and opentelemetry-api 1.45.2 #138's deploy (live82f35a8, 2026-11-16T19:35Z). Consumer loop logs no recovery after a failure run (backoff resets silently) #134 changes logging only, so it needs no generation bump and no Watcher notice. Its deploy is a restart.uv lock --upgrade. The lock now carries[tool.uv] override-dependencies = ["soupsieve>=4.1,<5"], andtests/test_lock_floors.pyfails if it's lost.processing_failedfrom strict-:has()specs is a deterministic ack, not a consumer-loop error. It doesn't belong in Consumer loop logs no recovery after a failure run (backoff resets silently) #134's recovery records.#134 is the epic's last unblocked child. Its brief comes when the operator asks for it.
gregoryfoster commentedon Nov 16, 2026
Brief: #134, log the consumer loop's recovery (coordinator, epic #84, 2026-11-16)
Authorities: the issue body is the spec. This brief adds a pre-flight against
origin/main82f35a8, and the harvest comment above (6928583580) carries #138's notes. Where they disagree, stop and ask. Labels:[measured]read from the live system;[traced]followed through the repo or a sibling's;[stated]an issue or a person says so.What changed since filed
mainmoved34d0af0→82f35a8(docs: retire Watcher's co-gcs-blob-reader grant on co-gcs-processor (watcher#420) #130 docs, Lock advisories: soupsieve 4.1 (extraction child, override past co-core 0.38.0's cap; LOCAL_GENERATION +4) and opentelemetry-api 1.45.2 #138 lock), butconsumer.pydidn't[traced]. The body's line numbers are off by one at most: the reset isconsumer.py:139-140, and the failure log is:145-149.0.38.0+4(Lock advisories: soupsieve 4.1 (extraction child, override past co-core 0.38.0's cap; LOCAL_GENERATION +4) and opentelemetry-api 1.45.2 #138). This is a logging-only change, so there's no generation bump, no Watcher notice and no lock move. Its deploy is a restart.Pre-flight
[traced at 82f35a8 unless labelled]762fd1a,src/worker/policy.py:616-634,_recovered()). It logs one INFO withextra={"after_failures": n}when the count is non-zero, then returns 0. Its consume loop (loop.py:1431-1433) does the same inline aspoll cycle recovered. Port the shape: a counter besidebackoff; increment in theexcept; log and reset on the next good step.run()resets at:139-140, afterstep()returns.step()does a due reclaim, then one blocking read (read_block_ms, default 5,000), plus at most one command.ack: complete, thenrecovered.step()after the read, which spreads the counter across two methods.start()(ensure_group) fails inside the sametry. The failures count, and the first good step logsconsuming, then the recovery. That ordering is right.tests/test_consumer.py:337-378(test_a_broker_name_that_does_not_resolve_yet_is_retried) already forces two name-resolution failures before the loop starts, and asserts 2 WARNINGs.caplogis set at WARNING (:359). Lower it to INFO and assert exactly one recovery withafter_failures == 2.tests/test_consumer.py:881-907(test_a_backoff_is_progress_and_says_so) scripts a step sequence and assertsbacking_offper step ((False, True, True, True, False)). That's the right harness for the body's three cases: fail then succeed gives 1 record; two runs give 2; success alone gives 0. Add a fourth case: stop during a backoff gives no recovery record.stop.wait()ends the loop before any good step.consumer loop error; backing off. Recommendconsumer loop recovered, so onejqfilter onconsumer loopreads both ends.poll cycle recoverednames Replicator's loop, not ours; the shared key isafter_failures.consuming:scripts/deploy.sh:520selectsmessage == "consuming"as its readiness proof.processor.loggingcarriesextra=fields through, and INFO gives journaldPRIORITY=6(WARNING and ERROR records reach journald at INFO priority #76's mapping).Docs that name the loop's records (
git grepfor the record andbackoff)docs/NETWORK.md:123, § Tailnet flaps, signatureconsumer loop error; backing off… "until the path is back"consumer loop recovered,after_failuresndocs/NETWORK.md, same signature's last bulletNRestarts=0, andack: completeresumes"docs/MONITORING.md:100, § Liveness "To find out which"backing_off: true:391-400)0203bf1,42abfca). Lean: a short Amended entry plus a clause hereThe
plans/mentions are history: leave them.Hazards to weigh
processor runagainst the localredis-server, stop it, start it, and read oneconsumer loop recovered.[measured: journal]) or Redis 7.0 is upstream EOL (2024-07-29): decide the broker's next Redis broker#85's Valkey window.src/), about 20 s of bus absence. That restart has no failure run, so it logs no recovery record:consumingcomes on a fresh process. Expected.last_progress,backing_offor the liveness check-in.tests/test_liveness.pyshould not move.processor, and it's not a broker outage either. Ship before or after it; either order works.Template
Replicator's port above for the code. For the shape of a small
src/change with a deploy: #76 (PR #85).Out of scope
is_transient;gregoryfoster commentedon Nov 16, 2026
Brief check against
82f35a8(implementer, 2026-11-16)Brief holds; building to it: record
consumer loop recovered,after_failuresn, INFO, logged at the reset; no record on stop during a backoff. Corrections and additions:step()can run more than one command before the record. A due reclaim walks the whole PEL and runs every entry idle pastreclaim_min_idle_ms(consumer.py:156-158,reclaim()), then the read takes at most one new command. So the record trails the broker's return by the reclaim walk plus one read or command. In practice the PEL holds at most the in-flight entry. Doesn't change the recommendation: keep it at the reset.tests/test_consumer.pyis module-markedintegration(:49). The shipping gate never runs it; CI does, and so doesuv run pytest -m integrationhere. The new tests go there anyway (tests mirror source; the seeds are there). The pre-deploy proof is CI's integration job, plus a local run.docs/NETWORK.md:151reads the journal with-p warning; an INFOconsumer loop recovered(PRIORITY=6) never shows. Adding a line for it to that block, not dropping-p warning.docs/NETWORK.md:160, "Log a flap … the window (first WARNING to the firstack: completeafter it)". On an idle queue there's noack: complete; the recovery record ends the window either way. Changing it to the record (falling back toack: completebefore this deploy).consumer.py's module docstring (runbullet,:25-28) describes the loop's records; it gets a clause too.Spec: a short Amended 2026-11-16 entry plus the §3 Health clause, as the brief leans. MONITORING.md: one line. Plan:
docs/plans/2026-11-16-134-consumer-recovery-log.md(#76 had one).#134 fix: CR 1 — after_failures counts the loop's error records, not …
#134 fix: CR 2 — run_scripted stops on the script's last step; no sto…
gregoryfoster commentedon Nov 16, 2026
Post-merge host steps (implementer, 2026-11-16)
PR #141, final
1b5e306(CI run 43955801114 green: lint, test). Ship is held for the operator. Once it says go:/home/exedev/processor, onmain):git fetch origin && git merge --ff-only origin/134-consumer-recovery-log && git push origin main. Confirmgh pr view 141says MERGED.src/, so runscripts/deploy.sh. It's a restart, about 20 s of bus absence. No generation bump (processor_versionstays0.38.0+4), no Watcher notice, no lock move, no env or/etc/processor/change. Deploy within 8 h of the push, or the drift check alerts.cat /srv/processor/live/REVISIONis the merge SHA; deploy.sh's own proof (starting,child_containment: required, thenconsuming; the smoke run) passes. Noconsumer loop recoveredon this restart: a fresh process has no failure run. That's expected.consumer loop error; backing offWARNINGs, then one INFOconsumer loop recoveredwithafter_failuresequal to their count,PRIORITY=6.gregoryfoster commentedon Nov 16, 2026
Shipped (#134)
PR #141, FF-merged
1b5e306(CI run 43957304130 green onmain). Live1b5e30629b17since 2026-11-16T21:13:08Z:scripts/deploy.shexit 0; pid 294004 loggedstarting(0.38.0+4, unchanged;child_containment: required), thenconsuming; the smoke run passed;NRestarts=0. Noconsumer loop recoveredon the restart, as expected (no failure run).Consumer.run()counts failed steps in a row. The first good step after one or more logs one INFOconsumer loop recovered,extra={"after_failures": n}, then resets (#188: the fetch-policy tail logs its recovery after failed cycles replicator#193's shape). No record on a good step after none, nor when a stop ends a backoff (1e075c6).after_failures == 3, after the WARNINGs); two runs →[2, 1]; good only → none; stop during a backoff → none; the boot path (broker.invalid, 2 failures) → 1 record, afterconsuming.-p warninghides, the window ends at the record); MONITORING.md; plandocs/plans/2026-11-16-134-consumer-recovery-log.md.dd7685a,after_failurescounts the loop's error records, not every WARNING), CR 2 (1b5e306, test helper simplified); CR 3 stet (red and green in one commit; red run seen in session).Range:
82f35a8..1b5e306. Host steps 1–3 done; 4 is moot (shipped before the window); 5 is below (host steps).Follow-up:
after_s). The journal's timestamps bracket the run already, and Replicator logs the count only; not filed.Known limits:
processor runagainst local redis, stop/start) was skipped: the integration tests drive the real loop against the scratch Redis, the boot-path one through real name-resolution failures.