diff --git a/docs/MONITORING.md b/docs/MONITORING.md index d79c7b0..3451a73 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -97,7 +97,7 @@ Installed before the deploy that brings the timer, the timer's first run (at the - the key is missing or wrong, so the check-ins fail; - `co-processor` is down, the tailnet path to `status` is gone, or Status refuses. -It never means the queue is empty. A broker outage isn't it either: that backs off, which counts as consuming, and the variables say `backing_off: true`. To find out which: +It never means the queue is empty. A broker outage isn't it either: that backs off, which counts as consuming, and the variables say `backing_off: true`. The journal shows its end: one `consumer loop recovered`, with `after_failures` (#134). To find out which: ```bash systemctl status processor diff --git a/docs/NETWORK.md b/docs/NETWORK.md index 5a8bfe2..8d346ec 100644 --- a/docs/NETWORK.md +++ b/docs/NETWORK.md @@ -120,14 +120,14 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k - The journal: no entry at all 04:12:53–04:13:26Z (32.7 s, the length of the jump). An idle VM can be that quiet, so this suggests the pause rather than proving it. - kernel: `virtio_balloon virtio5: Out of puff! Can't get 1 pages` ×26, 04:13:26–38Z. The host was inflating the balloon, and the guest had no pages to give back. These were the only balloon messages on this boot (since 2026-10-03). -- `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s). +- `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s). Since #134 the run ends with one INFO `consumer loop recovered`, `after_failures` the number of `consumer loop error` records before it. - tailscaled: - `magicsock: endpoints changed`, with a second STUN address beside the usual one; - `LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]`. The wall clock jumped 33 s, which is what a paused VM looks like from inside; - `closing connection to derp-10 (rebind-ping-fail)`, then a reconnect; - `open-conn-track` timeouts. - `processor`: possibly one `liveness check-in failed` (the 10 s bound). The next check-in lands inside the 900 s grace, so nothing pages. -- Recovery on its own: `NRestarts=0`, and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back. +- Recovery on its own: `NRestarts=0`, `consumer loop recovered` (since #134), and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back. **The cohort saw the same event** (a Mayfly questionnaire, 2026-10-23; synthesis on #67, the broker's timeline on CannObserv/broker#118): - co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump; @@ -149,12 +149,13 @@ exe.dev's docs don't describe maintenance, host migration, ballooning or pauses. ```bash journalctl -u processor -p warning -o cat --since | jq -cR 'fromjson? | {timestamp, level, logger, message, error}' +journalctl -u processor -o cat --since | jq -cR 'fromjson? | select(.message == "consumer loop recovered") | {timestamp, after_failures}' journalctl -u tailscaled -o short-iso --since | grep -E 'LinkChange|endpoints changed|rebind-reason|closing connection to derp|open-conn-track' journalctl -k -o short-iso --since | grep -E 'Out of puff|balloon' systemctl show processor -p NRestarts tailscale status ``` -`-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines; for a window before #76's deploy, every record is `PRIORITY=6`, so drop `-p warning` and filter with `select(.level != "INFO")` instead (DEPLOYMENT.md § Operate → Logs). +`-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines. The recovery record is INFO, below `-p warning`: the second query reads it. For a window before #76's deploy, every record is `PRIORITY=6`, so drop `-p warning` and filter with `select(.level != "INFO")` instead (DEPLOYMENT.md § Operate → Logs). -Log a flap on the epic as: the window (first WARNING to the first `ack: complete` after it), the counts above, and which kind it was. +Log a flap on the epic as: the window (first WARNING to `consumer loop recovered`; before #134's deploy, to the first `ack: complete` after it), the counts above, and which kind it was. diff --git a/docs/plans/2026-11-16-134-consumer-recovery-log.md b/docs/plans/2026-11-16-134-consumer-recovery-log.md new file mode 100644 index 0000000..ea51b59 --- /dev/null +++ b/docs/plans/2026-11-16-134-consumer-recovery-log.md @@ -0,0 +1,38 @@ +--- +title: The consumer loop logs its recovery (#134) +date: 2026-11-16 +status: per #134's body, the brief (6928735158) and its check (6928995289) +--- + +# The consumer loop logs its recovery + +## Problem + +`Consumer.run()` logs every failed step (`consumer loop error; backing off`), but the next good step resets `backoff` and `backing_off` silently (`consumer.py:139-140`). After a broker outage the journal shows when failures stopped, never that the loop came back. + +## Approach (Replicator's shape, replicator#193) + +- `run()`: a `failures` count beside `backoff`. `+1` in the `except`. After a good step, if non-zero: one INFO `consumer loop recovered`, `extra={"after_failures": n}`, then reset to 0. +- Logged at the reset (after `step()` returns), not earlier: a step is the loop's unit of having come back. Ordering after an outage: WARNINGs, maybe `consuming` (boot path) and `ack: complete`, then `recovered`. +- No record on a good step after none, and none when a stop ends a backoff. +- Name: `consumer loop recovered`, so `consumer loop` matches both ends; never `consuming` (`scripts/deploy.sh`'s readiness proof). Shared key with Replicator: `after_failures`. +- No change to `last_progress`, `backing_off`, liveness, backoff timing, `is_transient`, the formatter. +- Docs: `consumer.py` docstring (`run` bullet); spec Amended entry + §3 Health clause; NETWORK.md § Tailnet flaps (signature, query block: an INFO line `-p warning` hides, the window's end); MONITORING.md § Liveness one line. + +## Tests (red first, `tests/test_consumer.py`, integration-marked) + +Scripted `step` over the real one, as `test_a_backoff_is_progress_and_says_so`: +- fail ×3 then succeed → exactly one record, `after_failures == 3`, INFO; +- fail ×2, succeed, fail ×1, succeed → two records, counts 2 then 1; +- succeed only → none; +- fail, then stop during the backoff → none. + +Boot path: `test_a_broker_name_that_does_not_resolve_yet_is_retried` at INFO asserts one record, `after_failures == 2`, after `consuming`. + +## Ship + +PR `Refs #134` → FF merge → `scripts/deploy.sh` (a restart; no generation bump, no Watcher notice). Pre-deploy proof: CI's integration job, and optionally a scratch-bus run (`processor run` on local redis, stop/start it, read one record). Live proof waits for the next real failure run: verification debt on epic #84. + +## Open questions + +None. diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 3f02609..1ec8b17 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -159,6 +159,13 @@ on `co-processor`. - opentelemetry-api 1.45.1 → 1.45.2 (1 advisory) is a floor only: it isn't in the stack. +**Amended 2026-11-16 (the consume loop logs its recovery, processor#134):** +- The first good step after a run of failed steps logs one INFO `consumer loop + recovered` with `after_failures` (the run's length), so the journal shows when + the loop came back, not only when failures stopped. None follows a stop during + a backoff. A log record only: no change to backoff, liveness or the wire + (Section 3, Health). + --- ## Why a new service @@ -397,7 +404,9 @@ wedge. The heartbeat checks in `ok` to `co-processor-live` every read_block_ms`, the reclaim's own bound for a dead command plus one read. Otherwise it stays silent: - an idle queue still turns the loop, so it never reads as an outage; -- a broker outage backs off and counts as alive; +- a broker outage backs off and counts as alive. Its end is logged: the first + good step after it logs `consumer loop recovered` with `after_failures` + (amended 2026-11-16, #134); - a wedged loop goes silent while the process stays up. The check-in is blocking `requests`, so it runs in a daemon thread, not the diff --git a/src/processor/consumer.py b/src/processor/consumer.py index e98c3b7..e978398 100644 --- a/src/processor/consumer.py +++ b/src/processor/consumer.py @@ -23,7 +23,9 @@ the command never runs again in this process, so no success follows its failure fact. - ``run``: ``step`` until stopped, backing off on any exception (connection loss, - ``NOPERM``, ``OOM``), never exiting on one. A stop also ends a reclaim between + ``NOPERM``, ``OOM``), never exiting on one. The first good step after a run of + failures logs ``consumer loop recovered`` with ``after_failures`` (#134); a stop + during the backoff logs none. A stop also ends a reclaim between messages: the in-flight command finishes, and the rest of a backlog stays pending for the next process rather than outlasting the unit's ``TimeoutStopSec``. - ``last_progress``: stamped after every step, backoff included, and every message, @@ -127,6 +129,7 @@ async def run(self, stop: asyncio.Event) -> None: """Consume until ``stop`` is set; the current message always finishes.""" self._stop = stop backoff = _BACKOFF_START_S + failures = 0 # failed steps in a row: the run the next good step ends (#134) started = False self.last_progress = time.monotonic() while not stop.is_set(): @@ -136,10 +139,14 @@ async def run(self, stop: asyncio.Event) -> None: started = True logger.info("consuming", extra={"group": GROUP, "stream": CONTENT_PROCESS}) await self.step() + if failures: + logger.info("consumer loop recovered", extra={"after_failures": failures}) + failures = 0 backoff = _BACKOFF_START_S self.backing_off = False self.last_progress = time.monotonic() except Exception as exc: + failures += 1 self.backing_off = True self.last_progress = time.monotonic() log = logger.warning if is_transient(exc) else logger.exception diff --git a/tests/test_consumer.py b/tests/test_consumer.py index 54549ba..8e72828 100644 --- a/tests/test_consumer.py +++ b/tests/test_consumer.py @@ -356,7 +356,7 @@ def getaddrinfo(name, *args, **kwargs): monkeypatch.setattr(socket, "getaddrinfo", getaddrinfo) monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01) - caplog.set_level(logging.WARNING, logger="processor.consumer") + caplog.set_level(logging.INFO, logger="processor.consumer") # `bus` holds the ACL user and its denial check; this client reaches it by name. client = redis_client(f"redis://{USER}:{PASSWORD}@{host}:6379/{DB}", read_block_ms=100) consumer = make_consumer(client, stores) @@ -364,7 +364,8 @@ def getaddrinfo(name, *args, **kwargs): task = asyncio.create_task(consumer.run(stop)) try: async with asyncio.timeout(10): # no pytest-timeout: a loop that never starts fails - while not await admin.exists(CONTENT_PROCESS): + # The group, then a good step: the step that ends the failure run. + while not (await admin.exists(CONTENT_PROCESS) and consumer.backing_off is False): assert not task.done(), "the loop exited on an unresolved name" await asyncio.sleep(0.01) finally: @@ -376,6 +377,11 @@ def getaddrinfo(name, *args, **kwargs): assert len(retries) == 2 # Logged as a broker fault (a warning), not as a bug with its traceback. assert {r.levelno for r in retries} == {logging.WARNING} + # The boot path: the group, then the first good step ends the run (#134). + messages = [r.getMessage() for r in caplog.records] + [recovered] = [r for r in caplog.records if r.getMessage() == "consumer loop recovered"] + assert recovered.after_failures == 2 + assert messages.index("consuming") < messages.index("consumer loop recovered") async def test_stop_during_a_reclaim_finishes_only_the_in_flight_command( @@ -907,6 +913,75 @@ async def step() -> None: assert backing_off[:5] == (False, True, True, True, False) +RECOVERED = "consumer loop recovered" + + +async def run_scripted(consumer: Consumer, script: str) -> None: + """Run ``consumer`` over a scripted ``step``: ``F`` raises, ``S`` steps for real. + + The stop is set on the script's last step, so a script ending in ``F`` stops the + loop during that failure's backoff. + """ + real_step = consumer.step + calls = itertools.count() + stop = asyncio.Event() + + async def step() -> None: + n = next(calls) + if n == len(script) - 1: + stop.set() + if script[n] == "F": + raise RedisConnectionError("broker gone") + await real_step() + + consumer.step = step + await asyncio.wait_for(consumer.run(stop), timeout=5) + + +def recoveries(caplog) -> list[logging.LogRecord]: + """The loop's recovery records (#134).""" + return [r for r in caplog.records if r.getMessage() == RECOVERED] + + +async def test_the_first_good_step_after_failures_logs_one_recovery( + bus, stores, monkeypatch, caplog +) -> None: + # After a broker outage the journal says the loop came back, with how many + # failed steps it took (#134; Replicator's `after_failures`, replicator#193). + monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01) + caplog.set_level(logging.INFO, logger="processor.consumer") + await run_scripted(make_consumer(bus, stores), "FFFSS") + [record] = recoveries(caplog) + assert record.levelno == logging.INFO + assert record.after_failures == 3 + # After the run's failures, never inside it. + messages = [r.getMessage() for r in caplog.records] + assert messages.index(RECOVERED) > max( + i for i, m in enumerate(messages) if m == "consumer loop error; backing off" + ) + + +async def test_each_failure_run_logs_its_own_recovery(bus, stores, monkeypatch, caplog) -> None: + monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01) + caplog.set_level(logging.INFO, logger="processor.consumer") + await run_scripted(make_consumer(bus, stores), "FFSFS") + assert [r.after_failures for r in recoveries(caplog)] == [2, 1] + + +async def test_good_steps_alone_log_no_recovery(bus, stores, caplog) -> None: + caplog.set_level(logging.INFO, logger="processor.consumer") + await run_scripted(make_consumer(bus, stores), "SSS") + assert recoveries(caplog) == [] + + +async def test_a_stop_during_a_backoff_logs_no_recovery(bus, stores, monkeypatch, caplog) -> None: + # The loop never came back: the stop ends it inside the backoff. + monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01) + caplog.set_level(logging.INFO, logger="processor.consumer") + await run_scripted(make_consumer(bus, stores), "SFF") + assert recoveries(caplog) == [] + + async def test_each_message_of_a_reclaim_walk_is_progress(admin, bus, stores) -> None: # A walk over a backlog takes one command's time per entry; stamped per step alone, # a long walk would read as a wedge.