Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/MONITORING.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ Installed before the deploy that brings the timer, the timer's first run (at the
- the key is missing or wrong, so the check-ins fail;
- `co-processor` is down, the tailnet path to `status` is gone, or Status refuses.

It never means the queue is empty. A broker outage isn't it either: that backs off, which counts as consuming, and the variables say `backing_off: true`. To find out which:
It never means the queue is empty. A broker outage isn't it either: that backs off, which counts as consuming, and the variables say `backing_off: true`. The journal shows its end: one `consumer loop recovered`, with `after_failures` (#134). To find out which:

```bash
systemctl status processor
Expand Down
9 changes: 5 additions & 4 deletions docs/NETWORK.md
Original file line number Diff line number Diff line change
Expand Up @@ -120,14 +120,14 @@ A flap is the tailnet path to `broker` going away for minutes while tailscaled k

- The journal: no entry at all 04:12:53–04:13:26Z (32.7 s, the length of the jump). An idle VM can be that quiet, so this suggests the pause rather than proving it.
- kernel: `virtio_balloon virtio5: Out of puff! Can't get 1 pages` ×26, 04:13:26–38Z. The host was inflating the balloon, and the guest had no pages to give back. These were the only balloon messages on this boot (since 2026-10-03).
- `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s).
- `processor`: `consumer loop error; backing off`, `error` `TimeoutError: Timeout reading from broker:6379`, then `Timeout connecting to server` until the path is back (backoff 1, 2, 4 … s). Since #134 the run ends with one INFO `consumer loop recovered`, `after_failures` the number of `consumer loop error` records before it.
- tailscaled:
- `magicsock: endpoints changed`, with a second STUN address beside the usual one;
- `LinkChange: major, rebinding … rebind-reason=[time-jumped(33s),ips-changed]`. The wall clock jumped 33 s, which is what a paused VM looks like from inside;
- `closing connection to derp-10 (rebind-ping-fail)`, then a reconnect;
- `open-conn-track` timeouts.
- `processor`: possibly one `liveness check-in failed` (the 10 s bound). The next check-in lands inside the 900 s grace, so nothing pages.
- Recovery on its own: `NRestarts=0`, and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back.
- Recovery on its own: `NRestarts=0`, `consumer loop recovered` (since #134), and `ack: complete` resumes. A command issued during the gap stays in the stream and is acked once the path is back.

**The cohort saw the same event** (a Mayfly questionnaire, 2026-10-23; synthesis on #67, the broker's timeline on CannObserv/broker#118):
- co-broker and co-replicator, behind the same egress, each logged the first balloon pressure of their boot (04:13:07Z and 04:13:39Z), then a 31–33 s time jump;
Expand All @@ -149,12 +149,13 @@ exe.dev's docs don't describe maintenance, host migration, ballooning or pauses.

```bash
journalctl -u processor -p warning -o cat --since <start> | jq -cR 'fromjson? | {timestamp, level, logger, message, error}'
journalctl -u processor -o cat --since <start> | jq -cR 'fromjson? | select(.message == "consumer loop recovered") | {timestamp, after_failures}'
journalctl -u tailscaled -o short-iso --since <start> | grep -E 'LinkChange|endpoints changed|rebind-reason|closing connection to derp|open-conn-track'
journalctl -k -o short-iso --since <start> | grep -E 'Out of puff|balloon'
systemctl show processor -p NRestarts
tailscale status
```

`-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines; for a window before #76's deploy, every record is `PRIORITY=6`, so drop `-p warning` and filter with `select(.level != "INFO")` instead (DEPLOYMENT.md § Operate → Logs).
`-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines. The recovery record is INFO, below `-p warning`: the second query reads it. For a window before #76's deploy, every record is `PRIORITY=6`, so drop `-p warning` and filter with `select(.level != "INFO")` instead (DEPLOYMENT.md § Operate → Logs).

Log a flap on the epic as: the window (first WARNING to the first `ack: complete` after it), the counts above, and which kind it was.
Log a flap on the epic as: the window (first WARNING to `consumer loop recovered`; before #134's deploy, to the first `ack: complete` after it), the counts above, and which kind it was.
38 changes: 38 additions & 0 deletions docs/plans/2026-11-16-134-consumer-recovery-log.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
---
title: The consumer loop logs its recovery (#134)
date: 2026-11-16
status: per #134's body, the brief (6928735158) and its check (6928995289)
---

# The consumer loop logs its recovery

## Problem

`Consumer.run()` logs every failed step (`consumer loop error; backing off`), but the next good step resets `backoff` and `backing_off` silently (`consumer.py:139-140`). After a broker outage the journal shows when failures stopped, never that the loop came back.

## Approach (Replicator's shape, replicator#193)

- `run()`: a `failures` count beside `backoff`. `+1` in the `except`. After a good step, if non-zero: one INFO `consumer loop recovered`, `extra={"after_failures": n}`, then reset to 0.
- Logged at the reset (after `step()` returns), not earlier: a step is the loop's unit of having come back. Ordering after an outage: WARNINGs, maybe `consuming` (boot path) and `ack: complete`, then `recovered`.
- No record on a good step after none, and none when a stop ends a backoff.
- Name: `consumer loop recovered`, so `consumer loop` matches both ends; never `consuming` (`scripts/deploy.sh`'s readiness proof). Shared key with Replicator: `after_failures`.
- No change to `last_progress`, `backing_off`, liveness, backoff timing, `is_transient`, the formatter.
- Docs: `consumer.py` docstring (`run` bullet); spec Amended entry + §3 Health clause; NETWORK.md § Tailnet flaps (signature, query block: an INFO line `-p warning` hides, the window's end); MONITORING.md § Liveness one line.

## Tests (red first, `tests/test_consumer.py`, integration-marked)

Scripted `step` over the real one, as `test_a_backoff_is_progress_and_says_so`:
- fail ×3 then succeed → exactly one record, `after_failures == 3`, INFO;
- fail ×2, succeed, fail ×1, succeed → two records, counts 2 then 1;
- succeed only → none;
- fail, then stop during the backoff → none.

Boot path: `test_a_broker_name_that_does_not_resolve_yet_is_retried` at INFO asserts one record, `after_failures == 2`, after `consuming`.

## Ship

PR `Refs #134` → FF merge → `scripts/deploy.sh` (a restart; no generation bump, no Watcher notice). Pre-deploy proof: CI's integration job, and optionally a scratch-bus run (`processor run` on local redis, stop/start it, read one record). Live proof waits for the next real failure run: verification debt on epic #84.

## Open questions

None.
11 changes: 10 additions & 1 deletion docs/specs/2026-09-29-processor-service-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,13 @@ on `co-processor`.
- opentelemetry-api 1.45.1 → 1.45.2 (1 advisory) is a floor only: it isn't in the
stack.

**Amended 2026-11-16 (the consume loop logs its recovery, processor#134):**
- The first good step after a run of failed steps logs one INFO `consumer loop
recovered` with `after_failures` (the run's length), so the journal shows when
the loop came back, not only when failures stopped. None follows a stop during
a backoff. A log record only: no change to backoff, liveness or the wire
(Section 3, Health).

---

## Why a new service
Expand Down Expand Up @@ -397,7 +404,9 @@ wedge. The heartbeat checks in `ok` to `co-processor-live` every
read_block_ms`, the reclaim's own bound for a dead command plus one read.
Otherwise it stays silent:
- an idle queue still turns the loop, so it never reads as an outage;
- a broker outage backs off and counts as alive;
- a broker outage backs off and counts as alive. Its end is logged: the first
good step after it logs `consumer loop recovered` with `after_failures`
(amended 2026-11-16, #134);
- a wedged loop goes silent while the process stays up.

The check-in is blocking `requests`, so it runs in a daemon thread, not the
Expand Down
9 changes: 8 additions & 1 deletion src/processor/consumer.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,9 @@
the command never runs again in this process, so no success follows its failure
fact.
- ``run``: ``step`` until stopped, backing off on any exception (connection loss,
``NOPERM``, ``OOM``), never exiting on one. A stop also ends a reclaim between
``NOPERM``, ``OOM``), never exiting on one. The first good step after a run of
failures logs ``consumer loop recovered`` with ``after_failures`` (#134); a stop
during the backoff logs none. A stop also ends a reclaim between
messages: the in-flight command finishes, and the rest of a backlog stays pending
for the next process rather than outlasting the unit's ``TimeoutStopSec``.
- ``last_progress``: stamped after every step, backoff included, and every message,
Expand Down Expand Up @@ -127,6 +129,7 @@ async def run(self, stop: asyncio.Event) -> None:
"""Consume until ``stop`` is set; the current message always finishes."""
self._stop = stop
backoff = _BACKOFF_START_S
failures = 0 # failed steps in a row: the run the next good step ends (#134)
started = False
self.last_progress = time.monotonic()
while not stop.is_set():
Expand All @@ -136,10 +139,14 @@ async def run(self, stop: asyncio.Event) -> None:
started = True
logger.info("consuming", extra={"group": GROUP, "stream": CONTENT_PROCESS})
await self.step()
if failures:
logger.info("consumer loop recovered", extra={"after_failures": failures})
failures = 0
backoff = _BACKOFF_START_S
self.backing_off = False
self.last_progress = time.monotonic()
except Exception as exc:
failures += 1
self.backing_off = True
self.last_progress = time.monotonic()
log = logger.warning if is_transient(exc) else logger.exception
Expand Down
79 changes: 77 additions & 2 deletions tests/test_consumer.py
Original file line number Diff line number Diff line change
Expand Up @@ -356,15 +356,16 @@ def getaddrinfo(name, *args, **kwargs):

monkeypatch.setattr(socket, "getaddrinfo", getaddrinfo)
monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01)
caplog.set_level(logging.WARNING, logger="processor.consumer")
caplog.set_level(logging.INFO, logger="processor.consumer")
# `bus` holds the ACL user and its denial check; this client reaches it by name.
client = redis_client(f"redis://{USER}:{PASSWORD}@{host}:6379/{DB}", read_block_ms=100)
consumer = make_consumer(client, stores)
stop = asyncio.Event()
task = asyncio.create_task(consumer.run(stop))
try:
async with asyncio.timeout(10): # no pytest-timeout: a loop that never starts fails
while not await admin.exists(CONTENT_PROCESS):
# The group, then a good step: the step that ends the failure run.
while not (await admin.exists(CONTENT_PROCESS) and consumer.backing_off is False):
assert not task.done(), "the loop exited on an unresolved name"
await asyncio.sleep(0.01)
finally:
Expand All @@ -376,6 +377,11 @@ def getaddrinfo(name, *args, **kwargs):
assert len(retries) == 2
# Logged as a broker fault (a warning), not as a bug with its traceback.
assert {r.levelno for r in retries} == {logging.WARNING}
# The boot path: the group, then the first good step ends the run (#134).
messages = [r.getMessage() for r in caplog.records]
[recovered] = [r for r in caplog.records if r.getMessage() == "consumer loop recovered"]
assert recovered.after_failures == 2
assert messages.index("consuming") < messages.index("consumer loop recovered")


async def test_stop_during_a_reclaim_finishes_only_the_in_flight_command(
Expand Down Expand Up @@ -907,6 +913,75 @@ async def step() -> None:
assert backing_off[:5] == (False, True, True, True, False)


RECOVERED = "consumer loop recovered"


async def run_scripted(consumer: Consumer, script: str) -> None:
"""Run ``consumer`` over a scripted ``step``: ``F`` raises, ``S`` steps for real.

The stop is set on the script's last step, so a script ending in ``F`` stops the
loop during that failure's backoff.
"""
real_step = consumer.step
calls = itertools.count()
stop = asyncio.Event()

async def step() -> None:
n = next(calls)
if n == len(script) - 1:
stop.set()
if script[n] == "F":
raise RedisConnectionError("broker gone")
await real_step()

consumer.step = step
await asyncio.wait_for(consumer.run(stop), timeout=5)


def recoveries(caplog) -> list[logging.LogRecord]:
"""The loop's recovery records (#134)."""
return [r for r in caplog.records if r.getMessage() == RECOVERED]


async def test_the_first_good_step_after_failures_logs_one_recovery(
bus, stores, monkeypatch, caplog
) -> None:
# After a broker outage the journal says the loop came back, with how many
# failed steps it took (#134; Replicator's `after_failures`, replicator#193).
monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01)
caplog.set_level(logging.INFO, logger="processor.consumer")
await run_scripted(make_consumer(bus, stores), "FFFSS")
[record] = recoveries(caplog)
assert record.levelno == logging.INFO
assert record.after_failures == 3
# After the run's failures, never inside it.
messages = [r.getMessage() for r in caplog.records]
assert messages.index(RECOVERED) > max(
i for i, m in enumerate(messages) if m == "consumer loop error; backing off"
)


async def test_each_failure_run_logs_its_own_recovery(bus, stores, monkeypatch, caplog) -> None:
monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01)
caplog.set_level(logging.INFO, logger="processor.consumer")
await run_scripted(make_consumer(bus, stores), "FFSFS")
assert [r.after_failures for r in recoveries(caplog)] == [2, 1]


async def test_good_steps_alone_log_no_recovery(bus, stores, caplog) -> None:
caplog.set_level(logging.INFO, logger="processor.consumer")
await run_scripted(make_consumer(bus, stores), "SSS")
assert recoveries(caplog) == []


async def test_a_stop_during_a_backoff_logs_no_recovery(bus, stores, monkeypatch, caplog) -> None:
# The loop never came back: the stop ends it inside the backoff.
monkeypatch.setattr(consumer_module, "_BACKOFF_START_S", 0.01)
caplog.set_level(logging.INFO, logger="processor.consumer")
await run_scripted(make_consumer(bus, stores), "SFF")
assert recoveries(caplog) == []


async def test_each_message_of_a_reclaim_walk_is_progress(admin, bus, stores) -> None:
# A walk over a backlog takes one command's time per entry; stamped per step alone,
# a long walk would read as a wedge.
Expand Down