Skip to content

Epic: backlog burn-down 2 — #143 open; #54 and #140 parked on triggers #84

Open

Description

@gregoryfoster

The whole open backlog after epic #24, plus the gaps a 2026-10-24 audit found that nothing tracked: host patching never run, a log-priority bug, a sibling's per-host ask, and cohort ports Processor lacks. Done when Tiers 1 and 2 are closed and live, and every Tier 3 line is closed or parked with a trigger.

Order

The operator's rule, verbatim: "Prioritize bugs, issues other repos are waiting on, and operational conformance and stability."

Tiebreaks the coordinator added, numbered so the operator can challenge one:

  1. Live security exposure ranks as a bug. Adopt host patching: needrestart l, posture scheduled, first window #77 sits in Tier 1 for that reason: 60 -security updates pending, including the OpenSSL that processor run links.
  2. A guard goes before what it guards. WARNING and ERROR records reach journald at INFO priority #76 goes before Adopt host patching: needrestart l, posture scheduled, first window #77, because a patch window's post-check reads journald priority. Inside Adopt host patching: needrestart l, posture scheduled, first window #77, needrestart's l mode is committed before the first apt run.
  3. A sibling wait that is only an operator host step (Disable exe-setup.service on co-processor (CannObserv/provisioner#4) #78) runs alongside the code child in flight. It doesn't take the implementer's slot.
  4. In Tier 3, shared files decide the order: the workflow files (see Overlaps). Bundling two children that share a file is offered at that child's gate, never assumed.
  5. A child that needs an operator decision (Lock audit: a weekly finding notifies nobody #75) gets its question one check-in before it is next.

Overrides: none yet.

Boundary decisions

  • Agent-tooling items (context cadence, SocratiCode bumps, the skills roster) are out of this epic unless a sibling asks something of Processor. Proposed 2026-10-24.
  • A cohort practice Processor lacks is Tier 3, unless it is a live exposure (tiebreak 1). Proposed 2026-10-24.
  • Close what is done or superseded. Park ideas, each with a re-entry trigger. Proposed 2026-10-24.

Exit criterion

Tiers 1 and 2 closed and live. Every Tier 3 line closed or parked with a trigger. #54 may stay parked: its trigger is the broker's, not ours.

Verify recipe

  • Gate: CI lint and test, by job, on the merge SHA and on main's tip. uv audit (report-only) runs alongside and is not a gate. Run at creation: green on db8531a, 2026-10-24.
  • Deploy: scripts/deploy.sh, only when the merge touches src/, deploy/, scripts/deploy.sh, pyproject.toml or uv.lock. Proof: /srv/processor/live/REVISION equals the merge, ActiveEnterTimestamp falls after it, NRestarts=0, and the drift check reports ok at its next hourly run.
  • Baseline: uv run pytest -q, no .env sourced: 748 passed, 5 skipped, containment required (Landlock ABI 6), 2026-10-24. The 5 skips are all tests/test_child.py:290, credential paths unreadable to exedev. Compare skips by id.
  • By deliverable:
    • host change: read before and after (systemctl is-enabled, stat, apt list --upgradable);
    • timer: its journal over one period;
    • log change: journalctl -o json PRIORITY on a real record;
    • workflow change: the job's check-run on the merge SHA, plus one scheduled or dispatched run where the trigger is a schedule;
    • docs: git diff against the claim.
  • Cross-repo probes: run scan.sh with --token-for cannobserv/cannobserv=GH_TOKEN_CANNOBSERV --token-for cannobserv/provisioner=GH_TOKEN_PROVISIONER. GH_TOKEN gets a 404 on those private repos; every scan before 2026-11-17 lacked the flags, and their refs were read by hand.

Tier 1: bugs and live exposure

Tier 2: other repos waiting on Processor

Tier 3: operational conformance and stability

Waits

Ref Who waits On what Unblock condition
CannObserv/broker#85 us (#54) the broker's Valkey 7.2 window. Its last gate, CannObserv/broker#95, closed 2026-11-03 (live 21:02:10Z). Next, the pre-window CLIENT SETINFO asks to archiver, watcher and replicator; Processor's answer stands from 2026-10-16. The window isn't scheduled yet (re-read 2026-11-16: open, last updated 2026-11-04) the post-window notice on CannObserv/broker#85
CannObserv/provisioner#16 them, then us provisioner's co-image-base assessment of co-processor (box unticked since 2026-09-29; 2026-10-30 note covers co-provisioner only) an assessment issue filed here, with the owner's approval
CannObserv/cannobserv#559 CannObserv/archiver#353 a cannobserv release, then Processor's co-core bump (#140, parked). Re-read 2026-11-16: open, no release a co-core tag carrying CannObserv/cannobserv#559; it triggers #140
CannObserv/cannobserv#530 us BlobStore.store reports created vs already stored. Released in co-core v0.41.0 (tag f7285e1, 2026-11-04/05) as store_reporting / store_file_reporting returning Stored(uri, created). store is unchanged adopt at #140; not a reason to bump on its own
gregoryfoster/skills#576 (and gregoryfoster/skills#334) us (#115's local workarounds) gregoryfoster/skills#576 closed 2026-11-05; its fix is live: the registration prefix is retired (PR #120) and the canonical hook opened PR #123 (2026-11-07). gregoryfoster/skills#334 (using-github-cli) is still open gh-env.sh and env-value.sh stay until gregoryfoster/skills#334 ships

Parked

Ref Re-entry trigger Last re-read
#54 The broker's post-window notice on CannObserv/broker#85. Gates are clear: CannObserv/broker#95 closed 2026-11-03, and CannObserv/broker#85's hand-off brief was posted 2026-11-04. The window is unscheduled. When it fires: #54's body is the brief (CI image, scratch server, the 7.0.15-only claims). For #54 (from #80): ci.yml's Redis fallback, Ubuntu 24.04's apt redis-server, holds only with runs-on pinned to 24.04 (#81's operator decision). The same window should discharge #134's debt: the broker going away is a failure run, so read the consumer loop recovered record then (query in Verification debt) 2026-11-16
#140 Whichever comes first: a new advisory in a package co-core 0.38.0 caps; soupsieve 4.1.2 on PyPI (upstream lists more quadratic-time fixes); or a co-core release carrying CannObserv/cannobserv#559 (CannObserv/archiver#353 waits on it). When it fires: #140's body is the brief (DEPENDENCIES.md § Bumping co-core; remove the override, which the guard test_co_core_still_needs_the_override forces; regenerate the goldens; Watcher notice). Adopt CannObserv/cannobserv#530's store_reporting at the same bump. Hazards: a catch-up across 0.39–0.4x with breaking changes outside extraction; strict :has() is already live (#138), so it isn't new at the bump. Evidence: 0 open alerts and the audit at 0 on 1b5e306 (2026-11-16); latest co-core tag v0.48.2 (read 2026-11-16, #140's body) 2026-11-16

Out of epic

  • SocratiCode 1.16.0 → 1.16.1: other repos are bumping, but nobody asked Processor to. Boundary decision 1.
  • context-cadence.yml and curating-context: add CannObserv/processor to .skills/cohort — pair it with broker, or unpaired gregoryfoster/skills#343 (Processor's context ledger): agent tooling. Boundary decision 1.
  • co-core 0.39.0 through 0.40.0: nothing Processor imports moved, so no bump. 0.40.0 (2026-10-30) tightens only ContentReplicateCommandEmit and SourceRevisionObservedEmit in co_core.pure.models.changes; Processor uses ContentProcessCommand and the Processing* models from that module [measured, compare v0.39.5…v0.40.0].

Verification debt

What is unverified Why it was skipped How to discharge it Discharged by
A live processor run or processor drift WARNING landing at PRIORITY=4 No forced warnings in production; a transient unit running the same code, user and sandbox proved it (#76) The first real warning after 2026-10-25T15:50:36Z: journalctl -u processor -p warning Discharged 2026-11-04: six real processor.consumer WARNINGs (consumer loop error; backing off), 05:02:10–05:03:31Z, all at PRIORITY=4, SYSLOG_IDENTIFIER processor, PID 103348 = MainPID
The refresh hook's merge_mode pr path end to end: a pr: line in .git/skills-update.log, then a PR from skills/submodule-refresh (#115) The first run after the merge (2026-11-05T01:22Z) found nothing to propose, so it never reached gh pr list. The token prefix and the user-less origin remain unexercised The first session on main, on a later UTC day, when upstream has moved; then #115 closes Discharged 2026-11-06T17:14:03Z: .git/skills-update.log logged the push of skills/submodule-refresh and pr: opened PR #119 (old hook, with the registration prefix). The canonical hook gets its own check after the retirement (#115)
#113's mechanism on a real window: due at the window's opening, the attended run, a record (first line ## Patch run record: co-processor, from a public member), then ok No window has opened since B went live Tue 2026-11-10: read journalctl -u processor-patching at about 05:01Z (kind due, checkin 202), the run and its record, then the next hourly check (ok), or missed at 10:00Z Discharged 2026-11-10: due at 05:00:32Z and hourly, missed at 10:00:34Z (all checkin 202); the out-of-window run's record (comment 6771344102, 15:24:47Z, MEMBER); ok at 15:28:01Z, checkin 202, citing it
The canonical refresh hook (no registration prefix) on a real bump (#115) PR #120 merged 2026-11-06T18:47Z; today's refresh lock was already spent by #119's run .git/skills-update.log: pr: gh's token comes from /home/exedev/processor/.env; --repo github.com/CannObserv/processor, then pr: opened … or pr: PR #N proposes the bump, and no doctor may have no token warning. Then close #115 Discharged 2026-11-07T02:20Z: pr: gh's token comes from .env; --repo github.com/CannObserv/processor (the checkout's .env, logged relative), pr: opened PR #123, no warning. #115 closed 06:48Z
A Dependabot uv security job resolving through the co-pypi index (#121) No advisory has landed since admission; the proof (run 42338249411) was a version job The next uv in /. security job ("command":"security"): its log shows registered gcp OIDC credentials for python index and no PathDependenciesNotReachable. Then land its PR by hand (docs/DEPLOYMENT.md § CI pins and Dependabot) Discharged 2026-11-16 (read then): the soupsieve security job of 2026-11-13 (run 43371303257, "command":"security") logged registered gcp OIDC credentials for python index and authenticating request with OIDC token (host: us-west1-python.pkg.dev), no PathDependenciesNotReachable; it failed only on co-core 0.38.0's soupsieve cap (#138). The opentelemetry-api job (43536925946) succeeded and opened PR #135, landed by hand in #138
The live consumer loop recovered record (#134): one INFO, after_failures equal to the run's consumer loop error; backing off records, PRIORITY=6 Nothing fails the loop on demand in production; the integration tests drive the real loop against scratch Redis, the boot path through real resolution failures At the next real failure run (a tailnet flap, CannObserv/broker#85's Valkey window): journalctl -u processor -o cat --since <start> | jq -cR 'fromjson? | select(.message | startswith("consumer loop")) | {timestamp, level, message, after_failures}' (#134 host steps, comment 6929280921) (open)

Watching

Standing checks

Overlaps

Surface Items
— none open

Measured state

Rewritten from scratch at each check-in, never appended to.

  • processor active, NRestarts=0, MainPID 338, live releases/1b5e30629b17 (processor_version 0.38.0+4) since the patch run's reboot (boot 2026-11-17T17:55:56Z, active 17:55:58Z). origin/main is 8a1b79f (3 commits ahead, none that runs); drift ok at 19:01:02Z. Kernel 6.12.115. Measured 2026-11-17T19:35Z.
  • CI green by job on 8a1b79f (push run 44106119940); the lock audit on the push (44106119997): no known vulnerabilities in 63 packages. Dependabot alerts: 0 open. Open PRs: none. Measured 2026-11-17.
  • Host: 0 packages upgradable after the 2026-11-17 run (record on Patching: a missed window goes unnoticed — solicit approval before each window, or automate unless overridden #113, comment 6956497896); processor-patching ok at 19:00:32Z. Open probe findings: named:docker, dormant:nginx (Prune the image's dormant docker and nginx from co-processor (or declare keep) #143). Measured 2026-11-17.
  • 0 processor records at -p warning since the reboot; 4 since 2026-11-04T05:04:00Z, all reported before. Measured 2026-11-17.
  • Root disk 64% (6.9G free). Measured 2026-11-17T19:35Z.

Log

Activity

  1. gregoryfoster commented on Oct 27, 2026

    @gregoryfoster
    MemberAuthor

    #82 check-in pending (implementer, 2026-10-27): #95 merged (875a18c) and deployed. All host steps are done, and co-processor-disk-headroom is enabled and its rehearsal cleared. One thing is left for the coordinator: after the GC's first prune, at 2026-10-28T05:38:06Z, run the checklist on #82 (#82 (comment)). It covers the run result, what was removed, df /, the sensor, and MemoryPeak (for #83). Then record the result and close #82.

  2. gregoryfoster commented on Nov 5, 2026

    @gregoryfoster
    MemberAuthor

    cannobserv v0.41.0: BlobStore can report whether a store created the object (cannobserv#530)

    v0.41.0 is tagged at cannobserv f7285e1e. It carries cannobserv#530, requested in processor#26.

    • Added: store_reporting(data, fingerprint, media_type) and store_file_reporting(path, fingerprint, media_type) on the BlobStore Protocol, LocalBlobStore and GcsBlobStore. Each returns a frozen Stored(uri, created) from co_core.pure.util.blobstore. store / store_file are unchanged and return the twin's .uri.
    • What created guarantees: on GCS, only created=True is exact. False can follow this call's own create when the SDK retried it into a 412 after a lost response. On the filesystem it's the other way round: False is exact, and two racing writers can both report True. For a log field saying whether output was new or already stored, the rare GCS false negative is acceptable.
    • When: at Processor's next planned co-core bump. It pins ==0.38.0 on main 420cc19e. Nothing changes until it calls a twin.

    The jump from 0.38.0 crosses MINORs. The Emit twins Processor builds (ProcessingComplete / ProcessingFailed) are unchanged, and its bump needs the usual wheel-to-wheel diff of the modules it imports.

  3. gregoryfoster commented on Nov 10, 2026

    @gregoryfoster
    MemberAuthor

    For the coordinator: co-core now pins its build backend (CannObserv/cannobserv#605)

    What changed upstream. Each cannobserv workspace package pins requires = ["hatchling==1.33.0"] in its [build-system]. A test keeps the four in agreement. It's merged to cannobserv main as 594c59de (CannObserv/cannobserv#649) and ships with the next co-core release; it has no tag of its own yet. A later bump of the pin is its own reviewed PR, with a CHANGELOG entry that names the version.

    What it means for Processor: nothing to change.

    docs/plans/2026-10-28-79-87-wheelhouse.md stays as history; it describes the retired path.

  4. changed the title [-]Epic: backlog burn-down 2 — bugs, cohort waits, operational conformance[/-] [+]Epic (parking lot): backlog burn-down 2 — #54 and #140 parked on triggers[/+] on Nov 16, 2026
  5. changed the title [-]Epic (parking lot): backlog burn-down 2 — #54 and #140 parked on triggers[/-] [+]Epic: backlog burn-down 2 — #143 open; #54 and #140 parked on triggers[/+] on Nov 17, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions