diff --git a/.skills/context-claims-ok b/.skills/context-claims-ok index efa00eb..ac1e1e8 100644 --- a/.skills/context-claims-ok +++ b/.skills/context-claims-ok @@ -1,11 +1,2 @@ # curating-context claims warrants (prove-no-loss.sh --claims). Grammar: `--help`. -# #126: links into docs/DEPLOYMENT.md sections that the same change split out. -AGENTS.md :: retarget :: DEPLOYMENT.md#bumping-co-core -AGENTS.md :: retarget :: DEPLOYMENT.md#ci-pins-and-dependabot -AGENTS.md :: retarget :: DEPLOYMENT.md#disk-gc -AGENTS.md :: retarget :: DEPLOYMENT.md#extraction-dependencies -AGENTS.md :: retarget :: DEPLOYMENT.md#liveness -AGENTS.md :: retarget :: DEPLOYMENT.md#shell-lint -AGENTS.md :: retarget :: DEPLOYMENT.md#tailscale-dns -AGENTS.md :: retarget :: DEPLOYMENT.md#the-drift-check -AGENTS.md :: retarget :: DEPLOYMENT.md#the-lock-audit +# None for #126 part 2: no line was tightened. Part 1's are in git history. diff --git a/.skills/context-loss-ok b/.skills/context-loss-ok index fd78ae6..ea5f2f3 100644 --- a/.skills/context-loss-ok +++ b/.skills/context-loss-ok @@ -1,47 +1,39 @@ # prove-no-loss.sh warrants (curating-context). Grammar: `--help`. -# #126: anchor links re-aimed because the same change split their target out of -# docs/DEPLOYMENT.md; each line now names the doc its target moved to. -docs/DEPLOYMENT.md :: retarget :: Processor runs as one systemd unit, `processor`, on exe.dev VM `co-processor`, as its own user `processor`, from a release that [`scripts/deploy.sh`](#deploy-a-change-scriptsdeploysh) builds from a commit on `origin/main` (#2). What runs is `/srv/processor/live/REVISION`. An hourly timer, `processor-drift.timer`, reports through Status when that lags `origin/main` ([The drift check](#the-drift-check), #35). `processor run` itself checks in to Status every 5 min while it is consuming ([Liveness](#liveness), #39). Design: [the spec](specs/2026-09-29-processor-service-design.md) §2 (identities, grants), §4 (failure handling), §6 (cutover). -docs/DEPLOYMENT.md :: retarget :: | Tailscale `--accept-dns=true`, and tailscaled's OOM drop-in | this VM | done 2026-10-01, [Tailscale DNS](#tailscale-dns) (#8) | -docs/DEPLOYMENT.md :: retarget :: | Status key at `/etc/processor/status-checkin.key` (root 600; `processor-drift.service`'s `LoadCredential=`) | this VM | #35 gate 3, [The drift check](#the-drift-check) | -docs/DEPLOYMENT.md :: retarget :: The URL names the broker, never its address: see [Tailscale DNS](#tailscale-dns). -docs/DEPLOYMENT.md :: retarget :: - **No deploy:** an action bump touches `.github/` alone, which is outside `RUNTIME_*` ([§ The drift check](#the-drift-check)). +# #126 part 2, judged 2026-11-10 against origin/main 1c1312c: § Patching and +# § Node.js moved from docs/DEPLOYMENT.md to docs/PATCHING.md. Part 1's entries +# (judged against e81eb9f) are in git history. -# #126: AGENTS.md's pointers into docs/DEPLOYMENT.md, retargeted at the docs its -# sections moved to. The Infrastructure rows moved verbatim to each doc's -# `## At a glance`, their DEPLOYMENT.md links now `§ below`; AGENTS.md keeps a -# shorter row. The Detail Docs line became one line per doc. -AGENTS.md :: retarget :: **co-core is pinned exactly (`==0.38.0`); Processor owns the pin** (spec D5, §5; since watcher#350 nothing else extracts). `processor_version` = `"+"`; a pin test makes every bump a deliberate, test-failing act. Never let a dependency bot or `uv lock --upgrade` move co-core. A bump gives Watcher notice (it re-baselines on `processor_version`). It is coordinated with Watcher only if it changes the contract: the `content.process` / `content.derived` models or `canonical_text` semantics. Procedure: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#bumping-co-core). The extraction stack (co-core's `extract` extra, closure: pypdf, lxml, …) is held too: moving it outside a co-core bump bumps `LOCAL_GENERATION` (#71, [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#extraction-dependencies)). -AGENTS.md :: retarget :: | Broker | `broker:6379`, by MagicDNS name, never by address. Tailscale runs `--accept-dns=true`, with an OOM drop-in for tailscaled: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#tailscale-dns) | -AGENTS.md :: retarget :: | Node.js | 24 LTS from NodeSource apt, agent tooling only (mayfly, SocratiCode): `deploy/nodesource.sh install\|check`, [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#nodejs-agent-tooling-only) | -AGENTS.md :: retarget :: | CI | GitHub Actions [.github/workflows/ci.yml](.github/workflows/ci.yml) on `ubuntu-24.04`, the node's image: lint (ruff, shellcheck), then the full suite against `redis:7.0.15` (the broker's, not 7.2: CLIENT SETINFO; upstream EOL, the next is broker#85's call), pulled from `mirror.gcr.io`, never anonymously from Docker Hub (#43). Keyless WIF to `co-pypi-reader`, whose token only `uv sync` gets (#121), needs the org variable `GCP_WIF_PROVIDER` shared with this repo and a `roles/iam.workloadIdentityUser` binding for `principalSet://…/attribute.repository/CannObserv/processor`. Actions SHA-pinned; Dependabot version updates `github-actions` only, `uv` security updates landed by hand ([docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#ci-pins-and-dependabot), #80). Shell lint, with an opt-in pre-commit hook: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#shell-lint) (#81) | -AGENTS.md :: retarget :: | Lock audit | [.github/workflows/audit.yml](.github/workflows/audit.yml): `uv audit --frozen`, report-only, outside the deploy gate (#73). The weekly patch window reads the live lock (#75): [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#the-lock-audit) | -AGENTS.md :: retarget :: | Drift check | `processor-drift.timer`, hourly: does live lag `origin/main` in code that runs (`processor.drift.RUNTIME_*`)? Checks in to Status's `co-processor-drift` (`http://status:9000`, status#24); `alert` past 8 h. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#the-drift-check) (#35) | -AGENTS.md :: retarget :: | Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades (13:00 UTC, no restart, no reboot); the rest in one approved run a week, `Tue 05:00–10:00` UTC (Monday night Pacific), as the broker's window closes; `processor-patching.timer` asks for it through Status. Never a bare `apt-get upgrade`. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#patching) (#77) | -AGENTS.md :: retarget :: | Disk GC | Provisioner's GC and headroom sensor (`maintaining-host-disk`): a daily prune of tooling under `/home/exedev`, and a check-in to Status's `co-processor-disk-headroom` every 10 min (85% / 80%). Config [deploy/disk-gc/disk-gc.conf](deploy/disk-gc/disk-gc.conf); patch runs read its line as a floor. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#disk-gc) (#82) | -AGENTS.md :: retarget :: | Liveness | `processor run` checks in `ok` to Status's `co-processor-live` every 5 min, only while the consume loop progressed within 605 s; silence past the 900 s grace pages. Daemon thread, 10 s bound: Status never touches consumption. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#liveness) (#39) | -AGENTS.md :: retarget :: 1. `/etc/processor/` (root, dir 700, files 600) — production: `.env` (`CO_PROCESSOR_*`, the broker credential), read by systemd, never by the service user; the GCS key and the Status key, handed over by `LoadCredential=` (#2, #35, #39). Managed by the operator. Commands needing it run as the service: `processor_cli` in [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md). -AGENTS.md :: retarget :: - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — prerequisites, the broker credential handoff, `/etc/processor/`, releases and `scripts/deploy.sh`, CI pins and Dependabot, shell lint, the lock audit, the drift check, liveness, containment, install and rollback, operate, co-core bumps, extraction dependencies, TLS trust, Tailscale DNS, tailnet flaps (signature, rule), patching (posture, window, the run), the disk GC, memory (no earlyoom), Node.js -docs/DEPLOYMENT.md :: retarget :: Run as `exedev`, from `~/processor`, after the PR merges. It needs `git`, `uv`, `jq`, `curl` and `flock` (util-linux) on the `PATH`, HTTPS to `api.github.com`, and for a build the co-pypi reader's key (§ GCP provisioning) with HTTPS to `oauth2.googleapis.com` (the mint) and `us-west1-python.pkg.dev` (the index), plus `sudo` for `systemctl`, `systemd-run` and the unit file; `co-processor` has all of them. The script fetches `origin` itself: -docs/DEPLOYMENT.md :: retarget :: - `processor` (adj -500) at 342, tailscaled at -950 (§ tailscaled's OOM rank); -# Bare `§ X` references re-aimed as `DOC.md § X` (fresh-eyes review, finding 2): -docs/DEPLOYMENT.md :: retarget :: Status's CI gate (status#11) is adopted as § The CI gate (#34), and its drift check (status#12) as § The drift check (#35), which reports to Status instead of healthchecks.io. -docs/DEPLOYMENT.md :: retarget :: A merge that changes nothing under `src/`, `deploy/`, `scripts/deploy.sh`, `pyproject.toml`, `uv.lock` or `.skills/patching-hosts` (the patch-due check reads the release's knob, #113) needs no deploy. That list is `RUNTIME_DIRS` and `RUNTIME_FILES` in `src/processor/drift.py`, and the drift check alerts on it (§ The drift check). -docs/DEPLOYMENT.md :: retarget :: - **The lock audit is not in the gate.** `audit.yml` is its own workflow, so its run never counts, and a finding never blocks a deploy (§ The lock audit). -docs/DEPLOYMENT.md :: retarget :: - **Drift:** `journalctl -u processor-drift` (§ The drift check). -docs/DEPLOYMENT.md :: retarget :: - **Liveness:** `journalctl -u processor -o cat | jq -cR 'fromjson? | select(.logger == "processor.liveness")'` (§ Liveness). -docs/DEPLOYMENT.md :: retarget :: - **Dead letters:** `processor_cli dlq list | show | drop ` (§ `/etc/processor/`). There is no replay. An entry failed to decode, was not a command, or is a command that raised outside the handler on every attempt (`reason` starts `gave up on attempt`): that one is a bug to fix. Before dead-lettering such a command, Processor published a terminal `dead_lettered` whose `detail` starts `dead-lettered:` (best effort), so Watcher has closed it (#17; `extraction_error` before co-core 0.37.0, #21). The journal's `dead-lettering` record says which: `failure_fact` is `published`, `skipped` (a fact had already gone out) or `refused`. On a command given up at the cap, `handle_skipped` is `true` when the record retries a give-up that did not finish (the failure fact refused transiently, or the dead-letter refused), with no re-run of the command and the original traceback (#28). The same entry logging that every ~11 min while it stays pending means `content.process.dlq` refuses it: `journalctl -u processor -o cat | jq -cR 'fromjson? | select(.handle_skipped) | {timestamp, command_id, message_id, reason}'`. -docs/DEPLOYMENT.md :: retarget :: # CO_PROCESSOR_LIVE_INTERVAL_S=300 # change co-processor-live's interval with it (§ Liveness) -docs/DEPLOYMENT.md :: retarget :: - `processor-drift.service`, which reads no `.env`, so the drift check never holds the broker credential (§ The drift check); -docs/DEPLOYMENT.md :: retarget :: - `processor.service`, since #39, for the liveness check-in (§ Liveness). Landlock refuses it to the child, as it refuses the GCS key; -docs/DEPLOYMENT.md :: retarget :: - `processor-patching.service`, since #113, which reads no `.env` either (§ The patch-due check). -docs/DEPLOYMENT.md :: retarget :: - **The weekly window, the human read** (#75, operator 2026-10-26). Each run's after-probe (§ Patching, step 9) runs `uv audit --frozen` on `/srv/processor/live/uv.lock`, the **live** lock, as the lock file's owner, `exedev`, on a copy, never in the tree. It asks OSV live, so a disclosure `audit.yml`'s Monday 14:37Z run saw is read again in the window, which opens 14 h later. The count goes in the run's record, and the operator reads it. `tests/test_patching_knob.py` keeps the units' `WorkingDirectory=`, the knob's `owner` line and `scripts/deploy.sh`'s live link equal, so a refactor can't move the probe off the lock unnoticed. -docs/DEPLOYMENT.md :: retarget :: - **Outside the deploy gate.** The gate reads `ci.yml`'s push run alone (§ The CI gate); `tests/test_lock_audit.py` holds both workflows to that. -docs/DEPLOYMENT.md :: retarget :: | behind in code for more than 8 h | `alert` (`lag`), every hour until deployed. It names `main`'s CI result | `scripts/deploy.sh`. If `main`'s CI isn't green, § The CI gate refuses: fix `main` first | -docs/DEPLOYMENT.md :: retarget :: - **GitHub's budget:** unauthenticated, 60 requests an hour per address, shared with § The CI gate. A run costs 1 request in sync or behind in docs only, 2 behind in code, and at most 10 past the grace. -docs/DEPLOYMENT.md :: retarget :: **`missing` on `co-processor-drift` means the check isn't reporting.** The cause is one of: the timer is stopped, live was rolled back past #35 (§ Rollback), `co-processor` is down, the tailnet path to `status` is gone, the key is wrong or missing, Status refuses, or GitHub stayed silent for two runs in a row. It says nothing about `processor run` (that is `co-processor-live`, § Liveness). To find out which: -docs/DEPLOYMENT.md :: retarget :: - **The key** is the drift check's, the same file (§ `/etc/processor/`). If it's missing, the service still starts. `starting` then says `liveness: off: missing the status-checkin-key credential`, an ERROR record `liveness check-in off` follows, and the monitor goes `missing`. -docs/DEPLOYMENT.md :: retarget :: **With § Patching:** -docs/DEPLOYMENT.md :: retarget :: | The 2026-10-22 balloon event | co-processor took the same balloon (`Out of puff` ×26, § Tailnet flaps) with no earlyoom: no OOM kill and no allocation failure. status (7.9 GiB, as sized in its `earlyoom.default`) and notifier (3.8 GiB), both swapless, run earlyoom, and it killed a user process on each at its floor | -docs/DEPLOYMENT.md :: retarget :: A flap is the tailnet path to `broker` going away for minutes while tailscaled keeps running. The consumer backs off and retries, with no restart (§ Liveness). It has ridden out both flaps so far: -docs/DEPLOYMENT.md :: retarget :: `-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines; for a window before #76's deploy, every record is `PRIORITY=6`, so drop `-p warning` and filter with `select(.level != "INFO")` instead (§ Operate → Logs). +# AGENTS.md's Patching and Node.js rows now link docs/PATCHING.md; the Detail +# Docs line for DEPLOYMENT.md lost patching and Node.js to a PATCHING.md line. +AGENTS.md :: retarget :: | Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades; the rest in one approved run a week, `Tue 05:00–10:00` UTC. Never a bare `apt-get upgrade`. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#patching) | +AGENTS.md :: retarget :: | Node.js | 24 LTS from NodeSource apt, agent tooling only: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#nodejs-agent-tooling-only) | +AGENTS.md :: retarget :: - [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — releases and `scripts/deploy.sh`, the CI gate, rollback, operate, patching (posture, window, the run), Node.js + +# The runbook's topic index, extended in place to link PATCHING.md. +docs/DEPLOYMENT.md :: index :: The rest of the runbook, by topic (#126): [PROVISIONING.md](PROVISIONING.md) (prerequisites, `/etc/processor/` and `processor_cli`, containment, the install), [DEPENDENCIES.md](DEPENDENCIES.md) (CI pins and Dependabot, shell lint, the lock audit, co-core and extraction bumps, TLS trust), [MONITORING.md](MONITORING.md) (the drift check, liveness, disk GC, memory), [NETWORK.md](NETWORK.md) (Tailscale DNS, tailnet flaps). + +# Anchor links out of the moved block re-aimed at the doc that holds the target +# (DEPLOYMENT.md's own sections, or through its stubs to their homes); step 8's +# bare `§ Tailscale DNS` now names NETWORK.md. +docs/DEPLOYMENT.md :: retarget :: | `health` ×2 | The unit is active, and nothing at `-p warning` in 6 min ([§ Operate → Logs](#operate); valid since #76) | +docs/DEPLOYMENT.md :: retarget :: | `owner /srv/processor/live self` | The release has no git origin; its `uv.lock` is this repo's ([§ The lock audit](#the-lock-audit)) | +docs/DEPLOYMENT.md :: retarget :: - `tailscale`, if pending: a 2.3 s tailnet blip, sockets survive ([§ Tailnet flaps](#tailnet-flaps), "A tailscaled restart here"). +docs/DEPLOYMENT.md :: retarget :: 8. After it: `probe.sh --post-boot --run `, § Tailscale DNS → *Check after a reboot*, `NRestarts=0`, and drift and liveness `ok` in Status. The disk sensor's first tick after boot logs `checked in ok (ok): HTTP 202` within 10 min ([§ Disk GC](#disk-gc)). Read `journalctl -u processor -p warning --since ` raw (systemd's own lines, such as a failed stop) and through `-o cat | jq -cR 'fromjson? | …'`. A crash is `PRIORITY=6`, so check `NRestarts` too. +docs/DEPLOYMENT.md :: retarget :: 9. The after-probe, last, after the reboot or the no-reboot restart below: `probe.sh --refresh-into `. It is what's left pending, by class, for the record, and the only step that audits the live `uv.lock` (`pending.language`). The dry run and `--post-boot` never ask OSV ([§ The lock audit](#the-lock-audit)). +docs/DEPLOYMENT.md :: retarget :: **Deploys don't wait for a window** (operator, 2026-10-26, #77). The skill's policy.md says a deploy that restarts a declared `service` takes one (gregoryfoster/skills#366). Here `scripts/deploy.sh` runs after each merge, as before: it restarts `processor` alone, verifies the new build and switches back on failure ([§ Deploy a change](#deploy-a-change-scriptsdeploysh)). A merge that lands during a window is deployed after the post-boot checks, so a flap stays attributable. +docs/DEPLOYMENT.md :: retarget :: - **Language:** `/srv/processor/live/uv.lock`, its OSV advisory count, owner `self`. Run.md §7 asks for the owning repo's update mechanism; here that is `audit.yml`, report-only ([§ The lock audit](#the-lock-audit)). Dependabot's version updates cover Actions only, never `uv.lock`, and its security updates, resolving through the co-pypi index since #121, are landed by hand ([§ CI pins and Dependabot](#ci-pins-and-dependabot)). A count above 0 goes to that section's "Where a finding goes" and is linked from the record. A tree listed as not audited, or the class in `not_read`, is recorded as no read, never as 0. +docs/DEPLOYMENT.md :: retarget :: Read a run with `journalctl -u processor-patching -n 3 -o cat`: one `patching check` record, its `kind`, `window`, `record` and `checkin` (Status's 202). A deliberate alert (`kind: test`, fault `test`, asking GitHub nothing) proves the channels, as the drift check's does ([§ The drift check](#the-drift-check)): + +# Refuted on co-processor 2026-11-10: apt-daily.timer is enabled and active, +# APT::Periodic::Enable and Update-Package-Lists are "1" (#113), so the lists +# are refreshed daily at 12:00 UTC. Both lines now say so. +docs/DEPLOYMENT.md :: disproven :: 1. `sudo apt-get update`. The maintenance dry run reads the host's lists, which nothing else refreshes (gregoryfoster/skills#467). +docs/DEPLOYMENT.md :: disproven :: **An out-of-cycle fix** (a Node security release): `sudo bash deploy/nodesource.sh install`. It refreshes the package lists first, which nothing else here does (`apt-daily.timer` is masked and `APT::Periodic::Enable` is 0), so a bare `apt-get install nodejs` answers "already the newest version" while the release sits unfetched. It then takes the newest `nodejs` candidate, at adj 0 with needrestart listing only, and ends with `check`. `check` alone reads the lists as of the last update. + +# Fresh-eyes review: the host-config line now links where each one installs, +# since PATCHING.md took the last install steps DEPLOYMENT.md held. +docs/DEPLOYMENT.md :: retarget :: Then the host configs under `deploy/` (`HOST_CONFIGS`: tailscaled's drop-in, the apt files and the apt timers' drop-ins, needrestart's, the disk GC's) are compared with their installed copies. A difference is a note, never an install, and is journaled, as is an installed unit the release lacks. The deploy keeps the 5 most recently deployed releases plus `live`. + +# Refuted on co-processor 2026-11-10 (fresh-eyes review): both apt timers have +# stamps under /var/lib/systemd/timers/ since their first runs (12:14Z, 13:42Z). +docs/DEPLOYMENT.md :: disproven :: The first unattended run reads in `journalctl -u apt-daily-upgrade` and `/var/log/unattended-upgrades/unattended-upgrades.log` (root). No apt timer has a stamp under `/var/lib/systemd/timers/` (measured 2026-11-05), so `Persistent=true` has no missed run to catch up: enabling fires nothing, and the first run is the next 13:00. **To go back to `scheduled`:** `sudo systemctl disable --now` and `mask` both timers, remove `processor-periodic`, and set the knob's `posture`. diff --git a/.skills/context-metrics.jsonl b/.skills/context-metrics.jsonl index 117cbac..c17cc9b 100644 --- a/.skills/context-metrics.jsonl +++ b/.skills/context-metrics.jsonl @@ -1,3 +1,5 @@ {"actions": ["baseline:pre-curation"], "budget": 6000, "bytes": 15801, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": null, "delta_tokens": null, "docs_orphaned": 0, "docs_total": 3, "docs_unindexed": 0, "file": "AGENTS.md", "lines": 127, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": true, "repo": "processor", "repo_commit": "e81eb9f", "seams": null, "seams_acked": null, "skill_commit": "04c4bc87", "skill_version": "1.31", "tokens": 6663, "tokens_exact": true, "tokens_live": 60780, "top_section": "Infrastructure", "top_section_share": 41, "ts": "2026-11-08"} {"actions": ["split:docs/DEPLOYMENT.md", "demote:Infrastructure", "demote:Parity goldens", "tighten:Infrastructure", "restructure:Hard Rules", "fix:retarget-links", "index:Detail Docs", "wire:write-guard", "wire:cadence"], "budget": 6000, "bytes": 13978, "claims_dropped": 0, "claims_warranted": 9, "counts": 0, "counts_acked": 3, "delta_days": 0, "delta_tokens": -734, "docs_orphaned": 0, "docs_total": 7, "docs_unindexed": 0, "file": "AGENTS.md", "lines": 131, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 38, "note": "docs/DEPLOYMENT.md still over (17.6k): § Patching and § Node.js held until #113's 2026-11-10 window closes; they move in #126's second PR. Phase 8: write guard and weekly cadence installed (operator, 2026-11-08)", "over_budget": false, "repo": "processor", "repo_commit": "595ae1b", "seams": 0, "seams_acked": 90, "skill_commit": "04c4bc87", "skill_version": "1.31", "tokens": 5929, "tokens_exact": true, "tokens_live": 64626, "top_section": "Hard Rules", "top_section_share": 17, "ts": "2026-11-08"} {"actions": ["baseline:scheduled"], "budget": 6000, "bytes": 13978, "claims_dropped": null, "claims_warranted": null, "counts": 0, "counts_acked": 3, "delta_days": 0, "delta_tokens": 0, "docs_orphaned": 0, "docs_total": 7, "docs_unindexed": 0, "file": "AGENTS.md", "lines": 131, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "processor", "repo_commit": "25467d4", "seams": 0, "seams_acked": 10, "skill_commit": "04c4bc87", "skill_version": "1.31", "tokens": 5929, "tokens_exact": true, "tokens_live": 64626, "top_section": "Hard Rules", "top_section_share": 17, "ts": "2026-11-08"} +{"actions": ["baseline:pre-curation"], "budget": 6000, "bytes": 13978, "claims_dropped": null, "claims_warranted": null, "counts": null, "counts_acked": null, "delta_days": 2, "delta_tokens": 0, "docs_orphaned": 0, "docs_total": 7, "docs_unindexed": 0, "file": "AGENTS.md", "lines": 131, "links_dead": 0, "links_dead_anchors": 0, "no_loss": null, "no_loss_warrants": null, "note": null, "over_budget": false, "repo": "processor", "repo_commit": "1c1312c", "seams": null, "seams_acked": null, "skill_commit": "b51f3c24", "skill_version": "1.31", "tokens": 5929, "tokens_exact": true, "tokens_live": 64626, "top_section": "Hard Rules", "top_section_share": 17, "ts": "2026-11-10"} +{"actions": ["split:docs/DEPLOYMENT.md", "fix:false-fact", "fix:retarget-links", "index:Detail Docs", "fix:runtime-citations"], "budget": 6000, "bytes": 14033, "claims_dropped": null, "claims_warranted": null, "counts": 0, "counts_acked": 3, "delta_days": 0, "delta_tokens": 16, "docs_orphaned": 0, "docs_total": 8, "docs_unindexed": 0, "file": "AGENTS.md", "lines": 132, "links_dead": 0, "links_dead_anchors": 0, "no_loss": "ok", "no_loss_warrants": 16, "note": "#126 part 2: § Patching and § Node.js to docs/PATCHING.md after #113's 2026-11-10 window; every live doc under budget", "over_budget": false, "repo": "processor", "repo_commit": "2f414c6", "seams": 0, "seams_acked": 52, "skill_commit": "b51f3c24", "skill_version": "1.31", "tokens": 5945, "tokens_exact": true, "tokens_live": 65146, "top_section": "Hard Rules", "top_section_share": 17, "ts": "2026-11-10"} diff --git a/.skills/context-seams-ok b/.skills/context-seams-ok index 57fc242..e6e8671 100644 --- a/.skills/context-seams-ok +++ b/.skills/context-seams-ok @@ -27,7 +27,6 @@ tests/test_drift.py :: "CLAUDE.md", tests/test_drift.py :: r"deploy only if the merge touched (.+?) \(", (root / "AGENTS.md").read_text() tests/test_lock_audit.py :: """The summary and AGENTS.md link DEPENDENCIES.md's sections by anchor, and tests/test_nodesource.py :: assert f"{EXPECTED_MAJOR} LTS from NodeSource" in (ROOT / "AGENTS.md").read_text() -tests/test_patching_knob.py :: """The knob, AGENTS.md and docs/SKILLS.md link DEPLOYMENT.md's sections by anchor.""" tests/test_pin.py :: "restore it to the contract (spec §5, D5; AGENTS.md; DEPENDENCIES.md step 4; " tests/test_skills.py :: """A superpowers skill's ``superpowers:`` is ``skills/``; AGENTS.md maps strays.""" tests/test_skills.py :: agents = (ROOT / "AGENTS.md").read_text() @@ -37,29 +36,21 @@ tests/test_skills.py :: agents = (ROOT / "AGENTS.md").read_text() .gitignore :: # keeps its files for a rollback past #121 (docs/DEPLOYMENT.md § Rollback). No .socraticodecontextartifacts.json :: "path": "./docs/DEPLOYMENT.md", docs/DEPENDENCIES.md :: - **Outside the deploy gate.** The gate reads `ci.yml`'s push run alone (DEPLOYMENT.md § The CI gate); `tests/test_lock_ -docs/DEPENDENCIES.md :: - **The weekly window, the human read** (#75, operator 2026-10-26). Each run's after-probe (DEPLOYMENT.md § Patching, st docs/DEPENDENCIES.md :: What Processor pins and how each pin moves: the CI workflow's actions and images, shellcheck, the lock audit, co-core, t -docs/DEPLOYMENT.md :: In one commit, edit `node_.x` in `deploy/apt/nodesource.sources`, `EXPECTED_MAJOR` in `tests/test_nodesource.py`, thi docs/DEPLOYMENT.md :: | Service | systemd unit `processor` — [deploy/processor.service](../deploy/processor.service), runbook [docs/DEPLOYMENT -docs/MONITORING.md :: **With DEPLOYMENT.md § Patching:** docs/MONITORING.md :: **`missing` on `co-processor-drift` means the check isn't reporting.** The cause is one of: the timer is stopped, live w docs/MONITORING.md :: - **GitHub's budget:** unauthenticated, 60 requests an hour per address, shared with DEPLOYMENT.md § The CI gate. A run docs/MONITORING.md :: What checks in to Status, and the host's resources: the drift check, liveness, the disk GC and its headroom sensor, memo docs/MONITORING.md :: | behind in code for more than 8 h | `alert` (`lag`), every hour until deployed. It names `main`'s CI result | `scripts/ docs/NETWORK.md :: `-p warning` does the level filter (#76) and `fromjson?` drops systemd's own warning lines; for a window before #76's de -docs/PROVISIONING.md :: - `processor-patching.service`, since #113, which reads no `.env` either (DEPLOYMENT.md § The patch-due check). docs/PROVISIONING.md :: What the operator set up once, on `co-processor` and around it: grants and credentials, `/etc/processor/`, the child's c docs/SKILLS.md :: **gregoryfoster-skills (16):** `auditing-ci-cost` (CI since #12), `coordinating-epics` (#24 since #52), `curating-contex docs/SKILLS.md :: - **Superpowers' `using-git-worktrees` and `writing-plans` are not linked**: gregoryfoster's own, same-named, win. So br docs/SKILLS.md :: None since #62. `brainstorming` was one, synced from v6.4.2 to move the design doc to `docs/specs/`, until v7 deferred t docs/SKILLS.md :: The post-merge pass runs as written. Its `git pull --ff-only` is a no-op after the push. Step 8 then offers `run-deploy- docs/SKILLS.md :: `curating-context` holds `AGENTS.md` to 6,000 tokens and each live doc under `docs/` (not `plans/`, `specs/`) to 10,000 -tests/test_apt_automatic.py :: Tracked in ``deploy/``, installed by hand (docs/DEPLOYMENT.md § Patching) as: -tests/test_apt_automatic.py :: pytest.fail(f"{installed} is not installed on {NODE} (docs/DEPLOYMENT.md § Patching)") tests/test_deploy.py :: by its own commit's deploy.sh (docs/DEPLOYMENT.md § Rollback).""" tests/test_drift.py :: world.lag(hours=72, files=["docs/DEPLOYMENT.md", "tests/test_drift.py"]) -tests/test_needrestart.py :: pytest.fail(f"{INSTALLED} is not installed on {NODE} (docs/DEPLOYMENT.md § Patching)") -tests/test_nodesource.py :: """The NodeSource apt setup is pinned and self-consistent (docs/DEPLOYMENT.md, #6). # A moved subsection, already named with the doc it moved to (or its own doc). .socraticodecontextartifacts.json :: "description": "The tailnet path to the broker: Tailscale DNS, tailscaled's OOM rank and boot orderi @@ -72,34 +63,46 @@ scripts/co_pypi_token.py :: f" {type(exc).__name__}: {exc}. The key: docs/PROVIS scripts/co_pypi_token.py :: gets there: docs/PROVISIONING.md § GCP provisioning. tests/test_main.py :: # NETWORK.md "At boot" (#8, #15): a start whose preflight still fails after the -# Runtime paths (processor.drift.RUNTIME_*): editing one is a deploy, and this -# change is docs only. Each still resolves: a section that stayed, or its stub -# under DEPLOYMENT.md's "Moved sections". Repointing them is #126's remaining work. -deploy/apt-daily.timer.d/processor-window.conf :: # /etc/systemd/system/apt-daily.timer.d/processor-window.conf (docs/DEPLOYMENT.md). -deploy/apt/apt.conf.d/processor-periodic :: // by hand as /etc/apt/apt.conf.d/processor-periodic (docs/DEPLOYMENT.md § Patching). -deploy/disk-gc/disk-gc.conf :: # (root, 0644); scripts/deploy.sh compares the two. docs/DEPLOYMENT.md § Disk GC. -deploy/needrestart.conf.d/processor.conf :: # Restarts on this node belong to the patch window (docs/DEPLOYMENT.md § Patching). -deploy/nodesource.sh :: # the processor service never runs it. docs/DEPLOYMENT.md -> Node.js; #6. +# Runtime paths (processor.drift.RUNTIME_*) that name docs/DEPLOYMENT.md: each +# points at what stayed there (the deploy, the CI gate, rollback, host configs). +# The rest were repointed in #126 part 2's deploy. deploy/processor-drift.service :: # release and enables the timer the first time it installs it (docs/DEPLOYMENT.md). -deploy/processor-drift.service :: Documentation=https://github.com/CannObserv/processor/blob/main/docs/DEPLOYMENT.md -deploy/processor-patching.service :: Documentation=https://github.com/CannObserv/processor/blob/main/docs/DEPLOYMENT.md deploy/processor.service :: # the deploy installs this file from that release (docs/DEPLOYMENT.md). deploy/processor.service :: Documentation=https://github.com/CannObserv/processor/blob/main/docs/DEPLOYMENT.md -deploy/tailscaled.service.d/90-processor-oom.conf :: # Install: docs/DEPLOYMENT.md. OOMScoreAdjust= applies at exec, so tailscaled -pyproject.toml :: # version docs/DEPLOYMENT.md § Shell lint names, which tests/test_shellcheck.py holds. pyproject.toml :: readme = "AGENTS.md" scripts/deploy.sh :: "restarted on the unit it replaced. docs/DEPLOYMENT.md § Rollback" scripts/deploy.sh :: "sudo systemctl disable --now $name (docs/DEPLOYMENT.md § Rollback)" scripts/deploy.sh :: # (#34, as status#11; docs/DEPLOYMENT.md § The CI gate). scripts/deploy.sh :: # (its spec's R1-R13; docs/DEPLOYMENT.md says where processor differs). The unit scripts/deploy.sh :: die "Roll back to a kept release, or build this one with its own commit's scripts/deploy.sh (docs/DEPLOYMENT.md § Rollba -scripts/deploy.sh :: note "systemctl enable --now $name failed; enable it by hand (docs/DEPLOYMENT.md § The drift check)" -scripts/deploy.sh :: say "host config deploy/$rel differs from $dest; install it by hand (docs/DEPLOYMENT.md)" -scripts/deploy.sh :: say "host config deploy/$rel is not installed at $dest (docs/DEPLOYMENT.md)" -src/processor/checkin.py :: "reinstall it (docs/DEPLOYMENT.md § The drift check)" src/processor/drift.py :: #: AGENTS.md's rule for when a merge needs ``scripts/deploy.sh``, defined here -src/processor/patching.py :: "isn't recorded: approve it in co-processor's session (docs/DEPLOYMENT.md " -src/processor/patching.py :: "or record why not (docs/DEPLOYMENT.md § Patching)", # Moved verbatim from DEPLOYMENT.md; renaming it would break its anchor. docs/PROVISIONING.md :: Broker credential handoff (hash-only, broker#75 as of 2026-09-30) + +# #126 part 2 (judged 2026-11-10): PATCHING.md's intro and its links to sections +# that stayed in DEPLOYMENT.md, or to its own; mentions of the patch-due check +# as a thing, not a section; its tests and manifest entry. +docs/PATCHING.md :: Keeping `co-processor`'s packages current: the posture and its two lanes, the weekly attended run, the patch-due check, +docs/PATCHING.md :: | `health` ×2 | The unit is active, and nothing at `-p warning` in 6 min ([§ Operate → Logs](DEPLOYMENT.md#operate); val +docs/PATCHING.md :: | `origin deb.nodesource.com follow` | Node, agent tooling ([§ Patching Node](#patching-node)) | +docs/PATCHING.md :: **Deploys don't wait for a window** (operator, 2026-10-26, #77). The skill's policy.md says a deploy that restarts a dec +docs/PATCHING.md :: **The record** goes on the issue (`records CannObserv/processor`), per run.md §7, and ends with the next window's date. +docs/PATCHING.md :: In one commit, edit `node_.x` in `deploy/apt/nodesource.sources`, `EXPECTED_MAJOR` in `tests/test_nodesource.py`, thi +docs/DEPLOYMENT.md :: The rest of the runbook, by topic (#126): [PROVISIONING.md](PROVISIONING.md) (prerequisites, `/etc/processor/` and `proc +docs/DEPLOYMENT.md :: A merge that changes nothing under `src/`, `deploy/`, `scripts/deploy.sh`, `pyproject.toml`, `uv.lock` or `.skills/patch +docs/PROVISIONING.md :: - `processor-patching.service`, since #113, which reads no `.env` either (PATCHING.md § The patch-due check). +.socraticodecontextartifacts.json :: "description": "Keeping co-processor patched: the automatic posture and its two lanes, the weekly attended run, the patc +src/processor/__main__.py :: — the drift check; ``processor patching`` — the patch-due check.""" +src/processor/patching.py :: """Has this week's attended patch run been recorded? The patch-due check (#113). +tests/test_nodesource.py :: it. Moving the major line is a deliberate act: edit ``EXPECTED_MAJOR`` here in the same +tests/test_patching_knob.py :: for heading in ("## Patching", "### Patching Node"): + +# Prose that happens to contain a moved heading ("removing it"). +.claude/hooks/skills-submodule-update.sh :: # Judging a marker dead and then removing it are two steps, and two sessions +.claude/hooks/skills-submodule-update.sh :: # Taken, not removed (CR 1897): between judging it dead and removing it, + +# Fresh-eyes review: the host-config notes name § Deploy a change, which stayed +# and links where each one installs. +scripts/deploy.sh :: say "host config deploy/$rel is not installed at $dest (where each is installed: docs/DEPLOYMENT.md § Deploy a change)" +scripts/deploy.sh :: say "host config deploy/$rel differs from $dest; install it by hand (where: docs/DEPLOYMENT.md § Deploy a change)" diff --git a/.skills/context-token-counts b/.skills/context-token-counts index c2fb291..7ae394e 100644 --- a/.skills/context-token-counts +++ b/.skills/context-token-counts @@ -3,11 +3,12 @@ # The offline estimators prefer a file's own last exact measurement to the # repo-wide figure in .skills/context-token-ratio, falling back to it for a # file never counted exactly or since drifted far from the size recorded here. -13978 5929 AGENTS.md -20182 8178 docs/DEPENDENCIES.md -43700 17639 docs/DEPLOYMENT.md -18503 7633 docs/MONITORING.md +14033 5945 AGENTS.md +20180 8174 docs/DEPENDENCIES.md +23401 9473 docs/DEPLOYMENT.md +18501 7629 docs/MONITORING.md 14366 5801 docs/NETWORK.md -19540 8303 docs/PROVISIONING.md -16292 6674 docs/SKILLS.md +21352 8690 docs/PATCHING.md +19538 8299 docs/PROVISIONING.md +16288 6666 docs/SKILLS.md 11647 4469 docs/SOCRATICODE.md diff --git a/.skills/doc-sections b/.skills/doc-sections index 155aab3..aefa97a 100644 --- a/.skills/doc-sections +++ b/.skills/doc-sections @@ -8,6 +8,7 @@ docs/specs/2026-09-29-processor-service-design.md: a behaviour change under src/ AGENTS.md: the co-core pin and the co-pypi index under Environment & Tooling (pyproject.toml, uv.lock); the Infrastructure table (deploy/, .github/, scripts/deploy.sh); Hard Rules, its containment and parity paragraphs (src/processor/_child.py, _contain.py, tests/fixtures/parity/); Detail Docs when a doc is added or renamed docs/DEPENDENCIES.md "Bumping co-core", docs/DEPENDENCIES.md "Extraction dependencies": pyproject.toml, uv.lock, the co-pypi index and scripts/co-pypi.sh, LOCAL_GENERATION docs/DEPLOYMENT.md "Deploy a change: `scripts/deploy.sh`", docs/MONITORING.md "The drift check", docs/MONITORING.md "Liveness": deploy/ units, scripts/deploy.sh, processor.drift.RUNTIME_* +docs/PATCHING.md "Patching", docs/PATCHING.md "The patch-due check", docs/PATCHING.md "Node.js (agent tooling only)": .skills/patching-hosts, deploy/apt/, the apt timers' drop-ins, deploy/needrestart.conf.d/, deploy/nodesource.sh, src/processor/patching.py docs/DEPENDENCIES.md "CI pins and Dependabot", docs/DEPENDENCIES.md "Shell lint", docs/DEPENDENCIES.md "The lock audit": .github/, scripts/shellcheck.sh, scripts/audit_lock.sh, .githooks/ docs/SKILLS.md: the inventory, the refresh hook, Merging under pr and the doc-check lists (skills/, .claude/, .gitmodules, the .skills/ knobs) docs/SOCRATICODE.md: the store, the index scope and the context manifest (.socraticode.json, .socraticodecontextartifacts.json, .socraticodeignore, .claude/settings.json's env block) diff --git a/.skills/patching-hosts b/.skills/patching-hosts index d1e4ebe..2ceb690 100644 --- a/.skills/patching-hosts +++ b/.skills/patching-hosts @@ -24,14 +24,14 @@ window Tue 05:00-10:00 # and a Status check-in. processor-patching.timer (#113) ticks on the hour, the # same read and check-in, and reads this file's weekday window lines from the # release: a change here is a deploy (RUNTIME_FILES). Liveness checks in every 5 min from inside -# `processor run`; a stop past its 900 s grace pages (docs/DEPLOYMENT.md). +# `processor run`; a stop past its 900 s grace pages (docs/MONITORING.md § Liveness). # --- the service ---------------------------------------------------------- # Stopped in the reboot chain, and checked after boot for its first start and # NRestarts. KillMode=mixed + TimeoutStopSec=240 let the in-flight command # finish; one cut off stays pending and is reclaimed (store -> publish -> ack). # A deploy restarts it outside any window, by the owner's choice (2026-10-26): -# docs/DEPLOYMENT.md § Patching. +# docs/PATCHING.md § Patching. service processor.service # An extraction mid-run: its child is a `python -I -m processor._child`. @@ -61,7 +61,7 @@ hold tailscale tailscale # --- owners --------------------------------------------------------------- # The release is an extract with no git origin: its uv.lock is this repo's, -# updated through scripts/deploy.sh (the lock audit: docs/DEPLOYMENT.md). +# updated through scripts/deploy.sh (docs/DEPENDENCIES.md § The lock audit). owner /srv/processor/live self records CannObserv/processor diff --git a/.socraticodecontextartifacts.json b/.socraticodecontextartifacts.json index eca13b8..902c359 100644 --- a/.socraticodecontextartifacts.json +++ b/.socraticodecontextartifacts.json @@ -13,7 +13,12 @@ { "name": "deployment-runbook", "path": "./docs/DEPLOYMENT.md", - "description": "Deploying and operating processor on co-processor: releases and scripts/deploy.sh, the CI gate, rollback, operate, patching, Node.js; stubs for the sections that moved (#126)." + "description": "Deploying and operating processor on co-processor: releases and scripts/deploy.sh, the CI gate, rollback, operate; stubs for the sections that moved (#126)." + }, + { + "name": "patching-doc", + "path": "./docs/PATCHING.md", + "description": "Keeping co-processor patched: the automatic posture and its two lanes, the weekly attended run, the patch-due check (processor patching), and Node.js from NodeSource." }, { "name": "provisioning-doc", diff --git a/AGENTS.md b/AGENTS.md index 0146178..7bc4a6a 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -41,8 +41,8 @@ Python 3.12, uv, hatchling src layout, pytest, ruff. | CI | [.github/workflows/ci.yml](.github/workflows/ci.yml): lint (ruff, shellcheck), then the full suite against `redis:7.0.15`, the broker's, pulled from `mirror.gcr.io`, never anonymously from Docker Hub (#43). Pins, keyless WIF to `co-pypi-reader`, Dependabot (`github-actions` only): [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md#ci-pins-and-dependabot); shell lint: [§ Shell lint](docs/DEPENDENCIES.md#shell-lint) | | Lock audit | [.github/workflows/audit.yml](.github/workflows/audit.yml), report-only, outside the deploy gate: [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md#the-lock-audit) | | Status | Check-ins: drift (`processor-drift.timer`, hourly), liveness (`processor run`, 5 min), disk headroom ([deploy/disk-gc/disk-gc.conf](deploy/disk-gc/disk-gc.conf)): [docs/MONITORING.md](docs/MONITORING.md) | -| Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades; the rest in one approved run a week, `Tue 05:00–10:00` UTC. Never a bare `apt-get upgrade`. [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#patching) | -| Node.js | 24 LTS from NodeSource apt, agent tooling only: [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md#nodejs-agent-tooling-only) | +| Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades; the rest in one approved run a week, `Tue 05:00–10:00` UTC. Never a bare `apt-get upgrade`. [docs/PATCHING.md](docs/PATCHING.md#patching) | +| Node.js | 24 LTS from NodeSource apt, agent tooling only: [docs/PATCHING.md](docs/PATCHING.md#nodejs-agent-tooling-only) | ## Hard Rules @@ -98,7 +98,8 @@ scripts/deploy.sh [] # ship: CI green, then build a release ## Detail Docs - [docs/specs/2026-09-29-processor-service-design.md](docs/specs/2026-09-29-processor-service-design.md) — the spec: decisions, grants, runtime, failure table, versioning, cutover, testing, contract quick reference -- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — releases and `scripts/deploy.sh`, the CI gate, rollback, operate, patching (posture, window, the run), Node.js +- [docs/DEPLOYMENT.md](docs/DEPLOYMENT.md) — releases and `scripts/deploy.sh`, the CI gate, rollback, operate +- [docs/PATCHING.md](docs/PATCHING.md) — patching (posture, window, the run, the patch-due check), Node.js - [docs/PROVISIONING.md](docs/PROVISIONING.md) — prerequisites, the broker credential, GCP, `/etc/processor/` and `processor_cli`, containment, install - [docs/DEPENDENCIES.md](docs/DEPENDENCIES.md) — CI pins and Dependabot, shell lint, the lock audit, co-core bumps, extraction dependencies, TLS trust - [docs/MONITORING.md](docs/MONITORING.md) — the drift check, liveness, the disk GC, memory (no earlyoom) diff --git a/deploy/apt-daily.timer.d/processor-window.conf b/deploy/apt-daily.timer.d/processor-window.conf index 7925064..fed63c1 100644 --- a/deploy/apt-daily.timer.d/processor-window.conf +++ b/deploy/apt-daily.timer.d/processor-window.conf @@ -3,7 +3,7 @@ # own list refresh. apply.sh doesn't read this timer, so its range is ours to keep # clear. 12:00-12:30 UTC daily (04:00 PST / 05:00 PDT), before the upgrade at 13:00. # tests/test_apt_automatic.py holds it outside every knob window. Installed by hand as -# /etc/systemd/system/apt-daily.timer.d/processor-window.conf (docs/DEPLOYMENT.md). +# /etc/systemd/system/apt-daily.timer.d/processor-window.conf (docs/PATCHING.md). [Timer] # An empty OnCalendar= drops the stock times; a drop-in's own would add to them. OnCalendar= diff --git a/deploy/apt/apt.conf.d/processor-periodic b/deploy/apt/apt.conf.d/processor-periodic index 1576f66..314dd55 100644 --- a/deploy/apt/apt.conf.d/processor-periodic +++ b/deploy/apt/apt.conf.d/processor-periodic @@ -1,6 +1,6 @@ // posture automatic (#113; patching-hosts policy.md § automatic): unattended-upgrades // takes -security daily, from apt-daily-upgrade.timer, and never reboots. Installed -// by hand as /etc/apt/apt.conf.d/processor-periodic (docs/DEPLOYMENT.md § Patching). +// by hand as /etc/apt/apt.conf.d/processor-periodic (docs/PATCHING.md § Patching). // apt reads apt.conf.d in name order: this name sorts after the image's // docker-disable-periodic-update, which sets Enable "0". A 99… name sorts before it. // The two lines 20auto-upgrades already sets are restated, so this file alone says diff --git a/deploy/disk-gc/disk-gc.conf b/deploy/disk-gc/disk-gc.conf index e4d37b3..d18f958 100644 --- a/deploy/disk-gc/disk-gc.conf +++ b/deploy/disk-gc/disk-gc.conf @@ -2,7 +2,7 @@ # The GC, its tiers and its rules are Provisioner's (provisioner#61), installed # by gregoryfoster/skills maintaining-host-disk; this file starts from VERSION 2's # disk-gc.conf.example. Installed by hand as /etc/cannobserv/disk-gc/disk-gc.conf -# (root, 0644); scripts/deploy.sh compares the two. docs/DEPLOYMENT.md § Disk GC. +# (root, 0644); scripts/deploy.sh compares the two. docs/MONITORING.md § Disk GC. # Anything unknown or malformed fails the GC's run, and stops a patch run too. [gc] diff --git a/deploy/needrestart.conf.d/processor.conf b/deploy/needrestart.conf.d/processor.conf index e45b2fa..5a1c572 100644 --- a/deploy/needrestart.conf.d/processor.conf +++ b/deploy/needrestart.conf.d/processor.conf @@ -6,7 +6,7 @@ # libssl) outside a window. tailscaled is static Go and maps none: only its own # postinst restarts it, in the held tailscale step. With the key set, needrestart # prints "Disabling Ubuntu mode, explicit restart mode configured" and only lists. -# Restarts on this node belong to the patch window (docs/DEPLOYMENT.md § Patching). +# Restarts on this node belong to the patch window (docs/PATCHING.md § Patching). # # Keep `$nrconf{ui}` unset: setting it without this key forces interactive mode. # A parse error here makes needrestart die, which the hook swallows with diff --git a/deploy/nodesource.sh b/deploy/nodesource.sh index 144d521..0ad0c97 100755 --- a/deploy/nodesource.sh +++ b/deploy/nodesource.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # Node.js from NodeSource's apt repo: agent tooling only (using-mayfly-chat, SocratiCode); -# the processor service never runs it. docs/DEPLOYMENT.md -> Node.js; #6. +# the processor service never runs it. docs/PATCHING.md -> Node.js; #6. # # Usage: # sudo bash deploy/nodesource.sh install # key (fingerprint-checked), source, pin, nodejs diff --git a/deploy/processor-drift.service b/deploy/processor-drift.service index 9eb04a6..7e3a96b 100644 --- a/deploy/processor-drift.service +++ b/deploy/processor-drift.service @@ -12,7 +12,7 @@ # its exit code and nothing else (#35 trap 4). [Unit] Description=Processor: does live lag origin/main? Check in to Status -Documentation=https://github.com/CannObserv/processor/blob/main/docs/DEPLOYMENT.md +Documentation=https://github.com/CannObserv/processor/blob/main/docs/MONITORING.md Wants=network-online.target tailscaled.service After=network-online.target tailscaled.service diff --git a/deploy/processor-patching.service b/deploy/processor-patching.service index bd29b21..a5db277 100644 --- a/deploy/processor-patching.service +++ b/deploy/processor-patching.service @@ -13,7 +13,7 @@ # this unit its exit code and nothing else. [Unit] Description=Processor: is this week's patch run recorded? Check in to Status -Documentation=https://github.com/CannObserv/processor/blob/main/docs/DEPLOYMENT.md +Documentation=https://github.com/CannObserv/processor/blob/main/docs/PATCHING.md Wants=network-online.target tailscaled.service After=network-online.target tailscaled.service diff --git a/deploy/tailscaled.service.d/90-processor-oom.conf b/deploy/tailscaled.service.d/90-processor-oom.conf index 2ccf1c5..27cf058 100644 --- a/deploy/tailscaled.service.d/90-processor-oom.conf +++ b/deploy/tailscaled.service.d/90-processor-oom.conf @@ -8,7 +8,7 @@ # processor.service (-500), whose extraction child volunteers at +1000. Not # -1000: that exempts it outright. # No MemoryLow: without a system.slice grant it is bounded to 0 (watcher#309). -# Install: docs/DEPLOYMENT.md. OOMScoreAdjust= applies at exec, so tailscaled +# Install: docs/NETWORK.md. OOMScoreAdjust= applies at exec, so tailscaled # must restart, which blips DNS and the tailnet for a second or two. [Service] OOMScoreAdjust=-950 diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index 2c25ef7..6d3e40a 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -114,7 +114,7 @@ They call `uv run --frozen --only-group lint`, without the project, so a clone n Disclosures arrive without commits: #71's 12 advisories surfaced only because someone ran `uv audit` by hand. Two things read the lock against OSV now: - **`audit.yml`, the signal.** `.github/workflows/audit.yml` runs `scripts/audit_lock.sh` on every push to `main`, every PR, weekly (`audit.yml`'s cron), and on dispatch (#73, ported from broker#86). A push or PR run is seen by whoever made the change. A scheduled run that finds something is still green, so it notifies nobody (#75). -- **The weekly window, the human read** (#75, operator 2026-10-26). Each run's after-probe (DEPLOYMENT.md § Patching, step 9) runs `uv audit --frozen` on `/srv/processor/live/uv.lock`, the **live** lock, as the lock file's owner, `exedev`, on a copy, never in the tree. It asks OSV live, so a disclosure `audit.yml`'s Monday 14:37Z run saw is read again in the window, which opens 14 h later. The count goes in the run's record, and the operator reads it. `tests/test_patching_knob.py` keeps the units' `WorkingDirectory=`, the knob's `owner` line and `scripts/deploy.sh`'s live link equal, so a refactor can't move the probe off the lock unnoticed. +- **The weekly window, the human read** (#75, operator 2026-10-26). Each run's after-probe (PATCHING.md § Patching, step 9) runs `uv audit --frozen` on `/srv/processor/live/uv.lock`, the **live** lock, as the lock file's owner, `exedev`, on a copy, never in the tree. It asks OSV live, so a disclosure `audit.yml`'s Monday 14:37Z run saw is read again in the window, which opens 14 h later. The count goes in the run's record, and the operator reads it. `tests/test_patching_knob.py` keeps the units' `WorkingDirectory=`, the knob's `owner` line and `scripts/deploy.sh`'s live link equal, so a refactor can't move the probe off the lock unnoticed. The script and its workflow: diff --git a/docs/DEPLOYMENT.md b/docs/DEPLOYMENT.md index 03c2f43..d4fb1d5 100644 --- a/docs/DEPLOYMENT.md +++ b/docs/DEPLOYMENT.md @@ -2,15 +2,13 @@ Processor runs as one systemd unit, `processor`, on exe.dev VM `co-processor`, as its own user `processor`, from a release that [`scripts/deploy.sh`](#deploy-a-change-scriptsdeploysh) builds from a commit on `origin/main` (#2). What runs is `/srv/processor/live/REVISION`. An hourly timer, `processor-drift.timer`, reports through Status when that lags `origin/main` ([The drift check](MONITORING.md#the-drift-check), #35). `processor run` itself checks in to Status every 5 min while it is consuming ([Liveness](MONITORING.md#liveness), #39). Design: [the spec](specs/2026-09-29-processor-service-design.md) §2 (identities, grants), §4 (failure handling), §6 (cutover). -The rest of the runbook, by topic (#126): [PROVISIONING.md](PROVISIONING.md) (prerequisites, `/etc/processor/` and `processor_cli`, containment, the install), [DEPENDENCIES.md](DEPENDENCIES.md) (CI pins and Dependabot, shell lint, the lock audit, co-core and extraction bumps, TLS trust), [MONITORING.md](MONITORING.md) (the drift check, liveness, disk GC, memory), [NETWORK.md](NETWORK.md) (Tailscale DNS, tailnet flaps). +The rest of the runbook, by topic (#126): [PROVISIONING.md](PROVISIONING.md) (prerequisites, `/etc/processor/` and `processor_cli`, containment, the install), [DEPENDENCIES.md](DEPENDENCIES.md) (CI pins and Dependabot, shell lint, the lock audit, co-core and extraction bumps, TLS trust), [MONITORING.md](MONITORING.md) (the drift check, liveness, disk GC, memory), [NETWORK.md](NETWORK.md) (Tailscale DNS, tailnet flaps), [PATCHING.md](PATCHING.md) (the posture, the weekly run, the patch-due check, Node.js). ## At a glance | Thing | Value | |---|---| | Service | systemd unit `processor` — [deploy/processor.service](../deploy/processor.service), runbook [docs/DEPLOYMENT.md](DEPLOYMENT.md); live since 2026-10-02. Runs as user `processor` (no sudo, not in `docker`, no home) from `/srv/processor/live` → `releases/`, a read-only release `scripts/deploy.sh` builds from a commit on `origin/main` (#2; the cohort standard, broker#22 / status#9). **Never from a checkout** | -| Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades (13:00 UTC, no restart, no reboot); the rest in one approved run a week, `Tue 05:00–10:00` UTC (Monday night Pacific), as the broker's window closes; `processor-patching.timer` asks for it through Status. Never a bare `apt-get upgrade`. [§ below](#patching) (#77) | -| Node.js | 24 LTS from NodeSource apt, agent tooling only (mayfly, SocratiCode): `deploy/nodesource.sh install\|check`, [§ below](#nodejs-agent-tooling-only) | ## Releases, not a checkout @@ -79,7 +77,7 @@ A merge that changes nothing under `src/`, `deploy/`, `scripts/deploy.sh`, `pypr **On success,** a timer this deploy installed for the first time is enabled (`systemctl enable --now`), so a deploy that was switched back never ran it. A timer installed before is never enabled again, so an operator's `disable` sticks. A timer that won't enable or re-arm is a note, never a switch back: it watches the deploy, it never decides one. Still by hand: `sudo systemctl reenable ` after its `[Install]` section changes, and retiring a unit (`disable --now`, remove the file, `daemon-reload`), since a unit gone from `deploy/` stays installed. -Then the host configs under `deploy/` (`HOST_CONFIGS`: tailscaled's drop-in, the apt files and the apt timers' drop-ins, needrestart's, the disk GC's) are compared with their installed copies. A difference is a note, never an install, and is journaled, as is an installed unit the release lacks. The deploy keeps the 5 most recently deployed releases plus `live`. +Then the host configs under `deploy/` (`HOST_CONFIGS`: tailscaled's drop-in, [NETWORK.md § tailscaled's OOM rank](NETWORK.md#tailscaleds-oom-rank); the apt files, the apt timers' drop-ins and needrestart's, [PATCHING.md § Patching](PATCHING.md#patching) and [§ Node.js](PATCHING.md#nodejs-agent-tooling-only); the disk GC's, [MONITORING.md § Disk GC](MONITORING.md#disk-gc)) are compared with their installed copies. A difference is a note, never an install, and is journaled, as is an installed unit the release lacks. The deploy keeps the 5 most recently deployed releases plus `live`. **Interrupted** (INT, TERM or HUP: Ctrl-C, or an agent's tool call timing out), the deploy says where it stopped once the step in progress ends: - before the switch: `interrupted before the switch; nothing switched`, exit 1, nothing journaled; @@ -148,160 +146,6 @@ A stop mid-reclaim finishes the command in hand and leaves the rest pending. A c - **Lag / missing group:** the broker probe watches `processor.process` (broker#75). - **Strikes** are counted in memory. A restart resets them, so a poison command gets at most 3 more attempts. -## Patching - -Posture **`automatic`** since #113 (operator, 2026-11-03; `scheduled` from #77), under gregoryfoster/skills `patching-hosts` ([skills/patching-hosts](../skills/patching-hosts/SKILL.md)), read from the knob [.skills/patching-hosts](../.skills/patching-hosts). Two lanes (the skill's policy.md): - -- **Security, daily and unattended.** `apt-daily-upgrade.timer` runs `unattended-upgrade` on `-security` (and the release pocket and ESM security, its stock origins) at **13:00 + up to 60 min UTC**, with needrestart listing restarts and performing none, and `Automatic-Reboot "false"`. Nothing restarts `processor` and nothing reboots: a maintainer script's own restart still happens (the scratch `redis-server`'s). A `-security` libssl or libc leaves the service on the old library until the next deploy, attended run or reboot: the attended run's step 6 picks it up. -- **Maintenance, weekly and attended** (`cadence weekly`). The owner approves a run **every week**, in **`Tue 05:00–10:00` UTC**: Monday night Pacific, 21:00–02:00 PST or 22:00–03:00 PDT (operator, 2026-10-26). It takes the security lane's set plus `-updates` plus the followed origins, `--lane maintenance`, and any reboot or restart due. The window opens as the broker's closes (`Mon 23:00–05:00`, CannObserv/broker#67). Both VMs hold the bus link, so the hosts never patch at once and a flap stays attributable to one host. Each run is approved in this repo's session, like the first. - -**The timers are moved, not only unmasked.** `apply.sh` refuses any step whose span meets `apt-daily-upgrade.timer`'s range, its calendar time plus its random delay, and refuses while either apt service runs. The stock `06:00` + 60 min sits inside the window, so the drop-ins below move it to 13:00, and `apt-daily.timer` (stock: any hour) to 12:00 + 30 min, before it. `tests/test_apt_automatic.py` holds both outside every knob window: a window change that meets them fails CI. **Never a bare `apt-get upgrade`:** it takes every origin. The skill's [run.md](../skills/patching-hosts/references/run.md) is the procedure; this section is what is particular to this host. - -| Knob line | Why | -|---|---| -| `posture automatic`, `cadence weekly` | The security lane is the apt timers'; the maintenance lane is the weekly attended run in the `window` | -| `service processor.service` | The reboot chain stops it. `--post-boot` checks its first start and `NRestarts` | -| `inflight pgrep … processor\._child` | An extraction in progress. The chain checks it again before it stops anything | -| `health` ×2 | The unit is active, and nothing at `-p warning` in 6 min ([§ Operate → Logs](#operate); valid since #76) | -| `origin deb.nodesource.com follow` | Node, agent tooling ([§ Patching Node](#patching-node)) | -| `origin pkgs.tailscale.com follow`, `hold tailscale tailscale` | Its postinst restarts tailscaled, the only path to the broker, so it is a held step approved on its own | -| `owner /srv/processor/live self` | The release has no git origin; its `uv.lock` is this repo's ([§ The lock audit](#the-lock-audit)) | -| no `datastore` | The scratch `redis-server` holds test data only. It stays the skill's default `redis` held step, because its postinst restarts it | -| no `quiet` | `processor-drift.timer` ticks drift (`OnUnitActiveSec=1h`), and liveness runs inside the service | - -**Host files** (compared after each deploy, installed by hand): - -| Tracked | Installed | Why | -|---|---|---| -| [deploy/needrestart.conf.d/processor.conf](../deploy/needrestart.conf.d/processor.conf) | `/etc/needrestart/conf.d/` | `$nrconf{restart} = 'l'`: apt's hook lists restarts and performs none. With it unset, Ubuntu mode restarts `processor` by itself after a libssl update (its `python3.12` maps the library). `tailscaled` is static Go and maps none; only its own postinst restarts it, in the held `tailscale` step. `NEEDRESTART_MODE=l` is now a backup, not the guard | -| [deploy/apt/tailscale.pref](../deploy/apt/tailscale.pref) | `/etc/apt/preferences.d/` | Scopes the followed origin to `tailscale` and its keyring (policy.md) | -| [deploy/apt/apt.conf.d/processor-periodic](../deploy/apt/apt.conf.d/processor-periodic) | `/etc/apt/apt.conf.d/` | `APT::Periodic::Enable "1"` and `Automatic-Reboot "false"` (#113). The name sorts after the image's `docker-disable-periodic-update`, which sets 0: apt reads the directory in name order, so a `99…` name would lose | -| [deploy/apt-daily-upgrade.timer.d/processor-window.conf](../deploy/apt-daily-upgrade.timer.d/processor-window.conf) | `/etc/systemd/system/apt-daily-upgrade.timer.d/` | 13:00 + 60 min UTC, clear of the window (above) | -| [deploy/apt-daily.timer.d/processor-window.conf](../deploy/apt-daily.timer.d/processor-window.conf) | `/etc/systemd/system/apt-daily.timer.d/` | 12:00 + 30 min UTC: the lists, before the upgrade | - -Install once, **before any apt run**, from the deploy checkout's root, after `git pull --ff-only` (until then it lags `origin/main` and has neither file): - -```bash -sudo install -m 644 deploy/needrestart.conf.d/processor.conf /etc/needrestart/conf.d/processor.conf -sudo needrestart -m u -b -r l # "Disabling Ubuntu mode, explicit restart mode configured"; never without -r l -sudo install -m 644 deploy/apt/tailscale.pref /etc/apt/preferences.d/tailscale.pref -sudo tailscale set --auto-update=false # the tailnet's switch is off (broker#67); this is the node's own -# posture automatic (#113): the files first, then the timers -sudo install -m 644 deploy/apt/apt.conf.d/processor-periodic /etc/apt/apt.conf.d/processor-periodic -sudo install -D -m 644 deploy/apt-daily.timer.d/processor-window.conf /etc/systemd/system/apt-daily.timer.d/processor-window.conf -sudo install -D -m 644 deploy/apt-daily-upgrade.timer.d/processor-window.conf /etc/systemd/system/apt-daily-upgrade.timer.d/processor-window.conf -sudo systemctl unmask apt-daily.timer apt-daily-upgrade.timer && sudo systemctl daemon-reload -sudo systemctl enable --now apt-daily.timer apt-daily-upgrade.timer # unattended-upgrades.service (the shutdown hook) stays masked -uv run pytest tests/test_needrestart.py tests/test_tailscaled.py tests/test_apt_automatic.py # on the node, the installed and live tests run and pass -``` - -The first unattended run reads in `journalctl -u apt-daily-upgrade` and `/var/log/unattended-upgrades/unattended-upgrades.log` (root). No apt timer has a stamp under `/var/lib/systemd/timers/` (measured 2026-11-05), so `Persistent=true` has no missed run to catch up: enabling fires nothing, and the first run is the next 13:00. **To go back to `scheduled`:** `sudo systemctl disable --now` and `mask` both timers, remove `processor-periodic`, and set the knob's `posture`. - -**The probe escalates by itself.** `probe.sh` runs its root readings through `sudo -n` whenever that works, without asking. So every run, including `--dry-run-into` and `--post-boot`, is a sudo step and needs the owner's approval. For a non-root read, put a `sudo` that exits 1 first on `PATH`; the root readings then land in its `not_read`. - -**This host's run** (each sudo step approved in this repo's session; `` = `/var/backups/patching-hosts-`). Run the scripts under `skills/patching-hosts/scripts/` from the deploy checkout's root, pulled: each defaults its knob to `.skills/patching-hosts` at `git rev-parse --show-toplevel`, or else the working directory, and with no knob found it falls back to `automatic`, report-only. - -1. `sudo apt-get update`. The maintenance dry run reads the host's lists, which nothing else refreshes (gregoryfoster/skills#467). -2. `probe.sh --lane maintenance --dry-run-into `, in the background: 0 findings expected, beside a `lists:stale` risk that reads the security archive's own publish date, not when the lists were refreshed (2026-10-26). Read the `redis-server` candidate; it must still be a `7.0.15` build, not Valkey, until #54. -3. The needrestart proof, as above. -4. `apply.sh --approve --step bulk --lane maintenance --dry-run `, in the background, within 24 h of the dry run. It restarts nothing. With no data store there is no recovery point: `recovery-point.sh` refuses ("nothing to dump"), and the bulk records the before-versions, the rollback path, in the `` it creates and prints. -5. Held steps, each approved, each `apply.sh --approve --step --run `: - - `redis`: the scratch server restarts; - - `docker`: `containerd`. Docker and containerd are disabled and inactive here, so nothing is running to restart; - - `tailscale`, if pending: a 2.3 s tailnet blip, sockets survive ([§ Tailnet flaps](#tailnet-flaps), "A tailscaled restart here"). -6. The reboot and restart decision, from the host, never from this run's set: under `automatic` a `-security` library may have landed any day since the last run. `sudo needrestart -b -r l` and `sudo grep -c '(deleted)' /proc/1/maps`. The kernel never decides one on exe.dev. Expect a reboot when `systemd` or `libsystemd0` is in the set: after those families, needrestart listed dbus and logind on co-broker and co-status, and PID 1 mapped deleted files. -7. The reboot: `reboot-chain.sh --run ` prints the chain; `--approve` launches it. It stops `processor` (up to `TimeoutStopSec=240`), then reboots in-guest. **Never `exe.dev restart`**: that is a hard reset (#83). -8. After it: `probe.sh --post-boot --run `, § Tailscale DNS → *Check after a reboot*, `NRestarts=0`, and drift and liveness `ok` in Status. The disk sensor's first tick after boot logs `checked in ok (ok): HTTP 202` within 10 min ([§ Disk GC](#disk-gc)). Read `journalctl -u processor -p warning --since ` raw (systemd's own lines, such as a failed stop) and through `-o cat | jq -cR 'fromjson? | …'`. A crash is `PRIORITY=6`, so check `NRestarts` too. -9. The after-probe, last, after the reboot or the no-reboot restart below: `probe.sh --refresh-into `. It is what's left pending, by class, for the record, and the only step that audits the live `uv.lock` (`pending.language`). The dry run and `--post-boot` never ask OSV ([§ The lock audit](#the-lock-audit)). - -**With no reboot,** restart the service anyway when needrestart lists it, or after a libssl update by either lane: `/usr/bin/python3.12`'s `_ssl` links the system `libssl.so.3`, so the old library stays mapped until a restart. `sudo grep -c 'libssl.so.3 (deleted)' /proc/$(systemctl show -p MainPID --value processor)/maps` must read 0 afterwards. - -**Liveness's grace** is 900 s. The heartbeat ticks as soon as the service starts, so the silence is at most the time since the last tick (300 s), plus the stop (240 s), plus the boot. That stays inside the grace, so Status needs a warning only if a step stalls. - -**Deploys don't wait for a window** (operator, 2026-10-26, #77). The skill's policy.md says a deploy that restarts a declared `service` takes one (gregoryfoster/skills#366). Here `scripts/deploy.sh` runs after each merge, as before: it restarts `processor` alone, verifies the new build and switches back on failure ([§ Deploy a change](#deploy-a-change-scriptsdeploysh)). A merge that lands during a window is deployed after the post-boot checks, so a flap stays attributable. - -**The record** goes on the issue (`records CannObserv/processor`), per run.md §7, and ends with the next window's date. **Its first line is `## Patch run record: co-processor`**, then anything (a date, the lane), posted by an `OWNER` or a `MEMBER` whose membership is public (§ The patch-due check): that line is what [§ The patch-due check](#the-patch-due-check) reads as this week's run, a nothing-pending run's included. A run that can't take its window still posts one, saying why, or the monitor stays `missed` until the next. What each run leaves pending, by class, comes from step 9 and includes: -- **Ubuntu Pro / ESM:** Pro isn't attached, and 25 esm-apps updates were visible on 2026-10-26. -- **Language:** `/srv/processor/live/uv.lock`, its OSV advisory count, owner `self`. Run.md §7 asks for the owning repo's update mechanism; here that is `audit.yml`, report-only ([§ The lock audit](#the-lock-audit)). Dependabot's version updates cover Actions only, never `uv.lock`, and its security updates, resolving through the co-pypi index since #121, are landed by hand ([§ CI pins and Dependabot](#ci-pins-and-dependabot)). A count above 0 goes to that section's "Where a finding goes" and is linked from the record. A tree listed as not audited, or the class in `not_read`, is recorded as no read, never as 0. - -### The patch-due check - -`processor patching`, hourly on the hour from `processor-patching.timer` (#113), asks for the attended run, since nothing else does (the 2026-11-03 window passed unnoticed). It checks in to Status's `co-processor-patching` (tenant `co-processor`, the drift check's key and `LoadCredential=`): - -| Kind | Status | When | -|---|---|---| -| `ok` | ok | a record has been posted since the window last opened | -| `due` | alert | the window is open and no record yet: **the request for the run**, within a minute of 05:00Z | -| `missed` | alert | the window closed with no record, until it next opens. A new fault, so it reports at once | -| `knob` | alert | the window can't be read from the release's knob | -| `test` | alert | `processor patching --test-alert`, by hand | -| — | none | GitHub didn't answer: silence, which the monitor's grace turns into `missing`, never a false `ok` | - -- **The window** is the release's own `.skills/patching-hosts`, its weekday `window` lines. `read-knob.sh` isn't in a release (a `git archive` leaves the submodule empty), so `src/processor/patching.py` parses them, and `tests/test_patching.py` holds the parse equal to the skill's reader. A dated line is a one-off and opens no due period. A `[host` section is unread (`knob`). The knob is a `RUNTIME_FILES` path: a change to it is a deploy, or the drift check alerts. -- **The record** is an issue comment anywhere in this repo whose first line is `## Patch run record: co-processor`, from an `OWNER` or `MEMBER` (the repo is public: anyone else's comment never counts), created at or after the opening. **The read is anonymous, so a member counts only while their CannObserv membership is public:** a private member reads as `CONTRIBUTOR` (measured 2026-11-06 for user id 14647, before the operator made theirs public). A recorder whose membership goes private turns every week `missed`: a false alert, never a false `ok`. Check with `curl -s -o /dev/null -w '%{http_code}' https://api.github.com/orgs/CannObserv/public_members/` (204: public). One unauthenticated request an hour, newest first; past 300 comments since the opening it can't say, and is silent. -- **What it doesn't report:** a restart or reboot due. The service user can't read `processor`'s maps (it's undumpable) or run needrestart, so the weekly run's step 6 picks that up. - -Read a run with `journalctl -u processor-patching -n 3 -o cat`: one `patching check` record, its `kind`, `window`, `record` and `checkin` (Status's 202). A deliberate alert (`kind: test`, fault `test`, asking GitHub nothing) proves the channels, as the drift check's does ([§ The drift check](#the-drift-check)): - -```bash -sudo systemd-run --quiet --pipe --wait --collect -p User=processor -p Group=processor \ - -p LoadCredential=status-checkin-key:/etc/processor/status-checkin.key \ - -p "$(systemctl show -p Environment processor-patching.service)" -p WorkingDirectory=/srv/processor/live \ - /srv/processor/live/.venv/bin/processor patching --test-alert -``` - -Status's side (status#95): `enabled: false` stops only its `missing` sweep, never an alert, so a check-in's `alert` pages whatever the flag. An `ok` after an alert sends Status's own `cleared` notice, so the proof is two messages: *co-processor patching: test (…)*, then *cleared* at the next hourly `ok`. - -**Silence it** for a week you mean to skip: `sudo systemctl stop processor-patching.timer`, and the monitor goes `missing`, which is honest; `start` resumes it. Better: post a record saying why, which reads `ok`. - -## Node.js (agent tooling only) - -Node 24 LTS from NodeSource's apt repo, for `using-mayfly-chat` and SocratiCode (`npx`). The `processor` unit never runs it. The cohort survey and the reasons for NodeSource and for 24 are in #6. - -```bash -sudo bash deploy/nodesource.sh install # key, source, pin, nodejs; ends with check -bash deploy/nodesource.sh check # exit 0 in sync (a pending update is a note, not drift); 3, naming each drift -``` - -| Installed | From | Why | -|---|---|---| -| `/etc/apt/keyrings/nodesource.gpg` | `https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key`, refused unless its fingerprint is `6F71 F525 2828 41EE DAF8 51B4 2F59 B5F9 9B1B E0B4` | `Signed-By:` binds the key to this one source | -| `/etc/apt/sources.list.d/nodesource.sources` | [deploy/apt/nodesource.sources](../deploy/apt/nodesource.sources) | `node_24.x`: the major line is pinned | -| `/etc/apt/preferences.d/nodesource.pref` | [deploy/apt/nodesource.pref](../deploy/apt/nodesource.pref) | `nodejs` from NodeSource at 600: above universe's 18 (500), never a downgrade. Everything else the host serves (`nsolid`) at 100, so the lane's `site=` takes only Node | - -`node` is the package's `/usr/bin/node`, so it is on every PATH: hooks, systemd user units, VS Code sessions. Installed 2026-09-30: 24.21.0, with npm 11.19.0 bundled (no Ubuntu `npm`); nothing restarted. - -**SocratiCode** (#5) runs as `exedev`, pinned to `socraticode@1.16.0` with a capped pre-install under `~/.socraticode/pin`, against co-index's shared store; it never touches the release, and the service can't read `exedev`'s home (`ProtectHome=yes`). They share 7.7 GiB with no swap, and `processor.service` carries no `MemoryLow=`: inert on this host, and the unit's `OOMScoreAdjust=-500` is the protection that counts. Pin, link stubs, memory reasoning: [docs/SOCRATICODE.md](SOCRATICODE.md#repo-specific-notes). - -### Patching Node - -unattended-upgrades takes Ubuntu's security origins alone ([§ Patching](#patching)), so nothing updates Node on its own. NodeSource ships security fixes in its own repo, never `noble-security`, so its policy is **follow**: `nodejs` rides the weekly maintenance lane. Never a bare `apt-get upgrade`. - -**Select it by host, not by origin.** NodeSource's Release file says `Origin: . nodistro`, aptly's default, which every aptly-published `nodistro` repo shares. So the knob names the site, `origin deb.nodesource.com follow` (a value holding a dot is read as a site: the skill's knob.md), and the lane's `APT_CONFIG` takes: - -``` -Unattended-Upgrade::Origins-Pattern { - "origin=Ubuntu,archive=${distro_codename}-updates"; - "site=deb.nodesource.com"; -}; -``` - -Proven 2026-09-30, with `nodejs` stepped back to 24.20.0: `unattended-upgrade --dry-run` selected it under `site=deb.nodesource.com` and not under the stock security-lane config. `origin=. nodistro` also selects it, but can't tell NodeSource from any other aptly repo. The skill's policy.md cites `nodesource.pref` as its example of scoping a followed origin (gregoryfoster/skills#344). - -**An out-of-cycle fix** (a Node security release): `sudo bash deploy/nodesource.sh install`. It refreshes the package lists first, which nothing else here does (`apt-daily.timer` is masked and `APT::Periodic::Enable` is 0), so a bare `apt-get install nodejs` answers "already the newest version" while the release sits unfetched. It then takes the newest `nodejs` candidate, at adj 0 with needrestart listing only, and ends with `check`. `check` alone reads the lists as of the last update. - -### Moving the major line - -In one commit, edit `node_.x` in `deploy/apt/nodesource.sources`, `EXPECTED_MAJOR` in `tests/test_nodesource.py`, this section's "Node LTS" and table row, and AGENTS.md's Node.js row; the test fails until all of them agree. Then `sudo bash deploy/nodesource.sh install`. Read `init-socraticode`'s preflight first: Node 26 crashes SocratiCode older than 1.13. - -### Removing it - -```bash -sudo apt-get purge nodejs -sudo rm /etc/apt/keyrings/nodesource.gpg /etc/apt/sources.list.d/nodesource.sources /etc/apt/preferences.d/nodesource.pref -sudo apt-get update -``` - ## Moved sections These sections left this runbook (#126); each heading keeps its old anchor. @@ -377,3 +221,11 @@ These sections left this runbook (#126); each heading keeps its old anchor. ### Memory: no earlyoom [MONITORING.md](MONITORING.md#memory-no-earlyoom) + +### Patching + +[PATCHING.md](PATCHING.md#patching) + +### Node.js (agent tooling only) + +[PATCHING.md](PATCHING.md#nodejs-agent-tooling-only) diff --git a/docs/MONITORING.md b/docs/MONITORING.md index 41724e3..d79c7b0 100644 --- a/docs/MONITORING.md +++ b/docs/MONITORING.md @@ -141,7 +141,7 @@ sudo install -D -m 0644 -o root -g root deploy/disk-gc/disk-gc.conf /etc/cannobs uv run pytest tests/test_disk_gc.py # on the node: the installed copy matches, and the installed GC accepts it ``` -**With DEPLOYMENT.md § Patching:** +**With PATCHING.md § Patching:** - **patching-hosts reads `[sensor] mount` and `alert_at_percent` as a floor** (gregoryfoster/skills#390). It refuses a step whose download and install would take `/` to 85%, and stops on a config the GC would refuse. - **The GC's daily run (05:15–05:45) falls inside the Tuesday window.** It never touches apt's trees, and it yields: `Nice=10`, idle I/O. - **After a reboot,** if the GC runs before VS Code reconnects, it finds no build running and keeps the newest (VERSION 3, #99). Under VERSION 2 it kept none, and the next session re-downloaded its build (about 640M). diff --git a/docs/PATCHING.md b/docs/PATCHING.md new file mode 100644 index 0000000..f498c15 --- /dev/null +++ b/docs/PATCHING.md @@ -0,0 +1,164 @@ +# Patching + +Keeping `co-processor`'s packages current: the posture and its two lanes, the weekly attended run, the patch-due check, and Node.js from NodeSource (agent tooling only). Deploying and operating: [DEPLOYMENT.md](DEPLOYMENT.md). + +## At a glance + +| Thing | Value | +|---|---| +| Patching | `automatic` (knob `.skills/patching-hosts`, #113): `-security` daily by unattended-upgrades (13:00 UTC, no restart, no reboot); the rest in one approved run a week, `Tue 05:00–10:00` UTC (Monday night Pacific), as the broker's window closes; `processor-patching.timer` asks for it through Status. Never a bare `apt-get upgrade`. [§ below](#patching) (#77) | +| Node.js | 24 LTS from NodeSource apt, agent tooling only (mayfly, SocratiCode): `deploy/nodesource.sh install\|check`, [§ below](#nodejs-agent-tooling-only) | + +## Patching + +Posture **`automatic`** since #113 (operator, 2026-11-03; `scheduled` from #77), under gregoryfoster/skills `patching-hosts` ([skills/patching-hosts](../skills/patching-hosts/SKILL.md)), read from the knob [.skills/patching-hosts](../.skills/patching-hosts). Two lanes (the skill's policy.md): + +- **Security, daily and unattended.** `apt-daily-upgrade.timer` runs `unattended-upgrade` on `-security` (and the release pocket and ESM security, its stock origins) at **13:00 + up to 60 min UTC**, with needrestart listing restarts and performing none, and `Automatic-Reboot "false"`. Nothing restarts `processor` and nothing reboots: a maintainer script's own restart still happens (the scratch `redis-server`'s). A `-security` libssl or libc leaves the service on the old library until the next deploy, attended run or reboot: the attended run's step 6 picks it up. +- **Maintenance, weekly and attended** (`cadence weekly`). The owner approves a run **every week**, in **`Tue 05:00–10:00` UTC**: Monday night Pacific, 21:00–02:00 PST or 22:00–03:00 PDT (operator, 2026-10-26). It takes the security lane's set plus `-updates` plus the followed origins, `--lane maintenance`, and any reboot or restart due. The window opens as the broker's closes (`Mon 23:00–05:00`, CannObserv/broker#67). Both VMs hold the bus link, so the hosts never patch at once and a flap stays attributable to one host. Each run is approved in this repo's session, like the first. + +**The timers are moved, not only unmasked.** `apply.sh` refuses any step whose span meets `apt-daily-upgrade.timer`'s range, its calendar time plus its random delay, and refuses while either apt service runs. The stock `06:00` + 60 min sits inside the window, so the drop-ins below move it to 13:00, and `apt-daily.timer` (stock: any hour) to 12:00 + 30 min, before it. `tests/test_apt_automatic.py` holds both outside every knob window: a window change that meets them fails CI. **Never a bare `apt-get upgrade`:** it takes every origin. The skill's [run.md](../skills/patching-hosts/references/run.md) is the procedure; this section is what is particular to this host. + +| Knob line | Why | +|---|---| +| `posture automatic`, `cadence weekly` | The security lane is the apt timers'; the maintenance lane is the weekly attended run in the `window` | +| `service processor.service` | The reboot chain stops it. `--post-boot` checks its first start and `NRestarts` | +| `inflight pgrep … processor\._child` | An extraction in progress. The chain checks it again before it stops anything | +| `health` ×2 | The unit is active, and nothing at `-p warning` in 6 min ([§ Operate → Logs](DEPLOYMENT.md#operate); valid since #76) | +| `origin deb.nodesource.com follow` | Node, agent tooling ([§ Patching Node](#patching-node)) | +| `origin pkgs.tailscale.com follow`, `hold tailscale tailscale` | Its postinst restarts tailscaled, the only path to the broker, so it is a held step approved on its own | +| `owner /srv/processor/live self` | The release has no git origin; its `uv.lock` is this repo's ([§ The lock audit](DEPENDENCIES.md#the-lock-audit)) | +| no `datastore` | The scratch `redis-server` holds test data only. It stays the skill's default `redis` held step, because its postinst restarts it | +| no `quiet` | `processor-drift.timer` ticks drift (`OnUnitActiveSec=1h`), and liveness runs inside the service | + +**Host files** (compared after each deploy, installed by hand): + +| Tracked | Installed | Why | +|---|---|---| +| [deploy/needrestart.conf.d/processor.conf](../deploy/needrestart.conf.d/processor.conf) | `/etc/needrestart/conf.d/` | `$nrconf{restart} = 'l'`: apt's hook lists restarts and performs none. With it unset, Ubuntu mode restarts `processor` by itself after a libssl update (its `python3.12` maps the library). `tailscaled` is static Go and maps none; only its own postinst restarts it, in the held `tailscale` step. `NEEDRESTART_MODE=l` is now a backup, not the guard | +| [deploy/apt/tailscale.pref](../deploy/apt/tailscale.pref) | `/etc/apt/preferences.d/` | Scopes the followed origin to `tailscale` and its keyring (policy.md) | +| [deploy/apt/apt.conf.d/processor-periodic](../deploy/apt/apt.conf.d/processor-periodic) | `/etc/apt/apt.conf.d/` | `APT::Periodic::Enable "1"` and `Automatic-Reboot "false"` (#113). The name sorts after the image's `docker-disable-periodic-update`, which sets 0: apt reads the directory in name order, so a `99…` name would lose | +| [deploy/apt-daily-upgrade.timer.d/processor-window.conf](../deploy/apt-daily-upgrade.timer.d/processor-window.conf) | `/etc/systemd/system/apt-daily-upgrade.timer.d/` | 13:00 + 60 min UTC, clear of the window (above) | +| [deploy/apt-daily.timer.d/processor-window.conf](../deploy/apt-daily.timer.d/processor-window.conf) | `/etc/systemd/system/apt-daily.timer.d/` | 12:00 + 30 min UTC: the lists, before the upgrade | + +Install once, **before any apt run**, from the deploy checkout's root, after `git pull --ff-only` (until then it lags `origin/main` and has neither file): + +```bash +sudo install -m 644 deploy/needrestart.conf.d/processor.conf /etc/needrestart/conf.d/processor.conf +sudo needrestart -m u -b -r l # "Disabling Ubuntu mode, explicit restart mode configured"; never without -r l +sudo install -m 644 deploy/apt/tailscale.pref /etc/apt/preferences.d/tailscale.pref +sudo tailscale set --auto-update=false # the tailnet's switch is off (broker#67); this is the node's own +# posture automatic (#113): the files first, then the timers +sudo install -m 644 deploy/apt/apt.conf.d/processor-periodic /etc/apt/apt.conf.d/processor-periodic +sudo install -D -m 644 deploy/apt-daily.timer.d/processor-window.conf /etc/systemd/system/apt-daily.timer.d/processor-window.conf +sudo install -D -m 644 deploy/apt-daily-upgrade.timer.d/processor-window.conf /etc/systemd/system/apt-daily-upgrade.timer.d/processor-window.conf +sudo systemctl unmask apt-daily.timer apt-daily-upgrade.timer && sudo systemctl daemon-reload +sudo systemctl enable --now apt-daily.timer apt-daily-upgrade.timer # unattended-upgrades.service (the shutdown hook) stays masked +uv run pytest tests/test_needrestart.py tests/test_tailscaled.py tests/test_apt_automatic.py # on the node, the installed and live tests run and pass +``` + +The first unattended run reads in `journalctl -u apt-daily-upgrade` and `/var/log/unattended-upgrades/unattended-upgrades.log` (root). No apt timer had a stamp under `/var/lib/systemd/timers/` when they were enabled (measured 2026-11-05), so `Persistent=true` had no missed run to catch up: enabling fired nothing, and the first run was the next 13:00. Both have stamps since their first runs (2026-11-10), so a timer disabled and enabled again now catches up a run it missed. **To go back to `scheduled`:** `sudo systemctl disable --now` and `mask` both timers, remove `processor-periodic`, and set the knob's `posture`. + +**The probe escalates by itself.** `probe.sh` runs its root readings through `sudo -n` whenever that works, without asking. So every run, including `--dry-run-into` and `--post-boot`, is a sudo step and needs the owner's approval. For a non-root read, put a `sudo` that exits 1 first on `PATH`; the root readings then land in its `not_read`. + +**This host's run** (each sudo step approved in this repo's session; `` = `/var/backups/patching-hosts-`). Run the scripts under `skills/patching-hosts/scripts/` from the deploy checkout's root, pulled: each defaults its knob to `.skills/patching-hosts` at `git rev-parse --show-toplevel`, or else the working directory, and with no knob found it falls back to `automatic`, report-only. + +1. `sudo apt-get update`. The maintenance dry run reads the host's lists, which `apt-daily.timer` last refreshed between 12:00 and 12:30 UTC the day before (gregoryfoster/skills#467). +2. `probe.sh --lane maintenance --dry-run-into `, in the background: 0 findings expected, beside a `lists:stale` risk that reads the security archive's own publish date, not when the lists were refreshed (2026-10-26). Read the `redis-server` candidate; it must still be a `7.0.15` build, not Valkey, until #54. +3. The needrestart proof, as above. +4. `apply.sh --approve --step bulk --lane maintenance --dry-run `, in the background, within 24 h of the dry run. It restarts nothing. With no data store there is no recovery point: `recovery-point.sh` refuses ("nothing to dump"), and the bulk records the before-versions, the rollback path, in the `` it creates and prints. +5. Held steps, each approved, each `apply.sh --approve --step --run `: + - `redis`: the scratch server restarts; + - `docker`: `containerd`. Docker and containerd are disabled and inactive here, so nothing is running to restart; + - `tailscale`, if pending: a 2.3 s tailnet blip, sockets survive ([§ Tailnet flaps](NETWORK.md#tailnet-flaps), "A tailscaled restart here"). +6. The reboot and restart decision, from the host, never from this run's set: under `automatic` a `-security` library may have landed any day since the last run. `sudo needrestart -b -r l` and `sudo grep -c '(deleted)' /proc/1/maps`. The kernel never decides one on exe.dev. Expect a reboot when `systemd` or `libsystemd0` is in the set: after those families, needrestart listed dbus and logind on co-broker and co-status, and PID 1 mapped deleted files. +7. The reboot: `reboot-chain.sh --run ` prints the chain; `--approve` launches it. It stops `processor` (up to `TimeoutStopSec=240`), then reboots in-guest. **Never `exe.dev restart`**: that is a hard reset (#83). +8. After it: `probe.sh --post-boot --run `, NETWORK.md § Tailscale DNS → *Check after a reboot*, `NRestarts=0`, and drift and liveness `ok` in Status. The disk sensor's first tick after boot logs `checked in ok (ok): HTTP 202` within 10 min ([§ Disk GC](MONITORING.md#disk-gc)). Read `journalctl -u processor -p warning --since ` raw (systemd's own lines, such as a failed stop) and through `-o cat | jq -cR 'fromjson? | …'`. A crash is `PRIORITY=6`, so check `NRestarts` too. +9. The after-probe, last, after the reboot or the no-reboot restart below: `probe.sh --refresh-into `. It is what's left pending, by class, for the record, and the only step that audits the live `uv.lock` (`pending.language`). The dry run and `--post-boot` never ask OSV ([§ The lock audit](DEPENDENCIES.md#the-lock-audit)). + +**With no reboot,** restart the service anyway when needrestart lists it, or after a libssl update by either lane: `/usr/bin/python3.12`'s `_ssl` links the system `libssl.so.3`, so the old library stays mapped until a restart. `sudo grep -c 'libssl.so.3 (deleted)' /proc/$(systemctl show -p MainPID --value processor)/maps` must read 0 afterwards. + +**Liveness's grace** is 900 s. The heartbeat ticks as soon as the service starts, so the silence is at most the time since the last tick (300 s), plus the stop (240 s), plus the boot. That stays inside the grace, so Status needs a warning only if a step stalls. + +**Deploys don't wait for a window** (operator, 2026-10-26, #77). The skill's policy.md says a deploy that restarts a declared `service` takes one (gregoryfoster/skills#366). Here `scripts/deploy.sh` runs after each merge, as before: it restarts `processor` alone, verifies the new build and switches back on failure ([§ Deploy a change](DEPLOYMENT.md#deploy-a-change-scriptsdeploysh)). A merge that lands during a window is deployed after the post-boot checks, so a flap stays attributable. + +**The record** goes on the issue (`records CannObserv/processor`), per run.md §7, and ends with the next window's date. **Its first line is `## Patch run record: co-processor`**, then anything (a date, the lane), posted by an `OWNER` or a `MEMBER` whose membership is public (§ The patch-due check): that line is what [§ The patch-due check](#the-patch-due-check) reads as this week's run, a nothing-pending run's included. A run that can't take its window still posts one, saying why, or the monitor stays `missed` until the next. What each run leaves pending, by class, comes from step 9 and includes: +- **Ubuntu Pro / ESM:** Pro isn't attached, and 25 esm-apps updates were visible on 2026-10-26. +- **Language:** `/srv/processor/live/uv.lock`, its OSV advisory count, owner `self`. Run.md §7 asks for the owning repo's update mechanism; here that is `audit.yml`, report-only ([§ The lock audit](DEPENDENCIES.md#the-lock-audit)). Dependabot's version updates cover Actions only, never `uv.lock`, and its security updates, resolving through the co-pypi index since #121, are landed by hand ([§ CI pins and Dependabot](DEPENDENCIES.md#ci-pins-and-dependabot)). A count above 0 goes to that section's "Where a finding goes" and is linked from the record. A tree listed as not audited, or the class in `not_read`, is recorded as no read, never as 0. + +### The patch-due check + +`processor patching`, hourly on the hour from `processor-patching.timer` (#113), asks for the attended run, since nothing else does (the 2026-11-03 window passed unnoticed). It checks in to Status's `co-processor-patching` (tenant `co-processor`, the drift check's key and `LoadCredential=`): + +| Kind | Status | When | +|---|---|---| +| `ok` | ok | a record has been posted since the window last opened | +| `due` | alert | the window is open and no record yet: **the request for the run**, within a minute of 05:00Z | +| `missed` | alert | the window closed with no record, until it next opens. A new fault, so it reports at once | +| `knob` | alert | the window can't be read from the release's knob | +| `test` | alert | `processor patching --test-alert`, by hand | +| — | none | GitHub didn't answer: silence, which the monitor's grace turns into `missing`, never a false `ok` | + +- **The window** is the release's own `.skills/patching-hosts`, its weekday `window` lines. `read-knob.sh` isn't in a release (a `git archive` leaves the submodule empty), so `src/processor/patching.py` parses them, and `tests/test_patching.py` holds the parse equal to the skill's reader. A dated line is a one-off and opens no due period. A `[host` section is unread (`knob`). The knob is a `RUNTIME_FILES` path: a change to it is a deploy, or the drift check alerts. +- **The record** is an issue comment anywhere in this repo whose first line is `## Patch run record: co-processor`, from an `OWNER` or `MEMBER` (the repo is public: anyone else's comment never counts), created at or after the opening. **The read is anonymous, so a member counts only while their CannObserv membership is public:** a private member reads as `CONTRIBUTOR` (measured 2026-11-06 for user id 14647, before the operator made theirs public). A recorder whose membership goes private turns every week `missed`: a false alert, never a false `ok`. Check with `curl -s -o /dev/null -w '%{http_code}' https://api.github.com/orgs/CannObserv/public_members/` (204: public). One unauthenticated request an hour, newest first; past 300 comments since the opening it can't say, and is silent. +- **What it doesn't report:** a restart or reboot due. The service user can't read `processor`'s maps (it's undumpable) or run needrestart, so the weekly run's step 6 picks that up. + +Read a run with `journalctl -u processor-patching -n 3 -o cat`: one `patching check` record, its `kind`, `window`, `record` and `checkin` (Status's 202). A deliberate alert (`kind: test`, fault `test`, asking GitHub nothing) proves the channels, as the drift check's does ([§ The drift check](MONITORING.md#the-drift-check)): + +```bash +sudo systemd-run --quiet --pipe --wait --collect -p User=processor -p Group=processor \ + -p LoadCredential=status-checkin-key:/etc/processor/status-checkin.key \ + -p "$(systemctl show -p Environment processor-patching.service)" -p WorkingDirectory=/srv/processor/live \ + /srv/processor/live/.venv/bin/processor patching --test-alert +``` + +Status's side (status#95): `enabled: false` stops only its `missing` sweep, never an alert, so a check-in's `alert` pages whatever the flag. An `ok` after an alert sends Status's own `cleared` notice, so the proof is two messages: *co-processor patching: test (…)*, then *cleared* at the next hourly `ok`. + +**Silence it** for a week you mean to skip: `sudo systemctl stop processor-patching.timer`, and the monitor goes `missing`, which is honest; `start` resumes it. Better: post a record saying why, which reads `ok`. + +## Node.js (agent tooling only) + +Node 24 LTS from NodeSource's apt repo, for `using-mayfly-chat` and SocratiCode (`npx`). The `processor` unit never runs it. The cohort survey and the reasons for NodeSource and for 24 are in #6. + +```bash +sudo bash deploy/nodesource.sh install # key, source, pin, nodejs; ends with check +bash deploy/nodesource.sh check # exit 0 in sync (a pending update is a note, not drift); 3, naming each drift +``` + +| Installed | From | Why | +|---|---|---| +| `/etc/apt/keyrings/nodesource.gpg` | `https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key`, refused unless its fingerprint is `6F71 F525 2828 41EE DAF8 51B4 2F59 B5F9 9B1B E0B4` | `Signed-By:` binds the key to this one source | +| `/etc/apt/sources.list.d/nodesource.sources` | [deploy/apt/nodesource.sources](../deploy/apt/nodesource.sources) | `node_24.x`: the major line is pinned | +| `/etc/apt/preferences.d/nodesource.pref` | [deploy/apt/nodesource.pref](../deploy/apt/nodesource.pref) | `nodejs` from NodeSource at 600: above universe's 18 (500), never a downgrade. Everything else the host serves (`nsolid`) at 100, so the lane's `site=` takes only Node | + +`node` is the package's `/usr/bin/node`, so it is on every PATH: hooks, systemd user units, VS Code sessions. Installed 2026-09-30: 24.21.0, with npm 11.19.0 bundled (no Ubuntu `npm`); nothing restarted. + +**SocratiCode** (#5) runs as `exedev`, pinned to `socraticode@1.16.0` with a capped pre-install under `~/.socraticode/pin`, against co-index's shared store; it never touches the release, and the service can't read `exedev`'s home (`ProtectHome=yes`). They share 7.7 GiB with no swap, and `processor.service` carries no `MemoryLow=`: inert on this host, and the unit's `OOMScoreAdjust=-500` is the protection that counts. Pin, link stubs, memory reasoning: [docs/SOCRATICODE.md](SOCRATICODE.md#repo-specific-notes). + +### Patching Node + +unattended-upgrades takes Ubuntu's security origins alone ([§ Patching](#patching)), so nothing updates Node on its own. NodeSource ships security fixes in its own repo, never `noble-security`, so its policy is **follow**: `nodejs` rides the weekly maintenance lane. Never a bare `apt-get upgrade`. + +**Select it by host, not by origin.** NodeSource's Release file says `Origin: . nodistro`, aptly's default, which every aptly-published `nodistro` repo shares. So the knob names the site, `origin deb.nodesource.com follow` (a value holding a dot is read as a site: the skill's knob.md), and the lane's `APT_CONFIG` takes: + +``` +Unattended-Upgrade::Origins-Pattern { + "origin=Ubuntu,archive=${distro_codename}-updates"; + "site=deb.nodesource.com"; +}; +``` + +Proven 2026-09-30, with `nodejs` stepped back to 24.20.0: `unattended-upgrade --dry-run` selected it under `site=deb.nodesource.com` and not under the stock security-lane config. `origin=. nodistro` also selects it, but can't tell NodeSource from any other aptly repo. The skill's policy.md cites `nodesource.pref` as its example of scoping a followed origin (gregoryfoster/skills#344). + +**An out-of-cycle fix** (a Node security release): `sudo bash deploy/nodesource.sh install`. It refreshes the package lists first, which `apt-daily.timer` does only daily, between 12:00 and 12:30 UTC, so a bare `apt-get install nodejs` can answer "already the newest version" while the release sits unfetched. It then takes the newest `nodejs` candidate, at adj 0 with needrestart listing only, and ends with `check`. `check` alone reads the lists as of the last update. + +### Moving the major line + +In one commit, edit `node_.x` in `deploy/apt/nodesource.sources`, `EXPECTED_MAJOR` in `tests/test_nodesource.py`, this section's "Node LTS" and table row, and AGENTS.md's Node.js row; the test fails until all of them agree. Then `sudo bash deploy/nodesource.sh install`. Read `init-socraticode`'s preflight first: Node 26 crashes SocratiCode older than 1.13. + +### Removing it + +```bash +sudo apt-get purge nodejs +sudo rm /etc/apt/keyrings/nodesource.gpg /etc/apt/sources.list.d/nodesource.sources /etc/apt/preferences.d/nodesource.pref +sudo apt-get update +``` diff --git a/docs/PROVISIONING.md b/docs/PROVISIONING.md index b99f23c..5a5a996 100644 --- a/docs/PROVISIONING.md +++ b/docs/PROVISIONING.md @@ -155,7 +155,7 @@ CO_PROCESSOR_BUS_URL=redis://processor:@broker:6379/0 - **The Status key** is `/etc/processor/status-checkin.key`. Each unit that checks in loads it as `LoadCredential=status-checkin-key:…`: - `processor-drift.service`, which reads no `.env`, so the drift check never holds the broker credential (MONITORING.md § The drift check); - `processor.service`, since #39, for the liveness check-in (MONITORING.md § Liveness). Landlock refuses it to the child, as it refuses the GCS key; - - `processor-patching.service`, since #113, which reads no `.env` either (DEPLOYMENT.md § The patch-due check). + - `processor-patching.service`, since #113, which reads no `.env` either (PATCHING.md § The patch-due check). A command that needs the service's settings runs as the service, through systemd. Don't load the file into a shell: diff --git a/docs/SKILLS.md b/docs/SKILLS.md index 6d7210d..074e66a 100644 --- a/docs/SKILLS.md +++ b/docs/SKILLS.md @@ -41,7 +41,7 @@ The first session in a fresh clone or worktree fails the hook with exit 127 (hoo ## Selection -**gregoryfoster-skills (16):** `auditing-ci-cost` (CI since #12), `coordinating-epics` (#24 since #52), `curating-context`, `enforcing-architecture` (`reviewing-architecture` delegates to it), `init-socraticode`, `maintaining-host-disk` (the host's disk GC and headroom sensor since #82: [MONITORING.md § Disk GC](MONITORING.md#disk-gc)), `managing-skills`, `orchestrating-issue-backlog`, `patching-hosts` (the host's patch posture since #77: `.skills/patching-hosts`, [DEPLOYMENT.md § Patching](DEPLOYMENT.md#patching)), `reviewing-architecture`, `reviewing-code` (the stack-agnostic fallback, linked in #77 for #76's CR 11), `reviewing-code-python-fastapi`, `shipping-work` (since #115; was its FastAPI variant), `using-git-worktrees`, `using-mayfly-chat`, `writing-plans`. +**gregoryfoster-skills (16):** `auditing-ci-cost` (CI since #12), `coordinating-epics` (#24 since #52), `curating-context`, `enforcing-architecture` (`reviewing-architecture` delegates to it), `init-socraticode`, `maintaining-host-disk` (the host's disk GC and headroom sensor since #82: [MONITORING.md § Disk GC](MONITORING.md#disk-gc)), `managing-skills`, `orchestrating-issue-backlog`, `patching-hosts` (the host's patch posture since #77: `.skills/patching-hosts`, [PATCHING.md § Patching](PATCHING.md#patching)), `reviewing-architecture`, `reviewing-code` (the stack-agnostic fallback, linked in #77 for #76's CR 11), `reviewing-code-python-fastapi`, `shipping-work` (since #115; was its FastAPI variant), `using-git-worktrees`, `using-mayfly-chat`, `writing-plans`. **obra-superpowers (12):** `brainstorming`, `dispatching-parallel-agents`, `executing-plans`, `finishing-a-development-branch`, `receiving-code-review`, `requesting-code-review`, `subagent-driven-development`, `systematic-debugging`, `test-driven-development`, `using-superpowers`, `verification-before-completion`, `writing-skills`. diff --git a/pyproject.toml b/pyproject.toml index bd5e9fe..c3aca19 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -48,7 +48,7 @@ dev = [ # co-pypi index and no token (#121). shellcheck-py's wheel carries the shellcheck binary, so the lock # hash-pins it: no root install on the node, one version everywhere. A bump moves the # gate: `uv lock --upgrade-package shellcheck-py`, then fix what it finds and the -# version docs/DEPLOYMENT.md § Shell lint names, which tests/test_shellcheck.py holds. +# version docs/DEPENDENCIES.md § Shell lint names, which tests/test_shellcheck.py holds. lint = ["ruff>=0.16.8,<0.17", "shellcheck-py>=0.11.0.1,<0.12"] # scripts/co_pypi_token.py's own dep, locked (#79's rule, #121): scripts/co-pypi.sh # runs it `--frozen --only-group co-pypi`, before any co-core file is fetched, and diff --git a/scripts/deploy.sh b/scripts/deploy.sh index 23c6680..0c2c337 100755 --- a/scripts/deploy.sh +++ b/scripts/deploy.sh @@ -371,10 +371,10 @@ else fi touch "$release" # prune by last deploy, not first build -# The unit's user must exist before anything switches (DEPLOYMENT.md § Install). +# The unit's user must exist before anything switches (PROVISIONING.md § Install). user="$(sed -n 's/^User=//p' "$release/deploy/$UNIT.service")" getent passwd "$user" >/dev/null || - die "the unit runs as $user, who does not exist here; nothing switched (DEPLOYMENT.md § Install)" + die "the unit runs as $user, who does not exist here; nothing switched (PROVISIONING.md § Install)" # --- switch, verify, or switch back ---------------------------------------- @@ -498,7 +498,7 @@ enable_new_timers() { note "$name enabled; its first run: journalctl -u ${name%.timer} -n 5" logger -t processor-deploy "$name enabled" || true else - note "systemctl enable --now $name failed; enable it by hand (docs/DEPLOYMENT.md § The drift check)" + note "systemctl enable --now $name failed; enable it by hand (docs/MONITORING.md § The drift check)" fi done <"$backup/units.added" } @@ -588,9 +588,9 @@ compare_host_configs() { rel="${entry%%=*}" dest="$ETC/${entry#*=}" [[ -f "$release/deploy/$rel" ]] || continue if [[ ! -e "$dest" ]]; then - say "host config deploy/$rel is not installed at $dest (docs/DEPLOYMENT.md)" + say "host config deploy/$rel is not installed at $dest (where each is installed: docs/DEPLOYMENT.md § Deploy a change)" elif ! cmp -s "$release/deploy/$rel" "$dest"; then - say "host config deploy/$rel differs from $dest; install it by hand (docs/DEPLOYMENT.md)" + say "host config deploy/$rel differs from $dest; install it by hand (where: docs/DEPLOYMENT.md § Deploy a change)" fi done return 0 diff --git a/src/processor/checkin.py b/src/processor/checkin.py index 99877ee..74f83ba 100644 --- a/src/processor/checkin.py +++ b/src/processor/checkin.py @@ -63,7 +63,7 @@ def post_checkin( if not _KEY.fullmatch(key): raise CheckinFailed( f"the {CREDENTIAL_NAME} credential is not one token of printable ASCII; " - "reinstall it (docs/DEPLOYMENT.md § The drift check)" + "reinstall it (docs/MONITORING.md § The drift check)" ) url = f"{base_url.rstrip('/')}/api/v1/monitors/{monitor_id}/checkin" body: dict[str, object] = {"status": status, "variables": variables} diff --git a/src/processor/patching.py b/src/processor/patching.py index 5be1e45..2d8e1e5 100644 --- a/src/processor/patching.py +++ b/src/processor/patching.py @@ -247,14 +247,14 @@ def _assess(get: Get, this: Opening, now: datetime) -> Verdict: return Verdict( "due", f"the patch window is open until {closes}, and this week's attended run " - "isn't recorded: approve it in co-processor's session (docs/DEPLOYMENT.md " + "isn't recorded: approve it in co-processor's session (docs/PATCHING.md " "§ Patching)", window, ) return Verdict( "missed", f"the patch window closed at {closes} with no run recorded: run it out of window " - "or record why not (docs/DEPLOYMENT.md § Patching)", + "or record why not (docs/PATCHING.md § Patching)", window, ) diff --git a/tests/test_apt_automatic.py b/tests/test_apt_automatic.py index 1ad2dd7..ed9bfae 100644 --- a/tests/test_apt_automatic.py +++ b/tests/test_apt_automatic.py @@ -12,7 +12,7 @@ ``apply.sh`` doesn't read ``apt-daily.timer``; its ``apt update`` can still meet the bulk's own list refresh, so it moves too. -Tracked in ``deploy/``, installed by hand (docs/DEPLOYMENT.md § Patching) as: +Tracked in ``deploy/``, installed by hand (docs/PATCHING.md § Patching) as: - ``apt/apt.conf.d/processor-periodic`` -> ``/etc/apt/apt.conf.d/`` - ``apt-daily.timer.d/processor-window.conf`` -> ``/etc/systemd/system/apt-daily.timer.d/`` @@ -228,7 +228,7 @@ def test_installed_copy_matches_tracked(tracked: Path) -> None: try: text = installed.read_text() except FileNotFoundError: - pytest.fail(f"{installed} is not installed on {NODE} (docs/DEPLOYMENT.md § Patching)") + pytest.fail(f"{installed} is not installed on {NODE} (docs/PATCHING.md § Patching)") assert text == tracked.read_text() diff --git a/tests/test_needrestart.py b/tests/test_needrestart.py index 8cac434..e1132b7 100644 --- a/tests/test_needrestart.py +++ b/tests/test_needrestart.py @@ -90,7 +90,7 @@ def test_installed_copy_matches_tracked() -> None: try: installed = INSTALLED.read_text() except FileNotFoundError: - pytest.fail(f"{INSTALLED} is not installed on {NODE} (docs/DEPLOYMENT.md § Patching)") + pytest.fail(f"{INSTALLED} is not installed on {NODE} (docs/PATCHING.md § Patching)") assert installed == DROPIN.read_text() diff --git a/tests/test_nodesource.py b/tests/test_nodesource.py index 45efd02..12d128e 100644 --- a/tests/test_nodesource.py +++ b/tests/test_nodesource.py @@ -1,4 +1,4 @@ -"""The NodeSource apt setup is pinned and self-consistent (docs/DEPLOYMENT.md, #6). +"""The NodeSource apt setup is pinned and self-consistent (docs/PATCHING.md, #6). Node.js is agent tooling only (using-mayfly-chat, SocratiCode); the service never runs it. Moving the major line is a deliberate act: edit ``EXPECTED_MAJOR`` here in the same @@ -65,19 +65,19 @@ def test_script_installs_what_the_repo_versions() -> None: assert _script_value("FINGERPRINT") == FINGERPRINT -def test_deployment_doc_names_the_same_fingerprint() -> None: - doc = re.sub(r"\s", "", (ROOT / "docs" / "DEPLOYMENT.md").read_text()) +def test_patching_doc_names_the_same_fingerprint() -> None: + doc = re.sub(r"\s", "", (ROOT / "docs" / "PATCHING.md").read_text()) assert FINGERPRINT in doc def test_docs_follow_the_major_line_and_the_installed_paths() -> None: """A bump or a rename that leaves the docs behind fails here, not in an operator's shell.""" - deployment = (ROOT / "docs" / "DEPLOYMENT.md").read_text() - assert f"Node {EXPECTED_MAJOR} LTS" in deployment - assert f"node_{EXPECTED_MAJOR}.x" in deployment + patching = (ROOT / "docs" / "PATCHING.md").read_text() + assert f"Node {EXPECTED_MAJOR} LTS" in patching + assert f"node_{EXPECTED_MAJOR}.x" in patching for name in ("KEYRING", "SOURCES", "PREF"): # The table and the removal block both name each path. - assert deployment.count(_script_value(name)) >= 2, name + assert patching.count(_script_value(name)) >= 2, name assert f"{EXPECTED_MAJOR} LTS from NodeSource" in (ROOT / "AGENTS.md").read_text() diff --git a/tests/test_patching_knob.py b/tests/test_patching_knob.py index 0bcf305..583b1d4 100644 --- a/tests/test_patching_knob.py +++ b/tests/test_patching_knob.py @@ -149,9 +149,9 @@ def test_the_language_read_finds_the_live_lock() -> None: def _patching_steps() -> str: """§ Patching's numbered run, from "This host's run" to the next paragraph.""" - deployment = (ROOT / "docs" / "DEPLOYMENT.md").read_text() - assert "\n## Patching\n" in deployment, "DEPLOYMENT.md has no ## Patching" - section = deployment.split("\n## Patching\n", 1)[1].split("\n## ", 1)[0] + patching = (ROOT / "docs" / "PATCHING.md").read_text() + assert "\n## Patching\n" in patching, "PATCHING.md has no ## Patching" + section = patching.split("\n## Patching\n", 1)[1].split("\n## ", 1)[0] assert "**This host's run**" in section, "§ Patching has no **This host's run**" run = section.split("**This host's run**", 1)[1] return run.split("\n\n**", 1)[0] @@ -166,10 +166,14 @@ class in ``not_read``, and no probe reads the lock, so the runbook names it (#75 def test_the_runbook_sections_exist() -> None: - """The knob, AGENTS.md and docs/SKILLS.md link DEPLOYMENT.md's sections by anchor.""" - deployment = (ROOT / "docs" / "DEPLOYMENT.md").read_text() + """AGENTS.md and docs/SKILLS.md link PATCHING.md's sections by anchor; links from + before #126 reach them through DEPLOYMENT.md's stubs, so both keep the headings.""" + patching = (ROOT / "docs" / "PATCHING.md").read_text() for heading in ("## Patching", "### Patching Node"): - assert re.search(rf"^{heading}$", deployment, re.M), heading + assert re.search(rf"^{heading}$", patching, re.M), heading + deployment = (ROOT / "docs" / "DEPLOYMENT.md").read_text() + for heading in ("### Patching", "### Node.js (agent tooling only)"): + assert re.search(rf"^{re.escape(heading)}$", deployment, re.M), heading def test_the_gate_reads_the_extraction_and_the_unit() -> None: diff --git a/tests/test_tailscaled.py b/tests/test_tailscaled.py index c16be4b..d4453dc 100644 --- a/tests/test_tailscaled.py +++ b/tests/test_tailscaled.py @@ -43,6 +43,7 @@ "DEPENDENCIES.md", "MONITORING.md", "NETWORK.md", + "PATCHING.md", ) ] RESOLV_BACKUP = "/etc/resolv.pre-tailscale-backup.conf" diff --git a/tests/test_units.py b/tests/test_units.py index 80f94d1..5edc031 100644 --- a/tests/test_units.py +++ b/tests/test_units.py @@ -259,6 +259,7 @@ def test_the_install_never_globs_inside_etc_processor() -> None: "DEPENDENCIES.md", "MONITORING.md", "NETWORK.md", + "PATCHING.md", ) ) assert "/etc/processor/*" not in runbook