Repository navigation
Description
The 2026-11-17 patch run (#113, record) left two findings open, both dormant components:
named:docker: idle, butdeploy/names it, so the operator reads what that says, then either proposes the prune or declareskeep:docker.dormant:nginx: idle, nothing names it. The skill's verdict is to prune it or declarekeep:nginx.
They are new findings, not new state: the 11-10 run reported none. The probe has reported them since the vendored skill was bumped (217364d, 8a1b79f). Until each is pruned or excepted, every patch run's before- and after-probe lists both.
What the host shows (read 2026-11-17, non-root unless noted)
Both came with the image. One apt-get install -y … nginx … docker.io docker-buildx docker-compose-v2 … at 2026-09-29 17:12Z, the exeuntu build. The skill's profile says the same of nginx: it "ships with the image, disabled, and was in the security set on six hosts: patched for nothing."
docker:
- Installed:
docker.io29.1.3,containerd2.2.1,runc1.3.4,docker-buildx0.30.1,docker-compose-v22.40.3. docker.service,docker.socketandcontainerd.serviceare alldisabled.docker.socketwas disabled 2026-10-05 (Contain the extraction child: it runs as the service user and can reach the service's secrets #2 gate 1, docs/PROVISIONING.md), since/run/docker.sockis root to thedockergroup.- The probe read it as inactive for 49 days, never started since boot, 0 containers, 0 volumes.
exedevis still in groupdocker(gid 107). That is root-equivalent the moment Docker comes back. A purge deletes the group (pruning.md,group; cf. co-watcher leftovers: docker group (exedev member) and the retired notifier backup key watcher#336).- What the repo says about it is never a use on this host:
deploy/processor.service:processoris not in groupdocker;deploy/apt/apt.conf.d/processor-periodic: the name sorts after the image'sdocker-disable-periodic-update. That file is owned by no package (dpkg -Sfinds none), so a purge leaves it, andtests/test_apt_automatic.pykeeps reading it;deploy/disk-gc/disk-gc.conf: nodockertier, because the daemon is inactive;tests/test_child.py: the child refusessocket()before any connect, so/run/docker.sockneed not exist;- CI's
redis:7.0.15service runs on GitHub's runners, not here. Tests here use the aptredis-server.
nginx:
- Installed:
nginxandnginx-common1.24.0-2ubuntu7.19. nginx.serviceisdisabled. The probe read it as inactive 48 days;/var/log/nginxis all empty files, the newest 2026-09-29.sites-enabled/defaultis the image's link. Nothing in this repo names nginx.
Proposed route (the vendored patching-hosts skill, references/pruning.md)
For each component, one stage at a time. Each stage gets its own approval in this repo's session, and every prune-plan.sh/prune.sh run escalates through sudo -n, so each is a sudo step.
- Plan:
prune-plan.sh --component docker --out <file>and--component nginx --out <file>. They change nothing. Attach the JSON or summarise it here:units,packages(remove/purge sets, held),autoremove_would_take,reverse_depends,purge_scripts,paths,conffiles,group.- Read
autoremove_would_takeclosely for docker. On noble, docker.io with its recommends orphans about 47 packages,iptables,gitandopenssh-clientamong them. Heregitandopenssh-clientwere named in the same install line, so they should be manual and stay. The plan confirms it.
- Decide
docker: prune, orexception keep:docker <review-by> <reason>. Nothing found here needs it. - Disable:
- Command:
prune.sh --stage disable --plan <fresh plan> --approve. It stops, disables and masks every unit the packages ship. - Knob line:
exception disabled:<name> <review-by> <reason>. Its review-by date is the soak: 30 days by default. - The knob is a
RUNTIME_FILESpath, so the knob change is a PR and a deploy.
- Command:
- Savepoint (optional, preferred), taken during the soak:
prune.sh --stage savepoint --plan <fresh plan> --approve. It tarballs the config and conffiles at mode 600. With it, the remove stage is skipped, and norcresidue is left holding config. - Purge, once the soak ends:
- Command:
prune.sh --stage purge --plan <fresh plan> --approve --savepoint <dir>. - nginx-common's purge script deletes
/etc/nginxand/var/log/nginx, so its purge needs--purge-data /var/log/nginx. - docker.io's purge leaves
/var/lib/docker, which holds 0 containers and 0 volumes here. Name it with--purge-dataonly if the plan lists it and the operator wants it gone. - Knob line:
exception purged:<name> <review-by> <reason>, which stays until the image stops shipping it.
- Command:
- Docs:
- docs/PROVISIONING.md: the
docker.socketrow, and thedockergroup note. - docs/MONITORING.md and
deploy/disk-gc/disk-gc.conf: the "daemon is inactive" reason for nodockertier. - docs/PATCHING.md: the knob table, for the new exception lines.
- docs/PROVISIONING.md: the
Run it outside a patch window, never during one, and pick the order with the operator. nginx is the simpler first: no group, no reverse dependants expected.
The base image
pruning.md: "Where the component shipped with the image on every host, the prune belongs in the base image instead … Until it gets there, a prune on one host is drift from that image, and the knob declares it." Both ship in exeuntu on every cohort host. A note to the image's owner is a cross-repo post, so it is drafted here first and posted only with the operator's approval (AGENTS.md § Cross-repo Policy).
Done when
- The probe on co-processor reports neither
named:dockernordormant:nginxas a finding, either because each is purged (knobpurged:) or because each is declaredkeep:with a reason. exedevis out of groupdocker, orkeep:dockerstates why it stays.- The docs above match the host.
Activity
gregoryfoster commentedon Nov 17, 2026
Brief: #143, prune the dormant docker and nginx (coordinator, epic #84, 2026-11-17)
Authorities: this issue's body is the spec. This brief adds a pre-flight against
origin/main8a1b79fand the host. The vendored skill at063ba5db(skills/patching-hosts/references/pruning.md) governs the mechanics. Where the body and the skill disagree on mechanics, follow the skill. Where they disagree on scope, stop and ask. Labels:[measured]read from the host or a registry in this pre-flight;[traced]followed through the repo or a sibling's;[stated]an issue or a person says so.What the host shows now
[measured, 2026-11-17 ~20:00Z, non-root]The body's inventory holds:
docker.io29.1.3,containerd2.2.1,runc1.3.4,docker-buildx0.30.1,docker-compose-v22.40.3,nginxandnginx-common1.24.0-2ubuntu7.19.docker.service,docker.socket,containerd.serviceandnginx.serviceare eachdisabledandinactive. None is masked yet./run/docker.sockis absent.docker:x:107:exedev.Two corrections and one refinement:
The
dockergroup adds no access forexedevhere.exedevis in groupsudo, andsudo -nworks, so it already has root.processor(uid 997) is in no group but its own. So the group is hygiene, not exposure: deleting it at the purge is right, and Tier 3 is the right tier. The body's "root-equivalent the moment Docker comes back" is true in general, but grantsexedevnothing it lacks.Upstream's "about 47 orphans" for docker.io doesn't apply here.
apt-get -s purge --autoremoveon the five docker packages takes 9 packages:pigz, the only new orphan;bridge-utils,dns-root-dataanddnsmasq-base.git,openssh-client,docker.io,docker-buildx,docker-compose-v2andnginxare marked manual.iptablesstays:tailscaleandubuntu-standarddepend on it.bridge-utils,dns-root-dataanddnsmasq-baseare orphans already.apt-get -s autoremovetakes them today, before any prune. That's why the nginx purge simulation also lists them.prune.shnever autoremoves (pruning.md: "No autoremove, ever"), so they stay unless the plan takes them by name with--take-orphans. The weighed choice is below.prune-plan.shis the reading of record. Compare itsautoremove_would_takeandpackagesagainst the above, and say on the issue where they differ.Cohort precedents
[traced: each repo's .skills/patching-hosts]disabled:nginxanddisabled:docker, masked 2026-11-13disabled:nginx, masked 2026-11-17disabled:nginxtests/test_patching_knob.py::test_the_knob_reads_cleanassertsfindings == []. Adisabled:line's review-by passing turns into a finding, and CI goes red by calendar. Archiver'stests/deploy/test_patching_knob.py(_assert_clean,PRUNE_STAGES = ("disabled:", "removed:")) lets an expired exception through only on a prune-stage line, and keeps every other finding red. pruning.md asks for exactly that: "A repo that tests its knob allowssoak-endedon adisabled:orremoved:line". Port it test-first, before the firstdisabled:line lands. Whether the reader reports it assoak-endedorexpired-exceptionat this pin is for you to read fromread-knob.sh; archiver's filter matchesexpired-exception.Shape: two phases, two deploys
.skills/patching-hostsis aRUNTIME_FILESpath (src/processor/drift.py:65). So each knob change is a PR, thenscripts/deploy.sh, which restartsprocessorfor a few seconds. Deploys don't wait for a window (PATCHING.md).prune-plan.shfor each component;disablefor both;exception disabled:<name> <review-by> <reason>plus the ported test, as one PR and one deploy, with the docs left as they are (the host still matches them while the units are only masked);Refs #143, notCloses.purgewith--savepoint;exception purged:<name> <review-by> <reason>, since exeuntu keeps shipping both;Operator decisions
keep:docker— recommend prune. Nothing on this host uses it (the body's survey, andprocessorisn't in the group), and archiver pruned both.]--take-orphans— recommend no. It would namepigz(docker's one new orphan) and pull in the three orphans that predate the prune. They're harmless, and taking them widens the change past the issue.]--purge-data /var/lib/docker— recommend yes, if the plan lists it. It's root-only, with 0 containers and 0 volumes per the patch run's probe[stated: #113 comment 6956497896].--purge-data /var/log/nginxis required, per the body.]Starting the thread approves the recommendations. Every
prune-plan.shandprune.shrun escalates throughsudo -nby itself (this issue's body; PATCHING.md:61 forprobe.sh), so each one is a sudo step that needs approval in this repo's session. A plan binds one stage and expires in 24 h: take a fresh one for each stage.Every mention on
origin/main[traced: git grep -i 'docker|nginx', skills-vendor and uv.lock excluded]The body names PROVISIONING, MONITORING,
disk-gc.confand PATCHING. The search finds more. Each row is a recommendation to weigh:docs/PROVISIONING.md:19,:175,:193docker.socketrow, its rationale, and theis-enabled docker.socketcheckdpkg -s docker.ioabsentdocs/PROVISIONING.md:18,AGENTS.md:40,deploy/processor.service:12processor… not indocker"deploy/disk-gc/disk-gc.conf:12,docs/MONITORING.md:132dockertier "because the daemon is inactive"tests/test_disk_gc.py:68still holdsdocs/PATCHING.md:71docker: "Docker and containerd are disabled and inactive"[traced: grep]docs/SOCRATICODE.md:144-145, not in the bodydocker.socketis disabled (#2)"tests/test_child.py:380/run/docker.sockparameter, "each when present"socket()before any pathtests/test_apt_automatic.py:55,deploy/apt/apt.conf.d/processor-periodic:5docker-disable-periodic-update/etc/apt/apt.conf.d/docker-*[measured: dpkg -S]; they're the image's apt files, so the purge leaves them:345,:365;src/processor/_contain.py:14;docs/plans/*.github/workflows/ci.yml,tests/test_ci.pyDoc headroom (exact counts from the 2026-11-10 measurement; sizes unchanged or smaller since): PATCHING.md 8,690 / 10,000; PROVISIONING.md 8,299; MONITORING.md 7,629; AGENTS.md 5,945 / 6,000. Phase 2 should be net-neutral.
The base image
CannObserv/provisioner's co-image-base is
FROMexeuntu with no apt removal (image/Dockerfile)[traced]. So moving co-processor onto it (CannObserv/provisioner#16, group 3) drops neither package, and thepurged:lines outlive the move. No CannObserv/provisioner issue asks the image to drop them yet[measured: issue search]. Archiver, watcher and index carry the same drift. Draft the note on this issue and stop. It's a cross-repo post, and the operator approves it (AGENTS.md § Cross-repo Policy). It doesn't block either phase.Who closes, and when
named:dockernordormant:nginx. Its closing comment names the final SHA and the probe's run directory.Out of scope
--take-orphans);gregoryfoster commentedon Nov 17, 2026
Phase 1 pre-flight: corrections to the brief (2026-11-17 ~21:10Z)
Read against
origin/main8a1b79f, the skill at063ba5db, and the host. The operator approved the brief's four recommendations (prune docker, 7-day soak, no--take-orphans,--purge-data /var/lib/dockerif listed) and the twoprune-plan.shruns below.The plans
[measured: prune-plan.sh, 2026-11-17T21:06Z, both exit 0, not_read empty]They match the brief on packages, units, holds and the group:
nginx,nginx-common. No drag-along, nothing held, no reverse dependants.autoremove_would_takeis empty: the three pre-existing orphans aren't nginx's, so the plan leaves them out.docker.io,containerd,runc,docker-buildx,docker-compose-v2.autoremove_would_take=pigzonly. Groupdocker, membersexedev.nginx.service,docker.socket,docker.service,containerd.service, eachdisabled/inactive, unit files under/usr/lib/systemd/system, so the disable masks all four.declared: none,next: disable.Corrections
soak-ended, notexpired-exception, at this pin._knob-lib.sh:657-670(#636) gives a passeddisabled:/removed:line kindsoak-ended. Every other passed line,purged:included, staysexpired-exception. knob.md:65 asks a repo's test to allow exactly that. Archiver's_assert_cleanmatchesexpired-exception, so copied verbatim it would go red on 2026-11-25. The port matchessoak-endedon a prune-stage line and keeps everything else red.--todaypast it):disabled:andremoved:pass,purged:andkeep:stay red. That's red before the helper exists, so the test comes first, ahead of thedisabled:lines./var/lib/containerd(96 KiB) as data, beside/var/lib/docker(184 KiB, only abuildkit/dir). Both were created 2026-09-30 19:50Z, so the daemon ran once the day after the image build, and never since boot./etc/dockeris empty./var/www(16 KiB, an emptyhtml/) as data. nginx-common's purge script deletes/var/lib/nginx(empty),/var/log/nginxand/etc/nginx. So--purge-data /var/log/nginxis required, as the body says./var/wwwstays unless named./etc/docker. nginx has four:/etc/default/nginx,/etc/init.d/nginx,/etc/logrotate.d/nginxand/etc/ufw/applications.d/nginx. The savepoint keeps them.today > review-by), so:kept, not as findings (probe.sh:120);soak-ended, and with it the purge, is due from 2026-11-25, clear of the window.docker.socketreadsmasked, notdisabled. That affects docs/PROVISIONING.md:194's check (systemctl is-enabled docker.socket # disabled) and its row at :19. The brief leaves the docs to phase 2, so phase 1 leaves them, and the check's comment is stale while the unit is masked. Nothing automated runs that check (git grepfinds no other caller), so no test goes red. Phase 2 retires it.src/processor/patching.pyparses onlywindowlines (parse_windows), soexceptionlines can't trip the hourly patching check. The knob is still aRUNTIME_FILESpath, so the PR needs a deploy.No scope disagreement found, so phase 1 goes ahead as briefed.
gregoryfoster commentedon Nov 17, 2026
DRAFT, not posted: proposed new issue on CannObserv/provisioner (the base-image note)
Title: co-image-base: exeuntu's dormant nginx and docker, pruned host by host, are drift from the image
Body:
Every exeuntu build installs
nginx/nginx-commonanddocker.io,containerd,runc,docker-buildxanddocker-compose-v2. They come in oneapt-get install; on co-processor that ran 2026-09-29 17:12Z. co-image-base isFROMexeuntu, andimage/Dockerfilecarries no apt removal, so moving a host onto it (#16) keeps both.Hosts that pruned them, or are pruning them, through patching-hosts'
prune.sh(pruning.md: "a prune on one host is drift from that image, and the knob declares it"):disabled:nginx,disabled:docker; purge planned (archiver#512)disabled:nginxdisabled:nginxdisabled:nginx,disabled:docker, masked 2026-11-17; purge from 2026-11-25Each ends with an
exception purged:<name>line that stays as long as the image ships the component.Two things limit what the image can take on:
image/Dockerfile, owner 2026-10-01): "only fixes every host needs", and "COPY and LABEL only, no RUN". Anapt-get purgeis a RUN, so a removal in the image breaks the rule as it stands.docker buildx(#40 test: deterministic reclaim test; sweep of timing margins in tests/ #41). nginx has no exe.dev cohort user that we know of.Asks, in your order of preference:
purged:nginxline as declared drift.purged:dockerare the intended shape for every other host.image/README.mdnaming the hosts that prune them would let the next image bump re-check the drift.Not blocking anything: #143's phase 2 goes ahead either way.
gregoryfoster commentedon Nov 17, 2026
Phase 1: done on the host, PR open, shipping paused (2026-11-17)
PR #145 (
Refs #143), head961ab930274edafc4b2950c7cbfe7125187b4b3a(11e3f1c..961ab93, 5 commits). Not merged yet: the operator ships it. Corrections to the brief: the pre-flight comment.prune-plan.sh, 21:06Z) match the brief. docker's only new orphan ispigz; nginx takes none.prune.sh --stage disable --approveat 21:57Z maskednginx.service,docker.socket,docker.serviceandcontainerd.service. Each wasdisabled/inactivebefore. Logs:/var/backups/patching-hosts-prune-{nginx,docker}/disable.log.config.tar(/etc/nginxplus 4 conffiles) is sha256120a38d7245c8877cd1ce95ce7d83210eb8fcb51109d1100d5746b14788ad1b8. docker's (/etc/docker, empty) is85c544c770f5475cce3c6a0058fea0911ab2e56ba1363c3e98070e4c1372e9a6.exception disabled:nginxanddisabled:docker, review-by 2026-11-24. The knob test now allowssoak-endedon adisabled:/removed:line and keeps every other expiry red (purged:included), test-first. CR 1–3 fixed, 4 stet, 5 held (PR body).prune-plan.shandprune.shrun (12) was approved one by one in the session, and all ran outside the Tue 2026-11-24 05:00–10:00Z window.The soak
Review-by 2026-11-24 for both. The reader expires a line only after its date. So the 11-24 window's probe reads both as
kept, andsoak-ended(the purge's prompt) is due from 2026-11-25, clear of the window.Post-merge host steps (operator, once CI is green on
961ab93)/home/exedev/processor:git fetch origin && git merge --ff-only origin/143-prune-docker-nginx && git push origin main(docs/SKILLS.md § Merging underpr; never GitHub's button)..skills/patching-hostsis aRUNTIME_FILESpath (src/processor/drift.py:65). Runscripts/deploy.shin the background and readjournalctl -t processor-deployfor the verdict.processorrestarts for a few seconds; no window needed. Skip it and the drift check alerts 8 h after the push.cat /srv/processor/live/REVISIONreads961ab9302740;systemctl is-active processoris active;bash skills/patching-hosts/scripts/read-knob.sh --host co-processorfrom the checkout givesfindings: [];systemctl is-enabled nginx.service docker.socket docker.service containerd.servicereadsmaskedfor each.probe.shdry run (a sudo step) should list docker and nginx underdormantwith verdictkept, and neithernamed:dockernordormant:nginxas a finding.sudo systemctl unmaskthe unit. Every unit wasdisabled/inactivebefore, so nothing to enable or start. Then drop its knob line in a PR.Follow-up:
prune.sh --stage purge --approve --savepoint /var/backups/patching-hosts-prune-<name>with--purge-data; the knob'spurged:lines; the docs (the brief's table); one PR and one deploy; then close Prune the image's dormant docker and nginx from co-processor (or declare keep) #143.Known limits:
keptcomes from probe.sh:120 andread-knob.sh, not from a probe reading.is-enabled docker.socket # disabled) and its row at :19 now mismatch the host (masked) until phase 2 retires them (CR 5, held to the brief's scope).prune.shsays to dump the data off the node before the purge:/var/www(an emptyhtml/) and/var/log/nginx(empty logs),/var/lib/docker(buildkit/only) and/var/lib/containerd. Measured near-empty; phase 2 decides whether a dump is worth it.exedevstays in groupdockeruntil the purge'sgroupdel. It already hassudo, so the group adds nothing (brief correction 1).For #143:
--purge-data:/var/log/nginxis required: nginx-common's purge script deletes it, and the purge refuses until it's named./var/lib/docker./var/lib/containerd(96 KiB) and/var/www(16 KiB), which no decision covers yet. Ask the operator.test_the_dormant_components_are_prunedacceptspurged:lines and a staggered order (CR 1).test_the_knobs_own_soaks_end_greenskips once nodisabled:line is left.For #84:
soak-endeddate and phase 2's re-brief. Ship #143 prune docker and nginx, phase 1: disabled: lines and the soak-safe knob test #145 first; the deploy is the only open phase-1 step.gregoryfoster commentedon Nov 18, 2026
Phase 1 shipped (2026-11-18): PR #145 fast-forwarded
mainto961ab930274edafc4b2950c7cbfe7125187b4b3a; CI passed on it (run).scripts/deploy.shexited 0:/srv/processor/live/REVISION=961ab930274e, containmentrequired, the smoke run passed,processoris active with 0 warnings since. The live knob declaresdisabled:nginxanddisabled:docker, andread-knob.shgivesfindings: []. All four units readmasked. Post-merge steps 1–3 are done; step 4 (a probe) didn't run. #143 stays open for phase 2, from 2026-11-25.gregoryfoster commentedon Nov 18, 2026
The base-image note drafted above (comment 6964631233) is filed as CannObserv/provisioner#165, with the operator's approval (2026-11-18). One change from the draft: the archiver purge ref is written in full (CannObserv/archiver#512) so it links from the provisioner repo. It blocks nothing here; phase 2 goes ahead from 2026-11-25 either way.
gregoryfoster commentedon Nov 18, 2026
Operator decision (2026-11-18), for phase 2's
--purge-data: the purge also deletes/var/lib/containerdand/var/www.So phase 2's paths are:
--purge-data /var/log/nginx(required: nginx-common's purge script deletes it) and--purge-data /var/www.--purge-data /var/lib/docker(decided 2026-11-17) and--purge-data /var/lib/containerd.As measured on 2026-11-17 by phase 1's plans
[stated: the pre-flight comment above]:/var/lib/containerd: 96 KiB;/var/lib/docker: 184 KiB, abuildkit/directory only;/var/www: 16 KiB, an emptyhtml/;/var/log/nginx: empty logs.Phase 2 takes fresh plans, so re-read each path's contents from them before the purge. If any path holds more than the above (a container, a volume, a non-empty site, a non-empty log), stop and ask before deleting it. With these contents, no off-node dump is planned; the savepoints keep the config.