Repository navigation
Closed
Description
Epic #84, Tier 1 (live exposure). Filed by the coordinator on 2026-11-15, with the operator's decision to slot it. This body is the brief. Labels: [measured] read from the live system or a registry; [traced] followed through code; [stated] an issue, a changelog or a person says so.
What's open
Dependabot alerts on uv.lock [measured]:
| Alert | Advisory | Severity | Package, locked | Fixed in |
|---|---|---|---|---|
| 12 | GHSA-mmhr-3c5m-f64f, polynomial-time DoS in the descendant combinator | high | soupsieve 2.10 | 4.0.0 |
| 14 | GHSA-6655-6qjp-jx7x, exponential ReDoS in selector compilation | high | soupsieve 2.10 | 3.0.2 |
| 13 | GHSA-7g44-g9rc-x53w, uncontrolled recursion in the CSS parser | medium | soupsieve 2.10 | 3.0.2 |
| 15 | GHSA-mf5v-w3w3-f6cg, quadratic-time traceparent parsing |
medium | opentelemetry-api 1.45.1 | 1.45.2 |
- The audit: its runs on PRs chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 and chore: update skills submodules #136 list the three soupsieve advisories.
main's last push run (34d0af0, 11-11) predates them. The scheduled run on Mon 2026-11-16 at 14:37Z will list them too. - Dependabot can't fix soupsieve: its security jobs (run 43371303257 and two more on 11-13) fail with "co-core[extract]==0.38.0 depends on soupsieve>=2.0,<3". co-core 0.38.0's
extractextra declaressoupsieve<3,>=2.0[measured: the installed METADATA]. - Exposure
[traced]: co-core'sHtmlExtractor(co_core/pure/extract/html.py) runssoup.select(sel)with Watcher's spec selectors over untrusted documents, inside the extraction child. The child's timeout andRLIMIT_ASbound the damage to one command's terminal failure, not the service. opentelemetry-api comes in through google-api-core, in the parent, and isn't part of the extraction stack.
The fix upstream, and the path here
- co-core v0.46.1 (CannObserv/cannobserv#667, 2026-11-13) requires
soupsieve>=4.1,<5. Its changelog[stated]reportsHtmlExtractor's output unchanged on 2.10, 4.0 and 4.1, and saysHtmlExtractoruses onlysoupsieve.compileandSelectorSyntaxError, which 4.1 keeps. html.pyis byte-identical in v0.38.0, v0.46.1 and our venv[measured: diffed 2026-11-15]. So co-core's compatibility check applies to our pin unchanged.- [Operator decision: how Processor takes soupsieve ≥4.1. Recommended:
[tool.uv] override-dependencies = ["soupsieve>=4.1,<5"]past co-core 0.38.0's cap, withLOCAL_GENERATION3 → 4. The alternative is a co-core bump 0.38.0 → 0.46.1+: 8 minors, breaking ones among them (none in extraction), plus regenerated goldens. That belongs in its own issue.] Starting this thread approves the recommendation.- A constraint can't do it:
constraint-dependenciesonly narrows, and co-core's<3stays. It has to be an override. - Remove the override at the next co-core bump to ≥0.46.1. Record that in
docs/DEPENDENCIES.md§ Bumping co-core, so the bump procedure removes it.
- A constraint can't do it:
- opentelemetry-api: an ordinary floor. Raise
constraint-dependenciestoopentelemetry-api>=1.45.2. Per DEPENDENCIES.md, "A security PR lands by hand, never as proposed", so close Dependabot's PR chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 in favour of this one.
Steps (docs/DEPENDENCIES.md § The lock audit, § Extraction dependencies; template #112 / PR #114)
pyproject.toml: the override (with a comment naming the advisories, co-core's cap and the removal trigger), and the opentelemetry-api floor raised to 1.45.2.bash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api, never--upgrade. The diff should move those two alone. Check every co-coresourceis still the index URL.tests/test_lock_floors.py: rows for both. Weigh a guard that fails once co-core's own requirement makes the override redundant, so a later bump remembers to remove it.tests/test_pin.py:EXTRACTION_STACKsoupsieve4.1(or whatever locks), plusLOCAL_GENERATIONandGENERATION3 → 4 (src/processor/processors/extract.py:26).processor_versionbecomes0.38.0+4.- The spec: an Amended entry, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112 made for
+3.docs/DEPENDENCIES.md: the "Since #…" line.
Hazards to weigh
- The parity goldens must pass unchanged. This is the equivalence check between soupsieve 2.10 and 4.1 on real inputs. Never regenerate them here: they're regenerated only at a co-core bump. A failing parity test means output diverged, so stop and report.
- Containment. Run the whole-corpus test in
tests/test_child.pyunderrequiredonco-processor. Its allowlist comes fromsys.path, so a pure-Python upgrade should pass, but the test is the proof. - One refusal message changes
[stated, co-core changelog]. For a selector ending in a combinator ("a >"), theUnsupportedExtractionErrorreason readsInvalid character ' ' position 1, notExpected a selector at position 3. No Processor test pins that text[traced: grep]. Aprocessing_failedreason for such a spec reads differently. That is expected. - The Watcher notice, before the deploy: the new
processor_version(0.38.0+4) and the evidence, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112 did on [Epic] Burn-down: backlog outside #368 → bug fixes, sibling waits, operational stability (2026-10) watcher#394. That's a cross-repo post: draft it and stop. The operator approves that post. - The deploy is needed (
pyproject.toml,uv.lock,src/). It also restartsprocessor, which picks up the glibc that unattended-upgrades installed on 2026-11-13: the running PID still maps the oldlibc.so.6. That settles the restart decision raised in the idle reports. It's an expected side effect, not a separate step. - After the merge: Dependabot alerts 12–15 should close on their own once
main's lock moves, and the audit onmainshould read 0. If an alert stays open, report it; don't dismiss it.
Done when
- soupsieve ≥4.1 and opentelemetry-api ≥1.45.2 locked on
main, with the floors and override tested. PR chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 closed in favour of this one. -
LOCAL_GENERATION4. The parity, containment and full suites green. CI green by job on the final SHA. - Watcher's notice posted, with approval, before the deploy.
- Deployed by
scripts/deploy.sh. The live release venv shows the new soupsieve and opentelemetry-api, and the logs showprocessor_version0.38.0+4. - The audit on
mainreads 0, and Dependabot alerts 12–15 are closed.
Out of scope
- the co-core bump itself;
- Consumer loop logs no recovery after a failure run (backoff resets silently) #134;
- Watcher's spec selectors, i.e. whether any use a descendant combinator. Unmeasured, and moot once this lands.
Metadata
Metadata
Assignees
Labels
No labels
Activity
gregoryfoster commentedon Nov 16, 2026
Pre-flight: #138 (coordinator, epic #84, 2026-11-16)
Re-checked against
origin/main263f3ae, which hasn't moved since the body was filed. The body still holds. This comment adds what a probe measured, one correction and two hazards. The probe ran in a throwaway copy oforigin/mainin the coordinator's scratch space; nothing was written to the repo.Probe: the body's recipe, run
[measured, 2026-11-16]override-dependencies = ["soupsieve>=4.1,<5"]and changed the floor to"opentelemetry-api>=1.45.2", then ranbash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api. It resolved 64 packages and moved exactly two:sources are still the co-pypi index URL;[manifest]recordsoverrides = [{name = "soupsieve", specifier = ">=4.1,<5"}].uv sync --locked: co-core 0.38.0 with soupsieve 4.1.1. The pytest header readchild containment: required (Landlock ABI 6). Results:tests/processors/test_extract.py(37 parity goldens, 12 of them refusals) andtests/test_gen_parity_goldens.pytests/test_child.py, including the whole-corpus run under containmenttests/test_pin.pysoupsieve 2.10 → 4.1.1. Fixed by step 4So soupsieve 4.1.1 changes no golden on the parity corpus. That is the equivalence the body asks for, measured ahead of the build. Rerun it in the build; don't rely on this probe.
Correction
4.1(or whatever locks)". SoEXTRACTION_STACKreads"soupsieve": "4.1.1". Keep the override's floor at>=4.1, which is co-core v0.46.1's.Hazards the body didn't name
[traced]scripts/audit_lock.sh:39-65prints "Where a finding goes" plus a::warningsaying "Re-lock with a constraint-dependencies floor".tests/test_lock_audit.py:175-190asserts that routing text.every, stack = [lines starting "- "]unpacks exactly two bullet lines. A third bullet for "a package capped by co-core: anoverride-dependenciesentry until the co-core bump" breaks that unpacking. Change the test with it.docs/DEPENDENCIES.md§ The lock audit (:125) and the test together, or leave the audit generic and document the override only in § Extraction dependencies and § Bumping co-core.tests/test_lock_floors.pychecksconstraint-dependencies. An override has no floor row there by default. Body step 3's "rows for both" means a check that readsoverride-dependencies(or the lock's[manifest] overrides) for soupsieve. The weighed guard, which fails once co-core's own requirement makes the override redundant, would sit beside it.The rest of the body stands:
processoronto the 11-13 glibc. The patch window opens Tue 2026-11-17 05:00Z; if Lock advisories: soupsieve 4.1 (extraction child, override past co-core 0.38.0's cap; LOCAL_GENERATION +4) and opentelemetry-api 1.45.2 #138 deploys first, the window's restart step finds nothing left forprocessor.gregoryfoster commentedon Nov 16, 2026
Corrections before the build (#138)
I checked the body and the pre-flight (6923328878) against
origin/main263f3ae, co-core's tags and soupsieve's changelog. Worktree138-lock-advisories. The operator decision stands (override; LOCAL_GENERATION 3 → 4), and so do the steps. The pre-flight's probe reproduces: the lock moves soupsieve 2.10 → 4.1.1 and opentelemetry-api 1.45.1 → 1.45.2, nothing else, and all three co-coresources stay the index URL. Corrections and additions:1. soupsieve 3.0 has a breaking change the brief misses: strict
:has()[measured]:has()is CSS-strict by default. A complex selector inside it (div:has(p a)) and a nested:has()(div:has(div:has(p))) now fail to compile.UnsupportedExtractionError, and so to a terminalprocessing_failed. That is a change in result, not just in a refusal's wording. So the hazard "one refusal message changes" understates it. Measured on 4.1.1:div:has(p a)→Using strict rules as proposed by CSS, the complex selectors at postition 9 is not allowed;div:has(div:has(p))→:has() was found to be nested under :has() at position 16;div:has(> p)still compiles: a relative selector inside:has()is still allowed.div.content,div.view-display-id-bill_action_notice_block) are unaffected, and the parity cases use#sidealone. Watcher's live specs are unmeasured from here. The notice asks Watcher to check for a:has(with a combinator inside it, or a nested one.custom.2. 4.0's
<template>break: co-core's claim holds[measured]soup.select("div.c")no longer matches inside a<template>(it did on 2.10).HtmlExtractortakes no text from a template's contents on either version. A probe with selectors,exclude_selectorsand no selectors at all gave identical text on 2.10 and 4.1.1.<iframe>break is moot under lxml, which escapes an iframe's contents (also probed).3. The procedure's wider-corpus step is missing from the steps
[measured]HtmlExtractorover the parity HTML, the real corpus, and every HTML file over 3 KB on the host;:has,:lang, attributes,nth-*;div:has(p a)anddiv:has(div:has(p)): 289 files each go from a result on 2.10 toUnsupportedExtractionErroron 4.1.1 (288 extracted, 1 timed out; see the next bullet). This is correction 1, measured."a >": refused on both versions, with the reworded reason co-core names (289).nodejs/api/all.html), 2 of them the nested and complex:has();:lang(en),divwithignore_selectorsp:first-child). That's size, not a regression;:has()selectors at compile time;4. Audit state
[measured]main's last push run (34d0af0, 11-11) predates them" is stale.main's push run on263f3ae(43757382623, 2026-11-15T20:19Z) already lists all four advisories.mainreading 0 is checked on this PR's merge push.5. Scope the brief leaves implicit
LOCAL_GENERATIONline and §7's pin-test line ("0.38.0+3",:716) move, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112's did.test_lock_floors.py's docstring name opentelemetry-api 1.45.1. Both move to 1.45.2 and gain the override.scripts/audit_lock.sh's two routing lines, and the test that unpacks them, stay unchanged. DEPENDENCIES.md carries the override route instead:tests/test_lock_floors.pyreads the installed co-core'sRequires-Distfor soupsieve underextra == "extract". Once co-core's own floor is ≥ 4.1, the guard fails, naming the override to remove. Beside it go a lock and venv floor of 4.1, and a check that the override is declared.6. Notes, no change
:lang(),:-soup-contains()). Once it ships, picking it up is another stack move, and so another generation bump.docs/plans/2026-11-16-138-soupsieve-override.mdon the branch.Building now.
#138 fix: CR 1 — the override guard fails once co-core admits the flo…
#138 fix: CR 2 — the override's comment names when the test fails
#138 fix: CR 3 — exact name and extra matching in the floor tests
#138 fix: CR 4 — the Amended entry's results read as a list, one of t…
gregoryfoster commentedon Nov 16, 2026
Post-merge host steps (#138, PR #139)
Final SHA
82f35a8. CI is green by job on it (run 43925929773:lintandtest, 1026 passed,child containment: required (Landlock ABI 8)). The PR's audit (43925929751) readsFound no known vulnerabilities … in 63 packages. In order, from/home/exedev/processorasexedev:Merge. FF from the deploy checkout:
gh pr view 139says MERGED.CI on
main's push.ci.ymlgreen by job on82f35a8;scripts/deploy.sh's gate reads that run.The Watcher notice on [Epic] Burn-down: backlog outside #368 → bug fixes, sibling waits, operational stability (2026-10) watcher#394. The operator approves this post. It goes up before the deploy.
Deploy:
scripts/deploy.sh. Read the verdict line:journalctl -t processor-deploy -n 5 -o cat.The live release:
cat /srv/processor/live/REVISIONreads82f35a8bd94b9669aa54e82e282da3a332b8564d;/srv/processor/live/.venv/bin/python -I -c "import importlib.metadata as m; print(m.version('soupsieve'), m.version('opentelemetry-api'))"reads4.1.1 1.45.2;journalctl -u processor -o cat --since -10min | jq -cR 'fromjson? | select(.message=="starting") | {processor_version, build}'readsprocessor_version0.38.0+4.glibc, as a side effect of the restart:
sudo grep -c 'libc.so.6 (deleted)' /proc/$(systemctl show -p MainPID --value processor)/mapsreads0. With that, Tuesday's window (2026-11-17 05:00Z) finds no restart left forprocessor.Health:
systemctl show -p NRestarts processorreads0;journalctl -u processor -p warning --since -10minis empty;processor-drift.timer, hourly) and liveness areokin Status;content.derivedfacts carry0.38.0+4(anack: completewith noprocessing_failedspike).Advisories:
main's push run of82f35a8reads 0.gh api 'repos/CannObserv/processor/dependabot/alerts?state=open' --jq lengthreads0. If one stays open, report it, don't dismiss it.gregoryfoster commentedon Nov 16, 2026
Shipped in #139 (
2775d73..82f35a8, FF-merged;mainCI 43940376546 green by job). Deployed byscripts/deploy.sh:live is on 82f35a8bd94b: pid 247114 logged starting (child_containment required), then consuming; the smoke run passed.Live:
REVISIONreads82f35a8bd94b;4.1.1, opentelemetry-api1.45.2and co-core0.38.0;startingloggedprocessor_version0.38.0+4;ack: completeat 2026-11-16T19:36:03Z;NRestarts=0, and no warnings since the restart. No mapping of a deleted glibc remains, so Tuesday's window has no restart left forprocessor.Advisories:
main's push (43940376617) readsFound no known vulnerabilities … in 63 packages;Notices: Watcher on watcher#394, posted before the deploy. co-core's changelog gap on cannobserv#667.
soupsieve 2.10 → 4.1.1, by
[tool.uv] override-dependencies = ["soupsieve>=4.1,<5"]past co-core 0.38.0's<3(6a29956). opentelemetry-api 1.45.1 → 1.45.2, by its constraint floor. The lock moved those two alone.LOCAL_GENERATION3 → 4.EXTRACTION_STACKsoupsieve4.1.1.tests/test_lock_floors.pyholds:cebed8d,2f24925).Docs: the spec's Amended entry, §5 and §7. DEPENDENCIES.md gets the override route, its removal at the co-core bump, and the Since line (
fdb7f5e,82f35a8).Evidence:
required;:has(), and the reworded"a >"reason.Review: CR 1–4 fixed. CR 5 (no separate red commit) declined:
mainholds every commit green.Follow-up:
:lang(),:-soup-contains()), not yet on PyPI. Taking it is a stack move, and so a generation bump; or it rides the co-core bump.:has(), which Archiver's catch-up (archiver#454) inherits too. Noted on cannobserv#667; no change asked.Known limits:
:has()with a combinator inside it or nested. A spec using one now gives a terminalprocessing_failedinstead of a fact. The notice on watcher#394 asks Watcher to check.:has()refusal: a new golden is generated only at a co-core bump.82f35a8, on the PR and onmain.For #134:
processor_versionis0.38.0+4from this deploy. Consumer loop logs no recovery after a failure run (backoff resets silently) #134 changes logging only, so no generation bump and no Watcher notice; its deploy is a restart.uv lock --upgrade. A re-lock keepsoverride-dependencies, andtests/test_lock_floors.pyfails if it's lost.processing_failedafter this deploy, from strict-:has()specs, is a deterministic ack, not a consumer-loop error. It won't show in Consumer loop logs no recovery after a failure run (backoff resets silently) #134's recovery records.