Skip to content

Lock advisories: soupsieve 4.1 (extraction child, override past co-core 0.38.0's cap; LOCAL_GENERATION +4) and opentelemetry-api 1.45.2 #138

Closed

Description

@gregoryfoster

Epic #84, Tier 1 (live exposure). Filed by the coordinator on 2026-11-15, with the operator's decision to slot it. This body is the brief. Labels: [measured] read from the live system or a registry; [traced] followed through code; [stated] an issue, a changelog or a person says so.

What's open

Dependabot alerts on uv.lock [measured]:

Alert Advisory Severity Package, locked Fixed in
12 GHSA-mmhr-3c5m-f64f, polynomial-time DoS in the descendant combinator high soupsieve 2.10 4.0.0
14 GHSA-6655-6qjp-jx7x, exponential ReDoS in selector compilation high soupsieve 2.10 3.0.2
13 GHSA-7g44-g9rc-x53w, uncontrolled recursion in the CSS parser medium soupsieve 2.10 3.0.2
15 GHSA-mf5v-w3w3-f6cg, quadratic-time traceparent parsing medium opentelemetry-api 1.45.1 1.45.2
  • The audit: its runs on PRs chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 and chore: update skills submodules #136 list the three soupsieve advisories. main's last push run (34d0af0, 11-11) predates them. The scheduled run on Mon 2026-11-16 at 14:37Z will list them too.
  • Dependabot can't fix soupsieve: its security jobs (run 43371303257 and two more on 11-13) fail with "co-core[extract]==0.38.0 depends on soupsieve>=2.0,<3". co-core 0.38.0's extract extra declares soupsieve<3,>=2.0 [measured: the installed METADATA].
  • Exposure [traced]: co-core's HtmlExtractor (co_core/pure/extract/html.py) runs soup.select(sel) with Watcher's spec selectors over untrusted documents, inside the extraction child. The child's timeout and RLIMIT_AS bound the damage to one command's terminal failure, not the service. opentelemetry-api comes in through google-api-core, in the parent, and isn't part of the extraction stack.

The fix upstream, and the path here

  • co-core v0.46.1 (CannObserv/cannobserv#667, 2026-11-13) requires soupsieve>=4.1,<5. Its changelog [stated] reports HtmlExtractor's output unchanged on 2.10, 4.0 and 4.1, and says HtmlExtractor uses only soupsieve.compile and SelectorSyntaxError, which 4.1 keeps.
  • html.py is byte-identical in v0.38.0, v0.46.1 and our venv [measured: diffed 2026-11-15]. So co-core's compatibility check applies to our pin unchanged.
  • [Operator decision: how Processor takes soupsieve ≥4.1. Recommended: [tool.uv] override-dependencies = ["soupsieve>=4.1,<5"] past co-core 0.38.0's cap, with LOCAL_GENERATION 3 → 4. The alternative is a co-core bump 0.38.0 → 0.46.1+: 8 minors, breaking ones among them (none in extraction), plus regenerated goldens. That belongs in its own issue.] Starting this thread approves the recommendation.
    • A constraint can't do it: constraint-dependencies only narrows, and co-core's <3 stays. It has to be an override.
    • Remove the override at the next co-core bump to ≥0.46.1. Record that in docs/DEPENDENCIES.md § Bumping co-core, so the bump procedure removes it.
  • opentelemetry-api: an ordinary floor. Raise constraint-dependencies to opentelemetry-api>=1.45.2. Per DEPENDENCIES.md, "A security PR lands by hand, never as proposed", so close Dependabot's PR chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 in favour of this one.

Steps (docs/DEPENDENCIES.md § The lock audit, § Extraction dependencies; template #112 / PR #114)

  1. pyproject.toml: the override (with a comment naming the advisories, co-core's cap and the removal trigger), and the opentelemetry-api floor raised to 1.45.2.
  2. bash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api, never --upgrade. The diff should move those two alone. Check every co-core source is still the index URL.
  3. tests/test_lock_floors.py: rows for both. Weigh a guard that fails once co-core's own requirement makes the override redundant, so a later bump remembers to remove it.
  4. tests/test_pin.py: EXTRACTION_STACK soupsieve 4.1 (or whatever locks), plus LOCAL_GENERATION and GENERATION 3 → 4 (src/processor/processors/extract.py:26). processor_version becomes 0.38.0+4.
  5. The spec: an Amended entry, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112 made for +3. docs/DEPENDENCIES.md: the "Since #…" line.

Hazards to weigh

  • The parity goldens must pass unchanged. This is the equivalence check between soupsieve 2.10 and 4.1 on real inputs. Never regenerate them here: they're regenerated only at a co-core bump. A failing parity test means output diverged, so stop and report.
  • Containment. Run the whole-corpus test in tests/test_child.py under required on co-processor. Its allowlist comes from sys.path, so a pure-Python upgrade should pass, but the test is the proof.
  • One refusal message changes [stated, co-core changelog]. For a selector ending in a combinator ("a >"), the UnsupportedExtractionError reason reads Invalid character ' ' position 1, not Expected a selector at position 3. No Processor test pins that text [traced: grep]. A processing_failed reason for such a spec reads differently. That is expected.
  • The Watcher notice, before the deploy: the new processor_version (0.38.0+4) and the evidence, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112 did on [Epic] Burn-down: backlog outside #368 → bug fixes, sibling waits, operational stability (2026-10) watcher#394. That's a cross-repo post: draft it and stop. The operator approves that post.
  • The deploy is needed (pyproject.toml, uv.lock, src/). It also restarts processor, which picks up the glibc that unattended-upgrades installed on 2026-11-13: the running PID still maps the old libc.so.6. That settles the restart decision raised in the idle reports. It's an expected side effect, not a separate step.
  • After the merge: Dependabot alerts 12–15 should close on their own once main's lock moves, and the audit on main should read 0. If an alert stays open, report it; don't dismiss it.

Done when

  • soupsieve ≥4.1 and opentelemetry-api ≥1.45.2 locked on main, with the floors and override tested. PR chore: bump opentelemetry-api from 1.45.1 to 1.45.2 #135 closed in favour of this one.
  • LOCAL_GENERATION 4. The parity, containment and full suites green. CI green by job on the final SHA.
  • Watcher's notice posted, with approval, before the deploy.
  • Deployed by scripts/deploy.sh. The live release venv shows the new soupsieve and opentelemetry-api, and the logs show processor_version 0.38.0+4.
  • The audit on main reads 0, and Dependabot alerts 12–15 are closed.

Out of scope

Activity

  1. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Pre-flight: #138 (coordinator, epic #84, 2026-11-16)

    Re-checked against origin/main 263f3ae, which hasn't moved since the body was filed. The body still holds. This comment adds what a probe measured, one correction and two hazards. The probe ran in a throwaway copy of origin/main in the coordinator's scratch space; nothing was written to the repo.

    Probe: the body's recipe, run [measured, 2026-11-16]

    • I added override-dependencies = ["soupsieve>=4.1,<5"] and changed the floor to "opentelemetry-api>=1.45.2", then ran bash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api. It resolved 64 packages and moved exactly two:
      • soupsieve 2.10 → 4.1.1;
      • opentelemetry-api 1.45.1 → 1.45.2.
    • In the new lock:
      • all three co-core sources are still the co-pypi index URL;
      • the lock's [manifest] records overrides = [{name = "soupsieve", specifier = ">=4.1,<5"}].
    • After uv sync --locked: co-core 0.38.0 with soupsieve 4.1.1. The pytest header read child containment: required (Landlock ABI 6). Results:
    Tests Result
    tests/processors/test_extract.py (37 parity goldens, 12 of them refusals) and tests/test_gen_parity_goldens.py pass, goldens untouched
    tests/test_child.py, including the whole-corpus run under containment pass, 40 passed, 7 skipped (credential files this user can't read, and no docker socket: environmental)
    tests/test_pin.py 2 fail as designed: the lock and the venv both show soupsieve 2.10 → 4.1.1. Fixed by step 4

    So soupsieve 4.1.1 changes no golden on the parity corpus. That is the equivalence the body asks for, measured ahead of the build. Rerun it in the build; don't rely on this probe.

    Correction

    • The locked version is 4.1.1, not "4.1". The body's step 4 says "4.1 (or whatever locks)". So EXTRACTION_STACK reads "soupsieve": "4.1.1". Keep the override's floor at >=4.1, which is co-core v0.46.1's.

    Hazards the body didn't name [traced]

    1. The lock audit's routing says "a constraint-dependencies floor", and a test pins its shape.
      • scripts/audit_lock.sh:39-65 prints "Where a finding goes" plus a ::warning saying "Re-lock with a constraint-dependencies floor". tests/test_lock_audit.py:175-190 asserts that routing text.
      • Note: every, stack = [lines starting "- "] unpacks exactly two bullet lines. A third bullet for "a package capped by co-core: an override-dependencies entry until the co-core bump" breaks that unpacking. Change the test with it.
      • Weigh it: either add the override route to the audit text, docs/DEPENDENCIES.md § The lock audit (:125) and the test together, or leave the audit generic and document the override only in § Extraction dependencies and § Bumping co-core.
      • Lean: document it in DEPENDENCIES.md, and leave the audit's two lines alone. A capped package is rare, and the audit's job is to point at the section.
    2. tests/test_lock_floors.py checks constraint-dependencies. An override has no floor row there by default. Body step 3's "rows for both" means a check that reads override-dependencies (or the lock's [manifest] overrides) for soupsieve. The weighed guard, which fails once co-core's own requirement makes the override redundant, would sit beside it.

    The rest of the body stands:

  2. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Corrections before the build (#138)

    I checked the body and the pre-flight (6923328878) against origin/main 263f3ae, co-core's tags and soupsieve's changelog. Worktree 138-lock-advisories. The operator decision stands (override; LOCAL_GENERATION 3 → 4), and so do the steps. The pre-flight's probe reproduces: the lock moves soupsieve 2.10 → 4.1.1 and opentelemetry-api 1.45.1 → 1.45.2, nothing else, and all three co-core sources stay the index URL. Corrections and additions:

    1. soupsieve 3.0 has a breaking change the brief misses: strict :has() [measured]

    • co-core's changelog checks 4.0's two breaks and nothing else. soupsieve 3.0 has two more (upstream changelog). The one that matters here: :has() is CSS-strict by default. A complex selector inside it (div:has(p a)) and a nested :has() (div:has(div:has(p))) now fail to compile.
    • Through co-core, such a spec goes from extracting (on 2.10) to UnsupportedExtractionError, and so to a terminal processing_failed. That is a change in result, not just in a refusal's wording. So the hazard "one refusal message changes" understates it. Measured on 4.1.1:
      • div:has(p a) → Using strict rules as proposed by CSS, the complex selectors at postition 9 is not allowed;
      • div:has(div:has(p)) → :has() was found to be nested under :has() at position 16;
      • div:has(> p) still compiles: a relative selector inside :has() is still allowed.
    • So Watcher's selectors aren't moot (Out of scope, bullet 3). The real corpus's two selectors (div.content, div.view-display-id-bill_action_notice_block) are unaffected, and the parity cases use #side alone. Watcher's live specs are unmeasured from here. The notice asks Watcher to check for a :has( with a combinator inside it, or a nested one.
    • 3.0's other break (custom selectors must come in order) can't reach us: co-core passes no custom.

    2. 4.0's <template> break: co-core's claim holds [measured]

    • On 4.1.1, soup.select("div.c") no longer matches inside a <template> (it did on 2.10).
    • But HtmlExtractor takes no text from a template's contents on either version. A probe with selectors, exclude_selectors and no selectors at all gave identical text on 2.10 and 4.1.1.
    • So the break moves no output here. The <iframe> break is moot under lxml, which escapes an iframe's contents (also probed).

    3. The procedure's wider-corpus step is missing from the steps [measured]

    • DEPENDENCIES.md § Extraction dependencies step 3 asks for the moved package run over a wider corpus on both versions, as pypdf 6.19.0 in uv.lock carries 11 advisories (loops, runtimes, memory); 6.21.2 is in-cap #71 and Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112 did. Done for soupsieve:
      • Inputs: HtmlExtractor over the parity HTML, the real corpus, and every HTML file over 3 KB on the host;
      • Configs: 34 per file: selectors, exclude and ignore, combinators, :has, :lang, attributes, nth-*;
      • Runs: two venvs that differ in soupsieve alone.
    • Result: 289 files × 34 configs = 9,826 runs per version. 0 fingerprints moved. 867 runs differ, every one a refusal:
      • div:has(p a) and div:has(div:has(p)): 289 files each go from a result on 2.10 to UnsupportedExtractionError on 4.1.1 (288 extracted, 1 timed out; see the next bullet). This is correction 1, measured.
      • "a >": refused on both versions, with the reworded reason co-core names (289).
    • Timing, with a 30 s limit per run:
      • 2.10: 4 timeouts, all on one 10 MB page (nodejs/api/all.html), 2 of them the nested and complex :has();
      • 4.1.1: 2 timeouts, the same page and the same 2 configs (:lang(en), div with ignore_selectors p:first-child). That's size, not a regression;
      • 4.1.1 refuses the 2 :has() selectors at compile time;
      • total time: 1,154 s on 2.10, 1,012 s on 4.1.1.
    • The PR body carries the counts, and the spec's Amended entry the summary.

    4. Audit state [measured]

    • "main's last push run (34d0af0, 11-11) predates them" is stale. main's push run on 263f3ae (43757382623, 2026-11-15T20:19Z) already lists all four advisories.
    • Monday's scheduled run hadn't started by 17:36Z today. GitHub delays schedules: 11-09's 14:37Z run started at 23:32Z.
    • The audit on main reading 0 is checked on this PR's merge push.

    5. Scope the brief leaves implicit

    • The spec: besides the Amended entry, §5's LOCAL_GENERATION line and §7's pin-test line ("0.38.0+3", :716) move, as Lock advisories: pypdf 6.21.4 (extraction child, LOCAL_GENERATION +3) and pyasn1 0.6.5 #112's did.
    • The pyproject comment and test_lock_floors.py's docstring name opentelemetry-api 1.45.1. Both move to 1.45.2 and gain the override.
    • The override's route in docs: I'm taking the pre-flight's lean. scripts/audit_lock.sh's two routing lines, and the test that unpacks them, stay unchanged. DEPENDENCIES.md carries the override route instead:
      • § Extraction dependencies gets a step for a package co-core caps;
      • § The lock audit's "Where a finding goes" and § CI pins' "lands by hand" each get one clause pointing there;
      • § Bumping co-core gets the removal.
    • The guard (body step 3): tests/test_lock_floors.py reads the installed co-core's Requires-Dist for soupsieve under extra == "extract". Once co-core's own floor is ≥ 4.1, the guard fails, naming the override to remove. Beside it go a lock and venv floor of 4.1, and a check that the override is declared.

    6. Notes, no change

    • PyPI's latest soupsieve is 4.1.1. Upstream's changelog has an unreleased 4.1.2 with more quadratic-time fixes (:lang(), :-soup-contains()). Once it ships, picking it up is another stack move, and so another generation bump.
    • Plan: docs/plans/2026-11-16-138-soupsieve-override.md on the branch.

    Building now.

  3. added 4 commits that reference this issue on Nov 16, 2026
  4. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Post-merge host steps (#138, PR #139)

    Final SHA 82f35a8. CI is green by job on it (run 43925929773: lint and test, 1026 passed, child containment: required (Landlock ABI 8)). The PR's audit (43925929751) reads Found no known vulnerabilities … in 63 packages. In order, from /home/exedev/processor as exedev:

    1. Merge. FF from the deploy checkout:

      git fetch origin && git merge --ff-only origin/138-lock-advisories && git push origin main
    2. CI on main's push. ci.yml green by job on 82f35a8; scripts/deploy.sh's gate reads that run.

    3. The Watcher notice on [Epic] Burn-down: backlog outside #368 → bug fixes, sibling waits, operational stability (2026-10) watcher#394. The operator approves this post. It goes up before the deploy.

    4. Deploy: scripts/deploy.sh. Read the verdict line: journalctl -t processor-deploy -n 5 -o cat.

    5. The live release:

      • cat /srv/processor/live/REVISION reads 82f35a8bd94b9669aa54e82e282da3a332b8564d;
      • /srv/processor/live/.venv/bin/python -I -c "import importlib.metadata as m; print(m.version('soupsieve'), m.version('opentelemetry-api'))" reads 4.1.1 1.45.2;
      • journalctl -u processor -o cat --since -10min | jq -cR 'fromjson? | select(.message=="starting") | {processor_version, build}' reads processor_version 0.38.0+4.
    6. glibc, as a side effect of the restart: sudo grep -c 'libc.so.6 (deleted)' /proc/$(systemctl show -p MainPID --value processor)/maps reads 0. With that, Tuesday's window (2026-11-17 05:00Z) finds no restart left for processor.

    7. Health:

      • systemctl show -p NRestarts processor reads 0;
      • journalctl -u processor -p warning --since -10min is empty;
      • the next drift check (processor-drift.timer, hourly) and liveness are ok in Status;
      • the next content.derived facts carry 0.38.0+4 (an ack: complete with no processing_failed spike).
    8. Advisories:

      • The audit on main's push run of 82f35a8 reads 0.
      • Dependabot alerts 12–15 close on their own once the default branch's lock moves: gh api 'repos/CannObserv/processor/dependabot/alerts?state=open' --jq length reads 0. If one stays open, report it, don't dismiss it.
  5. gregoryfoster commented on Nov 16, 2026

    @gregoryfoster
    MemberAuthor

    Shipped in #139 (2775d73..82f35a8, FF-merged; main CI 43940376546 green by job). Deployed by scripts/deploy.sh: live is on 82f35a8bd94b: pid 247114 logged starting (child_containment required), then consuming; the smoke run passed.

    • Live:

      • REVISION reads 82f35a8bd94b;
      • the release venv has soupsieve 4.1.1, opentelemetry-api 1.45.2 and co-core 0.38.0;
      • starting logged processor_version 0.38.0+4;
      • the first fact on it was an ack: complete at 2026-11-16T19:36:03Z;
      • NRestarts=0, and no warnings since the restart. No mapping of a deleted glibc remains, so Tuesday's window has no restart left for processor.
    • Advisories:

    • Notices: Watcher on watcher#394, posted before the deploy. co-core's changelog gap on cannobserv#667.

    • soupsieve 2.10 → 4.1.1, by [tool.uv] override-dependencies = ["soupsieve>=4.1,<5"] past co-core 0.38.0's <3 (6a29956). opentelemetry-api 1.45.1 → 1.45.2, by its constraint floor. The lock moved those two alone.

    • LOCAL_GENERATION 3 → 4. EXTRACTION_STACK soupsieve 4.1.1. tests/test_lock_floors.py holds:

      • both floors and the override;
      • a guard that fails once co-core's own requirement admits soupsieve ≥ 4.1, whether its floor reaches it or its cap lifts (cebed8d, 2f24925).
    • Docs: the spec's Amended entry, §5 and §7. DEPENDENCIES.md gets the override route, its removal at the co-core bump, and the Since line (fdb7f5e, 82f35a8).

    • Evidence:

      • the goldens and the real corpus are unchanged;
      • whole-corpus containment passes under required;
      • over 289 HTML files × 34 selector configs, 0 fingerprints moved. The only differences are refusals: strict :has(), and the reworded "a >" reason.
    • Review: CR 1–4 fixed. CR 5 (no separate red commit) declined: main holds every commit green.

    Follow-up:

    • Bump co-core 0.38.0 → ≥ 0.46.1 and drop the soupsieve override (#138) #140: the co-core bump to ≥ 0.46.1, which removes the override. The guard fails at it.
    • soupsieve 4.1.2: upstream's changelog lists more quadratic-time fixes (:lang(), :-soup-contains()), not yet on PyPI. Taking it is a stack move, and so a generation bump; or it rides the co-core bump.
    • co-core's v0.46.1 changelog (cannobserv#667) checks 4.0's breaks only. It misses soupsieve 3.0's strict :has(), which Archiver's catch-up (archiver#454) inherits too. Noted on cannobserv#667; no change asked.

    Known limits:

    • Watcher's live specs are unmeasured for :has() with a combinator inside it or nested. A spec using one now gives a terminal processing_failed instead of a fact. The notice on watcher#394 asks Watcher to check.
    • The wider corpus is the host's HTML docs plus 9 real inputs with 2 selectors, not Watcher's watched pages. 0 moved there, but that can't show no watched page moves; the bump follows the rule.
    • No parity case pins the strict-:has() refusal: a new golden is generated only at a co-core bump.
    • Integration tests weren't run locally; CI ran them on 82f35a8, on the PR and on main.
    • The Status side of the drift and liveness checks wasn't read in this session. The deploy's smoke run passed, and liveness logged its check-in.

    For #134:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions