diff --git a/docs/DEPENDENCIES.md b/docs/DEPENDENCIES.md index 6d3e40a..7ceec20 100644 --- a/docs/DEPENDENCIES.md +++ b/docs/DEPENDENCIES.md @@ -45,7 +45,7 @@ gh api repos///git/ref/tags/ --jq '.object.type + " " + .obj - Dependency graph: **on**. It is the `Graph Update: uv in /.` run on each push, and opens nothing. - Dependabot alerts: **on**. These are the operator's. - Dependabot security updates: **on**, and left on (operator, 2026-11-03, #112). The 2026-10-29 reading said off. A security update ignores `dependabot.yml`: it is a `uv` job (`"command":"security"`) per advisory. Until #121 every one failed before it opened anything (`PathDependenciesNotReachable: --find-links`; the first two on the push of `312552a`, 2026-11-03T01:27Z). Since #121, co-core resolves from the co-pypi index, which the updater should read over OIDC through the org registry `PYTHON_INDEX_OIDC_GCP` once this repository (ID `1396635239`) is admitted (cannobserv `docs/DEPENDABOT.md`, step 3). Proven 2026-11-07, after admission: a temporary `uv` version entry allowing co-core alone, run [42338249411](https://github.com/CannObserv/processor/actions/runs/42338249411), logged `registered gcp OIDC credentials for python index`, PyPI's 404, `authenticating request with OIDC token (host: us-west1-python.pkg.dev)`, `200` and `Latest version is 0.41.0`; its `uv lock` fetched all three co-core wheels the same way. It opened nothing: `co-core-aio==0.38.0 depends on co-core>=0.38,<0.39`, so a proposal must move the three names together (#121). The entry was then removed. - - **A security PR lands by hand,** never as proposed. Add the `[tool.uv] constraint-dependencies` floor and its `tests/test_lock_floors.py` row in the same merge; in the extraction stack, bump `LOCAL_GENERATION` too ([§ The lock audit](#the-lock-audit)'s "Where a finding goes"). Then check the proposed `uv.lock`: every co-core `source` is still the index URL. Dependabot's lookup asks PyPI first, and the co-core names are unclaimed there (DEPENDABOT.md's accepted limit; CannObserv/replicator#169). + - **A security PR lands by hand,** never as proposed. Add the `[tool.uv] constraint-dependencies` floor and its `tests/test_lock_floors.py` row in the same merge (an `override-dependencies` entry where co-core caps the fix: § Extraction dependencies); in the extraction stack, bump `LOCAL_GENERATION` too ([§ The lock audit](#the-lock-audit)'s "Where a finding goes"). Then check the proposed `uv.lock`: every co-core `source` is still the index URL. Dependabot's lookup asks PyPI first, and the co-core names are unclaimed there (DEPENDABOT.md's accepted limit; CannObserv/replicator#169). No test can see these settings, and our token gets 403 on both APIs. Re-read them in the UI whenever the repo's security settings change. @@ -122,7 +122,7 @@ The script and its workflow: - **Outside the deploy gate.** The gate reads `ci.yml`'s push run alone (DEPLOYMENT.md § The CI gate); `tests/test_lock_audit.py` holds both workflows to that. - **No secret, no WIF.** `uv audit --frozen` reads the lock without fetching a package (measured with no co-pypi token, #121). OSV's text is echoed between `::stop-commands::` and a random token, so it can't issue workflow commands. - **By hand:** `bash scripts/audit_lock.sh` from any directory audits this checkout's lock. -- **Where a finding goes:** every finding is `uv lock --upgrade-package ` plus a `[tool.uv] constraint-dependencies` floor with its check in `tests/test_lock_floors.py`, so a later re-lock can't undo it. A package in the extraction stack also bumps `LOCAL_GENERATION`, with the notice: § Extraction dependencies (#71 did both for pypdf). Re-locks touch `uv.lock`, so deploy after the merge. +- **Where a finding goes:** every finding is `uv lock --upgrade-package ` plus a `[tool.uv] constraint-dependencies` floor with its check in `tests/test_lock_floors.py`, so a later re-lock can't undo it. Where co-core's own requirement caps the fix, the floor is an override instead (§ Extraction dependencies, #138). A package in the extraction stack also bumps `LOCAL_GENERATION`, with the notice: § Extraction dependencies (#71 did both for pypdf). Re-locks touch `uv.lock`, so deploy after the merge. What the window doesn't cover: - A disclosure mid-week waits for the next window, at most 7 days. Between windows, `audit.yml` alone sees it. @@ -139,6 +139,7 @@ co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-1 1. Update the pin in `pyproject.toml` (all three names) and `EXPECTED` in `tests/test_pin.py`, then `bash scripts/co-pypi.sh uv lock --upgrade-package co-core --upgrade-package co-core-aio --upgrade-package co-core-sync`, never `--upgrade`. Nothing else moves: no wheelhouse, no digest table (#121). - **The lock carries the digests.** Each co-core file gets its sha256 from the index, and `uv sync --locked` refuses a miss. cannobserv's publish run fails unless the index lists every file under its local sha256 (from v0.41.0, CannObserv/cannobserv#554). - **Check the diff:** each co-core `source` is the index URL, and nothing outside the bump moved. A moved extraction-stack package belongs to the bump (§ Extraction dependencies). + - **Drop an override the new co-core makes redundant.** `tests/test_lock_floors.py` fails when co-core's own requirement admits an `override-dependencies` floor: remove the entry and its `OVERRIDDEN` row, and keep the floor as a constraint if co-core's own is lower. At ≥ 0.46.1 that is soupsieve's (`>=4.1,<5` in co-core itself; #138). 2. A release that adds an essence or an extraction rule gets its cases in `tests/fixtures/parity/cases.json` first, so their goldens come from co-core. Then regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved, and every case that started or stopped being refused. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. A catch-up across releases reads each CHANGELOG's Downstream lines for the extract, bus and blob-store surfaces, and diffs those packages' code between the tags (#29 did 0.19.7 → 0.32.0). 3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved. 4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models and `canonical_text` semantics. A change to essence resolution is notice, not coordination (spec §5, #57). @@ -148,12 +149,12 @@ co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-1 `processor_version` sees co-core and `LOCAL_GENERATION`, nothing else in the lock. The extraction stack (the transitive closure of co-core's `extract` extra: pypdf, lxml, bs4, html5lib, openpyxl, …) turns bytes into text too, and co-core floors it loosely, so a re-lock can move a fingerprint under an unchanged version. Watcher reads that as a content change and notifies. So moving the stack outside a co-core bump is a generation bump (spec §5, amended 2026-10-24, #71): -1. `uv lock --upgrade-package `, never `uv lock --upgrade`; the diff moves what you named. A security floor goes in `[tool.uv] constraint-dependencies`, with its check in `tests/test_lock_floors.py`. +1. `uv lock --upgrade-package `, never `uv lock --upgrade`; the diff moves what you named. A security floor goes in `[tool.uv] constraint-dependencies`, with its check in `tests/test_lock_floors.py`. If co-core's `extract` extra caps the package below the fix (soupsieve `<3` in 0.38.0), a constraint can't lift it: the floor goes in `override-dependencies`, which replaces co-core's requirement, with an `OVERRIDDEN` row in the same test. Its guard fails once co-core's own requirement admits the floor, so the co-core bump removes the override (§ Bumping co-core). The alternative is the bump itself, its own issue. 2. `tests/test_pin.py` fails, naming what moved. Update `EXTRACTION_STACK`, and bump `LOCAL_GENERATION` in `src/processor/processors/extract.py` and `GENERATION` in the test. At a co-core bump the version moves anyway: update the table, leave the generation. 3. Measure what moved: the parity tests (goldens, real corpus) must pass unchanged (a golden that moves is a co-core-pin question, not this one), and run co-core's extractor for the moved package over a wider corpus on both versions (#71 used pypdf's and py-pdf's sample PDFs: 4 of 106 moved). Put the result in the PR. 4. **Notice to Watcher** before deploy: the new `processor_version` and the evidence. Merge, then `scripts/deploy.sh`. -Since #71: `0.38.0+2`, pypdf 6.21.2, opentelemetry-api 1.45.1 (not stack; a floor only). Since #112: `0.38.0+3`, pypdf 6.21.4, pyasn1 0.6.5 (not stack; a floor only). +Since #71: `0.38.0+2`, pypdf 6.21.2, opentelemetry-api 1.45.1 (not stack; a floor only). Since #112: `0.38.0+3`, pypdf 6.21.4, pyasn1 0.6.5 (not stack; a floor only). Since #138: `0.38.0+4`, soupsieve 4.1.1 by override past co-core's `<3` (no fingerprint moved over 289 HTML files × 34 selector configs; a complex or nested `:has()` now refuses), opentelemetry-api 1.45.2 (not stack; a floor only). ## Parity goldens diff --git a/docs/plans/2026-11-16-138-soupsieve-override.md b/docs/plans/2026-11-16-138-soupsieve-override.md new file mode 100644 index 0000000..1eb5bfd --- /dev/null +++ b/docs/plans/2026-11-16-138-soupsieve-override.md @@ -0,0 +1,47 @@ +--- +title: soupsieve 4.1.1 by override and opentelemetry-api 1.45.2; LOCAL_GENERATION 4 (#138) +date: 2026-11-16 +status: per #138 (filed by the burn-down coordinator, #84; pre-flight comment 6923328878; corrections on the issue); procedure per #71 (docs/DEPENDENCIES.md § Extraction dependencies) +--- + +# soupsieve 4.1.1, opentelemetry-api 1.45.2 + +## Problem + +Four open Dependabot alerts on `uv.lock`, and `main`'s audit lists them all (run 43757382623, `263f3ae`): +- **soupsieve 2.10** has 3: GHSA-mmhr-3c5m-f64f (polynomial-time descendant combinator, fixed in 4.0), GHSA-6655-6qjp-jx7x (ReDoS in selector compilation, 3.0.2) and GHSA-7g44-g9rc-x53w (parser recursion, 3.0.2). co-core's `HtmlExtractor` compiles and runs Watcher's spec selectors over untrusted documents in the extraction child. The child's timeout and `RLIMIT_AS` bound the damage to one command. +- **opentelemetry-api 1.45.1** has 1: GHSA-mf5v-w3w3-f6cg (quadratic `traceparent` parsing). It runs in the parent, through google-api-core. + +co-core 0.38.0's `extract` extra declares `soupsieve>=2.0,<3`, so neither a constraint nor Dependabot can reach a fix. co-core v0.46.1 raised it to `>=4.1,<5` (cannobserv#667). `html.py` is byte-identical in v0.38.0, v0.46.1 and the venv. + +## Approach + +Operator decision (#138): override past co-core's cap. The co-core bump is out of scope and gets its own issue. + +1. `[tool.uv] override-dependencies = ["soupsieve>=4.1,<5"]`. A constraint only narrows, and co-core's `<3` would stay. The `opentelemetry-api` floor in `constraint-dependencies` goes to 1.45.2. +2. `bash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api`, never `--upgrade`. Expected diff: soupsieve 2.10 → 4.1.1, opentelemetry-api 1.45.1 → 1.45.2, the manifest's `overrides` entry and the floor's specifier. Every co-core `source` stays the index URL. +3. `tests/test_lock_floors.py`: + - the opentelemetry-api floor moves to 1.45.2; + - soupsieve gets a lock and venv floor of 4.1, and a check that the override is in `override-dependencies`; + - a guard reads the installed co-core's `extract` requirement on soupsieve. Once that requirement alone floors at 4.1 or higher, the override is redundant, so the guard fails until the co-core bump removes it. +4. `tests/test_pin.py`: `EXTRACTION_STACK` soupsieve `4.1.1`, `GENERATION` 3 → 4. `LOCAL_GENERATION` 3 → 4, so `processor_version` = `0.38.0+4`. +5. Docs: + - The spec: an **Amended** entry, §5's generation line, §7's pin-test line. + - DEPENDENCIES.md: + - § Extraction dependencies: the "Since" line, plus a step for a package co-core caps (it takes an override, not a floor); + - § Bumping co-core: drop the override at the bump to co-core ≥ 0.46.1; + - § The lock audit's "Where a finding goes" and § CI pins' "lands by hand": one clause naming the override route. + - The audit script's two routing lines stay as they are (pre-flight hazard 1): the section they link carries the override route. + +## Evidence + +- The parity goldens and the real corpus pass unchanged, and `tests/test_child.py` passes under `required` containment. +- Wider corpus, as #71 and #112 did: co-core 0.38.0's `HtmlExtractor`, then `canonical_text_fingerprint`. Inputs: the parity HTML, the real corpus, and every HTML file over 3 KB on the host, each under a battery of 34 selector configs. It runs in two venvs that differ in soupsieve alone (2.10 and 4.1.1). The PR body has the counts. +- What soupsieve's changelog breaks between 2.10 and 4.1.1, and how each break lands here: + - 3.0's strict `:has()`: a complex or nested `:has()` now refuses; + - 4.0's `