Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 5 additions & 4 deletions docs/DEPENDENCIES.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,7 +45,7 @@ gh api repos/<owner>/<action>/git/ref/tags/<tag> --jq '.object.type + " " + .obj
- Dependency graph: **on**. It is the `Graph Update: uv in /.` run on each push, and opens nothing.
- Dependabot alerts: **on**. These are the operator's.
- Dependabot security updates: **on**, and left on (operator, 2026-11-03, #112). The 2026-10-29 reading said off. A security update ignores `dependabot.yml`: it is a `uv` job (`"command":"security"`) per advisory. Until #121 every one failed before it opened anything (`PathDependenciesNotReachable: --find-links`; the first two on the push of `312552a`, 2026-11-03T01:27Z). Since #121, co-core resolves from the co-pypi index, which the updater should read over OIDC through the org registry `PYTHON_INDEX_OIDC_GCP` once this repository (ID `1396635239`) is admitted (cannobserv `docs/DEPENDABOT.md`, step 3). Proven 2026-11-07, after admission: a temporary `uv` version entry allowing co-core alone, run [42338249411](https://github.com/CannObserv/processor/actions/runs/42338249411), logged `registered gcp OIDC credentials for python index`, PyPI's 404, `authenticating request with OIDC token (host: us-west1-python.pkg.dev)`, `200` and `Latest version is 0.41.0`; its `uv lock` fetched all three co-core wheels the same way. It opened nothing: `co-core-aio==0.38.0 depends on co-core>=0.38,<0.39`, so a proposal must move the three names together (#121). The entry was then removed.
- **A security PR lands by hand,** never as proposed. Add the `[tool.uv] constraint-dependencies` floor and its `tests/test_lock_floors.py` row in the same merge; in the extraction stack, bump `LOCAL_GENERATION` too ([§ The lock audit](#the-lock-audit)'s "Where a finding goes"). Then check the proposed `uv.lock`: every co-core `source` is still the index URL. Dependabot's lookup asks PyPI first, and the co-core names are unclaimed there (DEPENDABOT.md's accepted limit; CannObserv/replicator#169).
- **A security PR lands by hand,** never as proposed. Add the `[tool.uv] constraint-dependencies` floor and its `tests/test_lock_floors.py` row in the same merge (an `override-dependencies` entry where co-core caps the fix: § Extraction dependencies); in the extraction stack, bump `LOCAL_GENERATION` too ([§ The lock audit](#the-lock-audit)'s "Where a finding goes"). Then check the proposed `uv.lock`: every co-core `source` is still the index URL. Dependabot's lookup asks PyPI first, and the co-core names are unclaimed there (DEPENDABOT.md's accepted limit; CannObserv/replicator#169).

No test can see these settings, and our token gets 403 on both APIs. Re-read them in the UI whenever the repo's security settings change.

Expand Down Expand Up @@ -122,7 +122,7 @@ The script and its workflow:
- **Outside the deploy gate.** The gate reads `ci.yml`'s push run alone (DEPLOYMENT.md § The CI gate); `tests/test_lock_audit.py` holds both workflows to that.
- **No secret, no WIF.** `uv audit --frozen` reads the lock without fetching a package (measured with no co-pypi token, #121). OSV's text is echoed between `::stop-commands::` and a random token, so it can't issue workflow commands.
- **By hand:** `bash scripts/audit_lock.sh` from any directory audits this checkout's lock.
- **Where a finding goes:** every finding is `uv lock --upgrade-package <name>` plus a `[tool.uv] constraint-dependencies` floor with its check in `tests/test_lock_floors.py`, so a later re-lock can't undo it. A package in the extraction stack also bumps `LOCAL_GENERATION`, with the notice: § Extraction dependencies (#71 did both for pypdf). Re-locks touch `uv.lock`, so deploy after the merge.
- **Where a finding goes:** every finding is `uv lock --upgrade-package <name>` plus a `[tool.uv] constraint-dependencies` floor with its check in `tests/test_lock_floors.py`, so a later re-lock can't undo it. Where co-core's own requirement caps the fix, the floor is an override instead (§ Extraction dependencies, #138). A package in the extraction stack also bumps `LOCAL_GENERATION`, with the notice: § Extraction dependencies (#71 did both for pypdf). Re-locks touch `uv.lock`, so deploy after the merge.

What the window doesn't cover:
- A disclosure mid-week waits for the next window, at most 7 days. Between windows, `audit.yml` alone sees it.
Expand All @@ -139,6 +139,7 @@ co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-1
1. Update the pin in `pyproject.toml` (all three names) and `EXPECTED` in `tests/test_pin.py`, then `bash scripts/co-pypi.sh uv lock --upgrade-package co-core --upgrade-package co-core-aio --upgrade-package co-core-sync`, never `--upgrade`. Nothing else moves: no wheelhouse, no digest table (#121).
- **The lock carries the digests.** Each co-core file gets its sha256 from the index, and `uv sync --locked` refuses a miss. cannobserv's publish run fails unless the index lists every file under its local sha256 (from v0.41.0, CannObserv/cannobserv#554).
- **Check the diff:** each co-core `source` is the index URL, and nothing outside the bump moved. A moved extraction-stack package belongs to the bump (§ Extraction dependencies).
- **Drop an override the new co-core makes redundant.** `tests/test_lock_floors.py` fails when co-core's own requirement admits an `override-dependencies` floor: remove the entry and its `OVERRIDDEN` row, and keep the floor as a constraint if co-core's own is lower. At ≥ 0.46.1 that is soupsieve's (`>=4.1,<5` in co-core itself; #138).
2. A release that adds an essence or an extraction rule gets its cases in `tests/fixtures/parity/cases.json` first, so their goldens come from co-core. Then regenerate the goldens from co-core directly: `uv run python scripts/gen_parity_goldens.py tests/fixtures/parity`. It prints every digest that moved, and every case that started or stopped being refused. Paste that list into the bump note: each line is a deliberate re-baseline. Never regenerate to make a parity test pass outside a bump. A catch-up across releases reads each CHANGELOG's Downstream lines for the extract, bus and blob-store surfaces, and diffs those packages' code between the tags (#29 did 0.19.7 → 0.32.0).
3. The real corpus (`tests/fixtures/parity/real/`) is never regenerated. It passes unchanged, or the bump note says which recorded fingerprints moved.
4. **Notice to Watcher** before deploy: the new `processor_version`, and the moved digests. A change to the contract needs coordination with Watcher first, not just notice. The contract is the `content.process` / `content.derived` models and `canonical_text` semantics. A change to essence resolution is notice, not coordination (spec §5, #57).
Expand All @@ -148,12 +149,12 @@ co-core is pinned `==`, and Processor owns the pin (spec D5, §5; amended 2026-1

`processor_version` sees co-core and `LOCAL_GENERATION`, nothing else in the lock. The extraction stack (the transitive closure of co-core's `extract` extra: pypdf, lxml, bs4, html5lib, openpyxl, …) turns bytes into text too, and co-core floors it loosely, so a re-lock can move a fingerprint under an unchanged version. Watcher reads that as a content change and notifies. So moving the stack outside a co-core bump is a generation bump (spec §5, amended 2026-10-24, #71):

1. `uv lock --upgrade-package <pkg>`, never `uv lock --upgrade`; the diff moves what you named. A security floor goes in `[tool.uv] constraint-dependencies`, with its check in `tests/test_lock_floors.py`.
1. `uv lock --upgrade-package <pkg>`, never `uv lock --upgrade`; the diff moves what you named. A security floor goes in `[tool.uv] constraint-dependencies`, with its check in `tests/test_lock_floors.py`. If co-core's `extract` extra caps the package below the fix (soupsieve `<3` in 0.38.0), a constraint can't lift it: the floor goes in `override-dependencies`, which replaces co-core's requirement, with an `OVERRIDDEN` row in the same test. Its guard fails once co-core's own requirement admits the floor, so the co-core bump removes the override (§ Bumping co-core). The alternative is the bump itself, its own issue.
2. `tests/test_pin.py` fails, naming what moved. Update `EXTRACTION_STACK`, and bump `LOCAL_GENERATION` in `src/processor/processors/extract.py` and `GENERATION` in the test. At a co-core bump the version moves anyway: update the table, leave the generation.
3. Measure what moved: the parity tests (goldens, real corpus) must pass unchanged (a golden that moves is a co-core-pin question, not this one), and run co-core's extractor for the moved package over a wider corpus on both versions (#71 used pypdf's and py-pdf's sample PDFs: 4 of 106 moved). Put the result in the PR.
4. **Notice to Watcher** before deploy: the new `processor_version` and the evidence. Merge, then `scripts/deploy.sh`.

Since #71: `0.38.0+2`, pypdf 6.21.2, opentelemetry-api 1.45.1 (not stack; a floor only). Since #112: `0.38.0+3`, pypdf 6.21.4, pyasn1 0.6.5 (not stack; a floor only).
Since #71: `0.38.0+2`, pypdf 6.21.2, opentelemetry-api 1.45.1 (not stack; a floor only). Since #112: `0.38.0+3`, pypdf 6.21.4, pyasn1 0.6.5 (not stack; a floor only). Since #138: `0.38.0+4`, soupsieve 4.1.1 by override past co-core's `<3` (no fingerprint moved over 289 HTML files × 34 selector configs; a complex or nested `:has()` now refuses), opentelemetry-api 1.45.2 (not stack; a floor only).

## Parity goldens

Expand Down
47 changes: 47 additions & 0 deletions docs/plans/2026-11-16-138-soupsieve-override.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,47 @@
---
title: soupsieve 4.1.1 by override and opentelemetry-api 1.45.2; LOCAL_GENERATION 4 (#138)
date: 2026-11-16
status: per #138 (filed by the burn-down coordinator, #84; pre-flight comment 6923328878; corrections on the issue); procedure per #71 (docs/DEPENDENCIES.md § Extraction dependencies)
---

# soupsieve 4.1.1, opentelemetry-api 1.45.2

## Problem

Four open Dependabot alerts on `uv.lock`, and `main`'s audit lists them all (run 43757382623, `263f3ae`):
- **soupsieve 2.10** has 3: GHSA-mmhr-3c5m-f64f (polynomial-time descendant combinator, fixed in 4.0), GHSA-6655-6qjp-jx7x (ReDoS in selector compilation, 3.0.2) and GHSA-7g44-g9rc-x53w (parser recursion, 3.0.2). co-core's `HtmlExtractor` compiles and runs Watcher's spec selectors over untrusted documents in the extraction child. The child's timeout and `RLIMIT_AS` bound the damage to one command.
- **opentelemetry-api 1.45.1** has 1: GHSA-mf5v-w3w3-f6cg (quadratic `traceparent` parsing). It runs in the parent, through google-api-core.

co-core 0.38.0's `extract` extra declares `soupsieve>=2.0,<3`, so neither a constraint nor Dependabot can reach a fix. co-core v0.46.1 raised it to `>=4.1,<5` (cannobserv#667). `html.py` is byte-identical in v0.38.0, v0.46.1 and the venv.

## Approach

Operator decision (#138): override past co-core's cap. The co-core bump is out of scope and gets its own issue.

1. `[tool.uv] override-dependencies = ["soupsieve>=4.1,<5"]`. A constraint only narrows, and co-core's `<3` would stay. The `opentelemetry-api` floor in `constraint-dependencies` goes to 1.45.2.
2. `bash scripts/co-pypi.sh uv lock --upgrade-package soupsieve --upgrade-package opentelemetry-api`, never `--upgrade`. Expected diff: soupsieve 2.10 → 4.1.1, opentelemetry-api 1.45.1 → 1.45.2, the manifest's `overrides` entry and the floor's specifier. Every co-core `source` stays the index URL.
3. `tests/test_lock_floors.py`:
- the opentelemetry-api floor moves to 1.45.2;
- soupsieve gets a lock and venv floor of 4.1, and a check that the override is in `override-dependencies`;
- a guard reads the installed co-core's `extract` requirement on soupsieve. Once that requirement alone floors at 4.1 or higher, the override is redundant, so the guard fails until the co-core bump removes it.
4. `tests/test_pin.py`: `EXTRACTION_STACK` soupsieve `4.1.1`, `GENERATION` 3 → 4. `LOCAL_GENERATION` 3 → 4, so `processor_version` = `0.38.0+4`.
5. Docs:
- The spec: an **Amended** entry, §5's generation line, §7's pin-test line.
- DEPENDENCIES.md:
- § Extraction dependencies: the "Since" line, plus a step for a package co-core caps (it takes an override, not a floor);
- § Bumping co-core: drop the override at the bump to co-core ≥ 0.46.1;
- § The lock audit's "Where a finding goes" and § CI pins' "lands by hand": one clause naming the override route.
- The audit script's two routing lines stay as they are (pre-flight hazard 1): the section they link carries the override route.

## Evidence

- The parity goldens and the real corpus pass unchanged, and `tests/test_child.py` passes under `required` containment.
- Wider corpus, as #71 and #112 did: co-core 0.38.0's `HtmlExtractor`, then `canonical_text_fingerprint`. Inputs: the parity HTML, the real corpus, and every HTML file over 3 KB on the host, each under a battery of 34 selector configs. It runs in two venvs that differ in soupsieve alone (2.10 and 4.1.1). The PR body has the counts.
- What soupsieve's changelog breaks between 2.10 and 4.1.1, and how each break lands here:
- 3.0's strict `:has()`: a complex or nested `:has()` now refuses;
- 4.0's `<template>`: its contents give no text on either version;
- 4.0's `<iframe>`: lxml escapes its contents.

## Ship

PR (`Refs #138`: live checks remain after the merge) → CI → FF merge → `git pull --ff-only` → notice to Watcher on watcher#394 (approved post) → `scripts/deploy.sh` → the verdict line, `processor_version` `0.38.0+4` in the logs, the new versions in the release venv, drift `ok`, then the audit on `main` reads 0 and alerts 12–15 close. Close PR #135 in favour of this one.
30 changes: 25 additions & 5 deletions docs/specs/2026-09-29-processor-service-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -144,6 +144,21 @@ on `co-processor`.
names no reader; Section 1 and the prefix note name only Archiver and
Provisioner (Decisions, Sections 1 and 2).

**Amended 2026-11-16 (soupsieve past co-core's cap, processor#138):**
- soupsieve 2.10 → 4.1.1 for 3 selector DoS advisories. co-core 0.38.0's
`extract` extra caps it `<3`, so `[tool.uv] override-dependencies` replaces
that requirement until the co-core bump to ≥ 0.46.1 removes it.
`LOCAL_GENERATION` 3 → **4**, and facts report `0.38.0+4` (Sections 5 and 7).
- Results:
- the goldens and the real corpus held;
- over 289 HTML files × 34 selector configs, no fingerprint moved;
- one selector shape now refuses: soupsieve 3.0's strict `:has()` refuses a
complex or nested `:has()` (`div:has(p a)`, `div:has(div:has(p))`). A spec
with one moves from a fact to a terminal `processing_failed`;
- a trailing combinator's refusal reason is reworded; it was refused before too.
- opentelemetry-api 1.45.1 → 1.45.2 (1 advisory) is a floor only: it isn't in the
stack.

---

## Why a new service
Expand Down Expand Up @@ -534,9 +549,14 @@ reclaim re-running (not re-publishing) the extraction.
- A move gives Watcher notice, as a co-core bump does, with the evidence of
what moved. At a co-core bump the stack may move with it; the version moves
anyway, so the generation stays.
- **`LOCAL_GENERATION = 3`** since processor#112: pypdf 6.21.2 → 6.21.4 for
its advisories. Over the same corpora at HEAD, 0 of 110 PDFs extracted
differently; the goldens and the real corpus held.
- **`LOCAL_GENERATION = 4`** since processor#138: soupsieve 2.10 → 4.1.1 for
its advisories, by `override-dependencies` past co-core 0.38.0's `<3`. No
fingerprint moved over 289 HTML files × 34 selector configs, and the goldens
and the real corpus held. A complex or nested `:has()` now refuses
(soupsieve 3.0's strict `:has()`).
- 3 was processor#112: pypdf 6.21.2 → 6.21.4 for its advisories. Over the same
corpora at HEAD, 0 of 110 PDFs extracted differently; the goldens and the
real corpus held.
- 2 was processor#71: pypdf 6.19.0 → 6.21.2 for its advisories. Over pypdf's
and py-pdf's sample corpora, 4 of 106 PDFs extracted differently (word
spacing); the goldens and the real corpus held.
Expand Down Expand Up @@ -713,9 +733,9 @@ TDD, red first.
- Parity with Watcher's local path should be proven before shadow, not
discovered in it.
- **Pin test.** The lock's co-core equals the expected exact version and
`processor_version(LOCAL_GENERATION) == "0.38.0+3"` — a bump is a deliberate,
`processor_version(LOCAL_GENERATION) == "0.38.0+4"` — a bump is a deliberate,
test-failing act. So is moving the extraction stack (Section 5, processor#71,
processor#112).
processor#112, processor#138).
- **Shells — one test per Section 4 row.** Input/output stores: a local co-core
store on a temp dir behind the same interface, plus fakes raising GCS 5xx /
429 / `NotFound`. Bus: fakeredis (or co-core-aio's test double, if it ships
Expand Down
14 changes: 11 additions & 3 deletions pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -118,12 +118,20 @@ co-core-sync = { index = "co-pypi" }
# pypdf (via co-core[extract]) 6.21.2 fixes 11 advisories on 6.19.0 (loops, runtimes,
# memory, JPEG) on attacker-reachable PDFs (#71), and 6.21.4 7 more on 6.21.2 (memory and
# runtime blowups, two infinite loops; #112); opentelemetry-api (via google-api-core)
# 1.45.1 fixes unbounded tracestate parsing (#71); pyasn1 (via pyasn1-modules, google-auth)
# 0.6.5 fixes quadratic-time BER/DER decoding (#112). tests/test_lock_floors.py checks them.
# 1.45.1 fixes unbounded tracestate parsing (#71), 1.45.2 quadratic-time traceparent
# parsing (#138); pyasn1 (via pyasn1-modules, google-auth) 0.6.5 fixes quadratic-time
# BER/DER decoding (#112). tests/test_lock_floors.py checks them.
# pypdf is also extraction stack: moving it bumps LOCAL_GENERATION (tests/test_pin.py).
constraint-dependencies = [
"certifi>=2026.10.19",
"pypdf>=6.21.4",
"opentelemetry-api>=1.45.1",
"opentelemetry-api>=1.45.2",
"pyasn1>=0.6.5",
]
# soupsieve (via co-core[extract] and bs4) 4.1 fixes three selector DoS advisories on 2.10
# (GHSA-mmhr-3c5m-f64f, GHSA-6655-6qjp-jx7x, GHSA-7g44-g9rc-x53w; #138). co-core 0.38.0's
# extract extra caps it <3 and a constraint only narrows, so this overrides co-core's
# requirement. Remove it at the co-core bump to >=0.46.1, which declares >=4.1,<5 itself:
# tests/test_lock_floors.py fails once co-core's own requirement admits >=4.1, the cue.
# Extraction stack: LOCAL_GENERATION 4.
override-dependencies = ["soupsieve>=4.1,<5"]
4 changes: 2 additions & 2 deletions src/processor/processors/extract.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,8 +22,8 @@
# Processor's own logic (config merging, dispatch), or the locked extraction stack
# moving outside a co-core bump (tests/test_pin.py's EXTRACTION_STACK). 1 matched
# Watcher's LOCAL_EXTRACTION_GENERATION at the cutover; 2 is pypdf 6.19.0 -> 6.21.2 (#71);
# 3 is pypdf 6.21.2 -> 6.21.4 (#112).
LOCAL_GENERATION = 3
# 3 is pypdf 6.21.2 -> 6.21.4 (#112); 4 is soupsieve 2.10 -> 4.1.1 (#138).
LOCAL_GENERATION = 4
PROCESSOR_VERSION = processor_version(LOCAL_GENERATION)


Expand Down
Loading
Loading