diff --git a/docs/PROVISIONING.md b/docs/PROVISIONING.md index 5a5a996..4533f96 100644 --- a/docs/PROVISIONING.md +++ b/docs/PROVISIONING.md @@ -21,7 +21,7 @@ What the operator set up once, on `co-processor` and around it: grants and crede | Tailnet `tag:processor` → `tag:status:9000` | tailnet policy | #35 gate 2, done 2026-10-07: `http://status:9000/health` answered `production` from `co-processor` at 14:03Z | | Status tenant `co-processor`, its production key, and the monitor `co-processor-drift` (`01M4BHMGGTXWQ16J3G5WWDQRYQ`) | `co-status` | CannObserv/status#24 (#35 gate 1), done 2026-10-07 16:02Z | | Status key at `/etc/processor/status-checkin.key` (root 600; `processor-drift.service`'s `LoadCredential=`) | this VM | #35 gate 3, [The drift check](MONITORING.md#the-drift-check) | -| Watcher's reader: `objectViewer` on `co-gcs-processor`, bucket level, for `co-gcs-blob-reader`, the identity Watcher already reads `gs://` blobs with | GCP | done 2026-10-02 (in the bucket's IAM policy); Watcher confirmed that identity on watcher#325 the same day ([its answers](https://github.com/CannObserv/watcher/issues/325#issuecomment-5958475061)) | +| Watcher's reader: `objectViewer` on `co-gcs-processor`, bucket level, for `co-gcs-blob-reader`, the identity Watcher then read `gs://` blobs with | GCP | done 2026-10-02 (in the bucket's IAM policy); Watcher confirmed that identity on watcher#325 the same day ([its answers](https://github.com/CannObserv/watcher/issues/325#issuecomment-5958475061)). **Retired 2026-11-15:** binding removed, account deleted (CannObserv/watcher#420) | ### Broker credential handoff (hash-only, broker#75 as of 2026-09-30) @@ -74,13 +74,8 @@ gcloud storage buckets add-iam-policy-binding gs://co-gcs-processor \ gcloud storage buckets add-iam-policy-binding gs://co-gcs-blobs \ --member="serviceAccount:${SA}" --role=roles/storage.objectViewer -# 4. Watcher's reader. Proposed: co-gcs-blob-reader, which Watcher already reads -# gs:// blobs with (GCS_BLOB_CREDENTIALS). It gains nothing new, since the text -# is derived from blobs that identity can read. Run 2026-10-02, ahead of -# watcher#325's answer; if Watcher names another identity, grant that one too. -gcloud storage buckets add-iam-policy-binding gs://co-gcs-processor \ - --member="serviceAccount:co-gcs-blob-reader@${PROJECT}.iam.gserviceaccount.com" \ - --role=roles/storage.objectViewer +# 4. Retired 2026-11-15: Watcher's reader, co-gcs-blob-reader (granted 2026-10-02, +# removed with the account on CannObserv/watcher#420). Re-run nothing here. # 5. The key, straight onto co-processor over SSH stdin, never left on disk here. ( diff --git a/docs/specs/2026-09-29-processor-service-design.md b/docs/specs/2026-09-29-processor-service-design.md index 6229f37..2083de2 100644 --- a/docs/specs/2026-09-29-processor-service-design.md +++ b/docs/specs/2026-09-29-processor-service-design.md @@ -137,6 +137,13 @@ on `co-processor`. stays `0.38.0+3`. No behaviour changes: the same wheel bytes, which the lock holds to their sha256. +**Amended 2026-11-15 (Watcher's reader retired, processor#130):** +- Watcher no longer reads `gs://co-gcs-processor` (CannObserv/watcher#379). Its + `co-gcs-blob-reader` binding is removed and the account deleted + (CannObserv/watcher#420, read back 2026-11-15). Its Grants row is gone; D4 + names no reader; Section 1 and the prefix note name only Archiver and + Provisioner (Decisions, Sections 1 and 2). + --- ## Why a new service @@ -178,7 +185,7 @@ state and UI it should not have). | D1 | The service is **Processor**: repo `CannObserv/processor`, VM `co-processor`. `command.processor` names a transform *within* it (`"extract"` in v1). | | D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). | | D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. | -| D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. | +| D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so each reader's grant is a plain bucket-level `objectViewer`. *Amended 2026-11-15 (processor#130): was "Watcher's read grant"; Watcher's reader retired on watcher#420.* | | D5 | **co-core pinned exactly; Processor owns the pin** (0.38.0 today, processor#37). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models or `canonical_text` semantics. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged. Amended 2026-10-21 (processor#57): essence resolution left the contract, so a change to it is notice (Section 5).* | | D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. | | D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). | @@ -202,8 +209,8 @@ adapters later, selected by `command.processor` + spec); its bucket; draining **Does not own:** spec authoring (issuer), interpretation (issuers), fetching bytes (Replicator), change detection and diffing (Watcher), the durable registry -(Archiver), what readers do with its output. Watcher, Archiver and Provisioner -each read `blobs/` with their own `objectViewer` (Section 2). +(Archiver), what readers do with its output. Archiver and Provisioner each read +`blobs/` with their own `objectViewer` (Section 2). **v1:** Watcher's text extraction — #629's requirements 1–7 as amended here. **Later, each its own spec:** agenda adapters; Observo as an issuer; migrating @@ -238,14 +245,15 @@ processor-version pin on the command; no job API — the bus is the interface. | `co-gcs-processor-writer` | `gs://co-gcs-processor` | `objectCreator` + `objectViewer`; **no delete** — append-only, never deleted | | `co-gcs-processor-writer` | `gs://co-gcs-blobs` | `objectViewer` (Replicator's raw blobs — the input) | | Status tenant `co-processor` (production key, `/etc/processor/status-checkin.key`) | Status monitors `co-processor-drift` and `co-processor-live` | check-in only (`POST /api/v1/monitors/{id}/checkin`); Status's operator owns the monitors and their channels (CannObserv/status#24, #35, #39) | -| Watcher's service account, `co-gcs-blob-reader` (its `GCS_BLOB_CREDENTIALS`; confirmed on watcher#325, granted 2026-10-02) | `gs://co-gcs-processor` | `objectViewer`, bucket-level | | Archiver's `archiver-processor-text-reader@co-gcs.iam.gserviceaccount.com` (its notification composer: a size check, a ranged read and a hash check of `blobs/.bin`; CannObserv/archiver#367 D4). No binding on `gs://co-gcs-blobs`, read 2026-10-31 | `gs://co-gcs-processor` | `objectViewer`, bucket-level | | Provisioner's `co-gcs-provisioner-reader@co-gcs.iam.gserviceaccount.com` (`provisioner-consumer`, by `LoadCredential=`: the current text by `extracted_fingerprint`; CannObserv/provisioner's spec D6). Bound since Provisioner's go-live, recorded here 2026-10-31 (processor#106) | `gs://co-gcs-processor` | `objectViewer`, bucket-level | The GCP rows match the buckets' IAM policies as the operator read them on 2026-10-31 (`gs://co-gcs-processor` etag `CAY=`, `gs://co-gcs-blobs` etag -`CAk=`; processor#106). `gs://co-gcs-processor` also carries the defaults GCS -gives a new bucket, for project `co-gcs`: Owner and Editor hold +`CAk=`; processor#106), less Watcher's `co-gcs-blob-reader` binding, removed +2026-11-15 (read back on CannObserv/watcher#420, no etag recorded). +`gs://co-gcs-processor` also carries the defaults GCS gives a new bucket, for +project `co-gcs`: Owner and Editor hold `legacyBucketOwner` and `legacyObjectOwner`, Viewer holds `legacyBucketReader` and `legacyObjectReader`. So anyone with Viewer on `co-gcs` can read our objects. @@ -257,11 +265,10 @@ and the broker widens grants live. Bucket: uniform bucket-level access, public access prevention, no lifecycle rules, key scheme `blobs/.bin` (co-core's cohort scheme, -`DEFAULT_PREFIX = "blobs"`). Every reader's grant is bucket-level, so Watcher, -Archiver and Provisioner read every prefix, and all three read `blobs/` today. A -later processor whose output any of them must not read gets its own prefix then, -and the split names, for each of the three, whether its grant narrows to -`blobs/`. +`DEFAULT_PREFIX = "blobs"`). Every reader's grant is bucket-level, so Archiver +and Provisioner read every prefix, and both read `blobs/` today. A later +processor whose output either must not read gets its own prefix then, and the +split names, for each of the two, whether its grant narrows to `blobs/`. ## Section 3 — runtime and code structure