Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
11 changes: 3 additions & 8 deletions docs/PROVISIONING.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ What the operator set up once, on `co-processor` and around it: grants and crede
| Tailnet `tag:processor` → `tag:status:9000` | tailnet policy | #35 gate 2, done 2026-10-07: `http://status:9000/health` answered `production` from `co-processor` at 14:03Z |
| Status tenant `co-processor`, its production key, and the monitor `co-processor-drift` (`01M4BHMGGTXWQ16J3G5WWDQRYQ`) | `co-status` | CannObserv/status#24 (#35 gate 1), done 2026-10-07 16:02Z |
| Status key at `/etc/processor/status-checkin.key` (root 600; `processor-drift.service`'s `LoadCredential=`) | this VM | #35 gate 3, [The drift check](MONITORING.md#the-drift-check) |
| Watcher's reader: `objectViewer` on `co-gcs-processor`, bucket level, for `co-gcs-blob-reader`, the identity Watcher already reads `gs://` blobs with | GCP | done 2026-10-02 (in the bucket's IAM policy); Watcher confirmed that identity on watcher#325 the same day ([its answers](https://github.com/CannObserv/watcher/issues/325#issuecomment-5958475061)) |
| Watcher's reader: `objectViewer` on `co-gcs-processor`, bucket level, for `co-gcs-blob-reader`, the identity Watcher then read `gs://` blobs with | GCP | done 2026-10-02 (in the bucket's IAM policy); Watcher confirmed that identity on watcher#325 the same day ([its answers](https://github.com/CannObserv/watcher/issues/325#issuecomment-5958475061)). **Retired 2026-11-15:** binding removed, account deleted (CannObserv/watcher#420) |

### Broker credential handoff (hash-only, broker#75 as of 2026-09-30)

Expand Down Expand Up @@ -74,13 +74,8 @@ gcloud storage buckets add-iam-policy-binding gs://co-gcs-processor \
gcloud storage buckets add-iam-policy-binding gs://co-gcs-blobs \
--member="serviceAccount:${SA}" --role=roles/storage.objectViewer

# 4. Watcher's reader. Proposed: co-gcs-blob-reader, which Watcher already reads
# gs:// blobs with (GCS_BLOB_CREDENTIALS). It gains nothing new, since the text
# is derived from blobs that identity can read. Run 2026-10-02, ahead of
# watcher#325's answer; if Watcher names another identity, grant that one too.
gcloud storage buckets add-iam-policy-binding gs://co-gcs-processor \
--member="serviceAccount:co-gcs-blob-reader@${PROJECT}.iam.gserviceaccount.com" \
--role=roles/storage.objectViewer
# 4. Retired 2026-11-15: Watcher's reader, co-gcs-blob-reader (granted 2026-10-02,
# removed with the account on CannObserv/watcher#420). Re-run nothing here.

# 5. The key, straight onto co-processor over SSH stdin, never left on disk here.
(
Expand Down
29 changes: 18 additions & 11 deletions docs/specs/2026-09-29-processor-service-design.md
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,13 @@ on `co-processor`.
stays `0.38.0+3`. No behaviour changes: the same wheel bytes, which the lock
holds to their sha256.

**Amended 2026-11-15 (Watcher's reader retired, processor#130):**
- Watcher no longer reads `gs://co-gcs-processor` (CannObserv/watcher#379). Its
`co-gcs-blob-reader` binding is removed and the account deleted
(CannObserv/watcher#420, read back 2026-11-15). Its Grants row is gone; D4
names no reader; Section 1 and the prefix note name only Archiver and
Provisioner (Decisions, Sections 1 and 2).

---

## Why a new service
Expand Down Expand Up @@ -178,7 +185,7 @@ state and UI it should not have).
| D1 | The service is **Processor**: repo `CannObserv/processor`, VM `co-processor`. `command.processor` names a transform *within* it (`"extract"` in v1). |
| D2 | **Headless and stateless in v1**: no UI, no database. Idempotency is the write-if-absent store; correlation and dedupe are the issuer's (by `command_id`). |
| D3 | **Own identities**: broker ACL user, GCS service account, bucket, tailnet tag. Observo's broker#62 `observo` user is stripped. |
| D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so Watcher's read grant is a plain bucket-level `objectViewer`. |
| D4 | **Own bucket** `gs://co-gcs-processor`, derived outputs only, so each reader's grant is a plain bucket-level `objectViewer`. *Amended 2026-11-15 (processor#130): was "Watcher's read grant"; Watcher's reader retired on watcher#420.* |
| D5 | **co-core pinned exactly; Processor owns the pin** (0.38.0 today, processor#37). Bumps are deliberate (Section 5). Processor coordinates with Watcher only when a bump changes the contract: the `content.process` / `content.derived` models or `canonical_text` semantics. It **gives Watcher notice whenever `processor_version` moves**, because Watcher re-baselines on it. Watcher still pins co-core for contracts and the bus (`==0.19.7` on 2026-10-10), and that pin no longer has to equal Processor's. *Amended 2026-10-10 (processor#47, after watcher#350). Until then: matched to Watcher, in lockstep, since both sides extracted during shadow. Watcher agreed to that on 2026-10-02 (watcher#325), and neither side deployed a bump until the parity corpus passed on it unchanged. Amended 2026-10-21 (processor#57): essence resolution left the contract, so a change to it is notice (Section 5).* |
| D6 | **Extraction runs in a killable child process** with a timeout and an address-space limit, never a thread. |
| D7 | Broker `NOPERM` and `OOM` are **transient**; a reclaim **re-runs** the extraction (no dedupe key). |
Expand All @@ -202,8 +209,8 @@ adapters later, selected by `command.processor` + spec); its bucket; draining

**Does not own:** spec authoring (issuer), interpretation (issuers), fetching
bytes (Replicator), change detection and diffing (Watcher), the durable registry
(Archiver), what readers do with its output. Watcher, Archiver and Provisioner
each read `blobs/` with their own `objectViewer` (Section 2).
(Archiver), what readers do with its output. Archiver and Provisioner each read
`blobs/` with their own `objectViewer` (Section 2).

**v1:** Watcher's text extraction — #629's requirements 1–7 as amended here.
**Later, each its own spec:** agenda adapters; Observo as an issuer; migrating
Expand Down Expand Up @@ -238,14 +245,15 @@ processor-version pin on the command; no job API — the bus is the interface.
| `co-gcs-processor-writer` | `gs://co-gcs-processor` | `objectCreator` + `objectViewer`; **no delete** — append-only, never deleted |
| `co-gcs-processor-writer` | `gs://co-gcs-blobs` | `objectViewer` (Replicator's raw blobs — the input) |
| Status tenant `co-processor` (production key, `/etc/processor/status-checkin.key`) | Status monitors `co-processor-drift` and `co-processor-live` | check-in only (`POST /api/v1/monitors/{id}/checkin`); Status's operator owns the monitors and their channels (CannObserv/status#24, #35, #39) |
| Watcher's service account, `co-gcs-blob-reader` (its `GCS_BLOB_CREDENTIALS`; confirmed on watcher#325, granted 2026-10-02) | `gs://co-gcs-processor` | `objectViewer`, bucket-level |
| Archiver's `archiver-processor-text-reader@co-gcs.iam.gserviceaccount.com` (its notification composer: a size check, a ranged read and a hash check of `blobs/<sha256>.bin`; CannObserv/archiver#367 D4). No binding on `gs://co-gcs-blobs`, read 2026-10-31 | `gs://co-gcs-processor` | `objectViewer`, bucket-level |
| Provisioner's `co-gcs-provisioner-reader@co-gcs.iam.gserviceaccount.com` (`provisioner-consumer`, by `LoadCredential=`: the current text by `extracted_fingerprint`; CannObserv/provisioner's spec D6). Bound since Provisioner's go-live, recorded here 2026-10-31 (processor#106) | `gs://co-gcs-processor` | `objectViewer`, bucket-level |

The GCP rows match the buckets' IAM policies as the operator read them on
2026-10-31 (`gs://co-gcs-processor` etag `CAY=`, `gs://co-gcs-blobs` etag
`CAk=`; processor#106). `gs://co-gcs-processor` also carries the defaults GCS
gives a new bucket, for project `co-gcs`: Owner and Editor hold
`CAk=`; processor#106), less Watcher's `co-gcs-blob-reader` binding, removed
2026-11-15 (read back on CannObserv/watcher#420, no etag recorded).
`gs://co-gcs-processor` also carries the defaults GCS gives a new bucket, for
project `co-gcs`: Owner and Editor hold
`legacyBucketOwner` and `legacyObjectOwner`, Viewer holds `legacyBucketReader`
and `legacyObjectReader`. So anyone with Viewer on `co-gcs` can read our objects.

Expand All @@ -257,11 +265,10 @@ and the broker widens grants live.

Bucket: uniform bucket-level access, public access prevention, no lifecycle
rules, key scheme `blobs/<sha256>.bin` (co-core's cohort scheme,
`DEFAULT_PREFIX = "blobs"`). Every reader's grant is bucket-level, so Watcher,
Archiver and Provisioner read every prefix, and all three read `blobs/` today. A
later processor whose output any of them must not read gets its own prefix then,
and the split names, for each of the three, whether its grant narrows to
`blobs/`.
`DEFAULT_PREFIX = "blobs"`). Every reader's grant is bucket-level, so Archiver
and Provisioner read every prefix, and both read `blobs/` today. A later
processor whose output either must not read gets its own prefix then, and the
split names, for each of the two, whether its grant narrows to `blobs/`.

## Section 3 — runtime and code structure

Expand Down
Loading