diff --git a/.github/SECURITY.md b/.github/SECURITY.md index d5acdcc747..7315142a06 100644 --- a/.github/SECURITY.md +++ b/.github/SECURITY.md @@ -37,8 +37,8 @@ include: ## What Counts as a Vulnerability -See the [Security Guide](../docs/security.md) and -[Threat Model](../docs/threat-model.md) for the security architecture, trust +See the [Security Guide](../docs/security/security.md) and +[Threat Model](../docs/security/threat-model.md) for the security architecture, trust boundaries, and assumptions this section is based on. In scope: the Agent Substrate control plane (`ateapi`, `atecontroller`, @@ -81,7 +81,7 @@ from most to least severe: - Sample applications under `demos/` and load-testing harnesses under `benchmarking/`. - Documented scope boundaries and future capabilities listed in - [docs/security.md](../docs/security.md) at the time of your report. If you are + [docs/security/security.md](../docs/security/security.md) at the time of your report. If you are unsure whether a finding is covered by a documented limitation, report it privately anyway and let a maintainer make the call. diff --git a/.github/workflows/site.yaml b/.github/workflows/site.yaml new file mode 100644 index 0000000000..efca938169 --- /dev/null +++ b/.github/workflows/site.yaml @@ -0,0 +1,120 @@ +# Copyright 2026 Google LLC +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +# Builds the website in site/ on every pull request that could change it, and +# publishes it to GitHub Pages from main. +name: site +on: + pull_request: + paths: + - 'site/**' + - 'docs/**' + - 'demos/*/README.md' + - 'tools/site-content/**' + - 'tools/site-release-data/**' + - 'hack/site.sh' + - 'hack/test-site-templates.sh' + - '.github/workflows/site.yaml' + push: + branches: [main] + paths: + - 'site/**' + - 'docs/**' + - 'demos/*/README.md' + - 'tools/site-content/**' + - 'tools/site-release-data/**' + - 'hack/site.sh' + - 'hack/test-site-templates.sh' + - '.github/workflows/site.yaml' + workflow_dispatch: +permissions: + contents: read +concurrency: + group: ${{ github.workflow }}-${{ github.ref }} + cancel-in-progress: ${{ github.event_name == 'pull_request' }} +env: + # Keep in sync with module.hugoVersion.min in site/hugo.yaml. + HUGO_VERSION: 0.163.3 +jobs: + build: + runs-on: ubuntu-latest + timeout-minutes: 15 + steps: + - name: Checkout + uses: actions/checkout@fbc6f3992d24b796d5a048ff273f7fcc4a7b6c09 # v5.1.0 + with: + # site-release-data needs the latest release tag and its history. + fetch-depth: 0 + - name: Setup Go + uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + with: + # The root module, for tools/site-release-data. Hugo fetches Docsy + # with the same Go. + go-version-file: 'go.mod' + cache: false + - name: Setup Node + uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v7.0.0 + with: + node-version: 22 + cache: npm + cache-dependency-path: site/package-lock.json + - name: Install Hugo + run: | + base="https://github.com/gohugoio/hugo/releases/download/v${HUGO_VERSION}" + tarball="hugo_extended_${HUGO_VERSION}_linux-amd64.tar.gz" + cd "${RUNNER_TEMP}" + curl -sSfLO "${base}/${tarball}" + curl -sSfLO "${base}/hugo_${HUGO_VERSION}_checksums.txt" + grep " ${tarball}\$" "hugo_${HUGO_VERSION}_checksums.txt" | sha256sum -c - + mkdir -p hugo && tar -xzf "${tarball}" -C hugo hugo + echo "${RUNNER_TEMP}/hugo" >> "${GITHUB_PATH}" + - name: Install PostCSS + working-directory: site + run: npm ci + - name: Generate site content + run: hack/site.sh content + - name: Generate release data + run: hack/site.sh release-data + - name: Test templates + if: github.event_name == 'pull_request' + run: hack/test-site-templates.sh + - name: Configure Pages + id: pages + if: github.event_name != 'pull_request' + uses: actions/configure-pages@45bfe0192ca1faeb007ade9deae92b16b8254a0d # v6.0.0 + - name: Build + working-directory: site + env: + BASE_URL: ${{ steps.pages.outputs.base_url }} + run: hugo build --environment production --minify ${BASE_URL:+--baseURL "${BASE_URL}/"} + - name: Upload artifact + if: github.event_name != 'pull_request' + uses: actions/upload-pages-artifact@fc324d3547104276b827a68afc52ff2a11cc49c9 # v5.0.0 + with: + path: site/public + deploy: + if: github.event_name != 'pull_request' + needs: build + runs-on: ubuntu-latest + timeout-minutes: 10 + permissions: + pages: write + id-token: write + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@368f82528645a54fb793d4d04e342629a3f51346 # v5.0.1 diff --git a/AGENTS.md b/AGENTS.md index 30b87bdd71..3919a29188 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -132,7 +132,7 @@ Do not add `reserved` statements to protos, deprecated aliases, fallbacks for ol ## Version Skew -From v1.0.0, changes must keep upgrades and rollbacks working under [docs/version-skew-policy.md](docs/version-skew-policy.md). Read it before you change an RPC, a CRD, the worker pod template, a database migration, or file(s) that `atelet` and `ateworker` share on a node. +From v1.0.0, changes must keep upgrades and rollbacks working under [docs/reference/version-skew-policy.md](docs/reference/version-skew-policy.md). Read it before you change an RPC, a CRD, the worker pod template, a database migration, or file(s) that `atelet` and `ateworker` share on a node. ## Commit Messages @@ -149,7 +149,7 @@ Some attributes are defined in the registry but barred from metric labels: actor `docs/metrics/substrate.yaml` holds the rules Weaver cannot express: the cardinality rules, the known exceptions, and the subsystems that emit no metrics. Read `blind_spots` before you attribute a fault to a component. -See the [metric registry](docs/observability.md#the-metric-registry) section of the observability guide. +See the [metric registry](docs/guides/observability.md#the-metric-registry) section of the observability guide. ## Testing Instructions @@ -161,9 +161,9 @@ See the [metric registry](docs/observability.md#the-metric-registry) section of ## Security Considerations -Agent Substrate is built for zero-trust, multi-tenant execution of untrusted actors. Preserve and extend these invariants when modifying the codebase (see `docs/security.md` and `docs/threat-model.md` for the full architecture): +Agent Substrate is built for zero-trust, multi-tenant execution of untrusted actors. Preserve and extend these invariants when modifying the codebase (see `docs/security/security.md` and `docs/security/threat-model.md` for the full architecture): - **Workload & Storage Isolation**: Actors run strictly inside `gVisor` (`cmd/ateworker-gvisor`) or `Cloud Hypervisor` microVM (`cmd/ateworker-microvm`) sandboxes; plain Linux containers are not supported. `atelet` scrubs actor directories across lifecycle transitions (`resetActorDirs`), and snapshots are stamped with `ActorSandboxRuntime` metadata and transferred via `cmd/objectstore-plugin`. - **Component & Actor Identity (mTLS, JWTs, and JWKs)**: Inter-component communication uses mTLS backed by `cmd/podcertcontroller` (`podid`, `servicedns`, and `postgres` signers). Actor JWT signing (`internal/localjwtauthority`), verification (`internal/oidcjwt`), and key discovery (served via `cmd/ate-idp-server`) are built on the standard library's crypto packages. Extend those packages rather than adding `go-jose` or another JOSE library, so the code that decides whether a token is valid stays small and supports only the algorithms Substrate uses. -- **Control-Plane Authentication & Authorization**: `ate-api-server` checks every Control RPC against an embedded OpenFGA model (`cmd/ateapi/internal/authz/model.fga`), always enforced. Every served unary RPC needs an entry in `defaultRPCPermissions` (`registry.go`): an RPC without one is denied, and a test fails the build. Access is granted per global scope and atespace through AccessPolicy (`kubectl ate access-policy`); the install's bootstrap owner is the shared `ate-client` ServiceAccount that `kubectl ate` uses. See `docs/authorization.md` for roles and known gaps. Callers are authenticated via `SPIFFE` mTLS or configured OIDC `JWTProvider`s (`cmd/ateapi/internal/apiauthn`). +- **Control-Plane Authentication & Authorization**: `ate-api-server` checks every Control RPC against an embedded OpenFGA model (`cmd/ateapi/internal/authz/model.fga`), always enforced. Every served unary RPC needs an entry in `defaultRPCPermissions` (`registry.go`): an RPC without one is denied, and a test fails the build. Access is granted per global scope and atespace through AccessPolicy (`kubectl ate access-policy`); the install's bootstrap owner is the shared `ate-client` ServiceAccount that `kubectl ate` uses. See `docs/guides/authorization.md` for roles and known gaps. Callers are authenticated via `SPIFFE` mTLS or configured OIDC `JWTProvider`s (`cmd/ateapi/internal/apiauthn`). - **Network Isolation, Ingress, & Egress**: `manifests/ate-install/ate-system-networkpolicies.yaml` enforces default-deny ingress across `ate-system`, and `cmd/atecontroller` reconciles per-`WorkerPool` `NetworkPolicy` objects. Actors run in private `/30` point-to-point veth network namespaces (`internal/ateworker/actornet`) with link-local blocking, policy-enforced DNS filtering (`internal/atunnel/dns`), and zero ambient credentials. Outbound traffic traverses `atunnel` to `atenet-egress`, which enforces per-actor `EgressPolicy` and injects credentials out-of-band via `cmd/credential-provider/kubernetes-secrets` (`k8s-credential-provider`) or actor JWT minting. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 022d65cb3e..996d14f619 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,132 +1,3 @@ # How to Contribute -We would love to accept your patches and contributions to this project. Before -you spend a lot of time on a contribution, please review the following -guidelines. - -## Before you begin - -### Sign off your commits - -Every commit must be signed off under the -[Developer Certificate of Origin](https://developercertificate.org/) (DCO). -The `Signed-off-by` line certifies that you wrote the change or otherwise have -the right to submit it under the project's license. - -Sign off with your real name, not an anonymous ID or a false name. It doesn't -need to be your legal name, just the name the community knows you by. See the -[CNCF DCO guidelines](https://github.com/cncf/foundation/blob/main/policies-guidance/dco-guidelines.md). - -Git adds the line with `-s`, using `user.name` and `user.email` from your git -config. The name and email must match the commit author. - -```sh -git commit -s -m "fix typo in README" -``` - -The DCO check on your PR fails if any commit is missing a sign-off. To sign off -the commits already on your branch, then force-push: - -```sh -git rebase --signoff upstream/main -``` - -### Review our Community Guidelines - -This project follows [Google's Open Source Community -Guidelines](https://opensource.google/conduct/). - -### Set up a local development environment - -The [Quickstart (Development)](README.md#quickstart-development) in the README -covers bringing up a local cluster with the default (gVisor) runtime. To run -the microVM runtime locally — which needs `/dev/kvm`, or Lima nested -virtualization on Apple Silicon — see -[docs/dev/microvm-local.md](docs/dev/microvm-local.md). To measure where a -suspend or resume spends its time on such a cluster, follow -[docs/dev/suspend-resume-phase-breakdown.md](docs/dev/suspend-resume-phase-breakdown.md). - -## Contribution process - -This is a very new project, so we are still working out exactly how it is going -to be developed. For now, we are focused on iterating quickly to find the right -design and architecture. This has implications for contributors: - -1) Things are moving quickly, so PRs may need to be rebased or updated -frequently. Small PRs that are focused on a single issue or feature are easier -to review and update than large PRs that touch many different parts of the -codebase. - -2) While we welcome new contributors, we are really focused on the minimal -capabilities needed to make this project useful. Small fixes and cleanups can -go straight to a PR. Before you start anything larger, please discuss it with -us first (if there is an issue open, comment there and if not, open one), and -search open PRs so we are not duplicating work that is already in flight. We -want to make sure that your work is aligned with our near-term goals for the -project. - -3) PRs which are not aligned with our near-term goals may be closed without -extensive review. We are not trying to be discouraging, but we need to make -sure that we are focused on the most important work. - -If you are building something that runs *on* Substrate rather than changing -Substrate itself, see [Integration -Repositories](docs/integration-repos.md) for where that code should live. - -### Sizing PRs for review - -We optimize PRs for easy review — large PRs get broken -down, small PRs get merged. - -* **Large PRs**: split huge changes into a series of smaller PRs, each a - logically distinct feature. When the intermediate steps are not useful - on their own, keep the change as one PR split into commits at logical - break points for review. PRs are squash-merged, so only the branch - `HEAD` needs to build and pass tests. -* **Small and bulk PRs**: if you find a typo, review the whole file and - fix everything in one pass rather than sending the single edit. Group - related typo, doc, and single-line cleanup fixes into one PR rather - than opening several small ones for the same area. Maintainers may ask - you to consolidate fragmented PRs into one, or close them in favor of a - combined submission. - -As a rough scale: S is under 30 changed lines, M under 100, L under 500, XL under 1000. Most PRs should be L or smaller; XL and above are candidates for breaking down. - -### Code Reviews - -All submissions, including submissions by project members, require review. We -use [GitHub pull requests](https://docs.github.com/articles/about-pull-requests) -for this purpose. - -Changes that add or alter behavior should come with tests, and existing tests -should be updated when behavior they cover changes. We will not merge code that -causes tests to fail. Every test runs in CI on every PR, so please don't add -tests whose only purpose is to raise coverage numbers. Tests should be -inherently valuable for future changes. - -Follow the [PostgreSQL schema evolution rules](docs/dev/postgresql-schema-evolution.md) for each application schema change. - -Each pull request title becomes its line in the [release notes](docs/dev/release-notes.md). - -### Root-gated tests - -Tests that need root (overlay mounts, mknod, `trusted.*` xattrs, ...) call -`roottest.Require(t, ...)` from [internal/roottest](internal/roottest) as -their first statement. They skip in a plain `go test ./...`; CI reruns every -package whose tests import that package under `sudo`. To run them locally: - -```sh -hack/run-root-tests.sh -``` - -New privileged tests only need the `roottest.Require` call — no CI changes. - -### Copyright Headers - -Every file containing source code must include copyright and license -information. This includes any JS/CSS files that you might be serving out to -browsers. (This is to help well-intentioned people avoid accidental copying -that doesn't comply with the license.) - -Our standard headers for various filetypes can be found in -[./hack/boilerplate](./hack/boilerplate/). +Moved to [docs/project/contributing.md](docs/project/contributing.md). diff --git a/README.md b/README.md index 6e89e288e6..d6dd7a40ea 100644 --- a/README.md +++ b/README.md @@ -11,7 +11,7 @@ [![License](https://img.shields.io/badge/License-Apache_2.0-blue.svg)](https://opensource.org/licenses/Apache-2.0) -[Quickstart](#quickstart-development) · [Security](docs/security.md) · [Architecture](docs/architecture.md) · [Contributing](CONTRIBUTING.md) +[Quickstart](#quickstart-development) · [Security](docs/security/security.md) · [Architecture](docs/concepts/architecture.md) · [Contributing](CONTRIBUTING.md) @@ -25,12 +25,12 @@ Substrate is a low-opinion runtime, not an agent SDK. The workloads it runs do n ## Security Model -Substrate treats every actor as untrusted and mutually hostile, so tenants can share physical nodes without sharing a security boundary. Every control operates within the cold-start and resume budget. See the [Security Guide](docs/security.md) for the full list of security principles and threat model. +Substrate treats every actor as untrusted and mutually hostile, so tenants can share physical nodes without sharing a security boundary. Every control operates within the cold-start and resume budget. See the [Security Guide](docs/security/security.md) for the full list of security principles and threat model. * **Always sandboxed, VM-grade boundaries:** Every actor runs inside a dedicated [gVisor](https://gvisor.dev/) (`runsc`) or [Cloud Hypervisor](https://www.cloudhypervisor.org/) microVM sandbox with segregated compute, network, and storage. Plain Linux containers are not supported. * **Per-agent cryptographic identity:** Every active actor receives a short-lived [SPIFFE](https://spiffe.io/) X.509 certificate by default, and the egress gateway can mint per-actor JWTs on demand when an egress policy calls for one. Both carry the actor identity and atespace (the Substrate namespace grouping), and are designed to federate with external identity systems. -* **Zero credentials in the sandbox:** Actors never hold actor identity JWTs/certs, API keys, or cloud tokens. The built-in egress gateway ([`atenet-egress`](docs/egress-credential-injection.md)) injects credentials into approved outbound requests outside the sandbox, so a prompt-injected or compromised agent has nothing to exfiltrate. -* **Default-deny networking:** Actors start with a private network stack, no inbound path except through the ingress router (`atenet-router`), no lateral actor-to-actor or control-plane access, and outbound traffic [restricted by the egress gateway](docs/egress-traffic.md). +* **Zero credentials in the sandbox:** Actors never hold actor identity JWTs/certs, API keys, or cloud tokens. The built-in egress gateway ([`atenet-egress`](docs/guides/egress-credential-injection.md)) injects credentials into approved outbound requests outside the sandbox, so a prompt-injected or compromised agent has nothing to exfiltrate. +* **Default-deny networking:** Actors start with a private network stack, no inbound path except through the ingress router (`atenet-router`), no lateral actor-to-actor or control-plane access, and outbound traffic [restricted by the egress gateway](docs/guides/egress-traffic.md). * **Pluggable policy at the edge:** The egress gateway acts as an external Policy Enforcement Point (PEP) that is fully replaceable, allowing you to attach your own authorization, DLP, prompt-injection scanning, or cloud IAM policy engine without placing policy code inside the untrusted sandbox. ## Performance & Multiplexing @@ -145,7 +145,7 @@ kubectl ate proxy actor my-counter-1 -a ate-demo-counter > `{"name":"k8s.io"}` deploys the bundled credential provider with an empty, > default-deny namespace policy. Before an actor can have a credential > injected, grant its atespace the namespaces whose Secrets it may read; see -> [Enable it](docs/egress-credential-injection.md#enable-it). +> [Enable it](docs/guides/egress-credential-injection.md#enable-it). 3. In a **separate terminal**, send an HTTP request to increment the counter: ```shell @@ -190,18 +190,18 @@ curl -X POST -i http://localhost:8000/ `--ingress-allowed-spiffe-ids` lists the clients the router lets reach actors. The ID above is the one `kubectl ate proxy actor` presents. Without the flag no client reaches actors. See - [Ingress to actors](docs/operator-install.md#ingress-to-actors). + [Ingress to actors](docs/guides/operator-install.md#ingress-to-actors). `--credential-provider` is required. `{"name":"k8s.io"}` also deploys the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off - (see [docs/egress-credential-injection.md](docs/egress-credential-injection.md)). + (see [docs/guides/egress-credential-injection.md](docs/guides/egress-credential-injection.md)). `ATE_CREDENTIAL_PROVIDER` in `.ate-dev-env.sh` is the same choice. **Credential injection needs a namespace policy.** The bundled provider starts with an empty, default-deny one, so it resolves no Secrets until you grant each atespace the namespaces it may read; see - [Enable it](docs/egress-credential-injection.md#enable-it). + [Enable it](docs/guides/egress-credential-injection.md#enable-it). 5. You can then deploy the sample applications. See [demos/counter/README.md](demos/counter/README.md) or [demos/sandbox/README.md](demos/sandbox/README.md) for detailed walkthroughs. ```bash @@ -261,22 +261,22 @@ We provide several sample applications demonstrating Agent Substrate's capabilit 5. **[Request Parking](demos/parking/README.md)**: An oversubscribed pool where the router holds inbound requests until a worker frees up, instead of returning `503`. ### Documentation & Guides -* [Architecture](docs/architecture.md): How the control plane, node supervisor, and networking stack fit together. -* [API Configuration Guide](docs/api-guide.md): Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes. -* [Full CLI Documentation](cmd/kubectl-ate/README.md): Installation and usage for `kubectl-ate`. -* [Glossary](docs/glossary.md): Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateworker) and how they relate. -* [Integration Repositories](docs/integration-repos.md): Where integrations live, how their repositories are named, and how fixes flow back to core. -* [Observability Guide](docs/observability.md): Guide to actor logging, metrics, and distributed tracing. -* [Extending Substrate](docs/extensibility.md): How to augment or replace the built-in ingress and egress gateways. -* [Authentication Guide](docs/authentication.md): Configure trusted JWT providers and human credentials. -* [Egress Traffic](docs/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked. -* [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception. -* [Request Parking](docs/request-parking.md): How the router parks requests through transient worker-pool saturation. -* [Sandbox Asset Cache](docs/sandbox-asset-cache.md): How a node caches the sandbox binaries actors run on, decides which runtimes it needs, and evicts to a budget. -* [Upgrade Runbook](docs/upgrade.md): Upgrade a running install to a newer supported release, and roll back if needed. -* [Node Storage](docs/node-storage.md): How to give snapshot I/O its own disk. -* [Security Guide](docs/security.md): Security overview, security principles, and threat modeling. -* [Benchmarking Guide](benchmarking/README.md): Locust-based load tests, monitoring stack, and the orchestrated benchmark harness. +* [Architecture](docs/concepts/architecture.md): How the control plane, node supervisor, and networking stack fit together. +* [API Configuration Guide](docs/reference/api-guide.md): Detailed reference for configuring WorkerPools, ActorTemplates, Secrets, and Volumes. +* [Full CLI Documentation](docs/reference/kubectl-ate.md): Installation and usage for `kubectl-ate`. +* [Glossary](docs/reference/glossary.md): Core terms (Actor, Atespace, ActorTemplate, WorkerPool, Worker, ate-api-server, atenet, atelet, ateworker) and how they relate. +* [Integration Repositories](docs/project/integration-repos.md): Where integrations live, how their repositories are named, and how fixes flow back to core. +* [Observability Guide](docs/guides/observability.md): Guide to actor logging, metrics, and distributed tracing. +* [Extending Substrate](docs/security/extensibility.md): How to augment or replace the built-in ingress and egress gateways. +* [Authentication Guide](docs/guides/authentication.md): Configure trusted JWT providers and human credentials. +* [Egress Traffic](docs/guides/egress-traffic.md): Which protocols an Actor may reach the outside world with and which are blocked. +* [Enabling man-in-the-middle (MITM) interception for Actor Egress policy](docs/guides/egress-trust-bundle.md): Egress policies such as header injection depend on MITM interception of Actor traffic. This guide explains how an Actor should be configured to enable interception. +* [Request Parking](docs/concepts/request-parking.md): How the router parks requests through transient worker-pool saturation. +* [Sandbox Asset Cache](docs/concepts/sandbox-asset-cache.md): How a node caches the sandbox binaries actors run on, decides which runtimes it needs, and evicts to a budget. +* [Upgrade Runbook](docs/guides/upgrade.md): Upgrade a running install to a newer supported release, and roll back if needed. +* [Node Storage](docs/guides/node-storage.md): How to give snapshot I/O its own disk. +* [Security Guide](docs/security/security.md): Security overview, security principles, and threat modeling. +* [Benchmarking Guide](docs/guides/benchmarking.md): Locust-based load tests, monitoring stack, and the orchestrated benchmark harness. ## Tour @@ -291,7 +291,7 @@ We provide several sample applications demonstrating Agent Substrate's capabilit * `cmd/ateworker-microvm`: The microVM peer of `ateworker-gvisor`, running actors as cloud-hypervisor VMs. * `cmd/podcertcontroller`: A "polyfill" that provides Pod Certificate signers that will eventually ship in upstream Kubernetes (with different names). -* `cmd/kubectl-ate`: A CLI tool for managing Agent Substrate resources. See its [README](cmd/kubectl-ate/README.md). +* `cmd/kubectl-ate`: A CLI tool for managing Agent Substrate resources. See its [README](docs/reference/kubectl-ate.md). * `cmd/benchmarking`: Synthetic workloads used by the load tests, including `glutton`, which consumes RAM, disk, and file descriptors on demand. * `tools/setup-gcp`: A provisioning utility to set up the necessary GCP infrastructure resources (GKE, GCS, IAM). * `demos/`: Sample applications demonstrating Agent Substrate capabilities. diff --git a/api/ateapi/v1/AGENTS.md b/api/ateapi/v1/AGENTS.md index 511e64317d..d90bfa3edd 100644 --- a/api/ateapi/v1/AGENTS.md +++ b/api/ateapi/v1/AGENTS.md @@ -3,6 +3,6 @@ `ateapi.proto` defines Substrate's public API. - See `docs/api-style-guide.md` for the API style guide. -- See `docs/resource-model.md` for the resources and subresources the API exposes. Update it when you add, remove, or re-parent a resource. +- See `docs/reference/resource-model.md` for the resources and subresources the API exposes. Update it when you add, remove, or re-parent a resource. - See `docs/api-validation.md` for how to use declarative validation (DV) tags. - `*.pb.go` files are generated. Never edit them by hand. After changing `ateapi.proto`, run `hack/update/codegen.sh`, which also regenerates the DV code in `cmd/ateapi/internal/apivalidation`. diff --git a/benchmarking/README.md b/benchmarking/README.md index 803989d54b..50a9d9d397 100644 --- a/benchmarking/README.md +++ b/benchmarking/README.md @@ -1,1019 +1 @@ -# Substrate Benchmarking - -This is the nascent suite for benchmarking Substrate's performance at scale. - -The suite also measures the telemetry volume and the capacity of the OTel -collector: how much trace data and metric data substrate and its actors send, -and if the collector can accept it. To make a measurement, read -[telemetry/README.md](telemetry/README.md). For the prerequisites and the -scenario ladder, read [observability.md](observability.md). - -## Deploy benchmarks - -> [!IMPORTANT] -> Source the environment configuration file (e.g., `source .ate-dev-env.sh`) -> first so `PROJECT_ID` etc. are set. -> -> **For Local Kind Clusters**: Set `KO_DOCKER_REPO` to the local registry so images are pulled without remote cloud credentials: -> ```bash -> export KO_DOCKER_REPO="localhost:5001" -> ``` - -> [!IMPORTANT] -> The ate-api-server checks every call against -> [authorization](../docs/authorization.md), and Locust calls it as the -> `benchmarking/default` ServiceAccount. The Envoy ingress router also requires -> an allowlist of client SPIFFE IDs at install time. Install Substrate with -> `benchmarking/default` in both settings alongside the default `ate-client` -> owner: -> ```bash -> ATE_API_AUTHZ_BOOTSTRAP_OWNERS=system:serviceaccount:ate-system:ate-client,spiffe://cluster.local/ns/benchmarking/sa/default \ -> ATE_INGRESS_ALLOWED_SPIFFE_IDS=spiffe://cluster.local/ns/benchmarking/sa/default \ -> go run ./cmd/ate-setup deploy ate-system -> ``` -> The nightly automation sets both itself; see -> [automation/README.md](automation/README.md). - -Note that deploying the benchmarks does not run them. You must visit Locust's -web UI to start a test. - -A single wrapper deploys the scale workloads, builds and pushes the Locust -image, then deploys the Locust workers: - -```bash -./benchmarking/deploy_locust.sh --deploy -``` - -Useful flags: - -* `--worker-count N` — number of `WorkerPool` replicas (default 1). -* `--worker-memory SIZE` — memory request and limit for each `WorkerPool` pod - (default unset, the pod is unsized). Set it above half a node's allocatable - memory to get one worker per node. -* `--skip-build` — reuse the existing `:latest` locust image (skip the - `docker build && docker push` step). - -To tear everything down (locust then workloads, in reverse order): - -```bash -./benchmarking/deploy_locust.sh --delete -``` - -The same operations are also reachable from the top-level installer for -convenience: - -```bash -go run ./cmd/ate-setup deploy benchmarks -go run ./cmd/ate-setup delete benchmarks -``` - -`--skip-build` and `--worker-memory` are only available when invoking -`benchmarking/deploy_locust.sh` directly. - -### Settings - -`ate-setup` reads these settings in addition to those in -[docs/operator-install.md](../docs/operator-install.md#settings), through the same -flag, environment variable, and configuration document channels. - -| Key | Environment variable | Flag | Default | Applies to | -|--------------------------|-------------------------------|----------------------------|----------|------------------------------------------| -| `benchmark.actorMemory` | `BENCHMARK_ACTOR_MEMORY` | `--benchmark-actor-memory` | — | (global) | -| `benchmark.sandboxClass` | `ATE_BENCHMARK_SANDBOX_CLASS` | `--sandbox-class` | `gvisor` | `deploy benchmarks`, `delete benchmarks` | -| `benchmark.workerCount` | `ATE_BENCHMARK_WORKER_COUNT` | `--worker-count` | `1` | `deploy benchmarks`, `delete benchmarks` | - -## Running Tests - -### Locust Web UI -* Run `kubectl port-forward svc/locust -n benchmarking 8089:8089` -* Visit `http://localhost:8089` in your browser to configure and start the load test. - -The different user classes you can select are different types of load behaviors -you can throw at the system. Note that the "CounterUser" load type requires -that the counter demo be installed. - -You can also configure things like the number of users, how quickly those users -are spawned, the frequency with which requests are made and whether or not tracing is -enabled. - -User classes implemented in boomer rather than Python are selected at deploy -time — the stack runs one per deployment: - -```bash -./benchmarking/locust/deploy.sh --deploy --user-class durdir -``` - -### Headless (automation only) - -`runner.py` runs a test without the web UI, writing CSVs, logs and traces to -`--dest`. The nightly automation submits it as a Job on the test cluster; it is -not a local entry point. See [automation/README.md](automation/README.md). - -```bash -python3 runner.py -f tests/.py -t 1m -u 1 --name --dest /tmp/bench -``` - -Four flags control the optional post-run measurements described in -[Benchmark output files](#benchmark-output-files): - -* `--cluster-facts` / `--no-cluster-facts`: read node capacity and worker pod - count from the Kubernetes API once the run ends, to derive density frontiers, - and worker pod restarts and evictions around the run, for `oom_events`. - On by default. Pass `--no-cluster-facts` to skip Kubernetes API discovery. -* `--prometheus-url`: the Prometheus to harvest server-side telemetry from. - Defaults to the in-cluster service installed by - [Optional: Prometheus + Grafana](#optional-prometheus--grafana). -* `--atelet-lag-s`: how long to wait after the run before reading the - atelet's metrics. Defaults to 70. -* `--phase-breakdown` / `--no-phase-breakdown`: read the atelet and worker pod - logs once the run ends and add the suspend/resume phase percentiles to - `stats.jsonl`. On by default; needs `pods/log` access in `ate-system` and - `benchmark-workloads` (granted in `automation/manifests/runner-job.yaml.tmpl`). - See [analysis/README.md](analysis/README.md#automated-runs). - -Test-specific flags are appended to the same command; see the sections below. - -### DurDir Benchmark - -The DurDir benchmark evaluates actor suspend/resume performance, disk persistence overhead, -and state restoration latency when a durable directory is attached to the actor. - -#### DurDir Configuration Knobs - -* `--durdir-file-size-bytes`: Size in bytes of the data file (default `8388608` = 8 MiB). -* `--resume-mode`: Resume trigger mode: - * `explicit` (default): Client invokes the `ResumeActor` RPC before sending traffic. - * `implicit`: Client sends traffic through the router without an explicit wake RPC, testing traffic-triggered resume. -* `--durdir-read-mode`: Verification read mode: - * `data` (default): Server returns full payload bytes for client-side SHA-256 verification. - * `digest`: Server hashes the file and returns size and digest, reducing network transfer. -* `--durdir-template`: ActorTemplate name: - * `glutton-durdir-data` (default): Attaches a durable data directory without memory snapshot restore. - * `glutton-durdir-full`: Attaches a durable data directory and performs a full memory snapshot restore. - * `glutton-storage`: Attaches an external CSI volume instead of a durable - directory. `deploy_locust.sh` and `workloads/deploy.sh` deploy it only - when `--storage-class-name` names an existing StorageClass. To run it on - NFS, install with `go run ./cmd/ate-setup setup csi nfs` and pass - `--storage-class-name csi-nfs-sc`, the class the e2e tests also use. - -#### DurDir Reported Metrics - -* `DurDirWrite`: Initial truncate-write creating the data file. -* `DurDirServeInitial`: First read immediately following file creation. -* `SuspendActor`: Actor suspend latency (snapshot creation + persistence upload). -* `ResumeActor`: Actor resume latency. -* `DurDirServeAfterResume`: First read after resume (measures page faults / lazy load overhead on restored volume). -* `DurDirServeWarm`: Subsequent read within the same active cycle (cached state baseline). -* `DurDirOverwrite`: In-place file overwrite with checksum verification. - -### Sweperf Benchmark - -The sweperf benchmark replays a recorded SWE-Perf task inside an actor, suspending and -resuming between cycles to measure the cost of actor state transitions under a realistic -agent workload. One task is four cycles by default. Each task runs on a fresh actor: once -the last cycle finishes, the actor is deleted and the user creates a new one for the next -task. - -#### Sweperf Tasks - -Each task is one entry (name, digest-pinned image, step count) in -`workloads/manifests/sweperf/sweperf-images.json`. Its ActorTemplate, `sweperf-`, -is generated into `workloads/manifests/sweperf/generated/` and checked in. To add or change -a task, edit the catalog and regenerate: - -```bash -SWEPERF=benchmarking/workloads/manifests/sweperf -python3 -m venv "${SWEPERF}/venv" && "${SWEPERF}/venv/bin/pip" install -r "${SWEPERF}/requirements.txt" -"${SWEPERF}/venv/bin/python" "${SWEPERF}/generate_sweperf_templates.py" '*' -"${SWEPERF}/venv/bin/python" "${SWEPERF}/generate_sweperf_templates.py" --check # fail if generated/ is missing or stale -hack/verify/benchmarking-python.sh # unit tests, as CI runs them -``` - -Deploy a task and the sweperf user class: - -```bash -WORKLOAD_TEMPLATES=sweperf-django-11099 ./benchmarking/workloads/deploy.sh --deploy -./benchmarking/locust/deploy.sh --deploy --user-class sweperf -``` - -Then start the run with `--sweperf-template sweperf-django-11099` and -`--sweperf-total-steps 15`, set in the Locust web UI or through `LOCUST_SWEPERF_TEMPLATE` -and `LOCUST_SWEPERF_TOTAL_STEPS`. The client takes the step count from that option, not -from the template, so it must match the catalog. - -#### Sweperf Reported Metrics - -All rows are in milliseconds. CEL (command execution latency) is the time the trace commands -ran inside the sandbox, as reported by `replay.py`. - -* `ResumeToFirstExec`: Resume RPC start until the sandbox accepts the cycle's `/execute`. -* `CycleCEL`: CEL for one cycle. -* `TaskCEL`: CEL summed over one task. -* `TaskWallClock`: Client wall clock for one task, excluding inter-cycle think time and actor - startup. -* `CreateAtespace`, `CreateActor`, `ResumeActor`, `SuspendActor`, `DeleteActor`: Server-side - elapsed time for each control-plane RPC from the response trailer, or client time without one. -* `_rtt`: Client round trip for the RPC of the same name, recorded only when the trailer is - present, so network and queueing overhead stays visible separately. -* `Workload_Cycle_`: Client time for cycle `n`: `POST /execute` plus `/status` polling until - the job finishes. Polled every `--sweperf-poll-interval-ms` (default 100), so this row sits - up to one interval above the job's actual end. - -The liveness check on a new actor already has it running, so the first cycle of every task -runs on a freshly started actor and its resume is a no-op. Its successful `ResumeActor`, -`ResumeActor_rtt` and `ResumeToFirstExec` samples are not recorded; failures still are. - -The last cycle of a task is not suspended: the actor is deleted straight from running. A -task of `n` cycles therefore records `n - 1` `SuspendActor` samples, and `TaskWallClock` -does not include a final suspend. `DeleteActor` therefore times deleting a running actor, -including its sandbox teardown, so it is not comparable with older runs that suspended the -actor before deleting it. - -`locust_users` counts live actors rather than virtual users: it rises when a user starts an -actor and falls when that actor is deleted, so it dips between tasks while a user waits to -start its next actor. - -`CreateAtespace` is sent once per worker, not per task, and again only if `CreateActor` -reports the atespace missing. - -When the run ends, shutdown cancels cycles still in flight. Calls cut short this way record -nothing, and the interrupted task records no `TaskCEL` or `TaskWallClock` row, so the end of -a run does not show up as a burst of failures. - -#### Sweperf Log Lines - -The boomer log also carries per-task facts for post-processing such as the goodput script in -`tools/goodput`, which decides from them whether a task was good. The benchmark judges -nothing. Durations are in milliseconds. - -* `Completed sweperf cycle`: `actor`, `cycle`, `cel_ms` (0 when the execute failed), - `wall_ms` (resume start to cycle end), `ok` (the execute succeeded, and so did the suspend - if there was one), `resume_ms` (as on the `ResumeActor` row; omitted for a no-op resume) - and `ttfe_ms` (resume start to the asynchronous ack; omitted without one, and for a no-op - resume even when its execute got one). -* `ResumeActor failed in step`: also carries `wall_ms`. The cycle is retried on the next - iteration, and the failed attempt's time counts toward the task's `wall_ms`. -* `sweperf task record`, one per finished task: `v` (1), `user` (boomer goroutine ID, unique - per boomer process), `actor`, `template`, `cycles` and `steps` (the cycle and trace step - counts the task ran, resolved when it started, so a mid-run change to - `--sweperf-num-cycles` or `--sweperf-total-steps` applies from the next task; `cycles` is - capped at `steps`), `task_start` (when the session started) and `end`, `startup_ms` - (`CreateActor` until the sandbox answers `/status`), `cel_ms` and `wall_ms` (as in `TaskCEL` - and `TaskWallClock`), `think_ms` (think time between the task's cycles), `resume_max_ms` - (slowest non-no-op resume), `ttfe_max_ms` (slowest `ttfe_ms`; omitted when no cycle after - the no-op resume got an asynchronous ack, so always for a one-cycle task), `retries` - (failed resumes), `failed`, and the first failure's `fail_step` (`resume`, `execute` or - `suspend`) and `fail_cycle`. -* `sweperf on_start failed`: also carries `user`, to tie a failed start to its user. -* `sweperf shutdown started`: the worker began tearing down. Startups and cycles in flight - are canceled: interrupted tasks log no task record, and startup failures logged after it - belong to the teardown, not the run. - -### ate-env Benchmark - -The `ate-env` benchmark (`--user-class ate_env`) exercises the -[`ate-env-guest`](https://github.com/agent-substrate/env/tree/1c08a7a578afd896373511a83beb242aec1425c0/cmd/ate-env-guest) -sandbox image from `agent-substrate/env`. It follows `glutton`'s -wake/hold/hibernate lifecycle, but replaces synthetic HTTP `/ping` calls with a -real `"hello world"` process execution inside the guest (`StartProcess(["echo", -"hello world"])` + `StreamProcessOutput(follow: true)`) over gRPC directly -through `atenet-router` (`1 dns:///` gRPC channel per VU). - -Build the guest image from `agent-substrate/env` at commit `1c08a7a` (see -[`workloads/manifests/ate-env-template.yaml.tmpl`](workloads/manifests/ate-env-template.yaml.tmpl)), -then deploy the `ate-env` template and user class: - -```bash -ATE_ENV_IMAGE= WORKLOAD_TEMPLATES=ate-env ./benchmarking/workloads/deploy.sh --deploy -./benchmarking/locust/deploy.sh --deploy --user-class ate_env -``` - -#### ate-env Configuration Knobs - -Reuses `glutton`'s runtime and startup knobs: -`--min-wait-time`, `--max-wait-time`, `--min-live-time`, `--max-live-time`, -`--max-pings-per-wake`, `--lifecycle-mode` (`sync` | `async` | `none`), -`--trace-probability`, and `--actors-per-user`. - -#### ate-env Reported Metrics - -* `CreateAtespace`, `CreateActor`, `ResumeActorFirstResume`, `ResumeActor`, - `SuspendActor`, `SuspendActorAsync`, `DeleteActor`: Control-plane lifecycle - RPC latencies. -* `AteEnvExec`: Combined end-to-end latency of `"hello world"` (`StartProcess` + - `StreamProcessOutput` to exit with `exit_code == 0` and - `stdout == "hello world\n"`). -* `CrashCount`: Actors that entered `ACTOR_STATE_CRASHED` during resume. - -### Agent-Session Benchmark - -The agent-session benchmark (`--user-class agentsession`) emulates a fleet of -coding agents on Substrate. Each locust user is one session: an actor driven -through a scripted 20-step coding task ("clone a repo, build it, fix a test, -refactor, package it"), where every step costs the sandbox the CPU, memory, -disk, and network a real coding agent's action would. Between steps the agent -is "waiting for the LLM to think": the driver **suspends the actor** for the -step's think time, and the next step's first request **wakes it through the -atenet router** (request parking). Mostly-idle sessions plus fast wake is -exactly the oversubscription story this measures. - -The entire workload is a YAML script. The default, -[`internal/benchmarking/boomer/agentsession/scripts/coding-session.yaml`](../internal/benchmarking/boomer/agentsession/scripts/coding-session.yaml), -has one entry per step naming what the agent is doing and the resource ops -that act it out. To change the workload, edit it or add a sibling file and -select it with `--agentsession-script` (see [Writing a -script](#writing-an-agent-session-script)). Each session is an actor from -the stock `glutton` template; steps are sequences of glutton RPCs: - -| Step | The agent is… | Sandbox effect | -|---|---|---| -| 01_read_task | reading the task prompt | fill 32Mi RAM (agent context) | -| 02_clone_repo | `git clone` | 16Mi arrives over the network → disk; 0.5s CPU | -| 03_explore_tree | listing/grepping the tree | disk read (digest); 0.2s CPU | -| 04_read_key_files | opening files into context | disk read shipped back out; 8Mi RAM churn | -| 05_install_deps | `pip install` / `go mod download` | 32Mi over the network → disk; 1.5s CPU ×2 | -| 06_first_build | first full build | fill 64Mi RAM; 3s CPU ×2; 24Mi disk write | -| 07_run_unit_tests | running the test suite (one fails) | disk read; 2.5s CPU ×2 | -| 08_reason_about_failure | tracing the bug (long LLM turn, 8s think) | full RAM page-walk after the wake | -| 09_edit_source | applying the fix | 64Ki patch over the network; 4Mi disk write | -| 10_incremental_build | rebuilding changed packages | 1.2s CPU ×2; 8Mi disk write | -| 11_rerun_failed_test | re-running the failing test | 0.8s CPU | -| 12_write_new_tests | authoring regression tests (6s think) | 128Ki over the network; 2Mi disk write | -| 13_run_new_tests | running the new tests | disk read; 1s CPU | -| 14_full_test_suite | full-suite regression run | 4s CPU ×2; disk read; 8Mi RAM churn | -| 15_lint_format | lint + format pass | disk read; 0.9s CPU | -| 16_refactor | multi-file refactor (8s think) | 512Ki over the network; 6Mi disk write; 0.6s CPU | -| 17_rebuild | full rebuild | 32Mi RAM churn; 2s CPU ×2; 16Mi disk write | -| 18_final_test_suite | final full-suite run | 3.5s CPU ×2 | -| 19_package_artifact | building the release package | disk read; 1s CPU; 24Mi disk write | -| 20_commit_and_summarize | committing + summarizing | 256Ki disk write; 24Mi shipped back out; RAM walk | - -The script needs bigger actors than the 256Mi default: it holds ~96Mi of -RAM arrays + ~110Mi of tmpfs files, and the observed guest peak with -allocator transients is ~320Mi (512Mi OOMs). Deploy the workloads with -`--actor-memory 1Gi`. Every script declares that floor as -`min_actor_memory`; the worker reads the `glutton` template's memory limit -at start and refuses to run a script against a smaller actor, so a too-small -deployment fails loudly instead of showing up as OOM-flaky steps. - -```sh -./benchmarking/deploy_locust.sh --deploy --sandbox-class gvisor --actor-memory 1Gi -./benchmarking/locust/deploy.sh --deploy --user-class agentsession -``` - -#### Agent-Session Configuration Knobs - -* `--agentsession-script` — built-in script variant to run, by file name - under `internal/benchmarking/boomer/agentsession/scripts/` (default - `coding-session`). Resolved when a session starts, so a change takes - effect for sessions started after the next swarm; sessions already - running finish on the script they started with. -* `--agentsession-script-file` — path, on the boomer worker, of a script - YAML to run instead of a built-in variant; wins over - `--agentsession-script`. Normally set for you by - `locust/deploy.sh --agentsession-script FILE` (below). -* `--agentsession-think-scale` — multiplier on every think gap; 0.5 makes the - fleet twice as chatty, 4.0 models slow reasoning models (default 1.0). Each - gap gets ±20% jitter so sessions don't move in lockstep. -* `--resume-mode implicit|explicit` — implicit (default) lets the parked - first request wake the actor; explicit issues ResumeActor before traffic. -* `--lifecycle-mode sync|async|none` — synchronous suspend (`sync`, - uploads the snapshot before returning), asynchronous suspend (`async`, - default, uploads the snapshot in the background), or `none` to keep actors - resident in memory between steps without hibernating. -* `--agentsession-snapshot-steps` — comma-separated 1-based step numbers after - which the actor is hibernated (for example, `3,10,20,30,40,50,61`); between - all other steps the actor stays resident. -* `--agentsession-phase-in` — seconds over which session starts are spread: - each user waits a uniform random share of it before creating its actor - (default 0, every session starts at once). Locust's ramp spreads the fleet - over seconds and every session begins at step 1, so without it the - sessions reach each later step together; a step that dwells longer than - the ramp then asks every worker for more slots at once than it has. Set - it to a few think gaps for density runs. -* `--agentsession-resume-retry-budget` — seconds a step keeps retrying a - wake the fleet had no room for (ResumeActor `ResourceExhausted` or - `Unavailable`, router 503/504/429) with 50ms-to-1s backoff before giving - the step up until its next think gap (default 60; `0` gives the wake one - attempt). Each attempt is its own - `ResumeActor` or `WakeFirstTouch` sample; the whole wait lands in - `TimeToWake`. - -#### Writing an agent-session script - -A script is a named step list with a memory floor: - -```yaml -name: coding-session -min_actor_memory: 1Gi -steps: - - name: 01_read_task - agent: Boots, reads the task prompt, loads its context window - think: 2s - ops: - - fill_ram: {key: agent_context, size: 32Mi} - - ping: {} - - name: 02_clone_repo - agent: git clone of the target repository - think: 3s - ops: - - ingest: {key: repo_tarball, size: 16Mi} - - burn_cpu: {millis: 500, parallel: 1} -``` - -`think` is the suspended gap before the step (a Go duration, scaled by -`--agentsession-think-scale`); `name` keys the `Step_` stats row. -Sizes are Kubernetes quantities. Each op is one glutton RPC: - -| Op | Arguments | Sandbox effect | -|---|---|---| -| `ingest` | `key`, `size` | bytes cross the network from the driver and land in a file: a download | -| `burn_cpu` | `millis`, `parallel` (default 1) | spins `parallel` goroutines for `millis` of wall clock | -| `write_disk` | `key`, `size` | writes locally generated bytes to a file | -| `read_disk_digest` | `key` | reads and hashes a file; nothing ships back | -| `read_disk_data` | `key` | reads a file and returns its bytes: disk read plus egress | -| `fill_ram` | `key`, `size` | allocates a resident RAM array | -| `churn_ram` | `key`, `size` | re-randomizes part of an array in place, dirtying pages | -| `walk_ram` | `key` | touches one byte per page: demand-paging cost after a resume | -| `ping` | none | a minimal round trip through the router | -| `dwell` | `millis` | no request: the actor stays resident and idle, as a gateway does during a model round trip or a typing gap | - -Loading is strict: unknown op kinds or fields, an argument a kind does not -take, a read of a file nothing wrote, a walk of an array nothing filled, a -duplicate step name, or a `min_actor_memory` below the declared RAM plus -disk all fail before any actor is created. Built-in variants are checked by -`TestEmbeddedScriptsAreValid`, so a broken file cannot merge. - -#### Built-in script variants - -`coding-session` (default) is a coding agent working one task: 20 steps of -build, test and edit work, each preceded by a few seconds of LLM thinking -spent suspended; about 40 CPU-seconds of sandbox work per lap. - -`personal-assistant` is an always-on assistant of the OpenClaw or Hermes -Agent kind: one lap is one day in real time, with about 30 user messages -in 7 clusters, a heartbeat turn every 30 minutes while the user is awake, -hourly system-event crons, a morning briefing, model-catalog refreshes, -one context compaction and a nightly memory sweep (61 wakes, about 65 -CPU-seconds and 150 MiB written per lap, a 640Mi resident heap that grows -to about 960Mi). Each turn reads its session store, ships a 50 to 110 KiB -context out through the router, dwells for the model round trip, and -appends to its store, sized from an strace profile of OpenClaw; the -script's header comment lists the figures. Its think gaps are the real -idle gaps, so run it with a fractional think scale: -`--agentsession-script personal-assistant --agentsession-think-scale 0.02` -plays a day in about 30 minutes. Because every heartbeat and cron is a -wake, the script also shows what an always-on agent costs a system that -suspends it between events: an internal timer cannot fire in a suspended -sandbox, so each of those turns needs an external wake. - -`personal-assistant-light` is the same day with the RAM arrays scaled -down to 28Mi and everything else unchanged (files, reads, CPU, dwell and -the 61 wakes), for runs that want the always-on wake pattern with a small -memory image: about 40 to 95 MB per pause under gVisor, and under microVM -about 100 MB of guest before the script writes anything, bounded by the -actor's memory limit. It fits a 256Mi actor. Run it with the same -fractional think scale. - -To run a script of your own without rebuilding anything, hand it to the -locust deploy: - -```sh -./benchmarking/locust/deploy.sh --deploy --user-class agentsession --agentsession-script ./my-session.yaml -``` - -The script validates the file locally first (the same check the worker -runs, via `boomer-worker --check-agentsession-script`), uploads it as the -`agentsession-script` ConfigMap, mounts it into the boomer workers at -`/etc/agentsession/script.yaml`, and points the master's -`--agentsession-script-file` default at that path. Workers log the loaded -script's name, step count, and declared budgets on their first iteration. -To go back to a built-in variant, redeploy without the flag. - -#### Agent-Session Reported Metrics - -* `WakeFirstTouch`: latency of a dedicated ping sent before each step's ops — - in implicit mode that ping is what triggers the parked wake, so this row - **is** the user-visible wake latency, unpolluted by the step's own work. -* `TimeToWake`: the whole wake of a parked actor, from the first - `ResumeActor` attempt (in implicit mode, the first wake ping) to the ping - that found the actor serving, including every retry while the fleet had no - room. In implicit mode a first-attempt wake is the `WakeFirstTouch` sample; - in explicit mode it is `WakeFirstTouch` plus `ResumeActor`. A failed row is - a wake that failed: a step given up after - `--agentsession-resume-retry-budget`, or one that failed at once with an - error that is not a capacity error, such as a crashed actor or a router - 404 or 500. -* `Step_` (e.g. `Step_06_first_build`): wall time of that step's ops, - think gap excluded. -* `SuspendActor` / `ResumeActor` / `CreateActor` / `DeleteActor`: control-plane - lifecycle latencies. - -### Spawn Benchmark - -The Spawn benchmark creates a batch of actors once and measures how long each -actor takes from creation to its first answered ping, and how long the whole -batch takes. `tests.yaml` runs it as `spawn_smoke_10_actors` with -`shapes/spawn_shape.py`, which holds one user and ends the run once -`TimeToAllReady` is recorded. - -Each boomer process creates one batch; extra users in the same process do -nothing. Actors are named `spawn--` and deleted when boomer exits. - -#### Spawn Configuration Knobs - -* `--total-actors`: Actors in the batch (default `100`). -* `--spawn-concurrency`: Actors created concurrently (default `1`). -* `--actor-deadline`: Per-actor timeout in seconds, covering create, resume and - first ping (default `120`). - -The web UI shows the same fields; `0` keeps the value boomer-worker started with. - -#### Spawn Reported Metrics - -* `CreateActor`, `ResumeActor`, `GluttonPing`: Latency of each call. -* `ActorTimeToReady`: Per actor, from its first `CreateActor` attempt to its - first successful ping. -* `TimeToReady_pct` (`k` = 10, 20, … 100): From batch start until `k`% of - the batch was ready. -* `TimeToAllReady`: From batch start until the last ready actor answered. - Actors that failed show up as `ActorTimeToReady` failures instead. -* `CrashCount`: Actors that crashed during resume. - -The `actors_per_*` ratios in `trial_summary` are wrong for this test: they -count users × `--actors-per-user`, not `--total-actors`. - -### ResumePaused Benchmark - -The ResumePaused benchmark drives ate-api-server's cold resume path at a set -rate. A cold resume here is a ResumeActor of a suspended actor: ateapi takes the -actor lease, binds a worker, and restores the actor from its snapshot. A -ResumeActor of a running actor does none of that. Each user owns one actor -from the `glutton` template. On its first iteration the user creates the actor -and makes it cold with a resume and a suspend; every later iteration resumes the -actor and suspends it again, so every measured resume is cold. - -The warm-up runs in the step that added the user, unpaced, and is recorded -under its own names. - -It sends no requests to the actors themselves, so it runs on the fake data -plane, where it measures what a cold resume costs ate-api-server and Postgres -without the router or the restore in the way. The suspends run at the resume -rate, so a capacity it finds is one of resume and suspend cycles, not of resumes -alone. - -Actors are named `rp-` and deleted when boomer exits. If the atespace -cannot be ensured, boomer exits at startup. An actor that fails to warm is -deleted, and its user creates another on its next iteration. When a step lowers -the user count, the actors of the stopped users stay suspended, unused, until -boomer exits. - -To run the cold resume tests in `automation/tests.yaml` from a workstation, -see [automation/ateapi-cold-resume.md](automation/ateapi-cold-resume.md). - -#### ResumePaused Configuration Knobs - -* `--target-rps`: Cold resumes per second the whole boomer process paces to - (default `0`, unpaced). A `--ladder` step ending in `@target_rps`, as in - `200:3m@1000`, changes it as the step starts. Keep the users at or above the - rate times the p99 of one resume plus one suspend in seconds, or the step falls - short of its rate. - -The web UI shows the same field. - -Trim the start of each ladder step. A rate change reaches boomer at its next -poll of the master, within 10 s, and a step that adds users carries their -warm-up until they have all joined at the spawn rate. - -#### ResumePaused Reported Metrics - -* `ResumeActorCold`: Latency of each measured resume. A resume that finds its - actor already running fails, as no cold resume happened. -* `SuspendActorAsync`: Latency of the async SuspendActor that makes the actor - cold again. -* `SuspendActorAsyncRecovery`: After a failed cycle, the async SuspendActor that - makes the actor cold for the next one. It is load on the server, so it is - recorded, but apart from `SuspendActorAsync`. A recovery suspend that finds - the actor already suspended counts as a success. -* `CreateActor`, `ResumeActorWarmup`, `SuspendActorAsyncWarmup`: A user's - warm-up of its actor, in the step that added the user. - -#### Analyzing a ResumePaused Ladder Run - -`stats.csv` and the other rows of `stats.jsonl` cover a whole run, so they -blend every step of a ladder. After a ladder run the runner adds one -`stats.jsonl` row per step, with `metric` `ladder_step_001`, -`ladder_step_002`, and so on ([`locust/ladder_steps.py`](locust/ladder_steps.py)). -It reads each step's start and end from the run's log and queries the -benchmarking Prometheus over the step's steady window, from 30 s after the -step starts to 5 s before it ends (`window_start` and `window_end`). A row -holds the cold resume and pause rates, failures by operation -(`failures_`), client and server latency, the CPU and memory of -ate-api-server, Postgres and fake-atelet, and Postgres's connections, -commits and waits. - -The runner logs and skips a step with no end in the log and a step too short -to leave a window after the trim. A failed query adds no step rows at all -rather than a ladder with gaps, and leaves the run's other results as they -are. Read a step's figures against its `target_rps`: a `resume_rps` below it -means the step fell short of its rate. - -To pull the step rows out of a run's results: - -```bash -RUN=/runs//run_date=/run_ts=/run_tag= -gcloud storage cat "$RUN/stats.jsonl" | grep '"metric": "ladder_step_' -``` - -`` is the orchestrator's `--dest`, and its log has the full `RUN` path -on each `Uploaded ... -> gs://...` line. - -### Isolate Mode - -A real cold resume is limited by the node's restore capacity, which hides the -limits of ate-api-server and Postgres. Isolate mode takes the nodes out by -installing substrate with a fake data plane: - -```bash -go run ./cmd/ate-setup deploy ate-system --fake-data-plane -./benchmarking/workloads/deploy.sh --deploy --worker-count 200 \ - --fake-atelet-env FAKE_ATELET_DELAY=500ms -``` - -`deploy.sh` deploys the usual WorkerPool, but the install serves it with two -fakes from `cmd/benchmarking/isolate/` instead of worker pods: - -* `fake-workersync` stands in for ate-controller's WorkerPool controller and - worker syncer. It honors each pool's `spec.replicas` with fake Workers, none - backed by a pod, round robin on the nodes that run fake-atelet, and scales them - up and down with the pool, draining a Worker before deleting it. Each Worker - reports what a real worker would: cpu and memory from the pool's - `template.resources.limits`, or the node's allocatable without one, and 1000 - actors, ateworker's default, unless the pool's `ate.dev/fake-max-actors` - annotation says otherwise. It writes the pool's status, so - `kubectl get workerpool` and `kubectl scale workerpool` work. A node keeps - its Workers for `--node-grace`, five minutes by default, after its - fake-atelet pod goes, so a DaemonSet rollout does not move them; past that - they move to a node that still runs fake-atelet. -* `fake-atelet` takes atelet's place in its DaemonSet, on every node atelet - would run on, and answers every - lifecycle call with success after its `--delay`, without running or - saving any workload. It writes one placeholder object per snapshot, because - ate-api-server refuses to tag an empty one. Like ateworker, it mints the actor's - certificate from ate-api-server on every run and restore that names an - egress gateway, which the default install does; the mint is ate-api-server's - time, so it adds to `--delay`, the data plane's share of the call. - It also relays fake-workersync's capacity reports, as atelet relays ateworker's: - ate-api-server accepts a report only from the atelet on the Worker's node. - The relay admits only fake-workersync's pod identity. fake-atelet remembers - the epoch each Worker registers with and reports it from every run and - restore, as a worker process does; ate-api-server commits a resumed actor - only under its Worker's current epoch. It keeps the epochs in memory only, - and fake-workersync registers a Worker once, so a fake-atelet that restarts - after its Workers registered, in a DaemonSet rollout for example, fails - every resume on those Workers with Unavailable until they are replaced, - such as by redeploying the benchmark. - -The install also scales ate-controller to zero, whose worker syncer would -delete the fake Workers. Its other controllers stop with it: no NetworkPolicy -is made for a WorkerPool, and the egress MITM trust bundle is not derived. -Fake actors need neither. fake-atelet takes the storage settings of atelet's -objectstore-plugin sidecar, whose writes it takes over, so its snapshot -placeholders go to the storage the install points atelet at. The install -refuses a cluster whose WorkerPools still have worker pods, which would -outlive ate-controller. A normal `ate-setup deploy ate-system` -puts the real data plane back, and `delete ate-system` removes the fakes -either way. Never use the mode on a -cluster that serves real actors: actors placed on fake Workers do not exist. - -The installer only chooses the fake data plane; the benchmark tunes it. -fake-atelet reads each tuning flag from a `FAKE_ATELET_` variable in the -optional `fake-atelet-tuning` ConfigMap, which no install creates or resets: -`FAKE_ATELET_DELAY` sets `--delay`, `FAKE_ATELET_DELAY_RESTORE` overrides it -for Restore alone, and `FAKE_ATELET_MINT_ACTOR_CERTIFICATE=false` skips the -certificate mint. `deploy.sh --fake-atelet-env NAME=VALUE`, repeatable, writes -that ConfigMap to exactly the given entries and restarts fake-atelet when it -changes, before the WorkerPool is applied. - -Nothing answers behind the router, so the mode suits only a user class that -sends the actors no requests, such as `ate_api.py`, which creates, gets, -resumes and suspends actors. Every boomer user class pings its actors through -the router. In `tests.yaml`, an `isolate` block selects the mode, and its -keys become `deploy.sh --fake-atelet-env` entries: `delay` sets -`FAKE_ATELET_DELAY`, and so on. - -### Viewing Traces -You must have enabled otel tracing for your cluster to view traces. - -You can find trace IDs by viewing the `logs` tab in the Locust UI - -## Running at scale - -A dense run meets limits the demo path never reaches. The ones met so far, and -what to set before the run: - -* **Reaching the API and the router from outside the cluster.** The Locust - and boomer workers in this document run in the cluster and reach the - Services directly. A load generator outside it must not go through - `kubectl port-forward`: the forward resets under load and takes every - in-flight resume and pause down with it. Expose `svc/api` and - `svc/atenet-router` through a NodePort or LoadBalancer Service instead. - The port-forwards in this document are for the UIs only. -* **PostgreSQL connection slots.** Each ateapi replica opens up to - `pool_max_conns` connections in its read/write pool plus 5 for its owner - and watch pools. Unset, pgx sizes the pool at the node's CPU count (at - least 4), not the container's limit, so on a large node one replica can - take most of PostgreSQL's default `max_connections` of 100. Set it with - `--ateapi-postgres-pool-max-conns` on ate-setup and keep - `replicas x (pool max + 5)` under `max_connections` with slack for `psql` - and migrations (see - [connection sizing](../docs/dev/cloud-sql-scaling-guide.md#connection-sizing-for-a-target-throughput)). - To raise `max_connections`, add a `-c` argument to the StatefulSet in - `manifests/ate-install/postgres/postgres.yaml`, or install with - `--cluster-size size10`, which sets 300 and pins the pool at 64 but sizes - PostgreSQL for a dedicated node. When the slots run out, requests fail with - `INTERNAL` and `remaining connection slots are reserved`. -* **PostgreSQL probes.** Under load the probes miss their default 1 s - timeout. Three misses of the liveness probe (`pg_isready`) restart - postgres mid-run; three misses of the readiness probe (`nc`) drop the pod - from the Service, and ateapi loses its connections without a restart. Give - both `timeoutSeconds: 5` and `failureThreshold: 6`, and give the postgres - container CPU and memory requests sized for the run: on a saturated node - the request is its share of the CPU, and a starved postgres is what misses - the probes. -* **Parking budget.** `--parked-request-budget` (default `5s`) has to cover a - pause-and-resume cycle under load, tens of seconds, times the - oversubscription (live actors per worker slot); at 2x that is minutes. Watch - `atenet.router.parking.wait.duration{outcome="budget_exhausted"}`. See - [docs/request-parking.md](../docs/request-parking.md). -* **Synchronized bursts.** Clients that step in lockstep send one request per - actor at the same instant. Above the ext_proc circuit breaker Envoy answers - `503 ... reset reason: overflow` and the router never sees it. Jitter the - clients, or size `--extproc-max-requests`. -* **Kernel neighbor table.** Every sandbox has a link-local address, and - `net.ipv4.neigh.default.gc_thresh3` is global across network namespaces. - Above about a thousand sandboxes per host, raise the thresholds - (`gc_thresh1/2/3` = 4096/16384/32768). The symptom is an idle host with - `connection timed out` to its sandboxes and `arp_cache: neighbor table - overflow!` in `dmesg`. -* **Restores under full load.** A node's restore throughput falls as its - sandboxes burn CPU, and a restore wave onto a saturated node makes the - wakeup probe time out: the actor is crashed. Substrate does not steer - restores away from busy nodes, so keep CPU headroom instead: fewer resident - sandboxes per node, and admission (creates per second) paced to the restore - rate. -* **The kubelet.** Hundreds of sandboxes per node starve its heartbeat; after - the 300 s toleration the taint manager evicts every pod on the node. - Reserve CPU for it (`systemReserved`, `kubeReserved`) before going above a - few hundred sandboxes per node, and watch node `Ready`. - -Besides the Locust statistics, watch `ate.actor.crashes` (a crash is a finding, -not noise), `atenet.router.parking.*`, atelet restore and checkpoint p99, -`pg_stat_activity` against `max_connections`, node conditions, and `dmesg` for -OOM and neighbor-table lines. ateapi logs every RPC at INFO and its container -log rotates at 10 MiB, so read lifecycle events from the collector rather than -from `kubectl logs`. - -## Benchmark output files - -A run writes the following to `--dest`. Each run produces them fresh; none of -them are checked into the repository. - -* `status.json`: `locust_exit_code` and `stats_generated`. Deliberately just - those two keys, because it is what CI orchestration reads to decide whether a - trial ran at all. -* `stats.csv`, `stats_history.csv`, `failures.csv`, `exceptions.csv`: Locust's - own CSV output. `stats_history.csv` has a row per operation every second as - well as the `Aggregated` row, so each operation's latency can be charted - over the run; filter to `Aggregated` for run-wide numbers. -* `logs.txt`, `traces.txt`: the runner log, and the trace IDs seen during the run. -* `stats.jsonl`: one JSON object per line, one per metric. Every row carries - the same five keys: `timestamp`, `tag`, `test_name`, `metric`, and a flat - `measurements` map holding that metric's numbers. The `phase_*` rows and - the `phase_breakdown_summary` row are the suspend/resume phase percentiles - read from the node logs (see - [analysis/README.md](analysis/README.md#automated-runs)). -* `server_summary.json`: server-side telemetry harvested from Prometheus, - including the per-sample bin-packing timeseries. - -### Density frontiers - -With cluster discovery enabled, `stats.jsonl` gains a `trial_summary` row -describing how densely actors are packed onto the hardware. Its `measurements` -map holds the raw facts and the derived numbers side by side. - -* `machine_type`, `node_count`, `allocatable_cores`, `allocatable_ram_gb` - (GiB), `worker_pod_count`: the measured facts, before any arithmetic. - Capacity covers the nodes the worker pods are running on rather than the - whole cluster, so a separate infrastructure pool is not counted. They are - recorded so the ratios below can be re-derived later, or recomputed against - a different denominator. -* `kubernetes_version`: the worker nodes' Kubernetes version (kubelet version). -* `worker_actor_capacity`, `worker_memory_limit_gb` (GiB): the capacity each - worker pod declares, its ateworker `--max-actors` (1000 when unset) and its - container memory limit (null when unset, as the node bounds it instead). - `sandbox_class`: `gvisor` or `microvm`, from the worker image. A mixed pool - gives a sorted comma-joined list for each. An unreadable value is skipped - without affecting the others, and logged once. -* `multi_actor_worker`: `true` when `actors_per_pod_p50` is above 1. -* `run_start`, `run_end`: Unix seconds bracketing the run, for finding its - server-side metrics after the cluster is deleted. -* `metadata_`: one per key of the free-form `--metadata ''` object, - for facts only the caller knows. Non-string values are stored as JSON, and - null values are skipped. The orchestrator passes `cluster_name`, - `cluster_location` and `project_id` from the target cluster, plus any keys in - the `BENCHMARK_METADATA` environment variable (a JSON object) and in the - test's own `--metadata` flag, whose keys win on conflict. -* `actors_per_node`, `actors_per_vcpu`, `actors_per_gb_ram`: the most actors - Locust reported running, over the matching capacity. The `-u` flag only - stands in when no sample was read. -* `actors_per_pod_p50`, `actors_per_pod_p90`, `actors_per_pod_p99`: actors per - worker pod across the run. Reported as a distribution rather than one - average, and it spans ramp-up too, because a custom load shape has no - single user count to call steady. -* `aggregate_failure_ratio`: failures over requests for the run. -* `_failure_ratio`: the same ratio for every operation Locust - reported, so each test carries its own names through. The operation name is - lowercased with underscores, so `DurDirWrite` becomes - `dur_dir_write_failure_ratio`. A key is absent when the test has no such - row, and null when the row ran no requests. - -The six `actors_per_*` ratios rest on three assumptions. Read them before -comparing numbers across runs: - -* **Actors are derived, not counted.** Locust only sees virtual users, so the - numerator is the peak user count times `--actors-per-user`. No server-side - gauge counts resident actors: `ate.actor.stats.sampled_actors` drops any - actor without a live resource measurement, so suspended ones fall out. It - is recorded as `active_actors` in `server_summary.json`, a running-actor - count rather than a resident one. -* **The denominators are read once, after the run.** A cluster that autoscaled - mid-run is measured at its final size, so the ratio pairs a peak from one - moment with a capacity from another. The same holds for every worker and - node fact above, including `kubernetes_version` and the worker capacity, and - so for `multi_actor_worker`, which is derived from `actors_per_pod`. -* **The peak assumes every actor is alive at once.** A workload that creates - and deletes actors as it goes never holds them all at the same time, so its - real density is lower than reported. - -### Server ground truth - -With a reachable Prometheus, `server_summary.json` records what the server -actually did, independent of what the load generator reported. - -* `cluster_packing`: workers holding at least one actor (`partial` or - `at_capacity`) over the pool size at each sample (the sum of all - `ate_workerpool_workers` states), as a percentile `summary` plus the - per-sample `timeseries` it was computed from. -* `node_psi.cpu_stall_pct`, `mem_stall_pct`, `io_stall_pct`: kernel pressure - stall percentages on the nodes hosting a worker pod, with every node's - samples pooled. `node_psi.nodes` is how many nodes were seen. -* `pod_psi.cpu_stall_pct`, `mem_stall_pct`, `io_stall_pct`: the same for each - worker pod's cgroup slice, which also holds its gVisor sandbox, with every - pod's samples pooled. `pod_psi.pods` is how many pods were seen. -* `snapshots.size_p50_mb` through `size_p99_mb`: actor memory image sizes. -* `snapshots.size_avg_mb`: mean memory image size. -* `snapshots.restore_p50_s` through `restore_p99_s`, `restore_mean_s`, and the - same for `checkpoint_*`: how long the atelet took to resume or suspend an - actor (`ate_actor_{restore,checkpoint}_duration`, `total` phase). Every - snapshot kind counts, as in the sizes: the actor's own snapshot and a - template's first start from its golden snapshot. The buckets reach 60s, so a - slower operation reads as 60s. -* `snapshots.checkpoints_in_window`, `checkpoints_cumulative`: checkpoint - volume over the steady-state window, and since the atelet started. -* `snapshots.checkpoint_mb_s`: bytes written per second spent checkpointing, - not per second of wall clock. -* `active_actors`: running actors (`ate_actor_stats_sampled_actors`), as a - percentile `summary` of the cluster-wide count, `atelets` seen, and a - `timeseries` every 10s over the whole run (ramp-up included). Each point has - the cluster-wide count and `active_atelets` hosting an actor. As the - assumptions above note, the gauge drops actors without a live measurement, - so suspended ones fall out and the count dips while actors are suspended. - The atelet stops exporting when a node has no running actor, so at a step - where at least one atelet reported, a missing one counts as 0. A step where - none reported is skipped, not 0, since a failed scrape looks the same; this - mostly hits 1-user runs. If the metric never appeared during the run, every - field is `null`. `atelets` counts atelet processes seen in the run, not - nodes: the collector keeps an exited atelet's last value for about 5 - minutes, so a restart can briefly count twice, and an atelet scraped both - directly and through a collector counts twice. -* `working_set.node`, `worker`, `atelet`: memory working set in GiB from - cAdvisor on the nodes hosting a worker pod, per worker pod (every actor on it - included), and per atelet on those nodes. Each has a percentile `summary` - (plus `max`) over the steady-state window, `count` of series seen in it, and - a `timeseries` every 10s over the whole run with the sum (`total_gb`) and - largest (`max_gb`). A spike shorter than 10s can be missed. -* `working_set.actor`, `per_actor`: the same shape for the actors' own working - set as the atelet reports it (`ate_actor_stats_memory_working_set_bytes`), - per atelet with every template summed, and the average per measured actor - (`ate_actor_stats_sampled_actors`) on each atelet; `count` is atelet - processes, keyed like `active_actors`. As `per_actor` is an average per - atelet, its `max` is the highest atelet average, not the largest actor, and - its `total_gb` has no meaning. A gVisor actor is read from its sandbox cgroup - on the host, a microVM actor from inside the guest. As with `active_actors`, - suspended actors fall out, but through the telemetry-meter a series that - stops being exported keeps its last value for up to 5m. Missing samples are - skipped, not counted as 0. The atelet samples every - `--actor-stats-poll-interval` (default 1m), so these move in steps of a - minute or more. `actor` matches one worker pod's actors only when the node - runs one worker pod. -* `oom_events.node`, `pod`: OOM kills over the run, from cAdvisor's - `container_oom_events_total`. `node` counts only global OOMs. `pod` counts - kills inside a worker pod (actor sandboxes and the worker container) and is - best effort: a non-zero value is real, but a kill whose cgroup is gone before - the scrape is missed. A node-allocatable OOM (`oom_memcg=/kubepods.slice`) - isn't global, so it isn't in `node`, and `pod` usually misses it; the kubelet - evictions that often come with it show in `evicted`. OOMs inside a microVM - guest are invisible and show up as task failures. Memory PSI (`node_psi` / - `pod_psi`) is the early warning. -* `oom_events.worker_restarts`, `worker_oomkilled_pods`, `evicted`, - `worker_pods_lost`: from the Kubernetes API, read before and after the run. - Restarts on a pod deleted during the run are lost. `worker_oomkilled_pods` - counts pods whose last restart was `OOMKilled`, not kills; a group kill of - the whole worker (`memory.oom.group=1`) shows here. `evicted` is kubelet - eviction under node pressure (usually memory), not a kernel OOM kill. It also - counts replacement pods the kubelet refused to start while the node was under - pressure, as they get the same `Evicted` reason. It comes from `Evicted` - events, which last about 1h, so a run longer than that can undercount. - `worker_pods_lost` is pods gone by the end for any reason (eviction, - scale-down, recycling), not an OOM count; a pod created and deleted within - the run isn't counted. A node OOM can also show as a worker `OOMKilled` - restart, so never add the fields. A field is `null` when its Kubernetes read - fails, and all are with `--no-cluster-facts`. - -Every distribution reports p50, p90, p95 and p99 over the steady-state window. -The steady-state window runs from the first to the last Locust sample at 90% or -more of the run's peak user count, so ramp-up and teardown are left out. Under -a step-ladder load shape it covers only the top step. -Counts, means and `checkpoint_mb_s` come from the growth of the histogram's sum -and count across the window, and the size, restore and checkpoint percentiles -are interpolated inside its buckets. PSI is a one-minute rolling average read -every 10s. - -The atelet exports before Prometheus scrapes it, so the harvest waits -`--atelet-lag-s` seconds (default 70, enough for the OTel SDK's 60s default -export and a 10s scrape) and reads the snapshot, `active_actors` and actor -working set windows half that late, stamping `active_actors` and actor -working set points back by the same amount. The window ends at the last -full-load sample, so teardown suspends are left out. - -`metadata.start_ts` and `end_ts` bound the whole run, which the packing, -`active_actors` and `working_set` timeseries cover. `steady_start_ts` and -`steady_end_ts` bound the steady-state window. A flat subset of the numbers -also goes into a `server_summary` row in `stats.jsonl`. - -Neither the Kubernetes API nor Prometheus is required. If either is unreachable, -or discovery was skipped, the affected fields are written as `null` and the run -still succeeds. A `null` means the value was not measured. It never means zero. - -## Suspend/resume phase breakdown - -`SuspendActor` and `ResumeActor` latency can be attributed to the phases -inside them (object-storage transfer, hypervisor snapshot/restore, rootfs -assembly, ...) from the structured log records atelet and ateworker-microvm -emit. `analysis/collect_logs.sh` dumps the node logs of a run and -`analysis/phase_report.py` aggregates them into per-phase percentiles and -waterfalls of the slowest operations. See -[analysis/README.md](analysis/README.md). - -## Optional: Prometheus + Grafana - -Locust provides graphs, statistics, etc. via the UI. However, you -can install Prometheus/Grafana if you want richer details or -the ability to perform deeper analysis. Skip this section if -you're only using the Locust web UI. - -```bash -kubectl apply -f benchmarking/monitoring.yaml -``` - -Once installed: - -* Run `kubectl port-forward svc/grafana -n benchmarking 3000:3000` -* Visit `http://localhost:3000` in your browser. - -## Development - -### Generating gRPC Python clients - -The clients are not checked in. The locust and nighthawk-ingress images -generate them at build time, and `hack/verify/python-protos.sh` compiles them -on every PR, so a proto change needs no extra step. For local use, such as -editor completion, run `benchmarking/locust/codegen/generate.sh`. It manages -its own virtual environment under `locust/codegen/venv`. - -### Unit tests - -`locust/unit_tests` covers the runner's helpers and needs no cluster. From the -repository root: - -```bash -python3 -m unittest discover -s benchmarking/locust/unit_tests -``` - -`analysis/test_phase_report.py` covers the phase-log report's parser and -aggregation: - -```bash -python3 -m unittest discover -s benchmarking/analysis -``` +Moved to [docs/guides/benchmarking.md](../docs/guides/benchmarking.md). diff --git a/benchmarking/automation/README.md b/benchmarking/automation/README.md index 30a9dfe83c..ad54f51d67 100644 --- a/benchmarking/automation/README.md +++ b/benchmarking/automation/README.md @@ -27,7 +27,7 @@ To run the cold resume tests from your workstation instead, see images via `ko` as part of their deploy steps — there's no separate image build step). The install sets `ATE_API_AUTHZ_BOOTSTRAP_OWNERS` to the `ate-client` and `benchmarking/benchmark-runner` ServiceAccounts so the - runner's calls pass [authorization](../../docs/authorization.md), and sets + runner's calls pass [authorization](../../docs/guides/authorization.md), and sets `ATE_INGRESS_ALLOWED_SPIFFE_IDS` to `benchmarking/benchmark-runner` for the Envoy ingress router. For a `microvm` test the orchestrator also runs `hack/install-microvm-deps.sh --install` between the two, which diff --git a/cmd/ate-setup/commands.md b/cmd/ate-setup/commands.md index 8d090d24d6..62ddeb12f1 100644 --- a/cmd/ate-setup/commands.md +++ b/cmd/ate-setup/commands.md @@ -13,7 +13,7 @@ go run ./cmd/ate-setup [global flags] [flags] These configure every command, and may appear before or after it, except `--version`, which takes no command. -[`docs/operator-install.md`](../../docs/operator-install.md#settings) lists +[`docs/guides/operator-install.md`](../../docs/guides/operator-install.md#settings) lists every setting with its environment variable. | Flag | Notes | @@ -29,7 +29,7 @@ every setting with its environment variable. | `--ingress-allowed-spiffe-ids IDS` | Comma-separated SPIFFE IDs (exact match) of the clients the ingress router lets reach actors; any listed client may reach any actor. Unset, no client may reach actors through the router. Both dataplanes require a Pod Identity CA client certificate on HTTPS and TLS CONNECT. The SPIFFE URI SAN allowlist may be checked during or after TLS, including after CONNECT acceptance, but unlisted clients must be denied before actor routing. The podidentity signer issues `spiffe://cluster.local/ns//sa/`. `ATE_INGRESS_ALLOWED_SPIFFE_IDS` when the flag is absent | | `--egress-gateway-extra-spiffe-ids IDS` | Comma-separated SPIFFE IDs that hold ate-api-server's `egress_gateway` role next to atenet-egress, such as a credential provider that mints actor JWTs. ate-setup passes atenet-egress's ID and then each listed ID as repeated `--egress-gateway-spiffe-id` flags; unset passes none, leaving ate-api-server's default. `ATE_EGRESS_GATEWAY_EXTRA_SPIFFE_IDS` when the flag is absent | | `--experimental-additional-egress-extproc-service NS/SVC:PORT` | External processor authorization filter | -| `--credential-provider JSON` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. See [`docs/egress-credential-injection.md`](../../docs/egress-credential-injection.md). `ATE_CREDENTIAL_PROVIDER` when the flag is absent | +| `--credential-provider JSON` | Required by `deploy ate-system` and `deploy atenet`. A JSON object: `{"name":"k8s.io"}` deploys and uses the bundled Kubernetes Secrets provider, with a NetworkPolicy that admits only the egress gateway; `{"enabled":false}` turns egress credential injection off; `{"name":"","address":":"}` uses a provider you deploy yourself. See [`docs/guides/egress-credential-injection.md`](../../docs/guides/egress-credential-injection.md). `ATE_CREDENTIAL_PROVIDER` when the flag is absent | | `--ateapi-log-level LEVEL` | ate-api-server log level: `debug`, `info`, `warn`, or `error`. Written to the `ate-api-config` ConfigMap on every deploy when set; unset keeps the installed level (`info` on a new install). The server picks up a change without restarting. `ATE_API_LOG_LEVEL` when the flag is absent | | `--otlp-endpoint URL` | Send control plane telemetry to `URL` instead of the cluster default (see [`benchmarking/telemetry/README.md`](../../benchmarking/telemetry/README.md)). `ATE_OTLP_ENDPOINT` when the flag is absent | | `--preview GATES` | Preview gates passed as `--preview` to every substrate component: comma-separated gate names, or `ALL` for all. `ATE_PREVIEW` or `config.preview` when the flag is absent. An explicit `--preview=''` turns them all off, even when `ATE_PREVIEW` is set | diff --git a/cmd/atenet/README.md b/cmd/atenet/README.md index 3abe9e1a0e..afbc58a6ee 100644 --- a/cmd/atenet/README.md +++ b/cmd/atenet/README.md @@ -39,7 +39,7 @@ likely be split in the future for better scalability.) requests finish instead of resetting. The whole sequence must fit within `terminationGracePeriodSeconds` (see the manifest comments). An upgrade rolls the router as an ordinary Deployment update (see - [docs/upgrade.md](../../docs/upgrade.md)). The drain lets each old pod's + [docs/guides/upgrade.md](../../docs/guides/upgrade.md)). The drain lets each old pod's requests finish, except one still open about 15 seconds after the drain starts, which is cut. diff --git a/cmd/atenet/internal/router/README.md b/cmd/atenet/internal/router/README.md index 06777c6db6..3150638ef0 100644 --- a/cmd/atenet/internal/router/README.md +++ b/cmd/atenet/internal/router/README.md @@ -15,7 +15,7 @@ Router has several responsibilities: * Parks requests whose actor cannot be served immediately due to transient worker-pool saturation, retrying the resume until the actor is routable or a bounded wait elapses, instead of failing fast. See - [docs/request-parking.md](../../../../docs/request-parking.md). + [docs/concepts/request-parking.md](../../../../docs/concepts/request-parking.md). * Drains gracefully on SIGTERM: flips `/readyz` so the Service stops sending new connections, waits out endpoint propagation (`--drain-delay`), drains the dataplane's established connections (Envoy only — driven over its admin API; @@ -105,7 +105,7 @@ rule, `egress_passthrough` under a `tls_passthrough` rule, and `denied`, which matches no chain and closes the connection, when neither covers the SNI. The passthrough chain resolves the SNI and relays the bytes to it on the dialed port; the address the actor dialed plays no part (see -`docs/network-egress.md`). +`docs/guides/network-egress.md`). Identity on the request legs is `dev.ate.actor.identity`, the actor's SPIFFE ID that the outer chain set from the verified peer certificate and shares with @@ -125,7 +125,7 @@ deleted policy becomes a deny. Every policy denial answers a fixed Credential injection (`replace_headers`) runs on both request legs: a header the request carries is replaced with the credential from the provider, and a request without it is forwarded unchanged. See -`docs/egress-credential-injection.md`. +`docs/guides/egress-credential-injection.md`. ## adding a dataplane attribute diff --git a/cmd/kubectl-ate/README.md b/cmd/kubectl-ate/README.md index 2c86821b69..6439371119 100644 --- a/cmd/kubectl-ate/README.md +++ b/cmd/kubectl-ate/README.md @@ -1,363 +1 @@ -# `kubectl-ate` - -A Kubernetes-native CLI plugin for managing Substrate Actor and Worker lifecycles. - -## Running the CLI - -There are two ways to run the tool, depending on whether you are developing locally or installing it permanently. - -### 1. Install as a native `kubectl` Plugin -You can use `go install` to compile the tool and place the binary directly into your Go bin directory (which should be in your `$PATH`). Because the source folder is named `kubectl-ate`, Kubernetes will automatically recognize the resulting binary! - -```bash -go install ./cmd/kubectl-ate -``` -You can now run it seamlessly anywhere as a native Kubernetes command: `kubectl ate `. - -### 2. Run directly from source (Development) -If you are testing changes to the codebase, you can bypass compilation and run the CLI directly from the source tree: - -```bash -go run ./cmd/kubectl-ate -``` - -## Connection & Auto Port-Forwarding -By default, `kubectl-ate` will automatically read your `~/.kube/config`, discover the `ate-api-server` pods in your cluster, and establish a temporary background port-forward tunnel to execute gRPC calls securely. - -If you prefer to route traffic directly (e.g., through a LoadBalancer or when running natively inside a cluster pod), simply provide the `--endpoint` flag to bypass the tunnel. - -## Tracing -The CLI supports on-demand tracing using the `--trace` flag. When enabled, the CLI will generate a trace ID and signal to the server that it wants the request to be traced. - -**Prerequisites:** - -1. The Google Cloud project must have the **Cloud Trace API** enabled. You can enable it using: -```bash -gcloud services enable cloudtrace.googleapis.com --project=PROJECT_ID -``` - -2. The GKE cluster must have **Managed OpenTelemetry** enabled. Clusters created by -`setup-gcp create cluster` or `setup-gcp bootstrap` always enable it. For a cluster that does -not have it, enable it with: - -```bash -gcloud beta container clusters update CLUSTER_NAME \ - --project=PROJECT_ID \ - --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS \ - --location=LOCATION -``` - -**Local (kind):** - -The kind overlay installed by `ate-setup --kind deploy ate-system` already provisions an in-cluster OpenTelemetry Collector and Jaeger in the `otel-system` namespace. No additional setup is required. - -Port-forward the Jaeger UI and invoke any command with `--trace`: -```bash -kubectl port-forward -n otel-system svc/jaeger 16686:16686 & -kubectl ate get actor my-counter-1 --trace -# open http://localhost:16686 and search for the most recent trace -``` - -## Global Flags -These flags can be appended to any command: - -| Flag | Short | Description | Default | -|---|---|---|---| -| `--kubeconfig` | | Path to your kubeconfig file | `~/.kube/config` | -| `--context` | | Name of the kubeconfig context to use | current context | -| `--endpoint` | | Manual gRPC endpoint override (e.g., `localhost:8080`) | | -| `--token-file` | | Path to a bearer token for ate-api authentication, or `-` for stdin | Kubernetes ServiceAccount token | -| `--output` | `-o` | Output format (`table`, `json`, `yaml`) | `table` | -| `--trace` | | Enable on-demand tracing for the request | `false` | -| `--verbose` | `-v` | Increase output verbosity; may be repeated or stacked (e.g., `-v`, `-vv`, `-vvvvv`) | `0` | - -Pass `--version` (`kubectl ate --version`) to print the CLI version and exit. - ---- - -## Command Reference & Examples - -### Getting Resources -List and inspect the state of actors and workers across the cluster. - -```bash -# List actors in one atespace; -a is shorthand for --atespace -kubectl ate get actors --atespace -kubectl ate get actors -a - -# List actors across all atespaces -kubectl ate get actors -A - -# Get a specific actor by name and output as raw YAML -kubectl ate get actor --atespace -o yaml - -# List all physical workers and see which actors are assigned to them -kubectl ate get workers - -# Filter workers by Kubernetes namespace, assigned-actor atespace, or -# worker pool labels (same flags as `top workers`) -kubectl ate get workers -n -kubectl ate get workers -a -kubectl ate get workers -l -``` - -> **Note:** `get actors` requires either `--atespace ` / `-a ` (one atespace) or `-A`/`--all-atespaces` (all atespaces) — there is no default atespace. Getting a single actor always requires `--atespace`/`-a`, since an actor is addressed by `(atespace, name)`. `-a` (lower-case) scopes to one atespace; `-A` (upper-case) spans all. - -> **Note:** Actors, workers, and actor templates are not Kubernetes CRDs — they live in the Substrate control plane's PostgreSQL database, not `etcd`. `kubectl get actor`, `kubectl get worker`, and `kubectl get actortemplate` will not return anything; only `kubectl ate get …` queries the control plane (see [Actor Templates](#actor-templates)). `kubectl get workerpool` *does* work, because pools are CRDs. - -#### `kubectl ate get actor` output columns - -| Column | Meaning | -|---|---| -| `ATESPACE` | The atespace the actor belongs to. Part of the actor's identity; folded into the storage key as `actor::`. | -| `NAME` | The actor's name. User-provided for application actors; UUID for the golden actor that each template materialises while building its golden tag. | -| `TEMPLATE` | The `ActorTemplate` the actor references, displayed as `/`. | -| `STATE` | Current lifecycle state (`ACTOR_STATE_RESUMING`, `ACTOR_STATE_RUNNING`, `ACTOR_STATE_SUSPENDING`, `ACTOR_STATE_SUSPENDED`, `ACTOR_STATE_CRASHED`, `ACTOR_STATE_DELETING`, `ACTOR_STATE_REVERTING`). | -| `WORKER POD` | The worker pod (namespace/name) currently hosting the actor. Empty while suspended. | -| `WORKER IP` | The pod IP of that worker. Empty while suspended. | -| `VERSION` | Monotonic integer that increments on every state transition (resume / suspend / checkpoint). Useful for distinguishing snapshots. | -| `AGE` | Time elapsed since the actor was created. | - -#### `kubectl ate get worker` output columns - -| Column | Meaning | -|---|---| -| `NAMESPACE` | The `WorkerPool` namespace. | -| `POOL` | The `WorkerPool` name. | -| `POD` | The worker pod name. | -| `STATUS` | `FREE` (idle, ready to receive an actor) or `ASSIGNED` (currently hosting an actor). | -| `ASSIGNED ACTOR` | If `STATUS=ASSIGNED`, the actor reference `/