Repository navigation
Sync upstream main through 17fa8586e - #128
Merged
Merged
Conversation
…ent-substrate#3472) A routed request to an actor in ACTOR_STATE_CRASHED parks in atenet-router for the whole 5 s budget, makes about 19 `ResumeActor` calls, holds one parking slot throughout, and then answers 503. The state never clears on its own, so the wait buys nothing. Fixes agent-substrate#3269 Part of agent-substrate#1562 `retryable` in `cmd/atenet/internal/router/ingress/resumer.go` drops `FailedPrecondition` from the park arm, so the request fails on the first attempt with the same 503 and the same body. `ResourceExhausted` and `Unavailable` still park, `Aborted` is still always retried, and nothing changes with parking disabled. No ateapi change. Behavior change: every `FailedPrecondition` from `ResumeActor` now fails fast, not only CRASHED. That covers DELETING, SUSPENDING and REVERTING, a missing template, a volume without a storage ID, a worker without a node name, and a failed `DetachActor`. Two situations behind SUSPENDING and REVERTING: - An operation in progress. `SuspendActor`, `RevertActor` and `DeleteActor` hold the actor lease for their whole run, so a resume during one gets `Aborted`, which still parks. Nothing changes here. - A failed operation. A failed suspend/revert that fails at any step, leaves the actor SUSPENDING/REVERTING with no lease held. The resume then gets `FailedPrecondition` and now fails fast. A retried suspend or revert is what moves the actor on, never the resume retry. The status code and the body are unchanged. The e2e `TestRoutedRequestToCrashedActorFailsFast` crashes an actor by deleting its worker pod and routes one request: 503 naming the state within 2 s, where a parked request would first wait out the 5 s budget. It skips on the AgentGateway lane, whose resume path is outside this repo. `docs/concepts/request-parking.md` now lists `FailedPrecondition` under fail fast. Verified with `go test -race -count=3 -shuffle=on` on the router package, `hack/verify-all.sh`, and the demo e2e suite on kind. AI use: AI-assisted under my direction; I reviewed the diff and ran the tests. - [x] Tests pass - [x] Docs updated Signed-off-by: Yuan Gao <ypgao@google.com>
…ected at admission (agent-substrate#3487) Part of agent-substrate#3226. We only checked the shape of capability names, so a typo passed admission. In `add` it failed at actor start, and on microVM kata-agent panics converting it ([trans.rs](https://github.com/kata-containers/kata-containers/blob/03cf2ec06a5d302f23af811744f6bcea49988bfb/src/libs/protocols/src/trans.rs#L40-L52)). In `drop` it was ignored, so `drop: [NET_BIND_SERVIC]` kept `NET_BIND_SERVICE`. Names are now checked against the 41 Linux capabilities, the same set runsc ([specutils.go](https://github.com/google/gvisor/blob/760b80785dc53311d4c87bd2de167525a99709c1/runsc/specutils/specutils.go#L510-L552)) and kata's oci-spec ([capability.rs](https://github.com/youki-dev/oci-spec-rs/blob/1cc1afb0e9a68403d8fb5bde1a3d05db968f41d3/src/runtime/capability.rs#L26-L520)) accept. atelet runs the same check on Run and Restore. AI use: AI-assisted, I reviewed it. Breaking change: a template with an unknown name in `drop` is now rejected, check `drop` lists for typos. Unknown names in `add` already failed at actor start. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
…strate#2092) Stacked on agent-substrate#1838. ## Summary Ports the microVM dependency setup (`hack/install-microvm-deps.sh` and `hack/microvm-assets/`) to Go, and folds microVM asset assembly, staging, and `SandboxConfig` management directly into `deploy ate-system`, `deploy sandboxconfig`, and `delete ate-system`. ## Changes - **Fold microVM setup into control-plane deploy/delete (`cmd/ate-setup/internal/steps/deploy.go`, `delete.go`, `benchmarks.go`)**: - Calls `prepareMicroVMDeps` (before cluster resources are modified) and `applyMicroVMSandboxConfig` inside `DeployAteSystem` (after applying the system bundle so Kind's in-cluster `rustfs` Deployment and `rustfs-bucket-init` Job exist before asset staging) and inside `DeploySandboxConfig`. - Adds `--sandbox-classes` (default `gvisor,microvm`) to `ate-setup deploy ate-system` and `ate-setup deploy sandboxconfig` (`--sandbox-classes=gvisor` opts out of staging microVM assets and applying `SandboxConfig/microvm`). - Releases `SandboxConfig` protection finalizers inside `DeleteAteSystem` before CRD teardown when no `WorkerPool` references the config. - Updates `DeployBenchmarks` / `DeleteBenchmarks` since `deploy ate-system` now installs both `gvisor-default` and `microvm` `SandboxConfig`s up front, and checks that `sandboxconfig/microvm` exists when `--sandbox-class=microvm`. - **Remove `hack/install-microvm-deps.sh` and `hack/microvm-assets/`**: - Deletes `hack/install-microvm-deps.sh` and the entire `hack/microvm-assets/` directory (`assemble.sh`, `stage-to-gcs.sh`, `stage-to-rustfs.sh`, and `README.md`). - Updates `hack/run-microvm-demo.sh`, `benchmarking/automation/orchestrator.py`, and `.github/workflows/pr-workflow.yaml` to rely on `deploy ate-system`. - **Native Go multi-arch microVM asset assembly and staging (`cmd/ate-setup/internal/steps/microvm.go`)**: - Streams and extracts the Kata `.tar.zst` archive (`vmlinux.container`, `kata-containers-ubuntu.img` as `rootfs.img`, `virtiofsd`) and Cloud Hypervisor binary in Go (`runtime.GOARCH` on Kind; both `amd64` and `arm64` on non-Kind clusters) into `bin/microvm-assets/<arch>/`. - Checks the SHA-256 digests of all four assets (`cloud-hypervisor`, `virtiofsd`, `vmlinux`, `rootfs.img`) against the digests pinned in `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` (both after assembly and when reusing a cached `.asset-versions` directory). - Stages assets by content digest under `microvm-assets/<arch>/<sha256>` in Kind's `rustfs` bucket or GKE's `<bucket>-assets` GCS bucket, then renders and applies `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl`. - **Docs, manifests & unit tests**: - Moves `sandboxconfig-microvm.yaml.tmpl` to `manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` with per-arch `microvm-assets/<arch>/<sha256>` URLs and removes stale version-number comments. - Folds the remaining asset-set and snapshot-location notes from `hack/microvm-assets/README.md` into `docs/dev/microvm-local.md` and updates `docs/reference/api-guide.md`, `docs/guides/operator-install.md`, `cmd/ate-setup/README.md`, `cmd/ate-setup/commands.md`, `cmd/ate-setup/differences.md`, `demos/counter/README.md`, and `hack/release/draft-release.sh`. - Adds unit tests in `cmd/ate-setup/internal/steps/microvm_test.go`, `cmd/ate-setup/internal/steps/benchmarks_test.go`, `cmd/ate-setup/internal/kube/wait_test.go`, and `cmd/ate-setup/internal/cmd/deploy_test.go`. ## Testing - Run `go test -race ./cmd/ate-setup/... ./internal/e2e/...` and `hack/verify-all.sh`. - Local Kind cluster verification (`go run ./cmd/ate-setup --kind`): - Verified `deploy ate-system --sandbox-classes=gvisor` deploys the control plane and installs only `SandboxConfig/gvisor-default`. - Verified `deploy ate-system` assembles the microVM guest assets, stages them into the in-cluster `rustfs` bucket, and applies `SandboxConfig/microvm` alongside `SandboxConfig/gvisor-default` (reusing cached assets via `.asset-versions` on subsequent runs). - Verified `delete all` removes `SandboxConfig/microvm` during control-plane teardown before deleting CRDs. - Deployed `deploy demo counter` and ran `./hack/run-e2e-kind.sh ./internal/e2e/suites/demo`. ## Folded in from open PRs Because this PR replaces `hack/microvm-assets/stage-to-rustfs.sh` with `stageMicroVMAssetsToRustfs`, it folds in the `stage-to-rustfs.sh` fixes from two open PRs: - agent-substrate#2905 (@dims): look up the `app=rustfs` pod's `spec.nodeName` so the S3 upload container runs in the node network namespace admitted by rustfs's NetworkPolicy on multi-node Kind clusters. - agent-substrate#1122 (@ygao-g): use `net.JoinHostPort(svc.Spec.ClusterIP, "9000")` for `AWS_ENDPOINT_URL` so IPv6 ClusterIPs are bracketed. AI Use: AI assisted implementation & unit testing, human reviewed, refined, and locally tested. --------- Signed-off-by: Noah West <westnoah@google.com> Co-authored-by: jybao <jybao@google.com>
…share a CPU architecture (agent-substrate#3458) Part of agent-substrate#3230 A `SNAPSHOT_FIDELITY_ROOTFS` restore reuses the container's rootfs upper layer, which can contain architecture-specific binaries written on the worker where the snapshot was captured. Until the scheduler filters by CPU architecture on fallback, every `WorkerPool` selected by an `ActorTemplate` must run on the same CPU architecture. Previously, this requirement was only mentioned under `Snapshot Storage Layout` (`sandboxRuntime`) in `docs/reference/api-guide.md`. Surface it where operators and users configure pools and templates: - `README.md`: add `Known limitations` under `Status and compatibility`. - `docs/reference/api-guide.md`: note how to pin and label `WorkerPool`s in mixed-architecture clusters under `WorkerPoolPodTemplate`, and note the single-architecture requirement on `ActorTemplate.workerSelector`. - `api/ateapi/v1/ateapi.proto` (`ateapi.pb.go`): document the requirement on `ActorTemplate.worker_selector`. Used an AI assistant to draft the documentation wording and reviewed it locally. - [x] Tests pass - [x] Docs updated Signed-off-by: Shrutiya <shrutiyam@google.com>
<!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Very small change on docs, when I was starting on quickstart I've found it tells me to jump directly to Fixes # <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I wrote, AI reviewed the wording <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [ ] Tests pass - [X] Docs updated Signed-off-by: Ricardo Pchevuzinske Katz <ricardo@let.kube.rip>
…strate#3476) Follow-up to agent-substrate#3429, from @aramase's review: agent-substrate#3429 (comment) When the pod can lend a microVM snapshot write all the memory it needs, the actor's cgroup has no `memory.high` during the write, so it can end with a lot of dirty page cache. Putting `memory.high` back reclaims what it can, but gives up on pages still dirty or under writeback: https://github.com/torvalds/linux/blob/v7.0/mm/memcontrol.c#L4409-L4429 The memory guard resumes as soon as the write returns, before teardown, and its overshoot check counted that page cache. So it could kill the paused VM and report the actor crashed for exceeding its memory limit. The overshoot check now leaves reclaimable page cache out too, and both checks take what's left from one `memory.stat` read. The kernel only flushes `memory.stat` past a threshold of updates, so it can trail `memory.current`, and subtracting one from the other could count fresh page cache as used: https://github.com/torvalds/linux/blob/v7.0/mm/memcontrol.c#L561-L564 AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
…trate#3456) Pass [`swiotlb=noforce`](https://docs.kernel.org/admin-guide/kernel-parameters.html) on the microVM guest kernel command line to skip the 64 MiB software bounce buffer ([`SWIOTLB`](https://docs.kernel.org/core-api/swiotlb.html)) reserved at boot on `> 3 GiB` VMs (see agent-substrate#3444). ### Benchmark The change itself is tiny, but it gives us solid improvements! Having 64 MiB less uncompressed data in `memory-ranges` shrinks snapshots and cuts lifecycle latencies. To confirm this, here is a comparison against `main` on a `4 GiB` [SWE-bench workload](https://github.com/agent-substrate/substrate/blob/main/docs/guides/benchmarking.md#sweperf-benchmark): - **Setup**: GKE 1.35, `c3-highmem-192-metal` (bare metal), `hyperdisk-balanced` 200 GB - **Workload**: [`sweperf-astropy-7336`](https://github.com/agent-substrate/substrate/blob/main/benchmarking/workloads/manifests/sweperf/generated/sweperf-astropy-7336-template.yaml.tmpl) (`2 vCPU / 4 GiB` RAM), cold start + 4 cycles of `Resume → Execute → Suspend` (median of 4 cycles): | Metric | `main` | With `swiotlb=noforce` | Delta | | :--- | :---: | :---: | :---: | | **Golden `memory-ranges` populated size** | `223 MiB` | **`158 MiB`** | **`-64 MiB` (`-29%`)** | | **Golden cold boot (`RunWorkload`)** | `491 ms` | **`416 ms`** | **`-75 ms` (`-15%`)** | | **Suspend (`async`, on node) — `Resume` / `Suspend`** | `360 ms` / `383 ms` | **`314 ms` / `352 ms`** | **`-46 ms` / `-31 ms`** | | **Suspend (`sync`, to GCS) — `Resume` / `Suspend`** | `365 ms` / `1843 ms` | **`341 ms` / `1652 ms`** | **`-24 ms` / `-191 ms`** | Fixes agent-substrate#3444 AI use: AI-assisted for the benchmark. - [x] Tests pass Signed-off-by: Jiayi Bao <jybao@google.com>
…k/run-microvm-demo-kind.sh (agent-substrate#3509) Since agent-substrate#2092, `deploy ate-system` stages the microVM guest assets and applies the `microvm` SandboxConfig, so both scripts can be replaced by `deploy ate-system` now. This PR deletes `hack/run-microvm-demo.sh` and `hack/run-microvm-demo-kind.sh`. Tested on kind, this fixed the broken counter microVM demo, which failed before due to missing `ATE_CREDENTIAL_PROVIDER`. AI usage: test. - [ ] Tests pass - [ ] Docs updated Signed-off-by: Haven Xia <haoyuxia@google.com>
) `egress-credential-injection.md` - Injection also works on cleartext `http` rules. - Credential URIs: list the form the API accepts and what admission rejects. - The provider setting can also come from `ATE_CREDENTIAL_PROVIDER` or from `atenet.egress.credentialProvider` in the `--config` file. - The `ext-proc` log command works on the Envoy dataplane only. - Use `replaceHeaders` throughout, to match the examples. `egress-trust-bundle.md` - The example was a CRD-shaped manifest that `kubectl ate create actor-template -f` rejects, and its trailing-slash `mountPath` fails validation. It is now a protojson excerpt. - The `unusable contents` error does not exist. The doc now shows the error atelet returns when the egress ClusterTrustBundle is missing. - The wrapper script now reads the same path as the example. - System roots come from atelet's `distroless/base-debian13` image. `egress-traffic.md` - HTTPS whose SNI matches no rule is closed during the TLS handshake, without an HTTP response. - Add a row for `tls_passthrough`. It relays whatever runs inside the TLS without inspecting it. - A client that sends non-HTTP bytes first gets `400 Bad Request`. A client that waits for the server is closed after about 1 second. atunnel does not log either case. AI use: I used Claude Code to audit the egress docs against the code, run a test for all the outdated statement. I reviewed every change. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Yufan Su <yufans@google.com>
…t-substrate#3510) <!-- The PR title becomes the release note. Write it for users. --> Follow-up to agent-substrate#3498 (comment). Remove `hello from: %s` and `getCurrentIP()` from `demos/counter/counter.go`. Why? `counter.go` included `hello from: <ip>` in its response from early dyas of Substrate to show that an actor resumed on a different worker Pod IP. However, now every actor sandbox uses the fixed veth address `169.254.17.2` (`ActorVethIP` in `internal/ateworker/actornet/net.go`), so `getCurrentIP()` always returns `169.254.17.2` regardless of which worker Pod hosts the actor. Also updated documentations to remove mentioning of "the different worker Pod IP replied by counter indicates it was resumed on another pod". <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI-assisted to update the counter demo, docs, and tests; I reviewed the diff. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Zoe Zhao <zoezhao@google.com>
…ubstrate#3355) The security-status-report skill passes every column of `docs/threats.json` to its evaluators, so implementation-status claims and stale notes in the threat model were steering the scores instead of the code. This rewrites the suggested mitigations and notes as short design guidance with no status, and keeps threats and invariants unchanged except: - New Trust Boundaries section: the sandbox is the primary boundary, the worker Pod is trusted with the actors it hosts (its own confinement is defense in depth for the worker-to-node boundary), and a threat that needs another control to fail first is High priority at most - T-12 removed (documentation alone could never close it); other IDs keep their numbers - T-01 to T-03 lowered to High, since they require the outer network layer to fail first - T-11 invariant: the official path for secrets keeps them out of the sandbox - T-21 lowered to High, since it requires a sandbox escape first. Each worker permission beyond the container defaults needs a documented reason; the ones the sandbox runtime needs while workers share the host user namespace (root, runtime capabilities, AppArmor Unconfined) are accepted. A worker must not be able to modify or execute anything atelet or other workers trust - T-25 reworded to a malicious, not corrupt, snapshot, and lowered to Medium, since it requires control of snapshot storage first (same attackers as T-39) - T-26: enforcement points may cache policy as long as changes take effect within a short time, and must deny rather than serve stale policy indefinitely. Open connections and already-issued credentials are out of scope - T-30 lowered to Medium with the TBD invariant replaced, since nodes are not a boundary between actors Skill changes: evaluators score against the invariants rather than treating suggested mitigations as a checklist, don't penalize items a threat puts outside Substrate's scope or that its notes accept with a documented reason, and use a defined 0.0 to 1.0 scale. Each evaluator also writes a short mitigation summary and remaining gap; the new `render_table.py` renders them as a markdown table sorted by priority, and the chart now groups threats by priority. AI use: AI-assisted. I reviewed each section, and validated the rewrite with before/after evaluator runs and a full skill run. - [x] Tests pass - [x] Docs updated Signed-off-by: Greg Castle <gcastle@google.com>
<!-- The PR title becomes the release note. Write it for users. --> Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) This updates brotli to remove a CVE: CVE-2026-111497 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI assisted, I have reviewed the code. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [X] Tests pass - [X] Docs updated Signed-off-by: Max Smythe <smythe@google.com>
…nt-substrate#3464) The orchestrator ran every entry in the `--tests` file, so running one test meant copying it into a new file first. `--test-name NAME` runs only the named entries and can be repeated. The orchestrator now selects and validates the entries before it waits for Docker and clones the repo, so a mistyped name fails right away. A relative `--tests` path now resolves against the working directory instead of the clone. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI-assisted, I reviewed and tested it <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
…substrate#3223) BindActorToWorker and the actor update ran as separate writes, and so did ReleaseActorFromWorker and the actor update. A crash or lost lease between them could leave a worker assignment row the actor did not name, or an actor naming a worker that did not hold it. Recovering from that needed stale claim lookups, rebinds, and release-only paths. BindActorToWorker now applies the actor update in its transaction, and UpdateActorAndRelease replaces ReleaseActorFromWorker. An actor names a worker if and only if that worker holds its assignment. Remove the recovery code this makes unreachable. HostingNode reads the node from the actor's status. BindActorToWorker returns ErrWorkerNotFound for a missing worker, so the resume can tell it from a deleted actor. Fixes agent-substrate#1639 AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Scott Funkenhauser <sfunkenhauser@google.com>
…tenet-egress (agent-substrate#3417) `atenet-egress` runs `cmd/atenet router --mode=egress`, which previously initialized OpenTelemetry with the hardcoded service name `"atenet-router"`. - Set OTel `service.name` in `RouterServer.Run` based on `--mode` (`"atenet-egress"` for egress, `"atenet-router"` for ingress) and pass the meter to `extproc.NewRouteDurationHistogram`. - Remove the unused trace span in `handleStatusz`. - Include `atenet-egress` in `ate-setup`'s `otelOverrideDeployments` so `--otlp-endpoint` restarts it too. - Update the metrics registry/docs and E2E metrics test to verify `atenet-egress` telemetry is exported under `atenet-egress` and not `atenet-router`. Fixes agent-substrate#3308 <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [ ] Tests pass - [ ] Docs updated Signed-off-by: Bo-Cheng Chu <bochengchu@google.com>
…trate#2428) ## Summary Use fixed runtime-specific worker capability sets, following maintainer feedback that Substrate should own the minimum permissions needed to implement its sandboxes. Every worker drops ALL container-runtime default capabilities, adds back its runtime-specific set, and sets `allowPrivilegeEscalation: false`. | Component | Previous set | New set | Removed | | --- | --- | --- | --- | | gVisor worker | 13 capabilities | 9 capabilities (`0x802030cb`) | `MKNOD`, `SETPCAP`, `SYS_CHROOT`, `SYS_PTRACE` | | microVM worker | 15 capabilities | 11 capabilities (`0x882011db`) | `DAC_READ_SEARCH`, `NET_RAW`, `SYS_CHROOT`, `SYS_PTRACE` | | virtiofsd serving child | 9 capabilities | Its 8 default capabilities (`0x880000db`), empty bounding set | `DAC_READ_SEARCH` | Both workers still run as root and retain `SYS_ADMIN` and `NET_ADMIN`. This is incremental worker hardening; the existing deployment-permissions requirements still apply. Fixes agent-substrate#2422. That issue asked for these settings to be configurable per pool; following review, the workers get fixed, reviewed defaults instead. Worker seccomp is split out to agent-substrate#2749, with the profiles in draft agent-substrate#2750. ## Changes - Make the tested reduced capability sets fixed defaults rather than exposing capability drops per pool. - Remove the proposed WorkerPool security API, its validation and generated schema changes, and the example worker seccomp profiles and generator, which move to agent-substrate#2750. Outer worker seccomp and AppArmor remain Unconfined. - Keep `SETPCAP` on microVM workers so virtiofsd's serving child can drop its bounding set with `PR_CAPBSET_DROP`, which libcap-ng only issues while `CAP_SETPCAP` is effective. Keep `SETFCAP` on gVisor workers: runsc cannot start its gofer without it. - Run virtiofsd with `--inode-file-handles=never` and stop granting `DAC_READ_SEARCH`, which virtiofsd only needs to open file handles. The shared directory is on the worker's overlayfs rootfs, which cannot produce decodable file handles, so virtiofsd already used `O_PATH` fds. virtiofsd also explicitly selects its namespace sandbox and seccomp filter. - Verify exact runtime defaults and reconciliation after Deployment security settings drift. - `TestWorkerSecurityFilesystem` checks what the kernel enforces, not only the rendered PodSpec. A testserver `procstatus` subcommand runs in an ephemeral container in the worker's PID namespace and reports `/proc/<pid>/status` by process name. The test asserts the worker's permitted, effective and bounding sets equal the fixed mask with nothing ambient and `NoNewPrivs: 1`; runsc's gofer and sandbox run under seccomp; on microVM, virtiofsd's serving child holds its eight capabilities with an empty bounding set, and every cloud-hypervisor thread except main runs under seccomp. It still covers non-root-owned files through pause/resume and suspend/resume, and on microVM non-root access, cross-UID denial and device-node creation. - Update the API guide and security documentation, including when the change rolls worker pools, and the image cache README and comments that still described privileged workers needing `CAP_MKNOD`. Creating the 0:0 overlay whiteouts has not needed `CAP_MKNOD` since Linux 5.8. - Rebased on upstream `main` at `0ddbf78d`. Actor container capabilities remain configured separately through `ActorTemplate.containers[].securityContext.capabilities`. There is no new WorkerPool API or opt-in configuration. ## Upgrade The new capability sets change the worker pod template, so every WorkerPool rolls when `ate-controller` is upgraded, not only when its worker image changes. Follow the [upgrade runbook](docs/guides/upgrade.md) in its order: 1. Do steps 1 and 2 as usual. 2. In step 3, move every WorkerPool to the new worker image before you upgrade `ate-controller`. A microVM pool left on an older image fails at the next step: its virtiofsd exits with `can't apply the child capabilities`, and actors on it cannot start or restore. Check that no pool is left on the old image: ```bash kubectl get workerpools -A -o custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage' ``` 3. In step 4b, `go run ./cmd/ate-setup deploy ate-controller --config ~/ate-upgrade/new.yaml`. Unlike the runbook says for other releases, every pool rolls once here, with the same `SIGTERM` and 30 minutes per actor as step 3. Watch it with `kubectl get deploy -A -l ate.dev/worker-pool`, then check one worker of each class: ```bash kubectl -n $NS get pod <worker> -o jsonpath='{.spec.containers[0].securityContext.capabilities.add}' ``` gVisor lists 9 capabilities and microVM 11, without `DAC_READ_SEARCH`. Start one microVM actor to confirm virtiofsd comes up. 4. Finish steps 4c to 4e as usual. Rollback: roll back step 4 first, as the runbook says. The pools roll again to the old capability set, which runs both the new and old worker images. Then roll back step 3. Rolling back step 3 alone, with the new `ate-controller` still running, puts old microVM images on the new template and breaks them. Timing: if this lands in 1.0.0 there is no skew issue, because upgrades from 0.x are reinstalls. If it lands after 1.0.0, it changes the worker pod template within a major, which the [version skew policy](docs/reference/version-skew-policy.md) does not allow, and needs the `breaking-change` label. ## Validation On the current revision (`86e684c4`, on upstream `main` at `0ddbf78d`; `86e684c4` only changes docs, so the checks below ran on `6714b09b`): - `hack/verify-all.sh`, `go vet` and `go test -race` on the changed packages: passed. - Live on EKS (Kubernetes 1.37, AL2023, kernel 6.18), reinstalled from this revision with the v1 CRDs through `ate-setup deploy`, gVisor on `m6i.xlarge` and microVM on `i4i.metal`. These passed on both runtimes: `TestWorkerSecurityFilesystem`, `TestGoldenSnapshotLifecycle`, `TestActorSnapshotLifecycle`, `TestPauseResumeActor`, `TestRevertActorToMostRecentDurableSnapshot`, `TestGracefulWorkerTerminationSuspend`, `TestDurableDirLifecycle`, `TestMultipleDurableDirLifecycle`, `TestImageLayerOwnership`, `TestImageUserAndWorkdir`, `TestCombinedVolumes` (external cases skipped), `TestActorEgress`, `TestActorEgressHTTPS` and `TestActorIdentity_AfterRestore_IsOwnID_NotGolden`, plus `TestMicroVMClockAfterDelayedRestore` on microVM. - Process state read from `/proc` on the nodes: - gVisor worker `0x802030cb`; runsc sandbox `0x8001f` and gofer `0x4001f`, both with seccomp. - microVM worker `0x882011db`; virtiofsd serving child `0x880000db` with bounding set 0 and `Seccomp: 2`; cloud-hypervisor threads other than main at `Seccomp: 2`. - Both workers have no ambient capabilities and `NoNewPrivs: 1`. - Upgrade order: a WorkerPool on a microVM worker image from `main` under this `ate-controller` fails with `can't apply the child capabilities`, and recovers when moved back to this revision's image. - A syscall probe in the gVisor worker container gets `EPERM` from `userfaultfd`, which needs `CAP_SYS_PTRACE` while `vm.unprivileged_userfaultfd` is 0. It reached the kernel with the previous capability set. - The kind e2e lanes in `pr-workflow` cover both sandbox classes; they need a maintainer to approve the run on this fork PR. [Test plan and findings for the earlier opt-in revision (PDF)](https://github.com/tjorourke/substrate/blob/7e3040822b85ac0de2703bc8268e4e7f78f26e11/docs/validation/worker-security-test-report.pdf) - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR AI-assisted. --------- Signed-off-by: Tom O'Rourke <tom.orourke@solo.io>
…hs (agent-substrate#3249) <!-- The PR title becomes the release note. Write it for users. --> > Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) On microVM, an actor’s container runs as uid 0 in the guest kernel. Before this PR, the worker sent kata-agent empty masked and read-only path lists. That left the guest’s `/proc/sys` writable: the kernel lets uid 0 write sysctls without any capability. The kata-agent skips masked paths that are directories. Of the five directories, only `/proc/acpi` and `/sys/firmware` exist in the kata 4.3.1 guest kernel. We mount read-only tmpfs similar to [runc](https://github.com/opencontainers/runc/blob/4ca628d1d4c974f92d24daccb901aa078aad748e/libcontainer/rootfs_linux.go#L1250-L1258). **Breaking change**: a uid 0 entrypoint can no longer write guest sysctls, such as `sysctl -w vm.max_map_count=...`. This already fails on gVisor. Goldens and checkpoints taken before this change keep a writable `/proc/sys` until they are re-taken. Part of agent-substrate#3226. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: AI-assisted, I reviewed and tested it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [ ] Docs updated --------- Signed-off-by: Huy Pham <huypham@google.com>
…alkthroughs, and counter volume (agent-substrate#3500) The `sandbox`, `jupyter`, and `multi-template` tutorials linked to retired root `README.md` anchors, omitted required ingress and `ko` settings, and had walkthrough steps out of sync with current CLI and runtime behavior. `demos/multi-template/counter-template.yaml.tmpl` also omitted the `/home/counter` volume mount, so `counter` printed `preserved file counter: -1`. - Mount a `durableDir` volume at `/home/counter` in `demos/multi-template/counter-template.yaml.tmpl` and update the expected `curl` output for all three counters. - List the `atenet-router` `--ingress-allowed-spiffe-ids`, `SNAPSHOT_LOCATION`, and `KO_DOCKER_REPO` (`ko.dockerRepo`) prerequisites in all three tutorials (including the `ATE_INSTALL_KIND=true` defaults on `kind`), and point their install links to `docs/get-started/quickstart.md` and `docs/guides/operator-install.md`. - Clarify per-request subprocess execution in `demos/sandbox/README.md` (`sandbox-client` sends `sh -c` to `/process`), add a non-interactive piped `sandbox-client` command, and remove the duplicate `go install ./cmd/kubectl-ate` step. - Update `demos/jupyter/README.md` and `cmd/ate-setup/internal/demos/jupyter/jupyter.go` to remove unimplemented idle-suspend claims, use `quay.io/jupyter/base-notebook` and `ACTOR_STATE_SUSPENDED`, add a `curl -s http://localhost:8888/api` check, fix step numbering, and document full teardown. - Drop the retired actor DNS routing claim and inaccurate delete-suspend note in `demos/multi-template/README.md`, and stop the background `kubectl ate proxy actor` processes during cleanup. Fixes agent-substrate#3494 AI use: AI-assisted, I reviewed and verified the changes on `kind` and Kubernetes clusters and built the docsite locally. - [x] Tests pass (`./hack/verify/gofmt.sh`, `./hack/verify/boilerplate.sh`, `go test -race ./cmd/ate-setup/...`, `./hack/site.sh build`, and live walkthrough verification) - [x] Docs updated --------- Signed-off-by: Mofi Rahman <mofi@google.com>
…t-substrate#3498) `demos/counter/README.md` hid every walkthrough command inside 16 collapsed `<details>` blocks (leaving `<details>` tags in the docsite's raw Markdown outputs) and had outdated or incomplete instructions across the default, external volume, and microVM flows: - Inline the default `gvisor` walkthrough commands in the numbered steps, note the default `gvisor-default` sandbox upfront, move actor deletion after `## Reaching a non-default port`, and place the `microVM` and external CSI volume variants in dedicated sections at the end. - Document the `atenet-router` `--ingress-allowed-spiffe-ids` prerequisite for `kubectl ate proxy actor`, keeping `ATE_INSTALL_KIND=true` exported on `kind`, and point the Quickstart link to `docs/get-started/quickstart.md`. - Add the missing `ExternalVolumes` preview gate, `ate-setup setup csi`, `delete demo counter` and `kubectl wait --for=delete namespace/ate-demo-counter` before replacing the template, and `kubectl ate --preview create actor` steps for external volumes, and explain `--validate-existing-file-path=/external-data/test.txt`. - Add the `create`, `proxy`, `suspend`, and `delete` steps for `microVM` in `ate-demo-counter-microvm` (`my-counter-1`). - Document all three counters incremented by `counter.go` (`preserved memory count`, `preserved file counter`, and `preserved rootfs counter`) with expected `curl` output across suspend and resume. Fixes agent-substrate#3492 AI use: AI-assisted, I reviewed and verified the changes on `kind` and Kubernetes clusters and built the docsite locally. - [x] Tests pass: docs-only change (`./hack/site.sh build`, `./hack/verify/boilerplate.sh`, and live walkthrough verification) - [x] Docs updated --------- Signed-off-by: Mofi Rahman <mofi@google.com>
…rate#3511) `TestActorSnapshotLifecycle` created a `Tag` and resumed a clone from it, but did not verify in e2e that the tag's snapshot copy survives deleting the source actor or that the clone's own snapshot survives `DeleteTag`. Delete `sourceName` before creating the clone from `tagRef`, and after the clone runs, suspend it, wait for background detach, delete `tagRef`, and resume the clone to verify its counters continue at `(3, 3)`. AI use: AI-assisted, I reviewed and tested it - [x] Tests pass - [ ] Docs updated --------- Signed-off-by: Nayana Bidari <nybidari@google.com>
…ite (agent-substrate#2775) A new `apitool generate-docs` now generates reference documentation of the Substrate API. This PR has 4 commits: * Introduce changes to the style guide on how to write doc comments and a new `doc-format` linter rule to enforce them. * Prefactor: move methods around to reuse it in `apitool generate-docs`. * Add `apitool generate-docs` command. * Make `hack/site.sh content` run the generator. For now, only `latest` version is included in the site. This is controlled by the `releases-since` flag in `hack/site content` script. Once `v1.0.0` is available it will be picked up. AI use: AI-assisted. Manually reviewed, edited and tested it. --------- Signed-off-by: Julian Gutierrez Oschmann <juliangut@google.com>
…gent-substrate#3119) Part of agent-substrate#1464. - Rewrites `docs/guides/authentication.md` around who is calling ate-api: a person running `kubectl ate`, a pod in the Substrate cluster or another cluster, or a CI job, so people stop leaning on the shared `ate-client` token. - Adds pod certificates as an option for client pods (the benchmarks use them), the `ate.dev/api-access` namespace label the ate-system NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to reach ate-api. - Warns where an identity is weaker than it looks: another cluster's issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and basic Editor roles can act as any ServiceAccount, ate-system port-forward rights reach loopback-only Envoy admin APIs that can stop the data plane, and GitHub and Google tokens authenticate anyone. - Moves the per-user ServiceAccount recipe and the signing-in and install-time-provider gaps out of `authorization.md`, and drops its claim that a Secret-based token works there (ate-api rejects its issuer). The recipe and RBAC use agent-substrate#2851's names (`ate-users`, `ate-api-client`). For reviewers: agent-substrate#2851 (`--credential-plugin`) rewrites the same sections, and its paragraph calling GitHub and Google tokens unsafe needs to match this one. Testing: checked each claim against the code on main (GitHub, GKE and Kubernetes behavior against their docs or source) and that the relative links resolve. None of the commands or manifests ran on a cluster. AI use: AI-assisted. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Max Thompson <maxthomp@google.com>
Supersedes agent-substrate#3396 by @stshrive, rebased onto current `main`. The commit keeps his authorship and sign-off. The GKE provisioning and teardown tooling now lives in https://github.com/ai-on-gke/substrate-gke under tools/setup-gcp, so core no longer carries a copy. Docs and comments that pointed at the removed files now point at substrate-gke. go mod tidy drops the GCP client modules only setup-gcp used, and vendor/ and _LICENSES/ are regenerated to match. Covers agent-substrate#3390, except that hack/teardown.sh is removed here as well; substrate-gke's tools/setup-gcp/teardown.sh replaces it. Part of agent-substrate#2304 ### Rebase notes Conflicts were mostly from the Hugo docsite move (agent-substrate#1974): - `tools/setup-gcp/cloud-sql.md`: deleted, as in the original PR. - `docs/guides/operator-install.md`, `docs/guides/postgres.md`: the original edits, re-applied at the new paths. - `docs/guides/observability.md`: kept main's version, since agent-substrate#1974 already removed the Dashboards section the original PR edited. - `cmd/kubectl-ate/README.md`: kept main's stub that points to `docs/reference/kubectl-ate.md`. That page no longer has the setup-gcp line, so the edit had nowhere to go. --------- Signed-off-by: Steven Shriver <6007067+stshrive@users.noreply.github.com> Signed-off-by: Grant McCloskey <gmccloskey@google.com> Co-authored-by: Steven Shriver <6007067+stshrive@users.noreply.github.com>
…te#3147) Split out of the original version of this PR per the review comment: one area per PR, table-driven where the set-up is shared. Covers what a suspended actor's v1-stamped MEMORY snapshot restores at after its pool's SandboxConfig changes, on both lanes: | Edit | Restores at | Next snapshot stamped | |---|---|---| | default moves to v2, v1 still Enabled | MEMORY on v1 via restorable_runtimes | v1 | | v1 Disabled | ROOTFS on v2 | v2 | | v1 Disabled, actor moved to another template with UpdateActor | VOLUMES on v2 (template replacement takes precedence) | v2 | | version names swapped (bundles unchanged, with duplicate aliases) | MEMORY; worker lists the hash once | v1 | | assets edited under the same version name | ROOTFS | new hash | | Tag clone after v1 Disabled | ROOTFS on v2 | v2 | The fixture (`setupSuspendedV1Actor`, `updateVersions`, `waitForWorker`, `resumeAndCheck`) is what the follow-up PRs build on: paused actor across version changes, scheduler preferring the compatible pool, WorkerPool configRef changes and bad asset SHA256. The controlapi unit test change seeds a worker with a different default so the snapshot-stamp assertion actually discriminates. AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Shrutiya <shrutiyam@google.com>
… changes (agent-substrate#3370) `EnsureAPIServerPrerequisites` created `ConfigMap/ate-api-authentication` and `ConfigMap/ate-api-config` only when missing (unless `--ateapi-log-level` was passed), ignoring `--ateapi-expected-jwt-issuer` on re-deploy and leaving legacy or unknown `1.x` rollback fields in place where `ate-api-server`'s strict YAML/prototext loaders would `CrashLoopBackOff`. Additionally, `Deployment/ate-api-server`'s `ate.dev/env-hash` annotation only digested environment variables rather than the mounted `ate-api-authentication` and `ate-api-config` ConfigMaps. Fixes agent-substrate#3271 ## Summary ```text EnsureAPIServerPrerequisites ensureAPIAuthenticationConfig -> ateapiauth.ParseAuthenticationConfig (rewrites when missing, when --ateapi-expected-jwt-issuer is set, or when stored YAML fails strict schema validation; preserves valid custom issuers and extra jwtProviders) ensureAPIConfig -> apiconfig.ParseTextProto (rewrites when missing, when --ateapi-log-level is set, or when stored prototext fails strict schema/level validation; preserves valid customized log levels) DeployAPIServer / UpdateAPIAuthenticationConfig / UpdateAPIConfig apiServerEnvHash(env, ate-api-authentication, ate-api-config) -> annotates Deployment/ate-api-server pod template (ate.dev/env-hash) ``` ## Evidence - **Before:** Re-running `ate-setup deploy ate-system --ateapi-expected-jwt-issuer <issuer>` skipped `ConfigMap/ate-api-authentication`, and upgrading across schema removals or rolling back `1.x -> 1.0.0` with unknown fields in `authentication.yaml` or `apiconfig.textproto` left `ate-api-server` in `CrashLoopBackOff`. - **After:** Valid customized ConfigMaps (custom JWT issuers, extra external `jwtProviders`, hand-edited log levels) survive `ate-setup deploy` untouched when flags are unset; explicit flags or invalid/unknown rollback fields rewrite the ConfigMaps cleanly and update `ate.dev/env-hash`, rolling `Deployment/ate-api-server` to `Ready`. ## Merge Danger **Door:** Two-way **Blast Radius:** Small (`internal/ateapiauth`, `internal/apiconfig`, and `cmd/ate-setup` prerequisite reconciliation and `ate.dev/env-hash` computation) ## Rationale `api-server` crashes on any config field it doesn't recognize, and `ate-setup` never touched those config maps once they existed. the PR makes `ate-setup` rewrite the config maps when the server can't parse it, and keeps valid custom ones. However this need to be in `1.0,0` otherwise if `1.0.x` adds a config field, then someone rollback from `1.0.x` to `1.0.0`, the old server will fail the rollback as it cannot parse the yaml. AI use: AI-assisted implementation and unit tests; reviewed and verified by @moficodes on Kind and GKE. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <mofi@google.com>
…substrate#1607) ateapi stores resources as binary protobuf in `bytea` columns. Binary protobuf identifies a field only by its number. If a field is renumbered, existing rows lose its value. If a retired number is reused, existing rows decode into the wrong field. `atepg.go` unmarshals with `DiscardUnknown: true`, so the original bytes are dropped on the next read-modify-write. Nothing reports this at runtime. - `hack/tools/buf`: adds buf v1.72.0 as a tool module, run through `hack/run-tool.sh`. - `buf.yaml`: checks the protos under `api/`, `internal/proto/`, and `pkg/proto/` with buf's `PACKAGE` breaking rules. Within a proto package, declarations may move between `.proto` files, but messages, enums, services, RPCs, fields, enum values, and reserved ranges cannot be deleted, renamed, renumbered, or retyped. - `hack/verify/proto-breaking.sh`: runs `buf breaking` on the working tree against the merge base of `HEAD` and `origin/main` (or `main`). CI checks out with `fetch-depth: 0`, so the merge base is available there. It passes `--against-config buf.yaml` so a baseline commit without `buf.yaml` still builds. - `AGENTS.md`: removes the Backward Compatibility section. Breaking changes are no longer acceptable once this check is in. - `hack/verify-all.sh`: runs every verifier and lists the failures at the end, instead of stopping at the first one. This is to make sure the CI runs every verifier on a PR including breaking proto changes. I can split this into its own PR if you prefer. I used Claude Code to help write the change and this description. --------- Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
Why is this needed? The existing threat model diagram was out of date and the security guide lacked a high-level overview diagram illustrating Substrate's core isolation boundaries. This PR: 1. Adds `docs/assets/security-boundaries.svg` and a new "Security Boundaries" section to `docs/security.md`, defining the four core isolation boundaries (sandbox, gateways, control plane, storage) held by Substrate. 2. Replaces the single threat model diagram with two coordinated SVGs: - `docs/assets/actor-template-diagram.svg`: Details actor template creation (T1–T10), from image pull and golden actor warm-up to suspend, snapshot upload, and template readiness. - `docs/assets/threat-model-diagram.svg`: Details actor creation (A1–A2) and the lifecycle of an incoming request through outbound tool invocation (B1–B11). 3. Adds numbered step tables (`Step | What happens | What's checked`) below each diagram in `docs/threat-model.md`. 4. Refreshes the Key Components list in `docs/threat-model.md` to reflect current components (including `ate-idp-server`, `k8s-credential-provider`, and `atunnel`), sorted alphabetically. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Greg Castle <gcastle@google.com>
…upper back from disk (agent-substrate#3485) Why: In the microVM burn-in (GKE downstream testing), wake p50 went from ~350ms to ~1.1s (!), almost all of it in upper_join. Reported to me by @annapendleton though I'm not sure who originally found it. agent-substrate#3353 started writing out snapshot tars as they're written, so their dirty pages can't run a tight worker pod out of memory. It also dropped the written pages from the page cache, so a resume on the same node read the rootfs upper tar back from disk ... which is slow. It also made every suspend write the tars at disk speed. Fixes by commit order: 1. Pull summing memory.stat keys out of memoryUsed, no behavior change. 2. Clean pages are reclaimable, so dropping them is irrelevant for OOM safety. - The tars now stay cached. Write-out waits on the previous chunk instead of the one just written, so dirty pages stay at about two chunks without stalling on each one. 4. Start snapshot writeback after the OnDemand delta merge, which replaces memory-ranges, so it tracks the merged file instead of the unlinked delta. 5. Write snapshot files back in the background instead of with a blocking fdatasync. - Starting write-out can block for a long time on a busy disk (https://github.com/torvalds/linux/blob/v6.12/fs/sync.c#L315-L317), and holding deleted snapshot files open meanwhile leaked disk under load, so it starts write-out once, 64MiB at a time, and closes a file as soon as it's unlinked, which drops its dirty pages. 7. What snapshot writes can borrow leaves clean page cache out, so cached tars don't make a pod look "full". - A worker pod without a memory limit (e.g. the deploy_locust.sh default) has nothing to protect, so it lends all of it instead of capping every suspend as if the pod were full. 9. When the pod can lend memory for the whole archive, the tar skips the write-behind and goes to the page cache at full speed, and the loan comes back once it's on disk. Otherwise the write-behind still bounds its dirty pages. That said, with respect to workers without limits: We should probably consider recommending memory limits on worker pods. I will think about it overnight and propose docs updates tomorrow. Tested with locust burn-in tests (see comments in the PR). AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
…gent-substrate#3447) Part of agent-substrate#3095. The objectstore-plugin exchange read actor JWT keys from a file, a stopgap from before ate-idp-server. agent-substrate#3358 needs an issuer instead, so its `pkg/` API takes no key file ([thread](agent-substrate#3358 (comment))). - The exchange fetches keys through `<issuer>/.well-known/openid-configuration`. `--actor-jwt-jwks-file`, `oidcjwt.FileKeySource` and the `actor-id-jwt-jwks` volume are removed. - The sidecar verifies the issuer's certificate against the system roots plus `--actor-jwt-issuer-ca-file`, which is new and defaults to the service DNS trust bundle. The default ate-idp-server issuer verifies with or without an explicit `:443`, and a custom issuer can use a public certificate. ate-setup writes no CA setting for it. The sidecar re-reads the bundle when it changes and keeps the last good pool if a re-read fails. ate-api-server's `JWTProvider` CA files use the same reloading client and still replace the system roots there. - Nothing reads the pool Secret's `jwks` key now, so its writers are removed: the `admin` commands of kubectl-ate, ate-setup, `PublicJWKS` and `ConcretePool.JWKS`. Existing Secrets keep the stale key. - The sidecar fetches keys once at start. It exits on a certificate error and only warns on other errors, so an unreachable issuer fails exchanges with `UNAVAILABLE` until a fetch succeeds. Cached keys refresh after 5m, a failed refresh retries after 1m, and a token with an unknown key ID triggers at most one fetch per 1m. Until a fetch completes, including one the caller canceled, such a token gets `UNAVAILABLE`, so a newly rotated key ID is retried. Upgrade: drop `--actor-jwt-jwks-file` from custom sidecar configs. Set `--actor-jwt-issuer-ca-file` only if the issuer uses a private CA other than the service DNS one. Tests pass for `oidcjwt`, `credbundle`, `objectstoreplugin`, `cmd/objectstore-plugin`, `cmd/ate-setup` and `apiauthn`. The snapshotcredentials e2e needed no change. It compiles with `-tags=e2e` but has not run, as it needs GCS. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
…bstrate#3533) Part of agent-substrate#3226. `gvisor` and `microvm` previously mounted different filesystems at `/dev/shm`, and `/tmp`: - `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal `dev` filesystem, blocking non-root shared memory writes. - `/tmp` was part of the host-backed rootfs upper layer on microVM and a conditional `tmpfs` on gVisor when the image's `/tmp` was empty. - Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the image's rootfs, as on Kubernetes. Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`, `/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a volume at a system mount destination such as `/tmp`, `ocispec.Build` replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites `bind` mounts in place. AI use: AI-assisted, I reviewed and tested it. Breaking change: - Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading. - MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut. - MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed rootfs. - Its files count against the actor's memory limit. - It is capped at half of the guest's memory. - ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden. - Files the image ships under `/tmp` are hidden. - gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB. - gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when the image's `/tmp` was empty, so files the image ships under `/tmp` are now hidden. If an actor needs more space in `/dev/shm` or `/tmp`, or needs files there kept in ROOTFS snapshots, mount a `durableDir` volume at that path. It replaces the default tmpfs. Images that ship files under `/tmp` should create them at startup instead. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <nybidari@google.com>
…e#3450) Concurrency tests in agent-substrate#3255 showed that a request authorized against a deleted atespace can apply to a new atespace with the same name. A previous owner can regain `owner` on it or delete it. We decided not to fix this in code. Atespace names must not be reused, and nothing enforces it. This documents the limitation in the authorization guide and qualifies the claim that a recreated atespace starts with no policy. I used an AI assistant to analyze the issue and draft the doc text, and reviewed the result. --------- Signed-off-by: Shruti Nair <shrutinair@google.com>
…ate#3410) Part of agent-substrate#1661. - Documents the JSON array form of ate-setup's `--credential-provider` in `docs/guides/egress-credential-injection.md` and `cmd/ate-setup/commands.md`, and that the gateway sends each `ate-secret://` URI to the provider its host names. - Updates comments that still described one provider: the egress manifest's marker comment and `cmd/atenet/internal/router/egress/provider.go`. Docs and comments only, so it can merge during the freeze. AI use: AI-assisted draft, reviewed by me. - [x] Tests pass (no code changes) - [x] Docs updated Signed-off-by: Max Thompson <maxthomp@google.com>
…ubstrate#3486) Follow-up to agent-substrate#3323, as @SergeyKanzhelev suggested there. kubelet expects a device plugin to switch to a new socket when it re-registers. Since 1.37 it rejects a Register for a socket it still has connected, and two overlapping Registers for one socket can wedge the resource as "already connected" until kubelet restarts: kubernetes/kubernetes#139338 - the device plugin listens on a new `ate.dev-kvm-<random>.sock` for each registration, and removes sockets left over from earlier atelets - the device plugin stays up until atelet exits again. agent-substrate#3323 stopped it on SIGTERM to avoid overlapping with the replacement pod, but the DaemonSet only starts the replacement once the old pod is gone, so that just left `ate.dev/kvm` unhealthy for the whole drain <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to draft this and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
…e#3553) `delete demo` returned while the demo namespace was still terminating, so a `deploy demo` right after it failed: the API server refuses new objects in a terminating namespace. Delete now waits for the namespace to be gone, as `delete ate-system` does. Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Demo Fix. Fixes agent-substrate#3525 AI use: I wrote, claude polished. - [ ] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <haoyuxia@google.com>
The claude-code-multiplex demo doesn't work as written: 1. `deploy demo claude-code-multiplex` creates the three agent templates but no actors, so the dashboard has no agents and "Give a task" fails with `no agents available in namespace`. 2. Even with actors created by hand, the agents can't reach the Anthropic API. Every actor starts with **no egress policy**, and the egress gateway blocks all traffic. 3. The WorkerPool sets no resource limits, so each pod advertises the whole node and all three agents run at once. The demo is about a third agent waiting for a free pod, and that never happens. How this PR fixes it: - It builds on Adam's change from agent-substrate#2982 is folded in, with him as co-author. Deploy creates the three actors, the dashboard resumes an agent when its task starts and suspends it when the task ends, and each pod is sized for one agent. - When both pods are busy, a new task stays queued until one finishes, instead of suspending an agent mid-task. - Gives each agent an egress policy that lets TLS to api.anthropic.com:443 pass through the gateway. - The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps part of it for itself, so a 2Gi agent didn't fit. - Update the README to match the code. agent-substrate#2488 by @Ziy1-Tan documents the same setup as manual steps. With this PR those steps aren't needed. Testing: tested on kind. AI use: Claude assisted and I reviewed - [x] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <haoyuxia@google.com> Co-authored-by: Adam Wathieu <awathieu@google.com>
…-substrate#3517) 1. Removes the `status: planned` banner so the page no longer claims to be aspirational. 2. Rewrites the trade-offs section into present-tense implementation context (data locality, control-plane boundaries, and OpenTelemetry observability) and removes the outdated "haven't started to tackle" list. 3. Documents `atenet-egress`, DNS relay filtering, and `EgressPolicy` credential injection under the Networking Stack. 4. Replaces the inline Security & Isolation bullet list with links to the main Security Overview and Threat Model documents. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Greg Castle <gcastle@google.com>
agent-substrate#3391) Fixes agent-substrate#3287 `WorkerPool/parking` in `demos/parking/parking.yaml.tmpl` did not set container resource limits, so the Kubernetes Downward API exposed the node's allocatable capacity in `/run/worker-capacity`. As a result, 2 workers had room for all 4 `1000m` actors at once and request parking was never triggered. Set per-worker container limits to `1` CPU and `1Gi` memory so each worker hosts at most one actor. Also add a `wakeupProbe` on `/readyz` port `80` to `demos/parking/parking-template.yaml.tmpl` (matching `demos/counter/counter-template.yaml.tmpl`) so golden memory snapshots are captured after the counter HTTP server binds port 80. In `demos/parking/load.sh`: - Write `kubectl ate proxy actor` output to `.proxy` files instead of `.proxy.log` so `cat "${TMP}"/*.log` does not count proxy startup lines as failed requests. - Wait for each local proxy to start listening before generating load, and fail fast with the proxy output if a proxy exits early. - Exit `1` when any request fails with a non-200/503 status (`other > 0`) or when no requests succeed (`ok == 0`). Signed-off-by: Bo-Cheng Chu <bochengchu@google.com>
gvisor-default gains v7, release 20261102.0, as its default version. v6 stays enabled, so actors with memory snapshots taken on v6 still restore with their memory. The new gVisor release includes some reliability fixes such as: - Systrap no longer crashes sandboxes at high density. It now recognizes host stalls, such as a cgroup freeze, and stops killing stub threads that only look stuck (google/gvisor@a5b7936e84dd). - Resumed actors no longer fire all their pending timers at once, because CLOCK_MONOTONIC stops counting checkpointed time. Clock reads are fast right after a restore (google/gvisor@25172e526f45). - Apps no longer freeze while a large memory region loads during a background restore, which lowers resume latency for big actors (google/gvisor@ffc048011e1b, google/gvisor@ae286a28ef26). - Checkpoint no longer stats every cached file. For actors with many files, such as node_modules, the pause drops from seconds to milliseconds (google/gvisor@8229bcb7a7b6). - Disk-backed overlay pages no longer count as memory after a restore, and long-lived sandboxes no longer grow memory for each file they touch (google/gvisor@aa767e4c43d0, google/gvisor@b24cc9724102). AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Nayana Bidari <nybidari@google.com>
…ent-substrate#2605) Part of agent-substrate#1756. - Adds `docs/guides/actor-jwts.md`, so relying parties have one page that says what an actor JWT contains and how to verify it: the header and claims with a decoded example, the subject's length and incarnation limits, and the verification steps. - Documents the `https://<host>` issuer, the `actorJWT.issuer` setting, and that ate-api-server and ate-idp-server get the same value, since a mismatched issuer makes every token unverifiable. - Lists the discovery document and key set fields, and how to reach `idp` from inside the cluster (servicedns ClusterTrustBundle) and from outside (an Ingress the operator provides). - Gives the order of a key rotation with `kubectl ate admin add-jwt-key`, `activate-jwt-key` and `remove-jwt-key`. - Lists known limitations, so relying parties know what the tokens don't cover: no revocation, bearer tokens, no cloud token exchange yet (agent-substrate#1661), a stale `actorUID` after an actor is recreated (agent-substrate#2208), and manual rotation. - Links the page from the README's documentation list. For reviewers: weight 16 sorts the page after the existing guides, so no other page changes. Federation recipes and a rotation runbook come in agent-substrate#2911. Testing: checked each claim against main and that the relative links resolve; `go run ./tools/site-content` accepts the page. I didn't build the Hugo site; the site workflow does in CI. AI use: AI-assisted draft, reviewed by me. - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Max Thompson <maxthomp@google.com>
…bstrate#2911) Part of agent-substrate#1756. - Adds `docs/guides/actor-jwt-federation.md`, which serves the issuer on a public host and lists what a relying party outside the cluster needs: the issuer, the audience, authorization rules for the `actor/<atespace>/<name>` subjects that should have access, and RS256 if it rejects ES256. - Adds `docs/guides/actor-jwt-key-rotation.md`: the rotation steps and how long to wait between them. The waits come from the kubelet's Secret sync, the pool's one-minute reread, relying parties' key set caches, and the gateway reusing a token for two thirds of its lifetime. It also covers changing the algorithm and replacing a compromised key. - Links both from `actor-jwts.md` and the README. For reviewers: Nothing in Substrate presents an actor JWT to a cloud token exchange endpoint yet, since the gateway only replaces headers, so the federation guide has you check the setup by hand. Testing: docs only. AI use: AI-assisted draft, reviewed by me. - [x] Tests pass - [x] Appropriate changes to documentation are included in the PR --------- Signed-off-by: Max Thompson <maxthomp@google.com>
…e#3580) Fixes [agent-substrate#3579](agent-substrate#3579) ### Summary Restricts `securityContext.capabilities.add` to a hardened allowlist of 12 Linux capabilities and blocks dangerous capabilities such as `MKNOD`, `SYS_ADMIN`, and `SYS_RAWIO` before 1.0. In microVM sandboxes, granting `MKNOD` to a container running as root allows it to create block device nodes matching the guest's `virtio-pmem` root disk and directly write to the guest filesystem. Other capabilities like `SYS_ADMIN` and `SYS_RAWIO` expose dangerous guest kernel attack surface. ### Changes * **Capability Allowlist:** `securityContext.capabilities.add` now only permits 12 capabilities required for package installs, non-root user switches, and standard container workflows: * `AUDIT_WRITE` * `CHOWN` * `DAC_OVERRIDE` * `FOWNER` * `FSETID` * `KILL` * `NET_BIND_SERVICE` * `SETFCAP` * `SETGID` * `SETPCAP` * `SETUID` * `SYS_CHROOT` * **Admission Enforcement:** `ateapi` and `atelet` reject actor template creations and updates specifying capabilities outside this allowlist. * **Drop Unrestricted:** `securityContext.capabilities.drop` continues to accept any capability name. * **MicroVM Hardening:** * Volume bind mounts are now mounted with `nodev`, and the microVM guest mounts the virtio-fs share with `nodev` (covering automounted submounts and rootfs). `ShapeGVisor` strips `nodev`, keeping gVisor specs identical and preserving checkpoint/restore compatibility. * MicroVM guest sysctl `net.ipv4.ping_group_range = "0 2147483647"` is configured so standard unprivileged `ping` continues to work without requiring `NET_RAW`. * **Documentation & Schemas:** Updated `api-guide.md`, `sandbox-classes.md`, and regenerated protobuf bindings (`ateapi.pb.go`, `atelet.pb.go`). ### Breaking change: gVisor `MEMORY` snapshots taken before this release won't restore gVisor now gets `nodev` on volume and system mounts (`/proc`, `/dev/shm`, `/sys`, `/tmp`), matching microVM. runsc applies it to the host bind mount. `runsc restore` rejects a checkpoint whose mounts differ from the new spec (`"Mounts" does not match across checkpoint restore`), so gVisor `MEMORY` snapshots taken before the upgrade can't be restored. rc.2 already breaks these snapshots with [agent-substrate#3533](agent-substrate#3533) (tmpfs `/dev/shm` and `/tmp`), so this adds no new upgrade step on top of that. MicroVM actors are unaffected. **What breaks:** - Suspended gVisor actors whose latest snapshot is `MEMORY` fidelity (the default) move to `ACTOR_STATE_CRASHED` on `ResumeActor`. - New actors from gVisor templates created before the upgrade crash on first resume, because the template's golden snapshot is an old `MEMORY` checkpoint. **Upgrade steps:** 1. **Before upgrading,** resume any suspended gVisor actor whose in-memory state you need, and save that state to durable storage or let its work finish. 2. **Re-create gVisor templates.** `ActorTemplate` is immutable, so create a new template (e.g. `my-template-v2`), or delete and re-create it, to build a new golden snapshot. Point clients at the new template. 3. **Replace crashed actors.** `RevertActor` won't help: it reverts to the same pre-upgrade checkpoint, which crashes again. `UpdateActor --template` won't either, because resume restores the actor's own snapshot. Delete the actor and create it again from the new template. Its in-memory state and its snapshot's rootfs and volume contents are lost. ### Testing * **Unit Tests:** Updated resource validation tests, parity tests, and microVM spec tests (`go test ./...` passes in container). * **Live GKE Testing (`sgke-rc1`):** * Verified template rejection when adding `MKNOD` or `SYS_ADMIN`. * Verified unprivileged ICMP `ping` works in microVM guests without `NET_RAW`. * Verified that pre-existing gVisor actor snapshots restore cleanly. * **e2e Probe Coverage:** * Verified inside the guest that `mknod` fails with `EPERM`. * Verified that unprivileged users can open `SOCK_DGRAM`/`IPPROTO_ICMP` sockets across both runtimes. Thanks to [Fabiano Fidêncio (@fidencio)](https://github.com/fidencio) for the report on MKNOD that led us to make this change. --------- Signed-off-by: Greg Castle <gcastle@google.com>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bring fork
mainup to upstream17fa8586e, preserving the Helm charts, AgentGateway dataplane, PostgreSQL certificates, and fork CI/release behavior. This carries 41 upstream commits in one sync merge based on829e1a655.Merge with a merge commit to preserve upstream ancestry.
UpdateActorAndReleaseand observing it after commit.ate-setupfor microVM installation and keep RSA/EC private-key support alongside upstream system-root loading.Upstream's mount hardening breaks pre-upgrade gVisor
MEMORYsnapshots. Save needed state before upgrading, then recreate affected templates and actors. Changing an actor's template or callingRevertActordoes not repair its old snapshot. microVM snapshots are unaffected. Please apply thebreaking-changelabel.Worker release during suspend
sequenceDiagram participant W as Suspend workflow participant DB as Store participant C as Worker cache W->>DB: UpdateActorAndRelease DB->>DB: Commit actor state and worker release DB-->>W: Committed actor and released worker W->>C: Observe released workerValidation: full Go race suite, repository verifiers, Helm lint and 48 unit tests, manifest/CRD parity, and focused Helm tests with three shuffled race runs. Privileged tests and Helm runtime lanes covered gVisor, microVM, egress credential injection, and identity after restore. The gVisor metrics/networking suites passed after recreating old golden snapshots. Runtime validation used AgentGateway and S3; GCS exchange and runtime-specific cases were skipped. The refreshed chart renders equivalent runtime objects with the CI debug setting.
AI use: Codex helped resolve the merge, adapt fork integrations, review the changes, and run validation.