Repository navigation
Sync main with upstream development - #110
Merged
Merged
Conversation
`Control.MintActorCertificate` has no consumers: the feature that needed it was scoped out of GA. This removes the public RPC, its request/response types, signing code, validation, authorization rule, and docs, so a future consumer can define the API it needs. Actor JWT minting and the worker certificates used by atunnel remain in place. The internal `atelet.WorkerSupport.MintActorCertificate` is still used and forwards to `WorkerService.MintWorkerActorCertificate`. Fixes agent-substrate#3425 Breaking change: calls to the removed public RPC return `UNIMPLEMENTED`. Code referencing its generated client method or message types must remove those references before upgrading. <details> <summary>Certificate path retained for atunnel</summary> ```mermaid flowchart LR atunnel["atunnel"] -->|MintActorCertificate| atelet["atelet.WorkerSupport"] atelet -->|MintWorkerActorCertificate| worker["ateapi.WorkerService"] worker -->|Sign worker-for-actor certificate| ca["Actor identity CA pool"] ``` </details> AI use: Prepared with Codex. - [x] Focused tests pass (see validation above) - [x] Docs updated --------- Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Fixes agent-substrate#3272 Since 1.37, kubelet wedges a device plugin resource if a second Register for the same socket arrives while it is still closing the first stream. The failed connect deregisters both clients by socket path: https://github.com/kubernetes/kubernetes/blob/v1.37.0/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L82-L86 So when the first stream ends, kubelet never calls PluginDisconnected, and every later Register fails with "device plugin already connected" until kubelet restarts: https://github.com/kubernetes/kubernetes/blob/v1.37.0/pkg/kubelet/cm/devicemanager/plugin/v1beta1/handler.go#L137-L140 atelet keeps one fsnotify watcher across registration retries, so a kubelet.sock Create that was queued while it retried made it stop and re-register right after registering. It now ignores the Create if kubelet.sock is still the socket it registered with. A node that is already wedged still needs a kubelet restart. <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to trace this through the kubelet source and draft the fix, and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <bentheelder@google.com>
…ecurity overview (agent-substrate#3440) The zero-trust tagline said "no unauthorized way out of the sandbox" on the home page and "no way out of the sandbox" in the security overview, and the home page's sandboxing card was labelled "No way out". This changes all three to "no escape", which is a shorter and catchier way to say "no unauthorized" and makes the home page and security overview match. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Greg Castle <gcastle@google.com>
…gent-substrate#3193) The aggregation note from agent-substrate#3056 tells readers to take the `max` across ate-api-server replicas and then `sum`. As agent-substrate#3162 shows, the replicas read separate worker caches and can disagree for a few seconds: one reports `{ate_worker_state="partial"} 3` while the other still reports `{ate_worker_state="idle"} 3`, so `max by (ate_worker_state)` keeps both series and the pool sums to 6 workers. Take the average across replicas instead. It keeps the sum across the states equal to the pool size; the counts can be fractional for those few seconds. Updates the registry note and the observability guide. No code change: the locust collector already reads a single process's series, and nothing else aggregates this instrument. Fixes agent-substrate#3162 AI use: Claude Code analyzed the issue and wrote the change and this description. I reviewed the diff. Codex rebased the branch onto main and resolved the documentation conflicts. --------- Signed-off-by: cloudonly <enrondg@gmail.com>
…ate#3309) The locust runner uploads the same stats file twice, as `stats.jsonl` and `{name}.jsonl`. The second copy was kept for a data migration (see the removed TODO). Nothing in the repo reads it now. I used Claude Code to find the duplicate upload and make the edit. I reviewed the diff. Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
…3446) The v1.0.0-rc.1 release run (https://github.com/agent-substrate/substrate/actions/runs/40925903340) failed in **Publish images** with `DENIED: permission_denied`. ghcr.io can briefly deny pushes to a package it has just created, and this is the first release that creates them. `ate-setup publish release-images` now builds each ko image in its own `ko build`. If an image fails, it logs the error and retries that image, for up to 3 attempts, waiting 10s then 20s. The workflow is unchanged. --------- Signed-off-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com> Co-authored-by: Anna Pendleton <13793293+annapendleton@users.noreply.github.com>
…s kept (agent-substrate#2392) Containers ignored the image's `USER` and `WORKDIR` and ran as root in `/`, and atelet's unpack dropped the file owners image layers record. So an image that hands a directory to its user, such as jupyter's `/home/jovyan`, didn't work as built. - atelet applies each tar entry's uid/gid and keeps setuid, setgid and sticky bits. It adds `CHOWN`, `FOWNER` and `FSETID` to its `DAC_OVERRIDE`. The image cache layout moves to version 3. - The process runs as the image's `USER`, resolved as Docker does against the image's own `/etc/passwd` and `/etc/group`, and starts in its `WORKDIR` (default `/`). - The jupyter demo runs as jovyan and drops `DAC_OVERRIDE` and `--allow-root`. The `USER` and `WORKDIR` commits are @dims's from agent-substrate#1918, kept as written. This PR supersedes agent-substrate#1918. Fixes agent-substrate#2036 #### Breaking change Please add the `breaking-change` label. - After an upgrade or rollback, atelet refuses the existing image cache. Delete `/var/lib/ate/image-cache` on each node once no actor runs on it, as atelet's error says. After an upgrade, a node reboot also works. - A custom atelet securityContext needs `CHOWN`, `DAC_OVERRIDE`, `FOWNER` and `FSETID`. - Images with a `USER` stop running as root, and images with a `WORKDIR` stop starting in `/`. A non-root `USER` is expected to have no effective capabilities, so `securityContext.capabilities.add` no longer helps it. This is checked under runc and in the gVisor source, not yet inside a gVisor or microVM sandbox. - gVisor snapshots of such images don't restore, because `runsc restore` checks each container's user and working directory. Recreate those templates. MicroVM snapshots resume as they were. - Files an earlier run wrote as root into a durable dir stay root-owned. Tested: `go test -race` on atelet, imagecache, ocispec, tarutil, kata and the probe fixture, on macOS and on Linux as non-root, and imagecache with `-count=3 -shuffle=on`. The root-only imagecache and tarutil tests pass as root on a privileged Linux container. `hack/verify-all.sh` passes. The jupyter image serves `/api` under Docker as 1000:100 with every capability dropped. @dims ran the jupyter demo and a non-root node image as their users on kind, on both runtimes, before agent-substrate#1904 merged; on microVM uid 1000 couldn't bind :80 then. A new imagefs e2e starts an actor from a probe image with `USER nonroot:nonroot` and `WORKDIR /home/nonroot`. It checks uid and gid 65532, no supplementary group besides 65532, no effective capabilities, the working directory, and that the user can create a file there. CI runs it on gVisor and on microVM. At this head nothing ran in a gVisor or microVM sandbox yet: not the jupyter demo and not the imagefs e2e. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com> Signed-off-by: Davanum Srinivas <davanum@gmail.com> Co-authored-by: Davanum Srinivas <davanum@gmail.com>
<!-- The PR title becomes the release note. Write it for users. --> Fixes agent-substrate#3268 ### Problem Envoy asks for config in a fixed order: clusters, then the 2 secrets the clusters need, then listeners, then the 2 secrets the listeners need. It won't ask for listeners until every cluster is ready. Over state-of-the-world xDS, the router's ADS cache [answers a request that lists names only if it names every resource](https://github.com/envoyproxy/go-control-plane/blob/71637ad69bbc5f51fbb2562e612a4365292804a5/pkg/cache/v3/simple.go#L491). Envoy's first secret request names only the 2 cluster secrets, so it was ignored. Envoy waited out its 15s `initial_fetch_timeout`, marked the cluster ready anyway, then asked for listeners and all 4 secrets. With one replica, that meant 15s with no ingress on every start. ### Solution The router Envoy now uses delta xDS (`api_type: DELTA_GRPC`). The delta path in go-control-plane has no such check, and with `ads=true` it still orders updates across resource types, which we'll want if the router ever serves dynamic config. ### Result With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no timeout. `TestXdsServer_AnswersClusterSecretsBeforeListeners` checks the manifest uses delta and replays the startup over a delta stream. It fails on main. AI use: AI-assisted. I reviewed the change and ran it against real Envoy. - [x] Tests pass - [x] Docs updated (none needed) Signed-off-by: Chuang Wang <chuangw@google.com>
…te#3264) Expand the demo e2e suite to cover asynchronous suspend background upload, background detach, and CreateTag promotion. Lower atelet's snapshot detach delay in kind so the suite can wait for background detach. Checks that wait for a background upload or detach skip on installs with longer delays, and fail in CI. AI use: AI-assisted, I reviewed and tested it. - [ ] Tests pass - [ ] Docs updated Signed-off-by: msau42 <msau@google.com>
Follow the docsite to create a animated flow of substrate in README, add and some metric badges AI usage: apparently all done by AI. - [x] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <haoyuxia@google.com>
…gent-substrate#3435) `ate.dev/llms.txt` linked to full Docsy HTML pages instead of standalone Markdown files, omitted page status markers and schema links, and still used the old `ateom` component name. Meanwhile, `raw-markdown.html` wrote into an HTML template buffer and expanded `include-file` with `replaceRE`, which HTML-escaped code blocks and blockquotes across `ate.dev/llms-full.txt` and "Copy as Markdown" (`'`, `"`, `>`), stripped `$VAR` shell references as regexp capture groups, left relative links unrewritten, and kept decorative `<figure>` HTML blocks. - Add the `MARKDOWN` output format for doc pages (`site/layouts/docs/single.md`) so each page publishes a clean `index.md` alongside `index.html` and exposes `<link rel="alternate" type="text/markdown">` in `<head>`. - Return a plain string from `raw-markdown.html`, expand `include-file` with literal replacement, drop decorative diagram HTML, and rewrite relative links to absolute URLs across `llms-full.txt`, per-page `index.md`, and "Copy as Markdown". - Update `index.llms.txt` to link to each page's `index.md` build, include `[Status: ...]` labels from front matter, link the `ateapi.proto`, `ate.dev/v1` CRD, and OpenTelemetry Weaver metric schemas, and summarize the sandbox model and the split between `ate.dev/v1` Kubernetes CRDs and the `ate-api-server` gRPC API. - Replace leftover `ateom`, `ate-api`, and `herder` references in docs with `ateworker`, `atenet-router`, and `ate-api-server`, and add template tests in `hack/test-site-templates.sh`. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <mofi@google.com>
…rate#3474) `README.md` and `docs/concepts/architecture.md` claimed resume operations under 400ms and more than 1500 activations per second, while the website and get-started docs say under 500ms and 500+/s. Update both current-capability claims to less than 500ms and more than 500 activations per second, leaving the aspirational North Star targets in `docs/concepts/architecture.md` unchanged. AI use: AI-assisted, I reviewed and verified the changes. - [x] Tests pass: docs-only change - [x] Docs updated --------- Signed-off-by: Mofi Rahman <mofi@google.com>
…t-substrate#3468) Part of agent-substrate#3226 Add docs/reference/sandbox-classes.md comparing the gvisor and microvm sandbox classes across CPU sizing, memory limits, per-container limits, NET_RAW capabilities, system mounts, VOLUMES/ROOTFS snapshots and gVisor fscheckpoint compatibility requirement for cross-version filesystem restores. Update stale references across architecture, glossary, quickstart, and guides. AI use: AI-assisted, I reviewed and tested it. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <nybidari@google.com>
Snapshots are stored under
`<location>/atespaces/<atespace>/{actors,tags}/<uid>`. A
`storageLocation` that already contained that layout (for example
`gs://snaps/root/atespaces/team-a/actors/<uid>`) put one template's
snapshots inside another actor's prefix, grant and cleanup.
Where a location is set (template create, `--snapshot-location`,
`ate-setup deploy`), each path segment may now contain only ASCII
letters, digits, `_` and `-`, and may not be `atespaces` in any letter
case. The bucket is not restricted.
- Creating a tag checks the template's location against the same rules
and returns `FAILED_PRECONDITION` on a bad one.
- Parsing of recorded snapshot URIs stays lenient, so stored snapshots
still resume and get collected.
Breaking change: if the path of your `--snapshot-location`
(`SNAPSHOT_LOCATION`) has a segment outside `^[A-Za-z0-9_-]+$` (a dot,
`%`, an empty segment, non-ASCII) or equal to `atespaces` in any case,
change it before upgrading. Otherwise ate-api-server refuses to start.
- [x] Tests pass (touched packages, `hack/verify-all.sh`)
- [x] Docs updated
AI use: I used Claude Code to implement and review this change, and
reviewed the diff myself.
---------
Signed-off-by: Anish Ramasekar <anish.ramasekar@gmail.com>
### Summary
Adds `--agentsession-awake` (paired with `--target-rps`) to
`agentsession` so benchmarks can maintain a large pool of registered
sessions while keeping a fixed number resident and swapping idle awake
sessions with parked ones at a controlled rate.
### How It Works
- **Registration (`createSession`)**: When `--agentsession-awake > 0` or
`--lifecycle-mode none`, each session runs step 1 as soon as its actor
serves so the initial actor/snapshot holds real memory and disk state.
In `--lifecycle-mode none` at `--agentsession-think-scale 1`,
`personal-assistant-light` waits 55m before step 1, so a 20m resident
density run would otherwise measure empty sandboxes.
- **Fill**: Once session registration has been idle for 1 minute
(`registrationQuiet`), the conductor wakes `--agentsession-awake`
sessions (`fillConcurrency = 8` at a time) and logs `agentsession:
turnover fill done`.
- **Turnover (`--target-rps`)**: Each second, the conductor swaps
`--target-rps` idle awake sessions (`cmdPark`) with parked sessions
(`cmdWake`). Each woken session immediately runs its next scripted step
while marked `busy` and returns to `idle` when the step finishes.
- **Measurement isolation**:
- Registration parks record under `SuspendActor_Register` /
`SuspendActorAsync_Register` so `SuspendActor` and `SuspendActorAsync`
hold only turnover parks.
- `runner.py` parses `agentsession: turnover fill done`
(`locust/turnover_window.py`), starts `server_telemetry`'s steady-state
window there, and appends a `turnover_window` row to `stats.jsonl` with
per-request counts, failures, and rates from `stats_history.csv` over
the turnover phase.
- **Observability**: Exports
`agentsession_turnover_sessions{state="awake|parked|waking|parking"}`.
### Testing
```bash
go test -race -count=3 -shuffle=on ./internal/benchmarking/boomer/agentsession/...
python3 -P -m unittest discover -s benchmarking/locust/unit_tests -p "test_*.py"
```
---------
Signed-off-by: Chenyi Wang <chenyiwang@google.com>
…bstrate#3404) Several control-plane docs and the `kubectl-ate` reference still described earlier field names (`durableSnapshot.status`, `source_snapshot_tag`), empty-atespace rules, and older CLI columns and flags. Updates them to match the current implementation: - `docs/reference/glossary.md`: lists `Tag` under control-plane records, and notes that deleting an atespace requires its actors, actor templates, and tags to be gone. - `docs/reference/api-guide.md`: fixes `durableSnapshot.state`, documents deletion checks on `CreateActor` and `UpdateActor`, and adds `GetWorker` and `ListWorkerActorAssignments`. - `docs/api-style-guide.md`: renames `source_snapshot_tag` to `source_tag`. - `docs/guides/authorization.md`: `CreateActor` skips `can_use_tag` when `sourceTag` is omitted. - `docs/reference/kubectl-ate.md`: documents `--preview`, `--sandbox-class`, `top workers`, the current `get worker` and `get actor` columns, template removal before `delete atespace`, and automatic snapshot upload on `create tag`. AI use: AI-assisted to audit control-plane docs against the codebase and draft the documentation updates. I reviewed and verified the changes. - [x] Tests pass - [x] Docs updated Signed-off-by: Haiyan Meng <haiyanmeng@google.com>
Signed-off-by: Eitan Yarmush <eitan.yarmush@solo.io>
EItanya
marked this pull request as ready for review
October 30, 2026 15:46
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Sync fork main at
a57b0844bwith upstream main atf475d8b4c, preserving the fork's Helm, AgentGateway, credential-provider, deployment, and publishing behavior. Keep the shared release workflow from #106 and combine its source labels with upstream's per-image publishing retries.Give Helm's atelet the capabilities needed to preserve image ownership and permissions, and document the image-cache v3 upgrade. Set the snapshot detach delay for the new Helm E2E checks. Correct the credential-provider installation instructions and enable push validation on
release-1.0.This is one sync merge. Merge it with a merge commit to preserve upstream ancestry. Release cherry-picks are handled separately in #108.
Validation:
hack/verify-all.sh, Helm lint, all 41 chart tests, manifest rendering, CRD parity, migration immutability, and site-content generation pass. Independent review found no actionable issues.The full local race run has one failure:
TestUnpackLayer_UnprivilegedKeepsUnpackingUserexpects primary GID 1003, but this account can apply supplementary GID 1000. The test and ownership implementation match unmodified upstream0c662eba7, where this reproduced three times. CI passed, including gVisor and microVM with preview off/on and Helm runtime coverage.AI use: Codex audited the fork changes, prepared the merge, ran checks, and performed an independent review.