feat(compute): Lambda MicroVMs P2 — smoke parity foundations (#645) - #733
Open
dreamorosi wants to merge 6 commits into
Open
feat(compute): Lambda MicroVMs P2 — smoke parity foundations (#645)#733dreamorosi wants to merge 6 commits into
dreamorosi wants to merge 6 commits into
Conversation
…ples#645) Implement Phase P2 of ADR-021 short of the live smoke run: the agent is now fully launchable and observable on the lambda-microvm backend. Agent (agent/src/server.py): - platform_config delivery: deployment-specific, non-secret identifiers (13-key allowlist, 4 required) arrive in the /run envelope and are installed into the environment before any credential or pipeline initialization; unknown keys fail closed (400 MICROVM_RUN_PLATFORM_CONFIG_INVALID / _INCOMPLETE). Decided over image-baked env by configuration lifetime: platform values belong to the deployment, image versions to packaging - and external images cannot receive stack env at all - /validate (image hook): shallow, zero AWS calls - it runs under the build role; /terminate (runtime hook): best-effort flush, returns 200 for any body (raw-Request handler, structural guard against reintroducing a typed body model), never writes terminal task status - snapshot credential hygiene: /ready no longer spins up the CloudWatch writer (it pinned a build-time region + resolved credential chain into boto3.DEFAULT_SESSION - state a snapshot would replay into every MicroVM); /run pre-install logging is stdout-only until platform_config is installed (poisoned-seam + subprocess regression tests, mutation-checked) - the /run S3 payload fetch now uses the attributed client factory Infra (cdk): - platform_config producer in the strategy: closed map over the contract keys, env read at call time, required-key guard with remedy, boundary math includes the block, size-check before PutObject, key-names-only logging - execution-role runtime IAM parity (feature-based, not an ECS copy): GitHub PAT + channel-OAuth-prefix secrets, scoped Bedrock invocation, AgentCore Memory grantReadWrite, AZ describe; still no direct DynamoDB (tenant data flows through the SessionRole) - heartbeat liveness extended to lambda-microvm (exhaustive per-backend switch; agentcore byte-identical, ecs excluded; RUNNING-scoped so P3 suspends stay immune) - closes the hung-in-guest-pipeline blind spot behind a healthy substrate - hooks declared to match what the agent serves: ready+validate (image, 60s) and run+terminate (runtime, 60s/15s), with both-direction exact-set tests; packaging script hooks JSON updated Contracts: microvm_platform_config in contracts/constants.json is the single source of truth; both the agent and the CDK strategy derive from it and check-constants-sync validates shape and forbids literal re-declarations. Docs: ADR-021 amended in place (canonical wire shapes, per-phase hook table, platform-config delivery, dual-signal liveness); COMPUTE.md gains the Lambda MicroVMs column + classic-Lambda distinction (aws-samples#645 acceptance criterion) and backend overview; ORCHESTRATOR.md gains the dual-signal liveness and lifecycle sections; SECURITY.md covers all three compute roles. Remaining for P2 completion: the live smoke run (clone -> change -> PR with bgagent watch) and the deferred empirical items (suspend TTL >1h, SUSPENDED-vs-quota, microvmImageHooks API spelling, NO_INGRESS ARN) - tracked on aws-samples#645. Verification: cdk 3825, agent 1590 (coverage 82.8%), cli untouched; tsc/eslint/ruff/ty/vulture clean; drift-prevention (constants-sync, types-sync, pins) green; docs sync idempotent; docs:check clean. security:sast runs in CI (semgrep unavailable locally). Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
'sk-ant-secret' pattern-matches Anthropic key detectors (Code Defender warned; CI gitleaks could fail). Replaced with a non-matching dummy; test intent unchanged (secret values must not leak into session-start logs). Full P2 diff swept for other detector-pattern lookalikes: none. gitleaks local scan clean. Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
…#645) The Stage D live smoke run (evidence: aws-samples#645 thread) failed at turn 0 and surfaced five live-contract defects invisible to synth, unit tests, and cdk-nag. This lands all corrections: - P2-F1/F3: drop aws:SourceAccount from all MicroVM-facing role trust policies - the service presents no source key when assuming them (deterministic connector CREATE_FAILED; misleading caller- side PassRole denial proven by elimination). ADR security table states the limitation honestly with per-role passer accounting (CloudFormation passes build/operator roles; only the orchestrator passes the execution role) and the qualified Resource:* exceptions - P2-F5: cold 225 MiB claude binary killed every task (10s version probe vs lazy snapshot hydration). /ready now warms claude (required, 120s) then git/node (optional, shared 240s ceiling inside the 300s hook budget; hung optional can never starve the snapshot); runner probe raised to 60s. Fake-clock budget tests - P2-F2: CFN enforces API enums at change-set time, refuting the string-shape reasoning - ARM_64 + ENABLED hook states; routes live only in MICROVM_AGENT_HOOK_ROUTES; negative test keeps route strings out of the image resource; new drift-guard test diffs the script's actual flags against the synthesized template - P2-F4: execution role granted CreateLogStream/PutLogEvents on the application log group whose name travels in platform_config - P2-F6: ADR corrected - the service reaps run-hook FAILURES (4xx -> ~12s terminate); active terminate retained for success paths (after a 200 the service has no view of the guest) - P2-F8: empty terminate-hook microvmId is expected-normal; artifacts/trace bucket sameness documented as intentional Fixed-but-not-re-exercised: the CDK-managed image path and the warm-up's effect on snapshot warmth are proven against the run's verbatim errors; the follow-up live re-run converts them. cdk 3832, agent 1609, cli 736; build/drift-prevention/docs green. Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
Clears GHSA-5p4m-2wfm-xmqj (High) flagged by osv-scanner. The root resolutions range ^4.2.0 already admits 4.3.1; minimal single-entry lock refresh under the CI toolchain (Node 22.23.2 + Yarn 1.22.22), byte-identical across repeated installs. osv clean; cdk 3832 and cli 736 green. Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #733 +/- ##
=======================================
Coverage ? 91.83%
=======================================
Files ? 292
Lines ? 81935
Branches ? 8932
=======================================
Hits ? 75244
Misses ? 6691
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…-samples#645) The Stage D-redux smoke run achieved the P2 acceptance criterion (clone -> change -> PR on the lambda-microvm backend: dreamorosi/batch-sync-triage#6, 153s, $0.28, 12 turns) and converted four of five Stage E fixes live. It also disproved run 1's exoneration of the identity-side PassRole condition via a controlled two-arm experiment - run 1's control was contaminated by its own temporary unconditioned grant. This lands the residual fixes: - MicrovmPassExecutionRole: drop iam:PassedToService (denied on the RunMicrovm path; two-arm evidence in the reversed comment); the exact execution-role ARN remains the scoping. Tests assert no Condition, exact ARN, no wildcard - Bootstrap 1.4.0: new MicrovmPassRoles statement in the backend-conditional compute-lambda-microvm policy so CloudFormation can pass the build/connector-operator roles for the CDK-managed image path (three-leg evidence: byte-identical live policy, simulate allowed-with/implicitDeny-without, same-role out-of-band control). Execution role deliberately excluded; infrastructure's allowlisted IAMPassRole untouched and now test-pinned. Golden DEPLOYMENT_ROLES block + re-bootstrap callout; operators must re-bootstrap to >= 1.4.0 - agent_heartbeat_at now projected through toTaskDetail and shown in bgagent status/detail renderers (its absence caused run 1's wrong liveness conclusion while DynamoDB held a 6s-old value) - ADR: per-role passer accounting (orchestrator passes the execution role in practice; the deploy role's prefix grant technically matches it), smoke status corrected to record the passing run and the two fixes still awaiting live re-exercise, template-size consequence at 98.6% (aws-samples#735) Honest residuals: P2r2-F9/F10 fixes are evidence-based but not yet re-exercised live; suspend TTL beyond 1h and SUSPENDED-vs-quota remain open (AWS-side observability gaps). cdk 3841, cli 745, agent 1609; build/drift-prevention/bootstrap determinism/docs all green. Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
Two branches: the exhaustive unknown-compute-type guard in the heartbeat liveness check (rejects rather than bypassing), and run_agent's wiring of the extracted claude version probe. Patch coverage 100% on both files. cdk 3842, agent 1610. Refs aws-samples#645 Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Phase P2 of ADR-021 short of the live smoke run: the agent is now fully launchable and observable on the
lambda-microvmbackend. Follows #689 (P1). Honest scope note: the clone→change→PR smoke run itself (plus the deferred empirical items: suspend TTL >1h, SUSPENDED-vs-quota,microvmImageHooksAPI spelling,NO_INGRESSARN) executes against a live account after this lands and is tracked on #645 — this PR is the foundations, not the completion claim.Platform config travels with the task, not the image
MicroVM env vars are image-version-frozen, and external images can't receive stack env at all — so deployment identifiers (table names, secret ARNs, session-role ARN; 13-key allowlist, 4 required) now ride the
/runenvelope as aplatform_configblock. The agent installs only allowlisted keys into its environment before any credential/pipeline initialization and fails closed on unknown keys (env installation from a network payload = injection surface). The allowlist lives once incontracts/constants.json; both the agent and the CDK strategy derive from it andcheck-constants-syncforbids literal re-declarations. Canonical wire shapes are documented in ADR-021 §3.Snapshot credential hygiene (real defect found and fixed)
/readywas spinning up the CloudWatch debug writer, which resolved a credential chain and pinned the build-time region intoboto3.DEFAULT_SESSION— state the image snapshot would replay into every MicroVM. Build hooks and pre-install/runlogging are now stdout-only; a poisoned-seam test suite (every AWS/credential entry point armed to throw) plus a subprocess regression test lock the property, mutation-checked.Full hook set + IAM parity + dual-signal liveness
The image now declares exactly what the agent serves:
ready+validate(image hooks;/validatemakes zero AWS calls — it runs under the build role) andrun+terminate(runtime;/terminatereturns 200 for any body and never writes terminal task status — the orchestrator owns terminal state). The execution role gains feature-based runtime parity (GitHub PAT + channel-OAuth secrets, scoped Bedrock, Memory grantReadWrite, AZ describe — still no direct DynamoDB; tenant data flows through the SessionRole). Heartbeat-staleness liveness now coverslambda-microvm(exhaustive per-backend switch; agentcore byte-identical, ECS excluded, RUNNING-scoped so P3 suspends stay immune) — closing the hung-pipeline-behind-healthy-substrate blind spot.Docs
COMPUTE.md gains the Lambda MicroVMs column + explicit classic-Lambda distinction (the #645 acceptance criterion) with live-verified values only; ORCHESTRATOR.md gains the dual-signal liveness + lifecycle sections; SECURITY.md covers all three compute roles; ADR-021 amended in place (
proposed).Verification: cdk 3 825 / agent 1 590 (82.8% cov) / cli untouched; tsc, eslint, ruff/ty/vulture clean; drift-prevention green; docs sync idempotent + astro check clean; gitleaks clean.
security:sastruns in CI (semgrep unavailable locally).Refs #645