Skip to content

bootstrap-cli.md

Latest commit

 

History

History
428 lines (366 loc) · 24.1 KB

bootstrap-cli.md

File metadata and controls

428 lines (366 loc) · 24.1 KB
Raw

The bootstrap CLI (krops-bootstrap)

The imperative part of krops lives in one Rust binary under bootstrap-rs/. It implements the initial bootstrap, the default CAPI pivot into the self-managed management cluster, and teardown. After bootstrap and pivot finish, Flux owns the declared state until teardown.

The binary is a behavioral port of bootstrap.sh, pivot.sh, and teardown.sh. It preserves their step order, progress messages, environment interface, and safety guards, with two deliberate upgrades:

  • Reruns are safe by default. An existing healthy mgmt kind cluster is reused and each bootstrap or pivot step is idempotent. Pass --recreate to delete and rebuild the kind cluster instead.
  • Typed process execution. Tool arguments are passed as argv entries, secrets travel through stdin or the environment, and the GitHub and registry HTTP checks use reqwest with explicit timeouts.

Distribution and build

The primary distribution is the toolbox image, ghcr.io/polarsquad/krops-toolbox. It contains krops-bootstrap and the pinned tools required by the lifecycle. See Operations for the container invocation and host runtime contract.

The toolbox-release workflow runs on v* tags. It requires the tag to match bootstrap-rs/Cargo.toml, builds Linux amd64 and arm64 images each on a runner of that architecture, publishes X.Y.Z, X.Y, and stable latest tags, signs the image with GitHub OIDC, and attaches a Syft SPDX JSON SBOM attestation. The bootstrap-rs CI workflow also builds and smokes the arm64 image when its inputs change.

Each architecture builds natively rather than under QEMU: emulating the amd64 rustc binary on the arm64 runner segfaulted deterministically and blocked the v0.2.0 release (issue #254). The two single-arch images are merged into the multi-arch manifest with docker buildx imagetools create in plain bash rather than docker/metadata-action, after that action's tags output collapsed to a single tag on the v0.2.1 run and silently dropped the X.Y and latest aliases.

Verify a published image against the workflow's OIDC identity:

cosign verify ghcr.io/polarsquad/krops-toolbox:X.Y.Z \
  --certificate-identity-regexp \
    '^https://github.com/polarsquad/krops/.github/workflows/toolbox-release.yml@refs/tags/vX.Y.Z$' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com

Published releases carry the stable tags described above; build the current checkout as shown in Operations only for unreleased changes.

Build the CLI directly for native development:

cd bootstrap-rs
cargo build --locked
cd ..
./bootstrap-rs/target/debug/krops-bootstrap --help
./bootstrap-rs/target/debug/krops-bootstrap teardown --help

Run the binary from the repository root so its default ./bootstrap.toml path resolves. BOOTSTRAP_CONFIG selects another config path and works from any directory: the config's directory becomes the run's repository root for repository-owned files and mise tasks (see Repository configuration).

CI runs cargo fmt --check, clippy with warnings denied, a locked build, and the test suite. The toolchain is pinned in bootstrap-rs/rust-toolchain.toml; crate dependencies are locked in Cargo.lock.

Interface

krops-bootstrap [OPTIONS] [PROFILE] [COMMAND]
krops-bootstrap teardown [PROFILE]

Common examples:

krops-bootstrap                         # aws bootstrap, then pivot
krops-bootstrap local-host              # local-host bootstrap, then pivot
krops-bootstrap --recreate local-host   # rebuild the bootstrap kind cluster
krops-bootstrap teardown                # aws teardown
krops-bootstrap teardown local-host     # local-host teardown
  • PROFILE is the CLI's retained positional name. Its value names a section under [environments.*] in bootstrap.toml; the checked-in environments are those sections.
  • A non-empty KROPS_PROFILE overrides the positional profile. If neither is set, bootstrap.default-environment from bootstrap.toml is used.
  • --recreate applies to bootstrap only. Teardown is a subcommand and keeps its script-compatible controls in environment variables.
  • There is no pivot subcommand. Pivot is the default exit from bootstrap, and rerunning the normal command resumes an interrupted bootstrap or pivot.

Repository configuration

Repository-owned cluster names, paths, chart versions, provider manifests, and teardown targets live in bootstrap.toml. The binary retains generic fallback defaults and sequence-level contracts. It reads ./bootstrap.toml by default; BOOTSTRAP_CONFIG selects another path.

The loaded config's directory is the run's repository root (the file's parent directory, canonicalized). Repository-owned relative paths (provider-manifests, pivot-manifests, pivot-sops-secrets, and the move-fallbacks manifests) resolve against that root, and repository mise tasks (oci-push, aws-credentials, post-kind-create-task) run with it as their working directory; the binary never changes the process working directory, so selecting a checkout's config from another directory runs against that checkout (issue #260). A config stored outside a repository resolves those paths against its own directory, and the first use of a missing manifest fails naming the absolute resolved path. Caller-supplied paths keep their own semantics and are not re-rooted: AGE_KEY_FILE, a relative MGMT_KUBECONFIG, and KUBECONFIG resolve against the caller's working directory.

Runtime environment variables take precedence where an override exists. mise run validate parses the file and cross-checks its chart pins and teardown names against the Git manifests. Renovate updates the annotated chart pins together with their declarative counterparts. See Dependencies.

  • pivot-sops-secrets (optional, list): SOPS-encrypted manifests the pivot decrypts with SOPS_AGE_KEY_FILE (defaults to AGE_KEY_FILE) and applies to the target before clusterctl move. No checked-in environment uses it today; it is kept for environments whose moved objects reference a SOPS-encrypted Secret by name (see pivot-manifests as the secret-free replacement).
  • pivot-manifests (optional, list): plain (unencrypted) manifests applied to the target before clusterctl move, after the provider CRs. The workload-identity replacement for pivot-sops-secrets; used by azure for the secret-free aso-credentials Secret (issue #236). ${VAR} placeholders are substituted from the ConfigMaps in the Flux namespace of the bootstrap cluster (the values its Flux reconciled, e.g. azure-vars); the pivot fails naming any placeholder no ConfigMap provides.
  • post-kind-create-task (optional, string): name of a mise task in the active profile (mise.<env>.toml) run once the kind bootstrap cluster exists, on both the create and healthy-reuse paths; a non-zero exit aborts the bootstrap. azure uses it for Arc OIDC federation (issue #236).
  • teardown.manual (optional, string): when set, krops-bootstrap teardown refuses to run for that environment and prints the text. azure uses it until the live acceptance run defines the Azure orphan sweep.

Bootstrap and pivot controls

Variable Default Used by
BOOTSTRAP_CONFIG ./bootstrap.toml Repository configuration path; its directory is the repository root that repository-owned paths and mise tasks resolve against
KROPS_PROFILE positional profile, then bootstrap.default-environment (aws checked in) Environment selection
KROPS_KIND_CLUSTER config value mgmt Kind cluster name for the run; its kubeconfig context derives as kind-<name> unless BOOTSTRAP_KUBECONTEXT is set (concurrent runs on one host, #640)
REGISTRY_PORT 5001 Local-host registry host port
KROPS_REGISTRY_NAME config value krops-registry Local-host registry container name for the run (concurrent runs on one host, #640)
REGISTRY_READY_RETRIES 120 Local-host registry readiness attempts
LOCAL_RECONCILE_TIMEOUT 15m Local-host management and workload reconciliation waits
CONTAINER_ENGINE auto-detect Docker, then Podman kind and registry engine
GIT_REPO_URL required for GitHub-synced environments (aws, local-talos, azure, gcp) Management Flux Git source
GITHUB_TOKEN required for GitHub-synced environments (aws, local-talos, azure, gcp) PAT with read access to the repository
GITHUB_USER git Basic-auth username paired with the PAT
AGE_KEY_FILE age.agekey SOPS age private key loaded into sops-age; relative to the caller's working directory
AGE_PUBLIC_KEY derived from AGE_KEY_FILE Public key override during secret creation; must match the key file's public key when both are known (preflight fails fast on a mismatch)
OCI_REPOSITORY / OCI_TAG krops / latest Local-host OCI artifact name
BOOTSTRAP_PIVOT 1 Any value other than literal 1 skips pivot
MGMT_KUBECONFIG ~/.kube/krops-mgmt.yaml Exported management kubeconfig for native fallback runs; a relative value stays relative to the caller's working directory
MGMT_READY_TIMEOUT 40m for aws, 15m for local-host, 30m for local-talos (PXE install + first Talos boot) Management cluster definition and provisioning waits
MGMT_POLL_INTERVAL 10 seconds Management cluster definition, provisioning, and node-readiness polls
BOOTSTRAP_KUBECONTEXT kind-<KROPS_KIND_CLUSTER> when the cluster name is overridden, else the config value kind-mgmt Source context required by pivot; wins over the derived context
PIVOT_SKIP_DELETE 0 Literal 1 keeps kind after a successful pivot
KROPS_RUN_ID {profile}-{timestamp} Bootstrap/pivot run identifier for resource tagging; set to override auto-generation
KROPS_RUN_TTL 86400s (24h) Unsigned decimal seconds with optional s, m, or h (e.g. 3600, 2h); whitespace trimmed, empty defaults to 24h; none disables expiry
KROPS_REVISION Git branch HEAD SHA, else the forwarded KROPS_OCI_GIT_SHA checkout SHA, else unknown Git revision tag for resource tagging; auto-extracted from GitHub during bootstrap, else the checkout SHA the toolbox wrapper forwards
KROPS_RUN_KIND manual Bootstrap/pivot invocation kind for resource tagging (e.g. manual, scheduled, emergency)

The defaults above are the binary's generic fallbacks, not repository configuration: bootstrap.toml is the canonical source for the repository-owned values.

The pivot's target node-readiness wait uses a fixed 15m budget (MGMT_NODE_READY_TIMEOUT in pivot.sh and bootstrap-rs/src/main.rs). It is not an environment knob and is separate from MGMT_READY_TIMEOUT.

Concurrent runs on one host

Bootstrap runs on one host collide on the host-local names by default: the kind cluster (mgmt), its kubeconfig context (kind-mgmt), and the local-host registry container (krops-registry) all come from bootstrap.toml for every run. Two concurrent runs therefore target the same cluster and the same context, and the idempotent delete-before-create step makes them mutually destructive: whichever starts later deletes the other's in-flight management cluster mid-flight, orphaning whatever cloud resources the destroyed run's controllers had already reconciled (issue #640).

The CLI is multiplex-safe when each run uses distinct host-local names:

KROPS_KIND_CLUSTER=krops-a krops-bootstrap aws
KROPS_KIND_CLUSTER=krops-b krops-bootstrap gcp

KROPS_KIND_CLUSTER names the kind cluster for the run; its kubeconfig context follows kind's convention as kind-<name> (an explicit BOOTSTRAP_KUBECONTEXT still wins). A local-host run also separates the registry: KROPS_REGISTRY_NAME for the container and REGISTRY_PORT for the published host port. Pass the same values to the run's teardown and orphans invocations so they target the run's own cluster. A toolbox run receives the knobs through the container environment (raw docker run with explicit -e entries; the wrapper's allowlist does not forward them, see Operations).

The legacy shell scripts (bootstrap.sh, teardown.sh, pivot.sh) keep the fixed mgmt name and are not multiplex-safe: on a host that runs them, only one native bootstrap may be in flight at a time (the scripts are being retired in favor of the CLI, issues #92/#95/#100).

Toolbox runtime contracts

The toolbox runtime adds seven contracts:

  • KROPS_TOOLBOX=1 enables internal kind networking and disables host-only CAPD endpoint rewrites.
  • ENGINE_SOCK names the engine socket path as seen by the daemon. The wrapper resolves it for Docker Desktop, Docker contexts, rootful or rootless Podman, and podman machine.
  • KUBECONFIG must name one writable file, not a colon-separated list. The wrapper uses /workspace/.kube/kind.yaml.
  • CONTAINER_HOST points the Podman remote client at the mounted engine socket (unix:///var/run/docker.sock) when KROPS_TOOLBOX=1. The CLI sets it only if unset, before any Podman probe, so an operator-supplied value is kept regardless of the eventual CONTAINER_ENGINE.
  • AWS_PAGER is baked into the image as an empty string so the AWS CLI never pages its output through less (the toolbox is non-interactive and ships no pager). An operator-supplied value via -e AWS_PAGER=... overrides the default at runtime.
  • Every lifecycle aws invocation (the EIP quota preflight, the AWS orphan sweep, and the legacy shell scripts) carries MISE_ENV=aws scoped to the child process, so the in-image mise shim resolves the aws-cli pin from mise.aws.toml even though the parent process is not in the aws mise environment (issue #504); on native hosts the variable is inert for a plain aws binary. The preflight and the sweep probe aws --version before the first AWS call and fail with a message that names the shim, so a broken shim is never read as a credentials error or an empty listing.
  • Bare aws and clusterawsadm resolve image-wide through global mise bindings baked in at build time (mise -E aws use -g, with each pin read back from the aws env layer so it stays single-sourced in mise.aws.toml), because the checkout mounted at /workspace carries no aws env layer for a bare shim to resolve against (issue #635). A build-time smoke check runs both tools bare and fails the build if shim resolution regresses; lifecycle invocations still scope MISE_ENV=aws explicitly as above (issue #504).

The container reaches the local registry at krops-registry:5000. Its /root/.kube mount makes the management kubeconfig persist on the host as ./.kube/krops-mgmt.yaml. The wrapper's environment allowlist and override limitations are documented in Operations.

Design notes (bootstrap-rs/src/engine.rs): engine detection, socket resolution, CONTAINER_HOST defaulting, and kind-network attach/detach used to be implemented separately in preflight_checks, a detect_engine_for_network helper that read Config defaults instead of the engine preflight_checks actually resolved, teardown::detect_engine, and the toolbox shell entrypoint (which also detected and validated the engine before Rust ever ran). All of that now lives in one module: preflight_checks resolves the engine once and threads it through bootstrap, pivot, and teardown instead of re-detecting it, and toolbox-entrypoint.sh only sets KROPS_TOOLBOX=1 and execs. Inside the toolbox, podman info reports the mounted client socket (CONTAINER_HOST), not the daemon-side path kind's extraMounts need, so the module skips querying it there and uses the static fallback instead. kind-network attach/detach checks actual network membership rather than matching Docker's and Podman's differently worded "already connected" errors, so it doesn't depend on engine- or version-specific error text.

What bootstrap and pivot do

  1. Preflight: validate the environment and required tools, select a running container engine, and perform the GitHub token/age-key checks for GitHub-synced environments (aws, local-talos, azure, gcp). A fallback native run requires kind, helm, kubectl, clusterctl, and mise; OCI-synced environments (local-host) also require flux and curl. The aws profile additionally requires aws CLI on PATH. For the aws profile, preflight checks the EC2 EIP quota per region before any provisioning begins. The Rust CLI derives regions dynamically from bootstrap.toml; the shell scripts use a hardcoded region list and must be kept in sync manually if new AWS regions are added, making the Rust CLI the recommended path.
  2. Bootstrap kind: create or reuse mgmt, start the local registry for local-host, install the Flux Operator, create the Git and SOPS secrets (GitHub-synced environments) or publish the local OCI artifact, install the FluxInstance, and watch reconciliation.
  3. Pivot by default: wait for the Flux-created management Cluster definition (a clean first run polls through the reconciliation chain and surfaces failed Kustomizations on timeout), wait for the CAPI-managed management cluster, export its kubeconfig, wait for the target nodes (the poll tolerates a nodeless EKS start), install cert-manager, the CAPI operator, and provider CRs at the versions declared in bootstrap.toml, suspend Flux in kind, run clusterctl move, unpause the moved clusters, seed Flux on the target, and delete kind after the safety checks pass.

If a phase fails, fix the cause and rerun the same command. clusterctl move is re-runnable, and kind remains authoritative until the final deletion. Recovery details and the warning against deleting moved CAPI objects are in Pivot recovery.

Teardown controls and behavior

krops-bootstrap teardown [PROFILE]
Variable Default Effect
AWS_ONLY 0 Literal 1 skips Kubernetes steps and runs only the AWS orphan sweep; invalid with local-host and local-talos
FORCE_KIND_DELETE 0 Literal 1 removes the controller host even when CAPI cluster deletion was not confirmed
CLUSTER_DELETE_TIMEOUT 1200 seconds CAPI cluster deletion wait (aws workloads, local-talos management)
PROVIDER_DELETE_TIMEOUT 300 seconds CAPI provider deletion wait
MGMT_KUBECONFIG ~/.kube/krops-mgmt.yaml Post-pivot controller-host kubeconfig

Teardown checks required tools before mutation:

Mode Required tools
local-host kind, kubectl; AWS_ONLY=1 is rejected
local-talos kind, kubectl; AWS_ONLY=1 is rejected
normal aws kind, helm, kubectl, xargs; AWS CLI is optional and its absence skips the orphan sweep
AWS_ONLY=1 AWS CLI only

Teardown discovers where the CAPI controllers run: the pre-pivot kind cluster, the post-pivot self-managed management cluster, or no reachable cluster. In the toolbox, the aws path first joins the kind network so the pre-pivot kind endpoint resolves before discovery. The post-pivot local-host path joins the kind network as well whenever it still exists, because the self-managed management cluster's endpoint (the CAPD load balancer) is reachable only from that network. When no cluster is reachable and the run was not AWS_ONLY=1, teardown still runs the AWS orphan sweep (the recovery for a host that is already gone) but exits non-zero so automation does not read the skipped Kubernetes side as success; AWS_ONLY=1 is the explicit exit-0 recovery. It then preserves the shell implementation's reverse-order and best-effort cleanup semantics.

Teardown binds every kubectl call to that discovered target (an explicit kind context or the management kubeconfig; never the operator's current context) and treats a failed kubectl query as an unknown state, never as "nothing left to delete". A failed listing or lookup (auth, forbidden, API outage) therefore aborts the teardown with a nonzero exit and leaves the management cluster and its controllers intact, so in-flight CAPA/CABPT deprovisioning can continue; re-run it once the query works. A successful empty listing, a named lookup reporting NotFound, or the API server reporting the resource type as not installed is the only evidence accepted as "confirmed gone" by the deletion guard, which is what allows the management cluster to be removed.

  • local-host: suspend the workload Kustomization, delete the CAPD workload cluster and wait for its containers to disappear, remove kind or the self-managed management containers, then remove the local registry.
  • local-talos: suspend Flux and delete every CAPI Cluster, the management cluster included; the deletion IS the release, and CAPT returns the machine's Hardware to the Tinkerbell pool. The machine is never wiped: it keeps running Talos for the operator. AWS_ONLY=1 is rejected because there is no AWS orphan sweep for operator-owned hardware.
  • aws: suspend Flux, delete and wait for workload CAPI clusters, run the AWS orphan sweep for both workloads and the self-managed management cluster, remove CAPI providers and bootstrap Helm releases when the controller host is still reachable, and enforce the controller-host deletion guard. The sweep covers nodegroups, EKS clusters, RDS, CAPA-tagged VPC resources, versioned S3 buckets, IAM roles and users, and the clusterawsadm CloudFormation stack.

AWS_ONLY=1 is the recovery path when only AWS cleanup remains. A missing tool fails preflight before mutation; in the normal AWS path, a missing AWS CLI is reported and the orphan sweep is skipped rather than misclassified as an empty account.

Orphan discovery (read-only)

krops-bootstrap orphans [PROFILE]

Discovers unowned AWS resources without deleting anything. Supported profiles: aws only (local-host, local-talos, azure, gcp are refused with a clear error).

Variable Default Effect
ORPHAN_MIN_AGE_HOURS 6 Minimum age in hours; resources younger than this are "recent" not "orphaned"
ORPHAN_REPORT_JSON unset Optional path to write JSON report

Exit code is 0 when a report is produced (whether orphans are found or not); nonzero when a query failed or the profile is unsupported.

The report scans EKS clusters and nodegroups, RDS instances, VPCs with CAPA ownership tags, NAT gateways, Elastic IPs, and S3 buckets matching cluster name patterns. Markdown is printed to stdout and appended to $GITHUB_STEP_SUMMARY (if set). The report is read-only: see the teardown section for cleanup.

IAM roles/users and CloudFormation bootstrap stacks are not scanned (usually free resources).

Entry-point and parity status

scripts/toolbox-run.sh bootstrap, scripts/toolbox-run.sh pivot, and scripts/toolbox-run.sh teardown invoke this CLI in the toolbox container. The pivot wrapper verb is a named resume path for the rerun-safe default lifecycle; it does not select a separate CLI subcommand.

The three shell scripts remain as native reference and fallback paths until full parity runs pass for all environments. Local-host bootstrap, pivot, and post-pivot teardown have completed parity runs. AWS full-parity runs still gate script retirement. The local-talos environment has completed its hardware acceptance run (issue #105, closed) through the CLI; the documented PXE/Tinkerbell-Workflow provisioning transport still needs a run (issue #225). Toolbox releases are published (see above); no Podman-host acceptance run is recorded.

Run metadata is validated before the FluxInstance activates reconciliation. Pivot copies all four nonempty data keys exactly; bootstrap regenerates metadata only when a rerun ConfigMap is absent, and propagates read or decoding failures. Rust discovers the branch SHA for the revision default, then falls back to the KROPS_OCI_GIT_SHA checkout SHA the toolbox wrapper forwards, then unknown; shell uses unknown. Both accept KROPS_REVISION. TTL arithmetic and RFC3339 expiry are checked for overflow (maximum year 9999).